Evaluation of an Internet-based PgDL (CPE) Course

John Fairhurst
PgDL (CPE) Course Director
Department of Law
University of Huddersfield
j.fairhurst@hud.ac.uk
 

Abstract

The University of Huddersfield was the first institution to be validated to deliver a Postgraduate Diploma in Law (Common Professional Examination) course for delivery by open and distance learning, using the Internet as the principal method of delivery. The course came on-stream for the 1998/99 academic year and incorporated student interactivity and collaborative learning as one of its prime features. This article evaluates the strengths and weaknesses of the course (including a comparative analysis of students on this course with those on the conventional face-to-face part-time course), and offers an insight into the future delivery of internet-based law courses at both postgraduate and undergraduate level.

This is a Refereed Article published on 29 October 1999.

Citation: Fairhurst J, 'Evaluation of an Internet-based PgDL (CPE) Course', 1999 (3) The Journal of Information, Law and Technology (JILT). <http://elj.warwick.ac.uk/jilt/99-3/fa irhurst.html>. New citation as at 1/1/04: < http://www2.warwick.ac.uk/fac/soc/law/elj/jilt/ 1999_3/fairhurst/>

1. Introduction

The Department of Law at the University of Huddersfield has delivered a Postgraduate Diploma (Common Professional Examination) law-conversion course for a number of years in both full-time and part-time conventional face-to-face modes. The course was delivered part-time over two years by open and distance learning from September 1998 (having been validated by the CPE Board (which represents the two branches of the legal profession)). In order to take advantage of the advances in communication and information technology, and to offer an innovative course which could support student collaborative learning, it was decided to offer the course over the internet; the only such course to be delivered principally by this method.

The open and distance learning ('ODL') course is made up of various components: face-to-face tuition, lecture materials, tutorial submission and feedback, email discussion groups, and Subject Team email facility.

In its first year of operation it was decided to subject both the learning materials and the method of delivery to an evaluation, to ascertain their strengths and weaknesses, and where relevant to undertake a comparative analysis with students on Year One of the conventional part-time face-to-face course.

2. Background Information

2.1 Students on the Course

It is a condition of eligibility (imposed by the CPE Board) that a student is a graduate from a UK or Irish University or is accepted by one of the professional bodies as having a degree-equivalent qualification or as a mature student.

Given the nature of the course, there is a minimum requirement of basic Windows-based word-processing skills, together with a familiarity of the Internet and email system. Offers made are subject to an applicant satisfying this requirement.

Thirteen students initially enrolled on the ODL course (nine males and four females) in September 1998, but only eight completed the academic year. Of these eight, four were male and four were female. All eight who completed the first year of the course had a minimum of a 2ii degree classification (four had a 2i) and three had a higher degree or professional qualification. Subject areas were equally divided between the sciences and arts. One student had studied law as part of her undergraduate studies. All eight students were in full-time employment and two were raising families, one as a single parent.

This evaluation involves a comparative analysis of students on the ODL course with those on the conventional face-to-face part-time course. Fourteen students initially enrolled on Year One of the conventional part-time course in September 1998 (eight males and six females). Thirteen students (eight male and five female) completed the academic year, of whom one had a first class degree, ten had a 2ii, and two had been issued with a Certificate of Academic Standing from the Law Society on the basis of having degree-equivalent qualifications (one a Russian degree in Librarianship and Bibliography, and the other the English National Board's award for Nursing, Midwifery and Health Visiting). Nine of the thirteen had an arts background, with the remaining four a science. Two students had studied law at undergraduate level. All thirteen were in full-time employment.

2.2 Course Components (Year One)

2.2.1 Programme of Preliminary Study and the Introductory Programme

Prior to the face-to-face introductory programme (see below), students were given a programme of preliminary study which contained some elementary exercises to assist them in their study of law. This had to be completed prior to the introductory programme. Access to a computer and the Internet was not required at this stage. Upon enrolment students attended a compulsory face-to-face six-day introductory programme, the aim of which was to introduce them to the techniques of the study of law, legal skills and the English Legal System.

2.2.2 The Substantive Subjects

Following the introductory program, students commenced their study of the substantive subjects. In the first year these were:

• Criminal Law

• European Community Law

• Law of Contract

• Public Law

2.2.3 Teaching Strategy

Formal tuition for each of the four subjects was provided on the Internet.

Figure 1. The ODL Course's Homepage

Units

Each of the four subjects was divided into self-contained study units. A time period (i.e. start and end date) was laid down for the study of a specific unit.

Lecture materials (in Microsoft Word format) were prepared by the individual Subject Leader for the study of each unit.

Figure 2. Index of European Community Law

Interactive Tutorials and Collaborative Learning

At the end of each unit, students were required to complete a tutorial, applying their knowledge to a series of questions and case studies.

To assist students to complete the tutorial an email discussion group was available for each subject the aim of which was to enable students to discuss any particular problems with their peers. Emails were archived using Hypermail to enable students to review past contributions with ease.

At the outset, the ODL Course Team had sought to include an asynchronous bulletin board and a synchronous (i.e. real-time) chat room, but the University's Central Computing Services' policy was such as to exclude these facilities.

The purpose of the email discussion groups was to facilitate collaborative learning between students. Empirical studies have indicated that collaborative learning is superior to traditional learning methods and there is:

Although the aim of the discussion group was to promote collaborative learning between students, the activities of each discussion group were reviewed periodically by the Subject Team to give guidance to the group, as appropriate.

Students had the facility to email the Subject Team if they had a particular problem; there was a target response time of three working days.

At the end-date for completion of a unit, students were required to forward tutorial answers in Microsoft Word format, by email, to the Subject Team for marking and feedback. There was a target response time for providing feedback of ten working days.

Students could utilise the subject discussion groups and individual Subject Team emails during the revision period prior to their end-of-year examinations.

Face-to-face Tuition

The CPE Board requires a minimum of twelve days tutor/student face-to-face contact each year for distance learning courses, attendance at which is compulsory. The introductory programme at the start of Year One consisted of six days of face-to-face tuition leaving six further face-to-face tuition days to be scheduled over the academic year. Face-to-face study weekends (Saturday-Sunday) were held in November 1998, January 1999 and May 1999. Time was equally divided between each of the four subjects.

Assessment

Students completed one piece of coursework in each of the four subjects during the academic year (which accounted for 25% of the aggregate mark for the subject) at the end of which there was a three-hour unseen examination in each subject (which accounted for the remaining 75% of the aggregate mark for the subject).

Monitoring Mechanisms

The course was designed to include in-built monitoring mechanisms to ensure that tutors were complying with their obligations and also to identify any students who were struggling with the course or who were failing to submit tutorial work and courseworks.

2.3 Staff Involved in the Course

The Subject Team for each of the four subjects comprised of two full-time tutors who were involved in delivery of the conventionally taught modes. One member of each Subject Team was allocated sole responsibility for preparation of the unit materials and was designated the title 'Subject Leader'. As this was a new venture for the Department of Law and generally for the University, there was no knowledge as to how much time tutors would be engaged in course delivery. The Head of Law credited each member of a Subject Team with a 26-hour allowance on their individual timetable; no extra allowance was given to Subject Leaders.

The Course Director (the author of this article), assisted by the School's technician and the School's Central Computing Services' representative, attended to web-design and updating issues. The Course Director was solely responsible for all course administration.

2.4 The Conventional Part-time Course

The conventional face-to-face part-time course requires attendance at the University on two evenings each week for 3.25 hours each evening. Students on the conventional and ODL modes study the same four subjects during their first year. Both modes were delivered during the 1998/99 academic year, which allowed a comparative analysis to be undertaken between students on the two modes. This will be discussed further below.

3. Description of the Evaluation Study

3.1 What is Evaluation?

Thorpe has defined evaluation as:

3.2 The Program

Herman et al [( 1987 ) p.8] uses the term 'program' for the evaluation of something 'tangible' and/or 'intangible'.

For the purpose of this evaluation, the program comprises of:

• Tangible assets in the form of unit and tutorial materials (i.e. the learning materials); and

• Intangible assets in the form of the innovative mode of delivery (i.e. over the internet using email discussion groups; email questions to Subject Teams etc., as discussed above at section 2.2.3 ).

The evaluation is of the whole program i.e. the learning materials and the mode of delivery.

3.3 Types of Evaluation

There are two types of evaluation:

• Formative

• Summative

A 'formative' evaluation is concerned with an evaluation of the product throughout its developmental stages to ascertain if any revisions are necessary before the product is actually completed [see Tessmer (1993) p. 12].

A 'summative' evaluation, by comparison, takes place after the product has been completed and focuses on whether or not the product can deliver. It is usually comparative in nature, and seeks to ascertain whether the program (i.e. mode of delivery, as well as the learning materials) is superior to some other form of program (in terms of mode of delivery and learning materials).

3.3.1 Formative Evaluation or Summative Evaluation?

The purpose of conducting the program evaluation was two-fold:

• In its first year of operation, because it had not previously been subjected to a planned or structured evaluation, it was necessary to identify its strengths and weaknesses and to ascertain what revisions were necessary to rectify any weaknesses. This is a formative evaluation, even though the program had been completed [see Tessmer (1993) p. 14].
   

• To ascertain whether the program delivers. Did the initial intake of students complete the academic year (i.e. what is the drop-out rate?) and of those who completed the year how many successfully completed their assessments (coursework and examination in each of the four substantive subjects)? A comparative analysis of students on the ODL course with those on the conventional face-to-face part-time course could establish if there were any marked differences between the retention rate and the performance of students on the two modes. This is a summative evaluation in so far as it is an 'after-the-event' study, not really concerned with identifying revisions which are necessary.

3.4 What are the Evaluation Goals?

3.4.1 Improve the Program's Effectiveness

Tessmer has suggested that the main goal of a formative evaluation is to 'improve the effectiveness of the instruction [i.e. the program]' [ Tessmer (1993) p. 26].

With a particular reference to Computer Assisted Learning Programs, Jones A et al stated that an evaluation should enable '... the important issues regarding design, fitness for purpose, time and resource allocation and the process of development to emerge' [ Jones A et al (1998) p. 10].

This 'improvement of the program's effectiveness' goal could be subdivided into four sub-goals:

• Identify any deficiencies in the learning effectiveness of the product - where did the learners fail to learn?

• This required an evaluation of the learning effectiveness of the materials (i.e. lecture notes/directed reading and the tutorials), together with the mode of delivery (i.e. on the Internet using email discussion groups, email queries to Subject Teams etc.). Although formative in nature (because the purpose of the exercise was to ascertain what revisions, if any, were necessary) it could also fall within the remit of a summative evaluation to the extent that there would be an examination of student retention and whether or not students were successful in their assessments (see subsidiary goals below).

  If students who enrolled did not complete the academic year, or were unsuccessful in the coursework and examination assessments in each of the four subjects (a summative evaluation, as discussed above), then the question that would need to be addressed is why not? Was it a failing in the mode of delivery, or the learning materials? It could be that a student who was unsuccessful had failed to get involved in the email discussion groups the purpose of which was to support collaborative learning between students. Each of the elements which formed part of the educational experience would have to be taken into account to ascertain which students were, and which were not, fully taking part, when analysing the students' final performance.

  A procedural task analysis of the learning cycle for students on the course was as follows:

• Read unit materials and any directed reading.

• Start the tutorial.

• Make submissions to, and read and reply to submissions from, the email discussion group.

• Submit queries to the Subject Team.

• Submit the tutorial to the Subject Team.

• Receive feedback from the Subject Team.

• The only entry skills required were basic word-processing skills, together with a familiarity of the Internet and email. There were no pre-requisites for completion of a unit. Academically, students on the program were graduates (or had been approved by the Law Society or General Council of the Bar) and were therefore proven to have the academic ability to undertake the program, although there would be different levels of academic competence within the student body.

• Locate any ease of use problems

• This sub-goal concerned all 'users' of the program and therefore included both tutors and students. It involved issues concerning: the ease in accessing the program over the internet and navigating through it; the ease of using the email discussion groups and email questions to Subject Teams; the ease in accessing the lecture materials and tutorial questions; the student's ease in submitting tutorials and the tutor's ease in returning feedback to the student. Ease of use problems could affect the user's interest and motivation.

  The learning environment for tutors was the University's Department of Law, accessing the product via hardware and software in staff offices. Although the software used was standard, hardware specifications varied. The learning environment for students was either on-campus (for open learning students) using the University's hardware and software, or off-campus (for distance learning students) using their own hardware and software. Given the vast range of hardware and software specifications, problems could arise which might detract users from their learning experience.

• Evaluate the efficiency of the instruction

• This sub-goal related to the length of time tutors and students were engaged in completing the tasks and comparing it with that spent by tutors and students on the conventional face-to-face part-time course.

• Analyse instructional strengths

• In addition to identifying the program's weaknesses one of the aims of the evaluation was to identify its strengths. This could assist in rectifying any weaknesses within the program and could also be instructive in extending the mode of delivery to other areas (e.g. to undergraduate law modules on the LLB).

  In addition to this goal (and the four sub-goals) there were some subsidiary (or secondary) goals:

3.4.2 Comparative Analysis

As discussed above, one of the goals of the evaluation was to ascertain whether or not the program's mode of delivery had any marked difference on retention rates and ultimate success in student assessment compared to students on the conventional face-to-face part-time mode.

Rowntree [( 1992 ) p. 209] states that:

3.4.3 Miscellaneous

• Given that this course was the first of its kind to be validated by the CPE Board and that it was one of the first of its kind (in terms of mode of delivery) at the University of Huddersfield, there was an underlying 'political' goal of the evaluation: to provide these bodies with the necessary evidence to enable them to decide upon the educational effectiveness of the program.
   

• The evaluation sought to ascertain the workload of tutors in delivering the program, and how this compared to that on the conventional face-to-face part-time course, so that informed decisions could be made upon the cost-effectiveness of the program and to ensure that the workload of tutors was clearly reflected in allowances on their individual timetables.
   

• The requirement for twelve days face-to-face tuition per year is imposed by the CPE Board. This requirement impacted upon the recruitment of overseas students. The professional bodies do not impose a face-to-face requirement in respect of undergraduate distance learning law courses. Given the nature of this program, with continuous interaction through email discussion groups, the evaluation sought to establish whether or not there were sound educational reasons for making face-to-face tuition a mandatory requirement.

3.5 Information Sought from the Evaluation

The goals discussed above were formulated into a series of questions:

• Are there any differences in student retention rates and ultimate success in assessments between students on this program compared with those on the conventional face-to-face part-time course? If so, why?
   

• What are the weaknesses and strengths of the design of the program's website?
   

• What are the weaknesses and strengths of the learning materials (i.e. unit materials and/or directed reading)?
   

• What are the weaknesses and strengths of the email discussion group and the facility to email queries to the Subject Team?
   

• Does the email discussion group facilitate collaborative learning?
   

• What are the weaknesses and strengths of the tutorial submission and subsequent tutor feedback?
   

• Does a user's learning environment affect the ease in which they can use the program? If so, does this have an effect on the user's interest and motivation?
   

• Is the time in which it takes the student and tutor to complete a unit on this program different to that on the conventional face-to-face part-time course?
   

• Is face-to-face tuition necessary as a compulsory element of the program?

3.6 Evaluation Design

3.6.1 Data Gathering Methods and Tools

Two strategies could be employed for the kind of data that was to be collected:

• Quantitative

• Qualitative

In essence, the quantitative approach would seek to amass statistical data (usually from a large number of users) in the form of closed questions (e.g. questionnaires) and analysis of assessment results. There would be no mechanism for collating data which took into account the individual views of users which did not otherwise fit within the range of answers selected by the evaluator. Although the planning of an evaluation which employed a quantitative approach could be time-consuming, the time expended in the collection of the data would be minimal.

By comparison, the qualitative method would be more selective, involving a more in-depth and detailed approach. This approach could involve users being asked open questions, so that they could state their own views rather than have to select from a range of fixed responses (as is the case with closed questions). This form of data collection could be undertaken by, inter alia , questionnaire or one-to-one interview. This method could be time-consuming, especially if the collection of data involved individual face-to-face interviews. There would also be a greater demand on time when analysing the data, which by its very nature would not fit into a standard template of fixed responses.

Neither strategy has been definitively proven to be superior to the other and therefore in the evaluation of this program it was decided to employ both strategies through the use of questionnaires and selective individual one-to-one interviews. This would be similar to the approach adopted by the Open University [ Jones et al (1996) pp. 5-15], which Oliver et al has stated:

3.6.2 Questionnaires

All students on the ODL course were provided with a file containing a set of questionnaires which they were required to complete at various stages throughout the course. A user profile at the front of the file recorded details of the student's age, sex, academic and C&IT ability, and learning environment. The purpose of this was to enable these characteristics to be taken into account, where appropriate, when analysing the data. Students were provided with instructions to assist them in completing the questionnaires. The questionnaires were completed throughout the course, as follows:

• At the end of the face-to-face introductory course.

• At the end of each unit in each of the four substantive subjects.

• At the end of each of the three two-day face-to-face study weekends.

• At the end of the academic year.

In addition, all students on the first year of the conventional face-to-face part-time course were provided with a file which contained a set of questionnaires which they were required to complete at the end of each unit in EC Law and Public Law. A user profile, stored at the front of the file, recorded the individual student's sex, age, and academic qualifications. Students were provided with instructions to assist them in completing the questionnaires. Students completed a further questionnaire at the end of the academic year, which summarised the end-of-unit data.

EC Law and Public Law on both the conventional face-to-face and ODL courses were taught by the Course Director (i.e. the author). Students used the same lecture and tutorial materials. The purpose of this was to enable a comparative analysis to be conducted between students on the ODL course with those on the conventional face-to-face part-time course.

Tutors involved in course delivery were required to complete a log at the end of each unit to record any 'ease of use' difficulties, time expended by tutors in delivering the unit, and to ensure that the monitoring mechanisms (see section 2.2.3 above) were being complied with. This log also recorded students' tutorial success or failure, together with the individual involvement of students in the discussion group and questions to the Subject Team to ensure that full use was made of the facilities.

The questionnaires employed both quantitative and qualitative methods, asking closed and open questions. It was possible to employ both methods, given the relatively small size of students undertaking the program.

3.6.3 Interviews

As discussed above, interviewing students on a one-to-one basis could be a time-consuming affair. For this reason it was necessary to be selective and to recognise that it would not be possible to interview all students on the program. Tessmer [( 1993 ) p. 76] has suggested that three students should be the minimum number to use in one-to-one interviews, although he recognised that because of time-constraints, it may only be practicable to use one or two students. Students selected should be representative of the target population and have varying ability levels.

As far as ability level is concerned, all students on the program were graduates. However, it was recognised that within this body of students on the program, there would be different levels of academic ability. Given the innovative mode of delivery, students were required to have a minimum level of C&IT skills, however the level of such skills would likewise vary within the body of students. It was initially planned to use three students for one-to-one interviews. They were chosen primarily on the basis of level of academic ability (low, medium and high), but also with a consideration of their C&IT competence. In selecting the students consideration was also given to representative criteria including sex and age.

The program was delivered to students on the basis that they had no prior knowledge of law. Any student with prior legal knowledge was not selected for a one-to-one interview.

The interviews were planned to take place as follows:

• At the end of the face-to-face introductory course

• At the end of each of the three two-day face-to-face study weekends

• At the end of the academic year

It was intended that the interviews would be structured primarily around the issues addressed in the questionnaires, but by their very nature it was anticipated they would be more wide-ranging.

However, two of the three students selected for one-to-one interviews subsequently withdrew from the course. One-to-one interviews were therefore abandoned, and given the small size of the student body, whole-class discussions were undertaken.

In addition, the ODL Student Representative (who had been elected by the ODL students) was interviewed in-depth on two occasions; first midway through the academic year (14 November 1998), and second towards the end of the year (8 May 1999).

4. Results and Discussion

The results and discussion within this section are centred around the series of questions, set out at section 3.5 above, which represent the information sought from the evaluation.

4.1 Outcome Issues

• Are there any differences in student retention rates and ultimate success in assessments between students on this program compared with those on the conventional face-to-face part-time course? If so, why?

Given the small group of students involved, the following data is not statistically significant and it simply represents an overview of the delivery of the ODL and conventional part-time courses during the 1998/99 academic year. Any underlying factors which could have impacted upon students' performance will be discussed.

4.1.1 Student Retention

As stated at section 2.1 above, thirteen students enrolled on the ODL course and fourteen on the conventional part-time course. Of these students, five withdrew from the ODL course and one from the conventional part-time course.

Students contemplating withdrawal were counselled by the Course Director and reasons for their subsequent withdrawal were discussed in detail.

The five students who withdrew from the ODL course did so for the following reasons:

• A proposed move back to the UK from the USA had been delayed.
   

• Full-time employment was obtained within the voluntary sector (the student having previously been unemployed), requiring the student to work beyond his contractual hours and therefore impinging upon his study time.
   

• Family illness and work-related problems.
   

• Overseas employment was obtained, precluding the student from complying with the compulsory face-to-face attendance requirement.
   

• Demands of the course were much greater than the student had anticipated.

On the conventional course, the one student who withdrew stated work-related problems as the reason for her withdrawal.

On the ODL course, the higher withdrawal rate could possibly be related to the mode of delivery, in that the monitoring mechanisms incorporated within its structure makes it perfectly transparent once a student begins to fall behind. The requirement to submit tutorials at particular stages throughout the academic year (usually two each week) means that any student failing to submit will be immediately identifiable. Once a student had fallen behind, they would be contacted by the Course Director. This was generally the catalyst which subsequently resulted in the student's withdrawal. The intensive nature of the course was such that given their other commitments they did not think it prudent to proceed. All five students who withdrew were UK graduates: one had a 1 ^st class honours, two had a 2i, one a 2ii and the final one a 3 ^rd . Academic ability could not have been the reason for the majority of these students withdrawing from the course.

In comparison, students on the conventional part-time course have a weekly attendance requirement (6.5 hours in total) and are required to prepare for tutorials each week (usually two each week). However, there is no requirement that tutorials have to be submitted to tutors and a student could still attend even though the tutorial work had not been completed. Some students would occasionally fail to attend classes.

These factors might also be relevant when considering the final assessment performance of those students who completed the academic year. This will be considered further below.

4.1.2 Student Assessment

Of the eight students who completed the first year of the ODL course, seven were successful in all four subjects. One student had his examinations deferred due to illness during the examination period, and therefore he has been omitted from the statistics to avoid a distortion.

Figure 3: Distance Learning - Individual Subject Results

Figure 4: Distance Learning - Average per Subject

Of the thirteen students who completed the first year of the conventional part-time course, one student failed to sit the final examinations and was deemed to fail them. In order to avoid a distortion of the results this student has been omitted from the analysis. In addition, four students failed to sit the Contract Law examination for a variety of reasons. Similarly, these students' results in this one subject have been omitted from the analysis in order to avoid a distortion.

Figure 5: Part-time Conventional - Individual Subject Results

Figure 6: Part-time Conventional - Average per Subject

Figure 7: Average per Subject (ODL and Conventional)

Comparing the average results of students on the ODL and conventional part-time courses indicates a higher average in each of the four subjects by students on the ODL course. One underlying factor discussed above concerned the work-rate of students on the ODL course. Students were required to submit tutorials at the end of each unit and therefore it became apparent which students were not working to the required level. However, that said, only two students (numbered 4 and 7 on the 'Distance Learning - Individual Subject Results' bar chart above) regularly submitted their tutorials on time, and one student (numbered 5) regularly failed to submit tutorials. The other five students began to fall behind at various stages throughout the academic year, and while they usually submitted their tutorials late, towards the end of the academic year some tutorials were not submitted at all. Therefore this underlying factor should be placed in its context.

One other underlying factor relates to the academic ability of the students. Four of the seven ODL students completing the examinations had a 2i honours degree classification (of whom one had a higher degree); the remaining three had a 2ii (of whom one had a higher degree and one a professional qualification). This can be compared to the twelve students who completed the examinations on the conventional part-time course: one had a 1 ^st class honours, nine a 2ii and two had been issued with a Certificate of Academic Standing by the Law Society on the basis of having degree-equivalent qualifications. It could be argued that the ODL students were academically stronger than those on the conventional part-time course; an underlying factor which could have influenced the results.

Figure 8: Pass-rate per Subject (ODL and Conventional)

Not only were the average scores greater for the ODL students, but there was a 100% pass rate in respect of the ODL students. This was not the case in respect of students on the conventional part-time course.

4.2 Implementation Issues

Closed questions and students' answers from relevant sections of questionnaires will be set out below, together with students' answers to open questions and other data where appropriate. There were six students who completed the evaluation file on the ODL course and ten students on the conventional part-time course. Once again, given the small group of students involved, the following data is not statistically significant; it merely represents an overview of implementation issues during the 1998/99 academic year. Where appropriate there will be a discussion of any underlying factors which could have had an influence on the collated data.

• What are the weaknesses and strengths of the design of the program's website?

These results indicate a high level of user satisfaction with the site. The one possible 'weakness' where 42.9% of users were neutral, concerned the enjoyability of browsing the site. Although there were no negative recordings for this question, if the site is not enjoyable to browse, this could have a demotivating influence on students and ultimately affect their learning.

This enjoyability factor was noted in a couple of student responses to the open questions, for example:

The site is basic but functional. It had been decided to exclude complicated graphics from the site to facilitate high speed access. However, since all the latest browsers have the facility to disable the display of graphics, it is recognised that this may need addressing, so that those students who enjoy a 'more Homepage design' are not disappointed.

A strength of the site was the ease with which students could use all its facilities. A one-hour session was set aside during the introductory programme at the start of the academic year. All students had mastered the site by the end of the session. Most students were comfortable with the full range of facilities within 15 minutes.

Students at the start of the course were generally competent in the use of C&IT, as is evidenced from their user profiles:

• What are the weaknesses and strengths of the learning materials (i.e. unit materials and/or directed reading)?

1. The aims/objectives were clearly set out

2. The materials were well structured and clear

3. The materials were informative and useful

4. Having read the materials, the aims/objectives have been achieved

Student replies indicated that the time taken to access, download and read the materials was approximately three hours for each Public Law and Contract Law unit, four hours for each EC Law unit, and five hours for each Criminal Law unit.

Student replies to the closed questions indicated a high level of satisfaction with both EC Law and Public Law materials, less satisfaction with Criminal Law, and the least degree of satisfaction with Contract Law. With regard to Criminal Law, this could possibly be explained by the fact that the key text was disliked by the vast majority of students which resulted in them relying quite heavily on the tutor's lecture notes. In comparison students were generally satisfied with the EC Law and Public Law key texts. With Contract Law, no tutor's lecture notes were provided with the consequence that students were having to rely very much on the key text.

The next two questions are linked and therefore will be considered together:

• What are the weaknesses and strengths of the email discussion group and the facility to email queries to the Subject Team?
   

• Does the email discussion group facilitate collaborative learning?

The following closed questions related to the effectiveness of the email discussion group, the primary aim of which was to encourage collaborative learning between students.

1. The activities of the discussion group provided me with an opportunity to interact

2. The activities of the discussion group provided me with the motivation to complete tutorials

3. The activities of the discussion group were useful in assisting me to complete tutorials

Students were generally very negative about the email discussion groups. The concern expressed by students centred around the fact that quite early on students began working at different paces. Because the majority were in full-time employment, they did not have the time to contribute and to read the contributions of others. The discussion group activity soon began to dry up. By the end of the fourth week no contributions were being made, and this continued throughout the remainder of the academic year.

Usually two tutorials had to be submitted each week. Students would be working on their tutorials at different times during the week. Therefore, for example, a student working on one particular question during the week prior to submission would be unable to interact via the email discussion group unless another student was working on the same question at around the same time. Although the email discussion group is asynchronous, given the tight deadlines, unless students are working on the same tutorial question at about the same time, it is inevitable that interaction will falter. As the course progressed two other factors militated against the email discussion group working effectively:

• Student withdrawals resulted in only eight students completing the academic year, thus exacerbating the difficulty of ensuring that students would be working on the same tutorial question at around the same time.
   

• The majority of students soon began to fall behind with their tutorials, and therefore were not even working on the same tutorial.

In the interview with the ODL Student Representative on 14 November 1998, it was stated that 'There is concern that the email discussion groups are not working as effectively as they should be, and therefore the aim of encouraging collaborative learning is not being achieved'. On 8 May 1999 this had hardened to 'Students are no longer using the email discussion groups and therefore there is no student interaction when completing tutorials'. The Student Representative also commented that a bulletin board would probably have been a more effective mechanism for encouraging collaborative learning.

The comments from students recorded on their end-of-academic-year questionnaires echoed the same theme:

• 'There were not many contributions and they quickly disappeared altogether.'
   

• 'There was not enough content. There was no real discussion thread. Everyone seemed to be at different stages.'
   

• 'Everyone was working at a different pace and therefore if I was stuck I would telephone one of the other students.'

However, the two students who were regularly submitting their tutorials on time had set up their own group (sending one-to-one emails) and were assisting each other in the completion of tutorials. This worked quite well and their assessment results were at the top-end of the scale (students numbered 4 and 7 in the 'Distance Learning - Individual Subject Results' bar chart at section 4.1 above).

The following closed question related to the Subject Team email facility:

With regard to emails to the Subject Team, most students failed to rely upon this, except for clarification following receipt of their tutorial feedback. The issue had been raised by the Student Representative on 14 November 1998: 'The three-day target for a Subject Team to respond to a student's email query is considered to be too long. Most students start and finish their tutorial on the same day or over a period of a couple of consecutive days, and therefore if a query is raised with the Subject Team, the response would be received too late to be incorporated'. At the Course Director's suggestion, some students had started to put questions in their tutorial submissions where they were unclear about a particular point so that it could be addressed in the tutor's feedback.

That said, one student liked the email facility and stated that 'the answers were generally more reliable [than the email discussion group] and were answered with greater clarity'.

• What are the weaknesses and strengths of the tutorial submission and subsequent tutor feedback?

1. Tutorial questions were generally well structured and clear

2. I was sufficiently motivated to complete tutorials

3. Completing tutorials assisted my understanding

4. Feedback was useful and comprehensive

The only negative response concerned tutorial motivation in Criminal Law where 16.7% (i.e. one student) disagreed that they were sufficiently motivated to complete the tutorial. This negative response could be attributed to the fact that the recommended text was generally considered to be unsatisfactory and students found Criminal Law to be the most difficult of the four subjects. This was confirmed by the Student Representative during the two face-to-face interviews.

One request made by the Student Representative and by students in whole-group discussions, was for tutors to provide outline answers for tutorials, in addition to individual feedback. This will be discussed further below.

• Does a user's learning environment affect the ease in which they can use the program? If so, does this have an effect on the user's interest and motivation?

Only one student experienced problems with her learning environment caused by a fault with her ISP. Eventually this was sorted out, but further difficulty was encountered when the student's computer was infected with a virus. Nevertheless, the student completed the year and was successful in the assessments.

This student became demotivated during the period of difficulty, and in the end of year questionnaire declared a preference for communication via the telephone rather than by email!

All students undertook the course as distance learning students (as opposed to open learning students) using their own hardware and software. Students' user profiles indicated that all students had a minimum specification of a Pentium 233 and 56 Kbps modem. Their hardware was therefore of a high enough specification to prevent this being a cause for concern (unlike that of tutors).

Tutors generally agreed that their computers were of too low a specification and needed upgrading. Some staff initially experienced problems with the email system and keeping track of students. However, this was sorted out within the first month and staff ended the year with a high degree of confidence and satisfaction.

• Is the time in which it takes the student and tutor to complete a unit on this program different to that on the conventional face-to-face part-time course?

4.2.1 Students - ODL Course

Figure 9: Distance Learning Course - Average Time per Unit

Public Law

The average time spent upon a unit was 7 hours 55 minutes (broken down into 5 hours 20 minutes for accessing, downloading and reading the lecture materials, and 2 hours 35 minutes for completion of the tutorial). The highest composite time was 10 hours 30 minutes and the lowest was 4 hours.

EC Law

The average time spent upon a unit was 9 hours 10 minutes (broken down into 5 hours 35 minutes for accessing, downloading and reading the lecture materials, and 3 hours 35 minutes for completion of the tutorial). The highest composite time was 12 hours 30 minutes and the lowest was 7 hours.

Criminal Law

The average time spent upon a unit was 10 hours 55 minutes (broken down into 6 hours 10 minutes for accessing, downloading and reading the lecture materials, and 4 hours 45 minutes for completion of the tutorial). The highest composite time was 16 hours 30 minutes and the lowest was 7 hours.

Contract Law

The average time spent upon a unit was 8 hours 25 minutes (broken down into 5 hours 20 minutes for accessing, downloading and reading the lecture materials, and 3 hours 5 minutes for completion of the tutorial). The highest composite time was 13 hours and the lowest was 3 hours 30 minutes.

Students spent the greatest amount of time on Criminal Law. Student questionnaires (end-of-unit and end-of-academic-year) reflected this. Students generally commented that this was the most difficult subject. The ODL Student Representative, during the interview on 8 May 1999, commented that 'Students generally are devoting a disproportionate amount of time to Criminal Law'. As previously discussed there was a general disliking of the text which was considered to be very difficult for distance learners to grasp. Surprisingly, the lowest amount of time was spent on Public Law where there were only nine units compared to the other three subjects' twelve units.

4.2.2 Students - Conventional Part-time Course

Lecture/tutorial face-to-face contact was approximately three hours per unit. Students completing EC Law and Public Law on the conventional part-time course were required to keep a record of the total time they spent on each unit (including attendance and tutorial preparation), to ascertain whether or not there was any marked discrepancy between the ODL course and theirs.

Figure 10: Part-time Conventional - Average Time per Unit

The average time spent upon a Public Law Unit was 5 hours 15 minutes. The highest composite time was 8 hours and the lowest was 2 hours 30 minutes. There were only nine units in total in Public Law compared to twelve in each of the other three substantive subjects.

The average time spent upon an EC Law Unit was 5 hours 25 minutes. The highest composite time was again 8 hours and the lowest was again 2 hours 30 minutes.

Figure 11: Average Time per Unit (ODL and Conventional)

Students on the ODL course on average took considerably longer to complete a unit than their counterparts on the conventional part-time course. However, ODL students would save time travelling to the University two evenings each week throughout the academic year. On the conventional course, the impression of tutors was that few students had actually prepared sufficiently prior to a tutorial, whereas with the distance learners, the requirement to submit the tutorial in writing required full participation in this exercise (although, as discussed above, some ODL students submitted their tutorials late or failed to submit them altogether).

4.2.3 Tutors - ODL Course

The total time spent by tutors delivering Public Law over the whole academic year was 38 hours 20 minutes which averaged approximately 4 hours 15 minutes for each of the nine units:

Figure 12: Public Law

The total time spent by tutors delivering EC Law over the whole academic year was 51 hours 30 minutes which averaged approximately 4 hours 20 minutes for each of the twelve units:

Figure 13: EC Law

The total time spent by tutors delivering Criminal Law over the whole academic year was 62 hours which averaged approximately 5 hours 10 minutes for each of the twelve units:

Figure 14: Criminal Law

The total time spent by tutors delivering Contract Law over the whole academic year was 96 hours 20 minutes which averaged approximately 8 hours for each of the twelve units:

Figure 15: Contract Law

The greatest amount of time delivering the ODL course was spent by the Contract Team. In the initial tutor questionnaires, the Contract Law tutors indicated they were having to provide a large amount of feedback because students were misunderstanding important areas. As no tutor-prepared lecture notes were provided it is possible that this constituted a deficiency in the method of delivering Contract Law. EC Law and Public Law ran smoothly as far as tutors were concerned. Students had the benefit of comprehensive lecture notes and were also directed to key websites where further literature could be digested. Criminal Law was considered by students to be the most difficult of the four subjects. This possibly explains the reason why tutors' time was greater than that for Public Law and EC Law, despite comprehensive Criminal Law lecture notes being provided to students. The key text was disliked by students as being too difficult, whereas in comparison, the key text for Public Law and EC Law was unproblematic.

The experience of tutors generally was that the tutorial feedback given to students was often repeated. Students were generally making the same mistakes. Outline answers could therefore have reduced the time tutors had to spend typing out the same feedback to individual students. This issue could become even more critical if student numbers increased.

One other matter, discussed above, is that students tended to drift in and out of tutorials, and (with two exceptions) generally submitted their tutorials outside of the planned start and end dates, and sometimes not at all towards the end of the year. This could explain why later tutorials required less tutor time.

The course was delivered on the conventional part-time course by 1.5 hour lectures/tutorials over a 26-week period. Staff were therefore credited with 39 hours on their timetable. On the ODL course each tutor was credited with 26 hours on their timetable (total time for two tutors is 52 hours). This was sufficient for Public Law and EC Law, but was marginally inadequate for Criminal Law and substantially inadequate for Contract Law.

• Is face-to-face tuition necessary as a compulsory element of the program?

1. The session was informative and useful

2. The session helped me develop my own thinking

3. The session gave the opportunity for lively and relevant discussion

4. The session provided an invaluable opportunity for direct interaction with students and the tutor

5. The session should be optional rather than compulsory

6. If the session had been optional rather than compulsory I would definitely have attended

The above is illustrative of the fact that the students, almost unanimously, considered that the face-to-face sessions should remain compulsory rather than optional. They proved to be a vital element in the learning experience of students, providing the opportunity for tutors to motivate the students and for the students to engage in face-to-face interaction with their fellow students and tutors. Attendance at the face-to-face sessions was 100%, which is indicative of the fact that they were a vital component of the course. However, the students' responses to these questions may have been different had the interactive element of the course (i.e. email discussion groups) worked effectively!

5. Conclusions

The recorded data is not statistically significant due to the small number of students on the course and of students taking part in the evaluation study. However, it represents an interesting overview of the outcome and implementation measures of the course during its first year of operation. Underlying factors which could have an impact on the data have been discussed where relevant.

5.1 Outcome Measures

5.1.1 Retention Rate

The retention rate of students on the ODL course appears to be statistically poor. Of the 13 students who initially enrolled on the course in September 1998, only eight (61.5%) completed the academic year.

However, the withdrawals were all for reasons which made future participation on the course difficult, if not impossible. It does not seem to be appropriate that students should be encouraged to continue in such circumstances. Of the five students who withdrew, one deferred his studies until the 1999/2000 academic year. Some of the students who withdrew had completed a substantial part of the academic year, and therefore it might be appropriate to explore the option of such students deferring their studies so that the work undertaken prior to withdrawal is not wasted.

5.1.2 Assessment

Of the eight students who completed the academic year, seven entered the final examinations. The pass rate was 100%, which was greater than that for students on the conventional part-time course. The average mark in each of the four subjects was likewise greater than that of students on the conventional part-time course. Two underlying factors were highlighted which could have had an influence on this data:

Firstly, the requirement that ODL students regularly submitted tutorial work should have ensured that they covered the whole syllabus and that any mistakes or misconceptions were rectified in the tutor's feedback (this was not a requirement for students on the conventional part-time course). However there is a caveat to this in that, in practice, six of the eight ODL students who completed the year soon began to fall behind with tutorial submission and some of the later tutorials were not submitted.

Secondly, the academic qualifications of the ODL students were generally higher than those of the conventional part-time students, and therefore the ODL students might be considered to be academically stronger; this underlying factor could have had a significant effect on the data.

Nevertheless, a 100% pass rate in the first year of operation is indicative of the fact that the ODL course was educationally successful.

5.2 Implementation Measures

5.2.1 Website Design

The website was designed to be basic but functional. Its design was such as to avoid complicated graphics which could slow down accessibility and act as a demotivating influence on students and tutors. However, given the fact that the latest browsers have the facility to disable graphics display, it is recognised that the website should be redesigned so that it has a more stimulating and exciting visual impact. This itself could motivate the users.

All students and tutors could navigate the website with ease; this is an obvious strength and should be recognised when redesigning the website (see section 5.2.3 below).

5.2.2 Learning Materials

The learning materials for each subject were prepared by the respective Subject Leaders. EC Law, Public law and Criminal Law all consisted of substantive tutor-prepared materials. Students were also required to purchase a key text in each of these three subjects. In comparison, the Subject Leader for Contract Law considered the key text to cover the whole syllabus and to be suitable for distance learners. Therefore rather than 'reinvent the wheel' limited tutor-prepared materials were provided.

EC Law and Public Law were unproblematic for students with regard to the learning materials.

The Criminal Law tutor-prepared materials were also highly acceptable to the students. However there was a general complaint that the key text was far too difficult and unsuitable for distance learners. Questionnaires and interviews indicated that students found Criminal Law the most difficult of the four subjects, and the one which they were spending the most time on. The Criminal Law Subject Leader has subsequently reviewed the available Criminal Law texts and adopted one which is more suitable for distance learners.

Students expressed dissatisfaction with the Contract Law materials. However, it is recognised that students did not generally find this to be a difficult subject and therefore the limited nature of tutor-prepared materials did not appear to disadvantage students.

5.2.3 Email Discussion Groups

The aim of the email discussion groups was to engage students in collaborative learning when completing their tutorials so that they could learn from one another. This did not materialise and it has been highlighted as a major weakness with the program.

Although students used the facility with enthusiasm in the first few weeks of the academic year, this came to a sudden halt after week four and no further contributions were made during the remainder of the academic year. The reasons for this were identified as being two-fold:

• Student withdrawals resulted in only eight students completing the academic year, which exacerbated the difficulty of ensuring that students would be working on a tutorial at around the same time.
   

• Students soon began to submit their tutorials late, and therefore all students were generally working at different paces.

Widdison et al have reviewed the use of email as a medium for small group teaching of law at various institutions throughout the 1990's. One weakness exposed by their research is that 'Exchanges of email - particularly non-instantaneous exchanges - are generally slow and inflexible' [ Widdison et al (1998) p. 5]. This weakness possibly contributed to the failure of this element of the ODL course.

Herberger et al have investigated the use of the internet to support collaborative student learning in legal education. This weakness was also noted by them:

This weakness therefore needs to be addressed, and a number of options are available:

At the outset, the Course Team sought to include a bulletin board and/or synchronous chat room, but because of the University's Central Computing Services' policy this was not possible.

By incorporating options 3 and 4 (and implementing options 1 and 2) tutors could direct students to use the range of computer mediated tutoring systems in the initial tutorials in order to ascertain which ones were most effective in promoting interaction and collaborative student learning. Once students had become acquainted with using the various systems as part of the learning cycle, the tutor's presence could be reduced.

In order to circumvent the current Central Computing Services' policy it would be necessary to convince them that the policy should be changed (which is unlikely), or for the Business School to purchase its own server and thus be free to depart from the policy. The latter option has been adopted by other Schools within the University and therefore is preferred.

The Business School has since purchased its own server and learning environment (CourseInfo) which incorporates, inter alia , an asynchronous discussion group, a synchronous chat room, and a group work facility.

5.2.4 Emails to Subject Team

This facility was seldom used by students. Usually it was used for clarification purposes after a tutorial had been submitted and feedback had been received. It was also used during the revision period.

The ODL Student Representative stated that it was rarely used before a tutorial was submitted because the three-day response rate was unsatisfactory. Students would usually submit a query on the day they were submitting their tutorial. Students therefore would circumvent this by incorporating 'questions for tutor' within their tutorial submission, so that any problem areas could be addressed by the tutor in the feedback.

This worked satisfactorily, and it is unlikely that the three-day response rate could be reduced without placing an undue burden on tutors. The facility will be retained and students will be informed that they can place 'questions for tutor' within their tutorial submission.

5.2.5 Tutorials and Tutorial Feedback

The only negative response recorded by students was with regard to tutorial motivation in Criminal Law where one student disagreed that he was sufficiently motivated to complete Criminal Law tutorials.

Generally there was a high degree of satisfaction with the tutorial and feedback component of the course, although students would generally welcome being provided with an outline answer in addition to the feedback.

Subject Teams will in future provide outline tutorial answers in addition to feedback. This will be considered further below ( section 5.2.7 ).

5.2.6 Users' Learning Environment

Students generally had no problems with their hardware and software (with the exception of one student who experienced difficulties with her ISP and then had problems with a computer virus). However, all students during this academic year were undertaking the course as distance learners using their own equipment, the hardware of which had a minimum specification of a Pentium 233 with 56Kbps modem. It may be appropriate for students to be advised of minimum hardware specifications to avoid any future problems.

In future years, any student completing the course on an open learning basis using the University's on-campus hardware and software, will need to be carefully monitored to ensure that accessibility to the equipment, and hardware and software specifications, are suitable.

The tutors were concerned that their own hardware was outdated and required upgrading. An internal evaluation of current equipment has since been undertaken and upgrades have been sanctioned.

5.2.7 Time to Complete a Unit

Students on the ODL course generally took longer to complete a unit than their conventional part-time counterparts. However the time expended was not excessive.

Tutors had different experiences. Public Law and EC Law tutors found the delivery of the course less time-consuming and demanding than the Criminal Law and Contract Law tutors. The Criminal Law tutors' time was greater possibly because students found this a difficult subject and the key text was inappropriate. A greater amount of feedback was required to correct students' mistakes and misconceptions. Contract Law tutors again were finding themselves having to provide students with a vast amount of feedback, despite the fact students generally expressed the view that they did not find this a difficult subject.

It has been decided to provide students with outline tutorial answers in addition to feedback (see section 5.2.5 above). This could reduce the amount of time spent on individualised feedback. Where the same mistake or misconception is repeated by students, instead of repeating the same advice, the tutor will simply be able to refer to the outline answer.

One other factor which could have impinged upon tutors' delivery time could be the failure of the email discussion group. The aim of this was to facilitate collaborative student learning. If this had been achieved it could have resulted in a higher standard of submitted work with the resultant effect that feedback could have been less detailed (and therefore less time consuming). Rectification of this weakness has been discussed at section 5.2.3 above.

The total time spent by tutors delivering the course was 38 hours 20 minutes for Public Law, 51 hours 30 minutes for EC Law, 62 hours for Criminal Law and 96 hours 20 minutes for Contract Law. There are two tutors delivering each subject and therefore the individual tutor time is 19 hours 10 minutes, 25 hours 45 minutes, 31 hours, and 48 hours 10 minutes respectively. Individual tutors are credited with 26 hours on their timetable to cover delivery of the course. Therefore although this was sufficient for the Public Law and EC Law tutors, it was marginally insufficient for the Criminal Law tutors and substantially insufficient for the Contract Law tutors. This situation will be exacerbated if student numbers increase. However, revisions to the program (as discussed above) could have a downward effect on tutors' delivery time.

Tutors will engage in time recording during the 1999/2000 academic year to ascertain if there are any arguments for the timetable credit to be reviewed; this will take into account implementation of the revisions discussed above.

5.2.8 Face-to-face Tuition

The CPE Board, which externally validates the course, stipulates that students have to attend a minimum of twelve face-to-face tuition days each academic year. Students almost unanimously recorded the view that the sessions should remain as a compulsory element of the course, and that even if they were optional they would have attended.

The sessions have proved educationally rewarding for students. However, the compulsory nature of the tuition undoubtedly deters the recruitment of overseas students. It also goes against the rationale for introducing this mode in the first place (flexibility for those in full-time employment; with family commitments; with a disability etc.).

If the new learning environment (CourseInfo, discussed at section 5.2.3 above) successfully leads to interactive and collaborative student learning, it may well be that the students' views on the compulsory nature of the sessions change. Although it is recommended that the sessions should be retained, it must be questioned whether or not, in these circumstances, they should remain as a compulsory element of the course.

This issue therefore needs to be kept under review.

5.3 Subsequent Evaluations

This should only be seen as the first evaluation in a series.

Deficiencies in the program have been identified and need rectification before it can be declared to be a success. Strengths which have been identified may not warrant further evaluation. Therefore a partial evaluation during the 1999/2000 academic year could concentrate on the key weaknesses of the program:

• Interaction and collaborative student learning:

• Has the new learning environment (CourseInfo) resulted in student interactivity and collaborative learning?

• Tutors' delivery time:

• How has this changed in the 1999/2000 academic year compared to the 1998/99 academic year?

  Have outline answers reduced the volume of feedback?

  If collaborative student learning is now taking place, has this affected the need for detailed feedback?

• Learning materials:

• How have Year One students reacted to the learning materials during the 1999/2000 academic year bearing in mind the revisions made to the program?

• Face-to-face tuition:

• Have the views of students during the 1999/2000 academic year changed with regard to the compulsory nature of the sessions?

• Retention rate/assessment:

• What are the Year One and Year Two retention rates and assessment results, compared to those on the conventional part-time course?

  Is a pattern beginning to emerge?

This article has indicated that it is possible to successfully deliver a PgDL (CPE) course using the internet as the principal mode of delivery. However the necessity for ongoing evaluation is essential to ensure that all weaknesses within the program are eliminated. Once the program is considered to be operating effectively, consideration could then be given to extending the program to modules on the undergraduate LLB degree.

References

Herberger M., Scheuermann F. and Kaufmann I. (1998) 'Collaborative Learning via WWW in Legal Education' 2 The Journal of Information, Law and Technology < http://www2.warwick.ac.uk/fac/soc/law/elj/jilt/1998_2/herberger/>.

Herman J., Morris L. L. and Fitz-Gibbon C. T. (1987) Evaluator's Handbook (London: SAGE Publications Ltd.).

Jones A. et al (1996) 'Evaluating CAL at the Open University: 15 Years On' Vol 26 Computers in Education pp. 5-15.

Jones A., Greenwood S., Edwards J., McCulloch A., Tobias J. and Nicholson H. (1998) 'Using Special Study Modules to Develop CAL: An Evaluation of Student Author Benefits' No 8 Active Learning (CTI) pp. 9-14.

Oliver M. and Conole G. (1998) 'Evaluating Communication and Information Technologies: A Toolkit for Practitioners' No 8 Active Learning (CTI) pp. 3-8.

Rowntree D. (1992) Exploring Open and Distance Learning (London: Kogan Page Ltd.).

Tessmer M. (1993) Planning and Conducting Formative Evaluations (London: Kogan Page Ltd.).

Thorpe M. (1988) Evaluating Open and Distance Learning (Harlow: Longman).

Widdison R. and Schulte R. (1998) 'Quarts into Pint Pots? Electronic Law Tutorials Revisited' 1 The Journal of Information, Law and Technology < http://www2.warwick.ac.uk/fac/soc/law/elj/jilt/1998_1/widdison1/>.

Bibliography

Dearing R et al (1997) Higher Education in the Learning Society: Report of the National Committee of Inquiry into Higher Education (London: HMSO and NCIHE Publications) < http://www.leeds.ac.uk/educol/ncihe/>.

Forsyth I. and Jolliffe A., and Stevens D. (1995) Evaluating a Course: Practical Strategies for Teachers, Lecturers and Trainers (London: Kogan Page Ltd.).

Henderson et al (1987) How to Measure Attitudes (London: SAGE Publications Ltd.).

Jones R. and Scully J. (1998) 'Effective Teaching and Learning of Law on the Web' 2 Web Journal of Current Legal Issues < http://webjcli.ncl.ac.uk/1998/issue2/jones2.html >.

Morris L.L., Fitz-Gibbon C. T. and Freeman M. E. (1987) How to Communicate Evaluation Findings (London: SAGE Publications Ltd.).

Oppenheim A. N. (1992) Questionnaire Design, Interviewing and Attitude Measurement (London: Pinter Publishers Ltd.).

Patton M. Q. (1987) How to Use Qualitative Methods in Evaluation (London: SAGE Publications Ltd.).

Riachi R. (1998) 'ALT-C 98 and Learning Outcomes' No. 8 Active Learning (CTI) p. 65.