JILT 1996 (3) - Hegarty & Routen
STATUTOR: Intelligent Tutoring System?
Caitriona Hegarty and Tom Routen
De Montfort University
twr@dmu.ac.uk
Abstract
This paper describes some of the issues that have arisen from the development of an Intelligent Tutoring System to educate students in the statute law domain.The originality of the system consists of a graphical environment, in which the student can represent valid legal arguments by constructing complex graphical structures.A brief description of the system is given, mainly the graphical environment or tutorial module, the authoring module and the expert system module.The three modules share the same knowledge base providing an interesting example of reusability of declarative knowledge.Problems encountered throughout the project includng the rule-based representation of the statute and a suitable representation format for the graphical display are outlined.Reference is also made to interface developments including the type of assistance available to the student both during the process of constructing an argument and at the end of the exercise.The results of an interim evaluation of the system are also provided along with an indication of where the project might go in the future.
Contents
- 1. Introduction
- 2. The Architecture of Statutor
- 2.1 Domain Knowledge
- 2.2 Student Model
- 2.3 Tutor Model
- 3. A Statutor Exercise
- 4. User Interface Developments
- 4.1 The forwards only exercise dialogue
- 4.2 Student Feedback
- 4.3 System Feedback
- 5. Evaluation
- 6. Conclusion
- References
Downloa Date of publication : 30 September 1996
Citation: Hegarty C (1996) 'Statutor: Intelligent Tutoring System?', BILETA '96 Conference Proceedings, 1996 (3) The Journal of Information, Law and Technology (JILT). <http://elj.warwick.ac.uk/elj/jilt/bileta/1996/3hegarty/>. New citation as at 1/1/04: <http://www2.warwick.ac.uk/fac/soc/law/elj/jilt/1996_3/special/hegarty/>
Intelligent Tutoring Systems (ITSs) aim to provide students with individualised, dedicated tutoring based partly upon an analysis of the procedures followed by the user and AI techniques which may provide some assistance on how the user should progress. This can only be achieved if ITSs know what they teach, who they teach and how to teach it. The use of an ITS shell in a teaching domain has several advantages. A practical advantage is that the user of the shell only has to implement a knowledge base for the new domain and not a complete new system. A second, theoretical advantage is that the main knowledge of the shell can be oriented toward general theories and strategies of teaching whilst at the same time providing a means to test their generality in different domains. An important role in ITSs is played by the instructional environment.
In 1992 a prototype ITS shell called Statutor (Statute-Tutor) was designed. The aim of the project was to support, amongst other things, the reuse of formalisations of statute law in the presentation of exercises analogous to the familiar case analysis exercises of traditional legal education. In 1994, the Joint Information Systems Committee of the UK Higher Education Funding Councils under its New Technologies Initiative awarded support for a two-year project aimed at developing the prototype and producing a finished system which could be made available to law schools. This paper reports on the progress of this project, its evaluation and future development.
2. The Architecture of Statutor
The system provides a graphical environment in which the user can represent a legal argument by means of a proof tree. The graphic tree is composed of a series of inter-related text boxes representing facts, negated facts, conditions and conclusions. The text boxes are linked to the primary statutory source. There are three main components to Statutor: domain knowledge, a tutor model and a student model. (See Figure 1)
Figure 1
2.1 Domain Knowledge
One aim of this project was to set up a realistic knowledge base and associated exercises to enable an effective evaluation of the potential use of the system within a law curriculum. The Data Protection Act 1984 (DPA) was chosen to form the basis for the development of the knowledge base. In developing the realistic knowledge base several problems came to light. Firstly, it was apparent that the system needed to be flexible. Negated conditions in an argument had to be recognisable. Although the main idea behind Statutor was to treat the system's ability to construct a proof-tree as analogous to answering a case analysis exercise, it was also apparent that students could not always be expected to complete an exercise to the level of detail expected by the system. Statutor could not be adjusted to suit the needs of individual users. As a result the knowledge domain is limited to specific sections of the Data Protection Act 1984. It contains sufficient knowledge to generate a series of exercises with system generated solutions. It does not provide individualised explanations as to the appropriateness of any solutions.
2.2 Student Model
A student model aims to meet the needs of the user. Such an objective may be achieved by providing support during the attempted construction of an argument. Alternatively support may be displayed upon completion of the exercise by means of a comparison between the system generated proof-tree and the tree constructed by the student. The prototype was designed only to provide support on completion of an exercise. Changes have been implemented to extend the help facilities available during the construction of a graphic tree.
2.3 Tutor Model
The tutoring model provides the tutor with the facility to set up a series of exercises which the student may attempt. In the course of constructing an exercise the tutor utilises the inference engine and the domain knowledge to answer a series of questions posed during the reasoning process until a conclusion is reached. The tutor does not have the flexibility to vary the phraseology of an argument nor to add further comment. Within the tutoring model the student is asked to demonstrate an understanding of the logical structure of a statutory provision by means of a graphically constructed argument.
Figure 2 illustrates a typical exercise as it is presented to the student. The conclusion the student has to prove is graphically displayed in a text box at the centre top of the figure (text-box in bold characters). The student is then free from constraint to select from and connect in the appropriate way the conditions shown in the text boxes displayed directly below the conclusion. There are three different types of conditions, namely facts (text-boxes in underlined characters), negated facts (as facts but with a NOT flag), and conditions. Conditions are proved by using facts, negative facts or indeed other normal conditions (the real system makes full use of colours to distinguish among the different types of conditions). There are also red herrings or conditions which are not required in the specific argument. A number of functions are available to aid the student. These include the tools on the left hand side of the figure. The open book icon represents a source tool. This tool enables the student to view and add the source text.
Figure 2
A student may draw a number of inferences in order to prove a conclusion. An inference being a reasoning step taken from a number of facts to a conclusion. Figure 3 illustrates that at the bottom level inference the student states three conditions are required to prove the intermediate condition. This rule originates from the Data Protection Act, section 1, subsection (5), paragraph (h), subparagraph (ii). In the top level inference the student believes that a single condition is required to prove the conclusion according to DPA S.1(5). It is not necessary to provide evaluable conditions which are automatically generated by the system. However, the student user must check that the remaining conditions are such that the evaluable conditions are satisfied. Evaluable conditions are not required when the calculations involved are complex and the tutor concludes that such a task is unnecessary or if the facts as stated in the evaluable conditions are obvious from other conditions. Students can view the content of the evaluable conditions by clicking on a small circular symbol displayed on the screen.
Figure 3
Fig. 4 provides an illustration of when it is unnecessary to prove an evaluable condition. There is an obvious derivation of the other conditions.
Figure 4
The on line source text may be accessed by clicking, on the labels referring to the applied rule in the system proof tree. For example, clicking on the label DPA 1(5)© in Figure 3 above will result in the following extract from the statutory provisions, being displayed on screen. Paragraph © will be highlighted:
(5) " Data user " means a person who holds data, and a person " holds " data if:
(a) the data form part of a collection of data processed or intended to be processed by or on behalf of that personas mentioned in subsection (2) above; and
(b) that person (either alone or jointly or in common with other persons) controls the contents and use of the data comprised in the collection; and
(c) the data are in the form in which they have been or are intended to be processed as mentioned in paragraph (a) above or (though not for the time being in that form) in a form into which they have been converted after being so processed and with a view to being further so processed on a subsequent occasion.
Alternatively the student could access the representation of data holder:
srule(['DPA','1','5','©'],
holds_data(USER, DATA),
[controls_content_and_or_use_of(USER, DATA),
convertible (DATA, CONVERTED_DATA),
different (DATA, CONVERTED_DATA),
data (CONVERTED_DATA)] ).
4. User Interface Developments
The graphic interface serves two functions. Firstly, to represent the system generated exercises and associated information to the student. Secondly, to translate the student responses which are to be processed by Statutor. In the prototype of Statutor emphasis was placed on the student modelling facility and the tutoring model. However in view of the argument that a system appears only as intelligent as its interface, the decision was taken to enhance the user interface as an important component in the architecture of the Statutor ITS.
4.1 The forwards only exercise dialogue
In the exercise dialogue illustrated by Figure 2, the student is given access to all the facts, conditions and the conclusion. The student will be expected to link the text boxes appropriately in the construction of a proof-tree. There are a number of options available to the user. The student may opt for the forwards only exercise. By making this selection an unknown conclusion will be displayed as a text box with a question mark (See Figure 5). Alternatively the student may opt to be provided with only the facts and a set of rules. If this type of exercise is selected the student cannot reason backwards.
Figure 5
The student links a set of facts and their appropriate rule to the unknown conclusion, and then asks the system to produce the conclusion. (See Figure 6).
Figure 6
If the student's inference is correct, the unknown conclusion will be changed into the conclusion derived by the inference.Once a correct inference is made, a new unknown conclusion is generated by the system. The student is free to continue the forward reasoning process until the final conclusion is reached (See Figure 7). When the exercise has been successfully completed the unknown conclusion is not generated by the system. There may however be more than one conclusion.
Figure 7
4.2 Student Feedback
In any tutoring system it is important to consider not only how feedback is given to the student but also when it is most useful. Whilst assistance may be vital it should not be too disruptive. Excessive interruption can be tedious and soul destroying. In the development of Statutor the graphic interface aims not only to present information and exercises generated by the system to the student, but also to translate the student's actions to be processed by the system. It was decided that the student would benefit most from being able to reference three types of material. The primary source material (the statute), the rules which apply to the construction of a particular exercise in addition to an immediate graphical diagnosis of the progress being made in the construction of the graphical tree. The graphical feedback on an exercise would indicate whether the conditions provided are correct or incorrect(crossed box). If any conditions are missing this is indicated by a shaded text box (See Figure 8).
Figure 8
A diagnosis of the accuracy of the statute labelling is also possible. The reference diagnosis can be seen more explicitly by clicking on the symbolised diagnosis, as in Figure 9.
Figure 9
4.3 System Feedback
The feedback described so far is student controlled feedback. The student is free to choose when feedback is required. Statutor also provides some kind of system controlled feedback. By recording the various diagnoses made during the student's problem solving activity, the system can evaluate the student not only on the basis of the present state of the exercise, but considering also the process through which the present state was reached. The mechanism provides a means of controlling the type of feedback provided to the student. In contrast with immediate feedback where the system intervenes after each error made by the student, and final feedback, where feedback is given only at the end of the exercise without considering the intermediate steps, this mechanism may mean that the student makes significant errors before there is any intervention.
The student requiring frequent feedback is most likely to be a user with limited knowledge of the rule examined in a particular exercise. By requiring assistance the student has indicated doubts but the assistance may reduce the degree of error in the final proof-tree. Student diagnosis of this nature can be used in selecting the next exercises to be given to the student and indeed the format in which these exercises should be presented. A student's lack of knowledge of the use of a particular rule will lead to subsequent exercises where that rule can be re-examined. When the student appears to have mastered the use of a particular rule, the following exercises may not deal with the same rule, or, if they do, the rules will not have to be evaluated. The conclusion of the rule will be represented as a fact. Collapsing the expansion of a mastered rule into a simple fact can be achieved dynamically or by the human tutor using the authoring mode. In fact the human tutor can decide before constructing the new exercise which rules are to be expanded and which are not. Another beneficial result of collapsing mastered rules into simple facts lies in avoiding the repetition of the same rule expansion within a single proof-tree.
There are no general guidelines or standards for the evaluation of ITSs. As the main concern was to identify system related problems, a formative evaluation of the system was carried out in July 1995. A number of students some with prior knowledge of the DPA were guided in a rather formal manner through exercises with the system, and subsequently questioned about their impressions and ideas concerning the applicability of the system. Furthermore, the system was demonstrated to a number of law tutors in an attempt to elicit potential tutor reaction to the system.
The general impression of the system was fairly consistent. All test subjects understood the presentation of the legal argument with the help of the graphical representation used. They claimed to have developed new perspectives on the logical construction of a legal argument. The use of the graphical presentation would appear to be useful. Some students suggested that there was some utility in using Statutor as an alternative introduction to statute law and the Data Protection Act. Such an introduction should be supported and supervised by a human tutor, but restricted to an introductory stage.
The subjects managed to complete the given exercises in a reasonable time. Subjects with no previous exposure to the Data Protection Act concentrated on the graphical feedback and solved the exercises mainly with graphical help, without referring to the source material. Students who had some prior knowledge of the DPA attempted to solve the exercises by relying solely on their previous knowledge and the statute text. They accessed the graphical feedback only to verify and correct their proposed solution. This supports the conclusion that students require some type of introduction to the statutory framework prior to attempting the system generated exercises.
Response to the graphical diagnosis and the system proof trees was positive. In fact students indicated an interest in being able to construct their own exercises. Students' logical perception of the arguments appeared to improve. Those with no prior exposure to the domain managed to solve a paper based exercise without much difficulty at the end of the trials.
The main criticism of the system, as expressed by both students and tutor, concerned the rule-based representation of the statute law. Students did not make much use of, and did not understand, the feedback providing them with the rule-format of the exercise they were attempting to construct. Students and tutors would like to see the argument's text being phrased in a more articulate way, and not being restricted to the rigid format required by the expert system. Furthermore, there was consensus that the knowledge domain needs to be extended.
During these evaluations, we came upon a valuable but paradoxical insight: that by making the system simpler, it could be made more useful. We are used to expecting that injecting a little intelligence into the system will enhance its capabilities, but in the case of the present system, for the domain of statute law at least, it may well be that it is possible to separate the elements of the system which law tutors have responded positively to from those elements of the system which seem to restrict its applicability. In short, as long as we maintain the interface but dispense with the knowledge based aspects, then we may well be able to have more impact in presenting technological assistance for law tutors.
As a result of this insight we are investigating the possibility of a new system, implemented in Visual Basic, which would apply the same principles as the current system, but without requiring a rule-based representation of the law domain. In the new system the authoring tool permits the human tutor freely to edit both the argument's texts and structure. The ITS could be embedded within a hypertext environment. In brief, the tutor will construct his own graphical proof-tree, editing the text boxes and structuring the argument as he wishes. The only drawback of this approach is that the human tutor will not be provided with the argument structure, but will have to construct it himself. Much of the functionality of the system as discussed in this paper; feedback, tree-comparison and tutorial action, can then be applied by using the graphical argument structure as constructed by the human tutor. With this approach human tutors can build structured arguments related to any piece of law, importantly including case law, without the requirements of the law having been previously formalised.
This project was predicated on the idea that the ITS shell is of interest to law tutors, but it was also predicated on the idea that it could be of interest in other subject domains too. Initial positive responses were not related to the fact that the system could answer its own questions and had an expert system component. They were related to the interaction which the system provided through the graphic interface. We hope to be in a position in the near future, with two well developed systems, one knowledge-based and the other much simpler technology, to be able to provide a conclusive answer to this question.
Anderson, 1985. J.R. Anderson, C. F. Boyle, G. Yost, 1985. The Geometry Tutor. In : Proceeding of International Joint Conference on Artificial Intelligence. 1985.
Anderson, 1990. John R. Anderson, Eric G. Patterson, Albert T. Corbett. Student Modelling and Tutoring Flexibility in the Lisp Intelligent Tutoring System.In: INTELLIGENT TUTORING SYSTEMS (At the Cross-roads of Artificial Intelligence and Education) Edited by: Claude Frasson, Gilles Gauthier. Ablex Publishing Corporation. Norwood, New Jersey. 1990.
Burton, 1982. Richard. R. Burton. Diagnosing bugs in a simple procedural skill. In: INTELLIGENT TUTORING SYSTEMS. Edited by: D. Sleeman and J.S. Brown ACADEMIC PRESS INC. (LONDON) LTD. 1982.
Burton, 1988. Richard. R. Burton. The Environment Module of Intelligent Tutoring Systems. In: Foundations of INTELLIGENT TUTORING SYSTEMS. Edited by: Martha C. Polson, J. Jeffrey Richardson. Lawrence Erlbaum Associates. Hillsdale, N.J. 1988.
Clancey, 1982. William J. Clancey. Tutoring rules for guiding a case method dialogue.In: INTELLIGENT TUTORING SYSTEMS. Edited by: D. Sleeman and J.S. Brown. ACADEMIC PRESS INC. (LONDON) LTD. 1982.
Frasson, 1990. C. Frasson, G. Gauthier. Introduction. In: INTELLIGENT TUTORING SYSTEMS (At the Cross-roads of Artificial Intelligence and Education) Edited by: Claude Frasson, Gilles Gauthier. Ablex Publishing Corporation. Norwood, New Jersey. 1990.
Lawler, 1987. Robert W. Lawler, Masoud Yazdani. Introduction. In: Artificial Intelligence and Education. Volume One. Edited by: Robert W. Lawler, Masoud Yazdani. Ablex Publishing, Norwood, New Jersey. 1987.
McKendree, 1990. Jean McKendree. Effective Feedback Content for Tutoring Complex Skills.In: HUMAN-COMPUTER INTERACTION, 1990, VOLUME 5, PP. 381-413. Lawrence Erlbaum Associates, Inc.
Routen, 1991. Tom Routen. Complex Input: A Practical Way of Increasing the Bandwidth for Feedback and Student Modelling in a Statute-Based Tutoring System. In: Proceeding of the Third International Conference on Artificial Intelligence and Law. Oxford, England, 1991.
Routen, 1992. Tom Routen. Reusing formalisation of legislation in a tutoring system. In: Artificial IntelligenceReview 6, 145-159, 1992. Kluwer Academic Publishers, Kluwer.
Sleeman, 1987. D. Sleeman. PIXIE: A SHELL FOR DEVELOPING INTELLIGENT TUTORING SYSTEMS. In: Artificial Intelligence and Education. Volume One. Edited by: Robert W. Lawler, Masoud Yazdani. Ablex Publishing, Norwood, New Jersey. 1987.