JILT 2009 (3) - Bromby
[c]
Virtual Seminars: Problem-based Learning in Healthcare Law and EthicsMichael BrombyDivision of LawGlasgow Caledonian UniversityAbstractA series of problem-based learning scenarios were introduced using asynchronous discussion boards as a substitute for tutor-led face-to-face classroom seminars on and undergraduate LLB 'healthcare law and ethics' elective module. The scenarios contained ethical dilemmas, many of which could be solved by a number of alternative means. Student responses, therefore, were not 'right' or 'wrong' in the traditional sense but a variety of responses could be seen as appropriate, given that a supportive ethical framework or argument was presented within an answer. Feedback suggested that the students enjoyed the experience and gained a deeper understanding of the topics through advanced preparation and the ensuing discussion. Full student evaluation was conducted to evaluate the project on completion. The aim was to encourage greater student participation and co-operation in a class where many students had been extremely reluctant to offer an opinion or to challenge each other's views. Written communications appeared to create a more thoughtful discussion and reduce confrontations when discussing potentially controversial topics. Following the precepts set out in Kolb's learning cycle, the students were given the opportunity to reflect on their own group's findings in light of the other groups' feedback and comments. By confronting all sides of the debate and examining sources which may or may not support their own reading the students have brought about a transformation in their existing knowledge; a goal of constructivist learning. This is a Refereed Article published on 22nd December 2009. Citation: Bromby, Michael, 'Virtual Seminars: Problem-based Learning in Healthcare Law and Ethics', 2009(3) Journal of Information, Law & Technology (JILT), <http://go.warwick.ac.uk/jilt/2009_3/bromby> KeywordsProblem-based learning, online seminars, asynchronous discussion boards, evaluation 1. IntroductionThis paper describes how 'virtual' seminars have been employed to enhance and direct learning as an alternative to classroom-based seminar discussions using problem-based learning.[1] The transition was designed to replicate the original classroom learning activity as closely as possible, but to include flexibility for student participation and allow a longer reflective discussion period to foster critical thinking and encourage dialogue online. Classroom discussions were seen to have time limitations and were frequently cut short; the limited time available for feedback was also seen as a weakness for the classroom exercise. Feedback on performance was restricted both in relation to an individual's participation, and that of a small group. Time constraints also prevented the tutor from reporting on inter-group variation in conclusions or discussions in any meaningful depth. Problem-based learning (PBL) has origins in medical education[2], although has been used pervasively across many other disciplines.[3] The aim of PBL is to promote discovery-based learning and place the student at the centre of the learning activity, rather than teach didactically with lectures, case studies (real or hypothetical) or other forms of delivered learning. Staff take on the role of facilitator[4] rather than instructor to ensure that groups (or indeed singular students) are progressing. The nature of teamwork, cooperation and collaboration is also frequently cited as generic and discipline specific skills that can be acquired through the use of PBL.[5] A series of PBL scenarios were introduced as a substitute for tutor-led face-to-face classroom seminars on an undergraduate level 3 elective module entitled 'healthcare law and ethics'. The scenarios contained ethical dilemmas, many of which could be solved by a number of alternative means. Student responses, therefore, were not 'right' or 'wrong' in the traditional sense but a variety of responses could be seen as appropriate, given that a supportive ethical framework or argument was presented. Topics varied, but included a wide range of scenarios covering consent, medical negligence, infertility and childlessness, genetics, and end of life issues including abortion and euthanasia. Such use of problem scenarios was already embedded within the course and tutor-led classroom seminars had made use of the very same scenarios in previous years [6]. Participation was high and student motivation to marshal thoughts or prepare independently researched notes was clearly evident. The tutor played a passive role and was infrequently involved in the substantial issues that were debated: the tutors' role was reduced primarily to answering questions or monitoring the discussions in general. Any substantive questions were generally acknowledged, yet avoidance of a direct answer by the tutor ensured that the group addressed the problem without significant tutor assistance. Groups varied substantially with their responses - a mixture of unanimous, majority, minority and singular views were often expressed. Differences between the groups were also seen, although to a lesser extent: frequently each group would be representative of the entire seminar cohort. Anonymous feedback suggested that the students enjoyed the experience and gained a deeper understanding of the topics through advanced preparation and the ensuing discussion. A full student evaluation was conducted using questionnaires and a focus group to appraise the project upon completion of the first academic year-group. Alterations were subsequently made to formalise group feedback and facilitate inter-group communications. A nominated weekly reviewer was appointed in advance to summarise the discussion from each group in the form of a gobbet[7]. As each group's discussion took place within a private message board available only to the group members, the weekly summaries were posted to a separate, globally accessible message board. These summaries were therefore available to the entire cohort, and permitted individuals to make inter-group comments and responses. The aims were to encourage greater student participation and co-operation in a class where many students had been extremely reluctant to offer an opinion or to challenge each other's views. The module content covered many social, religious and legal dilemmas, therefore written communications were seem as an alternate mechanism to create a more thoughtful discussion and reduce confrontations when discussing such controversial topics. Placing students in random groups encouraged 'mutual learning', giving each student the chance to introduce their ideas among peers in a non-threatening environment. Following the precepts set out in Kolb's learning cycle[8], the students were given the opportunity to reflect on their own group's findings in light of the other group's feedback and comments. By confronting all sides of the debate and examining sources which may or may not support their own reading the students will have also hopefully brought about a transformation in their existing knowledge; a goal of constructivist learning. In sum, the module delivery team predicted five hypotheses that would enhance the learning and teaching aspects of the module, and the student experience in general. These hypotheses are:
These hypotheses will be tested and discussed in the results section below, following a brief discussion of the methodology behind the problem-based learning scenarios. 2. MethodologyThe set of problem scenarios were developed in-line with the previous style of traditional classroom seminar topics. The content was deliberately brief and insubstantial in many areas of both descriptive content and substantive law as Cruickshank[9] states that PBL problems should not be obvious, or that a solution can be immediately identified. Students must therefore make some suppositions or consider alternatives in the absence of specific information. When considering PBL for legal education, commentary from Macfarlane and Manwaring [10] illustrates the main differences between hypothetical scenarios that aim to test student knowledge about a specific and particular area, and PBL where a wider scope of possible responses should be envisaged. They suggest that a PBL problem should contain only some of the facts, and that part of the learning exercise is the students' discovery of further information. Scenarios should contain a number of diverse elements, making the problem multi-topic: "The advantages of making the problem multi-topic (or multi-issue) are several. First, this allows students to integrate ideas and materials between related areas in a way that traditional law teaching and assessment models otherwise do little to encourage. Second, it allows us to incorporate a range of policy and ethical issues into the problem we design and encourage students to consider these also. This encourages the development of a critical and practical framework for analysis. Third, this form of problem design explicitly gives students permission to be creative and to individualise (within the constraints of a group setting) their approach to the problem."[11] The scenarios were time-released on Blackboard on a weekly basis. Wednesday was seen as an appropriate start date to allow students the option of studying over the weekend if they so wished. The suggestive weekly timetable indicated that 'first thoughts' should be posted late in the week and in-depth discursive analysis should be developed early the following week. As a mere guide, it was suggested that the participation in the project would loosely follow the following pattern:
Groups of 6-7 students were provided with a private, asynchronous discussion board each week. Anonymous postings were not permitted and students were given a three-week period of face-to-face seminars to allow informal introductions and rapport building to establish a cohort identity. Groups were informed of the responsibilities for developing their own group discussion. The tutor had access to all discussion boards and occasionally made some comments during the first few scenarios to demonstrate both presence and willingness to engage if required. The first scenario to be discussed was not a substantive law issue, but a discussion of the importance of the discussion board rules[12], for two reasons. Firstly, as a tool to ensure that each student had a least located and read the rules, and secondly as a task to evaluate the students' interpretation as how important these house rules were, giving their individual reasons. Data collection was aided through the Course Statistics tool within the Control Panel for backboard to collate general trends regarding access and usage by content area. An additional evaluation survey at the end of the module was developed to gather anonymous data in relation to the students' perceptions and experiences. An anonymous online survey consisting of yes/no, multiple choice and free text responses was developed to gather general statistics on the students' background and IT competencies, and to evaluate their experiences of using discussion boards as an alternative to traditional classroom seminars. Beyond this, a focus group consisting of volunteer students was conducted to provide a more qualitative analysis of themes that emerged from the survey responses. 3. Results and AnalysisThe output from the study, in terms of participation and coursework assessment grades, was successful. No students in the sessions 2006 or 2007 declined to take part, left the course, or failed to engage as a direct result of using the discussion boards for the eight week period. Figure 1 below demonstrates that although the number of discussion board posts per day declined towards the end of the period, a substantial number of posts were consistently made on a daily basis. Indeed, for both 2006 and 2007 one single date attracted over 50 posts, created by a class of 47 and 44 respectively. Several other dates approached or achieved 30 posts within a single 24 hour period. Whilst quantity does not necessarily equate to quality, this is a good indicator of engagement.
Figure 1: Total discussion board posts per day for Semester B of 2006 and 2007
This section will now review the five hypotheses postulated in the introduction above to evaluate the learning experience of the students in more detail.
3.1 Greater flexibility of learning will be evident
Flexible delivery, as an 'Enhancement Theme', was announced by the QAA for Scottish Higher Education in 2004, and their report concluded in 2006 [13] with recommendations for the sector to support flexible programme entry, delivery and assessment. Flexible learning is seen as a significant benefit to the students at Glasgow Caledonian University as the internal Student Evaluation Project [14] identified.
"74% of our students work alongside their studies and more than half of these work more than 15 hours per week. Just over 10% of all students and 20% of those in employment report working over 20 hours a week. Two thirds of those working report that it affects their studies."
In order to balance some of these external commitments with their study time, the asynchronous nature of the discussion boards allows for greater flexible learning as timetabled seminars within the institution were no longer required. Figure 2 illustrates the spread of discussion board hits distributed according to the hour of the day. The distribution of hits is broad and covers the whole 24 hour period, although the majority of hits are between the hours of 10am and 1am.
Figure 2: Discussion board hits (viewing, not posting) measured over time.The spread of hits across the average week indicates that students were not as active over the weekend when compared with weekdays. This may, however, be biased by the scenario release time and summary deadline on Wednesday, inspiring a hive of activity to complete the discussion and frequent hits by the person allocated to construct the summary. Nevertheless, a number of hits arise from the time periods when the university is physically closed and a significant number arise from periods outwith timetabled hours. This suggests that students were indeed utilizing the flexible nature of the module and were not restricted to interactions solely within the timetabled hours.
The number of posts per day did tail off over the semester, as previously indicated by figure 1 above, suggesting that participation became less of a priority over time - although participation was still relatively high across the whole semester. Peaks of activity clearly identify the eight individual assignments over an eight week period suggesting that students did not disengage with the discussion boards as a gimmick or novelty in the early parts of the course. Research has shown that the implementation of alternative or new technologies has, on occasion, found that:
"if… overused or poorly used it can be seen by students as a gimmick or pro forma exercise in polling (Angelo & Cross, 1993). Murphy and Wolf (2005) found that as the semester progressed a few students became "bored"… and gave rushed and trivial responses…" [15]
Although peaks of activity cannot evidence whether postings were rushed or trivial, this will be dealt with in more detail below.
The focus group and survey responses indicated that student satisfaction with the ability to log on and 'chat' at their own pace was a significant benefit. The added benefit to this flexibility was the additional time spent on background reading and preparation for the discussion boards which could be done immediately prior to the posting of messages, rather than students having to 'hold their thoughts' until the timetabled seminar class. Additionally, flexible delivery benefitted staff members who were also able to access the boards remotely at any time point.
3.2 Students will become more articulate and self-reflective, independent learners
The module delivery team predicted that students could become more articulate in their presentation of argument. Firstly, the written word encourages students to have a greater awareness of the meaning of their written posts, and any possible misinterpretation. It was thought that students would be more considerate in what they wrote as once posted, the text was irreversibly published for staff and their peer group to see. This would facilitate the discussion of sensitive topics and reduce the potential dominance of a single individual during discussions. Secondly, the absence of any timetabled classes for seminar discussions would lead students to become self-managing and the reduction of the tutor from classroom discussion leader to passive discussion board viewer would encourage students to become independent learners, either individually or as a group.
Szabo,[16] citing the work of Cruickshank, lists a number of advantages to using PBL in the law curriculum. Many of these relate to the independent or group-based learning outcomes discussed by Astin, above. PBL can provide motivation to research materials as they are not provided or even listed for students to locate, and this leads to an expansion of learning resources. It also encourages students to develop working relationships and small group skills to share acquired information.
Results showed that individual students viewed a substantial number of posted created by their peers, thereby keeping abreast of the discussion and learning from each other within the group setting. The average number of discussion board posts per student per week remained constant at around 1.74, which suggests that students were principally offering comments, and replying to feedback regarding their initial posts to a lesser extent. However, some students made several posts per week and some made only one: this may be indicative of time management issues and failure to log on and check the boards regularly. The frequency of viewing the summary boards was initially high during the first few weeks before tailing off after four weeks, as seen in figure 3. The most likely cause of this effect is student familiarization with a new instrument of assessment and an interest in how other students had written and presented their summaries in the first few weeks. Discussion board activity was higher on a Monday and Tuesday towards the end of each discussion topic. This suggests that students were spending more time on the reflective and evaluative aspect of the project and not solely on creating a post to satisfy the assessment criteria of participation.
Feedback was given by the tutor in response to the summary posts to encourage self-reflection. Few students responded with written responses to the tutor's comments on their own summary, but occasionally students would respond to the summaries produced by a different group. The low level of comments in response to summaries can, in part, be due to the onset of the next seminar problem and the shifting focus to the next formal task. It is more likely, however, that students may have been reluctant to comment in public on what is essentially the feedback from an assessed piece of work.
Although difficult to measure, the module delivery team felt that the students were more confident in their ability to express an opinion, in their replies to comments from other group members, and in their general participation. Certainly students participated more (in terms of frequency of posting and viewing) when compared to traditional seminar attendance and participation, although the asynchronous nature of a week-long debate may contribute to the increase, and as such is a confounding factor.
3.3 A greater level of knowledge and understanding will develop
Oliver and Shaw cite a common problem with the summative assessment of discussion boards: "contributions were not strongly interactive and that students were simply 'playing the game' of assessment, making postings that earned marks but rarely contributing otherwise" [17]. Despite a low average weekly posting (1.74, above) it was evident that students were exceedingly well prepared for the online discussions. The sample discussion below illustrates typical themes that emerge from written discussions and type of skills and resources employed to gain a deeper understanding.
Not all discussion boards followed this format, but the flow from analysis through critique to reflection was a general trend with queries arising at several points, often breaking the reflection stage into three or more parts as and when alternative issues were identified. Meyer's evaluation of doctoral-level discussions noted that "54.3% [of postings] were at levels four through six in Bloom's taxonomy... which was seen as appropriate to the level of response expected of doctoral students" [18]. Bloom's taxonomy of educational objectives are commonly cited and consist of the following hierarchical terms: Know, Understand, Apply, Analyze, Evaluate and Create. [19] Most of the student discussion boards reached level 4, with several boards creating inferences, evaluations and solutions to associated problems that were not necessarily evident from the problem scenarios themselves.
The ability to insert hyperlinks to online resources was a frequently used tool. Not only were students referring to materials and providing a facility for their group members to locate and access online resources, but group members were indeed doing so and returning to the discussion with commentary and analysis of the materials in context. This was seen as a significant difference to the activities of classroom seminars. Where students, in a classroom, may refer to materials they have read, other students cannot necessarily respond in depth if they have not accessed the materials themselves. Whilst this is not generally a problem for materials listed as core texts, it does stifle the group discussion of materials independently researched.
In relation to the student perception of online learning, the survey allowed for a free text response to indicate their preference for modes of learning. Overwhelmingly, the students enjoyed the experience and felt they had made substantial gains that would not have been achievable in a traditional setting:
"I prepared more thoroughly and other group members' postings encouraged me to learn more to reply to them… I also think my seminar time stretched to at least four or five hours per week."
"I also feel I have learnt more and there is the opportunity to go back and look at information. In class it is not always possible to take down what everyone says… it helps with studying."
"I ended up thinking, reading and talking to other students more about the topics than I would have done for face-to-face seminars."
3.4 Feedback for participation and assessment can be formalised and more effective
The summary tasks were assessed as part of the coursework component for the module. A feedback mechanism was developed to enhance the learning experience and provide formal feedback to students above and beyond traditional classroom seminars.
The tutor responded publicly to each summary, giving constructive written feedback on the substantive issues. If groups arrived at different conclusions or adopted a novel approach to the problem scenario, then comparative comments were also made. On occasion, students would respond to the tutor's comments thereby generating discussion outwith the allocated time period, generally on reflective issues. Private individual performance was also generated along with grades for participation which was made available at the end of the course.
The tutor found that their role as a classroom discussion facilitator was no longer required and that this time could be spent providing feedback, therefore no overall increase in time commitment to the module was evident. In fact, comments from the tutor suggested that less time was spent on a more effective task than preparation and attendance at timetabled classroom seminars. 80% of students found the tutor's comments to be useful, especially in relation to aspects that were not discussed across all groups - this will be discussed in more detail in the next hypothesis.
3.5 The students will generate a knowledge base for coursework preparation and examination revision
It was not anticipated from the start that students would be creating a knowledge base, but this became apparent during the first few weeks of discussion when the use of hyperlinks and authoritative quotes emerged as a consistent trend. It was hypothesized that students would then use the summaries as a starting point or guide for exam revision as tutor comments were to be found here, rather than in the main body of the group discussions.
75-80% of students read the summaries of other groups and found them to be useful. Responses from the survey also indicated that the summaries, and perhaps more surprisingly the discussion boards, would assist in exam preparation.
Other than student comments, it is difficult to evidence that students used the boards as revision aides. From the Blackboard statistics tool, it was apparent that students did access the system beyond the last day of teaching. The module tutors suggest that such unexpected accesses relate to revision activities - whether such use is indicative of the boards acting as a knowledge base or as a starting point to access cited materials remains unclear. Certainly students were viewing both the summaries and the original discussion from their group during the period between the end of teaching and the exam date, although the number of hits was relatively low given the number of students matriculated on the module. It may be that a majority of students had previously downloaded or printed copies of the discussion and/or summary posts during the discussion period.
4. Conclusions and Review
In review, there is strong evidence for the five hypotheses postulated by the module team. The QAA subject benchmark statement for law has three subject specific areas of performance: Knowledge, Application and Problem Solving, and Sources and Research. [20] The added benefit of an asynchronous online seminar has contributed equally to all three areas, despite an expectation to focus primarily on the problem solving aspect. The students' knowledge may be enhanced due to innovative and exciting modes of delivery, however the shift from tutor-led seminars to self-directed learning has improved the ability of students to answer complex problems. The main benefit, in QAA terms, is the increasingly independent abilities of students to demonstrate the key elements of Sources and Research:
Whilst these are achievable in traditional seminar settings, they can be demonstrated to a greater extent in the online environment where the distribution and sharing of sources can be facilitated thus broadening the knowledge of students and enhancing further their ability to apply knowledge and solve problems in a group setting.
Much of the success may lie in the nature of the problems set for the groups to discuss. Lewis[21] discusses the concept of wicked problems:
"The debate and discussion is wicked in that the problem and question keep 'morphing' as science develops, attitudes evolve and economic interests shift… there is no single 'answer' that is demonstrably superior to many others".
Lewis applies this theory to the collection, maintenance, ownership, control and use of human tissue, which is a pertinent topic for the Healthcare Law and Ethics module, yet also observes that the theory of wicked problem analysis is less frequently applied to social sciences. Ten typical characteristics of wicked problems are reviewed by Lewis in relation to human tissue and many of these characteristics apply to the weekly problem scenarios used by the module team as a starting point for online discussions. It is therefore suggested that problem scenarios be difficult to define simplistically, have no stopping rule (i.e. no definable solution), no immediate test of a given solution, and a variety of known and unknown consequences; thereby ensuring a high level of discussion on a variety of intellectual levels.
Conversely, a study by Williams and Pury[22] found student uptake to be poor, fewer postings than expected and low student satisfaction. They did, however, find a benefit to students when discussing topics that may be controversial as this study also had elements of medical and socio-legal interest such as birth control and gender issues. It would appear that their use of simplistic and direct questions, for example "if you had an infant son, would you have him circumcised? why or why not?", may have been too personal, and this was later altered in subsequent iterations of the taught course to a series of less personal problem scenarios that appear to have some of the 'wicked' characteristics discussed above. This also compels students to interact, rather than post personal singular replies.
From this, it may be suggested that using technologies to avoid face to face discussions on controversial or possibly personal topics and move debate towards an online environment, where all comments are permanent in written form, that benefits can be gained by avoiding heated or personal debates. Whilst all academic teaching should strive to avoid such confrontations, the topics included within the study of healthcare law and ethics (and other topics within the law curriculum and elsewhere) can have the potential to cause distress. No doubt there is also the potential for equal distress, if not greater, to be caused by the use of written, online discussion boards and the asynchronous nature may not necessarily provide a timely response, remedy or rebuke if one is indeed necessary. The findings in this project, as discussed above, certainly show that students took great care in their written work and this suggests that a mechanism is in place for self-checking and peer regulation of the discussion boards.
To conclude, the transition from classroom seminars to asynchronous discussion boards has brought considerable benefit to the student learning experience and has also been well received by staff delivering the module. Recommendations can be drawn from this study that are broadly applicable to both legal education and higher education in general. The use of written group summaries has encouraged a style of altruistic participation as student posts are not 'mark-seeking', although posts are necessary in order for one group member to be able to construct a summary of the debate. Wicked problems may well be commonly found in the law, although taught modules that do not focus on jurisprudential aspects may struggle to avoid scenarios or set questions that do not have a definitive or easily identifiable answer unless a wider context such as public policy or ethics is incorporated.
References
[1] This study was funded and supported by the Caledonian Academy's Scholars and Associates Scheme during 2007/08 and 2008/09. The author would like to express his thanks to Alison Britton and Linda Creanor from Glasgow Caledonian University, and Patricia McKellar from UKCLE for their support.
[2] Neufeld, V & Barrows. H (1974) “The McMaster Philosophy: an Approach to Medical Education”, in 49 Journal of Medical Education 1040
[3] See for example: Savin-Baden M (2000) Problem-based learning in higher education: untold stories Buckingham: Open University Press and Schwartz P et al (eds) (2001) Problem-based learning: case studies, experience and practice London: Kogan Page
[4] Pallie, W & Carr, D (1987) “The McMaster Education Philosophy in Theory, Practice and Historical Perspective”, 9 Medical Teacher 59
[5] Astin, A (1987) competition or Cooperation? Teaching Teamwork as a Basic Skill 19(5) Change 12
[6] Sample scenarios problems and other related materials can be viewed online at http://bromby-scholar.pbworks.com
[7] A gobbet is frequently used in the history discipline (although this exercise was not an analysis of original documents) where students are required to write a short commentary on an assigned primary source. It should include a review of context, content and note the wider implications. They are generally brief (~500 words) and writing an introduction or conclusion is not required. See also: http://users.ox.ac.uk/~magd0968/gobbets.html
[8] Kolb, D, (1984) Experiential Learning: experience as the source of learning and development New Jersey: Prentice-Hall (0 13 295261 0)
[9] Cruickshank, D. (1996) “Problem-based learning in legal education” in J Webb and C Maughan (eds) Teaching lawyers' skills London: Butterworths
[10] MacFarlane J & Manwaring J (1998) “Using problem-based learning to teach first year Contracts”, in Journal of Professional Legal Education 16:271-298
[11] Ibid at page 285
[12] Also available online at http://bromby-scholar.pbworks.com/Rules
[13] Mayes, T (2006) Flexible Delivery - An Overview of the Work of the Enhancement Theme 2004-06, Quality Assurance Agency for Higher Education QAA123 08/06. See http://www.enhancementthemes.ac.uk/documents/flexibleDelivery/flexible_delivery_QAA_123.pdf
[14] The Student Evaluation Project is a university funded research initiative to identify further information about Glasgow Caledonian University students. It was set up in 2001 and to date has published reports on the student body at the university, what they like or dislike, and life in general at GlasgowCaledonianUniversity.
[15] Jones, D “Enhancing the learning journey for distance education students in an introductory programming course”, (http://davidtjones.wordpress.com/publications/enhancing-the-learning-journey-for-distance-education-students-in-an-introductory-programming-course/) , citing Angelo, T., & Cross, K. Classroom Assessment Techniques: A Handbook for College Teachers (2nd ed.). San Francisco: Jossey-Bass. and Murphy, L., & Wolff, D. (2005). “Take a minute to complete the loop: using electronic Classroom Assessment Techniques in computer science labs”, Journal of Computing Sciences in Colleges, 21(1), 150-159.
[16] Szabo A (1993) “Teaching substantive law through problem-based learning”, in Journal of Professional Legal Education 11:195
[17] Oliver, M. & Shaw, G. (2003), "Asynchronous Discussion in Support of Medical Education", in Journal of Asynchronous Learning Networks , vol.7, no.1.: http://www.aln.org/publications/jaln/v7n1/v7n1_oliver.asp
[18] Meyer, K.A. (2004), "Evaluating online discussions: Four different frames of analysis", in Journal of Asynchronous Learning Networks , vol. 8, no. 2 (April 2004): http://www.aln.org/publications/jaln/v8n2/pdf/v8n2_meyer.pdf
[19] Anderson, L. W., & D. R. Krathwohl (Eds). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. New York: Longman, 2001
[20] Quality Assurance Agency for Higher Education (2000) Academic Standards - Law http://www.qaa.ac.uk/academicinfrastructure/benchmark/honours/law.pdf
[21] Lewis, S (2008) “Wicked problem” 48 Jurimetrics 193
[22] Williams, S. & Pury, C. (2002), "Students' attitudes toward and participation in electronic discussions", in International Journal of Educational Technology, vol.3, no.1: http://www.ao.uiuc.edu/ijet/v3n1/williams/bromby_3.html
|