United Kingdom Journals

Benchmarking: A Pedagogically Valuable Process?

John Bell

Faculty of Law
University of Leeds

<j.s.bell@leeds.ac.uk>

Copyright © 1999 John Bell.
First Published in Web Journal of Current Legal Issues in association with Blackstone Press Ltd.

Summary

The QAA has embarked on a process of replacing teaching quality assessment by a process of benchmarking. This looks at standards of student achievement and whether they are appropriate to the award which a student is given. In this paper it is argued that giving thought to the standards which students should attain can be pedagogically beneficial for law teachers and students alike.

Contents

Introduction

Learning Outcomes

Articulating standards

Benchmarking as a way of encouraging articulation of standards

Setting a benchmark

A Holistic Perspective?

Accentuating the Positive

What do we claim? What do we assess?

Conclusion

Bibliography

Appendix - Benchmark Standards

Draft Statement for a Law Degree

Guidance Note for Law Schools

Example of Levels of Achievement: Skills

Introduction

Under the influence of the Dearing Report, the Quality Assurance Agency has embarked on a process of replacing teaching quality assessment by a process of benchmarking which looks at the standards of student achievement and whether they are appropriate to the award which a student is given. There is a clear political imperative to address the standards issue, clearly seen in the Dearing Report, which recommended that benchmark information should be produced to 'identify expectations for awards and achievements at the threshold and highest end of the spectrum for different types of programme (Dearing, 1997, para. 10.91; see also Ch. 10 and recommendation 25, and DfEE 1998, para. 4.4). For good or ill, the expansion of higher education in recent years has led some to question whether standards are slipping. Those who argue that `more means worse' in terms of student achievement are hard pressed to present evidence to show that students, for example with a 2.1 degree, are not as able as they used to be, but that does not stop the doubts. In addition, there have been a number of spectacular failures of the higher education sector to maintain standards properly, the most recent being Thames Valley University. At a time when students, their parents and sponsors are putting more of their own money into paying for higher education, there is a perceived need for systems which provide public reassurance that the qualifications received are worthwhile. There is also the political issue of whether a process which assures standards of student achievement will be seen as an adequate replacement for the much-criticised process of teaching quality assessment. But my concern in this brief paper is with the pedagogic rationale for benchmarking. The argument will be that giving thought to statements on the standards which students should attain can be pedagogically beneficial for teachers and students.

Top | Contents | Bibliography

Learning Outcomes

Many of us have become familiar with the practice of setting out learning outcomes when we design modules or programmes. Indeed, these have often replaced `aims' and `objectives' as headings, and are more specific. By saying what students ought to achieve at the end of the module, we set out a standard against which the course content and teaching methods can be judged (see Otter, 1992 and Jones, 1994).

The first report of the Lord Chancellor's Advisory Committee on Legal Education and Conduct on the initial stage of legal education, published in April 1996, suggested a radical change in emphasis from a focus on the content of the law curriculum to a focus on learning outcomes. The suggestion was that a university programme would satisfy the requirements for being a qualifying law degree if certain outcomes related to general transferable intellectual skills and some broad definitions of subject content were demonstrated in a programme which contained at least two-thirds of law subjects. Unlike the Joint Announcement, there was no specification of the content of legal subjects (see ACLEC, 1996, p72). This was welcomed by legal academics as providing much greater freedom to define the content and the assessment of the legal curriculum. The secretariats of the professions also welcomed this as a way forward and the revised Joint Announcement is being drafted in discussions between the professions and representatives of academics on that basis.

The tradition of law teaching in this country has focused heavily on content. This has been reflected in the requirements for qualifying law degrees (Bar Council, 1995; for earlier versions see Birks, P. (1992). The `foundations of legal knowledge' in the current Joint Announcement are much less prescriptive than what was found in the past when more detailed syllabuses where produced by the professional bodies. But even academic practice has focused on a core of knowledge which is required to have a proper foundation in law (see Birks, P. (1996) xii-xvii). A focus on learning outcomes which the student must demonstrate moves the focus from content to what the student should be able to do as a result of her studies. In Leeds, we have for some time had a statement of competences which a student should have achieved as a result of her degree programme. But we have also noted that, mapping these outcomes onto our traditional course provision with subjects defined in traditional ways, there are some achievements which are given great attention and others very little. If we have taught contract, torts and criminal law in much the same way, and assessed them in much the same way, then we cannot be surprised that duplication occurs in terms of learning outcomes. We say that students should be able to conduct research, but have not, until recently, made a dissertation or even the study of research methods an essential part of the programme of study. I would argue that the focus on content is not so much the product of the influence of the professions, but stems from an academic culture in which the content of `knowledge' takes precedence. In the model presented by more than one Vice-Chancellor, the university in a `knowledge society' has the function of generating, conserving and transmitting knowledge. Attention is then paid to defining the information which needs to be known. But this idea needs to be questioned. As the Moser Report pointed out: `What is not acceptable is [an academic] department which merely recycles what is known and fails to equip students with the spirit of inquiry which will inspire them to continue to learn throughout life.' (Moser, 1994, p309).

Now in Law, more than many other disciplines, we know that knowledge is ephemeral. It is the ability to find, analyse and criticise new legislation and case-law which is critical in a good lawyer, rather than just knowledge of established law. To describe the achievement of a good student in terms of learning outcomes forces us to pay attention to what a student can do with knowledge, rather than just on what she knows. Even those who are wary of an emphasis on `employability skills' as an appropriate function of higher education do insist that it involves more than knowledge and will emphasise the importance of critical thinking and other such intellectual attributes which university education should foster (see Barnett, 1994, ch.7).

It is therefore not surprising that the culture of learning outcomes has gone hand-in-hand with greater attention to the abilities and skills developed through education. When we describe the attributes of a student, we typically focus on these skills to show how well they understand the content of what they have learned. As we seek to define more carefully the skills which are supported through higher education, we are better able to explain the richness of the learning experience. We also relate more effectively to the requirements of employment, where over 40% of graduates will go into work which is unrelated to their degree subject. For this reason skills must be a major part of the learning agenda (see CVCP (1998).

It has to be recognised that, though learning outcomes have the advantages of making more explicit and transparent the way in which student learning is promoted and assessed, there are disadvantages. In particular, there is a concern that the outcomes will be too specific and that the unexpected and creative outcomes of the learning process might be unsupported or under-valued. Are the outcomes too constraining a framework? (see Atkins et al, (1993), pp45-6).

Top | Contents | Bibliography

Articulating standards

The process of requiring learning outcomes to be set down has also been part of a process of trying to articulate the standards which we deploy. Standards are defined by HEQC as `explicit levels of academic attainment that are used to describe and measure academic requirements and achievements of individual students and groups of students' (see HEQC, 1997a, vol 1, para 4.1). They are the way in which the achievement of learning outcomes are measured. The articulation of standards is seen as good practice in ensuring that appropriate criteria are used to judge students and that there is consistency of approach. Articulating the criteria through marking schemes can be a way of determining what staff treat as key criteria in marking and these can be communicated helpfully to students (UK Universities' Staff Development Unit, 1993, section 3.6). The Graduate Standards Report on degree classification by Phil Jones and others (HEQC, 1997b) examined marking criteria used by institutions and assessment practice. In particular, it reports the lack of articulated criteria in many institutions, but there was reliance on shared assumptions which were increasingly under threat from changes in the system (HEQC, 1997b, ch.3). Many assessment practices assume that internal and external examiners share assumptions about what is a good answer in a particular subject. But as practices across institutions diverge and the sector has expanded, there has been less tacit agreement about what a proper answer should be in relation to the type of syllabus in place. Indeed, that report noted that many practices which had served to develop shared understandings, such as scrutiny meetings involving all staff, no longer took place. All the same, the report does note that the implicit ideas of staff, which are sometimes articulated in marking criteria, do already conform closely to ideas of `graduateness' which are found in reports such as the HEQC report on Graduate Standards (HEQC, 1997b, ch.5). There is thus the basis for taking further ideas of graduateness by formalising them in a way which is grounded in the assumptions and actual practices of the academic community.

In the past, feedback on assessment might not even have included marks on individual papers and certainly not comments. How then were students supposed to understand what they had to do in order to achieve good marks? They relied on formative assessment from tutors and discussion with them. But the value of such feedback depended both on the fullness and the timeliness of what the tutor wrote, and the consistency of that feedback with the approach taken by examiners. With more students and other pressures, the amount of detailed feedback is likely to decline. Expressing standards enables a more systematic way of articulating to students what is required of them, especially in a system where face-to-face discussion on a small scale is declining.

Top | Contents | Bibliography

Benchmarking as a way of encouraging articulation of standards

Benchmarking is a process of articulating standards at national level, using the language of learning outcomes. Like any academic process, benchmarking has been based on research and discussion. This has been supplemented by consultation. The development of benchmarks brought together expertise developed in four separate exercises in recent years. In the first place, the ACLEC first report (ACLEC, 1996) was produced by a group of people who had experience in producing outcome statements in relation to law, and its approach, structure and terminology was an obvious starting point for further work. Secondly, the Graduate Standards Report has already been mentioned, with its work on degree classification. The larger Graduate Standards Report had led to work done by consortia of subject associations. In Law a short feasibility study was conducted producing a report (Graduate Standards in Law). The group conducting this work had developed some broad criteria of graduate standards at modal level based on consultation with a group of experienced external examiners. This demonstrated clearly that conceptions of graduateness held by legal academics were set at modal level, but were largely implicit. All the same, when confronted with actual scripts, there was much similarity among law examiners for whom accurate content was a major criterion for marks. There was a view common among many colleagues that law does not have a progression of levels as in standard CATS systems, but a single standard to be applied, that of competence in the subject which was applied to all law students, such that a student studying contract in the first year would be distinguished from a third year student doing that course by the marks achieved rather than by a distinct level of achievement. There was also some sense that there were differences in expectation of students on law programmes and non-law programmes attending the same modules. The fourth element was the DfEE sponsored Discipline Network in Law on General Transferable Skills in the Law Curriculum. This had produced a report in March 1998 (General Transferable Skills in the Law Curriculum; see also Bell 1997) after two years which set out some basic statements of requirements in relation to core general transferable skills which the law curriculum might develop. This study had been the subject of both questionnaires and workshops involving a wide range of law schools.

The combination of these materials enabled the subject benchmarking group to have some background research and information on opinions in the sector and to provide materials encapsulating ideas on standards with specific reference to law which could be discussed. Although there had been some consultation or enquiry with the sector which had produced feedback on both the Graduate Standards in Law report and the General Transferable Skills in the Law Curriculum, it would be wrong to say that the ideas in those reports were widely shared and applied in the sector.

In any case, a consensus among law teachers would not be a satisfactory basis for a statement of standards which would re-assure the wider public. It is here that problems are encountered. There are numerous studies of the employability of students, but no agreement among employers surveyed about what they expect students to demonstrate at the end of their course (see e.g. Hawkins and Winter 1995). Once one moves to the wider public, then this is even more diffuse. As a result, it is not surprising that the Benchmarking Panel did not contain employers from outside the legal professions, since they did have specific ideas about what students should achieve at the end of a Law degree.

As was highlighted in the report on Graduate Standards in Law, there is a need for staff development to move into this approach. It was interesting that the majority of those who thought the initial statement produced by the Subject Benchmarking Panel in July 1998 was pitched too high came from the old university sector, less familiar with the use of learning outcomes, whereas many of those from former CNAA tradition institutions argued that, if anything, it was pitched too low.

Top | Contents | Bibliography

Setting a benchmark

A benchmark is in essence a standard against which one measures oneself. In the Law Benchmarking panel, it was recognised that the vast bulk of final year law students achieve 2.2 degrees or better. All the same, it was decided to set the published benchmark at the minimum honours pass. The Subject Benchmarking Group took as its remit to set minimum standards for law degrees (a statement of `what a law student is at least able to do'). It was concerned to address the wider public concern that degrees could be obtained too easily, e.g. in overseas franchises or validated degrees. From the findings of the Graduate Standards Programme and the study of graduate standards in law, the Group accepted that academics think in modal terms, concerned to identify the typical student and defining achievement in relation to that person. The Group considered that, with the diversity of law programmes, a modal statement might be difficult to write in a non-contentious form. It took the view that the minimum could be defined in prescriptive terms, and then in future modal descriptions could be produced for illustrative purposes. Given the concern among law schools to obtain greater flexibility in the design of their programmes, they were wary of anything equivalent to a national curriculum for Key Stage 7. A prescriptive modal statement might appear too close to a national curriculum for a first step in the circumstances. Furthermore, there was concern about how a modal statement could be used to judge a degree programme. If a modal statement means that `a student is typically able to...', then what public reassurance does it provide? The employer who is told that law students from Leeds are usually able to work with primary legal sources is not going to be amused to be told that the student he has recruited and who cannot begin to work with primary sources is one of the unfortunate minority of students who cannot perform what typical Leeds students can do! The matter to be judged by benchmarking is the performance of students, rather than the aspirations of law schools.

On the other hand, it was fully expected that law schools would set their own statements of standards at a modal level. Indeed, statements sent by institutions as examples were often pitched at a 2.1 standard. For them, the benchmark would not so much be a standard to measure up to, but a standard below which they could not fall. It would serve a starting-point for their own reflection. That said, it would also serve as a framework of issues which any institution's own statement of standards would have to address. It is perhaps in this second area that the element of challenge is contained in the statement.

Naturally, the professional bodies would wish to pitch their minimum requirements for entry at a higher level of achievement than the threshold. This matches their practice of setting a 2.2 as a minimum entry qualification for entry to the Legal Practice Course, and (in practice) operating with a 2.1 requirement to obtain entry to major law firms and chambers. It was not felt that this level of achievement should be the focus of discussions about who should obtain an honours degree. Any professional statement would select among those who had achieved a law degree, rather than define the groups of students who merited such a qualification.

Among legal educators, the idea of benchmark standards fitted into the same approach as ACLEC in focusing on generic outcomes, rather than predominantly on specific content. Unlike the Benchmark standards in Chemistry which are very specific on content, or the Benchmark in History, which has a lot to say about the learning and assessment process, the Law Benchmark is deliberately flexible in setting out what students should learn and how they should learn it. A national curriculum (though perhaps without a daily literacy hour) is not what professionals in higher education wish to see imposed.

Top | Contents | Bibliography

A Holistic Perspective?

Staff traditionally decide whether a student obtained a particular class of degree on the basis of a number of credits passed on a range of modules. While some institutions are used to mapping intellectual and other skills onto their modules and developing a holistic view of a student's achievement, most are not. A student achieves a degree (and in particular a qualifying law degree) by passing a range of modules defined by their content (contract, torts, etc.), rather than by their skills (analytical ability, independent research, etc.). The working concept of a bare honours pass for most law teachers is thus experienced in terms of individual modules, rather than in terms of the student performance as a whole. But, when writing references on students, law teachers are often required to comment on the skills of students in communication, analysis, evaluation, team working and so on. The law degree is thus a process through which a variety of judgements are actually being made on the abilities of students which are reported to the non-academic world, not merely those based on examination performance. Benchmarking tries to make coherent sense of these claims which are being made for law graduates and to encourage law schools to make more explicit the claims which they do wish to make for their graduates and the evidence base on which they rely. As such, it serves as a focus on the student learning experience and achievement, rather than on the modular form of delivery.

There have been dangers that the modular system, while expanding student choice and flexibility, has meant that institutions have lost sight of the individual as she progresses. Benchmarking helps us to refocus attention on the learner and what she is like at the end of the process.

Top | Contents | Bibliography

Accentuating the Positive

In Law, it has been difficult to `touch bottom' with a formulation which captured the qualities of the bare honours pass student. By and large, it was relatively easy to set minimum standards in key skills or in the range of knowledge which had to be exhibited. The problem area was the intellectual skills. How much ability to integrate material in a coherent way is it reasonable to expect when incoherence is one of the traits associated with the student at this bottom level? Issues such as `creativity' were a problem. Original and creative thinking are associated with high performance. Thus originality is a particular merit of the first class candidate. On the other hand, the research of Holmes and others associates `graduateness' with `transformation potential' (Harvey et al, 1997, p77) and this is reflected in modal statements produced by the feasibility study where it is suggested that a graduate in law should be `able to create new or imaginative solutions through approaching a problem or using material in different ways'. The law teachers have a tendency to use the first class or upper second class students as a reference point and so how far the bare honours pass student fell below that, rather than distinguish the graduate and the non-graduate. Since many law teachers have no experience of teaching at non-degree level, their ability to define what is special about graduates is limited and this perhaps undersells the value of graduates. A benchmark statement is meant to indicate what students can do at the end of the degree-level education and this serves to justify the public expenditure involved in that process. The benchmarking statement and the individual institutions' own statements of standards are meant to articulate the value-added by academic legal education.

The approach thus requires one to move away from the description of the third class student in deficiency terms (e.g. `misses key points, contains important inaccuracies' (HEQC, 1997b, Appendix A). If a degree is an important social achievement, we need to be able to describe in positive terms what those achievements are. Describing such threshold honours pass students in positive terms was a marked departure from the various classification criteria which were available to the Group. These criteria were at their least precise in describing the third class student. The effort to identify a third class student's abilities provides a platform from which to build up the model of achievement of the better students. The initial work of the Group in defining the bare pass student in a prescriptive form was then built upon in the Guidance Note by the provision of an illustration of how better levels of performance might map onto this. Like the Chemists, the lawyers did not wish to focus too precisely on the traditional class boundaries used in classification schemes. By describing the higher levels as `proficient' and `very proficient' there was sufficient scope for flexibility in the use of class boundaries as appropriate to the different kinds of degree scheme. If the future work of the Subject Benchmarking Group is to develop further criteria beyond the threshold level, then the clear relationship to the threshold in terms of additional characteristics or higher levels of performance will be crucial. It may be that this in itself helps to publicise the value of a graduate beyond the legal profession.

Top | Contents | Bibliography

What do we claim? What do we assess?

One major issue which was much debated concerned assessment. If institutions were to claim that students had achievements in certain areas, did they have to assess all these areas? The answer from the Group was that there needed to be evidence on which an institution could state with confidence that a student had reached the right level, but that this need not be assessed in a conventional way. A report in a student progress file would be sufficient for many of the areas. After all, the Dearing Report did expressly recommend that student learning should be supported and recorded in this way and the matter is being taken forward by QAA (a development paper on this issue will be on the Sheffield University web-site from Spring 1999; on the general issue see Dearing 1997, paras. 9.42-9.50 and recommendation 20). The lack of familiarity with this idea, despite its currency for many years, led law schools to limited thinking in the area of assessment. It is right that if we claim that students have certain achievements on graduation, then we should also have evidence to support that claim. But there is significant progress being made in defining processes which can produce this result without trying to load subject syllabuses and assessment procedures with more and more items.

Top | Contents | Bibliography

Conclusion

This article has tried to demonstrate the importance of learning outcomes as a method by which staff can articulate the learning objectives of their modules and programmes. They can be a way of explaining to students what the course is designed to achieve and the standards by which their performance will be judged. Learning outcomes should be intimately linked to assessment criteria. The benchmarking process seeks to build on this important development in supporting student learning by identifying certain specific areas of performance in which students should be expected to demonstrate achievement. The benchmark standard is not so much a model to be copied by law schools, but a starting point for their own reflection on what a particular law school wishes its students to achieve, and what it is prepared to say to the outside world that they have achieved. Naturally, this will lead law schools to reflect on the ways in which different aspects of the student profile are going to be demonstrated. Which aspects should be demonstrated through summative assessment in specific areas of substantive law and which might be demonstrated through a progress file or similar record of achievement developed with support either from the law school or from the higher education institution more generally. In this way we may be able to support student development in a holistic fashion. It would be a mistake to make compliance with an external (and perhaps rather unfocused) agenda on quality the principal objective of a law school's activity. Rather we should look to promoting student learning as our principal objective.

Top | Contents

Bibliography

ACLEC, (1996) Lord Chancellor's Advisory Committee on Legal Education and Conduct, First Report on Legal Education and Training (ACLEC, London 1996), See also <http://www.law.warwick.ac.uk/ncle/html/aclec_report.html>
Atkins, M. et al. (1993) Assessment Issues in Higher Education (Newcastle)
Bar Council (1995) Bar Council for England and Wales and Law Society, Joint Announcement on Qualifying Law Degrees (January 1995).
Barnett, R. (1994) The Limits of Competence (Open UP, London).
Bell, J. (1997) `General transferable skills and the law curriculum' II Contemporary Issues in Law 1.
Birks, P. (ed.), (1992) Examining the Law Syllabus. The Core (Oxford U.P.,Oxford).
Birks, P. (1996) `Editor's Preface', What are Law Schools for? (Oxford U.P., Oxford).
CVCP, (1998) Skills Development in Higher Education (London 1998).
Dearing (1997) National Committee of Inquiry into Higher Education, (The Dearing Committee) Higher Education in the Learning Society (HMSO, London).
DfEE, (1998) The Learning Age: Higher Education for the 21st Century. Response to the Dearing Report (HMSO, London).
Graduate Standards in Law:- <http://www.law.warwick.ac.uk/ncle/html/grad_standards.html>
General Transferable Skills in the Law Curriculum:- <http://www.law.warwick.ac.uk/ncle/html/ldn.html>
Harvey, L. et al., (1997) Graduates' Work: Organisational Change and students' attributes (CRQ).
Hawkins, P. and Winter, J. (1995) Future Graduate Skills in the 21st Century (Association of Graduate Recruiters, London).
HEQC, (1997a) Graduate Standards Programme, Final Report (London).
HEQC, (1997b) Assessment in higher education and the role of `graduateness' (HEQC, Bristol).
Jones, P.A. (1994) Competences, Learning Outcomes and Legal Education (IALS and DfEE, London).
Moser,(1994) Report of the National Commission on Education, (the Moser Commission) Learning to Succeed (Hamlyn, London).
Otter, S. (1992) Learning Outcomes (DfEE, London).
UK Universities' Staff Development Unit, (1993) Handbook for External Examiners in Higher Education (TLTP Project Alter, Sheffield).