|
INQUIRY INTO A LEVEL STANDARDS Interim Report Mike Tomlinson September 2002 Please find herewith my initial
report into the events surrounding the grading of A2 and AS units
in this year’s GCE examinations. I
ask that you consider as a matter of In the light of this report I will be considering if I need to ask you to refine the terms of reference for the second stage of my inquiry. I will write to you separately on this matter in due course.
INTRODUCTION
1.
This initial report into the setting of the A level standards
in July 2002 results from the independent inquiry I was asked to undertake
by the Secretary of State on the 19th September. This follows concerns
expressed publicly by a number of headteachers of independent and
maintained schools following the publication of results.
2.
The precise Terms of Reference for the inquiry are: 1. To investigate allegations about the
setting of standards for A-level grades this year. In particular, to make sure that the conversion
from marks to grades was determined according to proper standards
and procedures. A first report
on this will be provided to the Secretary of State by Friday 27 September. 2. To investigate
the arrangements at QCA and the awarding bodies for setting, maintaining
and judging A level standards, which are challenging, and ensuring
their consistency over time; and to make recommendations by November
to the Secretary of State and Ken Boston, Chief Executive of the QCA,
for action with the aim of securing the credibility and integrity
of these exams.
3.
This report is concerned only with the first point. I must
stress that throughout the inquiry and in the preparation of this
report the paramount consideration has been those students who took
GCE A-level examinations this year, their parents and their teachers.
Even if the concerns expressed relate to a proportion A-level candidates
this year, all do deserve to be awarded a grade commensurate with
the standard of their work and one which is consistent with the standard
employed in previous years. I have not therefore regarded my inquiry
as an administrative exercise. Rather I have regarded it as vital
that the anxiety and concern felt by many students were if possible,
resolved quickly and conclusively.
4.
My inquiry has taken written evidence from a wide variety of
sources and has included face to face meetings with the Headmasters
and Headmistresses Conference (HMC), Secondary Heads Association (SHA),
National Association of Head Teachers (NAHT),
the teacher and lecturer associations, the Association of Colleges
(AoC), Chief Executives of three examination boards in England,
a sample (8 in total) of Chief Examiners, officials from the QCA,
including the Chairman and the former Chief Executive Professor Hargreaves,
officials from the Department for Education and Skills and Ministers.
5.
I must record my sincere thanks to everyone for their openness
and their readiness to provide me with data, papers and notes of meetings.
I also wish to thank all the headteachers, teachers and others who
have submitted written evidence. I intend making evidence available
in due course, where no desire for confidentiality was expressed by
the contributor.
6.
Finally, I would wish to record my thanks for the exceptional
hard work of the colleagues who have supported me in this inquiry:
Lisa Couchman, Darren Goff, Mark Hayward,
Nancy McLean, Peter O’Connor, Kate Taylor, and Matthew White. RECOMMENDATIONS
7.
I recommend to the Secretary of State that:
·
Students’ marked work in some A2 units in some subjects, where
changes to grade boundaries appear to have been out of line with historical
patterns for the legacy A level, is regraded
and their overall GCE A level grade adjusted if necessary. If agreed, I will be seeking further information
from the awarding bodies, by the end of 30 September, about changes
to the mark grade boundaries over recent years. With this information, and after discussion
with the awarding bodies and the regulator, I will advise the Secretary
of State by the end of Tuesday 1 October, on the extent and manner
of the recommended re-grading, including the subjects affected and
the number of students involved. At
the same time I hope to be able to offer advice on the timetable for
such an exercise: I believe it is likely to be done quickly;
·
The QCA be asked to provide additional guidance on the standard
associated with AS and A2 units, generically and in all subjects;
·
The awarding bodies jointly in association with the QCA, be
asked to undertake further work on the statistical issues associated
with the current assessment arrangements for the two awards;
·
Further training and guidance be made available to examiners
and teachers;
·
QCA be asked to revise as necessary the Code of Practice in
the light of the above work and the findings of my inquiry;
·
Efforts be made to improve the general understanding of “Curriculum
2000” and the difference between ‘maintaining a standard’ and ‘the
proportion of students meeting the standard’, as this is at the root
of the annual debate when public examination results are announced. BACKGROUND
TO THE DEVELOPMENT AND STRUCTURE OF THE CURRENT AS/A LEVEL SYSTEM
8.
The current “Curriculum 2000” reforms of the A level system
were introduced in September 2000 with the first AS and A level awards
being made in summer 2001 and summer 2002 respectively. The principles behind “Curriculum 2000” were
wholeheartedly endorsed during my inquiry.
9.
Because the structure of the new A level and the manner of
it’s implementation have emerged as significant issues in the course
of my inquiry, paragraphs 10 to 12 below set out the essential characteristics
of the new system.
10.
The current A level is divided into two parts: three units at AS level which, together, equate
to the first year of a traditional A level course and three A2 units
awarded during the second year of study.
Taken together these six units comprise a full GCE A level
and form the basis for an A level award.
The three units studied in the first year at AS level can,
if the student wishes, be “cashed in” to provide a certificated qualification
in its own right. Each unit of the award is equally weighted, with
the AS and A2 programmes each accounting for 50 per cent of the overall
grade.
11.
This system was established with the intention that students
would take a broad range of AS level courses during the first year
of study – up to four or five. They
are then able to narrow their studies in the second year by selecting
the subjects which they will pursue to the full GCE A level standard,
whilst receiving a qualification for subjects they pursue no further. Students may also retake units to seek to
improve their grade.
12.
These design features might reasonably have been expected to
lead to an increase, compared to the former “legacy” A levels, in
the proportion of full A-level candidates who achieved the GCE A level
standard without any change in the overall level of demand of the
qualification.
13.
Some aspects of the system were, from early in the implementation
phase, causing concern to some people involved with the system, including
at least one QCA board member (who later acknowledged that note had
been taken by QCA of his major concerns). For instance, I have collected
evidence that concerns were expressed, including at QCA seminars,
from 1998/99 about:
a.
the difficulty of, but vital need for, setting appropriate
standards for the AS and A2
units as a consequence of giving equal weighting to all units, even
though they require differing levels of demand;
b.
the complexity of the aggregation processes needed to achieve
the established A level standard, including the statistical challenges.
14.
The evidence suggests overwhelmingly that there is no clear,
consistent view among awarding body officials and many examiners and
teachers about the standard required at AS and A2 unit levels in order
to ensure that the overall GCE A level standard is maintained. This
concern had been identified specifically in relation to AS in the
review of Curriculum 2000 carried out by Professor David Hargreaves,
then Chief Executive of the QCA.
15.
This clearly has created a risk that differing interpretations
of the two standards exist. Such differing interpretations were strongly
evident during my hearings and in written evidence.
16.
Crucially, in order to reflect the different maturity levels
of the candidates during the first and second years of their study,
AS and A2 could be expected to have different levels of demand. My hearings and the written evidence revealed
some common understanding that AS should be set at a level of demand
less than the overall A level, to reflect candidates’ educational
development and maturity during the first year of an A level programme;
and that at A2 the level should be higher -
possibly higher than the A level Standard - to reflect candidates’
developing capacity during the second. The formal position is set
out in QCA’s publication, Managing
the Curriculum 2000 for 16-19 Students, that A2 should be more
demanding than the overall A level standard. In spite of this, there would appear to be
no common understanding on how much greater the demands of A2 units
should be compared to AS.
17.
QCA’s subject specific grade criteria setting
out the level of attainment which justifies the award of a particular
grade in each subject reflect only the level expected across the award
as a whole, and do not relate
to the assessment standard nor the actual attainment expected of students
in any one single unit. This
absence of a common standard for AS or for A2 contributes
significantly to a lack of common
understanding among those involved in teaching and examining
GCE A levels.
18.
AS units were piloted on a limited basis. A2 units were not, for reasons I have not had
time to ascertain. Therefore, before this summer, there was no practical
experience or relevant scripts to aid the grading process, or to illuminate
the challenges of the new grading and aggregation process across the
GCE A level as a whole. This resulted, in part, from the speed of
implementation of the policy as determined by Ministers.
19.
Evidence has been offered to me that not all the practical
and statistical issues implied by the new system had been fully and
comprehensively understood and worked through at critical stages in
the development and implementation process by the DfES, the QCA and
Examining Boards. For instance,
I would have expected much more comprehensive statistical modelling
than actually took place, particularly of the impact of differing
assumptions about the patterns of student choice and attainment before
introduction and after AS awards in 2001.
20.
The A level, GCSE and GNVQ Code of Practice governs the process
under which GCE A level awards should be made. A distinction must
be made between the marking of work, in which examiners award marks
to individual candidates for each unit, and the subsequent grading,
during which those marks are translated into grades for each unit
and ultimately for the subject as a whole.
21.
Under the QCA’s Code of Practice,
for each unit within a syllabus, an Awarding Meeting, chaired by the
Chair of Examiners, is responsible for establishing preliminary grade
boundary mark ranges for A and E for each unit and for recommending
the grade boundaries for A/B and E/U for each unit.
These recommendations need to be approved by the “accountable
officer”, usually the awarding body Chief Executive. Through that
mechanism consistency in the standard of the award is maintained.
Scrutineers under contract to the QCA attend
some awarding meetings, but play no active part, in order to monitor
application of the code of practice and report back to QCA.
The meeting:
·
checks that the required standards are reflected in each unit
and in the examination as a whole; and
·
assists the Chair of Examiners in arriving at recommended boundary
marks for the two key grades
in the examination (A and E).
22.
The meeting considers candidates’ work and compares it to that
from previous examinations, where possible, to ensure comparability,
and reviews statistics provided by the Awarding Body which are designed
to assist the Chair of Examiners in making a formal recommendation. This year more data were available than in
previous years, including for the first time analyses of predicted
A level grades based on candidates’ prior GCSE results. These additional data were welcomed by Chairs
of Examiners who found them a helpful additional source of information.
23.
The meeting recommends
A and E grade boundaries; other boundaries are then arithmetically
determined.
24.
The awarding body Accountable Officer (AO) is responsible for
reviewing the boundaries recommended by Chairs of Examiners with a
view to ensuring consistency of standards between subjects and awarding
bodies and over time.
25.
Different boards have different processes to support the AO
in making these judgements, but under the Code, the responsibility
rests in all cases with the AO. In making their decisions, AOs
do not generally consider students’ actual work, but they take account
of all other relevant information.
26.
The evidence put to me indicates variations in the ways these
tasks were approached within the three English awarding bodies this
year. Awarding meetings
27.
I received evidence from discussions with three OCR Chairs
of Examiners that they believed
that in making their recommendations they were expected to have very strong regard to the grade distributions
that emerged from the previous year’s A levels, and that these recommendations
should result in a distribution close to the previous year’s profile.
I have also received evidence from one QCA scrutineer
who recorded both in his report of the awarding meeting and in evidence
to me that the weight given to statistical evidence compared with
that given to professional judgements based on the actual work of
candidates was different this year. This may not be wholly unexpected
given that there were no archive scripts for A2 units.
It is a question as to whether or not the balance adopted was
appropriate. I am satisfied that there is evidence that there was
a perceived pressure to deliver outcomes in line with those of the
legacy A level in 2001. This
in at least one subject attended by a QCA scrutineer
resulted in the grade boundary decisions being “very much a statistically
driven operation”. The extent to which this occurred in other subjects
is not known precisely to me at this time.
28.
The Code of Practice does not specify the balance expected
between the various sources of information available to examination
boards. However, I am persuaded
that consideration was given in all cases to candidate’s work. I do not therefore conclude that the committees
and Chairs acted outside the boundaries of the Code of Practice. I am concerned however about the evidence that
the process leading to some grade boundary recommendations within
OCR, while seemingly permissible
under the Code of Practice, may have given undue weight to historical
and other statistical data. Edexcel
29.
Evidence from Edexcel Chairs of Examiners
indicated a significant and proper emphasis on the place of assessment
of students’ work in forming their grade boundary recommendations,
whilst acknowledging the relevance of statistical data in their judgements.
I have received no verifiable contrary indications from other
sources. Assessment and Qualification
30.
AQA Chairs of Examiners gave broadly similar accounts of their
role and the expectations upon them, and were clear that their deliberations
had given due weight to professional judgements based on candidates’
actual work. Accountable
Officer decisions
31.
Within Edexcel, consideration of
the Chair of Examiner’s recommended grade boundaries is a two stage
process. First, a Grade Evaluation
meeting consisting of senior awarding body officers considers the
recommendations and advises the AO on any changes.
The AO then makes a decision based on the Examiners’ recommendations
and this further advice. For
the 2002 GCE A levels, this process resulted in little change to recommended
grade boundaries; and most changes were of the scale of one or two
marks up or down and within the range recommended by the Chairs of
Examiners with the exception of Arabic and Russian, which were subsequently
agreed with the Chairs of Examiners after the GEM.
32.
A similar pattern is evident from AQA. In this awarding body the AO receives and considers
recommendations from the Chairs of Examiners without an intervening
advisory stage. It is usual
practice for proposed changes to be referred back to the relevant
chief examiner, which have in the first instance been considered in
detail my a senior officer of the board.
There is no further intervention at the advisory stage.
Data provided by the awarding body suggest a similar pattern
of grade boundary changes, with a total of 53, of which 26 reduced
the recommended mark. In only
three units were the changes to recommended mark grade boundaries
as much as 6 marks.
33.
In OCR, like Edexcel, there is an
intervening advisory stage – Grade Evaluation meetings (GEM) – which
are held before the AO confirms the grade boundaries.
I have received clear statistical evidence from the awarding body indicating that in a significant number
of units the GEM raised some of the relevant grade boundaries, and
the AO raised some of them yet again. [1]
Evidence from chief examiners in geography, French, German, Spanish,
history, psychology and government and politics expressed their concerns with these
changes. The same evidence
also indicates that in many cases the chief examiners were not made
aware of these changes. Their main impact has likely been on the proportion
of students passing at grade E, but this in itself has a knock on
effect on the higher grades.
34.
From the evidence I am clear that all three accountable officers
acted within their powers under the Code of Practice. I am, however, concerned that the OCR AO judged
that his duty to maintain the standard made it necessary to lift grade
boundaries in a significant number of units so substantially in the
light of statistical evidence. He
is quite clear that his actions were entirely based on a view that
the requirement to meet the A2 standard had been underestimated by
some examiners.
35.
At this stage I should point out that I have also received
unsolicited endorsements of this year’s processes from a number of
schools and examiners. This
is an important counterweight to some unjustified perceptions that
the whole system is failing to function.
36.
The evidence put to me suggests there may be a lack of consistency
in practices across the three English awarding bodies in the grading
process. In the time available
it has not been possible to investigate more fully this matter. I intend to do so in part two of my inquiry. WAS PRESSURE
BROUGHT TO BEAR ON THE AWARDING BODIES?
37.
In considering allegations of external pressure I have been
mindful of the context in which the A level system operates and in
particular the emphasis on year-on-year comparisons of results which
looms large in public debate about standards. That undoubtedly creates
an ongoing pressure within the system as a whole and raises sensitivities
among all concerned in relation to the maintenance of standards.
I have been careful to distinguish this type of ongoing background
noise from the possibility of distinct and identifiable actions which
might have added to this pressure this year. Meeting between
the awarding bodies and the QCA,
38.
This meeting was one in a series of regular and legitimate
meetings to discuss ongoing business between the English, Welsh and
Northern Irish awarding bodies and their regulators, QCA in
39.
Arising from the 12 March meeting, all three chief executives
of the English awarding bodies reported to me that the Chairman of
QCA had made clear to them what he expected of the awarding bodies
this summer in relation to the grading of GCE A levels.
Their perception was that they were being asked to give more
emphasis than perhaps was proper to statistical data and the need
to have an overall outcome similar to the 2001 legacy A levels.
40.
Both the awarding bodies
and the QCA Chairman reported specifically the latter’s comment that
there should be no “grade inflation” or “grade drift”. Both of these terms are used loosely to describe
more students passing and achieving higher grades year on year, either
as a result of changing standards in the qualification itself, or
in the level of performance of students.
In his evidence to me, the QCA Chairman was clear that he was
referring to the former interpretation.
I am equally clear that guarding against better grades based
on lower examining standards is a wholly proper and necessary concern
of the regulator.
41.
Following this meeting, the awarding bodies were sufficiently
concerned to write jointly to the QCA Chairman setting out their view
of their responsibilities with regard to the awarding process and
seeking his endorsement that this view was in line with the Code of
Practice. This he provided in a letter dated 19 April
which indicated clearly and properly
that “in this summer’s A level
awards, the change to new specifications means that awarders have
less evidence to assist them than in normal circumstances. In this situation, I do expect last year’s
A level results to provide a very strong guide to this year’s outcomes”
and concluded that “I am clear that grades for this summer’s A
level candidates can only be determined using a combination of professional
judgements and statistical evidence.”
42.
In at least one board, however, the earlier perception remained,
as is revealed in an internal note dated 29 April to OCR officers
from the OCR Chief Executive, which said of the 19 April letter: ‘This
latest letter is very carefully worded – but the underlying message
has not changed. “Very strong
guide” means something rather less than the most recent annual increases
in percentage achieving A or E!’
43.
There is evidence that in more than one board there was a perception
articulated by Chairs of Examiners during awarding meetings that there
was pressure being brought to bear for the candidates’ results in
their subjects to be close to those for the 2001 legacy A levels. In one case this perception led the QCA scrutineer present to contact QCA headquarters to check the
position. He understood the
perception was confirmed but I
have been unable to check elsewhere the substance of that conversation. The scrutineer (one
of 19) subsequently resigned in order to be able to make his concerns
known. 26 July meeting
between QCA and the awarding bodies
44.
This meeting was requested by the chief executives of the awarding
bodies in order to brief the QCA on provisional and emerging A level
results in a limited range of subjects.
These were revealing an increase in the pass rate and in the
proportion of candidates achieving an E grade in 2002 compared to
previous years.
45.
As explained earlier in this report, wholly legitimate improvements
in candidates’ performance might have been expected as a result of
the new AS/A2 system, in part because the A2 cohort in a subject might
change because those who had achieved a weak AS grade might not proceed
to A2, and in part because of the ability to re-sit units. Papers I have seen also indicate that the subject
grade profile for the new A levels was more affected by the statistical
phenomenon known as regression to the mean arising out of the modular
nature of the course, is taken into account in the grading process.
46.
The Chairman of QCA is clear that at the meeting, on the basis
of these data, he sought reassurance from the awarding bodies that
their awarding processes had maintained the standard. He also indicated that it might be necessary
to inquire into the standard which had been applied. I am clear this is a wholly proper position
for the Chairman to take in his role as regulator.
47.
The chief executives, however, had different perceptions of
the implications of this approach.
In varying degrees they all believed they were being put under
further pressure. For instance,
the internal note of the meeting from Edexcel
representatives reported to their chief executive that in their view
“Sir William Stubbs was not pleased with these outcomes…”, and that
he “indicated that if we went ahead with these predicted outcomes
he would announce an independent enquiry into the awarding bodies
and the Curriculum 2002 results.” Statements to me from the relevant Chief Executive
in the course of my inquiry confirm that this was interpreted within
Edexcel as a threat.
48.
Following the meeting an internal OCR e-mail gave another chief
executive’s view that “following his meeting with Joint Council colleagues
and the QCA”: “For any remaining AS/A2
awards, his [the OCR Chief Executive’s] advice is to try to get as
close as possible to the putatives grades
predicted on the basis of candidates’ GCSE results at ‘E’ – the phrase
he used was to push the boundary ‘until it squeaks’ – but not at the
expense of alienating awarders.”
49.
I am unable fully to resolve the clear differences in perception
of what was said at the meeting.
50.
The QCA explained that no official record of this meeting was made immediately. The record which I have been given has been
constructed this week from notes taken at the time by individual QCA
officers present.
51.
I have taken written
and oral evidence from Estelle Morris, David Miliband
and relevant DfES officials. My
inquiry has heard no evidence that Ministers or officials offered
any pressure or guidance on the grading process or the final outcomes
of this years examinations; nor was any present in the notes of meetings
between Ministers and QCA and officials and QCA.
I therefore conclude that there was none. CONCLUSIONS
·
From the evidence collected, it appears that the alleged problem
with the grading process this summer has its roots in decisions made
by the DfES and QCA about the structure of the AS and A-level awards,
the assessment model and the preparation for the introduction of the
new arrangements, particularly for A2.
The lack of a common understanding of the standard associated
with AS and A2 units, along with the challenges associated with aggregation
of the units, given all had equal weighting, played a significant
part in the problems experienced by the three examination boards during
the grading this year.
·
In relation to the grading process, there is no doubt in my
mind that to varying extents the three Chief Executives felt they
were being put under additional pressure to deliver outcomes largely
in line with the performance of students in 2001. I am equally clear
that the QCA, and in particular the Chairman, had stressed throughout
the need to maintain the A-level standard and prevent ‘grade drift’.
The differences in perception of what was said at meetings and in
subsequent letters is, however, a reality. The Chief Executives clearly
passed on their perception to staff and examiners.
·
I am satisfied that the requirements the QCA places on the
boards, and as set out in correspondence, were all proper and in line
with the regulatory responsibilities.
·
I am equally satisfied that the actions of the Chief Executives
of these boards, as accountable officers, were all done within the
parameters of the Code of Practice. In this regard, they acted with
integrity.
·
That said, the evidence strongly suggests that the actions
taken with regard to the marks to be associated with the key grade
boundaries (A/B and E/U) did vary across the three boards. In particular,
it seems the balance between judgement based on the standard of marked
work and the use of statistical information changed, and for more
subjects in one Board (OCR) than in the other two. I must, however,
stress that as the code of practice gives no detailed guidance on
this matter, all the actions must be seen as proper. The impact of
these significant changes to the marks associated with unit grade
boundaries, on the overall grades at GCE A level of the students affected
is impossible for me to say.
|