11 May 2015
Cherry Ridgway explains how comparable outcomes work for setting GCSE grades and looks ahead to the national reference tests, starting in 2017.
As many school leaders are aware, GCSE grade boundaries are currently set using a combination of a system called ‘comparable outcomes’, other statistical information and the professional judgement of examiners. It is important for school leaders, including governors, to understand how this system works in awarding GCSE grades.
The use of ‘comparable outcomes’ was developed by exam boards in 2002 to ensure that the first cohort to take reformed A levels was neither advantaged nor disadvantaged.
When there is a significant change in an examination system, as there was with the new A levels of curriculum 2000, it is essential to ensure that grade standards remain consistent, as there is always the risk that students will not achieve as well as they may have done if they had been taking a legacy exam.
The first time that students take a new examination it will also all be new to their teachers. Teachers will only have the sample assessment materials (SAMs) to help them prepare their students, as there will be no past examination papers. They will certainly not have seen examiners’ reports, detailing questions answered well or badly and not have had the chance to be an examiner for the new exam.
These factors could mean that students are not as prepared for the new exam as they would have been for those in an established syllabus.
Regarding the new GCSE and A levels there are some other risks. Students in some schools may not be used to sitting linear exam papers and may not be as prepared for a linear paper as they would have been for modular papers. Some younger teachers may never have taken a linear exam themselves but have to help students prepare for them for the first time.
In addition, the new GCSEs will be introduced without students having experienced all the new Key Stage 3 curriculum intended to be essential prior knowledge, understanding and skills.
All of these factors could put students at risk of not achieving as well in a new exam as they would have done in a legacy exam.
To an extent, the idea that a certain level of performance should lead to exactly the same grade as it had in previous years (that is, criterion referencing) disappeared some years ago as the key driver of determining examination grades.
In 2002, exam boards started to use ‘comparable outcomes’ to support the awarding process initially with the new A level syllabuses. This ensured that the national proportion of students with the same prior attainment gaining each grade in a particular subject was comparable with previous years.
This approach has been used ever since and was given increased emphasis by Ofqual at the start of the decade, as it was challenged by the government to tackle grade inflation. As a result, grade boundaries have since been set through calculations based on ‘comparable outcomes’ in a balance with examiners’ judgements. In fairness it should, however, be recognised that statistical information has been used in grade setting for more than 40 years.
In order to determine the percentage mark required to obtain a particular grade in a certain subject the following system is used:
First, looking at the year group in question, the ability profile of the national cohort is determined. This is done by looking at their Key Stage 2 assessment results. So, for current Year 11, their ability is determined by the national KS2 results for that year group when they were in Year 6.
Then, this is compared with the KS2 national ability profiles from previous years. The data is then examined, for each subject, what percentage of A*, A and so on were awarded for each KS2 score. They then determine the percentages at each grade for the current year group so they are the same within an allowed statistical variation. Examiners’ judgements are then used where they feel it is necessary to recommend any variation in grade boundaries.
The use of ‘comparable outcomes’ makes it very difficult to evidence whether or not the system is improving nationally. In order to try to address this, Ofqual is considering the use of national reference tests, which could begin in 2017, to help set grade boundaries from 2018.
The new tests will provide additional information to support the awarding of GCSEs. Their purpose is to provide evidence on changes in performance standards over time in GCSE English language and maths in England at the end of Year 11. The tests would provide an anchor for GCSE standards and should enable us to see over several years if there is genuine change in how students perform. In other words, it will help us to know whether the system is improving or not.
The new tests will be piloted over the next two years with the first ones planned to be taken to assist awarding in 2017. In future years, Ofqual and the exam boards would then take into account the information from the tests when GCSEs are awarded.
The national reference tests have been discussed at April Council and a further report will follow.
Cherry Ridgway is ASCL Curriculum and Assessment Specialist.
For further information, also see ASCL’s Timeline and Suggested Activities to Manage Curriculum, Assessment and Accountability