ASCL policy: Accountability Measures

Download ASCL policy paper: Accountability Measures

Accountability is the obligation of an individual and organisation to account for its activities, accept responsibility for them, and to disclose the results in a transparent manner. The highest form of accountability is the individual’s professional accountability for the quality of his or her own work and to the people who the profession serves. In a self-improving system, we believe that teachers and school leaders are agents of their own accountability.

Extract from ASCL’s Blueprint for a Self-Improving System

First principles

1. The government and public (both as parents and taxpayers) have a right to understand how effective schools and colleges are. There is a clear need for appropriate accountability mechanisms to support this.

2. While we believe that teachers and school leaders should be driven primarily by accountability to the people they serve, government has a role in defining a slim, smart and stable public accountability framework with a small number of ambitious goals, including a nationally determined progress measure to recognise improvement.

Analysis of the current system

3. Successive governments have determined which of the headline measures within the national performance tables they believe to be a national priority and/or best reflect their views on how to quantify effectiveness. These measures have changed over time and driven behaviour in schools because of the high-stakes nature of the accountability system.

Secondary phase

4. In recent years, whatever the focus, the nature of the accountability system and the consequences of poor performance against the headline measures has driven system behaviour and exposed whatever incentives and weaknesses are inherent in each measure. Indeed, because the focus has mainly been on a single measure, system behaviour has adapted to maximise performance against that measure.

5. For example, when a previous government determined that Contextual Value Added (CVA) was the priority, schools’ entries in qualifications which counted for the equivalent of more than one GCSE soared because the measure accommodated them. The more recent emphasis on five A*- C including English and mathematics resulted in a focus on students around the C/D borderline, especially in English and mathematics, in a drive to secure C grades where possible. There was a marked increase in early and additional exam entries, and a move to iGCSEs, both in an effort to gain more C passes for students.

6. To counter this, the government again made changes to the performance tables to deter schools from examination entry policies which it saw as undesirable, including limiting all qualifications to a volume of one, greatly reducing the number of non-GCSE qualifications which count in the performance tables and more recently publishing headline measures based only on the first qualification achieved by students.

7. As part of a complete review of accountability in all phases there is a new set of measures at Key Stage 4, with Progress 8 becoming the main headline indicator from 2016. This represents a shift in focus away from raw attainment in favour of progress through all grade boundaries in a number of subjects. In so doing, there are a number of principles of fairness to schools which the government has brought to bear.

8. The Progress 8 measure will undergo continual changes to its inputs and outputs because of changes to Key Stage 2 test reporting and to GCSE grading, such that a steady state will not be reached until 2019 at the earliest. It will not be possible to be clear whether this measure has achieved the intended effects, such as allowing schools with lower ability intakes the chance to demonstrate they have performed well.

9. Alongside Progress 8, there will be three other headline measures:

  • Attainment 8 - effectively the output component of Progress 8.

  • The ‘basics’ measure - the percentage of students achieving at least grade C in both English and mathematics.

  • The English Baccalaureate (Ebacc) – a performance measure and not a qualification.

10. These measures are not mutually exclusive. The basics measure, for example, is a component of the Ebacc, and Ebacc subjects themselves make up a significant part of Attainment 8. The lack of exclusivity of these measures brings with it the dangers of incentivisation and narrowness highlighted above.

11. However, it is already clear that the Progress 8 measure has started to impact on schools’ behaviour, with most schools adapting their curriculum offer to match the mix of subjects. Such behaviour will continue to affect the statistical modelling underpinning Progress 8. Schools’ convergence to a more common pattern of qualifications will in turn inflate average scores for any given level of prior attainment.

12. There is a range of views about the balance of qualifications in the Progress 8 mix and whether this is appropriate. Some argue that all students should pursue a curriculum with examinations largely weighted on traditional subjects. Others argue that this mix is too prescriptive and unsupportive of arts and technology subjects and vocational qualifications. In either case, the continued presence of the Ebacc measure alongside the Attainment 8 measure does not give sufficiently different information to schools or stakeholders about performance.

13. There are other problems. The percentage of children making ‘expected progress’ is a proxy for raw attainment and not a separate measure of progress. Statistical models for modern languages are inflated for students with low prior attainment by the inclusion of community languages, effectively condemning learners who are acquiring a language for the first time to score negative value added.

14. Measurement of the gap between the performance of disadvantaged students and their more advantaged peers is also ill-conceived. The headline figure of the proportion of five A*-C with English and mathematics does not register the progress students have made from lower starting points but who have not passed the grade C threshold. In this case the choice of an incorrect metric has led to the incorrect conclusion that the gap is not closing. Progress 8, or another metric which uses averages rather than thresholds, must be used to measure the gap.

Primary phase

15. The key primary accountability measure has for some years been the percentage of students achieving level 4 in reading, writing and maths at the end of Key Stage 2, with value added (between end of Key Stage 1 and end of Key Stage 2) as a subsidiary measure. To be above the floor standard, schools had to ensure 65 per cent of students reached at least level 4.

16. There are a number of issues with this approach. The prioritisation of the threshold measure has led to similar issues to those in the secondary phase, with schools effectively being incentivised to prioritise children at the level 3/4 border, at the expense of higher or lower attaining children. In addition, level 4 was felt to be too low an ambition, with modelling showing level 4b to be a better indicator of future success. Tracking progress only from the end of Key Stage 1 ignores the impact of the school on the crucial early years, and in fact actively discourages schools from achieving high Key Stage 1 results. And finally, the small numbers of students involved means the numbers are often of questionable statistical significance when looked at for a single year.

17. In 2013, the primary performance tables were tweaked to address some of these issues. The percentage of students achieving a level 4b or above was included for the first time, and three year rolling averages were introduced to give a more meaningful view.

18. From 2016, major changes to the primary accountability system will be introduced. With the demise of National Curriculum levels, the results of tests at the end of each key stage will be reported as a scaled score. Draft performance descriptors have been produced to inform end of key stage teacher assessment. A dual floor standard is proposed, with schools remaining above the floor if they either have 85 per cent of students achieving the new attainment target (believed to be equivalent to level 4b) across reading, writing and maths or reach a new progress target (for which the input measure will be a new baseline assessment in Reception).

19. A number of questions and concerns remain around these proposals, and a Commission on Assessment without Levels has been set up to address some of these. Outstanding issues include:

  • Concerns around the draft performance descriptors, particularly the way in which they are structured and some of the terminology used.

  • Potential negative effects of the new baseline assessment.

  • The length of time before we see the impact of the new progress measure.

  • How to ensure clarity and consistency between the ways in which externally-marked tests and teacher-assessed elements are reported to parents.

  • How to support schools in moving towards an assessment system based on fewer things in greater depth, and in understanding how a curriculum and assessment model based around mastery and ‘going deeper’ sits alongside an accountability model that prioritises progress.

Accountability in a self-improving system – policy proposals

For government

20. Central government should adopt a slim, smart and stable accountability framework. Working with the profession, the government should identify a small, stable core of measures in the national accountability framework that have broad based support, and then leave this in place for at least the term of government. These measures must be sophisticated enough to reach valid and reliable conclusions about a school’s performance. There is also a need to monitor the accountability framework to ensure that it drives positive system behaviour.

21. This new accountability framework should look at school’s performance across a number of years, in order to make judgements more statistically robust, particularly for smaller schools.

22. A ‘floor’ should no longer be necessary. This is because conclusions about effectiveness should never be based on a single data item or one centrally determined measure. This leads to out-of-kilter, high stakes accountability which drives perverse behaviours in the system.

23. ASCL will be doing further work on what this accountability framework might look like.

For the profession

24. The accountability framework must be designed in a way that gives school leaders confidence that they can design a curriculum which meets the needs of students rather than one that conforms to misaligned performance measures. Alongside the small, stable core of national measures, school leaders would be able to choose or design appropriate metrics. These would include the ‘headline’ measures (reflecting national priorities), ‘subsidiary’ (broader data still collated and published by government but not part of the headline set), ‘third party’ (such as that provided by FFT and other providers) or ‘bespoke’ (reflecting key priorities in the school, for example by constructing a measure on punctuality where no ‘off the shelf’ measure exists). Such measures will then be genuinely aligned to what is valued and less prone to distortion because of the range. School leaders should then look at the distributions of all measures, not merely the thresholds.1

25. School leaders would then be in a strong position to evidence through self-evaluation and in valid and reliable ways how successful the school is against both national expectations and its own vision and objectives. School improvement planning would be built around the right way to measure impact. This would involve the profession thinking harder and deeper about the construction and precision of its own metrics – those that generate the best possibilities for meaningful evaluation of impact.

26. Leaders of all-through schools or cross-phase multi-academy trusts (MATs) should consider how best to track progress and measure impact from entry to exit, and how such measures might contribute to new ways of assessing impact across the system as a whole.

For the inspectorate

27. The inspectorate needs to reform inspection methodology to build capacity and expertise in the analysis of school provided data. This will enable them to strengthen the reliability and validity of judgements through inspection (See ASCL’s separate policy paper on the future of inspection).

1 Coe, R. and Sahgren G.H, (2014) “Incentives and ignorance in qualifications, assessment, and accountability”.