Information paper: Decoding the Data

Download Information paper: Decoding the data

This information paper is designed for members of the senior leadership team and data managers, and seeks to explore the challenges facing school leaders as they respond to the ever more sophisticated range of data available to schools.

Ofsted, the DfE and local authorities often base judgements about schools on the messages that they derive from these data systems, so it is vital these systems are understood.

Throughout this information paper, ‘Percentage of cohort getting 5A*-C incl GCSE English and Maths’ is abbreviated to ‘%5A*-C inc En+Ma’. Also, ‘Percentage of cohort getting A*-C in both GCSE English and Maths’ is abbreviated to ‘%A*-C En+Ma’.

Overview

June 2016 will be the first year that Progress 8 takes effect for school accountability, along with other headline measures. This is a major step forward as it means that the results of every child at every grade in several subjects will count, rather than just a few around the threshold, and whilst also accounting for their prior attainment. But as with any change, it is important to understand the detail and be prepared.

This paper considers the following key questions and prompts:

Q1 Will you know your Progress 8 score on 25 August?

Q2 What will you be reporting to parents and the press on 25 August?

Q3 What difference will the change from %5A*-C inc En+Ma to %A*-C En+Ma make to you?

Q4 If your entries and results stay the same in June 2016 as June 2015, with a similar cohort prior attainment profile, what will happen to your Progress 8 score (compared with June 2015)?

Q5 Have you downloaded your Progress 8 data for 2015 and understand your school’s results?

Q6 Is the Overall % Expected Levels of Progress a school progress measure?

Q7 If your maths department has twice as many A* grades than the English department, is English underperforming?

Q8 What are Subject Transition Matrices (TMs)?

Q9 Will the change from KS2 levels to KS2 scaled scores mean an end to measuring progress?

Q1 Will you know your Progress 8 score on 25 August?

ASCL has liaised closely with the DfE from the beginning of the discussions about Progress 8. We have worked to ensure that, whatever views there may be about the formulation of Progress 8, the supporting documentation from the DfE and ASCL is consistent and that the results for 2014 and 2015 were made available to each school.

In particular, it was important to emphasise that Progress 8 is a relative measure, calculated each year on the basis of the actual results of all of the pupils taking exams at the end of KS4 that year. Therefore, nobody knows in advance of analysis on national results in August what the national average will be, and consequently, the answer to question 1 is ‘no’.

On the other hand, you will be able to calculate your Attainment 8 score and convert it to an average grade, which will be one of the new accountability measures (as will %A*-C En+Ma - see response to question 3 for further information).Back-to-the-top

Q2 What will you be reporting to parents and the press on 25 August?

This is a challenging question. The DfE’s Statement of Intent update for the June 2016 results, highlighting what will be appearing in the Performance Tables, usually appears towards the end of July. The latest guidance currently available (and updated 20 January 2016), is:

Progress 8 Measure in 2016, 2017, and 2018 (page 7) states:

What will be published in the performance tables and RAISEonline from 2016

The headline measures which will appear in the performance tables will be:

  • progress across eight qualifications

  • attainment across the same eight qualifications

  • percentage of pupils achieving the threshold in English and mathematics (currently a C grade, grade 5 when new GCSEs in English and mathematics are first reported in performance tables in 2017)

  • percentage of pupils achieving the English Baccalaureate

We are currently consulting on the addition of a fifth headline measure to show the percentage of pupils entered
for the English Baccalaureate. The consultation document can be found here and closes on 29 January 2016.

Parents and the local media are still probably expecting %5A*-C inc En+Ma, so you may wish to use one of the new headline measures %A*-C En+Ma, which actually is likely to be similar because they are both threshold measures.Back-to-the-top

Q3 What difference will the change from %5A*-C inc En+Ma to %A*-C En+Ma make to you?

Threshold measures: where a pupil has to reach a certain standard or threshold to count; this can then be
expressed as percentage of cohort etc, eg % of pupils gaining 5A*-C grades.

Note that in practice:

  1. results of a small range of pupils can affect a threshold measure eg those around C/D borderline for %A*-C, and

  2. one element of a multiple threshold can dominate, eg in practice there is very little difference between:

    %5A*-C inc En+Ma (av across schools = 57.1 per cent)

    and %A*-C En+Ma (av across schools = 59.4 per cent)

    (DfE SFR June 15 table 3a and 4a of pupils at end KS4 in state mainstream).

So, 96% of pupils who gained %A*-C En+Ma also gained %5A*-C inc En+Ma.

In 2013 figures were:

58.6% and 59.4% ie 99% of pupils who gained %A*-C En+Ma also gained %5A*-C inc En+Ma.

The slight drop from 2013 to 2015 is a consequence of the ‘Wolf’ reforms implemented in 2014.Back-to-the-top

Q4 If your entries and results stay the same in June 2016 as June 2015, with a similar cohort prior attainment profile, what will happen to your Progress 8 score (compared with June 2015)?

Chart1

Chart2

Graph 1

Graph 2

Graph 1 shows that big differences between the EBacc3 and Other3 ‘buckets’ (or elements) in 2014 and 2015. We know that most schools have changed their curriculum and exam entry policy for current Year 11 so that the majority of pupils are taking three EBacc subjects, so we are expecting the number of slots filled in the EBacc3 ‘bucket’ to increase in June 2016 (graph 2) with a concomitant rise in the national average score. So, if you are not changing your entry policy (and results are the same) then it is likely that your EBacc3 score will drop!Back-to-the-top

Q5 Have you downloaded your Progress 8 data for 2015 and understand your school’s results?

The 2015 results are still available on the DfE Tables Checking website. The ‘base’ data for June 2016 will be available between 13 and 27 June so that you can check who will be counted in the Performance Tables ‘cohort’ results – effectively the denominator in the percentages (your school should have received a letter from DfE).

David Blow has prepared spreadsheets downloadable from the ASCL website so that you can paste in the 2015 DfE pupil datafile (see Further information at the end of this paper).

It is particularly important for schools with the advantage of high prior attainment and very high threshold output attainment to look at these files carefully. Progress 8 will enable a fair and effective differentiation amongst schools which are currently clumped together with a high %5A*-C inc En+Ma figure. The Ofsted data dashboard quintiles were particularly flawed for these schools, and Progress 8 offers a fair way of analysing their relative performance.

Questions 1-5 relate mainly to Progress 8, but there are a number of other important questions relating to a genuine understanding of the data for your school, as detailed previously.Back-to-the-top

Q6 Is the Overall % Expected Levels of Progress a school progress measure?

Chart5

To many people’s surprise, the answer is ‘no’. Although ‘Overall % Expected Levels of Progress’ is being removed from the headline measures, there will still be the overall mindset change which is needed for schools to move from threshold measures to average measures, which take into account the result of every child at every grade.

Graph 3 shows the results for every school in the country for 2015, plotting the attainment measure %5A*-C inc En+Ma against ‘overall % making expected levels of progress’. There is a remarkable correlation, showing that overall % making expected levels of progress is actually a proxy for attainment.

The underlying reason is that a ‘level’ at KS2 is very different from a ‘level’ (or GCSE grade) at KS4. Most pupils on KS2 level 5 will gain a GCSE grade B (and thus make three levels of progress), whereas it is much more difficult for a pupil on KS2 level 3 to gain a GCSE grade D. Thus, schools with high prior attainment will have high %5A*-C inc En+Ma and overall % making expected levels of progress.

The importance of knowing the national picture is illustrated in question 7.Back-to-the-top

Q7 If your maths department has twice as many A* grades than the English department, is English underperforming?

Chart6

Actually, the English department is probably doing better than the maths department (although both might be over-performing or both underperforming). This is explained by graph 4 which shows the change in ‘%A* in maths and English’ in the last six years. In 2015, maths = 7.2 per cent, English = 3.2 per cent, ie more than twice as many maths* grades as English

How can you know how the maths and English departments are performing across the range of prior attainment, not just at the threshold of %A*-C? Look at the National Subject Transition Matrices and download the ASCL spreadsheet to enable you to compare the results from your school, not just English and maths, but in a total of 40 common GCSEs, against what they should be if in line with national expectation.Back-to-the-top

Q8 What are Subject Transition Matrices (TMs)?

Chart7

These are published each year by DfE and Ofsted for 40 subjects. They are also available by gender (which helps single-sex schools) and by disadvantage or not.

They give the actual number of students nationally gaining each GCSE grade in a particular subject from a given KS2 starting point. This enables fair comparisons to be made taking into account the prior attainment of the pupils involved. David Blow’s spreadsheet has the National TMs pre-loaded, and so all you have to do is a one-off pasting in of your school’s data from the DfE Tables Checking data file, to enable you to get a picture for each subject.Back-to-the-top

Q9 Will the change from KS2 Levels to KS2 scaled scores mean an end to measuring progress?

Not at all. Currently, the KS2 test marks are converted into a fine score which can then also be converted into a sub-level and level, but Progress 8 is based around the fine score. A similar process will take place once we have the marks and scaled score; there will still be a clear, finely gradated KS2 input value.

These examples provide a flavour of the complexity of making accurate judgements in the light of the many changes taking place each year.Back-to-the-top

Further information

This information paper has been written by David Blow, Head of The Ashcombe School in Surrey and an ASCL member. David has kindly provided a range of spreadsheets and information regarding performance measures, with links detailed below.

David is also a keynote speaker at our ASCL Professional Development Leadership of Data Conferences. Subsequent events will be available to book online at www.ascl.org.uk/pd

DfE Progress 8 School Performance Measure (20 January 2016)

DfE Tables Checking website

ASCL website, Progress 8 Toolkit

ASCL website, Updated KS4 Transition Matrices tool with 2015 national figuresBack-to-the-top