School league tables: Revealing or misleading?
New research claims that league tables mislead because they report past performance rather than indicating how schools might perform in the future
At the centre of the accountability framework for schools sit league tables. These tables have been changed recently to try to ensure that they are more comprehensible, relevant, accurate and transparent, so that parents can understand them better in order to support them in making informed choices. Indeed, each year parents are encouraged to use school league tables to help choose a secondary school for their children. Headteachers and governors, because of the league tables perceived importance, are highly sensitive to what the government measures in its school performance tables. But George Leckie and Harvey Goldstein argue that such comparisons are crude and ultimately misleading. The most high profile secondary school league tables are those publishing the percentage of children getting five A* to C grades at GCSE. However, Leckie and Goldstein claim ‘ this performance measure is an unfair way to compare schools as it says more about differences in schools’ intakes than it does about differences in their quality. Such league tables are also an unreliable way to compare schools as, with only around 200 students per school sitting GCSE exams each year, it is a ‘noisy’ measure of how well schools are truly performing.’ The so-called ‘contextual value-added’ (CVA) measure, published by the Government until it was abandoned in 2011*, was considered to be a better measure for comparing schools as it adjusted for differences in schools’ intakes and was published with error bars to communicate the statistical uncertainty in schools’ performances.
The authors write ‘Our research has focused on a fundamental problem in using secondary school league tables, value-added or otherwise, for school choice: school league tables report the past performance of schools, based on children who have just taken their GCSE exams, whereas what parents want to know is how schools will perform in the future when their own children take the exams. Consider parents who chose a secondary school for their child in autumn 2012. Their child will enter school in autumn 2013 and will take their GCSE exams in 2018. Thus, the information parents need when choosing is how schools are predicted to perform in 2018. However, the most recent information available to them is the school league table for how schools performed in 2011.’
They point out ‘There is therefore a seven-year gap between the available information and what parents want to know. Clearly, the more schools’ performances change over a seven-year period, the less reliable league tables will be as a guide to schools’ future performances.’
They continue ‘To examine how serious a problem this issue is, we first examined the official CVA school league table data. We found that many schools which were performing in the top quarter of schools seven years ago perform in the bottom half today. We then predicted schools’ current performances based only on data from seven years previously. We found that these predictions were so imprecise that almost no schools could be distinguished reliably from one another.
This means that, for choosing a school, the league tables carry very little useful information and, by not communicating this fundamental problem to parents, they are very likely to mislead.
More should be done, they conclude, by the Government and the media to communicate this important limitation to parents.
George Leckie is a Lecturer in Social Statistics at the Graduate School of Education, University of Bristol. Harvey Goldstein is a Professor in Social Statistics at the Graduate School of Education, University of Bristol.
*UPDATE=Thursday 24 January: This article was updated to clarify that from 2011 CVA was no longer used, though was in place when this research was originally carried out.
Department for Education (2012). Performance tables. Retrieved 21 December 2012.
Department for Education (2010). ‘Key Stage 2 to Key Stage 4 (KS2-KS4) Contextual Value Added Measure (CVA) Including English and Maths’ in 2005 Key Stage 4 Contextual Value Added (CVA) Pilot. Retrieved 21 December 2012.
Leckie, G. and Goldstein, H. (2009a). School league tables: Are they any good for choosing schools? Research in Public Policy, Bulletin of the Centre for Market and Public Organisation, 8, 6-9.
Leckie, G. and Goldstein, H. (2009b) The limitations of using school league tables to inform school choice. Journal of the Royal Statistical Society: Series A, 172, 835-851.
Leckie, G. and Goldstein, H. (2011). Understanding uncertainty in school league tables. Fiscal Studies, 32, 207-224.
Goldstein, H. and Leckie, G. (2008). School league tables: what can they really tell us? Significance, 5, 67-69.
Note. CVA measures are attractive to many but academics are still debating how best to measure value added as there are a number of different approaches to its measurement. No single model is universally accepted.