Pisa rankings can be misleading
So why do countries worry so much about them?
Every three years governments around the world wait – some eagerly, but most anxiously – for the latest results of the OECD Pisa survey. Those who fare badly in the tables often suffer what’s termed ‘Pisa shock’. In fact, roughly half of the governments affected change their education policies as a result of poor scores.
Take the US in 2013. Reacting to another poor showing – and a league-topping display by Shanghai – President Obama talked darkly of a generational “Sputnik moment” for his countrymen. America needed to catch up with the world’s best performers in maths and science, he said, and fast.
After its results, Norway, a laggard performer by high Nordic standards, immediately pledged new investment to boost teacher training, development and quality. And the Welsh government, with a mean score below the OECD average, talked of a grave situation and “systemic weaknesses”, committing to a reform programme focused on literacy and numeracy. Such self-flagellation is understandable. Pisa is an ambitious, large-scale attempt to measure and compare literacy, mathematics and science across education systems. First launched in 2000, the surveys are taken every three years, the latest in 2012, and countries slipping down the tables do so with the world looking on.
Moreover, the test aims to assess learners, aged fifteen, on their competence to address ‘real life’ challenges. Andreas Schleicher, the architect of Pisa – and the “the most important man in English education”, according to former education secretary Michael Gove – says that the world economy will not pay you for what you know, but rather for how you apply your knowledge. In that sense, Pisa scores speak directly to our deepest fears about future prosperity and a changing world order.
Don’t be Rasch
Pisa certainly generates significant amounts of useful data and evidence that can be used by policy makers and practitioners. But there are growing concerns about its methodology and the way its results are used. In principle, comparing yourself with the best in the world is good practice. But practically it’s a huge challenge to compare across different countries given the cultural and linguistic barriers.
Pisa’s comparison of countries relies on plausible student scores derived from the so-called Rasch model, which is not unchallenged. In short, pupils are not given identical questions but that is accounted for in the model, which seeks to iron out so-called ‘contextual’ features. However, some researchers question whether this scaling model is being applied reliably or consistently across the board. We probably can’t very accurately rank students across countries, as if they have all sat identical tests at the same time in an identical context, and that’s because they haven’t.
Cambridge University statistics professor David Spiegelhalter investigated Pisa for the BBC recently. He talked to experts including Svend Kreiner in Copenhagen, Harvey Goldstein at Bristol, Oxford’s Jenny Ozga and professor Alan Smithers of the University of Buckingham. His conclusion was that league tables are essentially misleading and unreliable, even though the data produced by Pisa is useful.
Gabriel Sahlgren, of the Centre for Market Reform, warns governments against using Pisa results to inform changes to their systems, because of its poor methodology. Pisa, he claims, confuses correlation with causation. And Professor Smithers has made an important point: the OECD cannot possibly know for sure whether a pupil does well in a test because of their schools system, or in spite of it. In the case of South East Asian countries, most of which rate highly in Pisa rankings, many pupils receive extra private tuition outside of school. And yet Pisa has no way of taking this into account.
The home front
Arguably, if you think that Pisa tests assess things that are worthwhile and you acknowledge their political consequences, then it makes sense to prepare your students for the test. Many countries do, of course; but in England, where the latest Pisa test results stagnated, that hasn’t happened.
Whether it ever will is questionable, too. After all, how will Pisa benchmarking fit with the reforms made to English education since 2010, whereby schools are increasingly autonomous and ‘self-improving’, with parents and teachers driving change?
Lessons from overseas are only useful if they can inform the English system. But this requires an institutional framework that enables findings to directly influence classroom practice. Neither the structural reforms to our system, nor changes to the accountability or assessment frameworks, were designed to improve our Pisa ratings. So the chances are they will not improve.
But, here is a heretical thought. Perhaps we should, given Pisa’s limitations, continue to use its data where appropriate, but concurrently seek ways to accurately measure and compare ‘soft skills’ in young people.
Communication, character and the ability to work in teams are just some of the skills employers desperately want, but find in shockingly short supply. Perhaps not surprisingly, none of those skills are measured by the Pisa test either.
Published- Education Investor- September 2014
Another League Table or ‘ Efficiency Index ‘has just been published by Gems. It purports to rate the efficiency of education systems around the world , relying inevitably on Pisa data. Korea, where parents have to invest $10bn in out of school private tutors to get their children up to the right level to progress, is ranked second. Surely an efficient state system would not drive parents to seek private sector support for tutoring the basics? And then there is the case of Germany. Despite having the best trained and motivated workforce in europe with a good balance between cognitive and non-cognitive skills, it is ranked 27th in the table. Sorry, I just dont buy it.