PISA UNDER ATTACK
Academics argue over the model it uses to compare performance
Is Pisa the right benchmark?
How well are our school children doing by International standards? Not well, would seem to be the answer, at least based on the latest OECD Pisa study of 2009 in which, for example, we came 28th in Maths out of 65 countries. Pisa measures maths, science and Reading and supposedly the ability of pupils to apply knowledge to resolve problems. However, a Danish Academic Professor Svend Kreiners is preparing a paper, shortly to be published, which allegedly rubbishes the statistical model on which all Pisa results are based. Pisa, he says, doesn’t compare like with like across all countries and is not therefore an objective performance benchmark. Andreas Schleicher – Head of the Indicators and Analysis Division (Directorate for Education) OECD who is responsible for Pisa, disputes this, and will issue a detailed rebuttal in due course-but statisticians are beginning already to take sides. The Education Secretary Michael Gove is an admirer of Schleicher dubbing him the most important man in British Education. In this country Professor Stephen Heppell has long contested the accuracy and usefulness of Pisa results and has a web site which cites research that questions Pisas methodology. Professor Alan Smithers doubts its ability to compare like with like. S J Prais (National Institute of Economic and Social Research London) has previously used the example of Englands results to demonstrate serious ﬂaws in the response rates and sampling, of Pisa which necessarily lead to biased results. Gjert Langfeldt (Agder Universitet) questions the validity and reliability claims made by PISA, pointing to ‘constructional constraints, methodological mishaps and the cultural bias embedded in the PISA design’. Svein Sjøberg at the University of Oslo analyzed PISA items and found some to have confusing and erroneous material. For example, Sjøberg observed that the title of an article about cloning, “A Copying Machine for Living Beings,” was translated literally word for word into Norwegian, rendering the title totally incomprehensible. And the questions posed by Pisa are supposed to be culturally neutral. PISA uses a statistical technique called the “One Dimensional Item Response Theory.” But Joachim Wuttke of Jülich Research Center in Munich contends that this is wholly inappropriate. “Items that did not fit into the idea that competence can be measured in a culturally neutral way on a one-dimensional scale were simply eliminated.” This appears to corroborate findings by the University of Oslo’s Rolf Olsen who also argues that “in PISA-like studies, the major portion of information is thrown away.”
Pisa results, say critics, just happen to support the Governments reform narrative of the moment which is why it is so frequently cited here, although Andreas Schleicher seems to suggest that there is no evidence of decline in English pupil performance merely a suggestion that we are stagnating. Gove uses Pisa statistics that go back to the 2000 and 2003 surveys to demonstrate a 10-year performance decline. However, the OECD itself eschews using its data for such comparisons saying that sampling in 2000 and 2003 was not up to scratch and is not keen for countries to use the data in this way because it is unreliable, although it hasn’t stopped our Government doing so. Professor Alan Smithers points out that UK schools had a very low response rate in the 2000 Pisa study and, of those who did respond, a disproportionate number came from the best state schools and independent schools. Smithers added (BBC Radio 4) that because schools don’t regard PISA as important they focus their efforts on trying to do well in the national tests where pupils are improving year on year. So, no special effort is put in for PISA (unlike some other countries) . Also given that there were 32 entries to Pisa in 2000 and in 2009 there were 65 it is hard to compare like with like. But possibly more serious is Professor Svend Kreiners (the University of Copenhagen) allegation that it is impossible to accurately compare items across all countries using Pisas methodology. To do so the items compared must be the same across all countries in the study. But he says this is not the case. So comparisons between countries are not objective and its ‘measurements’ are “neither valid, objective or sample free in the sense that we usually understand these terms.” He concentrated on researching different degrees of difficulty in Pisa Reading test items. PISA as we have seen uses the “One Dimensional Item Response Theory” which he also claims is inappropriate. The main charge is that items tested are not exactly the same across the countries being tested. Indeed Kreiners claims he found no two items exactly the same in all countries. So there is a real argument now among academics, getting up a head of steam, at a time when our Government is placing great significance on Pisa and using the Pisa results in a way that the OECD is clearly uncomfortable with.
What is slightly perplexing about all this is that there is rarely any mention of the alternative table, TIMSS which sheds a more positive light on this countries’ performance This is particularly ironic, as Warwick Mansell has pointed out , given that TIMSS is ‘ a closer test of pure curricular knowledge of the sort about which Mr Gove often enthuses – ie the problems could be seen as more “traditional” – than is PISA, which tests application of reading, maths and science understanding in “real world” scenarios.’ Comparing yourself against international benchmarks seems, on the face of it, a pretty sound idea. But if there is a question mark over the benchmark you choose, then it could become problematic, particularly if the whole education system is then oriented towards satisfying its criteria. Pisa measures just a very narrow range of outcomes. Is that really how we want to determine whether or not our schools are successful? And what if the two key international benchmarks are measuring altogether different things, which Pisa and TIMMSs appear to be doing? If you choose one benchmark then you are probably not going to improve much in the other as they are measuring different things. And if Pisa is so important you would have thought Government reforms would ensure that those things Pisa measures are targeted for improvement ie pupils ability not just to absorb facts but to apply those facts to resolve problems. Will the Ebacc help here? Or a more fact based curriculum delivering the essential body of knowledge which all children need to learn? Or giving schools more autonomy? Possibly, but it’s not absolutely clear how.