OECD and others are benefiting financially from PISA –some questions need to be answered.
Just ahead of the publication of the 2013 PISA league tables, India withdrew from the list of countries which featured in the tables. India’s decision was due, some said, to the fact that it would do badly (ie it chickened out) but others claimed that India had real doubts and worries about PISAs methodology. Some Indian educators claimed that its question papers were culturally biased and its rankings are counterproductive.
The PISA rankings are like many education league tables. They have influence, but for presentation purposes they present complex information simplistically and so they are imperfect. Although Andreas Schleicher of OECD habitually warns countries to be careful about how they interpret the PISA results, he knows full well that those countries that perform poorly tend to have post- mortems and adjust their policies and education reforms accordingly, in response to what is now termed ‘PISA shock.’
The Rasch model that PISA uses is supposed to iron out the contextual differences between the respective countries. But Professor Jenny Ozga, an expert in the field at the University of Oxford, told the TES , back in 2013 .“People have been struggling for decades to design tests that remove the contextual features that shape and support pupils’ performance. It cannot be done,”. Professor Svend Kreiner, of the University of Copenhagen, says this model can only work if the questions that PISA uses are of the same level of difficulty for each of the participating countries. He believes though that his research proves that this is not the case, and therefore the comparisons that PISA makes between countries are “useless”. Dr Hugh Morrison ,of Queens University Belfast, said that PISA and the Rasch model made the “impossible” claim of being able to measure ability independently of the questions that students answer. “I am certain this (problem) cannot be answered,” he told TES.
Countries are ranked separately in reading, maths and science, according to scores based on their students’ achievements in special PISA tests. These are representative rather than actual scores because they have been adjusted to fit a common scale – where the OECD average is always 500. So in the previous PISA assessment, for example, Shanghai finished top in reading with 556, the US matched the OECD average with 500 and Kyrgyzstan finished bottom with 314.
As the TES pointed our ‘ You might think that to achieve a fair comparison, and bearing in mind that culturally biased questions have been “weeded out”, that all students participating in Pisa would have been asked to respond to exactly the same questions.But you would be wrong. For example, in Pisa 2006, about half of the participating students were not asked any questions on reading and half were not tested at all on maths, although full PISA rankings were produced for both subjects. Science, the main focus of PISA that year, was the only subject on which all participating students were all tested.’
Professor Kreiner has looked in detail at the reading results for 2006 and noted that another 40 per cent of participating students were tested on just 14 of the 28 reading questions used in the assessment. So only approximately 10 per cent of the students who took part in PISAS were tested on all 28 reading questions. This in itself is ridiculous,” Kreiner told TES. “Most people don’t know that half of the students taking part in PISA (2006) do not respond to any reading item at all. Despite that, Pisa assigns reading scores to these children.”
People may also be unaware that the differences in questions don’t just occur between students within the same country. There are also between- country differences in the questions.
For example, eight of the 28 reading questions used in PISA 2006 were deleted from the final analysis in some countries. The OECD says that this is because they were considered “dodgy” and “had poor psychometric properties in a particular country”. However, in other countries the data from these questions did contribute to their Pisa scores.
The main point is, it is not up to the rest of the world to show they [the OECD] are wrong. It is up to PISA to show they are right. They are claiming they are doing something and they are getting a lot of money to do it, and they should support their claims.” said Professor Kriener
Dr Morrison has said “There are very few things you can summarise with a number and yet PISA claims to be able to capture a country’s entire education system in just three of them. It can’t be possible. It is madness.” He goes further, saying that the model PISA uses to calculate the rankings is, on its own terms, “utterly wrong” because it contains a “profound” conceptual error. For this reason, the mathematician claims, “PISA will never work”. Professor John Jerrim of the IOE, who looked at both PISA and TIMSS surveys , said that there are so many problems associated with these studies concerning missing data, procedures and target populations that it is impossible to draw firm conclusions from them ,and policy makers shouldn’t even try.
Gabriel Sahlgren ,who heads research at the CMRE think tank, claims that PISA has nothing, to say about effective practices and policies for raising performance. This is because it’s a non-academic report that can’t separate causation from correlation. If one is interested in understanding what works in PISA, an understanding of the academic research is key, he says.
As Pearson is developing the 2018 Student Assessment 21st century frameworks for OECD,its probably time for a thorough check of the OECDs methodology. The frameworks define what will be measured in PISA 2018, how this will be reported and which approach will be chosen for the development of tests and questionnaires. The main tasks will be to:
Redefine reading literacy, taking into account how young people are taught to approach the digital environment including how to recognise credible websites and online documents.
Review and where necessary adapt the frameworks for mathematics and science.
Develop the student questionnaire framework for the collection of contextual information and the measurement of other education outcomes which may have connections with performance.
Develop a framework for the measurement of global competence which will assess students’ awareness of the interconnected global world we live and work in and their ability to deal effectively with the resulting demands.
There are a lot of sound commercial reasons why both OECD and Pearson want to market PISA and expand its influence. But what about the education value? OECD is seeking to persuade schools to buy into its agenda and to adjust their curriculum offers to be more in line with what PISA is testing (for which it will benefit financially). PISA claims it is testing students ability to use knowledge to problem solve. This is not an obviously good trend and there are potential conflicts of interest here. It does seem time to have a more open and transparent debate about whether the methodology used by OECD is sufficiently robust, and whether whole systems should reform based on the latest PISA ratings. England is in a rather strange position in relation to PISA. Out politicians worry greatly about our rankings, but do very little to ensure our students do better in the tests. So almost certainly England will either stay where it is in the rankings, next time round, or drop in the tables.
According to OECD, only around 8% of the variation in student achievement is attributable to the school a child attends. The other 92% is outside the school’s control. Given the way politicians react to PISA results there is surely cause for concern. That is not to say that the information and statistics generated by the PISA process isn’t useful, because a lot of it is . It’s just that the way it is used to make crude comparisons and to generate league tables may well be counter-productive, over the longer term, and the way ,in particular, that politicians react to its headline conclusions rather than the detailed information in the PISA reports, can serve to compound the problems.