	worried about PISA? not now This web page was first posted in 2010. Since 2010 PISA has begun to change; I can unashamedly declare myself as much a fan of the new directions, as I was a worrier about the old. Andreas Schleicher at the OECD says. "We look very carefully at how the world and the skills that people need are changing and then we try to reflect that" and although that view of how the world may be changing is very (small C) conservative (where is the ability to work remotely but differently across longitude and latitudes, or big & small-data, or algorithms for example) it is a huge step forward.Read about PISA's promising progress here (from 2016) How things are today Put simply, OECD is now rich with useful and informative data. Viewing that data new directions have emerged: PISA has become more about ways of thinking, ways that involve creativity, critical thinking, problem solving and decision making. Speaking in Beijing in 2016 Andreas commented that: "we need to enable students to think for themselves and act for others, and educate the next generation who will create jobs, not seek jobs. Those are the reasons why we are putting global competency at the centre of the next PISA round". He also spoke of attributes like courage, integrity, curiosity, leadership, resilience or empathy, and about tools for working, and which woill include "not just the capacity to use technology but to recognise its potential for new ways of working". And I would say, as would anyone surely, hurrah to all that. Let's do it! How things were back in 2010 This section was written in 2010 and I leave it here because you can sense the progress from a fairly depressing base, to today's exciting new directions (above). Lately the results from the Programme of International Student Assessment (PISA) tests have been increasingly used to justify policy and political rhetoric in the UK and elsewhere. Many of us: teachers, professors, researchers, employers, harbour serious doubts about the foundation that PISA is built on. There is much to learn from PISA, of course, but many doubt its suitability as a platform for any international comparative league tables. Nor is it any kind of valid basis for criticisms of the weaknesses of educational systems - not that education is perfect of course. Because it is too often used and abused in those ways, we regard it as dangerous and indeed potentially damaging. This web page - which seems to be attracting a lot of visits - brings together, and will grow, a number of useful papers and references to help people who, like us, are astonished at the claims made for PISA's usefulness in policy making. Test results vary in the rankings of course. The 2007 TIMSS report suggested that UK students were "top" in Europe, for example. We are not though suggesting that policy makers pick and choose their tests, any more than they should pick and choose findings capriciously from within those tests. This all matters because you only get one chance at being a child and if a politician or policy maker messes up that chance with a capricious cherry picking of policy from what is already a deeply flawed comparative vehicle then we all need to shout out loud and clear that this is wrong. As many others have already observed, with scholarship to back their reflections, any policy making based on the PISA data (whether about school structures, standards or the curriculum) simply cannot be justified. Of course even where PISA with all its faults throws up a "league leader" the smart ones can see that its approach is less than useful in this new millennium. This web page hopefully helps everyone to be smart about the dangers of PISA. It is just a list, lightly annotated. Others have done the hard work, we are simply pointing a lens at their efforts and reflections. But it is a damning collection that should concern any parent, teacher, politician or student hearing the words PISA and policy together in the same speech. Ironically, one of the 'certainties' that are claimed to have emerged from PISA, although certainly ignored completely here in the UK in 2011, is the suggestion that the 'best' achievers are characterised by having collaboration rather than competition between their schools. Perhaps the very many critics of PISA should learn to collaborate more to magnify their voices of dissent. This is not a counsel of despair. Kenneth Chen is the inspirational Under Secretary for Education in Pisa top performer Hong Kong. Speaking at the recent regional leadership forum there in March 2011 he commented that: "learning to learn is the cornerstone of our education reforms in Hong Kong", and "Hong Kong does not come out on top of science teaching globally by continuing to teach science in the old ways", and "I would encourage you all to be moving away from a focus on Content Knowledge". He also reminded us that Singapore's mantra is currently "teach less, so that we might learn more". Wisdom indeed, and very quotable. Of our own criticisms (all of the below and more!) we would perhaps highlight the mismatch between the "met before" curriculum of PISA and the world of uncertainty and surprise that its learners are going out into. New technology allows people to move closer to the margins of what is possible and at those margins things fail spectacularly as we have seen often this century: volcanic ash clouds, deep ocean oil well leaks, a banking collapse. To be able to address these unexpected and surprising problems children need ingenuity, imagination and agility. PISA claims that they test "... students on their ability to adapt the knowledge they acquire at school to real-life situations as opposed to how they master a specific curriculum". The evidence is that it doesn't, or doesn't with enough validity to be useful for policy making. Prof Stephen Heppell and Carole Chapman originally posted February, 2010, last modified March 20, 2020

worried about PISA? not now

This web page was first posted in 2010. Since 2010 PISA has begun to change; I can unashamedly declare myself as much a fan of the new directions, as I was a worrier about the old.

Andreas Schleicher at the OECD says. "We look very carefully at how the world and the skills that people need are changing and then we try to reflect that" and although that view of how the world may be changing is very (small C) conservative (where is the ability to work remotely but differently across longitude and latitudes, or big & small-data, or algorithms for example) it is a huge step forward.Read about PISA's promising progress here (from 2016)

How things are today

Put simply, OECD is now rich with useful and informative data. Viewing that data new directions have emerged:

PISA has become more about ways of thinking, ways that involve creativity, critical thinking, problem solving and decision making. Speaking in Beijing in 2016 Andreas commented that:

"we need to enable students to think for themselves and act for others, and educate the next generation who will create jobs, not seek jobs. Those are the reasons why we are putting global competency at the centre of the next PISA round".

He also spoke of attributes like courage, integrity, curiosity, leadership, resilience or empathy, and about tools for working, and which woill include "not just the capacity to use technology but to recognise its potential for new ways of working".

And I would say, as would anyone surely, hurrah to all that. Let's do it!

How things were back in 2010

This section was written in 2010 and I leave it here because you can sense the progress from a fairly depressing base, to today's exciting new directions (above).

Lately the results from the Programme of International Student Assessment (PISA) tests have been increasingly used to justify policy and political rhetoric in the UK and elsewhere. Many of us: teachers, professors, researchers, employers, harbour serious doubts about the foundation that PISA is built on. There is much to learn from PISA, of course, but many doubt its suitability as a platform for any international comparative league tables. Nor is it any kind of valid basis for criticisms of the weaknesses of educational systems - not that education is perfect of course. Because it is too often used and abused in those ways, we regard it as dangerous and indeed potentially damaging.

This web page - which seems to be attracting a lot of visits - brings together, and will grow, a number of useful papers and references to help people who, like us, are astonished at the claims made for PISA's usefulness in policy making. Test results vary in the rankings of course. The 2007 TIMSS report suggested that UK students were "top" in Europe, for example. We are not though suggesting that policy makers pick and choose their tests, any more than they should pick and choose findings capriciously from within those tests.

This all matters because you only get one chance at being a child and if a politician or policy maker messes up that chance with a capricious cherry picking of policy from what is already a deeply flawed comparative vehicle then we all need to shout out loud and clear that this is wrong. As many others have already observed, with scholarship to back their reflections, any policy making based on the PISA data (whether about school structures, standards or the curriculum) simply cannot be justified. Of course even where PISA with all its faults throws up a "league leader" the smart ones can see that its approach is less than useful in this new millennium.

This web page hopefully helps everyone to be smart about the dangers of PISA. It is just a list, lightly annotated. Others have done the hard work, we are simply pointing a lens at their efforts and reflections. But it is a damning collection that should concern any parent, teacher, politician or student hearing the words PISA and policy together in the same speech.

Ironically, one of the 'certainties' that are claimed to have emerged from PISA, although certainly ignored completely here in the UK in 2011, is the suggestion that the 'best' achievers are characterised by having collaboration rather than competition between their schools. Perhaps the very many critics of PISA should learn to collaborate more to magnify their voices of dissent.

This is not a counsel of despair. Kenneth Chen is the inspirational Under Secretary for Education in Pisa top performer Hong Kong. Speaking at the recent regional leadership forum there in March 2011 he commented that: "learning to learn is the cornerstone of our education reforms in Hong Kong", and "Hong Kong does not come out on top of science teaching globally by continuing to teach science in the old ways", and "I would encourage you all to be moving away from a focus on Content Knowledge". He also reminded us that Singapore's mantra is currently "teach less, so that we might learn more". Wisdom indeed, and very quotable.

Of our own criticisms (all of the below and more!) we would perhaps highlight the mismatch between the "met before" curriculum of PISA and the world of uncertainty and surprise that its learners are going out into. New technology allows people to move closer to the margins of what is possible and at those margins things fail spectacularly as we have seen often this century: volcanic ash clouds, deep ocean oil well leaks, a banking collapse. To be able to address these unexpected and surprising problems children need ingenuity, imagination and agility. PISA claims that they test "... students on their ability to adapt the knowledge they acquire at school to real-life situations as opposed to how they master a specific curriculum". The evidence is that it doesn't, or doesn't with enough validity to be useful for policy making.

Prof Stephen Heppell and Carole Chapman
originally posted February, 2010, last modified March 20, 2020

This chapter is a great starting point. Stefan T. Hopmann/Gertrude Brinek's introduction to their (2007) book "PISA According to PISA – Does PISA Keep What It Promises?". Their introduction is a scholarly overview of everything from the cultural skew, through to PISA's criticised research conduct and significant flaws in the response rates and sampling (with its resulting bias and uncertainties).

http://www.univie.ac.at/pisaaccordingtopisa/introduction_pisaaccordingtopisa.pdf

Gerald W. Bracey - writing in Dissent Magazine in August 2008. "The Leaning (Toppling?) Tower of PISA: Facts and Doubts about International Comparisons in Education".
http://www.dissentmagazine.org/online.php?id=128

This article takes a bit of time to get going, but has some useful criticisms buried within it. For example

"PISA questions ramble discursively and sometimes contain irrelevant information and factually incorrect material. PISA's long questions, administered to 15-year-olds, mean that its assessment of science and math is hopelessly confounded with reading. The correlation between PISA's math and reading assessment, for instance, is .77, indicating a great deal of overlap."

Gerald offers some amusing examples about language to back that up (search article for "scholar's wife")...

Yong Zhao is currently Presidential Chair and Associate Dean for Global Education, College of Education at the University of Oregon, where he also serves as the director of the Center for Advanced Technology in Education (CATE). He is a fellow of the International Academy for Education. He has an honourable track record of criticism of PISA - see for example this blog post:

http://zhaolearning.com/2010/12/10/a-true-wake-up-call-for-arne-duncan-the-real-reason-behind-chinese-students-top-pisa-performance/

or this one: http://zhaolearning.com/2009/09/03/nature-article-interpreting-international-test-scores-is-the-u-s-really-behind/

or if you prefer video try No Child Left Behind and Global Competitiveness? Part of a film series produced by the Mobile Learning Institute's called "Technology and 21st Century Learning"

This misuse of PISA data is not necessarily PISA's own fault. OECD do offer caveats, whcih are then too often ignored. For example in the documentation which accompanies the 2009 PISA data the OECD state that no trends can be drawn between the latest data and those of a decade ago. They say: "Trend comparisons, which are a feature of the PISA 2009 reporting are not reported here because for the United Kingdom it is only possible to compare 2006 and 2009 data".

See a reflection on this from Full Fact . org "Promoting accuracy in public debate", here:
http://fullfact.org/factchecks/michael-gove-school-standards-pisa-oecd_2450

Pass rates in SATs in the UK have doubled in the last 15 years, but this is not matched by PISA data (for all the above reasons, and...). Here is the Local Schools Network's Francis Gilbert exploring "PISA versus SATs — which assess reading the best?"

http://www.localschoolsnetwork.org.uk/2010/12/the-pisa-versus-sats-which-is-the-better-test-of-reading-ability/

We'd do well to take our slice of the Pisa with a large pinch of salt - Peter Wilby towards the end of 2010 writing in the Times Educational Supplement's Connect section.

"The truth is that Pisa cannot offer a guide to the relative success of different school systems. It tests 15-year-olds on the "cumulative impact of learning ... experiences both in school and at home"

http://www.tes.co.uk/article.aspx?storycode=6065654

Donald Clarke doesn't pull any punches with his "Leaning tower of PISA – 7 serious skews" - but it covers some familiar criticisms with bravura.

http://donaldclarkplanb.blogspot.com/2011/03/leaning-tower-of-pisa-7-serious-skews.html

You may need cheering up after all this. We both enjoyed John Sener's Washington Post article Standardized tests prove I'm better than Michael Jordan in April 2011.

still more to come, this is enough for now...

this page last updated Friday, March 20, 2020 3:38 PM