Seven reasons not to trust PISA data
The new GCSE grades are to be benchmarked (at Grade 5) to international standards: the most likely being PISA, where the UK's performance has been static, while other countries have improved in the rankings. However, there are several flaws in the way the data is generated and comparisons made.
- The total number of countries in PISA has risen from 43 in 2000 to 55 in 2012 (was 65 in 2009, some lower performing ones dropped out) – it’s easier to come tenth in a league of 43 than it is in a league of 55.
- Nine countries (Chinese Taipei, Estonia, Hong Kong-China, Macao-China, Netherlands, Shanghai-China, Singapore, Slovenia and Vietnam) above England on the maths scale weren’t entered in 2000.
- In the 2000 wave the UK sample included just children from England and Northern Ireland, but, from 2003 onwards, it also included children from Wales, who have significantly lower levels of achievement.
- PISA data for England (but not other countries) have changed from an age‐based sample in 2000 and 2003 to what is now (effectively) a year group‐based sample (skewing data collection effectively by four months).
- Data collection in England (but not other countries) moved to five months earlier from 2006 onwards – Y11 students take the tests.
- UK response rates were so low in 2000 and 2003 that OECD said “data from the United Kingdom are not comparable to other countries”.
- The focus (major domain) of the study changes each year between English, maths and science, unlike TIMSS data, which shows the opposite trends in UK performance to PISA.
These are based on a round up of comments from a variety of sources.