Schoolzone blog: Seven reasons not to trust PISA data

Seven reasons not to trust PISA datasearch previous posts
Subscribe to get blog posts by email
Subscribe to get blog posts by RSS


17 Oct 2014

The new GCSE grades are to be benchmarked (at Grade 5) to international standards.When we asked a group of 18 senior leaders about this aspect of the reforms, some were in favour ("UK should be ranked against other countries"), some were against ("this idea is meaningless") and most were simply bemused ("does a link to PISA mean that the whole scale potentially shifts up or down every three years?").

Fortunately, Ofqual provided some clarification recently: To try and clarify this, it is not about putting in place any direct links or ties to any grades set elsewhere. Rather, where grade 5 sits within the grading scale will place it above a current grade C, and broadly in line with what the best available evidence tells us is the average performance of 16-year-olds in high performing countries.

So how do we know the average performance of 16-year-olds in high performing countries? The most likely source is PISA, where the UK's performance has been static, while other countries have improved in the rankings. However, there are several flaws in the way the data is generated and comparisons made.

  • The total number of countries in PISA has risen from 43 in 2000 to 55 in 2012 (was 65 in 2009, some lower performing ones dropped out) – it’s easier to come tenth in a league of 43 than it is in a league of 55.
  • Nine countries (Chinese Taipei, Estonia, Hong Kong-China, Macao-China, Netherlands, Shanghai-China, Singapore, Slovenia and Vietnam) above England on the maths scale weren’t entered in 2000.
  • In the 2000 wave the UK sample included just children from England and Northern Ireland, but, from 2003 onwards, it also included children from Wales, who have significantly lower levels of achievement.
  • PISA data for England (but not other countries) have changed from an age‐based sample in 2000 and 2003 to what is now (effectively) a year group‐based sample (skewing data collection effectively by four months).
  • Data collection in England (but not other countries) moved to five months earlier from 2006 onwards – Y11 students take the tests.
  • UK response rates were so low in 2000 and 2003 that OECD said “data from the United Kingdom are not comparable to other countries”.
  • The focus (major domain) of the study changes each year between English, maths and science, unlike TIMSS data, which shows the opposite trends in UK performance to PISA.

(These are based on a round up of comments from a variety of sources.)

Watch out for this, though: it seems on the face of it to be a low profile component of the GCSE reforms, but if the government is serious about driving up performance against international standards, it will obviously need to drive up response rates as well as performance, which presumably means there will be pressure to schools to do more.




HTML Comment Box is loading comments...

Schoolzone                                               Contact us
Court Mews 268 London Rd                 01242262906
Cheltenham GL52 6HS

Accessibility   Privacy and cookie policy   Terms and conditions  

© Ltd 2015
. All rights reserved