Making subjects comparable
4 Jan 2016
You'd be hard pushed to find anyone who thought that all subjects were comparable in terms of difficulty, who'd say that an A* in textiles was equally difficult to achieve as one in say physics or further maths. We all have our own perceptions of what's more difficult, or worth more: ask anyone giving careers or UCAS advice to students and they'll have a notional hierarchy in mind.
So, can Ofqual do anything to help balance the imbalance? Should they award more A* grades to difficult subjects? Or fewer? Should they offer the same percentage of students each year the same number of grades? They are currently considering four main alternatives (one of which is to do nothing) to see if they can make subjects comparable and have published a series of working papers to support the discussion - the most recent, being very interesting reading if, like me, you're an assessment geek.
At the heart of this discussion is the question, "what is assessment for?" and there are three main uses for it:
- so students can see where their strengths and weaknesses are
- so that the government can appraise the quality of teaching
- so that employers and universities can select applicants.
Ofqual's discussion focuses chiefly on the second and third and these are the uses that incline assessments to norm-referenced (stats based) approaches, whereas the first inclines us more towards criterion-referenced (what the student can do). We might think that the third use would incline towards criterion-referenced, but in practice, these groups just look at grades - grades have never really told anyone anything much about what students can actually do as it's way too complicated to understand the relationships between grades and assessment objectives.
Students choose subjects according to their preferences - those they like and / or are likely to do well in. For them, difficulty is a personal issue and it could be argued that they don't need any Ofqual comparability factors to balance the differences. But they do have to try to anticipate the values associated with subjects by employers and universities, so this makes it a gamble: do they go for an easy subject if it's going to have less currency?
The government has already gone some way to smoothing the differences between subjects, along with student prior attainment differences and relative "importance" of subjects, by introducing the new P8 measure. Though nothing at A-level to help with comparisons between schools, where raw performance will remain as the means to appraise the quality of teaching.
Which leaves us with universities and employers: do they need a scaling factor? If they're looking for applicants with physics A-level, for example, they don't generally care about whether someone with the same grade in textiles is "equivalent" or not. It's more important to them that the grades provide a differentiator (ie that they reflect the spread of ability), that they are equivalent year on year and between awarding bodies and so on.
So for us assessment geeks, it's really interesting to discuss the possibilities of solving the subject comparability issue, but none of Ofqual's suggested options seem terribly compelling as possible solutions because it's not clear what the problem is - there are many problems with grades, and there are different problems for different users.
Here's an alternative, which is suggested by early consultations for the P8 measure: individual subject progress measures. So students get their grade 2/B in GCSE/A-level physics, as now/soon, but they also get a grade compared to their scaled SATs scores: they could also see how well they performed in each subject compared to the others (value-added) and compared to the national performance for that year/all years etc. This requires nothing new and no further changes to anything before the first new GCSEs and A-levels are examined in 2017. Admittedly it means there's a lot of data for each student to carry around, but we live in a digital age where data is nothing if not portable.
So instead of just a grade 2 in GCSE physics, a student might have:
|ie can do what anyone else with this grade can do||ie did better in this subject than an average of the other subjects taken by this student||ie did a bit worse than they should have done compared to everyone with similar prior attainment|
Just a thought.