A few days ago, when the government (and Nick Clegg in particular) was launching its strategy for social mobility, there was a chart used that made me feel a little uneasy. I couldn’t initially put my finger on why that was the case, but it looked as if it supported the argument rather too well. The chart I’m talking about is reproduced below:
It was used to claim that initially better performing children from poorer families fall back compared with less well performing children from richer families as they get older and so justify action on social mobility. Without doubting for a moment that everyone should have equality of opportunity and be encouraged to be successful in life, somehow the chart looked a little bit suspicious to me.
Someone else had spotted a problem with it as well – a researcher from the business school at Warwick University. He pointed out what I’d been struggling towards recognising – it was more likely to be the result of a statistical phenomena known as regression towards the mean, rather than a genuine reflection of reality.
Regression towards the mean happens when you measure the performance of individuals at the extreme ends of two different groups. The most likely explanation for the pattern seen in the chart is that the good performances from both groups were over-estimates of the child’s ability. Over time, it’s therefore likely that a more realistic view of their performance is obtained through repeated testing – and this is never as good as their best performance that they were selected for right at the start of the process.
Similarly, those selected for particularly poor performances may just have been having an off-day (it happens with small children!) Over time, a more realistic view of their performance is also obtained through repeated testing – and this is never as bad as the performance that they were initially selected for.
Now, I have no doubt that the government’s social mobility strategy is needed and that the intentions behind it are honourable. But the misuse of statistics (and probably where no misuse is required) does nothing to further the cause of reform.
I could also argue that this incident provides a neat illustration as to why removing tuition fees and funding undergraduate courses at universities from general taxation to teach the sciences and social sciences should be seen as a priority …
16th April 2011
One further thing occurs to me this morning – how reliable is an assessment of a child’s performance on a test of ability or IQ at 22 months anyway? The methodological issues in undertaking such an assessment must be huge, be more a matter of subjective than objective measures and subject to a whole range of demand characteristics, from the expectations of the parents to the views of the health worker performing the procedure.