Lies, damned lies and social mobility statistics

A few days ago, when the government (and Nick Clegg in particular) was launching its  strategy for social mobility, there was a chart used that made me feel a little uneasy. I couldn’t initially put my finger on why that was the case, but it looked as if it supported the argument rather too well. The chart I’m talking about is reproduced below:

A problem of social mobility or regression towards the mean?
A problem of social mobility or regression towards the mean?

It was used to claim that initially better performing children from poorer families fall back compared with less well performing children from richer families as they get older and so justify action on social mobility. Without doubting for a moment that everyone should have equality of opportunity and be encouraged to be successful in life, somehow the chart looked a little bit suspicious to me.

Someone else had spotted a problem with it as well – a researcher from the business school at Warwick University. He pointed out what I’d been struggling towards recognising – it was more likely to be the result of a statistical phenomena known as regression towards the mean, rather than a genuine reflection of reality.

Regression towards the mean happens when you measure the performance of individuals at the extreme ends of two different groups. The most likely explanation for the pattern seen in the chart is that the good performances from both groups were over-estimates of the child’s ability. Over time, it’s therefore likely that a more realistic view of their performance is obtained through repeated testing – and this is never as good as their best performance that they were selected for right at the start of the process.

Similarly, those selected for particularly poor performances may just have been having an off-day (it happens with small children!) Over time, a more realistic view of their performance is also obtained through repeated testing – and this is never as bad as the performance that they were initially selected for.

Now, I have no doubt that the government’s social mobility strategy is needed and that the intentions behind it are honourable. But the misuse of statistics (and probably where no misuse is required) does nothing to further the cause of reform.

I could also argue that this incident provides a neat illustration as to why removing tuition fees and funding undergraduate courses at universities from general taxation to teach the sciences and social sciences should be seen as a priority …

16th April 2011

One further thing occurs to me this morning – how reliable is an assessment of a child’s performance on a test of ability or IQ at 22 months anyway? The methodological issues in undertaking such an assessment must be huge, be more a matter of subjective than objective measures and subject to a whole range of demand characteristics, from the expectations of the parents to the views of the  health worker performing the procedure.

Posts Tagged with…

Reader Comments

    • tim

      Interesting, but my wider point still holds – the graphs are meaningless as the samples are fatally biased to begin with. And with two sample groups, you will always have two means! Any conclusions as to the relative performance of individuals in the two samples is therefore moot – because the samples are meaningless to begin with.

      I’m not sure how you conclude that the graphs are showing brightness (or IQ perhaps) – they’re not. Look at the vertical axis. They are showing relative performance on a set of unspecified tests. As a child gets older, it’s highly likely that more advantaged parents will be able to use their resources to aid a child’s development more than those who are less advantaged. Which is why action on social mobility is required – we should be giving all children equality of opportunity.

      So the point I make in my original post still stands – dodgy statistics and the personal views of a professor that I don’t know notwithstanding. I have no doubt that the government’s social mobility strategy is needed and that the intentions behind it are honourable. But the misuse of statistics (and probably where no misuse is required) does nothing to further the cause of reform.

  1. Warwick professor thinks poor kids are stupid | Contrasting Sounds

    […] couple of fellow bloggers (Tim at Just One More Ten Pence Piece and Jonathan at Liberal England) seem to have been blinded by science. I don’t blame them, […]

  2. Six of the Best 150 | Henry Abraham

    […] (and probably where no misuse is required) does nothing to further the cause of reform.” Just One More Ten Pence Piece… introduces us to the concept of reversion to the […]

Your thoughts?