Tag Archives: policy

Lies, damned lies, and lying to ourselves with statistics

John Ewing, who served as the executive director of the American Mathematical Society for fifteen years, told me that he is perplexed by educators’ “infatuation with data,” their faith that it is more authoritative than using their own judgment. He explains the problem in terms of Campbell’s law, a principle that describes the risks of using a single indicator to measure complex social phenomena: the greater the value placed on a quantitative measure, like test scores, the more likely it is that the people using it and the process it measures will be corrupted. “The end goal of education isn’t to get students to answer the right number of questions,” he said. “The goal is to have curious and creative students who can function in life.”

I don’t know much about the US education system, but this lengthy feature in the New Yorker paints a hair-raising (and at times depressing) picture of a rigidly data-driven monster corrupting everyone it touches.

There’s a general principle here, that any time you build a system you have to think through what exploits of that system would look like, and how you’re going to guard against them. But what emerges here is less a picture of people conspiring for personal gain, rather people fighting the system to provide the evidence it requires that they’re doing the right thing anyway:

To [the mother of a teacher who corrected students’ exam papers], his decision to cheat was an act of civil disobedience. She told him that as soon as she heard about cheating in Atlanta she thought, “I bet my son was part of that.”

The villain of the story isn’t so much corrupt public officials as the insidious temptation of performance metrics. How close is the correlation between things you can measure, and the outcomes you want to achieve?

As the Wikipedia page on Campbell’s Law points out, there’s a parallel in the way the lofty goal of ‘evidence-based policy’ degrades into ‘policy-based evidence.’

Royal Society Vision for science & maths education

A quick post as I head out of the door this morning: the Royal Society have published their document on what they think STEM education should look like by 2030, and how they think we should get there.

They’re doing #ASEChat on Twitter next Monday (8-9pm BST), and in the autumn they’re planning a bunch of discussions about responses to the plan, and how it might be implemented.

I’ve only had chance to skip through the PDFs, but my first impression is: blimey, that looks expensive. I guess I’m not sure whether to cheer the sense of ambition or grouchily snark “Good luck with that.”

I’ll be interested to hear opinions as the discussion develops.