Lies, damn lies and poor interpretations of statistics

Funnily enough, when I was flying back from Japan, I saw a programme called Freakonomics based on the book of the same name. I hadn’t read the book but understood it was a lateral look at the myths that can abound when people make wrong inferences from data. Well, two outcomes that stuck in my somewhat jetlagged nut. One related to Japan. The other to New York.

First the Japanese angle. What the documentary showed was that when a society has, it imagines, in-built ethics or traditional cultural behaviours, it will not face the consequences of these being brought into disrepute. We saw sumo wrestlers. Here is an ancient art, a rule-ridden battle between men whose weight is more significant than their body shapes. Giants who are fed like the carp in Japanese ponds until their mountainous flesh vies with Fuji itself. They fight their way to the top where the pickings can be massive and include enamoured, beautiful, slight-bodied women. And NO-ONE could envisage even the sniff of corruption in this stylised, mannered world. No single event had come to light until an ex-sumo coach said that fights were fixed. Uproar. The media tried to shut him up. But our intrepid statisticians looked at the data and, over thousands of fights they found an anomaly. There was a huge bias where victories by opponents did not hurt the losers because they had already enough victories under their great leather belts to progress to the next championships. The Japanese federation accepted that in certain cases, injuries among sumo wrestlers (some died in training) were the result of abuse but would not accept the statistical pattern showing that a big proportion of fights was rigged. This notion that culture has its purities that we need to believe in, regardless of the true state of affairs, rings a bell, does it not? Politicians peddle it when they defend the police, secret services, religious schools, the royals, foreign conquest and so on. We are not expected to challenge these verities. Wikileaks exposed not only the malady in ambassadorial life but also the hypocrisies that abound when people like Hilary Clinton defend their nasty email injunctions to spy on America’s friends.

The other angle was the New York one. I have to admit that I was a sucker for the story about how New York was reformed by its Mayor and its policy of zero tolerance. By jumping on the small details of antisocial behaviour (breaking windows, throwing litter, petty stealing) the big ones do not occur in the same numbers. What a great breakthrough in public order. Well, our number crunchers looked at the data. What did they find? Following Ceaucescu’s fall from power in Romania it was discovered that he had forced women to have babies to swell the labour market. Twenty years later these unwanted babies had inflated the criminal ranks of the country. At around this time, the American Senate passed what seemed to be an unpopular law which allowed women to have abortion. Twenty years later, the crime figures for New York and elsewhere had dropped sharply. Unwanted pregnancies had been so diminished. Of course, religious fundamentalists don’t like it that abortion is a possible tool for ensuring babies are born to their mothers when they are ready to support them. Others on the liberal front attack the connection on the grounds that it appears to be biased against the poor, suggesting they are poor parents and irresponsible. But the causality is striking, make of it what you may.

Two examples from Freakonomics. You have to be brave to challenge social orthodoxies that underpin your culture and you have to control your own destiny, regardless of the State. But you may die for such causes. Take Punjab Governor Salman Taseer who was assassinated this week for wanting to repeal blasphemy laws. Laws that do not stem from the Koran but from bigots who want to limit freedom of speech and frank debate about religious and other pillars of Pakistan society.

Labels: Freedom of speech. Bad statistics-based policies. Being brave.