Let’s say the risk of having a heart attack in your fifties is 50 per cent higher if you have high cholesterol. That sounds pretty bad. Let’s say the extra risk of having a heart attack if you have high cholesterol is only 2 per cent. That sounds OK to me. But they’re the same (hypothetical) figures.Read more at location 3582
Out of a hundred men in their fifties with normal cholesterol, four will be expected to have a heart attack; whereas out of a hundred men with high cholesterol, six will be expected to have a heart attack. That’s two extra heart attacks per hundred. Those are called ‘natural frequencies’. Natural frequencies are readily understandable, because instead of using probabilities, or percentages, or anything even slightly technical or difficult, they use concrete numbers, just like the ones you use every day to check if you’ve lost a kid on a coach trip, or got the right change in a shop. Lots of people have argued that we evolved to reason and do maths with concrete numbers like these, and not with probabilities, so we find them more intuitive.Read more at location 3584
you could have a 50 per cent increase in risk (the ‘relative risk increase’); or a 2 per cent increase in risk (the ‘absolute risk increase’); or, let me ram it home, the easy one, the informative one, an extra two heart attacks for every hundred men, the natural frequency.Read more at location 3591
Try this, on bowel cancer, from the Today programme on Radio 4: ‘A bigger risk meaning what, Professor Bingham?’ ‘A third higher risk.’ ‘That sounds an awful lot, a third higher risk; what are we talking about in terms of numbers here?’ ‘A difference … of around about twenty people per year.’ ‘So it’s still a small number?’ ‘Umm … per 10,000…’Read more at location 3595
The reports were based on a study that had observed participants over four years, and the results suggested, using natural frequencies, that you would expect one extra heart attack for every 1,005 people taking ibuprofen.Read more at location 3605
‘British research revealed that patients taking ibuprofen to treat arthritis face a 24 per cent increased risk of suffering a heart attack.’ Feel the fear. Almost everyone reported the relative risk increases:Read more at location 3607
academics can themselves be as guilty as the rest when it comes to overdramatising their researchRead more at location 3614
Over a hundred years ago, H.G. Wells said that statistical thinking would one day be as important as the ability to read and write in a modern technological society. I disagree; probabilistic reasoning is difficult for everyone, but everyone understands normal numbers.Read more at location 3621
The Independent was in favour of legalising cannabis for many years, but in March 2007 it decided to change its stance. One option would have been simply to explain this as a change of heart, or a reconsideration of the moral issues. Instead it was decorated with science—as cowardly zealots have done from eugenics through to prohibition—and justified with a fictitious change in the facts.Read more at location 3627
Twice in this story we are told that cannabis is twenty-five times stronger than it was a decade ago.Read more at location 3634
The data from the Laboratory of the Government Chemist goes from 1975 to 1989. Cannabis resin pootles around between 6 per cent and 10 per cent THC, herbal between 4 per cent and 6 per cent. There is no clear trend. The Forensic Science Service data then takes over to produce the more modern figures, showing not much change in resin, and domestically produced indoor herbal cannabis doubling in potency from 6 per cent to around 12 or 14 per cent. (2003–05 data in table under references).Read more at location 3641
The rising trend of cannabis potency is gradual, fairly unspectacular, and driven largely by the increased availability of domestic, intensively grown indoor herbal cannabis.Read more at location 3647
‘Twenty-five times stronger’, remember. Repeatedly, and on the front page. If you were in the mood to quibble with the Independent’s moral and political reasoning, as well as its evident and shameless venality, you could argue that intensive indoor cultivation of a plant which grows perfectly well outdoors is the cannabis industry’s reaction to the product’s illegality itself.Read more at location 3653
In the mid-1980s, during Ronald Reagan’s ‘war on drugs’ and Zammo’s ‘Just say no’ campaign on Grange Hill, American campaigners were claiming that cannabis was fourteen times stronger than in 1970. Which sets you thinking. If it was fourteen times stronger in 1986 than in 1970, and it’s twenty-five times stronger today than at the beginning of the 1990s, does that mean it’s now 350 times stronger than in 1970? That’s not even a crystal in a plant pot. It’s impossible.Read more at location 3664
‘Use of the addictive drug by children doubles in a year,’ said the subheading. Was this true?Read more at location 3672
If you read the press release for the government survey on which the story is based, it reports ‘almost no change in patterns of drug use, drinking or smoking since 2000’. But this was a government press release, and journalists are paid to investigate:Read more at location 3673
You can download the full document online. It’s a survey of 9,000 children, aged eleven to fifteen, in 305 schools. The three-page summary said, again, that there was no change in prevalence of drug use. If you look at the full report you will find the raw data tables: when asked whether they had used cocaine in the past year, 1 per cent said yes in 2004, and 2 per cent said yes in 2005. So the newspapers were right: it doubled? No. Almost all the figures given were 1 per cent or 2 per cent.Read more at location 3676
The actual figures were 1.4 per cent for 2004, and 1.9 per cent for 2005, not 1 per cent and 2 per cent.Read more at location 3681
What we now have is a relative risk increase of 35.7 per cent, or an absolute risk increase of 0.5 per cent. Using the real numbers, out of 9,000 kids we have about forty-five more saying ‘Yes’ to the question ‘Did you take cocaine in the past year?’ Presented with a small increase like this, you have to think: is it statistically significant?Read more at location 3683
It’s just a way of expressing the likelihood that the result you got was attributable merely to chance. Sometimes you might throw ‘heads’ five times in a row, with a completely normal coin, especially if you kept tossing it for long enough.Read more at location 3686
The standard cut-off point for statistical significance is a p-value of 0.05, which is just another way of saying, ‘If I did this experiment a hundred times, I’d expect a spurious positive result on five occasions, just by chance.’Read more at location 3689
To ‘data mine’, taking it out of its real-world context, and saying it is significant, is misleading. The statistical test for significance assumes that every data point is independent, but here the data is ‘clustered’, as statisticians say. They are not data points, they are real children, in 305 schools. They hang out together, they copy each other, they buy drugs from each other, there are crazes, epidemics, group interactions.Read more at location 3695
Note: MA ATTENZIONE: SS ASSUME CHE I CASI OSSERVATI SIANO INDIPENDENTI E NN CONNESSI COME LO SONO NEL MONDO REALE. È TALMENTE VERO CHE LE REPLICHE NN RISPETTANO CERTO IL 5% CANONICO Edit
The increase of forty-five kids taking cocaine could have been a massive epidemic of cocaine use in one school,Read more at location 3698
As statisticians would say, you must ‘correct for clustering’. This is done with clever maths which makes everyone’s head hurt. All you need to know is that the reasons why you must ‘correct for clustering’ are transparent, obvious and easy, as we have just seenRead more at location 3704
When you correct for clustering, you greatly reduce the significance of the results.Read more at location 3707
Will our increase in cocaine use, already down from ‘doubled’ to ‘35.7 per cent’, even survive? No. Because there is a final problem with this data: there is so much of it to choose from. There are dozens of data points in the report: on solvents, cigarettes, ketamine, cannabis, and so on. It is standard practice in research that we only accept a finding as significant if it has a p-value of 0.05 or less. But as we said, a p-value of 0.05 means that for every hundred comparisons you do, five will be positive by chance alone. From this report you could have done dozens of comparisons, and some of them would indeed have shown increases in usage—but by chance alone, and the cocaine figure could be one of those.Read more at location 3708
Note: ALTRO PROB: QUANDO TESTI MOLTE RELAZIONI LA PROB CHE ALCUNE SIANO POSOTIVE PER PURO CASO AUMENTA E QUINDI LA TENTAZIONE DI ASSUMERLE SCARTANDO LE ALTRE. IL METODO SCIENTIFICO IMPONE INVECE DI FORMULARE PRIMA LE IPOTESI E POI VERIFICARE Edit
If you roll a pair of dice often enough, you will get a double six three times in a row on many occasions.Read more at location 3713
This is why statisticians do a ‘correction for multiple comparisons’, a correction for ‘rolling the dice’ lots of times. This, like correcting for clustering, is particularly brutal on the data, and often reduces the significance of findings dramatically.Read more at location 3714
Note: CORREZIONE DELLE COMPARAZIONI MULTIPLE. CORREZIONE DI DEONTOLOGIA METODOLOGICA PIÙ CHE FATTUALE. Edit
There are also some perfectly simple ways to generate ridiculous statistics, and two common favourites are to select an unusual sample group, and to ask them a stupid question. Let’s say 70 per cent of all women want Prince Charles to be told to stop interfering in public life. Oh, hang on—70 per cent of all women who visit my website want Prince Charles to be told to stop interfering in public life.Read more at location 3724
Telegraph in the last days of 2007. ‘Doctors Say No to Abortions in their Surgeries’ was the headline. ‘Family doctors are threatening a revolt against government plans to allow them to perform abortions in their surgeries,Read more at location 3729
‘Four out of five GPs do not want to carry out terminations even though the idea is being tested in NHS pilot schemes, a survey has revealed.’Read more at location 3731
It was an online vote on a doctors’ chat site that produced this major news story. Here is the question, and the options given: ‘GPs should carry out abortions in their surgeries’ Strongly agree, agree, don’t know, disagree, strongly disagree.Read more at location 3734
Is that ‘should’ as in ‘should’? As in ‘ought to’?Read more at location 3737
Are they just saying no because they’re grumbling about more work and low morale? More than that, what exactly does ‘abortion’ mean here?Read more at location 3739
In 2006, after a major government report, the media reported that one murder a week is committed by someone with psychiatric problems. Psychiatrists should do better, the newspapers told us, and prevent more of these murders. All of us would agree,Read more at location 3755
the blood test for HIV has a very high ‘sensitivity’, at 0.999. That means that if you do have the virus, there is a 99.9 per cent chance that the blood test will be positive. They would also say the test has a high ‘specificity’ of 0.9999—so, if you are not infected, there is a 99.99 per cent chance that the test will be negative. What a smashing blood test.* But if you look at it from the perspective of the person being tested, the maths gets slightly counterintuitive. Because weirdly, the meaning, the predictive value, of an individual’s positive or negative test is changed in different situations, depending on the background rarity of the event that the test is trying to detect. The rarer the event in your population, the worse your test becomes, even though it is the same test. This is easier to understand with concrete figures. Let’s say the HIV infection rate among high-risk men in a particular area is 1.5 per cent. We use our excellent blood test on 10,000 of these men, and we can expect 151 positive blood results overall: 150 will be our truly HIV-positive men, who will get true positive blood tests; and one will be the one false positive we could expect from having 10,000 HIV-negative men being given a test that is wrong one time in 10,000. So, if you get a positive HIV blood test result, in these circumstances your chances of being truly HIV positive are 150 out of 151. It’s a highly predictive test. Let’s now use the same test where the background HIV infection rate in the population is about one in 10,000. If we test 10,000 people, we can expect two positive blood results overall. One from the person who really is HIV positive; and the one false positive that we could expect, again, from having 10,000 HIV-negative men being tested with a test that is wrong one time in 10,000. Suddenly, when the background rate of an event is rare, even our previously brilliant blood test becomes a bit rubbish. For the two men with a positive HIV blood test result, in this population where only one in 10,000 has HIV, it’s only 50:50 odds on whether they really are HIV positive.Read more at location 3762
Let’s think about violence. The best predictive tool for psychiatric violence has a ‘sensitivity’ of 0.75, and a ‘specificity’ of 0.75. It’s tougher to be accurate when predicting an event in humans, with human minds and changing human lives.Read more at location 3778
Note: GLI PSICHIATRI HANNO TEST CON FALSI POSITIVI NOTEVOLI. IMPOSSIBILE AGIRE CON I POTENZIALI OMICIDI DATE LE FREQ DI BASE Edit
Let’s say 5 per cent of patients seen by a community mental health team will be involved in a violent event in a year. Using the same maths as we did for the HIV tests, your ‘0.75’ predictive tool would be wrong eighty-six times out of a hundred. For serious violence, occurring at 1 per cent a year, with our best ‘0.75’ tool, you inaccurately finger your potential perpetrator ninety-seven times out of a hundred. Will you preventively detain ninety-seven people to prevent three violent events?Read more at location 3780
In 1999 solicitor Sally Clark was put on trial for murdering her two babies.Read more at location 3790
At her trial, Professor Sir Roy Meadow, an expert in parents who harm their children, was called to give expert evidence. Meadow famously quoted ‘one in seventy-three million’ as the chance of two children in the same family dying of Sudden Infant Death Syndrome (SIDS).Read more at location 3792
The figure of ‘one in seventy-three million’ itself is iffy, as everyone now accepts. It was calculated as 8,543 × 8,543, as if the chances of two SIDS episodes in this one family were independent of each other. This feels wrong from the outset, and anyone can see why: there might be environmental or genetic factors at play, both of which would be shared by the two babies.Read more at location 3798
Many press reports at the time stated that one in seventy-three million was the likelihood that the deaths of Sally Clark’s two children were accidental: that is, the likelihood that she was innocent.Read more at location 3804
Once this rare event has occurred, the jury needs to weigh up two competing explanations for the babies’ deaths: double SIDS or double murder. Under normal circumstances—before any babies have died—double SIDS is very unlikely, and so is double murder.Read more at location 3808
If we really wanted to play statistics, we would need to know which is relatively more rare, double SIDS or double murder. People have tried to calculate the relative risks of these two events, and one paper says it comes out at around 2:1 in favour of double SIDS.Read more at location 3810
the rarity of double SIDS is irrelevant, because double murder is rare too.Read more at location 3817
You know, the most amazing thing happened to me tonight. I was coming here, on the way to the lecture, and I came in through the parking lot. And you won’t believe what happened. I saw a car with the license plate ARW 357. Can you imagine? Of all the millions of license plates in the state, what was the chance that I would see that particular one tonight? Amazing… Richard FeynmanRead more at location 3825
A nurse called Lucia de Berk has been in prison for six years in Holland, convicted of seven counts of murder and three of attempted murder. An unusually large number of people died when she was on shift, and that, essentially, along with some very weak circumstantial evidence, is the substance of the case against her.Read more at location 3829
The judgement was largely based on a figure of ‘one in 342 million against’.Read more at location 3832
It’s only weird and startling when something very, very specific and unlikely happens if you have specifically predicted it beforehand.Read more at location 3836
Imagine I am standing near a large wooden barn with an enormous machine gun. I place a blindfold over my eyes and—laughing maniacally—I fire off many thousands and thousands of bullets into the side of the barn. I then drop the gun, walk over to the wall, examine it closely for some time, all over, pacing up and down. I find one spot where there are three bullet holes close to each other, then draw a target around them, announcing proudly that I am an excellent marksman.Read more at location 3838
a cardinal rule of any research involving statistics: you cannot find your hypothesis in your results.Read more at location 3844
a rather complex, philosophical, mathematical form of circularity:Read more at location 3847
To collect more data, the investigators went back to the wards to see if they could find more suspicious deaths. But all the people who were asked to remember ‘suspicious incidents’ knew that they were being asked because Lucia might be a serial killer. There was a high risk that ‘an incident was suspicious’ became synonymous with ‘Lucia was present’.Read more at location 3848
