Free Bad Science by Ben Goldacre

the Jadad score tends toward the top mark of 5, as the trials become more of a “fair test,” the line tends toward showing that homeopathy performs no better than placebo. There is, however, a mystery in this graph, an oddity, and the makings of a whodunit. That little dot on the right-hand edge of the graph, representing the ten best-quality trials, with the highest Jadad scores, stands clearly outside the trend of all the others. This is an anomalous finding; suddenly, only at that end of the graph, there are some good-quality trials bucking the trend and showing that homeopathy is better than placebo.

     
    What’s going on there? I can tell you what I think: some of the papers making up that spot are rigged. I don’t know which ones, how it happened, or who did it, in which of the ten papers, but that’s what I think. Academics often have to couch strong criticism in diplomatic language. Here is Professor Ernst, the man who made that graph, discussing the eyebrow-raising outlier. You might decode his political diplomacy and conclude that he thinks there’s been a fix too.
    There may be several hypotheses to explain this phenomenon. Scientists who insist that homeopathic remedies are in every way identical to placebos might favor the following. The correlation provided by the four data points (Jadad score 1–4) roughly reflects the truth. Extrapolation of this correlation would lead them to expect that those trials with the least room for bias (Jadad score = 5) show homeopathic remedies are pure placebos. The fact, however, that the average result of the 10 trials scoring 5 points on the Jadad score contradicts this notion, is consistent with the hypothesis that some (by no means all) methodologically astute and highly convinced homeopaths have published results that look convincing but are, in fact, not credible.
     
    But this is a curiosity and an aside. In the bigger picture it doesn’t matter, because overall, even including these suspicious studies, the meta-analyses still show, overall, that homeopathy is no better than placebo.
    Meta-analyses?
    Meta-Analysis
     
    This will be our last big idea for a while, and this is one that has saved the lives of more people than you will ever meet. A meta-analysis is a very simple thing to do, in some respects: you just collect all the results from all the trials on a given subject, bung them into one big spreadsheet, and do the math on that, instead of relying on your own gestalt intuition about all the results from each of your little trials. It’s particularly useful when there have been lots of trials, each too small to give a conclusive answer, but all looking at the same topic.
    So if there are, say, ten randomized, placebo-controlled trials looking at whether asthma symptoms get better with homeopathy, each of which has a paltry forty patients, you could put them all into one meta-analysis and effectively (in some respects) have a four-hundred-person trial to work with.
    In some very famous cases—at least, famous in the world of academic medicine—meta-analyses have shown that a treatment previously believed to be ineffective is in fact rather good, but because the trials that had been done were each too small, individually, to detect the real benefit, nobody had been able to spot it.
    As I said, information alone can be lifesaving, and one of the greatest institutional innovations of the past thirty years is undoubtedly the Cochrane Collaboration, an international not-for-profit organization of academics that produces systematic summaries of the research literature on health care research, including meta-analyses.
    The logo of the Cochrane Collaboration features a simplified blobbogram, a graph of the results from a landmark meta-analysis that looked at an intervention given to pregnant mothers. When women give birth prematurely, as you might expect, the babies are more likely to suffer and die. Some doctors in New Zealand had the idea that giving a short,