Free I Think You'll Find It's a Bit More Complicated Than That by Ben Goldacre Page B

it definitely works, and we should use it everywhere. Again, this isn’t necessarily true. Firstly, all trials need to be run properly: if there are flaws in a trial’s design, then it stops being a fair test of the treatments. But more importantly, we need to think carefully about whether the people in a trial of an intervention are the same as the people we are thinking of using the intervention on.
    The Family Nurse Partnership is a programme that is well funded and popular around the world. It was first shown to be effective in a randomised trial in 1977. The trial participants were white mothers in a semi-rural setting in upstate New York, and people worried at the time that the positive results might have been exceptional, and occurred simply because the specific programme of social support that was offered had suited this population unusually well. In 1988, to check that the findings really were applicable to other settings, the same programme was assessed using a randomised trial in African-American mothers in inner-city Memphis, and was again found to be effective. In 1994, a third trial was conducted in a large population of Hispanic, African-American and Caucasian mothers from Denver. After this trial also showed a benefit, people in the US were fairly certain that the programme worked, with fewer childhood injuries, increased maternal employment, improved ‘school readiness’, and more.
    Now the Family Nurse Partnership programme is being brought to Britain, but the people who originally designed the intervention have insisted that a randomised trial should be run here, to see if it really is effective in the very different setting of the UK. They have specifically stated that they expect to see less dramatic benefits here, because the basic level of support for young families in the UK is much better than that in the US: this means that the difference between people getting the FNP programme and people getting the normal level of help from society will be much smaller.
    This is just one example of why we need to be thoughtful about whether the results of a trial in one population really are applicable to our own patients or pupils. It’s also an illustration of why we need to make trials part of the everyday routine, so that we can replicate them in different settings, instead of blindly assuming we can use results from other countries (or even other schools, if they have radically different populations). It doesn’t mean, however, that we can never trust the results of a trial. This is just another example of why it’s useful to know more about how trials work, and to be a thoughtful consumer of evidence.
    Lastly, people sometimes worry that trials are expensive and complicated. This isn’t necessarily true, and it’s important to be clear what the costs of a trial are being compared against. For example, if the choice is between running a trial, and simply charging ahead, implementing an idea that hasn’t been shown to work – one that might be ineffective, wasteful, or even harmful – then it’s clearly worth investing some time and effort in assessing its true impact. If the alternative is doing an ‘observational’ study, which has all the shortcomings described above, then the analysis can be so expensive and complex – not to mention unreliable – that it would have been easier to randomise participants to one intervention or the other in the first place.
    But the mechanics and administrative processes for running a trial can also be kept to a minimum with thoughtful design, for example by measuring outcomes using routine classroom data that was being collected anyway, rather than running a special set of tests. More than anything, though, for trials to be run efficiently, they need to be part of the culture of teaching.
    Making evidence part of everyday life
    I’m struck by how much enthusiasm there is for trials and evidence-based practice in some parts of teaching; but I’m also struck