Free The Sober Truth by Lance Dodes

method of multivariate analysis known as
structural modeling
, an approach that attempts to develop mathematical equations to explain existing data. 26 Such techniques are not uncommon in medicine, although doubts about their value persist. After a good deal of mathematical fireworks, the study’s authors determined that AA attendance was associated with a reduction in alcohol-related problems, but that reduced alcohol-related problems were not associated with AA attendance. In other words, AA attendance actually
caused
a reduction in alcohol-related problems, rather than simply correlating with them.
    Yet a closer look at the paper’s methodology raises some important questions about both the model and the generalizability of its findings. The researchers didn’t look at a representative sample of the general population. The study was populated by mostly (86 percent) single men, all of whom were veterans and all of whom had been in a 12-step in-patient program previously and were subsequently referred to AA. There were no controls or randomization. About one-quarter of all study participants dropped out and were not considered in the paper’s conclusions.
    This study’s findings hewed fairly closely to what we’ve seen before: at one-year follow-up, hazardous consumption decreased from 93 percent to 42 percent, and decreased further to 37.5 percent at two years (a further 10.7 percent drop); at the same one-year follow-up, 80 percent of the subjects were involved with AA (an increase of 24 percent over baseline). At two years, 68 percent were still involved.
    Yet these numbers, which suggest a strong correlation between AA attendance and sobriety, become less impressive when one looks more closely at the results. After one year, for instance, hazardous use dropped about 50 percent even though AA involvement increased by only 24 percent. It would therefore be difficult to attribute this improvement to AA alone. Far more likely is the possibility that a series of other factors lent a helping hand, including the hospitalization itself. Even
intensive
AA involvement—the kind most associated with better outcomes among AA members—during that first year was reported to be up by just 14 percentage points (from 9.2 percent to 23.3 percent) despite the 50 percent improvement, suggesting again that the improvement may have had to do with factors beyond AA. (More on this in a moment.)
    The key point is a statistical one. When AA involvement and better outcomes move in the same direction, even if they are out of proportion, that represents a plausible correlation that might indeed turn out to be a causal relationship. On the other hand, when the numbers move in
opposite
directions, that is considered a clear negative result. This is precisely what happened in the first two years of the McKellar study. The paper reports that AA involvement decreased by 12 percent, while “hazardous” drinking went down by 11 percent. (The authors defined hazardous drinking as consuming more than four drinks on a drinking day.) This is the reverse of what one would expect if AA were responsible for the improvement: as people dropped out of AA, drinking should have become worse, not better.
    As Harvard biostatistician Richard Gelber notes,
    That both alcohol-related problems and participation in AA seemed to decline between years 1 and 2 by the same amount raises questions about the conclusion from the structural modeling used. It does not pass the common sense test. . . . This direct evidence calls into question conclusions drawn from the structural modeling. 27
    Of course, the demographic issue alone, including the fact that everyone in the study had already been through 12-step treatment before, disqualifies this paper as a representative look at what happens when a cross-section of alcoholics is treated. McKellar and colleagues themselves noted, “Because individuals were not randomly assigned to attend self-help groups, one could argue that