Free A Field Guide to Lies: Critical Thinking in the Information Age by Daniel J. Levitin

error. For example, in a standard two-way poll (in which there are two possibilities for respondents), a random sample of 1,067 American adults will produce a margin of error around 3 percent in either direction (we write ±3%). So if a poll shows that 45 percent of Americans support Candidate A, and 47 percent support Candidate B, the true number is somewhere between 42 and 48 percent for A, and between 44 and 50 percent for B.Note that these ranges overlap. What this means is that the two-percentage-point difference between Candidate A and Candidate B is within the margin of error: We can’t say that one of them is truly ahead of the other, and so the election is too close to call.
    How confident are we that the margin of error is 3 percent and not more? We calculate a confidence interval. In the particular case I mentioned, I reported the 95 percent confidence interval. That means that if we were to conduct the poll a hundred times, using the same sampling methods, ninety-five out of those hundred timesthe interval we obtain will contain the true value.Five times out of a hundred, we’d obtain a value outside that range. The confidence interval doesn’t tell us how far out of the range—it could be a small difference or a large one; there are other statistics to help with that.
    We can set the confidence level to any value we like, but 95 percent is typical. To achieve a narrower confidence interval you can do one of two things: increase the sample size for a given confidence level; or, for a given sample size, decrease the confidence level. For a given sample size, changing your confidence level from 95 to 99 will increase the size of the interval. In most instances the added expense or inconvenience isn’t worth it, given that a variety of external circumstances can change people’s minds the following day or week anyway.
    Note that for very large populations—like that of the United States—we only need to sample a very small percentage, in this case, less than .0005 percent. But for smaller populations—like that of a corporation or school—we require a much larger percentage. For a company with 10,000 employees, we’d need to sample 964 (almost 10 percent) to obtain the 3 percent margin with 95 percent confidence, and for a company of 1,000 employees, we’d need to sample nearly 600 (60 percent).
    Margin of error and confidence interval apply to sampling of any kind, not just samples with people: sampling the proportion of electric cars in a city, of malignant cells in the pancreas, or of mercury in fish at the supermarket. In the figure here, margin of error and sample size are shown for a confidence interval of 95 percent.

    Theformula for calculating the margin of error (and confidence interval) is in the notes at the end of the book, and there are many online calculators to help. If you see a statistic quoted and no margin of error is given, you can calculate the margin yourself, knowing the number of people who were surveyed. You’ll find that in many cases, the reporter or polling organization doesn’t provide this information. This is like a graph without axes—you can lie with statistics very easily by failing to report the margin of error or confidence interval. Here’s one: My dog Shadow is the leading gubernatorial candidate in the state of Mississippi, with 76 percent of votersfavoring him over other candidates (with an unreported margin of error of ±76 percent; vote for Shadow!!!).
    Sampling Biases
    While trying to obtain a random sample, researchers sometimes make errors in judgment about whether every person or thing is equally likely to be sampled.
    An infamous error was made in the 1936 U.S. presidential election. The Literary Digest conducted a poll and concluded that Republican Alf Landon would win over the incumbent Democrat, President Roosevelt. The Digest had polled people who were magazine readers, car owners, or telephone customers, not a random sample. The conventional