Free Super Crunchers by Ian Ayres

in the average response rate between the two groups must be caused by the difference in their treatment.
    Of course, randomization doesn’t mean that those who were sent photos are each exactly the same as those who were not sent photos. If we looked at the heights of people who received photo solicitations, we would see a bell curve of heights. The point is that we would see the same bell curve of heights for those who received photos without solicitations. Since the
distribution
of both groups becomes increasingly identical as the sample size increases, then we can attribute any differences in the
average
group response to the difference in treatment.
    In lab experiments, researchers create data by carefully controlling for everything to create matched pairs that are identical except for the thing being tested. Outside of the lab, it’s sometimes simply impossible to create pairs that are the same on all the peripheral dimensions. Randomization is how businesses can create data without creating perfectly matched individual pairs. The process of randomization instead creates matched distributions. Randomization thus allows Super Crunchers to run the equivalent of a controlled test without actually having to laboriously match up and control for dozens or hundreds of potentially confounding variables.
    The implications of the randomized marketing trials for lending profitability are pretty obvious. Instead of dropping the interest rate five percentage points, why not simply include a picture? When Credit Indemnity learned the results of the study, they were about to start doing just that. But shortly after the tests were analyzed, the bank was taken over. The new bank not only shut down future testing, it also laid off tons of the Credit Indemnity employees—including those who had been the strongest proponents of testing. Ironically, some of these former employees have taken the lessons of the testing to heart and are now implementing the results in their new jobs working for Credit Indemnity’s competitors.
    You May Be Watching a Random Web Page
    Testing with chance isn’t limited to banks and credit companies; indeed, Offermatica.com has turned Internet randomization into a true art form. Two brothers, Matt and James Roche, started Offermatica in 2003 to capitalize on the ease of randomizing on the Internet. Matt is its CEO and James works as the company’s president. As their name suggests, Offermatica has automated the testing of offers. Want to know whether one web page design works better than another? Offermatica will set up software so that when people click on your site, either one page or the other will be sent at random. The software then can tell you, in real time, which page gets more “click throughs” and which generates more purchases.
    What’s more, they can let you conduct multiple tests at once. Just as Credit Indemnity randomly selected the interest rate and independently decided whether or not to include a photo, Offermatica can randomize over multiple dimensions of a web page’s design.
    For example, Monster.com wanted to test seven different elements of the employers’ home pages. They wanted to know things like whether a link should say “Search and Buy Resumes” or just “Search Resumes” or whether there should be a “Learn More” link or not. All in all, Monster had 128 different page permutations that it wanted to test. But by using the “Taguchi Method,” Offermatica was able to test just eight “recipe” pages and still make accurate predictions about how the untested 120 other web pages would fare. *1
    Offermatica software not only automates the randomization, it also automatically analyzes the Internet response. In real time, as the test was being conducted, Monster could see a continuously updating graph of not only how the eight recipe pages were faring, but also how untested pages would likely fare in