Read Super Crunchers for Free Online

insight they all share—is that data-based decision making doesn’t need to be limited to the conscious preferences of the masses. Instead, it is possible to study the results of decisions and tease out from inside the data the factors that lead to success. This chapter is about how simple regressions are changing decisions by improving predictions. By sifting through aggregations of data, the regression technique can uncover the levers of causation that are hidden to casual and even expert observation. And even when experts feel that a particular factor is an important determinant of some outcome, the regression technique literally can price it out.
    Just for fun, Garth Sundem, in his book
Geek Logik,
used a regression to create a formula to predict how long celebrity marriages will last. (It turns out that having more Google hits reduces a marriage’s chances—especially if the top Google hits include sexually suggestive photos!) eHarmony, Perfectmatch, and True.com are doing the same kind of thing, but they’re doing it for profit. These services are engaged in a new kind of Super Crunching competition. The game’s afoot and it’s a very different kind of game.
    Harrah’s Feels Your Pain
    The same kind of statistical matchmaking is also happening inside companies like Lowe’s and Circuit City, which are using Super Crunching to select job applicants. Employers want to predict which job applicants are going to make a commitment to their job. Unlike traditional aptitude tests that try to suss out an applicant’s IQ, the modern tests are much closer to eHarmony’s questionnaire in trying to evaluate three underlying personality traits of the applicants: their conscientiousness, agreeableness, and extroversion. Data mining shows that these personality traits are better predictors of worker productivity (especially turnover) than more traditional ability testing. Barbara Ehrenreich was appalled when she took an employment test at a Minneapolis Wal-Mart and was told that she had given the wrong answer when she agreed with the proposition “there is room in every corporation for a non-conformist.” Yet regressions suggest that people who think Wal-Mart is for non-conformists aren’t a good fit and are more likely to turn over. It’s one thing to argue that Wal-Mart and other employers should reorganize their mind-numbing jobs to make them less boring. But in a world where mind-numbing jobs are legal, it’s hard for me to see what’s wrong with a statistically validated test that helps match employees that are most compatible with those jobs.
    Mining for non-obvious predictors is not just about hiring good applicants. It’s also helping businesses keep their costs down, especially the costs of stagnant inventory. Businesses that can do a better job of predicting demand can do a better job of predicting when they are about to run out of something. And it can be just as important for businesses to know when they’re
not
about to run out of something. Instead of bearing the costs of large inventories lying around, Super Crunching allows firms to move to just-in-time purchasing. Stores like Wal-Mart and Target try to get as close as possible to having no excess inventory on hand at all. “What they have on the shelf is what they’ve got,” said Scott Gnau, general manager of the data-mining company Teradata. “If I buy six cans of yellow corn off the shelf, and there are now three cans left, somebody knows that happened immediately so they can make sure that the truck coming my way gets some more corn loaded on it. It’s gotten to the point that as you’re putting stuff in your trunk, the store is loading the truck at the distribution center.” These prediction strategies can be based on highly specific details about likely demand. Before Hurricane Ivan hit Florida in 2004, Wal-Mart already had started rushing strawberry Pop-Tarts to