The new age of algorithms: How it affects the way we live
'Big Data' impacts how we work, elect our presidents, and play tennis. It also affects the way we're watched.
(Page 4 of 7)
But isn't banality the point? Historians want to know not just what happened in the past but how people lived. It is why they rejoice in finding a semiliterate diary kept by a Confederate soldier, or pottery fragments in a colonial town.Skip to next paragraph
Subscribe Today to the Monitor
It's as if a historian today writing about Lincoln could listen in on what millions of Americans were saying on the day he was shot.
Youkel and Mandelbaum might seem like an odd couple to carry out a Big Data project: One is a career Library of Congress researcher with an undergraduate degree in history, the other a geologist who worked for years with oil companies. But they demonstrate something Babson's Mr. Davenport has written about the emerging field of analytics: "hybrid specialization."
For organizations to use the new technology well, traditional skills, like computer science, aren't enough. Davenport points out that just as Big Data combines many innovations, finding meaning in the world's welter of statistics means combining many different disciplines.
Mandelbaum and Youkel pool their knowledge to figure out how to archive the tweets, how researchers can find what they want, and how to train librarians to guide them. Even before opening tweets to the public, the library has gotten more than 400 requests from doctoral candidates, professors, and journalists.
"This is a pioneering project," Mr. Dizard says. "It's helping us begin to handle large digital data."
For "America's library," at this moment, that means housing a Gutenberg Bible and Lady Gaga tweets. What will it mean in 50 years? I ask Dizard.
He laughs – and demurs. "I wouldn't look that far ahead."
* * *
Arnold Lund is looking ahead. Lund has a Ph.D. in experimental psychology. He holds 20 patents, has written a book on managing technology design, and directs a variety of projects for General Electric.
Last year, a tree fell on power lines behind my house. As the local utility repaired things, an electrical surge crashed my computer, destroying all the contents. Lund's power line project has my attention.
"For power companies, one of the largest expenses is managing foliage," he says. "We lay out the entire geography of a state – and the overlay of the power grid. We use satellite data to look at tree growth and cut back where there's most growth. Then [we] predict where the most likely [problem] is. We have 50 different variabilities to see the probability of outage."
In that one compressed paragraph, I see three big changes Mr. Cukier and Mr. Mayer-Shönberger say Big Data brings to research. It's what we might call the three "nots."
Size, not sample. For more than a century, they point out, statisticians have relied on small samples of data from which to generalize. They had to. They lacked the ability to collect more. The new technology means we can "collect a lot of data rather than settle for ... samples."
Messy, not meticulous. Traditionally, researchers have insisted on "clean, curated data. When there was not that much data around, researchers [had to be as] exact as possible." Now, that's no longer necessary. "Accept messiness," they write, arguing that the benefits of more data outweigh our "obsession with precision."
Correlation, not cause. While knowing the causes behind things is desirable, we don't always need to understand how the world works "to get things done," they note.
Lund's lab exemplifies all three. First, his "entire geography" and 50 variables involve massive sets of data – information streaming in from sensors, satellites, and other sources about everything from forest density to prevailing wind direction to grid loads. Second, he looks for "probability" not "obsessive precision."
Correlation? Lund values cause, but the reason behind, say, tree growth interests him less than spotting correlations that might spur action. "Ah – that tree," he exclaims, as if he is an engineer in the field. "Better get the trucks out ahead of the storm!"