The new age of algorithms: How it affects the way we live
'Big Data' impacts how we work, elect our presidents, and play tennis. It also affects the way we're watched.
They work a few hundred yards from one of the Library of Congress's most prized possessions: a vellum copy of the Bible printed in 1455 by Johann Gutenberg, inventor of movable type. But almost six centuries later, Jane Mandelbaum and Thomas Youkel have a task that would confound Gutenberg.Skip to next paragraph
Subscribe Today to the Monitor
The researchers are leading a team that is archiving almost every tweet sent out since Twitter began in 2006. A half-billion tweets stream into library computers each day.
Their question: How can they store the tweets so they become a meaningful tool for researchers – a sort of digital transcript providing insights into the daily flow of history?
Thousands of miles away, Arnold Lund has a different task. Mr. Lund manages a lab for General Electric, a company that still displays the desk of its founder, Thomas Edison, at its research headquarters in Niskayuna, N.Y. But even Edison might need training before he'd grasp all the dimensions of one of Lund's projects. Lund's question:
How can power companies harness the power of data to predict which trees will fall on power lines during a storm – thus allowing them to prevent blackouts before they happen?
The work of Richard Rothman, a professor at Johns Hopkins University in Baltimore, is more fundamental: to save lives. The Centers for Disease Control and Prevention (CDC) in Atlanta predicts flu outbreaks, once it examines reports from hospitals. That takes weeks. In 2009, a study seemed to suggest researchers could predict outbreaks much faster by analyzing millions of Google searches.
Spikes in queries like "My kid is sick" signaled a flu outbreak before the CDC knew there would be one. That posed a new question for Dr. Rothman and his colleague Andrea Dugas:
Could Google help predict influenza outbreaks in time to allow hospitals like the one at Johns Hopkins to get ready?
They ask different questions. But all five of these researchers form part of the new world of Big Data – a phenomenon that may, for better or worse, revolutionize every facet of life, culture, and, well, even the planet. From curbing urban crime to calculating the effectiveness of a tennis player's backhand, people are now gathering and analyzing vast amounts of data to predict human behaviors, solve problems, identify shopping habits, thwart terrorists – everything but foretell which Hollywood scripts might make blockbusters. Actually, there's a company poring through numbers to do that, too.
Just four years ago, someone wanted to do a Wikipedia entry on Big Data. Wikipedia said no; there was nothing special about the term – it just combined two common words. Today, Big Data seems everywhere, ushering in what advocates consider some of the biggest changes since Euclid.