The new age of algorithms: How it affects the way we live
'Big Data' impacts how we work, elect our presidents, and play tennis. It also affects the way we're watched.
(Page 3 of 7)
Most of this data doesn't affect us. Amassing information alone doesn't mean it's valuable. Yet the new ability to mine the right information, discover patterns and relationships, already affects our everyday lives.Skip to next paragraph
Subscribe Today to the Monitor
Anyone, for instance, who has a navigation screen on a car dashboard uses data streaming from 24 satellites 11,000 miles above Earth to pinpoint his or her exact location. People living in Los Angeles and dozens of other cities now participate, knowingly or not, in the growing phenomenon of "predictive policing" – authorities' use of algorithms to identify crime trends. Tennis fans use IBM SlamTracker, an online analytic tool, to find out exactly how many return of serves Andy Murray needed to win Wimbledon.
When we use sites like SlamTracker, companies take note of our browsing habits and, through either the miracle or the meddling of Big Data, use that information to send us personal pitches. That's what happens when AOL greets you with a pop-up ad (Slazenger tennis balls – 70 percent off!).
In their book, "Big Data: A Revolution That Will Transform How We Live, Work, and Think," Kenneth Cukier and Viktor Mayer-Schönberger mention Wal-Mart's discovery, gleaned by mining sales data, that people preparing for a hurricane bought lots of Pop-Tarts. Now, when a storm is on the way, Wal-Mart puts Pop-Tarts on the shelves next to the flashlights.
But what excites and concerns people about Big Data is more far-reaching than that. One way of seeing the bigger picture: taking a closer look at some of the people in the digital trenches.
* * *
I follow Mandelbaum and Mr. Youkel down a corridor of the Library of Congress, past exhibits redolent of history and what you might expect from what we call "America's library," with its 38 million books on 838 miles of shelving.
They open a door. We pass behind people staring at huge computer screens and enter a room that doesn't look as if it belongs in a library at all. It's the size of a gym, with fluorescent lights overhead and tall metal boxes rising from the floor.
"The tweets come here," Mandelbaum says.
It's been three years since Twitter approached the library with a question. What the online networking service started in 2006 had become a new way of communicating. Would there, Twitter asked, be historical value in archiving tweets?
"We saw the value right away," says Robert Dizard, deputy director of the library. "[Our] mission is, preserve the record of America."
Certainly the record of what millions of Americans say, think, and feel each day would be a treasure-trove for historians. But was the technology feasible, and – important for a federal agency – cost-effective to handle the three V's that form the fingerprint of a Big Data project – volume, velocity, and variety?
The library said yes. But the task is daunting.
Volume? It will archive 172 billion tweets in 2013 alone, about 300 each from the world's 500 million-plus tweeters.
Velocity? That means absorbing more than 20 million tweets an hour, 24 hours a day, seven days a week, each stored in a way that can last.
Variety? There are tweets from a woman who may run for president in 2016 – and from Lady Gaga. And they're different in other ways.
"Sure, a tweet is 140 characters," says Jim Gallagher, the library's director of strategic initiatives. "But there are 50 fields. We need to record who wrote it. Where. When."
Because many tweets seem banal, the project has inspired ridicule. When the library posted its announcement of the project, one reader wrote in the comments box: "I'm guessing a good chunk ... came from the Kardashians."