Skip to: Content
Skip to: Site Navigation
Skip to: Search


The Wikipedia way to better intelligence

Open-source information gathering can rival, if not surpass, the clandestine intelligence produced by government agencies.

By Douglas Raymond, Paula Broadwell / January 8, 2007



SAN FRANCISCO; AND CAMBRIDGE, MASS.

The US State Department's effort last month to issue a travel ban on 12 Iranians suspected of supporting that nation's nuclear program wasn't big news at first. Shortly thereafter, it was revealed that the analysis supporting the ban was provided not by the CIA, but by a single junior analyst using Google searches.

Skip to next paragraph

The lesson? Advanced technology and Web-savvy citizenry now make it possible for open-source information gathering to rival, if not surpass, the clandestine intelligence produced by government agencies.

Indeed, open-source methods have already proved their worth in counterterrorism. Shortly after Sept. 11, Valdis Krebs, a security expert, re-created the structure and identities of the core Al Qaeda network using publicly available information accessed from the Internet. He started with two 9/11 hijackers, Nawaf al-Hazmi and Khalid Almihdhar, who were identified from a photograph taken while they attended a meeting with known terrorists in Malaysia in 2000. By scanning public sources for information linking these suspects to others, he re-created the social network identifying all 19 hijackers and described their relationships to their coconspirators, including the identification of Mohammed Atta as the ringleader.

A US-based research center, the Search for International Terrorist Entities (SITE) Institute, monitors the public communications of terrorist and extremist websites and has successfully penetrated password-protected Al Qaeda-linked sites. SITE has successfully accessed terrorists' propaganda, training manuals, and communications, offering insight into their activities that is difficult to obtain elsewhere. According to a Marine colleague who just returned from Iraq, information on the SITE website was used within hours of posting to prevent a terrorist attack in Iraq, demonstrating that third-party analysis has become a key component of intelligence.

A third example comes from a new database at the Jebsen Center for Counter- Terrorism Studies at Tufts University's Fletcher School in Medford, Mass. Researchers there have collected historical data on the life paths of hundreds of terrorists and analyzed their letters, wills, and interviews. This information, based on open-source data, is being used to identify the factors that tend to predict terrorist acts.

Technology that lets anyone analyze data

While motivated citizens and academics have often been able to generate analysis that rivals that of government experts, the difference today is that technology such as wikis and blogs allows thousands to contribute to an analysis. Readers can then "vote" the most accurate and relevant information to the top, giving them enough credibility to be taken seriously. Take, for example, the Wikipedia entry of Moqtada al-Sadr. Mr. Sadr's entry in this free encylopedia that anyone can edit has been modified approximately 500 times by about 50 people in the past three years. These motivated authors have expanded the entry and corrected hundreds of one another's errors and omissions. Thousands read the profile and hundreds of others have linked to it, making it the first entry in most search engines' results.

Blogs are another tool for massive parallel analysis and collaboration – a search for blogs dealing with terrorism generates nearly 1 million results.

While most bloggers generate little of value to intelligence analysis, the collaborative nature of the technology gives greater weight to the better analyses, pushing them to the top. Additionally, the increasing reliance of terrorist groups on the Internet provides these amateur intelligence specialists with tomes of data that will make it easier to understand terrorist goals and objectives, improving their ability to conduct pattern analysis. The result is that analysts have increasingly better access to data, and the consumers of their work have better tools for distinguishing great analyses from those that are merely good.

A disconcerting fact about the Iranian travel-ban event is that the State Department had repeatedly requested that list of names from the CIA, but was refused for reasons of secrecy.

How US intelligence can adapt

To be fair, the US intelligence community has taken some first steps in adopting collaborative technology by creating an "Intellipedia" – a secret, internal version of Wikipedia. However, the strength of Wikipedia is not the technology, but the massively collaborative effort that the technology enables. US intelligence agencies must adopt this collaborative spirit and become more adept at incorporating the increasingly valuable analysis produced in the public domain with their internal efforts. America will be a more secure country once it discards the notion that secrecy is equal to strength and begins harnessing the power of 100,000 bloggers.

Douglas Raymond is a former US Army captain, former member of the 66th Military Intelligence Group, and currently a term member of the Council on Foreign Relations. Paula Broadwell is a PhD student in counterterrorism policy studies at Harvard University and the deputy director of the Jebsen Center for Counter-Terrorism Studies at Tufts University's Fletcher School.

Permissions