A city locked out of its own data network

Officials in San Francisco charge that one of their own employees gave himself exclusive access to key switches on the city’s computer network.

Rich Pedroncelli/AP/file
Password, please: San Francisco Mayor Gavin Newsom told the San Francisco Chronicle that "There is nothing to be alarmed about, save the inability to get into and tweak the system. Nothing dramatic has changed in terms of our ability to govern the city."

[Update: Terry Childs hands over the codes. Full story at the Horizons blog.]

It sounds like a plot from Hollywood: A team of techies is busily trying to crack passwords to get access to parts of San Francisco’s computer network. They are doing so at the direction of city officials, who have discovered that they are locked out of parts of their new multimillion-dollar system.

But for the City by the Bay, it’s a story line they didn’t see coming.

Local officials charge that one of their own employees, a network administrator named Terry Childs, gave himself exclusive access to key switches on the network. After they discovered the problem, Mr. Childs was interrogated by the police, but unlike the disgruntled programmers in the movie “Office Space,” he apparently hasn’t been fazed by the threat of prison. Authorities say he first gave police bogus passwords and now sits in jail refusing to divulge his abracadabras.

Childs pleaded not guilty last Thursday to four felony counts of computer network tampering. His lawyer declared it all a big misunderstanding and called the $5 million bail inappropriate. But San Francisco officials aren’t sure what Childs has done behind password locks, and they worry he might have created back channels into city data.

So-called “malicious insiders” are surprisingly common, and they tend to be more harmful – and difficult to thwart – than outside hackers, say experts. Despite the threat, one recent study found that organizations are growing more lax in guarding against them.

“Most of the security solutions [deployed] are outward facing, focusing on the moat and the turrets, not determining if the threat can come from inside” the castle walls, says Tom Kellermann, a computer security expert formerly with the World Bank Treasury and now with Core Security Technologies in Boston.

Roughly a quarter of computer system attacks are inside jobs, according to the past two years of the E-Crime Watch Survey from CSO Magazine and the US Secret Service. Their most recent report in 2007 found steep drops over the previous year in the percentage of organizations taking common protective measures:

•Background checks on employees and contractors dropped from 73 to 57 percent.
•Employee monitoring went from 59 to 42 percent.
•Employee security training plummeted from 68 to 38 percent.

The report defines an insider as a current or former employee, services provider, or contractor. Outside technology vendors and partners who are given insider access constitute a fast-growing source of attacks, according to a new four-year study conducted by Verizon.

Ironically, San Francisco began building its network three years ago out of a desire to be less reliant on outside systems, says Ron Vinson, chief administrative officer for the city’s Department of Telecommunications and Information Services. Childs was a key developer on the project.

The network, called FiberWAN, currently encompasses 60 percent of the city’s internal and external business sprawling over 60 departments.

The lockout hasn’t disrupted city services, yet: Officials can still send e-mails across departments, and residents can still pay taxes and parking tickets online. But it has created no-go areas on the system where officials aren’t sure if sensitive data – such as e-mails and payroll records – have been compromised.

“We had control of the house,” Mr. Vinson says by way of analogy, “but there were certain rooms inside the house where we didn’t know what was going on and did not have access.” His team is trying to identify and access all the locked “rooms.”

The exclusive privileges that officials say Childs gave himself were discovered, Vinson says, after the city hired a security chief and she began upgrading security protocols. Prosecutors have said Childs locked out other administrators after a confrontation with the security head.

Vinson estimates the costs of the restoration work will be in the hundreds of thousands of dollars.
Nearly half of computer security breaches take weeks to mitigate, according to the Verizon report, with 14 percent taking months. Detection times are worse, with 63 percent of attacks going unnoticed for months. In 70 percent of cases, it’s a third party who notices first.

There’s no simple way to profile malicious insiders, says Mark Maybury, executive director of the IT division at the MITRE Corp., a nonprofit research-and-development group outside Boston. He has researched hundreds of insider cases with the aim of developing computerized sensors to detect them.

“Just as insiders are highly heterogeneous in their demographics, so too are they highly heterogeneous in their behaviors. Therefore, you can’t detect all insiders with one sensor,” Dr. Maybury says.

At this point, however, not much sensor software is commercially available, he says. Still, basic security protocols and simple attentiveness are crucial preventative measures, say experts.

Vinson’s department does conduct backups, he says. And there are fail-safe systems and disaster recovery plans – but they were designed with natural disasters in mind. “If an earthquake happened, we all have instructions about what to do. But we don’t have instructions for what to do when it’s one of your own employees,” says Vinson.

[Update: Terry Childs hands over the codes. Full story at the Horizons blog.]

of stories this month > Get unlimited stories
You've read  of 5 free articles. Subscribe to continue.

Only $1 for your first month.

Get unlimited Monitor journalism.