How a Texas city coped after debilitating ransomware hack

Cities in Texas were hit by Russian ransomware attacks before the United States fully acknowledged the cybersecurity threat. Here’s how they mended the damage without paying millions of dollars to overseas criminals. 

Chuck Burton/AP
Amanda Crawford (right) and Nancy Rainosek, pose for a photo inside the Information Resources Data Center in Austin, Texas, July 19, 2021. After cyberattacks struck Texan towns, many state responsibilities that were previously digitized had to be manually operated.

It was a steamy Friday two Augusts ago when Jason Whisler settled in for a working breakfast at the Coffee Ranch restaurant in the Texas Panhandle city of Borger. The most pressing agenda item for city officials like him that morning: planning for a country concert and anniversary event.

Then Mr. Whisler’s phone rang. Borger’s computer system had been hacked.

Workers were frozen out of files. Printers spewed out demands for money. Over the next several days, residents couldn’t pay water bills, the government couldn’t print checks, police officers couldn’t retrieve certain records. Across Texas, similar scenes played out in nearly two dozen communities hit by a cyberattack officials linked to a Russia-based criminal syndicate.

In 2019, ransomware had yet to emerge as one of the top challenges confronting the United States. But the attacks in Texas were a harbinger of the now-exploding threat and offer a case study in what happens behind the scenes when victims come under attack.

Texas communities struggled for days with disruptions to government services as workers in small cities and towns endured cascading frustrations brought on by the cyberattack, according to thousands of pages of documents reviewed by The Associated Press and interviews with people involved in the response. The AP also learned new details about the attack’s scope and victims, including an Air Force base where access to a law enforcement database was affected and a city forced to operate its water-supply system manually.

Recent ransomware attacks have led to gasoline shortages and threatened meat supplies. But the Texas attacks – which, unlike recent prominent cases, were resolved without a ransom payment – make clear ransomware need not hit vital infrastructure nor major corporations to interrupt daily life.

“It was just a scary feeling,” said Mr. Whisler, Borger’s emergency management coordinator.

Early on Aug. 16, as most Texans were still asleep, hackers half a world away were burrowing into networks.

As the attack’s impact became apparent, the city manager of Vernon emailed colleagues that the city could get back online by paying a $2.5 million ransom but that was “obviously” not the plan.

“Holy moly!!!!!” came the reply.

The culprits were affiliated with REvil, the Russia-linked syndicate that last spring extorted $11 million from meat-processor JBS and more recently was behind a Fourth of July weekend attack that crippled businesses around the globe.

The August 2019 hackers gained their foothold through an attack on TSM Consulting Services, a Texas firm that provides technology services to local governments. The attackers branched through screen-sharing software and remote administration to seize control of the networks of some of the company’s clients.

Within hours, state and federal officials were hunkered inside an underground operations center normally used for calamities like hurricanes and floods. Gov. Greg Abbott declared a cyber disaster. Texas National Guard cyber specialists were activated.

“Basically, if there’s a municipal function that you would go down to a city hall for, or that you would rely on the police department for, it wasn’t available,” said Andy Bennett, the state’s then-deputy chief information security officer.

In Borger, a city of fewer than 13,000, ransomware demands spat out of printers and flashed on some computer screens. Government files were encrypted, their titles replaced by gibberish combinations of letters and symbols, said city manager Garrett Spradling.

Vital records, like birth and death certificates, were offline. Signs posted on a drive-up window outside City Hall said the city couldn’t process water bill payments but that cutoffs would be delayed.

Because the city had paid for remote offsite backup, Borger could reformat servers, reinstall the operating system and retrieve data. The police department, however, retained its data locally and officers were unable to access previous incident reports, Mr. Spradling said.

Jeremy Sereno was working his civilian job at Dell when he was enlisted by the state to help. A lieutenant colonel and senior cybersecurity officer with the Texas Military Department, Mr. Sereno helped deploy Texas National Guard troops to hacked cities, where specialists worked to assess the damage, restore data from backed-up files, and retake control of locked systems.

One of the first areas of concern was a small North Texas city. The attack locked the “human-machine interface” workers used to control the water supply, forcing them to operate the system manually, Mr. Sereno said. Water purity was not endangered.

“That’s what’s considered critical infrastructure, when you talk about water,” he said.

AP is not identifying the city at the urging of state officials, who said doing so could draw new attacks on its water system.

In Graham, the ransomware attacked a police server housing body-camera videos, causing hundreds to be lost. Instead of using mobile data terminals to run checks on people they encountered, officers had to rely on requests to dispatchers at a local sheriff’s office unaffected by the attack, said Chief Brent Bullock.

The impact wasn’t limited to local governments. Sheppard Air Force Base confirmed to AP that its access to a statewide law enforcement database used for background checks was temporarily disrupted.

One complication: TSM’s client list was encrypted, officials said. State officials didn’t immediately know which communities had been victimized.

They had to call around, said Nancy Rainosek, Texas’ chief information security officer. “There was one place that we contacted and they said, ‘no, no, we’re not hit,’” she said. Days later, “they said, ‘yes, we were.’”

Fortunately for Borger, most city services were restored within days. The city has since invested in additional cybersecurity protections.

“When you complain about having to change your passwords, you complain a lot more when it’s never happened to you and you don’t have anything to relate it to,” Mr. Spradling said. “You tend to complain a little less after you’ve had to answer the phone and tell 300 people they couldn’t pay their water bill.”

Even now, Mr. Spradling said, officials will go to pull an old report or address record – only to find it isn’t there.

This story was reported by The Associated Press. 

You've read  of  free articles. Subscribe to continue.

Dear Reader,

About a year ago, I happened upon this statement about the Monitor in the Harvard Business Review – under the charming heading of “do things that don’t interest you”:

“Many things that end up” being meaningful, writes social scientist Joseph Grenny, “have come from conference workshops, articles, or online videos that began as a chore and ended with an insight. My work in Kenya, for example, was heavily influenced by a Christian Science Monitor article I had forced myself to read 10 years earlier. Sometimes, we call things ‘boring’ simply because they lie outside the box we are currently in.”

If you were to come up with a punchline to a joke about the Monitor, that would probably be it. We’re seen as being global, fair, insightful, and perhaps a bit too earnest. We’re the bran muffin of journalism.

But you know what? We change lives. And I’m going to argue that we change lives precisely because we force open that too-small box that most human beings think they live in.

The Monitor is a peculiar little publication that’s hard for the world to figure out. We’re run by a church, but we’re not only for church members and we’re not about converting people. We’re known as being fair even as the world becomes as polarized as at any time since the newspaper’s founding in 1908.

We have a mission beyond circulation, we want to bridge divides. We’re about kicking down the door of thought everywhere and saying, “You are bigger and more capable than you realize. And we can prove it.”

If you’re looking for bran muffin journalism, you can subscribe to the Monitor for $15. You’ll get the Monitor Weekly magazine, the Monitor Daily email, and unlimited access to CSMonitor.com.