Fixing Spirit, from 125 million miles away

For those working to unravel Spirit's babblings, it meant hours in the "sandbox" - the test bed where engineers work with a rover replica, pushing and prodding it to see what happens.

And for the most part, the most telling clues come not from any movement or action obvious to the untrained eye. Instead, they come in the lines of code that pop up on a computer screen every time the rover follows a command.

In a programming language that distills actions down to a jumble of words, brackets, and keyboard symbols, the rovers reveal their secrets, showing what parts of the rover were used to complete which commands.

With little more to go on than random bursts of data, Spirit's engineers started their search wide. Eventually, they narrowed the primary culprit down to the rover's flash memory, which stores data while the rover is sleeping.

The rover, engineers discovered, sensed some flaw in its flash memory, and when it couldn't resolve it, it shut down and tried to restart. In three days, it tried - unsuccessfully - to reboot itself more than 60 times.

Yet even in this apparent frustration is the kernel of what has made many of America's robotic missions so resilient. Spirit's ability to keep itself alive in a "safe mode" after sensing a fatal flaw gave engineers the time needed to diagnose the problem and devise a solution. "Russian probes never had this ability," says Dr. Oberg. "They broke and they died."

Things, after all, inevitably go wrong. On Voyager 2, for instance, a part of the camera assembly broke, meaning the camera could not pan correctly. Engineers trained it to move more modestly.

On Galileo, the large antenna designed to transmit scientific data never unfurled, so scientists reprogrammed an onboard tape recorder to store images until they could be sent on a smaller antenna. When the tape jammed, they taught the rewind mechanism to stop before it reached the sticky portion of the tape.

With Spirit, engineers have jury-rigged a "cripple mode" in which Spirit can reboot without using the flash memory. While Spirit can't perform all its functions in cripple mode, it can respond to commands and go to sleep when ordered, giving engineers more time and help in isolating Spirit's problem.

It's a process that could take weeks as they probe all the relevant lines of code to pinpoint exactly where and how the problem started - without triggering it again. If it's a flaw in the software - as is expected - engineers can eventually send up a new set of codes to replace the corrupt ones. If it's a problem caused by something physical in the machine, engineers can program the rover to avoid those sections.

Either way, the process is one that engineers have come to know well. And though Dr. Townsend knows her job is to deliver a perfect rover, there's something of a Martian car mechanic in her nature, and she can't help but relish the chance to tinker under the hood.

"A perfect mission is great, but we engineers love jumping on stuff like this," she says. "It's exciting to figure it out and make the craft work again."

Page: 1 | 2