 |
 |
||
 |
|
||
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
||||||||
 |
The Planetary Society WeblogGuest Blogger: Mark AdlerSeptember 18 - 24, 2006
|
||||||||||||||||
 |
As Chief Mission Concept Architect at the Jet Propulsion Laboratory, Mark Adler is currently responsible for the conception, design, and cost estimation for proposed new space missions across the lab. In past lives at JPL, Mark was the Mars Exploration Rover Spirit Mission Manager from launch through the first extended mission, and he was the Lead Mission Engineer on Cassini-Huygens. Though his work seems like a hobby to Mark, he also enjoys flying small planes, amateur theatre, and writing about himself in third person. |
Musings on the Moon
In January 2004, the president announced a vision for space exploration. It had been in work for some time. Once the first Mars Exploration Rover had landed successfully, and seemed to be working ok, they came out with it.
While I was mission manager on Spirit, I remember being told that if there was a major problem, to run it up the chain fast since the President was going to make an announcement soon. Fortunately for the vision, the major problem on Spirit didn't occur until a week after the announcement. More on that in another blog.
 A bird's eye view of Spirit This bird's-eye view combines a self-portrait of Spirit's deck and a panoramic mosaic of the Martian surface as viewed by the rover. The rover's solar panels are still gleaming in the sunlight, having acquired only a thin veneer of dust two years after the rover landed and commenced exploring the red planet. Spirit captured this 360-degree panorama on the summit of Husband Hill inside Mars' Gusev Crater. During the period from Spirit's Martian days, or sols, 583 to 586 (Aug. 24 to 27, 2005), the rover's panoramic camera acquired the hundreds of individual frames for this largest panorama ever photographed by Spirit. This image is an approximately true-color rendering presented with geometric seam correction. Color: True color. Created: 30 January 2006. Credit: NASA / JPL-Caltech / Cornell |
The Big Thing in the vision was the plan to send people to the
Moon in preparation for sending people to Mars.
Naturally, the first question to come up was: why the Moon? At first glance,
it seems like a good idea -- sort of a stepping stone to Mars. However it
would be extremely expensive to go to the Moon and would put off going to
Mars for decades. Couldn't we just go straight to Mars with the people? Proponents
for the Moon would say that we will practice on the Moon what we're going
to do on Mars. The go-to-Mars-now proponents would say, hogwash, everything
has to change for Mars since the environment and nature of the trip are completely
different -- if the objective is to go to Mars, then just go to Mars. We
have the technology today, why waste money and time on the Moon?
Of course, both sides' arguments are a bit of a stretch. Going to the Moon will of course develop some things that will be needed to go to Mars, to work there, and to get back. At least it will if you're thinking a little about the long-term objective while you're making design decisions. The best example is the heavy-lift, Saturn V class launch vehicle that will be developed, now called the Ares V. Another example is the Earth entry and landing vehicle for the humans, previously called the Crew Exploration Vehicle, now called Orion.
The Orion would require only some improvements to the heat shield to handle a landing from a Mars return trajectory as compared to a Lunar return. A significant benefit would be that we would (re)gain the operational experience of having people working on the surface of another body. And of course many of the subsystems, such as environmental control and life support, would carry over. But are all those really worth the cost and time of a Lunar campaign? You could just develop the launch vehicle and entry vehicle for Mars in the first place -- there is no magic there. Since we'll have to spend several months on Mars anyway for reasonable mission designs, then we can do our learning there for operations. We know enough about the Martian environment now to design robust equipment. Furthermore, a lot of the stuff you'd develop at great expense for the Moon would not be useful for Mars at all, such as the Lunar lander and return vehicle.
There have been some silly arguments as well for the Moon, which I should mention in passing. One is that a Moon base provides a departure point for a Mars mission. Well, that's like deciding to jump into a giant two-mile deep pit on your long hike to a distant destination, and having to climb out again, instead of just walking around the pit. The deep gravity well of the Moon negates any benefit of stopping there. Another is that we can develop the ability to convert Lunar material into fuel, oxygen, or water, and that that experience will apply to Mars. Of course, the resource processing on Mars would be completely different, and would require new designs with almost none of the risk retired by the Lunar experience. The resource processing experience would be more applicable if we found water ice on the Moon and decided to land near water ice on Mars. Though in that case, chances are we could find some water ice on Earth to test the equipment with.
Back to the more compelling arguments. Having worked on a surface mission, I'd have to say that there's a lot to be said for the operational experience that would be gained at the Moon. Much of that learning would give us a huge leg up for a Mars mission. But even there, you'd have to ask: what is the incremental cost of a second human mission to Mars compared to the entire Lunar campaign?
If we had to go Mars the first time to get the operational experience in order to make the second trip more productive, that wouldn't be so bad. I'm sure it would be cheaper than the extra expense of going to the Moon. And you'd get another Martian landing site out of the deal.
My diatribe so far sounds like I think you shouldn't go to the Moon first. Well, in fact I think not only that you should, but that it is necessary. Here's why.
Let's say that you've asked NASA how much it will cost to send people to Mars. They give you an answer. Would you believe it? Heck no!
Any sensible manager would look at the recent history of NASA's large programs, such as the shuttle and space station. The conclusion is that there's no reason to believe that the massive increases over the originally estimated costs would be any different this time. Furthermore, there are too few examples and too many differences to derive a simple multiplier from those experiences to apply to the Mars cost estimate. Any arguments about external influences, changes, international cooperation, etc. on those programs would be dismissed by the sensible manager. All of that will apply to the Mars endeavor as well. So there's no way to know what it will cost.
Not a good way to start a project.
Now you ask NASA if they have the management ability to pull together what it takes to succeed in something several times bigger than Apollo. Can they mobilize industry, establish the right partnerships, establish and enforce the necessary and sufficient lines of communication in a complex technical project, and resolve a dozen other organizational issues to meet schedule and cost and have safe and successful missions? If NASA said yes, we can send people to Mars now, just give us the money, would you, the sensible manager, believe them? Heck no!
So what do you do with someone whose performance you are unsure of? Well, you give them a smaller job and see how they do on that before you risk far more on the larger job. You give them a job roughly the same magnitude but a little bigger than jobs they've had before, and with some similarity to the job you eventually want them to do. Then they will both learn and demonstrate the managerial skills you want to see. If the job is similar enough, they will begin to build the infrastructure that will be needed for the bigger job. Especially if you tell them what the bigger job is and that they'll get that bigger job if they do good on this one. Furthermore, having done the smaller job, they'll have a far, far better idea what the bigger job will cost.
 Apollo 16 View of the lunar module and lunar rover. Credit: NASA |
That's is exactly the position that the US administration and congress are in,
if they want to send people to Mars. Our government, in stunning contradiction
to many other things they do, have done exactly the right thing by having NASA
send people to the Moon first. If we could run the experiment in two parallel
universes, I would predict that the skip-the-Moon universe would actually run
into far more delays and it would end up being more expensive to get to Mars
(if we got to Mars ever), than in the universe in which we go to the Moon first
and then Mars. Let's say, for example, that the Mars program is estimated to
be four times the cost of the Moon program. Then I might imagine I could save
20% by not going to the Moon. However not having that experience of going to
the Moon first could easily increase my cost overruns by much, much more than
the 20% I thought I was going to save.
Ok. So we should send people to the Moon first if we want to send people to Mars. That leaves an open question. Should we send people to Mars? Or more precisely, at what price is it worth sending people to Mars? I'll leave that as an exercise for the reader. :)
Oh No, Not The Pluto Thing AGAIN!
Since most of the other guest bloggers have opined about the whole whether-Pluto-is-a-planet thing, I figure I have to as well. Don't worry. I'll keep it short.
My position on the subject appears to have been grossly underrepresented in all the goings on, since I have not seen it discussed anywhere. (I'm sure readers will point me to where I've missed it.) My position was first stated, I think, by Isaac Asimov who said: "Our solar system consists of a star, four planets, plus debris." The four planets are, of course, Jupiter, Saturn, Uranus, and Neptune. That in my mind is the only definition of planet that could stand up to scientific scrutiny. The vast gulf in size between those four and the remaining debris in any sort of histogram of solar system objects makes the division startlingly obvious.
So any definition that does not have four planets, and that does not define Earth as a "dwarf planet", a "minor planet", "debris", or some such word would not get my vote. Not that anyone has asked me for my vote. In fact now you can probably see why nobody has asked me for my vote.
Two Tantalizing Glimpses
It's kind of nice to be operating about a dozen spacecraft around the solar system. Sometimes you find stuff.
We just got our first glimpse into Victoria Crater on Mars as the Opportunity rover closes in. I pulled the high-resolution mosaic off of the public JPL website and cropped out the left and right half of the far rim of Victoria as seen from Opportunity's current position. I also adjusted the contrast to bring out the features. I do that sort of thing often, and it's something that anyone out there can do if you have some simple image processing software such as Photoshop or the free GIMP software.
Each of the protrusions on the far rim are being called "capes" and some of the indents between them "bays". Some are getting names like Cape Verde and Duck Bay, and most are getting numbers, Cape4, Bay2.
It is critical for us to name them in order to be able to discuss the crater and what we plan to do there. The plan is to go in the crater. Though I'm not seeing an obvious on ramp here yet -- it looks kind of steep to me. Of course, we can't see much of the crater rim yet, so I'm not worried. We always find a way.
 Approaching Victoria Crater (looking left) Credit: NASA / JPL-Caltech / Mark Adler |
 Approaching Victoria Crater (looking right) Credit: NASA / JPL-Caltech / Mark Adler |
A bit farther away, we have found a new,
faint ring of Saturn near the orbits of the satellites Janus and Epimetheus.
Cassini was looking in the direction of the Sun through the rings, which
really brings out the features. In the image below, again I messed with the
image to bring out the ring. You should try this at home! It's fun.
The + marks the new ring. As far as I know, the new ring doesn't have a
name or a letter yet, but I'm sure some IAU committee somewhere is chomping
at the bit to name it. ("Is this a real ring, or is it just a dwarf ring?" I
can hear them asking ...)
It's common for satellites to lose material to space due to impacts, and
for the satellites to then "shepherd" the material into a coherent ring.
Still, this one was surprising. By the way, I call it a new ring, but really
it's just new to us. I'm sure it's been there for quite some time.
 A New Ring On September 15, 2006, Cassini revealed a new diffuse ring at Saturn, coincident with the orbits of Janus and Epimetheus. This observation occurred during a solar occultation, where the Sun passed directly behind Saturn and left Cassini in Saturn's shadow with brilliantly backlit rings. Usually, an occultation lasts only about an hour, but this time it was an unusual 12-hour marathon. The view looks down from about 15 degrees above the un-illuminated side of the rings. The image was taken in visible light with the Cassini spacecraft wide-angle camera on September 15, 2006, at a distance of approximately 2.2 million kilometers (1.3 million miles) from Saturn and at a Sun-Saturn-spacecraft angle of almost 179 degrees. Image scale is 130 kilometers (81 miles) per pixel. JPL's Chief Mission Concept Architect Mark Adler processed this image to enhance the details of the ring. Credit: NASA / JPL / Space Science Institute / Mark Adler |
Fear, the Great Motivator
I suspect that if you're reading this blog, you are among the folk that appreciate space exploration and might like to see it get more funding. I sometimes think about how to get that to happen. One idea that comes to mind is fear. Yes, fear. That's always worked well to provide some serious motivation. So how can we use fear in this context? It's not too hard.
The most obvious is the giant rock crashing into the Earth. It's often said that the reason the dinosaurs became extinct is that they didn't have a space program. Now while the probability of a giant rock hitting the Earth that's capable of global-scale consequences is vanishingly small in any given lifetime, you only need one to ruin your day. This is the sort of thing insurance is for. Something that will probably never happen, but if it does you sure would like to be covered.
In this case, the insurance is pretty darned cheap compared to the expense
of what could happen. Though it seems we haven't yet committed to paying the
premiums. We should be doing much more to find all the threats and send spacecraft
out there to characterize them. Then we can start thinking about what we'd
do if one was headed our way. The key here is to get a big head start. With
a warning of a few decades, which is not unreasonable, then I'm certain we'd
figure out a way to get the giant rock to miss.
Earth can also be hit by enormous solar flares. While no one on Earth will be hurt by them, they can sure mess up our electrical and communications systems. The fear here is that you'll lose power, or, even worse, our cell phones will stop working! Clearly a fate worse than death. Another concern to many businesses would be the loss of communications satellites and other Earth orbiting assets, either temporary or permanently.
 Double Solar Flare Credit: NASA |
Now we can't do anything to stop the solar flares, but a warning sure would
help. You can sort of batten down the hatches on the communication satellites,
and prepare your power and communications customers for the outages, and even
provide alternate paths to mitigate the impact. Again the key is having predictions
and warnings of these things. To really understand the Sun well enough to do
these sorts of weather predictions, we'd have to do what we do at Earth, which
would be to have a bunch of weather satellites around the Sun. They're looking
at rather different sorts of weather, but the idea is the same. It seems like
a good investment to me to place these sorts of solar activity sentinels out
there to give us that heads up. Another avenue of fear might actually make
this happen sooner rather than later. While solar flares don't hurt us on the
Earth, and even somewhat mitigated in low Earth orbit, you can really get zinged
in deep space. For example on your way to the Moon or Mars.
While the MER rovers were on their way to Mars, we got hit by the biggest solar flare ever recorded up to that time. Fortunately both spacecraft were designed for the radiation environment, so we only had to live without working star scanners for a few days while we weathered the storm, and reset some equipment after it was over.
However if humans had been on that trip, they would have gotten a very, very bad dose of radiation.
Again you can't stop the flares, but if you have sentinels to give you a few hours warning, you can imagine a storm shelter in the Moon or Mars ship where they are surrounded by their food and water to partially shield them from the deadly flare.
Another one that sure looks scary are gamma ray bursts. These have, fortunately, always been observed very far away and going off a very long time ago. Billions of years ago and billions of light years away. But if one went off nearby, and it was pointed even a little bit in our direction, we would be toasted, literally. The whole planet would be completely sterilized, ready to be packaged in a food container. This is a case where could do absolutely nothing, even with a warning.
 Hypernova |
But it would be nice to know more about these things. The theories indicate that
they can only happen relatively early in the evolution of the universe. More
observation to pin down that theory might at least ease our worries.
If you watch the movies, clearly the major threat from space are aliens. Nasty aliens with ray guns, force fields, and sharp teeth. Now here's something we can really grab onto for the fear factor.
Fortunately, this is right along the lines of many of our science objectives
-- understanding our origins by looking for evidence of extraterrestrial life.
One of my favorite "marketing" slides is an image from the movie Independence Day with the city-sized alien spaceship firing its giant ray gun into the Empire State Building. Over the image it says: "JPL: Finding Life Before It Finds Us".
Mars Sample Return
In this blog entry and the next, I will summarize my horribly failed attempts at bringing rocks back from Mars. The idea is to give you, our faithful blog readers, a little insight into the perils and pitfalls of trying to bring these grand and glorious space missions to reality, from a personal point of view. This story is the depressing downside of what goes on. If you'd like a story with a happier ending, though still with lots of perils and pitfalls, I recommend Steve Squyres' Roving Mars book, which details the story of the Mars Exploration Rover project. Now back to the Mars Sample Return story (so far).
At some point after I had gotten through several somewhat unrelated and largely unplanned college degrees, I actually starting thinking about what it was I wanted to do with my life. (I hear that most people try to figure that out before getting the degrees.) I wondered then, and still wonder now, how did we get here? By "we" I mean creatures who are capable of asking questions like "how did we get here?" It all seems so magnificently bizarre. I didn't think I could answer the question of course, but maybe I could contribute to some small part of the answer. It is a remarkable luxury that we can spend time on such questions, as opposed to, for example, having to hunt and kill dinner.
The real mystery in my mind is not the evolution of intelligence from single-celled organisms, though there's obviously a lot there to learn, but rather how the single-celled organisms came to be in the first place. None of my degrees was in biology or even chemistry, so I took a different tack on how I might contribute. I figured, as many others have, that if we could find another example of single-celled life distinct from Earth's, then we'd be getting somewhere.
It's hard to figure out anything from just one example. So where to look? Clearly not on Earth, since we seem to have a single biology that has infested the entire planet from stem to stern. There really is only one example of life here. That leaves some place other than Earth. Somewhere out there. So I decided to apply my degrees in physics, electrical engineering, and mathematics to exploring space.
The center of gravity of space exploration was at JPL, so that's where I went.
In early 1996 Donna Shirley (currently on the Planetary Society Advisory Council) asked me to work for her on the newly-formed "Mars Exploration Program" at JPL, which she was managing. I was to be the "Mars Exploration Program Architect". Among other things, I was to figure out how to bring selected samples of Mars rocks to Earth, and do it affordably. The idea is that if you pick up and deliver the right rocks, then by bringing to bear the full arsenal of Earth's scientific instruments on those rocks, you can find evidence of an extinct single-celled biology on Mars and learn something about it.
If it was ever there, that is. In fact, that is pretty much the only way to get a definitive answer to such a question in a reasonable amount of time. If you try to do it on Mars and the instrument says yes, there was life, then someone will come up with a challenge to that finding that will require sending another instrument to Mars to respond to that challenge. At best, six and a half years later (three Mars launch opportunities). Then the same thing happens again with that instrument's results. This could go on for some time. If you have the rocks here on Earth, then that challenge/response cycle time is much faster, on the order of months, and you can apply much more sophisticated instruments and sample processing. Including instruments we haven't invented yet. As Carl Sagan, one of the founders of the Planetary Society, once said; extraordinary claims require extraordinary evidence. In fact he said that a few months after I joined the Mars Program.
In August of 1996, the rock hit. A paper was published about the now famous ALH84001, a meteorite from Mars found in Antarctica. The claim was that several different lines of evidence all resulted in the conclusion that the rock contained evidence of ancient biological activity on Mars. Fortunately I had a few months head start working on Mars Sample Return, since now the interest in that mission suddenly got intense. In the meantime, using the process described above, many laboratories did their own analyses of pieces of ALH84001 to substantiate, or not, the claims. Even though the subsequent investigations showed that the necessary "extraordinary evidence" was unfortunately not there, the interest in Mars Sample Return did not subside. In fact it increased.
"Hey, wait a minute" you're thinking. "If you already have pieces of Mars here, why do you need a Mars Sample Return mission?" First off, they're not the right rocks. We'll need to go to one, or more likely several, very carefully selected spots on Mars to have even a fighting chance of finding the evidence we're looking for. We don't get to pick the rocks that come to us by accident. Those can come from anywhere. Furthermore those rocks get contaminated on Earth before we find them and pick them up. We should of course continue to do the really cheap Mars Sample Return missions to Antarctica, but unless we're really lucky, that's not going to give us the gold.
In 1997 we had several stunning successes at Mars, the Mars Pathfinder lander and Sojourner rover, and the Mars Global Surveyor orbiter. That really bolstered everyone's confidence that we could do even more ambitious things at Mars and succeed. Like Mars Sample Return.
So we had the motive and the means. We just needed the opportunity.
We developed a way to scale up the Mars Polar Lander design to land a much larger payload. We came up with a rover and a rocket to put on the lander. The rover to get the rocks, and the rocket to put them into Mars orbit fit in a grapefruit-sized container. And we designed a spacecraft that could go into orbit around Mars, find and capture the rock container, and bring it back to Earth. How hard could it be?
Even in those illusionary faster-better-cheaper days, it was still a very expensive mission. Fortunately the French space agency, CNES, decided to partner with us on the mission, providing a large Ariane V launch vehicle, the return orbiter, and a really talented team. NASA bought into the joint mission, and so in 1998 we were off and running with the first real Mars Sample Return Project, planning to launch in 2003 and 2005. I was the Mission and Systems Manager and Chief Engineer on the project. Below is a piece of artwork depicting that mission that I had commissioned for the project. (It was my idea to hide the rover behind a rock for the launch. :-) )
 Mars Sample Return Credit: NASA |
You probably know the next part of the story. In 1999 we lost both the Mars Climate
Orbiter and the Mars Polar Lander during their critical events at arrival. All
that blustery confidence that we had built up from Pathfinder and Mars Global
Surveyor vanished in an instant. We no longer had any business thinking that
we could pull off something as complicated and risky as Mars Sample Return. The
project was cancelled in 2000, a few months after the Mars Polar Lander failure.
The artwork and a lot of designs and documents was as far as we got.
At that point, Job #1 was to regain our confidence. To get back in the saddle. I pulled together a small team to propose what became the Mars Exploration Rover project with exactly that objective. Fast forward four years (see Steve's book for the details), MER has succeeded beyond our wildest expectations, and our confidence has been restored. In my next entry, I'll cover my second swing at Mars Sample Return starting in 2004.
Spirit Sol 18 Anomaly
In a previous blog entry I said I'd talk about what happened on Spirit about a week after the President announced the Vision for Space Exploration. So here it is: a little window into what it's like to operate a priceless national asset.
It was January 21st, 2004, 18 Martian days (sols) since Spirit landed successfully on Mars and about a week since Spirit successfully drove off of the lander onto the surface of Mars. Everything was going so incredibly well, we couldn't believe it. It's really odd to have the rover work so much better on Mars than it ever did in test. Spirit was engaged in real geological exploration on the surface of an alien world. We figured we were the luckiest people on Earth.
Well, our luck was about to change.
Jennifer Trosper and myself took turns as the tactical mission manager for Spirit. Jennifer was on duty for Sol 18, and it was an off day for me. I came to JPL anyway around noon for an interview for a documentary. As I was coming out of the interview, I ran into Steve Squyres, the Principal Investigator for MER. He was going in for his interview appointment. As he sees me coming out, Steve says "Have you heard about Spirit?"
Just from the question and Steve's serious tone of voice, I instantly awakened from my partially sleep-deprived state and my eyes flew wide open. "What do you mean" I asked, ""Have I heard about Spirit?'" Steve said that we did not get any signal from Spirit when we expected to, either from a high-gain antenna communication session direct to Earth, or from the subsequent relay communication session to the Odyssey orbiter.
Oh boy.
If only the high-gain session were missing in action, that could easily be due to weather, Deep Space Network problems, lots of things. It turns out that communicating over a few hundred million miles isn't easy, and you often have problems. But our experience up to then was that the relay sessions, only a few hundred miles to the orbiter, always worked. The Odyssey orbiter itself was working fine. For the relay to also fail right after the high-gain failure gave a very strong indication that the problem was in the rover, and that it was serious.
Space missions are risky. We're used to that. We especially think a lot about the most risky parts. For MER, the biggest risks were, in order, entry, descent, and landing, or the six minutes of terror as we would call it, with the next most risky being the launch from Earth, and finally the post-landing events which was driving off the lander and the rover unfolding and cable cutting that preceded that.
Launch, landing, and egress. All of them nail-biters.
We had made it through that with Spirit. All the risky stuff was behind us, we figured. From here on, with proper care and attention of course, there were no more big risks. It should all be gravy and smooth sailing. Which made this all the more alarming. What the heck happened?
I immediately headed over to the operations area, in which I and many others spent the next three days almost continuously. There were no off days at that point. Jennifer was the planning mission manager and I was the tactical mission manager for the next three sols. Planning is what you do during the Martian night, and tactical operations occur when the rover is awake during the day. So Jennifer and her team were figuring out what to do, and I and my team were doing it. Or at least trying to.
On Sol 19 we simply attempted to communicate with Spirit to see if we could get some data back. Before the tactical shift started, per tradition, I played a song in the mission control room related to the day's activities. For Sol 19 I played "S.O.S.", by Abba. And that's about all we got out of Spirit that day. Several attempts at communication that all would have worked fine normally instead all failed, except for one commanded beep. A beep is simply the rover turning on the radio carrier for five minutes, which we detect at Earth. While there is no data on the carrier, the carrier itself provided one vital piece of information: Spirit was still there. It wasn't completely dead. Though we didn't get any data, we called that a good day. I ended my mission manager report with the optimistic plan: "In the long term, restore the state of the vehicle, diagnose and correct what happened, and return to normal operations." In fact, I ended my reports for Sol 20 and 21 with the same words.
On Sol 20, we made an even more concerted attempt to coax data out of the rover. Without the data, we'd have no idea what to do, or try to do, to recover normal operations. It took many attempts and variations on the approach, but we managed to get actual data modulated on the carrier. Much of the data was repeating gibberish, itself a mystery, but we did get a complete enough packet to get some health information on Spirit. Getting the data was fantastic, but the data itself painted a bleak picture.
What we saw was much higher internal temperatures on the rover, and a much lower battery voltage than we expected. Those two together were a clear indication that the rover was not going to sleep like it was supposed to. Normally the rover computer is on only five or six hours out of the day. That conserves the precious solar energy stored in the battery, and also keeps the rover from overheating.
Well Spirit wasn't sleeping for long enough, or possibly not sleeping at all.
What we had on our hands was one sick rover. Spirit had insomnia, a fever, was getting weaker all the time, was babbling incoherently, and was largely unresponsive to commands.
Not good. We had one rover on Mars dying, and two days later Opportunity was going to go through it's risky entry, descent, and landing. Before the weekend we could easily go from having two Mars rovers to having none at all.
While we still had time to command Spirit on Sol 20, before the Earth set at the landing site, our priority was to get the rover to go to sleep. We planned the shutdown command to hit the rover while it was in a communications session. That way we could see the session end prematurely, which would verify that it got the command. So we sent the SHUTDWN_DMT_TIL command. That stands for shutdown dammit until, which is followed by the time we want the rover to wake up. The "dammit" means an immediate emergency shutdown without regard for whatever activities happen to be running on the rover. (There was a little bit of humor in the naming of the commands.)
We sent the shutdown dammit, and sure enough, the communications session ended as soon as the command hit the rover. So it started the shutdown. Whew. We got Spirit to go to sleep. To verify this, we sent a beep command. We shouldn't get a beep in return, since a sleeping rover can't receive or respond to a command.
We got the beep.
What the ...? Spirit was supposed to go to sleep! Well, it didn't. Spirit was going to burn the midnight oil for yet another night. The Earth set at Gusev Crater, and we had to wait until the next day to try more commanding. In the meantime, Spirit's battery would continue to head downhill, and the electronics would get even hotter inside. We were running out of time.
Sol 21. We had a plan. The prevailing theory, at least the one we could do something about, was that the rover computer was in a "reboot loop". The response of the software when it encounters a problem it can't solve is to reboot. Just like you would do with your computer when it seizes up. Since no one is there to press the reset button, the rover does it automatically. However if the software encounters a problem while rebooting, then it's stuck rebooting forever. The designers had the foresight to put a delay in the reboot cycle so that there was some time to try to talk to the rover between reboots. That would explain the intermittent commandability, since whether the rover responded to a command would depend on when the command hit in the reboot cycle.
The idea behind the reboot is that everything in the software starts over from square one, so that whatever caused the problem in the first place should be gone the second time around. But in this case, the problem persisted. So Spirit was remembering something between reboots that was the cause of the problem. That pointed to either the flash memory (like in your digital camera), another smaller piece of memory that retains its contents called the EEPROM, or a hardware failure. The flash memory on the rover is used like the hard drive on your computer -- the file system is kept there.
Again, the brilliant designers had built in a back-door for us. There was a way to get the rover to reboot without ever looking at the file system on the flash. The radio that receives commands from Earth has built into it the ability to decode a few commands all by itself, called hardware commands. It doesn't need the computer at all to figure out those commands and execute them. One of those commands sets a flag to tell the computer to not use the flash file system when booting. Another one of those commands hits the reset button on the computer to force a reboot.
So that's what we tried on Sol 21. It took a few attempts, but we eventually and victoriously rebooted Spirit into a somewhat sane state where it was responsive to commands and not babbling. What a relief that was. We retrieved some power history data for that last few days, removed some planned relay communication windows, and finally gave Spirit a long deserved and badly needed sleep. This time it worked.
Now we had the secret sauce to get Spirit working. The rover would still wake up every morning into the reboot cycle, but we could quickly send up the necessary commands to boot without the flash file system. Which is what we did for several more days. We had won the race with time, and now we could carefully and methodically figure out what happened, fix it, and go on with the mission.
So I went home and went to sleep. Very easily I might add. My alarm went off about five hours later. Why? So that I could go back to JPL for the Opportunity landing that night! Hours after we got Spirit under control, Opportunity came screaming into the Martian atmosphere at 12,000 mph. Opportunity landed successfully, and we were back to confidently having two rovers. Now with both of them safely on Mars. Wow, what a ride.
The end of Sol 21 was the turning point in the recovery. It still took another two weeks to complete the diagnosis, fix the problem which involved effectively reformatting the hard drive (the flash), and restoring Spirit to full science operations. As we were recovering some of the data taken before the anomaly, we got back this beautiful color picture of the U.S. flag on the Rock Abrasion Tool (RAT) on the arm. That flag was on a metal shroud that was made from scorched and torn remnants of the World Trade Center towers. The RAT was designed and built in Manhattan, blocks from the WTC site. We put that picture of the RAT with the stars and stripes on the big screens in our mission control room, and I played our national anthem. Everyone stood with hand on heart. It was a good moment.
 Credit: NASA / JPL / Cornell |
Spirit has operated just beautifully ever since, aside from some recent signs
of old age, like a bum wheel motor. As I write this, Spirit is on Sol 967. Nine
hundred and sixty seven?! Wait, that can't be right. Let me check that ... Yep,
it's right. Incredible.
You may be wondering what caused the Sol 18 problem in the first place. We eventually figured out that there was simply an error in the software on the rover that we didn't catch in test. Some memory was getting used up more and more each day as we collected data. On Sol 18 that fixed block of memory got filled up, and the boot process failed while trying to read the file system. We actually did think about the sort of failures that can occur in the accumulation of many sols of operations. To scare out those sorts of problems, we ran a 10-sol test before landing. But we didn't run an 18-sol test. Well, not until Spirit ran the test for us on Mars, that is. We will probably run into a bad software problem some other day on some other spacecraft, but I can guarantee you, we won't let that kind of bug get us again.
Mars Sample Return, Part Deux
While MER was going on from 2000 through 2004, Mars Sample Return (MSR) continued to be discussed and developed. After the success of MER, MSR found a new home in the plan for future missions, which was the 2013 opportunity. The Mars Program budget had been significantly augmented, in part in response to the president's Vision to prepare for eventual human missions to Mars with more detailed science investigations and engineering system characterizations. That allowed for funding an entirely U.S. Mars Sample Return mission.
Around the Fall of 2004, the manager of the Mars Exploration Program asked me to leave MER, then in its first extended mission and soon to start its second, and become the Pre-Project Manager for the 2013 MSR. While the launch was still nine years away, a great deal of preparation was needed. This included detailed design work and credible cost estimates so that we could understand if it really fit in the budget or not, a lot of technology development for several key capabilities in the long chain of MSR events, and -- interestingly -- the preparation for a sample receiving facility on Earth. The last one needed early up-front work simply because of the very long schedule to get approval for a facility to safely contain the Martian samples while we examined them for signs of highly unlikely, but not inconceivable, biological threats to Earth.
Despite my previous experience, I decided to take the job. (Think of Charlie Brown with Lucy pulling the football away every time ... ) There really was the right amount of money in place, and the right science community support and participation in place. This could work. We could really pull off MSR this time. So MER had succeeded in my original objective for it, which was to regain the confidence we needed to go forward with MSR. Sure enough, Mars Sample Return popped up on the horizon.
I dove in with the existing MSR team plus bringing in more folks, and it was looking better all the time. The new launch vehicle families, the Atlas Vs and Delta IVs, had much greater performance that what we had handy in the last round. With more mass, there should be no great miracles that would be required technically. We now had a well established rover technology from MER to build off of for sample collection.
In the last round, the sample rocket didn't have guidance on the second stage to in order to make it light enough, but we weren't sure if that would work well enough. Now with more mass, we could make the rocket heavier and put guidance on the second stage. Now that system looked much more robust. Lastly, the work on the 2009 Mars Science Laboratory had developed a new Mars landing system that looked like it could actually handle the large landed payloads that MSR needed, maybe with some work on a larger parachute. All the pieces were in place.
So we charged ahead, working the design and cost of the mission itself, putting in place technology development plans over the next three years, and beginning the planning for the sample receiving facility.
Well, it didn't take long for NASA to begin to understand the cost of the Vision on the desired schedule combined with keeping shuttle and station going. They noticed the Mars program windfall that we had from the Vision. Looking at their priorities, they quite reasonably concluded that that money would be better spent on their nearer-term objectives. The human Mars missions were a long way off. I would probably have done the same thing in their shoes.
Since MSR depended on that windfall, it could no longer fly in 2013. It was pushed out first to 2016, then 2018, and now it lives somewhere around 2022, after a few more budget cuts. When a mission is moving forward in time at a rate greater than one year per year, which MSR was and is, well then it's not a real mission anymore.
So that was my second swing at the ball. Or the second attempted kick at Lucy's football, whichever you like. Either way, it was a big whoosh.
I'm on now to other things. Perhaps my age and cynicism has taken hold, but now I am skeptical that MSR will even make the currently imagined schedule in the decade after next. What I see coming in the next decade is a ramp up in spending to support sending people to the Moon, an energy crisis significantly impacting the economy as the rate of new oil discoveries peaks and can no longer match the rate of increasing demands, and the government trying to figure out how to pay all the retirees expecting their social security benefits. It's going to be a tough decade. The last two do not bode well for discretionary spending like NASA's, and all three would likely defer an expensive mission like Mars Sample Return for a few more decades.
MSR's only hope will be the young, enthusiastic up and coming space explorers who don't know or care about any of that, and through sheer force of will make it happen anyway. Even in my cynicism, I hold on to that glimmer of a vision for the future.
[ what is this? ]