ELI40

Landing on the Moon

Part 2: Debrief - Apollo 11 post-mission notes

NASA

Post-mission debriefings are important. Evaluating what happened, what went right, and what went wrong, is the best way to improve future missions. Only after the fact, when there's time to study the details, do some subtle yet critical facts reveal themselves. Sometimes there was a problem that nobody had noticed. Other times, what seemed like a big problem, turns out to be no big deal.

Sure, you could skip the debrief, or you could look for improvements by going over the details again. That's the thing about knowledge, you'll never know what you don't know unless you stop and look for it.

descent
From Powered Descent Initiation (PDI) to the lunar surface took about 12 minutes. [4]

Down-range position error

The Apollo 11 LM landed about 3 nautical-miles off target. The guidance system was actually a combination of three separate systems: primary (PGNCS), alternate (AGS) and tracking data from Earth (MSFN). Back at mission control, all three systems were monitored constantly to see if any one of them disagreed with the others. All three systems performed as planned and data deviations stayed within acceptable limits.

The navigation error was caused by several small ΔV inputs to the spacecraft during coasting flight before the descent engine was started. In other words, something unexpectedly pushed the spacecraft off course. It wasn't an invisible space monster, it was small RCS attitude maneuvers and venting from the cooling system. These tiny little nudges during orbit were enough to cause the spacecraft to miss the landing site by 3 nautical miles.

The pre-selected landing area was nearly 10 nautical miles across. During mission planning, engineers had performed Monte Carlo analyses that predicted a 99% probability of landing within 3.6 nautical miles of the exact center. Saying they were off course isn't really accurate. The landing wasn't dead center but it was still within the permission mapped area. Mission planners knew there might be some last minute obstacles and when there were, it was not a problem.

Bingo fuel

Saturn V
As of 2017, the Saturn V is still the tallest, heaviest, and most powerful rocket ever built.

The Saturn V rocket used to go to the moon is still (as of 2017) the biggest rocket ever built. This is because it had to lift all the fuel and other equipment needed to get out of Earth orbit, into lunar orbit, down do the Moon's surface, back up again, then back to Earth. Every pound lifted into orbit requires an exponentially larger rocket. The Saturn V was at the limits of what a rocket could do. Because of this, there was no room for extra weight which meant no room for extra fuel while landing on the moon.

Still, not wanting to kill astronauts, NASA brought enough fuel to be safe, but no more. If everything went perfect, it would require approximately 12 minutes of fuel to land. Since things sometimes go wrong, there were a couple minutes of extra fuel so the astronauts could hover over to a new landing site if necessary. The fuel valve might not work perfectly so there was a contingency for that. Also, the fuel indicators also aren't perfect, especially when the fuel sloshes about, so there was a margin for that too.

NASA, following fighter pilot tradition, used a concept called Bingo fuel. Basically, it means "Almost out of fuel, land now or crash." For the Apollo missions, Bingo fuel officially meant 20 seconds of fuel remaining before needing to abort. So when CapCom called out "60 seconds" what they were really saying is "60 seconds until the 20 second Bingo fuel margin."

In addition to that, NASA was using the most conservative estimate for the amount of remaining fuel. If you had two fuel gauges, and one showed emptier than the other, which one would you bet your life on? NASA made the safe bet but after the landing, further investigation revealed that there was indeed more fuel remaining than indicated.

With all the hovering Neil Armstrong did looking for a good landing spot, he got right down to the last drop of fuel. That may seem reckless because with no tow trucks or ambulances nearby to help if something went wrong, he had no room for mistakes. However, the deeper story reveals how much fuel he really had left. Not only did he have the 20 seconds of Bingo fuel, he also knew there was probably a safety margin not shown on the gauges. Indeed, post analysis revealed incorrect readings due to fuel sloshing so there was actually about 45 seconds of fuel remaining. Most importantly though, Armstrong already had the LM down to about 50 feet above the surface. At that height, even with Bingo fuel, he knew he could still get on the ground safely. Even if they fell the last few feet, things fall six times slower on the moon than they do here on Earth.

Armstrong probably never had any intention of aborting. Imagine getting that close then giving up. For these guys, failure was not an option. Indeed, in a post mission statement, Armstrong said [3]:

I guess that, at that altitude, running out of fuel wasn't a consideration. Because we would have let it just quit on us, probably, and let it fall on in.

1202 & 1201 errors

Within 10 seconds of landing on the moon, the lab that had designed the guidance computer received an urgent phone call from NASA. They needed an explanation for those 1201 and 1202 program alarms. The mission would not be allowed to continue as planned without an explanation and reassurance that the computer was still functioning properly. They had less than 24 hours to figure it out.

The design team immediately ran to their simulation facility and worked all night, receiving another phone call from NASA about once every 15-30 minutes. The team went over old ground, new ground, brainstorms, crazy ideas, anything. They had to find an answer.

simulator
Armstrong and Aldrin practicing in the simulator.

The 1202 and 1201 alarms meant the computer was running out of memory. This was probably caused by something running too slow, exhausting available resources, and causing lower priority tasks to run out of memory and fail. But why was the computer running too slow?

Then someone remembered seeing a similar problem. Once, during a test with the Rendezvous Radar Switch on, the I/O system started stealing too many cycles while looking for radar data which then caused the other tasks to slow down. But why would the Rendezvous Radar Switch be on? That was only used for rendezvous during ascent, it shouldn't be on during descent. Had the astronauts made a mistake?

Either way, the first step was to see if the radar really was the problem. They sifted through the piles of telemetry data sent back to Earth from the LM, found the correct 16-bit word, found the correct bit, and... yikes!!! it showed that the radar was indeed on. Maybe an astronaut had made a mistake.

The switch [5]

Before accusing anyone, the engineers sifted through the 4-inch thick book of operating procedures and there it was, the checklist said to turn the radar switch on prior to descent. It wasn't the fault of the astronauts, they had followed procedure correctly.

So why hadn't this error ever been seen before? If this was standard procedure, why had none of the simulation tests ever shown this problem? As it turns out, since the simulator didn't use an active radar, that particular switch didn't really do anything. That's the danger of testing different systems in isolation, the problem only revealed itself when the systems were all operated together for the first time while descending to the moon.

Finally, an answer, and just in time. You can see it in the official transcript. Shortly before ascent back up to orbit, CAPCOM calmly gave Buzz Aldrin a couple checklist changes.

, CapCom: Got a couple changes to your surface checklist here. ... The main one being that we do not want the rendezvous radar on during the ascent, and we think that this will take care of some of the overflow of program alarms which we were getting during descent.

The astronauts hadn't made a mistake and neither had the software engineers. The guidance computer had been designed in such a way that it could handle unexpected errors just like this one. With no way to do a live test of landing on the moon without actually going to the moon, unexpected errors were inevitable. The entire world watched the first moon landing. Far fewer people understand how big of an achievement it really was.


References

  1. Apollo 11 mission timeline
  2. Apollo 11 official transcript, complete mission
  3. Apollo 11 landing transcript, with comments
  4. Apollo Lunar Descent and Ascent Trajectories
  5. Lunar lander control panel
  6. YouTube - Apollo 11 Landing from PDI to touchdown
  7. 1202 and 1201 alarms; Don Eyles, young engineer #1
  8. 1202 and 1201 alarms; Peter Adler, young engineer #2
  9. 1202 and 1201 alarms; Fred Martin, senior engineer
  10. Lunar Module renaming
  11. Google map of Apollo landing sites