We have an audio recording of Dave's presentation.
The C-17 program went through a major avionics upgrade with scheduled delivery of production planes in late 2006. Soon after these planes were delivered to the USAF customer, multiple instances of loss of all cockpit displays were encountered. The displays included the primary flight data and lasted from 10 to 15 seconds. The occurrence was random and could not be tied to pilot actions. Boeing and the USAF judged this to be a very serious safety of flight issue, and all C-17s with this avionics upgrade were severely restricted in use. These aircraft could not fly in bad weather, at night, over the ocean, or perform airdrops. These restrictions made the planes unusable for their intended purposes, and the USAF customer was justifiably outraged.
Boeing initiated an immediate analysis of the failures. The USAF was very cooperative in allowing test equipment to be installed on the aircraft that recorded all data on the avionics bus. It became obvious that contention for control of the avionics bus was happening between the two mission computers (MC). The system only allows one MC to control the avionics bus, with the other MC serving as a hot spare backup. For reasons unknown at the time, the bus controller MC was pausing in its bus activity long enough for the backup MC to believe the bus controller had failed. The backup MC then promoted itself to bus controller as appropriate. However, soon after this bus control switch over, the former bus controller came back in control of the bus, resulting in two bus controllers. Two bus controllers resulted in a completely incoherent avionics bus. Message traffic was garbled for the entire avionics system. The displays blanked per design, because in the absence of fresh data, the displays must blank to satisfy hazardously misleading information requirements.
The presentation will provide a lessons learned perspective by going through the analysis process, which included hardware review and scrutiny of the software updates. In the end, the root cause of the problem was an improper software definition that had existed in the software for 12 years without ever causing a problem. The presentation will identify how to investigate similar issues, how to avoid latent software errors, and what C-17 software development has done to their process to ensure such escaped defects are significantly less likely.
Our speaker, Dave Graf, is an Associate Technical Fellow at Boeing in Long Beach where he specializes in embedded real-time airborne avionics systems. He has been actively involved in the specification, design, implementation and delivery of avionics software for various aircraft over the past 26 years. He is currently the Chief Software Architect for the C-17 cargo plane.