Building a lifeboat: MSL's uplink and installation campaign to restore a failing backup computer

Abstract

Flight software updates are among the hardest and most dangerous activities for the Mars Science Laboratory (MSL) Curiosity team. While danger is often mitigated by backups, fallback strategies, and incremental installation with ground-in-the-loop cycles which provide a safety net for the installation process, the software update described in this paper was unable to use many of the common practices due to the nature of the fault addressed by the update. The MSL rover (landed August 2012) encountered a problem with one of its computer's non-volatile storage chips in 2019, requiring a swap to its backup computer and an urgent software upgrade called R-Hope. R-Hope, a lifeboat to be used in the event of primary computer issues, was written, tested, and sent to the rover in lightning speed of just 19 months. Multi-mission and team coordination allowed the 49 flight software image files to be uplinked to the rover over a 6-week period, using multiple paths and backup options for speedy delivery. In the end, the R-Hope software upgrade returned the computer to operation as a backup flight computer. The flight software transition was designed to impact science return as little as possible, and installation plans included science activities for the majority of MSL instruments. This paper describes the uplink and installation campaigns for R-Hope, and discusses the notable lessons learned by the operations team.