patching flight software on mars - nasa
TRANSCRIPT
![Page 1: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/1.jpg)
Patching Flight Software on Mars
Ed Benowitz / Mark Maimone MSL Flight Software Developers NASA/Jet Propulsion Laboratory California Institute of Technology
Copyright 2015 California Institute of Technology. Government sponsorship acknowledged.
Artist’s Concept. NASA/JPL-Caltech
![Page 2: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/2.jpg)
Agenda
• Spirit/Opportunity Rover Mission Overview • Curiosity Rover Mission Overview • Flight Software Architecture Overview • Options for Changing Software in Flight • Goals for Patching • General Patching Approaches • Trades • How a FSW Image is Loaded • Patch Scenarios • Lessons learned
Page 2
![Page 3: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/3.jpg)
Spirit Mission Overview
NASA/JPL - Caltech/Cornell
NASA’s Mars Exploration Rovers were designed to survive 90 days. But Spirit explored Mars for 6 years! A stalled wheel motor led to a serendipitous discovery proving Mars had freestanding pools of water. FSW patches helped keep Spirit going even with failed actuators. .
![Page 4: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/4.jpg)
Opportunity Overview
NASA/JPL-Caltech/Cornell
Opportunity is now in its 12th year of Mars Exploration! Opportunity holds the distance record for roving across a planetary surface (over 42km) There have been 6 major FSW updates and tens of hot or cold patches.
![Page 5: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/5.jpg)
Spirit and Opportunity Rover Mission Overview
• Size – About the size of a golf cart -- 5 feet long (not including the
arm), 6 feet wide and 5 feet tall
• Arm reach – About 3 feet
• Weight: – 185 kilograms (400 pounds)
• Mission: – Search for and characterize a wide range of rocks and soils
that hold clues to past water activity on Mars.
Page 5
![Page 6: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/6.jpg)
Curiosity Rover Mission Overview
Page 6
Artist’s Concept. NASA/JPL-Caltech
![Page 7: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/7.jpg)
Curiosity Rover Mission Overview
Page 7
NASA/JPL-‐Caltech/MSSS
![Page 8: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/8.jpg)
Curiosity Rover Mission Overview
• Size – About the size of a small car -- 10 feet long (not including the
arm), 9 feet wide and 7 feet tall or about the height of a basketball player
• Arm reach – About 7 feet
• Weight: – 900 kilograms (2,000 pounds)
• Mission: – To search areas of Mars for past or present conditions
favorable for life, and conditions capable of preserving a record of life
Page 8
![Page 9: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/9.jpg)
Flight Software Overview
• MER and MSL rovers all run the VxWorks operating system
– Supports executing shell commands • Change variables • Dynamic code loading and execution
• MSL has over 100 flight software modules, most run as their own task
• Communication between tasks – Inter-process communication
(IPC) messages • Send a message • Wait for a reply • Proceed
Page 9
![Page 10: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/10.jpg)
Options for Changing Software in Flight
• Uplink and install an entire flight software image • Patch an existing flight software image
– Modify or extend the existing onboard flight software image
• Extend FSW with a new software component – Dynamically load new code – Add a new .o and add new commands, telemetry, and data
products using a FSW-provided API
10/22/15 10
![Page 11: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/11.jpg)
Why Patch?
• Patching can: – Add entirely new functionality to flight software
• Could include new commands and telemetry • May be necessary to work around newly discovered hardware
behavior • Increase science return
– Fix a flight software bug • Patching potentially a faster process than a full FSW image
– A full FSW load typically requires that Validation and Verification (V&V) tests run against all FSW capabilities, even ones that are not changing
10/22/15 11
![Page 12: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/12.jpg)
General Patching Approaches
• Hot patch – Changes code in RAM while we are running – VxWorks also allows dynamic loading of .o files
• Cold patch – Modify the flight software image stored in a non-volatile storage area. – The new code image would be used on a subsequent boot – Does not change the currently running code
• Trades – Risk
• Simpler uninstall for a hot patch – Reboot would restore flight software back into a clean version.
– Testing effort – Uplink bandwidth
10/22/15 12
![Page 13: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/13.jpg)
General Patching Approaches
• Patches may need to – Add code to an existing function – Replace a buggy function with a new function – Remove code from an existing function – Add new global variables – Change or assign references to existing global variables,
e.g. function pointers – Add new commands and telemetry
• And just in case the patch doesn’t work – Have a plan for un-installing the patch
10/22/15 13
![Page 14: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/14.jpg)
Trades
• Trade: Should flight software provide commands for patching? – MER and MSL had some support for patches
• MER: Could apply patches to a copy of FSW in RAM and write to non-volatile storage
• MSL has a command for adding a new FSW component module – But hot patches that change existing code require back-door methods
• Issues with a general built in patch capability – Trades FSW development resources against Ops resources – A small code change can result in a large difference in the FSW
binary • If memory locations move, the size of a diff file can be large
– Both missions had small amounts of RAM • MSL could not add padding inside binaries to allow for reduced diff file
sizes • MER did have room to add padding
10/22/15 14
![Page 15: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/15.jpg)
MER Example Patch Differences
10/22/15
Green – No change Red – At least one bit changed
Memory Start
Memory End
![Page 16: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/16.jpg)
How MSL Boots
• The boot loader loads the FSW image from the currently selected NOR bank – Copy from NOR to RAM – Execute the image image from RAM
• If the image fails to load, the boot loader uses another NOR bank
10/22/15 16
![Page 17: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/17.jpg)
How MER boots normally
• MER had a Rad6K flight computer – FSW images were stored in EEPROM – Multiple, different FSW images could be stored
10/22/15 17
![Page 18: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/18.jpg)
How A Full New Image is Installed
• The ground uplinks a new binary image to the file system
• Upon command, the new image is saved – Flight software loads the image into RAM – Flight software burns the NOR zone with the image
• MER – In addition to the above, MER also allowed patching the
RAM image before burning to its nonvolatile storage in EEPROM
10/22/15 18
![Page 19: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/19.jpg)
MSL Patch Scenario: Remove code
• A bug affecting both Cruise and Surface operations required the removal of several lines of code – Cruise: The spacecraft is always on
• Hot patch approach: Patch the code in RAM only – Replace the original code with no-ops
» A memory assignment
– Surface: The rover has a wakeup/shutdown cycle • Cold patch approach: Change the image in NOR
– Load the original image into RAM – Modify the RAM image to add the no-ops. – Save the newly modified image into NOR – The bug fixes are retained across reboots
10/22/15 19
![Page 20: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/20.jpg)
MSL Patch Scenario: Replacing a function
• Add code to a function to fix a bug – The function referenced global variables
• Implemented as a hot patch. • Use a replacement function
– Create a new .o file to be dynamically loaded by Vxworks containing
• Replacement function • Pointers to the global variables
• Installation – Load the .o – Find the address of the old function – Poke an instruction into the old function to jump to the new function – Find the address of the global variables, and assign them to the
new global variable pointers • Install performed on every boot by a sequence
10/22/15 20
![Page 21: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/21.jpg)
MSL Patch Scenario: Adding new functionality
• MSL can add new functionality – Special commands can install a new FSW component .o
containing new code, commands, telemetry
• Registration to add new functionality – The new .o is dynamically loaded during the boot process – The new code may register new commands, telemetry, data
products – The new code can hook into the existing wakeup and
shutdown process • Functions are registered.
– The registration is performed once • Registration information is stored in non-volatile memory
10/22/15 21
![Page 22: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/22.jpg)
Lessons Learned
• We had to use back-door methods to change small parts of the existing flight code on MSL – Even though explicit commands were provided to add new
functionality and load an entire new flight software image • Onboard patching of small code changes was frequent enough
to consider adding a patch command in future missions. – Running hot patch setup sequences has become a standard
part of our nominal sequencing process
Page 22
![Page 23: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/23.jpg)
Thanks
• MSL flight software team – Led by Ben Cichy – Special thanks to Danny Lam
• MER flight software team – Led by Glenn Reeves
Page 23
![Page 24: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/24.jpg)
QUESTIONS?
Page 24
![Page 25: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/25.jpg)
BACKUP
Page 25
![Page 26: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/26.jpg)
Mars facts
• Size: – Half the size of Earth, but same land area
• Weight: – 1/10th of what the Earth weighs
• Gravity: – 38% as strong as on Earth
• Average Temperature: – -81 degrees Fahrenheit
• Atmosphere: – Mostly carbon dioxide, with some water vapor
• Moons: – 2, Phobos and Deimos
Page 26
![Page 27: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/27.jpg)
Mars facts
Page 27
![Page 28: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/28.jpg)
MSL Flight Computer
• Single Board Computer – RAD 750
• On-board memory includes 128 MB of volatile DRAM • 4 GB of NAND non-volatile memory on a separate card • Both with error detection and correction • Runs on two flight computers
– Prime and backup
Page 28
![Page 29: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/29.jpg)
Cameras
• Cameras – Four Pairs of Engineering Hazard Avoidance Cameras (Hazcams):
• Mounted on the lower portion of the front and rear of the rover, these black-and-white cameras use visible light to capture three-dimensional (3-D) imagery.
– Two Pairs of Engineering Navigation Cameras (Navcams): • Mounted on the mast (the rover "neck and head"), these black-and-
white cameras use visible light to gather panoramic, three-dimensional (3D) imagery. The navigation camera unit is a stereo pair of cameras, each with a 45-degree field of view
– Four Science Cameras: • MastCam (one pair), will take color images, three-dimensional stereo
images, and color video footage • The Mars Hand Lens Imager is the equivalent of a geologist's hand
lens • Chemcam
Page 29
![Page 30: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/30.jpg)
Arm and Hand
Page 30
![Page 31: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/31.jpg)
Drill
• Curiosity drilled on Mars
Page 31
![Page 32: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/32.jpg)
Instruments on Curiosity
• ChemCam – Will fire a laser and analyze the elemental composition of vaporized
materials from areas smaller than 1 millimeter on the surface of Martian rocks and soils.
• Chemin: – The Chemistry and Mineralogy instrument, or CheMin for short, will
identify and measure the abundances of various minerals on Mars. • SAM
– The Sample Analysis at Mars features chemical equipment found in many scientific laboratories on Earth. SAM will search for compounds of the element carbon.
• REMS – A weather monitoring station
• RAD – Measures radiation
Page 32
![Page 33: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/33.jpg)
Communication
• Curiosity can communicate directly to Earth • Typically Curiosity talks to satellites orbiting Mars
– These orbiters then forward data to Earth
Page 33
![Page 34: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/34.jpg)
Photos from Mars
Page 34
![Page 35: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/35.jpg)
Photos from Mars
Page 35
![Page 36: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/36.jpg)
Photos from Mars
Page 36
![Page 37: Patching Flight Software on Mars - NASA](https://reader031.vdocument.in/reader031/viewer/2022012015/615a42d183d85f2d9369b8af/html5/thumbnails/37.jpg)
References
[1] J. Grotzinger, J. Crisp, A. Vasavada, R. Anderson, C. Baker, R. Barry, D. Blake et al. “Mars Science Laboratory mission and science investigation,” Space science reviews 170, no. 1-4, 2012, pp. 5-56. [2] http://mars.jpl.nasa.gov/msl/ [3] http://mars.nasa.gov/mer/
Page 37