Metamorphic, Model-Based Testing of Autonomous Systems

Mikael LindvallAdam Porter

Gudjon Magnusson Christoph Schulze

Testing Autonomous Systems

§ Increasing interest in using autonomous systems to perform safety-critical missions in the real-world

§ Can’t test every possible scenario§ Difficult to determine correct test behavior

§ Systems designed to make their own decisions§ Non-deterministic control algorithms

§ Uses sense and avoid algorithm to autonomously navigate an unknown environment to deliver packages

§ Uses LIDAR and traditional cameras to avoid obstacles and identify landing pads

§ Used a simulation environment and a testing framework to generate numerous test cases

Testing an Autonomous Drone

Model-Based, Metamorphic Testing

§ Manually created test cases for a small set of interesting scenarios

§ Use model-based testing to generate a large number of variants from each manual scenario

§ Instrumented drone software to emit information about its internal state

Model-Based, Metamorphic Testing

§ Used metamorphic testing principles to identify anomalous behaviors in “equivalent” test cases

§ Assume small changes in test cases should lead to small changes in behavior

§ Flag anomalies to detect unsafe/unstable behavior for certain configurations

Metamorphic Equivalences

§ Behavior should be relatively independent of:§ Repeated runs§ Geographic rotation§ Geographic translation§ Obstacle locations § Obstacle formations

Comparing Behaviors

§ Flight path shape§ Experimental

§ Safety property violations§ Time in unsafe internal states§ Control state sequences

Example Test Scenario

Starthere(LandingpadA)

Landhere(LandingpadB)

Drone

Avoidobstacles

Analyzing a Test Run

Identifying an Anomaly

Original Rotated1º

Identifying Another Anomaly

360 Test Case Variants

Preliminary Findings

§ Model-based, metamorphic testing was practical and useful for safety testing of an autonomous system§ Identified unstable/unpredictable behaviors§ Identified corner cases with catastrophic failure and

landing problems§ Confirmed usefulness of safety-related features for

identifying anomalies (implemented as safety cages)

Future Work

§ Higher fidelity simulation§ Smarter (adaptive) test generation§ Automatically learn relationships between

changes in environment and changes in behavior

§ Define autonomy requirements