MASTER: Mobile Autonomous Scientist
for Terrestrial and Extra-terrestrial Research
Dr. Iain Wallace, Dr. Mark Woods
Autonomous & Intelligent Systems Group
Wednesday 13th May 2015
2
What is MASTER?
Why do we need it?
3
NASA/JPL
Missing the Obvious ?
Retracing its steps
4
Why are SCISYS doing this work?
5
Science Autonomy
Heritage
6
7
Neuro-Fuzzy, Computer Vision Based Science Classification
Used FL flight code
8
Scene Understanding
Autonomously detect
what’s interesting to
scientists
Autonomously and
safely work out how
to gather more data
We have a basis for
an autonomous
scientific – robotic
“apprentice”
+ Tactical Planning = Closed loop science
Fusing Technologies
9
STOP-PRESS: ProviScout win in FP 7
CREST – Autonomous Robotic Scientist Showed that autonomous, science data acquisition/investigation was feasible
10
WP0
WP1
WP2
WP4Out Crop
Unusual
Material
Unusual
MaterialInteresting Stop
WP3
Full Autonomy Trials in Etna, Iceland, or Tenerife
11
Step Back Need for fundamental approach and review
12
Objectives • Conceive a generic novelty detection system architecture
• Test with Planetary Science and EO examples
• But this is an open research problem
• Currently several algorithmic choices for detection pipeline
steps but which ones work best?
• MASTER is GSP Early TRL so we:
• built the prototype architecture implementation
• and a methodology to evaluate the algorithmic choices –
• System allows you to use alternate algs. for different steps
and therefore compare
13
MASTER Team • Prime Contractor, Req’s Definition, System Design
Development and Test – SCISYS
• State of the Art Literature Review – MRG Uni. of
Oxford
• Planetary Science Dataset – Uni. of Leicester
• Earth Observation Dataset – Pixalytics
14
Autonomy and Robotics Group Scope
Robotic Platform + low-level software
Payload
Insts.
Operators
Autonomous Navigation
Scene Analysis
Mission Planning
Autonomy Framework
Rob
otic A
RM
Mission Planning
3D Data Understanding
Data
Remote Operations
The MASTER Problem
• Can you spot what’s unusual in this image?
1 image
~10cm
traverse
~2.5s
operations
The MASTER Problem
• Can you spot it now?
• …and you know what you’re looking for.
9 images
~1m traverse
~25s
operations
The MASTER Problem
• Can you spot it now?
25 images
~2.5m
traverse
~1 min
operations
The MASTER Problem
• Can you spot it now?
100 images
~10m
traverse
~5 mins
operations
The MASTER Problem
• Miss anything else interesting?
• 250 images, ~25m traverse, ~15 mins operations
• How many can be saved/returned?
• How do you choose?
• A domain independent software architecture for
detecting scientific events in images.
» This was narrowed to detecting (and defining) novelty.
• Capturing expert scientist’s knowledge.
• Two test domains
» Earth Observation
» Planetary Science
• System output is location and extent of regions of
interest.
• Project output is a detailed analysis of the problem
and properties of the presented solution.
MASTER Objectives
• Expected – This case corresponds to easily classified phenomena seen in training.
• Unusual – Phenomena that can be classified, but an expert has somehow indicated would be unusual.
• Unexpected – The phenomena primarily applied to different contexts in training.
• Classified Novelty – Reasonably probable the classifier has identified the phenomena, but it is a statistical outlier.
• Unclassified Novelty – Where the output is above the threshold to create a new class, this is “unclassified novelty”.
Classes of Novelty N
ovel
Not
Novel
• NASA/JPL Photojournal Planetary Image archive
Test Datasets - PS
• Collected during ESA SAFER trial.
Test Datasets – PS SAFER
• Rover NAVCAM from ESA Seeker trial
Test Datasets - Seeker
• Processed MERIS Level 1 data
Test Datasets - EO
System Overview
Saliency
Example Saliency Maps
Left-to-right: Original image, scale-space saliency map, image-signature saliency map.
• What’s outstanding in this image?
» Looking for local interest.
• Many algorithms evaluated
» Complete system can select from a library.
• Performance varies by domain » Varies by phenomena of interest too.
• SAFER Dataset performance appears poor, but this is a relative measure not absolute.
» Reflects the relatively sparse labelling.
• Seeker data also exhibited very tight error bounds – due to very homogenous data.
• Operating point selection important, as saliency represents a bound on performance.
Saliency Evaluation Conclusions
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
SK Pssafer PS EO
AUC
• Well-proven HOG/Visual Words/Spatial histogram based features » Others also tested, ICF, ACF…
• SVM Classifiers » libSVM implementation
» One classifier per class
» Trained “one versus the rest”
• Parameter selection, then training on complete training set.
Classifiers
0.8
classifier
Feature Vector
“On average, they work most of the time”
Summary of Classifier Performance
• Performance on the whole good.
» Generality exceeded our expectations.
• Best performance with homogeneity between test
and training.
» As seen in SAFER and Seeker data.
• More training examples improve performance.
» 25 samples a sensible lower bound in homogenous data.
• Classes should be well defined.
» Poor relative performance of “roi” and “flare” classes.
Classification Conclusions
• Simple Novelty
» If the most confident classifier is above a threshold, assign
a not-novel class according to training frequency.
Otherwise Unclassified Novelty.
• Group Novelty Classification
» Similar to the above, but considers also many confident
classes (confusion) to be unclassified novelty, and defines
classified novelty as a large margin to an uncertain class
or many unsure classes.
Novelty Detection
• What is novelty, in relation to the training data?
• How important is the detection of novelty in an image?
• How important is identifying the location and extent of
any novelty?
• How desirable is it to classify regions of interest that are
not novel?
• What is the appropriate trade-off between detecting
novelty, and the cost of false positives?
• Is the problem one of selection – “is this image novel?”
• Is the problem one of ranking – “which is the most novel
image?”
System Evaluation
Varying Thresholds
0
0.2
0.4
0.6
0.8
1
0 0.5 1
Recall
False-Discovery Rate
Varying Novelty
0
0.2
0.4
0.6
0.8
1
0 0.5 1
Recall
False-Discovery Rate
Varying Saliency
Labelling – Future Possibilities
Carnot Discovery - Saliency
UKSA Chameleon - Classifiers
UKSA Chameleon - Performance
Data product or Test
average time from
command to capture
(seconds) std. dev
CPU
consumption
(watts) data product size (Mb)
sensor power
consumption
(hardware)
memory usage
(component
specified, Mb) notes
energy cost over
idle overseer
(joules, rough)
idle system na na 6.15 0 - n/a
running overseer, no
sensors or data capture,
average over 10 minutes na na 11.39 13.1Mb in 253s -
generating map xtion m0 0.047 10.8 -0.02773
imu 0.00009 0.00004 11.66 0.00008 42.96875 20Hz 0.0000243
SSC Mode 1 rectified
images PAIR 0.0088 12.91 0.3 - 86.91894531 15Hz 0.013376
hokuyo to PC 0.12 0.014 11.69 0.009 - 43.45703125 20Hz 0.036
XB3 rectified images 0.012 0.021 20.48 0.81 - 166.1591797 15Hz commanded 0.10908
map to occupancy grid
(includes dtm) 0.007 0.005 28.05 0.05 n/a 87.43457031 SSC 1 pose 5Hz, PC 0.5hz ROR, Mapping 0.5hz 1.5 dem refresh, 3.3m drive0.11662
SSC Mode 7 rectified images 0.022 17.08 1.12 - 15Hz 0.12518
Xtion QVGA@60Hz PC 0.017 30.22 0.69 - 50.02734375 mode 2, 60hz, SOR filtered 0.32011
XB3 PC 0.12 14.71 2.6 - 196.0117188 1Hz 0.3984
Xtion VGA@30Hz PC 0.031 25.65 4.6 - 70.484375 30hz, mode 0, SOR filtered 0.44206
SSC Mode 1 VO estimate 0.059 26.05 0.0003 - 92.69140625
15Hz
still camera on tripod 0.86494
teval 0.069 0.0029 24.08 0.0001 85.43066406 SSC mode1, 15Hz 0.87561
XB3 VO estimate 0.057 0.0056 27.94 0.0003 - 176.2138672 15Hz commanded 0.94335
SSC Mode 1 PC no filtering 0.3 15.77 7 - 121.6806641 1Hz (commanded) 1.314
SSC Mode 7 VO estimate 0.13 29.1 0.0003 - 142.0976563 15Hz 2.3023
mapping eval 0.16 29.57 0.0002 148.4599609 eval 1Hz, SSC mode1 for PC 1Hz no filter 2.9088
generating map SSC1 0.55 24.3 0.175 SSC 1Hz PC no filter, Mapping 1Hz 7.1005
generating map SSC7 1.11 35.3 0.14 SSC 1Hz PC no filter, Mapping 1Hz 26.5401
XB3 PC SOR filter 2 27.01 2.3 - 200.6005859 1Hz 31.24
XB3 PC ROR filter 3.1 27.33 2.6 - 199.5898438 1Hz 49.414
SSC Mode 7 PC 3.2 28.77 26 - 228.1054688 1Hz 55.616
SSC Mode 1 PC SOR 5.6 26.73 6.4 - 156.6689453 1Hz 85.904
SSC Mode 1 PC ROR 8.6 26.78 7 - 157.4609375 1Hz 132.354
SSC Mode 7 PC SOR filter 24.2 27.42 26 - 391.0273438 1Hz 387.926
SSC Mode 7 PC ROR filter 142 27.75 26 - 412.1826172 1Hz 2323.12
• General classifiers show good performance
• Visual saliency shows good utility.
• Evaluation of novelty detection is not simple.
• Labelling and annotation is a big deal.
» Several good datasets created.
• Future application performance baselines have
implications:
» Where all images are inspected – e.g. labelling
» Selecting a few images for downlink
» No images are inspected – e.g. navcams.
• Evaluation is slow, execution is fast.
General Conclusions
Thank you [email protected] www.scisys.co.uk
See our rovers in the exhibition area this morning! We are recruiting for autonomy R&D engineers.