MASTER: Mobile Autonomous Scientist

for Terrestrial and Extra-terrestrial Research

Dr. Iain Wallace, Dr. Mark Woods

Autonomous & Intelligent Systems Group

Wednesday 13th May 2015

2

What is MASTER?

Why do we need it?

3

NASA/JPL

Missing the Obvious ?

Retracing its steps

4

Why are SCISYS doing this work?

5

Science Autonomy

Heritage

6

7

Neuro-Fuzzy, Computer Vision Based Science Classification

Used FL flight code

8

Scene Understanding

Autonomously detect

what’s interesting to

scientists

Autonomously and

safely work out how

to gather more data

We have a basis for

an autonomous

scientific – robotic

“apprentice”

+ Tactical Planning = Closed loop science

Fusing Technologies

9

STOP-PRESS: ProviScout win in FP 7

CREST – Autonomous Robotic Scientist Showed that autonomous, science data acquisition/investigation was feasible

10

WP0

WP1

WP2

WP4Out Crop

Unusual

Material

Unusual

MaterialInteresting Stop

WP3

Full Autonomy Trials in Etna, Iceland, or Tenerife

11

Step Back Need for fundamental approach and review

12

Objectives • Conceive a generic novelty detection system architecture

• Test with Planetary Science and EO examples

• But this is an open research problem

• Currently several algorithmic choices for detection pipeline

steps but which ones work best?

• MASTER is GSP Early TRL so we:

• built the prototype architecture implementation

• and a methodology to evaluate the algorithmic choices –

• System allows you to use alternate algs. for different steps

and therefore compare

13

MASTER Team • Prime Contractor, Req’s Definition, System Design

Development and Test – SCISYS

• State of the Art Literature Review – MRG Uni. of

Oxford

• Planetary Science Dataset – Uni. of Leicester

• Earth Observation Dataset – Pixalytics

14

Autonomy and Robotics Group Scope

Robotic Platform + low-level software

Payload

Insts.

Operators

Autonomous Navigation

Scene Analysis

Mission Planning

Autonomy Framework

Rob

otic A

RM

Mission Planning

3D Data Understanding

Data

Remote Operations

The MASTER Problem

• Can you spot what’s unusual in this image?

1 image

~10cm

traverse

~2.5s

operations

The MASTER Problem

• Can you spot it now?

• …and you know what you’re looking for.

9 images

~1m traverse

~25s

operations

The MASTER Problem

• Can you spot it now?

25 images

~2.5m

traverse

~1 min

operations

The MASTER Problem

• Can you spot it now?

100 images

~10m

traverse

~5 mins

operations

The MASTER Problem

• Miss anything else interesting?

• 250 images, ~25m traverse, ~15 mins operations

• How many can be saved/returned?

• How do you choose?

• A domain independent software architecture for

detecting scientific events in images.

» This was narrowed to detecting (and defining) novelty.

• Capturing expert scientist’s knowledge.

• Two test domains

» Earth Observation

» Planetary Science

• System output is location and extent of regions of

interest.

• Project output is a detailed analysis of the problem

and properties of the presented solution.

MASTER Objectives

• Expected – This case corresponds to easily classified phenomena seen in training.

• Unusual – Phenomena that can be classified, but an expert has somehow indicated would be unusual.

• Unexpected – The phenomena primarily applied to different contexts in training.

• Classified Novelty – Reasonably probable the classifier has identified the phenomena, but it is a statistical outlier.

• Unclassified Novelty – Where the output is above the threshold to create a new class, this is “unclassified novelty”.

Classes of Novelty N

ovel

Not

Novel

• NASA/JPL Photojournal Planetary Image archive

Test Datasets - PS

• Collected during ESA SAFER trial.

Test Datasets – PS SAFER

• Rover NAVCAM from ESA Seeker trial

Test Datasets - Seeker

• Processed MERIS Level 1 data

Test Datasets - EO

System Overview

Saliency

Example Saliency Maps

Left-to-right: Original image, scale-space saliency map, image-signature saliency map.

• What’s outstanding in this image?

» Looking for local interest.

• Many algorithms evaluated

» Complete system can select from a library.

• Performance varies by domain » Varies by phenomena of interest too.

• SAFER Dataset performance appears poor, but this is a relative measure not absolute.

» Reflects the relatively sparse labelling.

• Seeker data also exhibited very tight error bounds – due to very homogenous data.

• Operating point selection important, as saliency represents a bound on performance.

Saliency Evaluation Conclusions

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

SK Pssafer PS EO

AUC

• Well-proven HOG/Visual Words/Spatial histogram based features » Others also tested, ICF, ACF…

• SVM Classifiers » libSVM implementation

» One classifier per class

» Trained “one versus the rest”

• Parameter selection, then training on complete training set.

Classifiers

0.8

classifier

Feature Vector

“On average, they work most of the time”

Summary of Classifier Performance

• Performance on the whole good.

» Generality exceeded our expectations.

• Best performance with homogeneity between test

and training.

» As seen in SAFER and Seeker data.

• More training examples improve performance.

» 25 samples a sensible lower bound in homogenous data.

• Classes should be well defined.

» Poor relative performance of “roi” and “flare” classes.

Classification Conclusions

• Simple Novelty

» If the most confident classifier is above a threshold, assign

a not-novel class according to training frequency.

Otherwise Unclassified Novelty.

• Group Novelty Classification

» Similar to the above, but considers also many confident

classes (confusion) to be unclassified novelty, and defines

classified novelty as a large margin to an uncertain class

or many unsure classes.

Novelty Detection

• What is novelty, in relation to the training data?

• How important is the detection of novelty in an image?

• How important is identifying the location and extent of

any novelty?

• How desirable is it to classify regions of interest that are

not novel?

• What is the appropriate trade-off between detecting

novelty, and the cost of false positives?

• Is the problem one of selection – “is this image novel?”

• Is the problem one of ranking – “which is the most novel

image?”

System Evaluation

Varying Thresholds

0

0.2

0.4

0.6

0.8

1

0 0.5 1

Recall

False-Discovery Rate

Varying Novelty

0

0.2

0.4

0.6

0.8

1

0 0.5 1

Recall

False-Discovery Rate

Varying Saliency

Labelling – Future Possibilities

Carnot Discovery - Saliency

UKSA Chameleon - Classifiers

UKSA Chameleon - Performance

Data product or Test

average time from

command to capture

(seconds) std. dev

CPU

consumption

(watts) data product size (Mb)

sensor power

consumption

(hardware)

memory usage

(component

specified, Mb) notes

energy cost over

idle overseer

(joules, rough)

idle system na na 6.15 0 - n/a

running overseer, no

sensors or data capture,

average over 10 minutes na na 11.39 13.1Mb in 253s -

generating map xtion m0 0.047 10.8 -0.02773

imu 0.00009 0.00004 11.66 0.00008 42.96875 20Hz 0.0000243

SSC Mode 1 rectified

images PAIR 0.0088 12.91 0.3 - 86.91894531 15Hz 0.013376

hokuyo to PC 0.12 0.014 11.69 0.009 - 43.45703125 20Hz 0.036

XB3 rectified images 0.012 0.021 20.48 0.81 - 166.1591797 15Hz commanded 0.10908

map to occupancy grid

(includes dtm) 0.007 0.005 28.05 0.05 n/a 87.43457031 SSC 1 pose 5Hz, PC 0.5hz ROR, Mapping 0.5hz 1.5 dem refresh, 3.3m drive0.11662

SSC Mode 7 rectified images 0.022 17.08 1.12 - 15Hz 0.12518

Xtion QVGA@60Hz PC 0.017 30.22 0.69 - 50.02734375 mode 2, 60hz, SOR filtered 0.32011

XB3 PC 0.12 14.71 2.6 - 196.0117188 1Hz 0.3984

Xtion VGA@30Hz PC 0.031 25.65 4.6 - 70.484375 30hz, mode 0, SOR filtered 0.44206

SSC Mode 1 VO estimate 0.059 26.05 0.0003 - 92.69140625

15Hz

still camera on tripod 0.86494

teval 0.069 0.0029 24.08 0.0001 85.43066406 SSC mode1, 15Hz 0.87561

XB3 VO estimate 0.057 0.0056 27.94 0.0003 - 176.2138672 15Hz commanded 0.94335

SSC Mode 1 PC no filtering 0.3 15.77 7 - 121.6806641 1Hz (commanded) 1.314

SSC Mode 7 VO estimate 0.13 29.1 0.0003 - 142.0976563 15Hz 2.3023

mapping eval 0.16 29.57 0.0002 148.4599609 eval 1Hz, SSC mode1 for PC 1Hz no filter 2.9088

generating map SSC1 0.55 24.3 0.175 SSC 1Hz PC no filter, Mapping 1Hz 7.1005

generating map SSC7 1.11 35.3 0.14 SSC 1Hz PC no filter, Mapping 1Hz 26.5401

XB3 PC SOR filter 2 27.01 2.3 - 200.6005859 1Hz 31.24

XB3 PC ROR filter 3.1 27.33 2.6 - 199.5898438 1Hz 49.414

SSC Mode 7 PC 3.2 28.77 26 - 228.1054688 1Hz 55.616

SSC Mode 1 PC SOR 5.6 26.73 6.4 - 156.6689453 1Hz 85.904

SSC Mode 1 PC ROR 8.6 26.78 7 - 157.4609375 1Hz 132.354

SSC Mode 7 PC SOR filter 24.2 27.42 26 - 391.0273438 1Hz 387.926

SSC Mode 7 PC ROR filter 142 27.75 26 - 412.1826172 1Hz 2323.12

• General classifiers show good performance

• Visual saliency shows good utility.

• Evaluation of novelty detection is not simple.

• Labelling and annotation is a big deal.

» Several good datasets created.

• Future application performance baselines have

implications:

» Where all images are inspected – e.g. labelling

» Selecting a few images for downlink

» No images are inspected – e.g. navcams.

• Evaluation is slow, execution is fast.

General Conclusions

Thank you iain.wallace@scisys.co.uk www.scisys.co.uk

See our rovers in the exhibition area this morning! We are recruiting for autonomy R&D engineers.