understanding underwater optical image datasetsoptical modems: 1,000,000 bits/s at 100 meters [farr...
TRANSCRIPT
1 PhD Thesis Defense - J.W. Kaeli, 2013
Computational Strategies for
Understanding Underwater
Optical Image Datasets
Jeffrey W. Kaeli
Advisor:
Hanumant Singh, WHOI
Committee:
John Leonard, MIT
Ramesh Raskar, MIT
Antonio Torralba, MIT
2 PhD Thesis Defense - J.W. Kaeli, 2013
“latency of understanding” in underwater robotics
3 PhD Thesis Defense - J.W. Kaeli, 2013
“latency of understanding” LOU 0 for ROVs
4 PhD Thesis Defense - J.W. Kaeli, 2013
what do we lose by “cutting the cord?”
5 PhD Thesis Defense - J.W. Kaeli, 2013
what can we gain by “cutting the cord?”
6 PhD Thesis Defense - J.W. Kaeli, 2013
AUVs generate large volumes of data
O(1,000s) images
1 image
3 seconds
3600 s
1 hour
O(hours)
x
x
O(10,000,000,000) bits
O(10,000,000) bits
1 image
x
O(1-10) GB
7 PhD Thesis Defense - J.W. Kaeli, 2013
underwater communication channels are limited
Optical Modems:
1,000,000 bits/s at 100 meters [Farr et al. 2010]
Acoustic Modems:
100 - 1,000 bits/s at kilometers
[Frietag et al. 2005]
8 PhD Thesis Defense - J.W. Kaeli, 2013
wavelet-based compression coded with
Set Partitioning in Hierarchical Trees (SPIHT)
smaller packet size optimized for acoustics
fully embedded progressive encoding
throughput of 1 megapixel image / ~15 minutes
acoustic transmission of imagery
[Murphy 2012, PhD Thesis]
9 PhD Thesis Defense - J.W. Kaeli, 2013
manual annotation is time consuming
[Ferrini et al., 2006] [Kaeli 2011] [Futrelle & York, 2012]
10 PhD Thesis Defense - J.W. Kaeli, 2013
automatic classification aids quantitative analysis
habitats [Loomis 2011] [Pizarro et al., 2009]
[Rigby et al., 2010] [Steinberg et al., 2011]
fish [Loomis 2011] [Spampinato et al., 2010]
scallops [Dawkins et al., 2013]
starfish [Clement et al., 2002]
coral [Johnson et al., 2006] [Kaeli et al, 2006]
[Purser et al., 2009] [Soriano et al., 2001]
[Shihavuddin et al., 2013] [Stokes & Deane, 2009]
11 PhD Thesis Defense - J.W. Kaeli, 2013
summary of latencies contributing to overall LOU
O(hours)
? O(days-weeks)
O(hours)
12 PhD Thesis Defense - J.W. Kaeli, 2013
what is the smallest latency with which we can
begin to “understand” the survey environment?
RAW IMAGERY
PRE-PROCESSING
SELECTION
COMPRESSION
TELEMETRY
UNDERSTANDING
{ [Murphy 2012,
PhD Thesis]
Chap. 1,2
Chap. 2,3
Chap. 3
13 PhD Thesis Defense - J.W. Kaeli, 2013
1. Image Correction
2. Computational Strategies
3. Understanding Underwater
Optical Image Datasets
outline of thesis chapters and contributions
14 PhD Thesis Defense - J.W. Kaeli, 2013
1. Image Correction
outline of thesis chapters and contributions
review underwater image formation and broad
range of existing correction techniques
estimate water column properties and unknown
vehicle parameters using multi-sensor fusion
use estimated values to correct images
15 PhD Thesis Defense - J.W. Kaeli, 2013
2. Computational Strategies
outline of thesis chapters and contributions
introduce novel multi-scale image processing
framework for efficient feature computations
discuss design of invariant features
16 PhD Thesis Defense - J.W. Kaeli, 2013
3. Understanding Underwater
Optical Image Datasets
outline of thesis chapters and contributions
propose lightweight keypoint detection and
description scheme for use in scene classification
modify surprise-based navigation summaries for
selecting images to transmit acoustically
demonstrate creation of low-bandwidth semantic
maps via cumulative image telemetry
17 PhD Thesis Defense - J.W. Kaeli, 2013
1. Image Correction
2. Computational Strategies
3. Understanding Underwater
Optical Image Datasets
outline of thesis chapters and contributions
18 PhD Thesis Defense - J.W. Kaeli, 2013
underwater image formation
19 PhD Thesis Defense - J.W. Kaeli, 2013
underwater robotic imaging platform assumptions
ignore gain
no natural light
assume white strobe
neglect spreading
no additive effects
ignore lens effects
20 PhD Thesis Defense - J.W. Kaeli, 2013
log color channel means as a function of altitude
21 PhD Thesis Defense - J.W. Kaeli, 2013
could additional sensors be useful for correction?
SeaBed - Autonomous
Underwater Vehicle
SeaSled – Towed
Camera System
22 PhD Thesis Defense - J.W. Kaeli, 2013
use overlapping imagery to constrain equations
3 equations (RGB), 6 unknowns
23 PhD Thesis Defense - J.W. Kaeli, 2013
use overlapping imagery to constrain equations
3 + 3 = 6 equations, same 6 unknowns!
24 PhD Thesis Defense - J.W. Kaeli, 2013
attenuation coefficient estimates
25 PhD Thesis Defense - J.W. Kaeli, 2013
beam pattern estimate in angular space
26 PhD Thesis Defense - J.W. Kaeli, 2013
artifacts from artificial lighting and attenuation
27 PhD Thesis Defense - J.W. Kaeli, 2013
comparison with our results
raw image
our method frame averaging
white balance (wb) homomorphic + wb
adapt. hist. eq.+ wb
29 PhD Thesis Defense - J.W. Kaeli, 2013
sample corrected imagery
30 PhD Thesis Defense - J.W. Kaeli, 2013
sample corrected imagery
31 PhD Thesis Defense - J.W. Kaeli, 2013
Demonstrated a multi-sensor fusion-based
approach to underwater image correction
Additional sensor information useful for correction,
but makes correction platform-dependent
[Rock et al., unpublished] [Kaeli et al., 2011] [Bryson et al., 2012]
Potential for historical water quality from past data
1. Image Correction
summary, conclusions, future work
32 PhD Thesis Defense - J.W. Kaeli, 2013
1. Image Correction
2. Computational Strategies
3. Understanding Underwater
Optical Image Datasets
outline of thesis chapters and contributions
33 PhD Thesis Defense - J.W. Kaeli, 2013
Convolution is still a major bottleneck in many
multi-scale image processing frameworks such as
fast keypoint detection [Calonder et al., 2010]
Graphics Processing Units (GPUs) have increased
“brute force” computing power [Loomis 2011]
Can we exploit pixel grid geometries that allow us to
substitute adds and bit shifts for multiples while still
approximating a Gaussian? [Viola et al., 2001]
Applications on low-power robotic imaging platforms
34 PhD Thesis Defense - J.W. Kaeli, 2013
the notion of “scale” is synonymous with blurring
Convolution can be thought as locally re-distributing
a signal based on some “kernel” or distribution
Place kernel 𝐾 𝑥 wherever 𝑆 𝑥 is, weight by 𝑆 𝑥 ,
them sum resulting functions
∗ = 𝑆(𝑥) ∗ 𝐾(𝑥) 𝐾(𝑥) 𝑆(𝑥)
35 PhD Thesis Defense - J.W. Kaeli, 2013
the notion of a “scale-space” is a family of images
parameterized by a one-dimensional scale factor [Koenderink 1984] [Babaud et al., 1986]
continuous scale-space [Lindeberg 1994]
discrete pyramids [Burt & Adelson, 1983]
36 PhD Thesis Defense - J.W. Kaeli, 2013
hierarchical discrete correlation (HDC) [Burt 1981]
37 PhD Thesis Defense - J.W. Kaeli, 2013
octagonal pyramid
38 PhD Thesis Defense - J.W. Kaeli, 2013
comparison of effective kernel distributions
𝜎 =1
2 𝜎 = 1 𝜎 = 2 𝜎 = 2 𝜎 = 8 𝜎 = 128
39 PhD Thesis Defense - J.W. Kaeli, 2013
comparison of effective kernel distributions
40 PhD Thesis Defense - J.W. Kaeli, 2013
octagonal pyramid decomposition
computed in 3P adds
and P bit shifts
traditional pyramids
with 5x5 separable
kernels use 3.3P
multiplies, 2.7 adds
for same 2 scale
resolution, 8 times
more operations
41 PhD Thesis Defense - J.W. Kaeli, 2013
invariant features for underwater images
42 PhD Thesis Defense - J.W. Kaeli, 2013
invariant features for underwater images
43 PhD Thesis Defense - J.W. Kaeli, 2013
invariant features for underwater images
44 PhD Thesis Defense - J.W. Kaeli, 2013
what is the smallest latency with which we can
begin to “understand” the survey environment?
RAW IMAGERY
PRE-PROCESSING
SELECTION
COMPRESSION
TELEMETRY
UNDERSTANDING
{ [Murphy 2012,
PhD Thesis]
Chap. 1,2
Chap. 2,3
Chap. 3
45 PhD Thesis Defense - J.W. Kaeli, 2013
2. Computational Strategies
summary, conclusions, future work
introduce novel multi-scale image processing
framework for efficient feature computations
log opponent colors for illumination invariance,
partial correction of attenuation
future work in spatially-varying white balance
[Hsu et al., 2008]
46 PhD Thesis Defense - J.W. Kaeli, 2013
1. Image Correction
2. Computational Strategies
3. Understanding Underwater
Optical Image Datasets
outline of thesis chapters and contributions
47 PhD Thesis Defense - J.W. Kaeli, 2013
create an image “fingerprint” based on histograms
of quantized keypoint descriptors [Sivic & Zisserman 2006]
48 PhD Thesis Defense - J.W. Kaeli, 2013
create an image “fingerprint” based on histograms
of quantized keypoint descriptors [Lowe 2004]
49 PhD Thesis Defense - J.W. Kaeli, 2013
keypoint detection using extrema of the difference
of Gaussian function [Lowe 2004]
50 PhD Thesis Defense - J.W. Kaeli, 2013
keypoint detection using extrema of the difference
of Gaussian function
51 PhD Thesis Defense - J.W. Kaeli, 2013
both methods detect similar keypoints
52 PhD Thesis Defense - J.W. Kaeli, 2013
Quantized Accumulated Histogram of Oriented
Gradients (QuAHOG) around each keypoint
53 PhD Thesis Defense - J.W. Kaeli, 2013
possible QuAHOG patterns
[Ojala et al., 2002]
54 PhD Thesis Defense - J.W. Kaeli, 2013
keypoints detected and described with QuAHOGs
55 PhD Thesis Defense - J.W. Kaeli, 2013
what is the smallest latency with which we can
begin to “understand” the survey environment?
RAW IMAGERY
PRE-PROCESSING
SELECTION
COMPRESSION
TELEMETRY
UNDERSTANDING
{ [Murphy 2012,
PhD Thesis]
Chap. 1,2
Chap. 2,3
Chap. 3
56 PhD Thesis Defense - J.W. Kaeli, 2013
surprise-based online navigation summaries
[Girdhard & Dudek, 2010, 2012]
[Itti & Baldi,2005]
Kullback-Leibler divergence
57 PhD Thesis Defense - J.W. Kaeli, 2013
the threshold for “surprise” grows as the vehicle
experiences more of the world
58 PhD Thesis Defense - J.W. Kaeli, 2013
modified navigation summaries for semantic maps
wait for surprise threshold to stabilize before
selecting first image to transmit (most represented)
once transmitted, a summary image should become
a static member of the summary set
non-summary images are attributed a summary
image to represent them
never remove summary images, merge with nearest
summary image, represent with more represented
59 PhD Thesis Defense - J.W. Kaeli, 2013
selecting which summary image to transmit
semantic mapping via cumulative image telemetry
60 PhD Thesis Defense - J.W. Kaeli, 2013
data collected by SeaBED AUV
in 2003 off Stellwagen Bank
can transmit approximately once
every 300 images
summary size set to 16,
approximately twice as many
images as will be sent
61 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping via cumulative image telemetry
62 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping via cumulative image telemetry
63 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping via cumulative image telemetry
64 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping via cumulative image telemetry
65 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping via cumulative image telemetry
66 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping via cumulative image telemetry
67 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping via cumulative image telemetry
68 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping with heuristic merging
69 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping with heuristic merging
70 PhD Thesis Defense - J.W. Kaeli, 2013
semantic mapping with heuristic merging
71 PhD Thesis Defense - J.W. Kaeli, 2013
post-mission analysis of entire summary set
72 PhD Thesis Defense - J.W. Kaeli, 2013
post-mission analysis of class membership
73 PhD Thesis Defense - J.W. Kaeli, 2013
post-mission analysis of class membership
74 PhD Thesis Defense - J.W. Kaeli, 2013
post-mission analysis of class membership
75 PhD Thesis Defense - J.W. Kaeli, 2013
post-mission analysis of class membership
76 PhD Thesis Defense - J.W. Kaeli, 2013
post-mission analysis of class membership
77 PhD Thesis Defense - J.W. Kaeli, 2013
what is the smallest latency with which we can
begin to “understand” the survey environment?
RAW IMAGERY
PRE-PROCESSING
SELECTION
COMPRESSION
TELEMETRY
UNDERSTANDING
{ [Murphy 2012,
PhD Thesis]
Chap. 1,2
Chap. 2,3
Chap. 3
78 PhD Thesis Defense - J.W. Kaeli, 2013
nDeployments ≥ nRecoveries… LOU ∞
0
2
4
6
8
10
12
14
16
18
20
Deployments
Recoveries
[Kimball 2013]
79 PhD Thesis Defense - J.W. Kaeli, 2013
3. Understanding Underwater
Optical Image Datasets
summary, conclusions, future work
propose lightweight keypoint detection and
description scheme for use in scene classification
modify surprise-based navigation summaries for
selecting images to transmit acoustically
demonstrate creation of low-bandwidth semantic
maps via cumulative image telemetry
80 PhD Thesis Defense - J.W. Kaeli, 2013
summary, conclusions, future work
we have proposed a realistic framework for
reducing the latency of understanding paradigm
in autonomous underwater robotics
future implementations in real AUV missions
more robust keypoint detectors/descriptors
“AUV Pandora” for adaptive mission planning
81 PhD Thesis Defense - J.W. Kaeli, 2013
acknowledgements
82 PhD Thesis Defense - J.W. Kaeli, 2013
questions?