pome a mobile camera system for accurate indoor...

© 2010. All rights reserved.

Paul Montgomery & Andreas Winter

November 2 2016

1

POME A mobile camera system for accurate

indoor pose


ICT – Intelligent Construction Tools

• A 50-50 joint venture between Trimble and Hilti

• Vision: revolutionize the way construction is done

– Many inefficiencies persist on the construction site

• The central technical problem - Accurate and robust indoor positioning

11/7/2016 2


Overview

1. ICT and the construction market

2. A brief history of indoor positioning

3. POME concept, theory of operation, system tradeoffs

4. POME error budget

5. System components

6. The usual problems

7. POME accuracy

8. The future

11/7/2016 3

static kinematic


Construction Site

11/7/2016 4


Construction tools today

• A precision instrument (not a tool)

• 5 arc sec sensor

– ~2.5mm at 100 m (Horizontal, Vertical)

• $30-60K (expensive)

• Requires experience to set up and use

– careful installation on a stable tripod

– correct referencing

• Single user

• Subject to line of sight occlusion

• Issues in tracking at close range

• Most sites still use traditional tools for most tasks

11/7/2016 5


Requirements • Accuracy (< 6mm)

• Robustness

– to occlusion

– to drop

– dust/dirt

• Cost

– BOM <= $400

• Ease of installation

– No cables

– Fast and reliable installation

• Room size > 30 meters

• Challenges

– rapidly changing environment

– variable lighting conditions

– many reflective surfaces

11/7/2016 6


Existing Solutions

11/7/2016 7

Hawk-Eye, which costs $100,000 and can

pinpoint a ball to within 5 millimeters.


Why POME?

• POME = Position & Orientation Measurement Engine

• Multi User

• Indoor (+ possibly outdoor)

• Low cost & fully solid state

• POME inside == GPS outside

– Similar weight & volume

– Similar cost

– Similar update rate

– Similar accuracy

• Like GPS, enables many applications, e.g.:

– Staked layout

– Projection systems

– Robotic systems

– Augmented Reality

11/7/2016 8

==

POME

inside

GPS

outside


POME Applications

11/7/2016 9


Principle of Operation

• Measure angles between known points

• Use redundant measurements

• Use least squares to solve a set of nonlinear equations

• Q: How accurately can you measure angles with a (wide angle) camera?

11/7/2016

10

Θ

Θ

Θ

A

B

Θ

Θ

Θ − 𝜹Θ

A

B

A Simplified 2D example


2D example: intersection of 2 circles

• Nonlinear problem

• Intersection of 2 circles gives candidate solutions X & Y

• Positions of A,B,C,D must be known

• Uncertainty in angle measurements results in a covariance ellipse

11/7/2016 11

Θ𝑨𝑩

D

B

C

A

Θ𝑪𝑫 X

Y


Error Budget

• require 0.2 pixel 1 sigma with multiple cameras

• 1 pixel = 2.2 um

• Error contributors:

– Achievable calibration accuracy

· Lenses

· mechanics

– Mechanical stability

– Centroid determination with:

· saturated signals

· weak signals

– Number and geometry of targets

– Accuracy of target survey

11/7/2016 12


System Design Considerations • Number of cameras

• Arrangement of cameras for best F.O.V.

• Type of image sensors

– Number / size of pixels

– Rolling / global shutter

– Color / monochrome

– Dynamic range of pixels

• Type of lens

– projection function

– Image sensor matching

• Type of target LED’s

– Visible / I.R., power, pattern

• Image processing considerations

– Image processing bandwidth

– Power

• Cost !!

11/7/2016 13

Early concept for 3 camera

overlapping F.O.V.


Active Targets

• Transmit at 850 nm (near IR)

• Approximately 350 mW

• Modulated intensity

11/7/2016 14


Projection from object space to image space

• Camera: a projection from object space to image space (x,y,z) -> (u,v)

• 1 to 1 mapping of rays (unit vectors) to image space points (u,v)

• Point of light becomes a “blob”

• a non-trivial mapping

• For pose calculation we need to convert from image space points to rays (angles)

• Mapping function is different for every camera => need to calibrate

11/7/2016

O

X

Y

optical axis

lens

Image sensor

𝜽

(u,v)

𝝓

Z

(x,y,z)

object

space

image

space

(x/z,y/z,1)

15


Fisheye (f-theta) lens projection

• For large F.O.V. pinhole camera needs a very large image sensor

• F-theta projection => equal angle increment maps to equal number of pixels

• Camera is an angle measuring sensor

11/7/2016 16

X

Y

optical axis

optical center

R Image sensor

𝜽

𝜽

𝑹 = 𝒇 ∗ 𝒕𝒂𝒏𝜽

f

X

Y

optical axis

optical center

R Image sensor

𝜽

𝑹 = 𝒇 ∗ 𝜽

f

pinhole projection f-theta projection


F-theta lens

• equal angle is mapped to equal distance on image sensor

• Camera measures angles

• Our cameras have ~14 pixels/degree

• => 1 pix = 250 arcsec

• => 0.2 pix = 50 arcsec

• ~ 5 mm @ 20 meters

11/7/2016 17

160 degree

Image circle

image sensor


Example blobs using off the shelf lenses

• Non symmetry of impulse response across the F.O.V

– Strongly affects centroid determination accuracy

• Non uniformity of energy distribution across the F.O.V.

– compounds near/far problem

11/7/2016 18


Custom Optics

• DSL627 optimized lens

11/7/2016 19


11/7/2016 20

30 deg. elev. circle

0 deg. elev. circle

-30 deg. elev. circle

F.O.V. boundary


Some “not very interesting” images

11/7/2016 21


Example blob

11/7/2016 22


Saturated blob

11/7/2016 23


Calibration residual

• After removing an f-theta model, we are left with residual

• Calibrate by fitting a function to the residual

• Blob centroid (u,v) -> -> -> angles -> pose solution

• Lens and camera mechanics must be stable over time and temperature

11/7/2016 24

inverse residual

function inverse f-theta


Shock and Vibration

11/7/2016 25

• Significant testing has been done to POME head to verify stability

– Below are test set up and results from shock and vibration testing with positive results

11/7/2016 25


Lens stability testing results

11/7/2016 26 11/7/2016 26


The Usual Problems

• Calibration + mechanical / thermal stability

• range ratio (near / far problem)

• Registration (target determination)

• Interference rejection (strong signals)

• Multipath rejection (with and without direct ray)

• Initialization

• Data rate (~3000 Mb/s) / image processing / power

• Solution (target modulation, synchronization)

11/7/2016 27


Accuracy Testing -- Warehouse

11/7/2016 28

PLT for truth validation

Indoor warehouse location with industrial and

natural lighting ~ 15 m x 10 m x 8m


Accuracy Testing

11/7/2016 29


Test results for unit 0x8004

11/7/2016 30

• 48 position stations

• 16 azimuth stations at each position station

• In table on following page

– Each row is one position station

– Each column is one azimuth station

• Numbers show the position error relative to the truth system (PLT) in units of mm

• Each number represents a static mode result

Target locations are shown with black dots and numeric ID

Robot trajectory is shown with blue ‘x’

PLT location is shown with a red dot

Position Station TP07 location shown with

Warehouse dimensions are in units of meters


11/7/2016 31


Why is it difficult to state performance?

• Performance is characterized by error statistics

– To have significance, statistics need many measurements to validate

• We have 6 errors to characterize at each point in space

– 3 components of position error

– 3 components of orientation error

• Errors are worse in some directions than in other directions

• There are a variable number of targets and variable working volume geometry

• There are different modes of operation (here, we document static and survey modes)

• We plot the worst direction 1 sigma errors in a square working volume

11/7/2016 32


Example of scatter plot ellipsoids • A scatter plot of results creates a clump of data points distributed around the truth value

• The statistics of the clump can be characterized by an error ellipsoid

• Shown below are example 1 sigma ellipsoids for position and orientation

11/7/2016 33

Position Error Ellipsoid Orientation Error Ellipsoid


Simulation Target Configurations • Nominal ‘square’ room of size 10x10

meters

• Consider 6 different target arrangements in room

• Targets installed at uniform height and positioned on walls around circumference of room

• Calculate position and orientation accuracy at grid points in the room

• Calculate solutions with:

– 1 azimuth station (static mode)

– 4 azimuth stations (survey mode)

• All simulations use 0.5 pixel 1 sigma, but we expect/hope to achieve 0.2 pixel 1 sigma in practice!!

11/7/2016 34


10x10 m room, 8 targets, 1 azim. station

• 0.5 pix 1 sigma

• 1 azimuth station

• 10x10 meter room

• Worst direction position

• Worst direction orientation

• 1 sigma results

11/7/2016 35


10x10 m room, 8 targets, 4 azim. stations

• 0.5 pix 1 sigma

• 4 azimuth stations

• 10x10 meter room

• Worst direction position

• Worst direction orientation

• 1 sigma results

11/7/2016 36


The future

• “In the words of Yogi Berra, ‘I never make predictions, especially about the future,'”

• Image sensors and image processing continue to develop quickly

– reduced cost

– sophisticated image processing of real images

– lenses and stability will remain challenges to accuracy

• Mobile cameras and lightweight infrastructure (scalability, infrastructure)

• Step 1: Reduce the number of required active targets, use natural features

• Step 2: sensor fusion with inertial, ranging camera, stereo camera, …

11/7/2016 37

pome a mobile camera system for accurate indoor...

Documents