CPSC 643

Aligning Windows of Live Video from an Imprecise Pan-Tilt-Zoom Robotic Camera

into a Remote Panoramic Display

Dezhen SongDepartment of

Computer Science and Engineering

Texas A&M University

Supported in part by

2

Network PTZ Robotic Camera for Nature Observation

Panosonic HCM 280

– PTZ Robotic Camera:

• 350° Pan, 120° Tilt, 42x Zoom

• 200° per second servo speed

– Network Video Camera:

• Built-in streaming server

• 640x480 pixels video

• >30 frames per second

– Low power consumption: <5

Watt

– Affordable price: $ 1.2 K

3

Real Time Panoramic Video

Tilt

Pan

Frame sequence

Panorama

Tilt

Time

Panorama

Live frame sequence

Updated Part in

Panorama

4

5

Related Work

– Multiple fixed cameras• [Swaminathan and Nayar

2000]

• [Tan et al. 2004]

• [Foote et al. 2000, 2001]

– Single wide angle camera• [Baker and Nayar 1999]

• [Nayar 1997]

• [Xing and Turkowski 1997]

6

Related Work: Image Alignment

– Direct Method• Use pixel intensity value

• Sensitive to luminance change

• Need good guess for initial parameters input

• Existing work– [Shum and Szeliski 1997] [Szeliski 1994, 1996]

– [Coorg and Teller 2000] [Kang and Weiss 1997]

– Frequency Domain Registration• Existing work

– [Castro and Morandi 1987] [Reddy and Chatterji 1996]

7

Related Work: Image Alignment

– Feature-based Image Registration• Use feature points: Harris corner point, SIFT

• Robust to luminance change

• Faster than direct method

• Existing work– [Torr and Zisserman 1997] [Brown and Lowe 2003]

– [Zoghlami et al. 1997] [Hu et al. 2001] [Cho et al. 2003]

– [Kanazawa and Kanatani 2002] [Zhang et al. 2002]

8

Comparison: Panoramic Video

System Resolution Bandwidth Live Motion Images

Our system Excellent Low Yes

Film-based panorama

Excellent Low No

Wide-angle systems

Poor Moderate Yes

Multi-cameras

Good Moderate to High

Yes

9

Assumptions

• Pan-tilt camera with a fixed base

• Known intrinsic camera parameters

– Calibrated camera before deployment

• Inaccurate pan-tilt readings

– May deteriorate over time

• Standard video camera with HFOV ≤ 45o

10

QKRI Wq

Review: Perspective Projection

Intrinsic

Parameters Extrinsic

Parameters

[Tsai86,

87]

11

Problem Definition

qtpMqRKKq BBB

BAB

A ),(1

Re-projection: Project image B onto image A plane:

Image alignment:

BAiBtBp

iA

AiB

BBB qqtpM2

Intensity,Intensitymin),(

12

Excessive Computation in Image Alignment

Speed slow down caused by coupling re-projection and SSD:

– Extensive float point computation

– Coupled with Sum of Squared Difference (SSD) operation

– A naive search takes O(km) re-projection operations

• k: number of candidate pan/tilt pairs over feasible solution

set.

• m: number of overlapping pixels

Proposed solution: decouple re-projection and SSD– Spherical re-projection – Cell-based Alignment

• Constant time alignment

13

• Project image onto a spherical surface• Image =(p, t) on local spherical coordinate system { }

Spherical Projection

22arctan

arctan

fu

vt

f

up

CY

CX

tp

u

v

)(~ ),,(

p,tq

zyxQC

),( vuq

OImage plane

f

CZ

I~

I~

14

0.8

0.4

-0.4

-0.8-1.5 -0.5-1 0 0.5

p

t 0

-400

-200

0

200

400

v

200u -200-600-1000

Planar Spherical

Two poses have 30o pan difference with the same 30o tilt value

Distortion under Re-projection

15

0.8

0.4

-0.4

-0.8-1.5 -0.5-1 0 0.5

p

t 0

Invariant under Spherical Re-projection

16

Re-projection after Spherical Projection

Define conversion between camera coordinate system and local spherical coordinate system

Re-projection function between two local spherical coordinate system

qP

pctcf

tsf

pstcf

z

y

x

Q ~1

QP

zxy

zx

t

pq

22arctan

arctan~

qRFqRPPQRPq BAB

BAB

BAB

A ~,~~ 1

17

• Rc is a 2x2 rotation matrix

• Cell distortion under re-projection is negligible.

Lemma 1: If the spherical cell is small , define

point

and its corresponding point

)5 and 5( cc tp

Cqq BBo

B ~ Aqq Ao

A ~~

qRq BC

A ~~

qqqqRF Ao

ABo

BAB

~~ ~~,

we have

18

Proof for Lemma 1:

qP

pctcf

tsf

pstcf

z

y

x

Q ~1

df

dt

dp

pctcpctfspstfc

tstfc

pstcpstfspctfc

Q 0

Introduce coefficient matrix HRadius f remains

same

f

f

f

H

00

00

00

0

0

33

23

13

p

p

mpctspstc

mtc

mpstspctc

H

z

y

x

19

Continue: Proof for Lemma 1

HFOV≤45o and VFOV ≤34o

0.956 ≤ cos(t) ≤ 1

Dropping cos(t) introduces ≤ 5% distortion for 20x20 cell

∆f=0 substitute [m13, m23, m33]T

0

0

33

23

13

p

p

mtspcps

mtc

mtspspc

H

z

y

x

tRpR

tcpctspcps

tstc

tcpstspspc

XY

0

20

Continue: Proof for Lemma 1

0

~

0

~ qR

q BA

oB

XoB

YABo

AYo

AX tRpRRpRtRR

10

0

21

12cRR

qtRpHRQ XY~ QRQ BA

BA

21

Lemma 2: Rotation angle Θ of Rc can be approximated by

where α is the dot product of Z axis of {CA} and {CB}

BoB

oA

AB

AoB

oA

AB

oB

oA

ABoB

oA

tcpspcpps

tcpcpspps

pspsppcpcpc

arccos

BAABBA tstsppctctc

22

Algorithm

Cj ε(Cj )

A~

B~

ojB q~

2

1

IntensityIntensitymin),(

c

BtBp

k

jjAjcB CCR

23

Cell based Image Alignment

Select kc cells from the overlapping region in O(1)

Sphere projection O(1)

Feature detection in the cell and searching regions O(1)

For each(δpB, δtB) O(1)

For each cell O(1)

Compute Cj

Compute , j=1, …, kc

Compute SSD between and

End For

Report sum of SSD across all cells

End For

Output solution with the minimum SSD

BCR jC

~

BCR jC

~1 AC j

~

24

Experiment and Results

Speed test:

– 881 milliseconds to align 21 320x240 images

– 4 seconds for Autostitch program on same data set

– Up to 25fps on a laptop PC

25

26

• Align video images from a camera that only differ in pan and tilt settings into a panorama at 25 frames per second.

• Alignment is performed on a spherical surface to avoid excessive distortion caused by homographic transformation.

• A constant time algorithm pre-rotates small pre-sampled squared patches on spherical surface for matching.

• Experiments show that the alignment speed is 4.5x faster than the best method available.

live video window

Thank You!