On-Demand Sharing of a High-Resolution Panorama Video from Networked Robotic

Cameras

Supported in part by

CPSC 643

Dezhen Song

Texas A&M University

Vide framesSphere

wrappingImage

alignment

1

i1

im N

Server side

IBBBBBBIBBBBBBIBBBBBB1

IBBB

i

1

IBBBBBB IBBBBBBN

……

Client i

IBBBBBBIBBBBBBIBBBBBB……

i

m

Client iClient iClient i

Decoding and composing panorama

Rendering, Display

User request [area i, at time t1~tk]

… …

Live video

Time k

BBB

2

Panosonic HCM 280

– PTZ Robotic Camera: • 350° Pan, 120° Tilt, 42x Zoom

• Maximum spatial resolution: 500 Megapixel per

steradian

• 3 Gigapixels panorama

– Network Video Camera:• Built-in streaming server

• 640x480 pixels video

• >30 frames per second

Network PTZ Robotic Camera for Nature Observation

3

4

5

Giga-pixel Motion Panorama VS. Fixed Lens Camera

• Fixed lens with mirror• 10M Pixel CCD• $ 20.0 K• 2M Pixel / Steradian

• Pan, Tilt, Zoom (21x)• 0.37M Pixel CCD• $ 1.2 K• 500M Pixel / Steradian

6

Existing Panoramic Video Systems

SystemCamer

a

Bandwidt

hVideo Output

Sample System

s

Wide angle lens/mirrors

Single fixed

Low Low quality live stream

[Baker 1999], [Nayar 1997], [Xiong

1997], [Ng 2005]

Multiple camera

panorama video

Multiple fixed

High Live panoramic video[Foote2001], [Swaminath

an 2000]

Panoramic video texture

Single pan

HighPseudo-live panorama

video by changing video temporal display

[Agarwala 2005]

Dynamosaics Single pan

HighPseudo-live panorama

video by changing space-time volume

[Rav2005]

Motion panorama

Single LowStatic panorama overlaid with living moving objects

trajectory

[Irani 1996], [Bartoli 2004]

Our systemPTZ

Cameras

Low Partial live panorama

7

Evolving Panorama: High Resolution Live Panoramic Video Using PTZ

Camera

Tilt

Pan

Frame sequence

Panorama

Tilt

Time

Panorama

Live frame sequence

Updated Part in

Panorama

8

robotic video cameras

Collaborative Observatories for Natural Environments (www.c-o-n-e.org)

motion sensors

timed checks

sensor networks

humans: amateurs and profs.

2005-2008

9

On-demand Panoramic Video

SharingVide frames

Sphere wrapping

Image alignment

1

i1

im N

Server side

I B B B B B B I B B B B B B I B B B B B B1

I B B B

i1

I B B B B B B I B B B B B BN

……

Client i

I B B B B B B I B B B B B B I B B B B B B

……

im

Client iClient iClient i

Decoding and composing panorama

Rendering and Display

User request [area i, at time t1~tk]

… …

Live video

Time k

B B B

10

Challenges:

– Dynamic video coverage

– High resolution panorama coverage

– Multiple different spatial-temporal client requests.

On-demand Panoramic Video

Sharing

11

User Request

Live Live

Timek-1 k

… ……

Live video

…

Client i

User i request: ri=[u, v, w, h, ts, te]

12

Camera Coverage

Camera Coverage

pan-180o -180o

60o

tilt

N

1 pjk

Patch-based Panorama Video

Snapshot at time k

Live patch Static patch

Camera Coverage

13

Patch-based Panorama Video

Live video

Live

Timek-1 k

… ……k-2

…

Camera coverage at time kPatch j at time k

14

On-demand Patch-based Panorama Video Sharing

Vide frames

Sphere wrapping

Image alignment

1

i1

im N

Server side

I B B B B B B I B B B B B B I B B B B B B1

I B B B

i1

I B B B B B B I B B B B B BN

……

Client i

I B B B B B B I B B B B B B I B B B B B B

……

im

Client iClient iClient i

Decoding and composing panorama

Rendering and Display

User request [area i, at time t1~tk]

… …

Live video

Time k

B B B

15

Frame Insertion Algorithm

Input: Ft

Output: Updated evolving panorama video

Wrap Ft onto the spherical surface;

Estimate Ft’s registration parameters by aligning it with previous frames;

Project Ft onto the sphere panorama surface;

for each pj and pj ∩ Ft ≠ Ø do

Insert pjt into pj’s GOP buffer;

for each pj, j=1, …,N do

if pj’s GOP buffer is full then

Encode patch video segment;

Store patch segment start position and time data into lookup table;

Reset GOP buffer for incoming data;

16

On-demand Patch-based Panorama Video Sharing

For User i request:

Send patch data:

ri=[u, v, w, h, ts, te]

ri ∩ Pt = { pjk | j Є {1,…,N}, k Є [ts, te], pjk ∩ ri ≠Ø , pjk ≠Ø }

17

User Query Algorithm

Input: ri

Output: ri ∩ P in MPEG-2 format

Identify patch set S= { pj | j Є { 1,…,N }, pj ∩ ri ≠Ø };

for each pj Є S do

Find the nearest I frame pjb earlier or equal to ts;

Find the nearest I frame pjc later or equal to te;

Transmit the patch segments between pjb and pjc;

18

Experiments and Results

Hardware configuration:

•Dell Dimension DX, 3.2Ghz Pentium dual-core processor, 2GB RAM

•Panasonic HCM 280A video camera

Software configuration:

•Visual C++ in Microsoft Visual Studio 2003 .NET

•MPEG-2 encoder/decoder from MPEG Software Simulation Group

Input data set: •Frame number: 609

•Frame resolution: 640x480 pixels

•Frame rate: 25 fps

•Raw RGB data size; 536 MB

•Panorama resolution: 2742x909 pixels

19

Experiments and Results

Storage and computation speed versus different patch sizes:

20

Experiments and Results

Bandwidth for a user query (800x600 pixel) versus different patch sizes:

21

Summary

Patch-based data representation and encoder provides on-demand sharing of a high resolution panoramic video from networked robotic Pan-Tilt-Zoom cameras with:

• Effective data organization

• Efficient data storage.

• Satisfy spatial-temporal user video requests.