being there: capturing and experiencing a sense of place richard szeliski microsoft research...

49
Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Upload: audra-dawson

Post on 11-Jan-2016

217 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Being There:Capturing and Experiencing a

Sense of Place

Richard Szeliski

Microsoft Research

Symposium on ComputationalPhotography and Video

Page 2: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Early art: events

Lascaux

Page 3: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Early art: events

Page 4: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Early art: events

Bayeux Tapestries

Page 5: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Renaissance art: events

Da Vinci

Page 6: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Baroque art: portraits

Rembrandt

Page 7: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Baroque art: landscapes

Rembrandt

Page 8: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Classical art: landscapes

Corot

Page 9: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Impressionism: landscapes

Monet

Page 10: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Modern art: multi-perspective

Picasso

Page 11: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Impressionism: landscapes

Monet

Page 12: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Photography: events

Henri Cartier-Bresson

Page 13: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Photography: landscapes

Ansel Adams

Page 14: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Photography: landscapes

Galen Rowell

Page 15: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Photography: landscapes

Art Wolfe

Page 16: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Interactive Visual Media

• Branching storytelling• Games:– problem solving, skill acquisition

– sense of place, navigation

Page 17: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Real-world “immersive” media

• Recapturing the sense of place

• QTVR: the first real breakthrough in modeling from reality?

• Fairly universal for high-end home and car sales, virtual tourism

• Limitations: mostly static, discrete jumps

Page 18: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Increasing realism

• Better:– field of view and resolution

Demo: Space Needle– dynamic range

• Additional cues/modalities:– motion / movement

Demo: Deception Falls– sound

Page 19: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

VR Tools

Page 20: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Fully automated stitch discovery

Demo

Page 21: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Image stitching — open issues

• Fully automated assembly (no ordering)

• Full 2D stitching (multiple rows)

• Double image fix-up (de-ghosting)

• Merging different exposures

• Automated grouping/clustering (detection)

Page 22: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

HDR: merging exposures

Inputs Tonemapped output:no motion compensation

or consistency check

Page 23: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Inputs Tonemapped output:global+local motion

compensation

HDR: merging exposures

Page 24: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Video Textures

• How can we turn a short video clip

into continuous video?

– Enhance sense of liveness

– Use in games and presentations

Page 25: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Video Textures

1. Find cyclic structure in the video

2. Play frames with random shuffle

3. Optional region-based analysis

4. Smooth over discontinuities (morph)

Page 26: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Animating Stills

• What if we only have a single beautiful photo (or painting)?

• Can we add so liveness to a photo/still?

Page 27: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Animating Stills

Page 28: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Increasing “immersion”

• Add continuous movement

Page 29: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

DemoBellevue Botanical Garden

Page 30: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

DemoHDR Home Walkthrough

Page 31: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Video-based walkthroughs

• Add continuous movement

• How did we do this?

• Is this 3D? 2D?Graphics? Video?

• Dimensionality and storage?

Page 32: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Limits of Video-Based Walkthroughs

• Video-based walkthroughs are “rail experiences” of a space-time slice

• How do we break the bounds?– Can’t move from side to side– If there is motion in the scene, don’t see it

• Capturing all points of view everywhere in space at all times is impractical

Page 33: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

A Practical 3D Video Camera

• Working volume?– Walls of a room: Virtualized Reality– 2D “window”: LightField Array– 1D “rail”: Virtual Viewpoint Video

Page 34: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

“3D” video

Page 35: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Virtual Viewpoint Video

Capture multiple synchronized video streams

Page 36: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Key to view interpolation: Geometry

Stereo

Camera 1 Camera 2

Image 1

Image 2

Virtual Camera

Page 37: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Don’t match pixels – match segments

• Segments contain more information, so they’re easier to match.

Page 38: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Iteratively update depths

Page 39: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Depth through time:

Page 40: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Matting

Some pixels get influence for multiple surfaces.

Background Surface

Foreground Surface

Image

Camera

Close up of real image:

Multiple colors and depths at boundary pixels…

Page 41: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Find matting information:1. Find boundary

strips using depth.

2. Within boundary strips compute the colors and depths of the foreground and background object.

Background

ForegroundStrip

Width

Page 42: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Why matting is important

MattingNo Matting

Page 43: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Representation

Color Depth

MAIN LAYER

Color Depth

Matting informationBOUNDARY LAYER

Page 44: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Rendering pipeline (GPU)

RenderMain Layer

RenderBoundary Layer

Composite

RenderMain Layer

RenderBoundary Layer

Page 45: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Real-time viewpoint control

“Matrix”-like effects – on demand!

(With much, much less hardware.)

Page 46: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Wrapping up…

Page 47: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Slow Glass

• What if we could captureall the photons and play them all backat a later time?

Light of Other DaysBob Shaw, © 1969http://www.scifi.com/scifiction/classics/classics_archive/shaw/shaw1.htm

Page 48: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Slow Glass

One could stand the glass beside, say, a woodland lake until the scene emerged, perhaps a year later. If the glass was then removed and installed in a dismal city flat, the flat would—for that year—appear to overlook the woodland lake. During the year it wouldn't be merely a very realistic but still picture—the water would ripple in sunlight, silent animals would come to drink, birds would cross the sky, night would follow day, season would follow season.

Apart from its stupendous novelty value, the commercial success of slow glass was founded on the fact that having a scenedow was the exact emotional equivalent of owning land. The meanest cave dweller could look out on misty parks—and who was to say they weren't his? A man who really owns tailored gardens and estates doesn't spend his time proving his ownership by crawling on his ground, feeling, smelling, tasting it. All he receives from the land are light patterns…

Page 49: Being There: Capturing and Experiencing a Sense of Place Richard Szeliski Microsoft Research Symposium on Computational Photography and Video

Being There

• Artists (and all of us) have always wanted to capture a sense of being there.

• Computational photography and video bring us a lot closer:– realism (field of view, resolution, contrast)– movement and sound– immersion and exploration

• What does the future hold?