IIIT

Hyd

erab

ad

Geometry Directed Browser for Personal Photographs

Center for Visual Information Technology

IIIT Hyderabad

Aditya Deshpande, Siddharth Choudhary, P J Narayanan, Kaustav Kundu, Krishna Kumar Singh, Aditya Singh, Apurva Kumar

IIIT

Hyd

erab

ad

We use SfM and other 3D computer vision techniques to provide intuitive Geometry Directed Photo Browsing.

Photo-Browsing

• Digital Photography - No hard copy - Capture photographs and relive later on display device

• Photo-Browsers are tools to view digital photographs. E.g. Windows Photo Viewer, iPhoto, FSpot, KSquirrel etc.

•Photo Browsing model has notevolved much.

IIIT

Hyd

erab

ad

Related Work

• Face Detection & Tagging on Social Networking Sites.

• [Zhang et al. MM’03], Automatic annotation of family albums.

• [Davis et al. MM’05], Additional contextual data viz. time of capture, geo-tag, indoor/outdoor scene, co-occurring faces.

Above techniques only improve photo-browsing experience of social engagements.

IIIT

Hyd

erab

ad

Our Goal

• Apart from social engagements, a large chunk of user’s personal photographs consist of tourist places & monuments.

• [Snavely et al. IJCV’08, SIGGRAPH’06] (Photosynth) - CPC Storage, local reconstruction to add new camera’s

• Choudhary et al., Li et al., Sattler et al., Irschara et al. etc. - Localize new query images w/o exhaustive search.

We combine SfM-Reconstruction + Localization to provide intuitive browsing of user photos in 3D space of the monument.

IIIT

Hyd

erab

ad

Assumptions

• Our target platform is an off-the-shelf laptop or a desktop.

• User is expected to click around 5-50 photographs for a particular monument.

• The system should localize these user photographs in areasonable time.

• The system should provide a smooth visualization / transitions of all user photos and ~105 points of the monument.

IIIT

Hyd

erab

ad

System Design(1) Heavy SfM Reconstruction done offline in the cloud

(2) GDBPackage : reconstruction + addnl. information downloaded to local disk

(3) User uploads personal photo’s through a camera / phone

(4) System registers user’sphotos to the point cloud andprovides 3D visualization.

IIIT

Hyd

erab

ad

System Block Diagram

GDBPackage

User Photos

RegistrationModule

VisualizationModule

System is divided in two parts : 1. Registration / Localization Module2. Visualization Module

2 1

EstimatedCamera’s

IIIT

Hyd

erab

ad

Localizing User Photo’s

• Trivial if photograph is taken from GPS enabled deviceand is geo-tagged!

• What if no geo-tag information?

• Two Localization Approaches :Image based search in a geo-tagged Image Dataset[Panda et al.] Geo-locate digital heritage site photos.

Using structure information in SfM Dataset[Irschara et al. CVPR’09], match to nearby similar images.[Li et al. ECCV’10], visibility prioritized 3D-2D matches. [Sattler et al. ICCV’11, ECCV’12], visual words to find 2D-3D matches.

IIIT

Hyd

erab

ad

Localization - Choudhary et al.

• [Choudhary et al. ECCV’12]- Triangulate a seed point in the user photograph.- Further 3D-2D search is guided by visibility probabilities.- Find ~20 independent matches.- Use RANSAC to estimate camera parameters.

Probability Guided 3D-2D correspondence

3D Position

Up Vector

View Direction

IIIT

Hyd

erab

ad

Advantages of Localization Method

• Data for Localization is stored in GDBPackage : (1) Cover Set (2) Visibility Matrix (3) Bi-Partite Visibility Graph

• CPC images need not be stored, data requirements are minimal.

• The method is fast and localizes images at the rate of 1sec/photo.

IIIT

Hyd

erab

ad

Non-Localizable Photographs

• In some cases the images lack sufficient monument geometryfor localization to work :- Occluded by people.- Noisy images of nearby scenery/smaller monuments.- Zoomed in images of smaller monument structures etc.

Zoomed In View of Small Structure

(Pantheon Dataset)

Completely Occludedby People

(Colosseum Dataset)

IIIT

Hyd

erab

ad

Non-Localizable Photographs

• Photographs have time of capture stored in their EXIF-tags.

• A non-localized image is placed at a position that is weightedaverage of its immediate known predecessor and immediateknown successor in time.

• Similarly, linear interpolation is also done for the view-direction vector to get the complete camera pose.

• The above method will not give the exact location, but placing it in temporal neighborhood suffices for display purposes.

IIIT

Hyd

erab

ad

Visualization Module

• 3D Viewer

• Mouse Navigation

• Button Navigation

• Add Screenshot

• Delete Path

• Generate Photo-Tour

• 2D Viewer

IIIT

Hyd

erab

ad

3D Photo Browser : Geometry Directed Photo-Browsing

• Initial Mode : 3D Model and small preview (thumbnails) of user photographs.

• Select Mode : Animate to clicked photo and detailed view.

• Linear quaternion interpolation of Rotation Matrix for smoothtransitions between images.

• Smooth transitions give a feel of the geometric space of the monument.

IIIT

Hyd

erab

ad

3D Photo Browser : Generating Custom Photo Tours

• User can save the current viewpoint (“Add Screenshots”)

• Once a set of viewpoints are saved, he can smoothly animateover viewpoints. (“Generate Photo-Tour / Animate Path”)

• User can delete the viewpoints and generate a new photo-tour.

• Photo-Tours are a good way to creatively view personalphoto’s taken at a tourist place.

IIIT

Hyd

erab

ad

ResultsMonument # Photos # Registered

PhotosReg. Time

(secs per photo)Colosseum 24 21 1.01Colosseum 19 14 0.97Pantheon 35 22 1.13

Stone Chariot(Hampi)

17 17 1.07

(a) Localization Module

(b) Visualization Module

IIIT

Hyd

erab

ad

Conclusion and Future Work

• Minimal System Requirements.

• Intuitive 3D Visualization of User Photographs.

• Pipeline for 3D personal photo-viewing from SfM reconstruction.

• Port our system to a mobile phone and have a touch/gesture interface.

3D Photo-Viewing & Localization App

IIIT

Hyd

erab

ad

Thank you. Questions?More Results

(a) Hampi Dataset (Stone Chariot)

(b) Pantheon Dataset

IIIT

Hyd

erab

ad

Platform Details

Item Specification

CPU Intel ® CORE ™ i5

Clock Speed 2.44GHZ

RAM 4GB

GPU Intel ® HD Graphics Accelerator