interactive 3d modeling of indoor environments with a consumer depth camera

Interactive 3D Modeling of Indoor Environments with

a Consumer Depth Camera

Hao Du3, Peter Henry1, Xiaofeng Ren2, Marvin Cheng1,Dan B Goldman4, Steven M. Seitz1, Dieter Fox1,2

1University of Washington: Computer Science & Engineering2Intel Labs Seattle (now ISTC at UW)

3Google4Adobe Systems

Outline1. Introduce RGB-D (Red, Green, Blue + Depth)

mapping with a Kinect camera2. Show how an interactive mapping system

improves usability and results3. Demonstrate potential applications to

localization, measurement, and interactive visualization

The Kinect

Guinness record: “fastest selling consumer device”

$150, more than 10 million sold as of March 2011

OpenNI and Kinect SDK allow for use by developers

RGB-D CameraRed Green Blue

Depth

[PrimeSense.com]

RGB-D Data

Interactive Mapping

RGB-D Frames

Local Alignment (RANSAC) Global Alignment

Loop ClosureDetection

Update Model

RGB-D Mapping*RGB-D frames are recordedThe sequence is then processed to obtain the

map

*RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments. Henry et al. International Symposium on Experimental Robotics (ISER), 2010.

Frame Matching with RANSAC(Random Sample Consensus)

Visual features (from image) in 3D (from depth)Figure out how the camera moved by matching

these features to the previous frame

Limitations of PreviousRGB-D Mapping

Limited to offline processing after recordingEven a small number of missing alignments

result in a uselessly disconnected mapExpert knowledge of the system is required to

get good resultsNo feedback on coverage of the environment

Interactive 3D Mapping

RGB-D Frames

Local Alignment (RANSAC) Global Alignment

Loop ClosureDetection

Update Model

RGB-D Frames

Failure Detection

Local Alignment (RANSAC) Global Alignment

Loop ClosureDetection

CompletenessAssessment

User Moves and Resumes Mapping

User Verifies and Adds Loop Closures

User Acquires Additional Frames

Visualization

User Controls Viewpoint

Update Model

Suggest Places to Visit

InteractiveRGB-D Mapping

User Interface

Examining Model Completeness

Completeness Hints

Interactive System Evaluation

4 users (2 novice, 2 experienced) were instructed to map three walls of a room both with and without the interactive system

All experienced failures without the interactive system (after a mean count of 40.75 frames)

The proposed interactive system caught 26 potential failures allowing all users to interactively go back and create accurate maps

Non-interactive

Interactive

Final Position Error (m)

1.13 0.08

Benefits of Interactive 3D Mapping

RobustnessReal-time feedback of mapping status and failuresRewind / Recover / ResumeDelete frames (if a person walks in front of you,

etc.)Remove false-positive loop closuresPropose additional loop closures

CompletenessSee the map as it is builtShow unexplored areas

Large Scale Map

Espresso Room Model

Applications1. Localization2. Measuring Dimensions3. Interactive Visualization

Localization

LocalizationCentimeter level accuracy within an existing

mapProvides full 3D translation and rotationRequires no instrumentation of the environmentAn alternative to RFID / 802.11 / GSM /

PowerLine based localization in GPS denied environments

Measuring Dimensions

Online real estate salesVirtual remodelingOnline furniture shoppingVirtual walkthrough for educationRemote telepresense

Interactive Visualization

Interactive gesture controlled visualization

ConclusionIncorporating user interaction into real-time 3D

mapping allows non-experts to create accurate maps using RGB-D cameras (such as the Kinect)

These maps have applications to localization, measurement, and visualization

Making this system publicly available will allow for a proliferation of 3D mapping and the potential for new context-aware applications

peter@cs.washington.edu