Download - Keynote from ISUVR'10
![Page 1: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/1.jpg)
Image-based modelling for augmented reality
Anton van den Hengel
Director, Australian Centre for Visual technologies
Professor, Adelaide University, South Australia
Director, PunchCard Visual Technologies
![Page 2: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/2.jpg)
3D Modelling for AR
AR needs modelsAR is about the interaction between the real
and the synthetic 3D modelling isn’t much fun
Even with the best interfaces invented 3D Studio Max? Blender?
![Page 3: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/3.jpg)
User-created content
2D UCC has changed the face of the web Blogs, Wikis, Social networking sites, Advertising, Fanfiction, News Sites, Trip
planners, Mobile Photos & Videos, Customer review sites, Forums, Experience and photo sharing sites, Audio, Video games, Maps and location systems and such, but more
Associated Content, Atom.com, BatchBuzz.com, Brickfish, CreateDebate, Dailymotion, Deviant Art, Demotix, Digg, eBay, Eventful, Fark, Epinions, Facebook, Filemobile, Flickr, Forelinksters, Friends Reunited, GiantBomb, Helium.com, HubPages, InfoBarrel, iStockphoto, Justin.tv, JayCut, Mahalo, Metacafe, Mouthshut.com, MySpace, Newgrounds, Orkut, OpenStreetMap, Picasa, Photobucket, PhoneZoo, Revver, Scribd, Second Life, Shutterstock, Shvoong, Skyrock, Squidoo, TripAdvisor, The Politicus, TypePad, Twitter, Urban Dictionary, Veoh, Vimeo, Widgetbox, Wigix, Wikia, WikiMapia, Wikinvest, Wikipedia, Wix.com, WordPress, Yelp, YouTube, YoYoGames, Zooppa
![Page 4: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/4.jpg)
User-created content for AR
![Page 5: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/5.jpg)
Google-created content for AR
![Page 6: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/6.jpg)
UCC for AR
Just using images is a good startBut limits interactions to 2D
Flexible AR requires 3D models Ubiquitous AR requires UCC Flexible ubiquitous AR requires 3D UCC
![Page 7: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/7.jpg)
3D UCC
3D has been limited by the lack of good UCC tools This is true for AR But also VR, 3D TV,
Second Life, Google Earth, Little Big Planet, 3D PDF, Adobe Premier, Unreal Tournament, Playstation, SGML, ...
![Page 8: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/8.jpg)
3D UCC
AR particularly needs to model the real world Images are a good source of 3D information
Easily accessible They’re typically captured anyway Almost everything has a camera attached
Humans are very good at interpreting them
Can AR be ubiquitous without UCC?
![Page 9: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/9.jpg)
Image-based 3D UCC
The image is the interfacePeople can’t help but see images in 3DMost image sets embody 3D
Powerful way to model real objectsVarying levels of interactionVarying types of models
Helps even in modelling imaginary objects
![Page 10: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/10.jpg)
Image-based modelling for AR
AR is largely about interactive imagesAny other mode of interaction adds
complexity The majority of the content is real 3D modelling from images seems a
natural fit with AR
![Page 11: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/11.jpg)
Image-based 3D modelling
AutomaticVery detailed models of everythingBut it’s getting better
InteractiveMeans you can specify
What you want to model What kind of model you want
![Page 12: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/12.jpg)
Videotrace
Interactive image-based modelling A familiar interface Image-based interactions
The image is the interface Generates low polygon count models with
textures
![Page 13: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/13.jpg)
Input
![Page 14: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/14.jpg)
Modelling
![Page 15: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/15.jpg)
Results
![Page 16: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/16.jpg)
Another example
![Page 17: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/17.jpg)
Interactive 3D modelling
3D modelling is critical to all sorts of application Special effects, but also mining, architecture, defence,
urban planning, … People are getting more visually sophisticated More 3D data is being generated
More cameras, but also scanners etc The interfaces of modelling programs are usually
very hard to fathom
![Page 18: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/18.jpg)
Low polygon-count models
Insert your own objects into a game Model an environment for AR Put your house into Google Earth Video editing
Cut and paste between sequencesRemove someone from your home videos
![Page 19: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/19.jpg)
Put your truck into a game
![Page 20: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/20.jpg)
Put your truck into a game
![Page 21: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/21.jpg)
Modelling for special effects
![Page 22: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/22.jpg)
Video editing requires models
![Page 23: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/23.jpg)
Video editing requires models
![Page 24: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/24.jpg)
Modelling architecture
![Page 25: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/25.jpg)
Modeling for virtual environments
![Page 26: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/26.jpg)
Modeling for virtual environments
![Page 27: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/27.jpg)
Modeling for virtual environments
![Page 28: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/28.jpg)
Modeling for virtual environments
![Page 29: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/29.jpg)
The process
Capture and import the video Run video through the camera tracker
Performs structure and motion analysis
Interact with the system to generate and edit the modelExport to your application
![Page 30: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/30.jpg)
The approach
Pre-compute where possibleStructure from motion (camera tracking)Superpixels
Then interact Interactions allow user to exploit precomputed
results
![Page 31: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/31.jpg)
Structure from motion
Camera tracking Calculates
Reconstructed point cloudCamera parameters
Location Orientation Intrinsics (eg. Focal length)
Informs interaction interpretation process
![Page 32: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/32.jpg)
Structure from motion
![Page 33: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/33.jpg)
Interactions Straight lines
Closed sets of lines define planar polygons Curves
For planar shapes with curved edges For NURBS surfaces
Mirroring Duplicates existing geometry
Extrusion Dense meshing
![Page 34: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/34.jpg)
Fitting planar faces
User specifies boundary Boundary specifies infinitely many planes Fitting similar to pre-emptive RANSAC
Generate bounded plane hypotheses from point cloud
Eliminate hypotheses that fail a series of tests Run simplest / most robust tests first
Generally 3d tests before 2d tests
![Page 35: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/35.jpg)
Image plane
Line of sight
Fitting planar facesFitting planar faces
Object points
![Page 36: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/36.jpg)
Hierarchical RANSAC Generate bounded plane hypotheses Tests
Support from point cloudReprojects within new image boundariesConstraints on relative edge length and face
sizeColour histogram matching on facesColour matching on edge projectionsReprojection is not self-occluding
![Page 37: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/37.jpg)
2D Curves
![Page 38: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/38.jpg)
3D Curves
![Page 39: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/39.jpg)
Mirroring
![Page 40: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/40.jpg)
Extrusion
![Page 41: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/41.jpg)
Dense surface reconstruction
![Page 42: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/42.jpg)
Live modelling
![Page 43: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/43.jpg)
Live modelling
Most geometry cannot be modelled beforehandYou can’t tell where it will beModelling the whole world won’t work
Need to generate models in-situWhile you’re there
![Page 44: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/44.jpg)
Live modelling in AR
Using VideoTrace to model geometry from live video To insert elsewhere in
the world So real objects can
occlude synthetic geometry
![Page 45: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/45.jpg)
Live modelling for AR
The camera tracking is performed live using SLAMSimultaneous Localisation and mapping
Markerless video tracking No prior model of the space
Using PTAM Parallel Tracking and Mapping Klein and Murray
![Page 46: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/46.jpg)
PTAM
![Page 47: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/47.jpg)
Videotrace - Live
![Page 48: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/48.jpg)
Occlusion
Low polygon count models?Needed for efficiencyNot accurate enough for occlusion
calculations SLAM errors also prevent direct occlusion
modelling
![Page 49: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/49.jpg)
Occlusion boundary refinement
The model of the foreground object is projected into the imageUsing the PTAM-estimated camera
parameters But there is always some misalignment Solve using a live segmentation of the real
object from the video
![Page 50: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/50.jpg)
Occlusion boundary refinement
Lay out nodes of a graph around the projected boundarySet foreground and background probabilities
per node from colour modelSet link weights from edge strengthSegment using max-flow algorithm
At frame rate
![Page 51: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/51.jpg)
Occlusion boundary refinement
![Page 52: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/52.jpg)
Occlusion boundary refinement
Graph cut means that model doesn’t need to be accurate Very low polygon
counts Very simple modelling
process More complex objects
possible
![Page 53: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/53.jpg)
Occlusion boundary refinement
Graph cut gives a hard segmentation
Fix with an alpha matte
Blends between foreground and synthetic object
Fixes some holes in the cut
![Page 54: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/54.jpg)
Live modelling for AR
![Page 55: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/55.jpg)
AR modelling for other purposes
![Page 56: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/56.jpg)
Minimal interaction AR modelling Use the camera as the modelling tool
The user only specifies the object, the rest is done with the camera
Projective texturingSome compensation for Visual Hull
![Page 57: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/57.jpg)
Silhouette modelling
![Page 58: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/58.jpg)
Minimal interaction modelling
![Page 59: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/59.jpg)
How to get Videotrace
It’s available on free beta testJust register at www.punchcard.com.auThey will email you a link It’s a real beta
Hopefully the final version will be free too
![Page 60: Keynote from ISUVR'10](https://reader036.vdocument.in/reader036/viewer/2022081518/547e0fecb4af9f00198b45da/html5/thumbnails/60.jpg)
What’s next?
New interactions, applications and data sources Interactive SFM, Better SLAM Videoshop