addressing challenges with augmented reality applications ...€¦ · reality applications on...
TRANSCRIPT
![Page 1: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/1.jpg)
Addressing Challenges with Augmented Addressing Challenges with Augmented Reality Applications on Reality Applications on SmartphonesSmartphones
Krzysztof Zienkiewicz
Vanderbilt University Nashville, Tennessee
Presented at Mobilware 2010
![Page 2: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/2.jpg)
Augmented Reality (AR)
• Azuma’s definition: AR is a system that has the following characteristics
• Combines real and virtual
• Interactive in real time
• Registered in 3D
• Milgram’s Continuum
![Page 3: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/3.jpg)
Components of an AR application
• Rendering 3D context-specific data
• Retrieving geotagged Points Of Interest (POIs)
• Frame of Reference
estimation
![Page 4: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/4.jpg)
Challenges
• Mobile 3D solutions are not optimal and hard to mesh with cameraimagery
• Filtering geotagged POIs by proximity is computationally intensive
• Real-time estimation of frame of reference is computationally demanding
• Geomagnetic sensor noise makes orientation estimation hard
![Page 5: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/5.jpg)
Mobile 3D solutions are not optimal
• AR needs precise 3D overlays to provide an intuitive interface
• Without accuracy, user experience deteriorates
• Meshing 3D data with camera preview is highly platform dependent
• Mobile GLs don’t offer a “pick”
mode
![Page 6: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/6.jpg)
Retrieving POIs by proximity is expensive
• Areas of interest may have a high density of POIs
• Retrieval by location requires many comparisons
• Bandwidth limitations
![Page 7: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/7.jpg)
Estimation of frame of reference
• Fiducial markers
• Feature extraction
![Page 8: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/8.jpg)
Geomagnetic sensor noise
• Compass used to estimate the field of vision
• Compass noise introduces jitter in the rendered overlays
• Savitzky-Golay smoothing filter is too slow
• Noise +/- 9 degrees
![Page 9: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/9.jpg)
Geomagnetic sensor noise
Device Steady
-10-8-6-4-202468
10
0 1 2 3 4 5 6 7
Time (s)
Ang
le o
ffse
t (de
gree
)
![Page 10: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/10.jpg)
Effects of geomagnetic sensor noise
Noise θ
![Page 11: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/11.jpg)
Effects of geomagnetic sensor noise Effect of Compass Noise
0
5
10
15
20
25
30
35
40
0 1 2 3 4 5 6 7 8 9 10
Noise (degrees)
Jitte
r (pi
xels
)
θα
tantan2
widthjitter =
![Page 12: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/12.jpg)
Solutions
• Using hardware accelerated 3D APIs to display rendered content
• A grid-based approach to data storage and retrieval
• Using GPS and geomagnetic sensors to estimate device position and orientation
• Statistical analysis filter
![Page 13: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/13.jpg)
Hardware accelerated 3D APIs
• Apple's UIKit allows a transformation matrix to be applied to views
• Allows for hit testing
• Other platforms will need to do hit testing by hand
• But get to use GLs for rendering
• Meshing achieved via layering
![Page 14: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/14.jpg)
Grid-based data storage and retrieval
• Mapping function takes location to grid indices
• Each square contains all POIs within the region
• No numeric comparisons are performed by the server
• POIs are downloaded in bulk and need to be filtered on the phone
![Page 15: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/15.jpg)
Using sensors to estimate field of view
• The computationally cheap solution
• Only need to update the transformation matrix
![Page 16: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/16.jpg)
Nature of the noiseNoise Normality Test
y = 1.0036x - 0.0075R2 = 0.9861
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
Expected z-score
Obs
erve
d z-
scor
e
![Page 17: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/17.jpg)
Filtering algorithm
• Small number of parameters
• Extendable for dynamic
parameterization
• Good results: 60% noise reduction
![Page 18: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/18.jpg)
Filtering algorithm
Device Steady
-10-8-6-4-202468
10
0 1 2 3 4 5 6 7
Time (s)
Ang
le o
ffse
t (de
gree
)
![Page 19: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/19.jpg)
Filtering algorithm
Device during Uniform Rotation
-5-4-3-2-1012345
0 1 2 3
Time (s)
Ang
le o
ffse
t (de
gree
)
![Page 20: Addressing Challenges with Augmented Reality Applications ...€¦ · Reality Applications on Smartphones Krzysztof Zienkiewicz krzysztof.k.zienkiewicz@vanderbilt.edu Vanderbilt University](https://reader034.vdocument.in/reader034/viewer/2022042709/5f40dc2bd45b426dd4479636/html5/thumbnails/20.jpg)
Thanks
Questions?