mobile ar lecture 8 - ar panoramas

Researcher: Zi Siang See

Lecture 8: Augmented Reality using High Fidelity

Spherical Panorama with HDRI

Research Presenta,on: Zi Siang See Human Interface Technology Laboratory New Zealand (HIT Lab NZ) Email (Industrial collabora,on): [email protected]

Zi Siang See, Mark Billinghurst, Adrian David Cheok (2015) Taken from: SIGGRAPH Asia 2015 Symposium On Mobile Graphics And Interac,ve Applica,ons

Zi Siang See, Mark Billinghurst, Adrian David Cheok (2015) Augmented Reality using High Fidelity Spherical Panorama with HDRI. SIGGRAPH Asia 2015 Symposium On Mobile Graphics And Interac,ve Applica,ons. hVp://dx.doi.org/10.1145/2818427.2818445 hVp://www.zisiangsee.com/research/zisiangsee_research_p009.pdf

Mobile-Based Augmented Reality Development


Research Presenta?on

1) Introduc?on, Related Work 2) Method and Apparatus 3) Discussion 4) Prac,cal Session


Figure: AR on Mobile

•  Mobile Augmented Reality (AR) mixes a live real-‐world view with virtual interac,ve content on a mobile device.

•  One of the key enablers for this is tracking technology, such as computer vision techniques for tracking off pre-‐defined markers or markerless images.

Introduc?on: Augmented Reality (AR) on Mobile Source: google search (keyword “augmented reality”)


Augmented Reality using High Fidelity Spherical Panorama with HDRI

High fidelity requirements – mobile AR environment recogni?on • Spherical Panorama • High Dynamic Range Imaging (HDRI) • Op,miza,on involves obtaining accurately reproduced spherical panorama with least acquisi,on ,me -‐ AR source tracking content has sufficient luminance by using HDRI having dynamic range extended from single acquired digital nega,ve for each angle in mul,row configura,on.


Figure: (a) AR using marker. (b) AR using panorama image, markerless experience.

•  There have been previous studies on using pre-‐produced panorama images for AR tracking (Arth et al 2011; DiVerdi et al. 2008; Langlotz et al. 2014; Ventura and Höllerer 2013; Wagner et al. 2010).

•  Most of related studies describe how the panorama images can be used for AR tracking.

•  Therefore, there is a need for specifying the method for high fidelity produc,on of the source panorama images for mobile AR.

Introduc?on: AR using Panorama Image (a) (b)


Case Study: Corporate Project Requirements


Townships

Landscape

Augmented /Virtual Other Scope

Construc?on

Panorama Image Reproduc?on, Projects Heavy Requirement & Workload/Quan?ty

Industrial Case Study: Amount, Time, Quality


Figure: Panorama image reproduc,on process for AR /VR projects.


Introduc?on: Spherical Panorama Acquisi?on


Figure: reproduc,on op,ons available for fast panorama image acquisi,on


LadyBug GoPro Rig Mobile (Handheld)


Figure: reproduc,on op,ons available for fast panorama image acquisi,on


LadyBug GoPro Rig Mobile (Handheld)

Parallax error Shake /Blur

Resolu?on Limited Dynamic Range

Inconsistent ligh?ng distribu?on for mul?ple angles

Nadir difficulty

Ghos?ng effect (HDRI)

Inconsistent white balance for mul?ple angles

Time management


•  Acquire wider panoramic visibility (Benosman and Kang, 2001) •  Major difficulty for spherical panorama reproduc,on may occur, this can

include limited dynamic range reproduced and parallax error (Brown and Lowe, 2006; Felinto et al, 2012)

Figure: panoramic imagery with visual abnormality -‐ loca,on in Sannomiya Kobe, Japan.

Introduc?on: Spherical Panorama Acquisi?on Camera shake

Limited dynamic range Visual abnormality

Parallax error


Figure: combined luminance (EV) range being covered by HDRI outcome.

Darker images (luminance) Brighter images

(luminance)

Introduc?on: HDRI (for mul?ple angle images)

•  High Dynamic Range Imaging (HDRI) in photography technique has a goal to increase the dynamic range resul,ng an improved luminance reproduced in shadow and highlight areas (Reinhard et al, 2010; Debevec and Malik, 1997; Fairchild, 2007), usually by combining mul,ple exposures consist of darker and brighter images.


•  Obstacles and issues of HDRI that involves mul,ple exposures can include lengthy acquisi,on (for each angle), ghos,ng effect and misalignment issue due to inconsistent acquisi,ons (Reinhard et al, 2010)

Figure: HDRI with visual abnormali,es, ghos,ng.

Introduc?on: HDRI (for mul?ple angle images)

Mul,ple exposures for HDRI Single exposure (LDR) Visual abnormality


•  Spherical panorama can be used for AR as source tracking image. •  Various studies (Felinto et al 2012, Gawthrop 2007, Brown and Lowe

2006; Schulz, 2012) have shown that photographic combina,on for HDRI and spherical panorama can be possible, however condi,onally tolera,ng with obstacles and issues.

Spherical Panorama (Mul?row) HDRI requires mul?ple exposures

-‐4, -‐3, -‐2, -‐1, 0, +1, +2, +3, +4 (EV)

Difficulty: Spherical Panorama and HDRI for AR?


Figure: example of source tracking images or scenarios for AR

•  For AR, Lieberknecht et al. [2009] has shown that it can be difficult to create source tracking images, or synthe,c images that reproduce the real effects of real-‐world phenomena such as ligh,ng, noise, mo,on blur, discre,za,on, blooming or limited color depths during real image acquisi,on.

•  Image elements that influence real-‐,me tracking results include texture richness, the texture repeatability of the objects to be tracked, the camera mo,on and speed, the changes of the object scale, and varia,ons of the ligh,ng condi,ons over ,me.

Difficulty: Source Tracking Images for AR Source: google search (keyword “augmented reality”)



1) Introduc,on, Related Work 2) Method and Apparatus 3) Discussion 4) Practical Session


.. no photographic process is complete un?l the final images can be proudly displayed (Phase One, 2015).


Method: Spherical Panorama

Addi,onal 3 nadir acquisi,on

Horizontal angles

Zenith

Nadir image acquired without including the base of the camera. Stable moun,ng is required for avoiding driring, shake and parallax error.

Nadir without obstacle, allow sufficient space.


Method: High Dynamic Range Image (HDRI)

Pixe

l Val

ue

Sequences of extendable dynamic range from single RAW (photographic recording instrument Nikon D3x)

•  Extending Dynamic Range from single acquired Digital Nega,ve (RAW). •  Consistent to the idea brought up by Debevec and Malik (1997) about

extending dynamic range can be possible from conven,onal single film processing -‐ mul,ple scanning method.



•  -‐2 EV and +2 EV, resul,ng an addi,onal increment from 8.5EV to 12.5EV. •  The finding in the study about na,ve EV of dynamic range reproduc,on

was observed to have similar approximated usable exposure values with the RAW produced by ISO100 according to the test by Rehm (2009).

Pixe

l Val

ue

Sequences of extendable dynamic range from single RAW (photographic recording instrument Nikon D3x)



Figure: an image from mul,ple angle before applying HDRI.

without


Figure: proposed mul,ple angles HDRI for high fidelity AR. (HDRI method that requires a single acquisi,on that extends the dynamic range from a digital nega,ve)

Method: High Dynamic Range Image (HDRI) with HDRI


Source Image for AR: High Fidelity Spherical Panorama with HDRI

mul?ple angles facilitated by HDRI required for producing the spherical panorama (least or near-‐zero visual abnormality)


Mul?ple angles HDRI of mul?row configura?on (from the proposed method) with reduced error or mistakes that requires rec?fica?ons, including near-‐perfect nadir angle reproduc?on.

Nadir Angle: Reproduc?on from the original loca?on-‐based scene, minimal post-‐produc?on correc?on required

cubic projec?on



mul?ple angles facilitated by HDRI required for producing the spherical panorama (least or zero visual abnormality)



HDRI ghos?ng effect using mul?ple exposures method.

HDRI reproduc?on from single acquisi?on, digital nega?ve (RAW) visual abnormality is eliminated or avoided (for mul?ple angles required for spherical panorama)



mul?ple angles facilitated by HDRI required for producing the spherical panorama (least or zero visual abnormality)



(a) LDR photo image (b) HDRI from single source acquisi,on (c) HDRI from mul,ple exposures

HDRI Panorama with reduced overall photographic produc?on processes and ?me.



Augmented Reality using High Fidelity Spherical Panorama with HDRI (Authoring process for AR environment recogni?on)

Spherical Panorama facilitated with HDRI (Cubic projec,on)

Cropped /selec,ve cubic facade working with image-‐based AR

Figure: augmenta,on can be assigned on any cubic facades, speculated to work on most of the AR browsers /workflow.


AR for Mobile and Wearable

Figure: handheld experience of markerless AR.



Figure: handheld experience of markerless AR (video).


Zeiss VR One

•  Carl Zeiss announced VR One, a virtual reality headset for use with a smartphone. It is a viewer designed to work with phones between 4.7 and 5.2 inches (Zeiss, 2014).

•  Options for HMD using mobile device as computing module are becoming available.


Samsung Gear VR

•  Samsung has launched Gear VR in October 2014, co-developed with Oculus. Gear VR can house specific mobile device (Note 4) for providing virtual reality and augmented user experience (Samsung, 2014)



Figure: stereo image of AR – adaptable for HMD such as Oculus or Google Cardboard.



Figure: HDRI spherical panorama sampled at Sannomiya Sta,on, Japan (01 Nov 2015).



Figure: stereo image of AR – adaptable for HMD such as Oculus or Google Cardboard.


Discussion

Poten?al obstacles and issues (before op?miza?on): •  Parallax error (especially during hand-‐held) •  Unstable image acquisi,on, driring mo,on (hand-‐held) •  Nadir angle difficulty •  Compromised geometrical registra,on in image •  Low dynamic range image (LDRI) •  High dynamic range requires mul,ple exposures •  Long acquisi,on ,me if mul,ple exposures •  High dynamic range ghos,ng with moving objects •  Mul,ple exposures misalignment •  Inconsistent ligh,ng distribu,on for mul,ple angles

Benefit of using our method and apparatus: •  Free from parallax error •  Stable image acquisi,on •  Nadir angle with near-‐error-‐free authen,city •  Accurate geometrical registra,on in image •  High dynamic range image (HDRI) •  High dynamic range from single RAW acquisi,on •  High dynamic range with least acquisi,on ,me •  High dynamic range with no ghos,ng •  High dynamic range with perfect alignment •  Consistent high dynamic range for mul,ple angles


Conclusion

•  Markerless AR. •  High fidelity spherical panorama with HDRI can provide a near-‐error-‐free

and dynamic range enhanced source of image-‐based AR tracking content. •  Source tracking content is reproduced with liVle distor,on, producing a

result very similar to the original scene condi,on. •  Without using extra sensors such as a GPS. •  Allows an AR experience to be delivered on a mobile device with a lower

processing requirement -‐ adaptable in other wearable system or devices.


Implica?ons

•  Android-‐based mobile with RAW capturing capability. •  Approach adapta,on in VR/AR workflow for mobile applica,ons.


Future Study

•  In the future we will conduct more extensive evalua,on studies to compare the tracking accuracy with the systems using our HDRI panorama images to other tradi,onal approaches.

•  We will also explore other solu,ons suitable for HDRI panorama video and hybrid approaches that combine panorama image tracking and sensor input.


Thank you

(Next) Practical and Authoring





HDRI Panorama reproduc,on Interac,ve panorama output AR authoring Tes,ng and simula,on Real-‐world tes,ng


Interac?ve Panorama Pano2vr (or other tools)



Figure: HDRI spherical panorama sampled at Sannomiya Sta,on, Japan (01 Nov 2015).


Augmented Reality using High Fidelity Spherical Panorama with HDRI (Authoring process for AR environment recogni?on)

Spherical Panorama facilitated with HDRI (Cubic projec,on)

Cropped /selec,ve cubic facade working with image-‐based AR

Figure: augmenta,on can be assigned on any cubic facades, speculated to work on most of the AR browsers /workflow.


Wikitude Studio hcp://studio.wikitude.com


Layar Creator hcp://www.layar.com


Load Panorama and Try AR for simula?on (also works on actual loca?on)


Thank you Research Discussion

collaboration /contact www.zisiangsee.com



Additional



Figure: early aVempt for HDRI computa,on from the source of single acquired digital nega,ve, RAW.

HDRI Computation Input: Source image in digital negative RAW format (e.g. NEF, DNG) Output: The HDRI is reproduced from a single acquired RAW image having its dynamic range extended from RAW

1. Three images, I, I+, I- are processed from the single source of RAW.

2. I+ and I- are respectively altered by +2EV and -2EV where dynamic range are extended. That is, the luminance lx of each pixel x is recalculated as (lx)new

= 2m (lx)old, where m is the change in EV.

3. A new image, I’+, is constructed out of I+, by preserving only the highlight regions of I+. Similarly, a new image I’- is constructed out of I-, by preserving only the non-highlight regions of I-.

4. I’+ is then overlaid on top of I’- with 50% opacity. The results of the two are then combined to form the final image I’.


mul?ple angles of mul?row configura?on, covering horizontal, zenith and nadir angles. (However having nadir angle difficulty, parallax error, low dynamic range)

mul?ple angle images



1/8 sec, f8 1/15 sec, f8 1/30 sec, f8 1/60 sec, f8 1/125 sec, f8 1/250 sec, f8 1/500 sec, f8 1/1000 sec, f8 1/2000 sec, f8

Scenario 1 (high brightness)

HDRI Acquisi?on: Ligh?ng Scenarios

Mul?ple exposures approach Our HDRI approach (dynamic range extended from single acquired digital nega?ve, RAW)



Scenario 1 (high brightness)



Single Acquisi,on (RAW)

Process: Extending Dynamic Range

Extended +2EV Original

0EV Extended -‐2EV

HDRI from single source

HDRI that requires single acquisi,on HDRI that requires mul,ple exposures

-‐4, -‐3, -‐2, -‐1, 0, +1, +2, +3, +4 (EV)



Scenario 2 (moderate brightness)




4 sec, f8 2 sec, f8 1 sec, f8 1/2 sec, f8 1/4 sec, f8 1/8 sec, f8 1/15 sec, f8 1/30 sec, f8 1/60 sec, f8

Scenario 3 (low light)




Additional (extended)