High resolution holographic image synthesis for future display eyeglasses

Praneeth Chakravarthula

UNC Chapel Hill

Future of Personal Computing

2

3

Wide field of view

High resolution

Accommodation support

Eyeglasses-Style Near-Eye Display Optics

Eyeglasses-Style Near-Eye Display Optics

4

Wide field of view

Moderate resolution

Accommodation support

Holography is the only demonstrated technology for getting everything

Maimone et al. 2017

Basics of Digital Holography

5

Principle of Holography

6

Principle of Holography

7

Principle of Holography

8

Step 1: Recording

Principle of Holography

9

Step 1: Recording

Principle of Holography

10

Step 1: Recording Step 2: Playback

Principle of Holography

11

Step 1: Recording

Principle of Holography

12Step 2: Playback

Step 1: Recording

13

Incident light modulated by Hologram H

Holographic Image Formation

14

Propagates to result in the final field z

and final image |z|2

Holographic Image Formation

Phase Hologram Reconstructed Image

Reference

Heuristic Hologram Phase Retrieval

Reconstruction

Double Phase Encoding Hologram

Phase Hologram Reconstructed Image

Reference Reconstruction

Wirtinger Holography

Phase Hologram Reconstructed Image

Reference Reconstruction

Wirtinger Holography

18

19

Differentiable forward model

Compute complex Wirtinger gradients

Optimize for phase holograms

Wirtinger Holography Overview

20

Differentiable forward model

Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image

Compute complex Wirtingergradients

Optimize for phase holograms

21

Differentiable forward model

Compute complex Wirtingergradients

Optimize for phase holograms

Step 1: For a penalty function f, compute the error between the holographic reconstruction and he target image

22

Differentiable forward model

Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image

Compute complex Wirtingergradients

Optimize for phase holograms

23

Differentiable forward model

Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image

Compute complex Wirtingergradients

Optimize for phase holograms

24

Differentiable forward model

Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image

Compute complex Wirtingergradients

Optimize for phase holograms

25

Differentiable forward model

Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image

Compute complex Wirtingergradients

Optimize for phase holograms

26

Differentiable forward model

Compute complex Wirtingergradients

Optimize for phase holograms

Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase

27

Differentiable forward model

Compute complex Wirtingergradients

Optimize for phase holograms

Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase

28

Differentiable forward model

Compute complex Wirtingergradients

Optimize for phase holograms

Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase

29

Differentiable forward model

Compute complex Wirtingergradients

Optimize for phase holograms

Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase

30

Differentiable forward model

Compute complex Wirtingergradients

Optimize for phase holograms

Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase

We can use standard optimizers if there is a gradient

31

Differentiable forward model

Compute complex Wirtingergradients

Optimize for phase holograms

Holomorphic function: Complex function that is complex differentiable

Derivative of holomorphic real-valued function is always ZERO

Derivatives of any order are NOT DEFINED for our objective function

Real-valued function

Complex-valued argument

Optimize for phase holograms

Differentiable forward model

Compute complex Wirtingergradients

32

Two important properties of gradient:

1) Direction of maximal rate of change

2) Is zero at stationary points

Step 3: Define approximate gradient and compute Wirtinger derivatives for each propagation model

REFER TO MY SIGGRAPH Asia 2019PAPER AND SUPPLEMENT

33

Differentiable forward model

Compute complex Wirtingergradients

Optimize for phase holograms

Standard off-the-shelf optimization methods

1) Quasi-Newton

2) Stochastic gradient descent

Step 4: Optimize for holograms using off-the-shelf methods

Simulation Results

34

Reference

Reconstruction

Reference Reconstruction

37

Simulation Results

TargetModified GS

(Peng et al. 2017)Double phase

(Maimone et al. 2017) Our Method

38

Simulation Results

TargetModified GS

(Peng et al. 2017)Double phase

(Maimone et al. 2017) Our Method

Prototype Hardware Results

41

Prototype Hardware Results

TargetModified GS

(Peng et al. 2017)Double phase

(Maimone et al. 2017) Our Method

42

TargetModified GS

(Peng et al. 2017)Double phase

(Maimone et al. 2017) Our Method

Prototype Hardware Results

43

TargetModified GS

(Peng et al. 2017)Double phase

(Maimone et al. 2017) Our Method

Prototype Hardware Results

Results Reference

Experiment

Reference Experiment

46

Perceptually Optimized Holography

Learned Perceptual Image Patch Similarity (LPIPS) ( Zhang et al. 2018 )

Optimize for deep learning based perceptual losses

47

Perceptually Optimized Holography

Target LPIPS optimizedMS-SSIM optimizedL2 optimized

48

Cascaded Superresolution Holography

49

TargetModified GS

(Peng et al. 2017)Double phase

(Maimone et al. 2017) Our Method

Prototype Hardware Results

Real World Deviations

51

Ideal Wave Propagation

52

Real World Wave Propagation

53

Compensating real world deviations viaHardware-in-the-loop phase retrieval

Target (Chakravarthula et al. 2019)Wirtinger HolographyDouble Phase Encoding

(Maimone et al. 2017) Our Method

Upcoming at SIGGRAPH Asia 2020

54

Target (Chakravarthula et al. 2019)Wirtinger HolographyDouble Phase Encoding

(Maimone et al. 2017) Our Method

Target (Chakravarthula et al. 2019)Wirtinger HolographyDouble Phase Encoding

(Maimone et al. 2017) Our Method

Compensating real world deviations viaHardware-in-the-loop phase retrieval

Upcoming at SIGGRAPH Asia 2020

Differentiable forward model

Compute complex Wirtinger gradients

Optimize for phase holograms

WirtingerHolography

www.cs.unc.edu/~cpkPraneeth Chakravarthula