vision-correcting displays @ siggraph 2014

Eyeglasses-free Display:

Towards Correcting Visual Aberrations with

Computational Light Field Displays

Fu-Chung Huang1,+ Gordon Wetzstein2,# Brian A. Barsky1 Ramesh Raskar2

University of California, Berkeley

MIT Media Lab

now at Microsoft

now at Stanford University

1

2

+

#

shown at 350mm

normal display

distance to display

focal

range

perceived image

normal display

distance to display

focal

range

pinhole array mask parallax barrier based light field display

25%

U.S. population of hyperopia (far-sightedness)

[Krachmer et al. 2005]

43%

age 40

U.S. population of presbyopia (need reading eyeglasses)

[Katz et al. 1997]

68%

age 80+

[Katz et al. 1997]

U.S. population of presbyopia (need reading eyeglasses)

43%

age 40

U.S. population of myopia (near-sightedness)

41.6%

[Vitale et al. 2009]

Myopia in some Asian countries

60% ~ 90%

[Rajan et al. 1995] [Wong et al. 2000]

[Takashima et al. 2001] [Lin et al. 2004]

Irregular Blurring in VisionPSF PSF PSF

caused by higher-order aberrations

Nirmud lens (?)

9th century

reading stone

1284

Salvino D’Armato

1508

concept

1760

Benjamin Franklin

1880

August Mueller

1983

PRK and LASIK

now

934 B.C.

Computational light field display

(eye-tracking)

(input data)

Prior Work

Projector Precompensation- Brown et al. [2006]

- Zhang and Nayer [2006]

- Oyamada et al. [2007]

- Grosse et al. [2010]

Computational Displays

- Lanman et al. [2010]

- Wetzstein et al. [2012]

- Maimone et al. [2013]

- Hirsch et al. [2014]

- Akeley et al. [2004]

Computational Vision Correction- Alonso and Barreto [2003]

- Yellot and Yellot [2007]

- Huang et al. [2012]

- Pamplona et al. [2012]

- Ji et al. [2014]

- Huang and Barsky [2011]

How to Build a Vision Correcting Display

Spatial domain Frequency domain

=

⊗ ∗

=

𝑖𝑚𝑔 ∗ 𝑝𝑠𝑓 = 𝑏𝑙𝑢𝑟𝑖𝑚𝑔 ⊗ 𝑝𝑠𝑓 = 𝑏𝑙𝑢𝑟


∗

=

−1 𝑖𝑚𝑔 ∗ 𝑝𝑠𝑓 = 𝑝𝑟𝑒

=

𝑖𝑚𝑔 ⊗ 𝑝𝑠𝑓 = 𝑝𝑟𝑒

⊗

−1


prefiltered

perceived

=

𝑖𝑚𝑔 ⊗ 𝑝𝑠𝑓 = 𝑝𝑟𝑒

⊗

−1

Spatial domain

plane of focus time-multiplexed

PSFPSF

[Huang et al. 2012]

without correction multilayer displayconventional display

(simple inversion)

7x7 views into the eye

[Pamplona et al. 2012]

without correction

corrected vision

pupil aperture

Inversely Prefilter the Light Field

target image prefiltered light field

Flatland Light Field Projection

retina

𝐼(𝑥) = −∞

+∞

𝑙 𝑥, 𝑢 𝐴 𝑢 𝑑𝑢Retinal image:

𝒙𝒖

𝒙

𝒖

display

focus plane

(1D image + 1D direction)

Light Field Projection

𝐼(𝑥) = −∞

+∞


𝒙

𝒖

retinadisplay

focus plane

𝒙𝒖

“Defocus” Light Field Projection

focus plane

𝒙

𝒖

retinadisplay

𝒙𝒖


focus plane

𝒙

𝒖

retinadisplay

𝒙𝒖

convolution


focus plane

𝒙

𝒖

retinadisplay

𝒙𝒖

convolution

𝝎𝒙

𝝎𝒖

frequency domain

analysis (in the paper)

Using a Light Field Display

𝒙

𝒖

𝒍𝒅

= −𝑟/2

𝑟/2

𝑙𝑑 Ψ𝑥𝑢

𝑑𝑢

𝐼(𝑥) = −∞

+∞


more degreesof freedom

focus plane

retina

𝒙𝒖

Using a Light Field Display𝒍𝒅

= −𝑟/2

𝑟/2

𝑙𝑑 Ψ𝑥𝑢

𝑑𝑢

𝐼(𝑥) = −∞

+∞


focus plane

retina

𝒙𝒖

?

Using a Light Field Display𝒍𝒅 𝒙𝒖

?𝐏 ∙ 𝐋𝒅 𝐈=

Using a Light Field Display𝒍𝒅 𝒙𝒖

?𝐋𝒅 𝐈= 𝐏−𝟏𝐏 ∙


𝒙

𝒖

𝒍𝒅


focus plane

retina

𝒙𝒖

𝐋𝒅 𝐈= 𝐏−𝟏


𝒙

𝒖

𝒍𝒅


focus plane

retina

𝒙𝒖

𝐋𝒅 𝐈= 𝐏−𝟏

become well-posed?

Experiments and Results

250 m

m

focus 3

80 m

m

f = 50 mma = 6 mm

without correction Pamplona et al. 2012multilayer prefilteringtarget image light field prefiltering

HDR-VDP2

Low error detection

Higher Order Aberrations

wit

ho

ut

co

rre

cti

on

co

nve

nti

on

al

dis

pla

y

lig

ht

fie

ld

dis

pla

y

* =

:

::

:

Display light fieldProjection matrices

Axial or lateralmovement

conventionaldisplay

multilayerdisplay

[Pamplona et al.2012][Huang et al.2012] Proposed method

Method Inverse prefiltering Direct ray tracing Prefiltered light field

Spatial Resolution Very High Very Low High

Image Contrast Very Low Full (100%) High

Building Cost High Very High Very Low

light fielddisplay

light fielddisplay

Shortcomings

• Contrast and brightness loss – Content-dependent

• Resolution loss– 3-to-1(DroidDNA), 5-to-1(iPhone)

– about 150 PPI

• Computation– GPU, Mobile

• Calibration– Eye-tracking

– Off-Axis Opt.

Future Work

• Higher Resolution & Large Display– e.g. tensor displays

• Multi-way correction

• Other applications– AR/VR, 3D, Cryptography

• Theoretical analysis– Higher order aberrations

Eyeglasses-free Display

http://web.media.mit.edu/~gordonw/VisionCorrectingDisplay/

http://graphics.berkeley.edu/papers/Huang-EFD-2014-08/

Fu-Chung Huang Gordon Wetzstein

Brian A. Barsky Ramesh Raskar

http://displayblocks.org/

Frequency Domain Analysis

𝝎𝒙

𝝎𝒖

(a) conventional display

(in-focus)

sp

atia

l d

om

ain

fre

qu

en

cy d

om

ain

no angular variations

only spatial energy

𝒙

𝒖

𝝎𝒙

𝝎𝒖

𝒙

𝒖


(in-focus)

sp

atia

l d

om

ain

fre

qu

en

cy d

om

ain

pupil function

is a rect(), in umultiplication

( just spreading )pupil response

is a sinc(), in 𝜔𝑢convolution

𝝎𝒙

𝝎𝒖

𝒙

𝒖


(in-focus)

sp

atia

l d

om

ain

fre

qu

en

cy d

om

ain

retinal projection

integration in u

slicing at 𝜔𝑢 = 0

𝐼 𝑥

𝐼 𝜔𝑥

Fourier Slice Theorem

𝝎𝒙

𝝎𝒖

𝝎𝒙

𝝎𝒖

𝒙

𝒖

𝒙

𝒖


(in-focus)

(b) conventional display

(out-of-focus)

sp

atia

l d

om

ain

fre

qu

en

cy d

om

ain

retinal projection

𝝎𝒙

𝝎𝒖

𝝎𝒙

𝝎𝒖

𝝎𝒙

𝝎𝒖

𝒙

𝒖

𝒙

𝒖

𝒙

𝒖


(in-focus)


(out-of-focus)

(c) multilayer display

(out-of-focus)

sp

atia

l d

om

ain

fre

qu

en

cy d

om

ain

retinal projection

𝝎𝒙

𝝎𝒖

𝝎𝒙

𝝎𝒖

𝝎𝒙

𝝎𝒖

𝒙

𝒖

𝒙

𝒖

𝒙

𝒖


(in-focus)


(out-of-focus)

(c) multilayer display

(out-of-focus)

sp

atia

l d

om

ain

fre

qu

en

cy d

om

ain

retinal projection

𝝎𝒖

(d) light field display

(out-of-focus)

𝒙

𝒖

𝝎𝒙

(d) light field display

(out-of-focus)

vision-correcting displays @ siggraph 2014

Technology