Applied Perception in Applied Perception in GraphicsGraphics

Erik ReinhardErik ReinhardUniversity of UtahUniversity of Utah

reinhard@cs.utah.edureinhard@cs.utah.edu

Computer GraphicsComputer Graphics

• Produce computer Produce computer generated imagery generated imagery that cannot be that cannot be distinguished from distinguished from real scenesreal scenes

• Do this in real-timeDo this in real-time

Trends in Computer GraphicsTrends in Computer Graphics• Greater realismGreater realism

– Scene complexityScene complexity– Lighting simulations Lighting simulations

• Faster renderingFaster rendering– Faster hardwareFaster hardware– Better algorithmsBetter algorithms

• Together: still too slow and unrealisticTogether: still too slow and unrealistic

Algorithm designAlgorithm design

• Largely opportunisticLargely opportunistic

• Computer graphics is a maturing fieldComputer graphics is a maturing field

• Hence, a more directed approach is Hence, a more directed approach is neededneeded

Long Term StrategyLong Term Strategy• Understand the differences between Understand the differences between

natural and computer generated scenesnatural and computer generated scenes

• Understand the Human Visual System Understand the Human Visual System and how it perceives imagesand how it perceives images

• Apply this knowledge to motivate Apply this knowledge to motivate graphics algorithmsgraphics algorithms

This Presentation (1)This Presentation (1)

Reinhard et. al., “Color Transfer between Images”, IEEE CG&A, sept. 2001.

This Presentation(2)This Presentation(2)

Reinhard et. al., “Photographic Tone Reproduction for Digital Images, SIGGRAPH 2002.

IntroductionIntroductionThe Human Visual System is evolved

to look at natural images

Natural Random

Human Visual SystemHuman Visual System

RetinaRetina

Color ProcessingColor ProcessingRod and Cone pigments

Color ProcessingColor Processing

Cone output is logarithmic

Color opponent space

Image StatisticsImage Statistics

• Ruderman’s work on color statistics:Ruderman’s work on color statistics:

– Principal Components Analysis (PCA) on Principal Components Analysis (PCA) on colors of natural image ensemblescolors of natural image ensembles

– Axes have meaning: color opponents Axes have meaning: color opponents (luminance, red-green and yellow-blue)(luminance, red-green and yellow-blue)

Color Processing SummaryColor Processing Summary• Human Visual System expects images with Human Visual System expects images with

natural characteristics (not just color)natural characteristics (not just color)

• Color opponent space has decorrelated axesColor opponent space has decorrelated axes

• Color space is logarithmic (compact and Color space is logarithmic (compact and symmetrical data representation)symmetrical data representation)

• Independent processing along each axis Independent processing along each axis should be possible should be possible Application Application

Color TransferColor Transfer • Make one image look like anotherMake one image look like another

• For both images:For both images:– Transfer to new color spaceTransfer to new color space– Compute mean and standard deviation along Compute mean and standard deviation along

each color axiseach color axis

• Shift and scale target image to have Shift and scale target image to have same statistics as the source imagesame statistics as the source image

LL Color Space Color Space

Convert RGB Convert RGB triplets to LMS triplets to LMS cone spacecone space

Take logarithmTake logarithm

Rotate axesRotate axes

Why not use RGB space?Why not use RGB space?Input images Output images

RGB

L

Color Transfer ExampleColor Transfer Example

Color Transfer ExampleColor Transfer Example

Color Transfer ExampleColor Transfer Example

Color Processing SummaryColor Processing Summary

• Changing the statistics along each axis Changing the statistics along each axis independently allows one image to independently allows one image to resemble a second imageresemble a second image

• If the composition of the images is very If the composition of the images is very unequal, an approach using small unequal, an approach using small swatches may be used succesfullyswatches may be used succesfully

Tone ReproductionTone Reproduction

Tone ReproductionTone Reproduction

Global vs. LocalGlobal vs. Local

• GlobalGlobal– Scale each pixel according to a fixed curveScale each pixel according to a fixed curve– Key issue: shape of curveKey issue: shape of curve

• LocalLocal– Scale each pixel by a local averageScale each pixel by a local average– Key issue: size of local neighborhoodKey issue: size of local neighborhood

Global OperatorsGlobal Operators

TumblinWard

Ferwerda

Global OperatorsGlobal Operators

TumblinWard

Ferwerda

Local OperatorLocal Operator

Pattanaik

Spatial ProcessingSpatial Processing• Light reaches the retina and is detected Light reaches the retina and is detected

by rods and conesby rods and cones

• The number of rods and cones is much The number of rods and cones is much larger than the number of nerves larger than the number of nerves leaving the eyeleaving the eye

• Hence, data reduction occurs in the Hence, data reduction occurs in the retinaretina

Spatial ProcessingSpatial Processing

• Certain aspects of natural images Certain aspects of natural images are more important than othersare more important than others

• For example, contrast edges need For example, contrast edges need to be detected with accuracy, to be detected with accuracy, whereas slow gradients do not whereas slow gradients do not need to be perceived at high need to be perceived at high resolutionresolution

Spatial ProcessingSpatial Processing• Circularly symmetric Circularly symmetric

receptive fieldsreceptive fields

• Centre-surround Centre-surround mechanismsmechanisms– Laplacian of GaussianLaplacian of Gaussian– Difference of GaussiansDifference of Gaussians– BlommaertBlommaert

• Scale space modelScale space model

Scale Space (Histogram Scale Space (Histogram Equalized Images)Equalized Images)

Tone Reproduction IdeaTone Reproduction Idea

• Modify existing global Modify existing global operator to be a local operator to be a local operator, e.g. Greg operator, e.g. Greg Ward’sWard’s

• Use spatial processing Use spatial processing to determine a local to determine a local adaptation level for adaptation level for each pixeleach pixel

5.2

4.0

4.0max

max

5.2

4.0

4.0max

max

),(219.12219.1),(

219.12219.1),(

yxL

L

LyxLL

L

L

LyxLL

a

d

doutput

w

d

doutput

Blommaert Brightness ModelBlommaert Brightness Model

22

2

22

1 skr

ii

iesk

R

ii RvuLsyxV ),(),,(

),,(2),,(),,()(),,(

12

21

syxVs

syxVsyxVsWsyxV

ns

s

syxVyxB0

),,(),(

Gaussian filter

Center/surround

Neural response

Brightness

BrightnessBrightness

ns

s

syxVyxB0

),,(),(

Scale Selection AlternativesScale Selection Alternatives

0

)),(,,(),(ssyxsyxVyxB

n

m

Mean value

Thresholded )),(,,(: yxsyxVs mm

How large should a local neighborhood be?

Mean ValueMean Value

0

)),(,,(),(ssyxsyxVyxB

n

m

ThresholdedThresholded

),,(: mm syxVs

Tone-mappingTone-mapping

5.2

4.0

4.0max

max

1

),(219.12219.1),(

)),(,,(),(

yxL

L

LyxLL

yxsyxVyxL

a

d

doutput

maLocal adaptation

Greg Ward’s tone-mapping with local adaptation

ResultsResults• Good results, but something odd about Good results, but something odd about

scale selection:scale selection:

• For most pixels, a large scale was For most pixels, a large scale was selectedselected

• Implication: a simpler algorithm should Implication: a simpler algorithm should be possiblebe possible

Simplify AlgorithmSimplify Algorithm

),(),(

),(1),(

),(219.12219.1),(

5.2

4.0

4.0max

max

yxLLayxL

yxLyxLL

yxL

L

LyxLL

ww

output

a

d

doutput

Greg Ward’s tone-mapping with local adaptation

Simplify

Fix overall lightness of image

Global Operator ResultsGlobal Operator Results

WardOur method

Global Operator ResultsGlobal Operator Results

WardOur method

Global Global Local Local

)),(,,(1),(

),(1),(

1 yxsyxVyxLL

yxLyxLL

moutput

output

Global operator

Local operator

Local Operator ResultsLocal Operator Results

Global

Local

Local Operator ResultsLocal Operator Results

Global Local Pattanaik

SummarySummary• Knowledge of the Human Visual System Knowledge of the Human Visual System

can help solve engineering problemscan help solve engineering problems

• Color and spatial processing Color and spatial processing investigatedinvestigated

• Direct applications shownDirect applications shown

Ongoing ResearchOngoing Research• Natural Image StatisticsNatural Image Statistics

• Applications:Applications:– Reconstruction filtersReconstruction filters– Perlin noisePerlin noise– Fractal terrainsFractal terrains

Ongoing ResearchOngoing ResearchImpoverished environments

Future WorkFuture Work

This presentation

AcknowledgmentsAcknowledgments• Thanks to my colaborators: Peter Thanks to my colaborators: Peter

Shirley, Jim Ferwerda, Mike Stark, Shirley, Jim Ferwerda, Mike Stark, Mikhael Ashikhmin, Bruce Gooch, Tom Mikhael Ashikhmin, Bruce Gooch, Tom TrosciankoTroscianko

• This work sponsored by NSF grants 97-This work sponsored by NSF grants 97-96136, 97-31859, 98-18344, 99-78099 96136, 97-31859, 98-18344, 99-78099 and by the DOE AVTC/VIEWSand by the DOE AVTC/VIEWS