CS 395/495-25: Spring 2004CS 395/495-25: Spring 2004

IBMR:IBMR:Poisson SolversPoisson Solvers

Can Reconstruct Images Can Reconstruct Images from their Changesfrom their Changes

Jack TumblinJack Tumblinjet@cs.northwestern.edujet@cs.northwestern.edu

Do pixels describe what we see?Do pixels describe what we see?

What We WantWhat We Want What We GetWhat We Get

What do you see?What do you see?

What part has constant intensity?What part has constant intensity?

AA

BB

What do you see?What do you see?Humans don’t sense intensities reliably, but infer them from changesHumans don’t sense intensities reliably, but infer them from changes

BB intensity is intensity is constantconstant, , AA is is darker on rightdarker on right

AA

BB

What do you see?What do you see?Humans don’t sense intensities reliably, but infer them from changesHumans don’t sense intensities reliably, but infer them from changes

(tol’djah!)(tol’djah!)

AA

BB

What do you see?What do you see?

What part has constant intensity?What part has constant intensity?

XX

YY

What do you see?What do you see?

What part has constant intensity?What part has constant intensity? NEITHER! NEITHER!

XX

YY

Humans don’t sense intensities reliably, but infer them from changesHumans don’t sense intensities reliably, but infer them from changes

What do you see?What do you see?

What part has constant intensity?What part has constant intensity? NEITHER! NEITHER!

XX

YY

ConstantConstant

Humans don’t sense intensities reliably, but infer them from changesHumans don’t sense intensities reliably, but infer them from changes

What do you see?What do you see?

Example: aren’t all the dots white? (http://udel.edu/~jgephart/fun2.htm)

Humans don’t sense intensities reliably, but infer them from changesHumans don’t sense intensities reliably, but infer them from changes

Why Pixels Could be Improved:Why Pixels Could be Improved:• People see (or think they see) People see (or think they see) changeschanges

finite features that may have infinite bandwidthfinite features that may have infinite bandwidth occlusion, depth, collision time, trajectory changes, occlusion, depth, collision time, trajectory changes,

corner, cone tip, boundaries, edges, occlusions, corner, cone tip, boundaries, edges, occlusions, shadow details, contact points, velocity & direction shadow details, contact points, velocity & direction changes...changes...

Pixels only Pixels only approximateapproximate changes, changes, and approximate discontinuous changes poorly;and approximate discontinuous changes poorly;

object boundaries, silhouettes, etc.object boundaries, silhouettes, etc.They force indirect estimation...They force indirect estimation...

How? Retinal Receptive Fields…How? Retinal Receptive Fields…• 130M Photoreceptors130M Photoreceptors1M optic nerve fibers1M optic nerve fibers

• Center-Surround Antagonism:Center-Surround Antagonism:Out Out Center - (avg surround) Center - (avg surround)

• Complementary Complementary ON-center, OFF-center typesON-center, OFF-center types

• Center responds quickly;Center responds quickly; Surround responds more slowlySurround responds more slowly

• Output: ‘recent local change’Output: ‘recent local change’

++ -- -- -- -- --

-- -- --

++++++ ++++

++ ++ ++--

Complementary Receptive FieldsComplementary Receptive Fields• Retina is ~differential for small signalsRetina is ~differential for small signals

– Better SNRBetter SNR– Can signal ambiguity (eyes closed, etc)Can signal ambiguity (eyes closed, etc)– Allows quality/fault detectionAllows quality/fault detection

++ -- -- -- -- --

-- -- --

++++++ ++++

++ ++ ++--

ctr/surrctr/surr

Firing Rate (Hz)Firing Rate (Hz)

1010

100100

5050

1010110.10.1ctr/surrctr/surr

1010

100100

5050Firing Firing Rate (Hz)Rate (Hz)

Yarbus (1950s): Pioneer of Yarbus (1950s): Pioneer of Retinal Stabilization ExperimentsRetinal Stabilization Experiments

(inspired a flood of others…)(inspired a flood of others…)

‘‘BUT BUT HEREHERE is a is a Big ring of Big ring of VERY strong change!’VERY strong change!’

‘Not much to see.(pink-ish?)’

‘mm, nothing much. (green-ish?)’

Strongly ImpliesStrongly Implies‘‘Filling In’ requiresFilling In’ requiresNystagmus forNystagmus fortemporal transients...temporal transients...

‘Not much to see.(pink-ish?)’

‘mm, Not much to see. (green-ish?)’

What ‘Changes’ do we Sense?What ‘Changes’ do we Sense?• Intensity (luminance) vs. local positionIntensity (luminance) vs. local position• Color (chrominance) vs. local positionColor (chrominance) vs. local position• Intensity vs. time (‘flicker’)Intensity vs. time (‘flicker’)• Color vs. timeColor vs. time• VERY weak, low-res: overall intensityVERY weak, low-res: overall intensity• Inertial changes: movement, velocity…Inertial changes: movement, velocity…

Compensated eye moves Compensated eye moves (saccade, glissade, smooth-pursuit…(saccade, glissade, smooth-pursuit…

• Higher-level attributes? Umm, er, uh,….Higher-level attributes? Umm, er, uh,….

‘‘Digital’ Image: a 2D Grid of NumbersDigital’ Image: a 2D Grid of Numbers• NO intrinsic meaning—use it for NO intrinsic meaning—use it for anything:anything: reflectance, transparency, illumination, normal reflectance, transparency, illumination, normal

direction, material, velocity. BUT usually direction, material, velocity. BUT usually ‘intensity’‘intensity’

xx

yy

xx

yy

2D Images Described by Change?2D Images Described by Change?• Image intensity as height field Image intensity as height field f(x,y)f(x,y)::• 11stst derivative— derivative—

Gradient: the ‘uphill’ vector at point x,y Gradient: the ‘uphill’ vector at point x,y = = f(x,y) = (f(x,y) = (f(x,y)f(x,y)//xx, , f(x,y)f(x,y)//yy) = ) = ff

xx

yy

f(x,y)

f(x,y)

2D Images Described by Change?2D Images Described by Change?• Image intensity as height field Image intensity as height field f(x,y)f(x,y)::• 22stst derivative— derivative—

Gradient: the ‘uphill’ vector at point x,y Gradient: the ‘uphill’ vector at point x,y = = f(x,y) = (f(x,y) = (f(x,y)f(x,y)//xx, , f(x,y)f(x,y)//yy) = ) = ff

xx

yy

f(x,y)

f(x,y)

Review: Div, Grad and CurlReview: Div, Grad and Curl• Formalized, computable ‘Local Change’Formalized, computable ‘Local Change’