Image Formation and CaptureImage Formation and Capture

Acknowledgment:Acknowledgment: some figures by B. Curless, E. Hecht, W.J. Smith, B.K.P. Horn, and A. Theuwissen some figures by B. Curless, E. Hecht, W.J. Smith, B.K.P. Horn, and A. Theuwissen

Image Formation and CaptureImage Formation and Capture

• DevicesDevices

• Sources of ErrorSources of Error

Real worldReal worldReal worldReal world OpticsOpticsOpticsOptics SensorSensorSensorSensor

OpticsOptics

• Pinhole cameraPinhole camera

• LensesLenses

• Focus, aperture, distortionFocus, aperture, distortion

PinholePinholeObjectObject

Pinhole cameraPinhole camera

Pinhole CameraPinhole Camera

• ““Camera obscura” – known since Camera obscura” – known since antiquityantiquity

ImageImage

Image planeImage plane

Pinhole CameraPinhole Camera

• ““Camera obscura” – known since antiquityCamera obscura” – known since antiquity

• First recording in 1826 onto a pewterFirst recording in 1826 onto a pewterplate (by Joseph Nicéphore Niepce)plate (by Joseph Nicéphore Niepce)

PinholePinholeObjectObject

ImageImage

Pinhole cameraPinhole camera

Image planeImage plane

Pinhole Camera LimitationsPinhole Camera Limitations

• Aperture too big: blurry imageAperture too big: blurry image

• Aperture too small: requires long Aperture too small: requires long exposure or high intensityexposure or high intensity

• Aperture much too small: diffraction Aperture much too small: diffraction through pinhole through pinhole blurry image blurry image

LensesLenses

• Focus a bundle of rays from a scene Focus a bundle of rays from a scene point onto a single point on the imagerpoint onto a single point on the imager

• Result: can make aperture biggerResult: can make aperture bigger

Camera AdjustmentsCamera Adjustments

• Iris?Iris?– Changes apertureChanges aperture

• Focus?Focus?– Changes Changes ddii

• Zoom?Zoom?– Changes Changes f f and sometimes and sometimes ddii

Zoom Lenses – VarifocalZoom Lenses – Varifocal

Zoom Lenses – ParfocalZoom Lenses – Parfocal

• For a given For a given ddii, “perfect” focus at only one , “perfect” focus at only one ddoo

• In practice, OK for some range of depthsIn practice, OK for some range of depths– Circle of confusionCircle of confusion smaller than a pixel smaller than a pixel

• Better depth of field with smaller aperturesBetter depth of field with smaller apertures– Better approximation to pinhole cameraBetter approximation to pinhole camera

• Also better depth of field with wide-angle Also better depth of field with wide-angle lenseslenses

Focus and Depth of FieldFocus and Depth of Field

Field of ViewField of View

• Q: What does field of view of camera Q: What does field of view of camera depend on?depend on?– Focal length of lensFocal length of lens

– Size of imagerSize of imager

– Object distance?Object distance?

Computing Field of ViewComputing Field of View

xoxo

didi

xixi

dodo

1/1/ddoo + 1/ + 1/ddii = 1/ = 1/ff

xxo o / / ddoo = = xxi i / / ddii

= 2 tan= 2 tan-1-1 ½ ½ xxi i

(1/(1/ff1/1/ddoo))Since typically Since typically ddoo >> >> f,f,

2 tan2 tan-1-1 ½ ½ xxi i / / ff

tan tan /2 = ½ /2 = ½ xxo o // ddoo

xxi i / / ff

ApertureAperture

• Controls amount of lightControls amount of light

• Affects depth of fieldAffects depth of field

• Affects distortion (since thin-lens Affects distortion (since thin-lens approximation is better near center of approximation is better near center of lens)lens)

ApertureAperture

• Aperture typically given as “Aperture typically given as “ff-number”-number”(also “(also “ff-stops” or just “stops”)-stops” or just “stops”)

• What is What is ff /4?/4?– Aperture is ¼ the focal lengthAperture is ¼ the focal length

SensorsSensors

• FilmFilm

• VidiconVidicon

• CCDCCD

• CMOSCMOS

VidiconVidicon

• Best-known in family ofBest-known in family of“photoconductive video cameras” “photoconductive video cameras”

• Basically television in reverseBasically television in reverse

Electron GunElectron Gun

Photoconductive PlatePhotoconductive Plate

Lens SystemLens System

+ + + ++ + + +

Scanning Electron BeamScanning Electron Beam

MOS CapacitorsMOS Capacitors

• MOS = Metal Oxide SemiconductorMOS = Metal Oxide Semiconductor

Gate (wire)Gate (wire)

SiOSiO22 (insulator) (insulator)

pp-type silicon-type silicon

MOS CapacitorsMOS Capacitors

• Voltage applied to gate repels positive Voltage applied to gate repels positive “holes” in the semiconductor“holes” in the semiconductor

+10V+10V

Depletion regionDepletion region(electron “bucket”)(electron “bucket”)

+ + + + + ++ + + + + ++ + + + + ++ + + + + +

MOS CapacitorsMOS Capacitors

• Photon striking the material createsPhoton striking the material createselectron-hole pairelectron-hole pair

+10V+10V

+ + + + + ++ + + + + +

PhotonPhoton

++

Charge TransferCharge Transfer

• Can move charge from one bucket to Can move charge from one bucket to another by manipulating voltagesanother by manipulating voltages

CMOS ImagersCMOS Imagers

• Recently, can manufacture chips that Recently, can manufacture chips that combine photosensitive elements and combine photosensitive elements and processing elementsprocessing elements

• Benefits:Benefits:– Partial readoutPartial readout

– Signal processingSignal processing

– Eliminate some supporting chips Eliminate some supporting chips low cost low cost

ColorColor

• 3-chip vs. 1-chip: quality vs. cost3-chip vs. 1-chip: quality vs. cost

Errors in Digital ImagesErrors in Digital Images

• What are some sources of error in this What are some sources of error in this image?image?

Sources of ErrorSources of Error

• Geometric (focus, distortion)Geometric (focus, distortion)

• Color (1-chip artifacts, chromatic Color (1-chip artifacts, chromatic aberration)aberration)

• Radiometric (cosine falloff, vignetting)Radiometric (cosine falloff, vignetting)

• Bright areas (flare, bloom, clamping)Bright areas (flare, bloom, clamping)

• Signal processing (gamma, Signal processing (gamma, compression)compression)

• NoiseNoise

Monochromatic AberrationsMonochromatic Aberrations

• Real lenses do not follow thin lens Real lenses do not follow thin lens approximation because surfaces are approximation because surfaces are spherical (manufacturing constraints)spherical (manufacturing constraints)

• Result: thin-lens approximation only Result: thin-lens approximation only valid iffvalid iffsin sin

Spherical AberrationSpherical Aberration

• Results in blurring of image, focus shifts Results in blurring of image, focus shifts when aperture is stopped downwhen aperture is stopped down

• Can vary with the way lenses are Can vary with the way lenses are orientedoriented

DistortionDistortion

• Pincushion or barrel Pincushion or barrel radial distortionradial distortion

• Varies with placement of apertureVaries with placement of aperture

• Varies with placement of apertureVaries with placement of aperture

DistortionDistortion

• Varies with placement of apertureVaries with placement of aperture

DistortionDistortion

• Varies with placement of apertureVaries with placement of aperture

DistortionDistortion

First-Order Radial DistortionFirst-Order Radial Distortion

• Goal: mathematical formula for distortionGoal: mathematical formula for distortion

• If small, can be approximated by “first-If small, can be approximated by “first-order” formula (like Taylor series order” formula (like Taylor series expansion):expansion):

• Higher-order models are possibleHigher-order models are possible

rr’ = ’ = rr (1 + (1 + rr22))rr = = ideal distance to center of imageideal distance to center of imagerr’ = ’ = distorted distance to center of imagedistorted distance to center of image

Chromatic AberrationChromatic Aberration

• Due to dispersion in glass (focal length Due to dispersion in glass (focal length varies with the wavelength of light)varies with the wavelength of light)

• Result: color fringesResult: color fringes

• Worst at edges of imageWorst at edges of image

• Correct by buildingCorrect by buildinglens systems withlens systems withmultiple kinds of glassmultiple kinds of glass

Correcting for AberrationsCorrecting for Aberrations

• High-qualityHigh-qualitycompound lensescompound lensesuse multipleuse multiplelens elements tolens elements to“cancel out”“cancel out”distortion anddistortion andaberrationaberration

• Often 5-10 elements, more for extreme wide Often 5-10 elements, more for extreme wide angleangle

Other Limitations of LensesOther Limitations of Lenses

• Optical vignetting: less power per unit Optical vignetting: less power per unit area transferred for light at an oblique area transferred for light at an oblique angleangle– Transferred power falls off as cosTransferred power falls off as cos44 – Result: darkening of edges of imageResult: darkening of edges of image

• Mechanical vignetting: due to aperturesMechanical vignetting: due to apertures

Other Limitations of LensesOther Limitations of Lenses

• Flare: light reflectingFlare: light reflecting(often multiple times)(often multiple times)from glass-air interfacefrom glass-air interface

– Results in ghost images or hazinessResults in ghost images or haziness

– Worse in multi-lens systemsWorse in multi-lens systems

– Ameliorated by optical coatings (thin-film Ameliorated by optical coatings (thin-film interference)interference)

BloomBloom

• Overflow of charge in CCD bucketsOverflow of charge in CCD buckets– Spills to adjacent bucketsSpills to adjacent buckets

– Streaks (usually vertical) next to bright Streaks (usually vertical) next to bright areasareas

• Some cameras have “anti-bloom” Some cameras have “anti-bloom” circuitrycircuitry

Flare and BloomFlare and Bloom

TanakaTanaka

Dynamic RangeDynamic Range

• Most common cameras have 8-bitMost common cameras have 8-bit(per color channel) dynamic range(per color channel) dynamic range– With gamma, this can translate to more With gamma, this can translate to more

than 255:1than 255:1

• Too bright: clamp to maximumToo bright: clamp to maximum

• Too dim: clamp to 0Too dim: clamp to 0

• Specialty cameras with higher dynamic Specialty cameras with higher dynamic range (usually 10-, 12-, and 16-bit)range (usually 10-, 12-, and 16-bit)

High Dynamic Range (HDR)High Dynamic Range (HDR)from Ordinary Camerasfrom Ordinary Cameras

• Take pictures of same scene with Take pictures of same scene with different shutter speedsdifferent shutter speeds

• Identify regions clamped to 0 or 255Identify regions clamped to 0 or 255

• Average other pixels, scaled by 1 / Average other pixels, scaled by 1 / shutter speedshutter speed

• Can extend dynamic range, but Can extend dynamic range, but limitations of optics and imager (noise, limitations of optics and imager (noise, flare, bloom) still applyflare, bloom) still apply

GammaGamma

• Vidicon tube naturally has signal that variesVidicon tube naturally has signal that varieswith light intensity according to a power law:with light intensity according to a power law:Signal = Signal = EE, , 1/2.5 1/2.5

• CRT (televisions) naturally obey a power law CRT (televisions) naturally obey a power law with gamma with gamma 2.5 2.5

• Result: standard for video signals hasResult: standard for video signals hasa gamma of 1/2.5a gamma of 1/2.5

• CCDs and CMOS linear, but gamma often CCDs and CMOS linear, but gamma often appliedapplied

NoiseNoise

• Thermal noise: in all electronicsThermal noise: in all electronics– Noise at all frequenciesNoise at all frequencies

– Proportional to temperatureProportional to temperature

– Special cooled cameras available for low Special cooled cameras available for low noisenoise

• Shot noise: discrete photons / electronsShot noise: discrete photons / electrons– Shows up at extremely low intensitiesShows up at extremely low intensities

– CCDs / CMOS can have high efficiency – CCDs / CMOS can have high efficiency – approaching 1 electron per photonapproaching 1 electron per photon

NoiseNoise

• 1/f noise – inversely proportional to 1/f noise – inversely proportional to frequencyfrequency– Not completely understood – shows up in Not completely understood – shows up in

semiconductorssemiconductors

– Can be dominant source of noiseCan be dominant source of noise

• All of the above apply for imager and All of the above apply for imager and amplifieramplifier

Filtering NoiseFiltering Noise

• Most common method – simple blurMost common method – simple blur– e.g., convolution with Gaussiane.g., convolution with Gaussian

• Adaptive filters to prevent bleed acrossAdaptive filters to prevent bleed across intensity edges intensity edges

• Other filters for specialized situationsOther filters for specialized situations– e.g., “despeckling” (median filters) for dead e.g., “despeckling” (median filters) for dead

pixelspixels