image formation & camera...

Image Formation & Camera FundamentalsRODNEY DOCKTER

APRIL 3 2019

1

Outline• Physics Review

• A Brief History Of Capturing Images

• How are images formulated?

• Camera Models: Lenses, Depth of Field, Field of View

• Human Eye

• Digital Image Representation

2

Electromagnetic Spectrum

3

Electromagnetic Spectrum

4

• Radio Waves – Wireless Communication

• Microwaves – Ramen Noodles

• Infrared – Heat Waves

• Visible Light – What your eyes see

• Ultraviolet – Sunburn, curing silicone, invisible patterns

• X-Rays – Penetrates Tissue

•Gamma Rays – Most Energetic

Visible Spectrum

5

Visible light extends in wavelength from ~400 to ~700 nanometers

Quantum Theory of Light

6

• Isaac Newton, 1704 - Light is a stream of particles traveling in a straight line. Each particle is called a quantum and each quantum of light is a photon. The intensity of light is measured by the number of photons.

• Refraction occurs when light enters a different medium causing the velocity of light to change. This change then bends the direction of the light.

• When white light enters a prism the light is split into the full spectrum. Shorter wavelengths (violet) are refracted at a greater angle than longer wavelengths (red).

How do we capture an image?

7

Image Formation• Two parts to the image formation process:

• (1) The Geometry, which determines where in the image the plane the projection of a point in the scene will be located.

• (2) The Physics of Light, which determines the brightness of a point in the image plane.

• Simple model: f(x,y) = i(x,y) r(x,y)

• i: illumination, r: reflectance

8

Image Formation• Design of a basic camera, Version 1:• Put a piece of film in front of an object

• Coat film with light sensitive material (Changes depending on light intensity)

• Do we get a reasonable image? Probably not… Blurring

9

Image Formation• Design of a basic camera, Version 2:• View through a hole: Reduces blurring

• The opening is known as aperture.

• How does this transform the image? (It flips it)

10

Image Formation• Camera Obscura (4th Century BC China -> 16th century Leonardo da Vinci)

• Limitations:

• Aperture too big: blurry image

•Aperture to small: Requires long exposure time

•Aperture much to small: diffraction through pinhole + blurry image.

11

Image Formation• Adding a lens

• Lens based Camera Obscura, 1568

12

Image Formation• 1826 – The first photograph – Joseph Nicéphore Niépce

• Camera Obscura focused onto a pewter plate coated with asphalt. ~8 hour exposure.

• Basic mathematical model – Pinhole Camera

13

Image Formation• Conventional Camera Design

• Early film based cameras, 1960’s

14

Image Formation

15

Animal Eye:Early AnatomicalStudies

Camera Obscura +Pinhole camera model4th-15th centuries

First PhotographicCamera. Niépce1820’s

Image Formation

16

Camera Models

17

Camera Models• Pinhole Camera Model

• The simplest method to form an image of a 3D scene on a 2D surface

• Rays of light pass through a “pinhole” and form an inverted images of the object on the image plane

18

http://web.anagram.at

Perspective Projection:

𝑥 =𝑓 𝑋

𝑍𝑦 =

𝑓 𝑌

𝑍

𝑃 𝑋, 𝑌, 𝑍 → 𝑝(𝑥, 𝑦)

f: focal length, distance between pinhole and image plane

Camera Models

19

• Is an image being formed on the screen?

• Yes, but not a clear one

𝒓′

𝑓′=𝒓

𝑧

𝑥′

𝑓′=𝑥

𝑧

𝑦′

𝑓′=𝑦

𝑧(𝑟′≠ 𝑓′)

Camera Models

20

• What is the effect of aperture size?

• Large aperture: light from the sourcespreads across the image (i.e. not properly focused) making it blurry

• Small aperture: reduces blurring butit limits the amount of light enteringthe camera and causes light diffraction

Camera Models

21

• Shrinking the aperture

•Why not make the aperture as small as possible?• Less light gets through

Camera Models

22

• Shrinking the aperture more

• What happens if we keep decreasing aperture size?

• When light passes through a small hole, it does not travel in a straight line and is scattered in many directions (diffraction)

Camera Models

23

• Pinhole too big – many directionsare averaged, blurring the image

• Pinhole too small – diffractioneffects blur the image

• Generally, pinhole cameras aredark because a very small amountof rays from a point in the scene reach the image plane

Camera Models

24

• Problems with pinholes

• Pinhole size (aperture) must be SMALL to obtain a clear image

• However as pinhole size gets smaller, LESSlight is received by the image plane

• If pinhole is comparable to wavelength ofincoming light, DIFFRACTION effects blur the image

• OPTIMAL pinhole size is obtained when:

pinhole diameter: 𝑑 = 2 𝑓′λ

Example: 𝑓′ = 50𝑚𝑚, λ = 600𝑛𝑚 (𝑟𝑒𝑑)𝑑 = 0.36𝑚𝑚

Lenses

25

• The reason for lenses:

• Gather more light from each point in the scenePinhole

Lens

Lenses

26

• Upgrading from a pinhole to a lens.

• Lens redirect light rays emanating from the object

• Lenses improve image quality, leading to sharper images

Lenses

27

• A lens focuses parallel rays onto a single focal point• Focal point at a distance ‘f’ beyond the plane of the lens

• ‘f’ is a function of the shape and index of the refraction of the lens

• Aperture of diameter ‘D’ restricts the range of the rays• Aperture may be on either side of the lens

• Lenses are typically spherical (Easier to produce)

Focal Point

Optical Center

Lenses

28

• Properties of “thin” lenses (ideal lenses)

• Light rays passing through the center are not deviated

• Light rays passing through a point far away from the center are deviated more

Lenses

29

• Properties of “thin” lenses (ideal lenses)

• All parallel rays converge to single point

• Rays passing perpendicular to the lens pass through the Focal Point

Lenses

30

• Properties of “thin” lenses (ideal lenses)

• The plane parallel to the lens at the focal point is the called the Focal Plane

• The distance between the lens and the focal plane is the Focal Length (f)

Lenses

31

• Thin lens equation:

• Any object satisfying this equation is in focus

1

𝑑𝑂+

1

𝑑𝑖=1

𝑓

Lenses

32

• Lets verify the thin lens equation

• Assume an object at distance u from the lens plane

Lenses

33

• Using similar triangles:

Lenses

34

• Using similar triangles:

Lenses

35

• Combining these equations:

Lenses

36

• The thin lens assumption assumes the lens has no thickness (this is not physically possible)

• When our lens does have thickness, the shape of the image plane becomes slightly concave

• By adding more elements to lens, we can correct for this and the focus region can be made roughly planar

Optical Axis

Aperture

D

f

Lenses

37

• Lens Aperture (f/Number)

• Lenses have the ability to control the amount of light that reaches the camera sensor.

• The f/number (𝑛𝑓) is the ratio of the focal length of the lens to the diameter of the aperture (𝐷 = 2𝑟)

• Lens Power (P): Reciprocal of focal length

𝑛𝑓 =𝑓

2𝑟

P =1

𝑓

Depth of Field

38

• A lens focuses light onto the film

• There is a specific distance at which objects are “in focus”

• Other points project to a “circle of confusion in the image

• Lens shape affects this distance

Depth of Field

39

• The size of the blur circle is proportional to aperture size

Depth of Field

40

• Changing the aperture size affects depth of field

• A smaller aperture increases the range in which an object is approximately in focus

• Small aperture also reduces the amount of light – increased exposure needed to compensate

Depth of Field

41

• Changing the aperture of a camera also changes the amount of the image that is in focus

• This amount (or range of depths) is called the depth of field

Depth of Field

42

• Changing the aperture size affects depth of field

• Smaller aperture increases the range in which an object is approximately in focus (but we need to increase exposure time)

• A larger aperture decreases the depth of field (but lower exposure time required

Depth of Field

43

• Smaller aperture increases the range in which an object is approximately in focus

Depth of Field

44

• The range of depths over which the scene is approximately sharp (i.e. in focus)

Depth of Field

45

Large aperture,Small depth of field

Small aperture,Large depth of field

Depth of Field

46

• Small Apetures(e.g. f11, f16, f22) only let a smallamount of light through

• Large Apertures(e.g. f4, f5.6, f8) let through a lot of light

• For use a sunny day, you mightneed to use a small aperture toget correct exposure

Field of View (Zoom)

47

• The cone of viewing directions of the camera

• Inversely proportional to focal length

Field of view is governed by the size of the camera retina:

𝜑 = 𝑡𝑎𝑛−1𝑑

2𝑓

Field of View (Zoom)

48

• Increasing focallength

• Decreased fieldof view…

Lens Flaws

49

• Chromatic Aberration

• Lens has different refractive indices for different wavelengths

• Can cause fringing: lens cannot focus all the colors from a single point

Lens Flaws

50

• Chromatic Aberration example:

Lens Flaws

51

• Radial Distortion

• Straight lines becomes distorted as we move further away from the center of the image

• Deviations are most noticeable for rays that pass through the edge of the lens

Lens Flaws

52

• Radial Distortion example

No Distortion Pin Cushion Barrel

Lens Flaws

53

• Tangential Distortion

• Lens is not exactly parallel to the imaging plane

• Image plane becomes skewed

The Human Eye

54

• Cameras are a copy of the human eye

The Human Eye

55

• We intentionally made cameras that act similar to the human eye

• … Because that’s what we understand

The Human Eye

56

• The primary difference is what captures the light

The Human Eye

57

• The other major difference is that the human eyes perceive depth very well

• Human vision system transforms light stimuli on the retina into a mental construct of a stable 3D world

• Our 3D perception of the world is invariant to a wide range of changes in illumination, size, shape, and brightness of the scene

• The best 3D cameras in the world do not come close to human 3D perception

Digital Image Formation

58

Digital Image(Grayscale)

Digital Image Formation• First digitally scanned photograph

• 1957, 176x176 pixels

• Origins of digital cameras come fromdigital scanners

59

Digital Image Formation• A digital camera replaces film with a sensor array

• Each cell in the array contains a light-sensitive diode that converts photons to electrons

• Two common varieties:

• Charge coupled Device (CCD)

• Complementary Metal OxideSemiconductor (CMOS)

60

Digital Image Formation

61

8 Bits/Pixel

0

128

255

Digital Image Formation• When a continuous scene is imaged on the array (grid) formed by a CMOS, the continuous image is divided into discrete elements

• The picture elements (pixels) that are captured represented a spatially sampled version of the image

62

Digital Image Formation• Our spatially sampled image can then be represented by numbers according to the brightness of each pixel

63

Digital Image Formation

64

Original Image Region of Interest Matrix Representation(R,G,B)

Digital Image Formation

65

Digital Image Formation• Digital Camera Properties

• Focus – Shifts the depth that is in focus

• Focal Length – Adjusts the zoom (wide angle or telephoto)

• Aperture – Adjusts the depth of field and amount of light let into sensor

• Exposure Time – How long an image is exposed. More exposure = more light, more blur

• ISO – Adjusts the sensitivity of the image sensor. Basically a gain function for digital cameras to increase brightness.

66

Digital Image Formation• Camera Exposure

• ISO Number• Sensitivity of the film or the sensor

• Can go as high as 1600 or 3200

• Shutter Speed• How fast the shutter is opened and closed

• f/stop• The size of the aperture

• 1.0 ~ 32

67

Digital Image Formation• Exposure:

• Defined as the total amount of light falling on the film

• Exposure = Illuminance * Time

68

Digital Image Formation• Exposure is controlled by the shutter. When closed the film is not exposed to light.

• Exposure time is simply the time interval between opening and closing the shutter

69

Digital Image Formation• Long Exposure

70

Digital Image Formation• Short Exposure

71

Digital Image Formation• Varying Exposure Levels

72

Image File Formats• Many image file formats adhere to the simple model shown below (line by line, no breaks)

• The header contains at least the width and height of the image

• Most headers begin with a signature or “magic number” (a short sequence of bytes for identifying the file format)

73

Image File Formats• GIF (Graphics Interchange Format)

• PNG (Portable Network Graphics)

• JPEG (Joint Photographic Experts Group)

• TIFF (Tagged Image File Format)

• PGM (Portable Gray Map)

• FITS (Flexible Image Transport System)

74

Image File Formats• PBM/PGM/PPM Format

• A very basic image format for greyscale images (Not really used anymore)

• Closely-related formats:• PBM (Portable Bitmap) for binary images (1 bit/pixel)

• PPM (Portable Pixelmap) for color images (24 bits/pixel)

75

Image File Formats• JPEG Format

• Comparatively complex image format (very common)

• Format consists of a sequence of markers or marker segments

• Each marker indicates the start of a marker segment that contains data bytes according to:• Marker: FF xx s1 s2 [data bytes]

76