computational photography - portal · computational photography • more than digital photography...
TRANSCRIPT
Computational Photography
Matthias Zwicker University of Bern
Fall 2012
Today • Course organization
• Course overview
• Image formation
Course organization Instructor
• Matthias Zwicker ([email protected])
Teaching Assistant
• Daniel Donatsch ([email protected])
Course organization Lecture
• Mondays, 14:00-16:00
• Engehaldenstrasse 8, Room 3
Exercises
• Mondays, 16:00-17:00
• Engehaldenstrasse 8, Room 3
Class web page • Class overview
http://www.cgg.unibe.ch/teaching/computational-photography
ILIAS • Use your campus account to log in
• Join course Magazin → Weitere Institutionen; Weiterbildungen und Studiengänge → BeNeFri Joint Master in Computer Science → HS2012 → 2012HS: 31051 Computational Photography
• Lecture slides
• Exercise description & material
• Additional reading material
• Forum
– Any questions and discussions related to class material and exercises
Exercises • 6 assignments
• Programming projects
– Matlab – Available in ExWi pool
• Exercises on paper
Exercises • Final grade: 40% exercises, 60% final exam • To qualify for final exam: need 70% of
exercise score • Late penalty
– 50% of original score – Exceptions for military service, illness
• Collaboration – Discussion among students is encouraged – Each student must write up and turn in his/her
own solution – If we detect copied material, you will need to talk
to us and explain your material in person; if we are not satisfied, you will not get credit
Final exam • Written, 105 minutes
• Bring two A4 sheets (4 pages) of hand written notes
• Relevant material: slides and exercises
– Wikipedia links not part of class material, but may be useful to better understand concepts discussed in class
• Date: February 2013
Prerequisites • Familiarity with
– Linear algebra (matrix calculations, linear systems of equations, least squares problems)
– Programming experience
Today • Course organization
• Course overview
• Image formation
Computational photography Topics of this class
• Role of computation, algorithms in digital photography today
• Algorithms to extend and improve capabilities of digital photography in the future
Photography Traditionally
• „Measuring light“
• Optics focuses light on sensor
• Sensor records image
• Sensors
– Digital – Film
http://en.wikipedia.org/wiki/Single-lens_reflex_camera http://en.wikipedia.org/wiki/Digital_single-lens_reflex_camera
Computational photography • More than digital photography
• Arbitrary computation between light measurement and final image – Light measured on sensor is not final image – Computation enhances and extends capabilities of
digital photography • Two „types“ of computation
1. Post-process after traditional imaging 2. Design of new camera devices that require
computation to form an image • Overview of recent research
http://en.wikipedia.org/wiki/Computational_photography
Removing imaging artifacts • Denoising & deblurring
http://www.cs.ust.hk/~quan/publications/yuan-deblur-siggraph07.pdf
Blurry + Output Noisy Algorithm
Removing imaging artifacts • High dynamic range images & tone mapping
Image manipulation • Panoramas
http://en.wikipedia.org/wiki/Image_stitching
Computational optics
Coded aperture
Captured image, slightly blurry everywhere
Computational optics
Recovered depth
Refocused image Sharp foreground, blurry background
Focus of class • „Fun with digital photography and
computer programming“
– Algorithms and computational techniques with potential applications in the consumer domain
– Mostly software, less hardware
• Recent research
What you will learn • Basic understanding of photography, light,
and color • Practical experience with implementation of
algorithms for image processing & computational photography
• Cool and creative applications of mathematical tools – Fourier transforms – Linear and non linear filtering – Optimization techniques (least squares, iteratively
re-weighted least squares, graph cuts) – Probabilistic models
• Many applications beyond processing images!
Related areas, not covered • Image processing for scientific applications
– Physics, biology, etc.
• Optics, lens design
• Photosensors, sensor design
• Computational imaging
– Tomography, radar, microscopy
• 3D imaging
• Using photo processing tools, e.g. Photoshop
• Artistical aspects of photography
Syllabus 1. Introduction, image formation 2. Color & color processing 3. Dynamic range & contrast 4. Sampling, reconstruction, & the frequency domain 5. Image restoration: denoising & deblurring 6. Image manipulation using optimization 7. Gradient domain image manipulation 8. Warping & morphing 9. Panoramas 10. Automatic alignment 11. Probabilistic image models 12. Light fields 13. Capturing light transport
http://www.cgg.unibe.ch/teaching/computational-photography
• Cameras, image artifacts
Image formation
Color • Color perception, color spaces, color
measurement, color processing
Dynamic range & contrast • HDR imaging
http://en.wikipedia.org/wiki/High_dynamic_range_imaging http://en.wikipedia.org/wiki/Tone_mapping
Sampling, reconstruction • Sampling artifacts
• Frequency domain analysis
Spatial Domain Frequency Domain
Image restoration • Denoising & deblurring
Blurry input Deblurred output
Estimated blur kernel (scaled) http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/
Image manipulation using optimization • Photomontage, matting, colorization
http://grail.cs.washington.edu/projects/photomontage/
http://www.cs.huji.ac.il/~yweiss/Colorization/
http://grail.cs.washington.edu/projects/digital-matting/image-matting/
Gradient domain manipulation • Poisson equation
http://portal.acm.org/citation.cfm?id=882269
Warping & morphing
Panoramas • Automatic alignment, stitching
http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/
Probabilistic models • Faces, textures
http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/Visio_SIG09.pdf
• Beyond 2D images
Light fields
http://www-graphics.stanford.edu/papers/fourierphoto/
Capturing light transport • Dual photography
http://www-graphics.stanford.edu/papers/dual_photography/
Today • Course organization
• Course overview
• Image formation
Models of light
Question • Why is there no image on a white piece of
paper?
Question • Why is there no image on a white piece of
paper?
• Receives all light rays
– Images from all viewpoints
• Need to select light rays for specifice image, viewpoint
• How?
• Invented by Alhazen, 10th century http://en.wikipedia.org/wiki/Pinhole_camera
Pinhole camera
Limitations • Small pinhole: sharper image, longer exposure
• Larger pinhole: blurrier image, shorter exposure
Camera model • Thin lens, aperture, shutter, film
Lenses • Gather more light
– Proportional to area of lens aperture
• Sharp image if focused
– Use refraction http://en.wikipedia.org/wiki/Refraction
Lens
http://en.wikipedia.org/wiki/Lens_(optics)
Scene point (emits or
reflects light) Image of
scene point
Lenses Pinhole Lens
6 sec. exposure 0.01 sec exposure
Thin lens model http://en.wikipedia.org/wiki/Thin_lens
• Theoretical model for well-behaved lenses
• Properties
1. All parallel rays converge at focal length
2. Rays through the center are not deflected
Same perspective image as pinhole placed at center of lens
Thin lens model • How are arbitrary rays deflected when
passing through a thin lens?
1. Parallel rays converge at focal length f
Thin lens model 2. Rays through center are not deflected
Thin lens model • Similar triangles
Thin lens model • More similar triangles
Thin lens model • Thin lens formula
• All rays passing through a single point y on a plane at distance in front of the lens will pass through a single point y’ at distance behind the lens
Thin lens model • Focus at infinity:
– Film at distance f
• Closest focusing distance:
– Film at infinity
Film plane
Object
Thin lens model • Out of focus film plane results in spherical
blur
Out of focus film planes
Spherical blur
Properties of real lenses • Mostly undesired!
• Aberrations
– Spherical aberration – Chromatic aberration
• Distortion
– Barrel distortion – Pincushion distortion
• Etc.
Barrel & pincushion distortion
Question • What‘s the advantage of a lens with a
short focal length? In what situation would this be useful?
• What‘s the advantage of a lens with a long focal length? In what situation would this be useful?
Camera model • Thin lens, aperture, shutter, film
Aperture • Amount of light captured at sensor
proportional to area of aperture
Aperture
Aperture • Blurriness of out of focus objects depends
on aperture size
• Aperture size determines depth of field: depth range that is sharp in image
Aperture
Depth of field
Circle of confusion • Also called „blur circle“
• Calculation of radius c – Lens focused at S1
– Object at S2 – Aperture A – Focal length f sensor
http://en.wikipedia.org/wiki/Circle_of_confusion
Proportional to A
f-number http://en.wikipedia.org/wiki/F-number
• Definition: (focal length) / (diameter of aperture)
• Large aperture means small f-number
• Practice: f-stops increase by factors of
– f/2.0, f/2.8, f/4, f/5.6, f/8 – Aperture area gets halved in each step
f-number • Smaller f-number
– Larger aperture – Capture more light – Small (shallow) depth of field
• Larger f-number
– Smaller aperture – Capture less light – Large depth of field
Camera model • Thin lens, aperture, shutter, film
Shutter speed http://en.wikipedia.org/wiki/Shutter_speed
• Determines time the film is exposed to light
• Amount of light captured is proportional to exposure time
• Long exposure leads to motion blur
Reciprocity • Amount of light captured stays
same if exposure is doubled and aperture area is halved (or vice versa)
Reciprocity • Which exposure/aperture combination?
Camera model • Thin lens, aperture, shutter, film
Film • Film/sensor responds roughly linearly to light
– „Double the amount of light leads to double the recorded value“
• Film speed: sensitivity of film to light
– Digital photography analog: sensor gain (scaling or amplification factor)
• Measured using ISO scale
– Linear: sensitivity is proportional to ISO value – „Double ISO value, halve the exposure time, get
same recorded value“
Film • Trade-off: higher gain, more noise
ISO 100 ISO 3200
Film • Underexposure
– Not enough light, image too dark • Overexposure
– Film or sensor is saturated – Clipping of highlight details
Good exposure Overexposure Underexposure
Conclusions • Simple camera model
– Thin lens, aperture, shutter, film
• Photographs often have undesired artifacts
– Distortions, color artifacts, blur, noise, under/overexposure
Goal
• Develop algorithms to remove artifacts after image is captured
References • „Photography“, by London, Upton, Stone
Next time • Color, color processing