Computational Photography and Image Manipulation

CMPT 469 / 985, Fall 2019

Instructor: Yağız Aksoy

3

Layer editing Compositing

Soft Color Segmentation

Aksoy, Aydin, Smolic, Pollefeys, “Unmixing-Based Soft Color Segmentation for Image Manipulation”, ACM Transactions on Graphics, 2017

4

5

6

Semantic Soft Segmentation

Input

Aksoy, Oh, Paris, Pollefeys, Matusik, “Semantic Soft Segmentation”, ACM Transactions on Graphics (Proc. SIGGRAPH), 2018

7

Computational Photography

Computational Photography

How can I use computational techniques to capture light in new ways?

How can I use computational techniques to breathe new life into the photograph?

How can I use computational techniques to synthesize and organize photo collections?

[Hays]

Comp Photo and Related Fields

• Computer Graphics: Models to Images

• Computer Vision: Images to Models

• Computational Photography: Images to Images

Google Pixel phones

Portrait mode iOS

Course outline(Tentative!)

[Wikipedia]

Alim Khan, emir of Bukhara,1911.

Week 1 – Imaging basics

Week 2 – Camera basics

The (in-camera) image processing pipeline

analog front-end

RAW image (mosaiced,

linear, 12-bit)white

balanceCFA

demosaicingdenoising

color transforms

tone reproduction

compressionfinal RGB

image (non-linear, 8-bit)

Week 3 – Frequency domain

Week 4 - High dynamic range (HDR) imaging

[example from www.dpreview.com] [Debevec and Malik, SIGGRAPH 1997]

Week 5 – Bilateral filtering

Week 6 - Color

Retinal vs perceived color.

Berlin, 1945/2010, Mehringdamm

Week 7 – Image compositions

Week 8 – Boundary minimization techniques

Scaling

Michael Rubinstein — MIT CSAIL – mrub@mit.edu

Week 9 – Transforms and panoramas

[Brown and Lowe, IJCV 2007]

+ =

Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

Week 10 - Stereo

Week 11 – Motion and Optical Flow

24G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics

14, 201-211, 1973.

Course structure

• Prerequisites: (i) A computer vision, computer graphics or visual computing course(ii) College-level linear algebra course

• Coding assignments - 4 x 15% = 60%Paper presentation - 30%Paper discussion - 3 x 3.333333% = 10%

Presentations

• Tuesdays: Paper presentations by you!

• CMTP 469: 8 minutes CMTP 985: 15 minutes

• Papers announced on Thursdays for the next Tuesday

• You can propose which papers you would like to present the week before your presentation related to the topic of the week

• SIGGRAPH, SIGGRAPH Asia, Transactions on Graphics, Eurographics, CVPR, ICCV, ECCV, ICCP

• Week assignments for everyone next week

Paper discussions

• 4 minutes discussion after each presentation

• Everyone will have 3 weeks assigned for paper discussion• 2, 5, and 7 weeks after your own presentation (roll over to week 1)

• Interesting closely-related work, shortcomings/strengths of the paper, application scenarios etc.

• You can switch your week assignments with each other, but notify me at least 1 week before the assignment.

Coding assignments

• 4 coding assignments, 2 weeks completion time for each

• Problems related to image processing and manipulation

• Code in whatever you want, but MATLAB is encouraged

• Late policy:• 2 days of penalty-free late submission throughout the semester

• 5% bonus if you don’t use them!

• After that, one-day late: 50% penalty, two-day late: 100% penalty• 1 hour late = 23 hrs late

• Plagiarism policy

Misc.

• We will use Piazza, Coursys, or Canvas, link will be posted later

• Course website: http://yaksoy.github.io/cpim/

• Please no laptops during lectures & presentations• The slides will be posted online

• The best way to take notes is on paper!

• Min. req.

http://yaksoy.github.io/cpim/

Textbook

http://szeliski.org/Book/

http://szeliski.org/Book/

Independent research projects

Slide credits

• Many slides thanks to courses by Alyosha Efros, James Tompkin, and Ioannis (Yannis) Gkioulekas along with their acknowledgements.

34