digital photography with flash and no-flash image pairs

Digital Photography with Flash and No-Flash Image Pairs

By: Georg Petschnigg Maneesh AgrawalaHugues Hoppe Richard Szeliski

Michael Cohen Kentaro Toyama, Microsoft Corporation

Presented by: Yael AmsterdamerAdvanced Topics in Visual Computing, Spring 2012

2Digital Photography with Flash and No-Flash Image Pairs

The Dilemma: to Flash or not to flash?

Introduction

• Natural lighting• Low signal-to-noise ratio

(SNR)• Loss of details• Longer exposure –

motion blur

• Harsh, unnatural lighting• High SNR• More details• May cause unwanted

artifacts (red eye, shadows, specularities)


Why not both?

• The idea: use the good features of each photo to create a better image

Introduction


Other works on combining images

Introduction

Different exposures

Panoramic view

Relighting

Current work • handles 2 images with specific

characteristics,• Envisioned for camera firmware


Different Applications

• Denoising• Detail transfer– Side requirement: handling flash induced

shadows and specularities• White Balancing• Continuous Flash Adjustment• Red Eye Removal

Introduction


Camera Adjustment to light• A certain amount of light is needed for a photo

• Taking pictures of a dark scene

– Increasing exposure time

– Opening the aperture

– Increasing ISO (sensor sensitivity)

• The problem – high ISO noise affects high-frequency details

Background


The Input

• F – the photo with flash• A – the no-flash (ambient) photo– Composed of 3 RGB intensity values for each pixel– Ap – one pixel in A

• Same camera position, focus, aperture– Otherwise, registration is required

• Adjusted ISO and exposure time• Undergo non-linear tone-mapping operations

Background


1st Application: Denoising

• A classic problem in image processing– The target: analytically reduce SNR

• One type of solution: smoothing filters• Bilateral Filter [Tomasi and Manduchi 1998]

– Fast, non-iterative– Averaging close-by pixels– Low-pass filter, combined with– Edge-stopping function

Denoising


Bilateral filter

• However, results still have noise or blur (or both)

Denoising

ambient

flashBilateral

filter


Improvement: Joint Bilateral Filter

• In the flash image there are much more details• Why not use F to find edges?

Denoising

Bilateral filter

Joint Bilateral filter

The difference


Not all edges are real …

• May cause over- or under-blur in joint bilateral filter• We need to eliminate their effect

Denoising


Detecting shadows• Observation: the pixels in the flash shadow should be similar to

the ambient image

• Not identical:

– Noise

– Inter-reflected flash

• Compute a shadow mask

• Take pixel p if

• is manually adjusted

• Mask is smoothed and dilated

Denoising

Pixels for which we will not use F


Detecting specularities• Take pixels where sensor input is close to maximum

(very bright)

– Over fixed threshold

• Create a specularity mask

• Also smoothed and dilated

• M – the combination of shadow and specularity masks

– Where Mp=1, we use ABase. For other pixels we use ANR.

Denoising


There is so much denoising can do• It cannot add details missing in the ambient image• Exist in flash image because of high SNR• We use a quotient image:

• Multiply with ANR to add the details• Masked in the same way

Detail Transfer

Bilateral filtered

Reduces the effect of

noise in F


Unified process• Denoising + detail transfer + masking shadows and

specularities

Detail Transfer


More examples

Detail Transfer


The opposite of preserving the atmosphere• The goal: simulate white light in the scene

– What color is the light in the scene?

• Flash image can help us:The difference between F and A is only the flash light

• Assume it is proportional to the surface albedo– Inherent property of the surface

– Assume the surface reflects light in the same color (will not work for plastic, glass…)

White Balancing


Computing the ambient light• We cancel the color of the surface

• The average c of Cp(col) for pixels with certain properties is the ambient light color

• Then we scale A to cancel the ambient light

White Balancing


Example

White Balancing

Estimated ambient light

Computed color

Ambient original

White-balanced


Combining flash and no-flash images• Providing complementary details about the scene

• Use for denoising and detail transfer

• Use for white balancing

• Future work

– More robustness

– Automatization of configuration

Summary

Thank you!