digital photography with flash and no-flash image pairs

Digital Photography with Flash and No-Flash Image PairsBy: Georg PetschniggManeesh AgrawalaHugues HoppeRichard SzeliskiMichael CohenKentaro Toyama, Microsoft Corporation

Presented by: Yael AmsterdamerAdvanced Topics in Visual Computing, Spring 2012

1The Dilemma: to Flash or not to flash?

Introduction

2Digital Photography with Flash and No-Flash Image PairsNatural lightingLow signal-to-noise ratio (SNR)Loss of detailsLonger exposure motion blurHarsh, unnatural lightingHigh SNRMore detailsMay cause unwanted artifacts (red eye, shadows, specularities)

2Why not both?The idea: use the good features of each photo to create a better image Introduction3Digital Photography with Flash and No-Flash Image Pairs

3Other works on combining images Introduction4Digital Photography with Flash and No-Flash Image Pairs

Different exposuresPanoramic viewRelightingCurrent work handles 2 images with specific characteristics,Envisioned for camera firmware 4Different ApplicationsDenoisingDetail transferSide requirement: handling flash induced shadows and specularitiesWhite BalancingContinuous Flash AdjustmentRed Eye RemovalIntroduction5Digital Photography with Flash and No-Flash Image Pairs5Camera Adjustment to lightA certain amount of light is needed for a photoTaking pictures of a dark sceneIncreasing exposure timeOpening the apertureIncreasing ISO (sensor sensitivity)The problem high ISO noise affects high-frequency details

Background6Digital Photography with Flash and No-Flash Image Pairs

6The InputF the photo with flashA the no-flash (ambient) photoComposed of 3 RGB intensity values for each pixelAp one pixel in ASame camera position, focus, apertureOtherwise, registration is requiredAdjusted ISO and exposure timeUndergo non-linear tone-mapping operationsBackground7Digital Photography with Flash and No-Flash Image Pairs71st Application: DenoisingA classic problem in image processingThe target: analytically reduce SNR One type of solution: smoothing filtersBilateral Filter [Tomasi and Manduchi 1998]Fast, non-iterativeAveraging close-by pixelsLow-pass filter, combined withEdge-stopping function

Denoising8Digital Photography with Flash and No-Flash Image Pairs8Bilateral filterHowever, results still have noise or blur (or both)Denoising9Digital Photography with Flash and No-Flash Image Pairs

ambientflash

Bilateral filter

9Improvement: Joint Bilateral FilterIn the flash image there are much more detailsWhy not use F to find edges?

Denoising10Digital Photography with Flash and No-Flash Image Pairs

Bilateral filterJoint Bilateral filter

The difference10Not all edges are real May cause over- or under-blur in joint bilateral filterWe need to eliminate their effect

Denoising11Digital Photography with Flash and No-Flash Image Pairs

11Detecting shadowsObservation: the pixels in the flash shadow should be similar to the ambient imageNot identical:NoiseInter-reflected flashCompute a shadow maskTake pixel p if is manually adjustedMask is smoothed and dilated

Denoising12Digital Photography with Flash and No-Flash Image Pairs

Pixels for which we will not use F12Detecting specularitiesTake pixels where sensor input is close to maximum (very bright)Over fixed threshold Create a specularity maskAlso smoothed and dilatedM the combination of shadow and specularity masksWhere Mp=1, we use ABase. For other pixels we use ANR.

Denoising13Digital Photography with Flash and No-Flash Image Pairs

13There is so much denoising can doIt cannot add details missing in the ambient imageExist in flash image because of high SNRWe use a quotient image:

Multiply with ANR to add the detailsMasked in the same way

Detail Transfer14Digital Photography with Flash and No-Flash Image Pairs

Bilateral filteredReduces the effect of noise in F

14Unified processDenoising + detail transfer + masking shadows and specularities

Detail Transfer15Digital Photography with Flash and No-Flash Image Pairs

15

More examples Detail Transfer16Digital Photography with Flash and No-Flash Image Pairs

16The opposite of preserving the atmosphereThe goal: simulate white light in the sceneWhat color is the light in the scene?Flash image can help us:The difference between F and A is only the flash lightAssume it is proportional to the surface albedoInherent property of the surfaceAssume the surface reflects light in the same color (will not work for plastic, glass)

White Balancing17Digital Photography with Flash and No-Flash Image Pairs17Computing the ambient lightWe cancel the color of the surface

The average c of Cp(col) for pixels with certain properties is the ambient light colorThen we scale A to cancel the ambient lightWhite Balancing18Digital Photography with Flash and No-Flash Image Pairs

18Example White Balancing19Digital Photography with Flash and No-Flash Image Pairs

Estimated ambient lightComputed colorAmbient original

White-balanced19Combining flash and no-flash imagesProviding complementary details about the sceneUse for denoising and detail transferUse for white balancingFuture workMore robustnessAutomatization of configuration

Summary20Digital Photography with Flash and No-Flash Image Pairs20

Thank you! 21