A Gentle Introduction

to Bilateral Filtering and its Applications

A Gentle Introduction

to Bilateral Filtering and its Applications

10/10: Conclusions

Jack Tumblin – EECS, Northwestern University

The Bilateral FilterThe Bilateral Filter

• As Simple asAs Simple asWeights from Weights from twotwo Gaussian Functions; Gaussian Functions;

• Has deep connections to PDE formalisms,Has deep connections to PDE formalisms,shock-forming, heat flow, and diffusionshock-forming, heat flow, and diffusion

• Speed? several Speed? several very fastvery fast versions versions

Bilateral FilterBilateral Filter

GOALS are Ambitious & Subjective, butGOALS are Ambitious & Subjective, but• Noise RemovalNoise Removal• ““Edge-Preserving Smoothing” Edge-Preserving Smoothing” • Locally Weighted Averaging with Outlier RejectionLocally Weighted Averaging with Outlier Rejection• Visual Separation of ‘fine details’ and ‘large features’Visual Separation of ‘fine details’ and ‘large features’

METHODS are Simple & Practical METHODS are Simple & Practical • BiBilateral Filter: lateral Filter: doublydoubly-weighted local average,-weighted local average,

High weights only for ‘ High weights only for ‘nearbynearby’, ‘’, ‘similarsimilar’ values’ values• YOUYOU decide parameters of what is decide parameters of what is “nearby”“nearby”, ,

of what is of what is “similar”“similar”

VERY Broad Range of Applications:VERY Broad Range of Applications:

• any task where any task where ‘edges’‘edges’ and and ‘similarities’‘similarities’carry useful informationcarry useful information

• data of arbitrary dimensions; data of arbitrary dimensions; M to N, time, color, M to N, time, color, image vector-spaces, etc,image vector-spaces, etc,

• even on manifolds even on manifolds 3D meshes, etc.3D meshes, etc.

Explosive Growth: at SIGGRAPH, Explosive Growth: at SIGGRAPH, EUROGRAPHCIS, CVPR, ICCV, ECCV…EUROGRAPHCIS, CVPR, ICCV, ECCV…• Image Deblurring with Blurred/Noisy Image PairsImage Deblurring with Blurred/Noisy Image Pairs L. Yuan, et al.L. Yuan, et al.• Multiscale Shape and Detail Enhancement from Multi-Light Image Collections Multiscale Shape and Detail Enhancement from Multi-Light Image Collections

R. Fattal et al.R. Fattal et al.• Joint Bilateral UpsamplingJoint Bilateral Upsampling J. Kopf et al.J. Kopf et al.• LDR2HDR: On-the-fly Reverse Tone Mapping of Legacy Video and PhotographsLDR2HDR: On-the-fly Reverse Tone Mapping of Legacy Video and Photographs

A. G. Rempel et al.A. G. Rempel et al.• Factored Time-Lapse VideoFactored Time-Lapse Video K. Sunkavalli et al.K. Sunkavalli et al.• Computational Time-Lapse VideoComputational Time-Lapse Video E. Bennett et al.E. Bennett et al.• Real-Time Edge-Aware Image Processing With the Bilateral GridReal-Time Edge-Aware Image Processing With the Bilateral Grid J. Chen et al.J. Chen et al.• Constant Time O(1) Bilateral Filtering Constant Time O(1) Bilateral Filtering Fatih PorikliFatih Porikli• A Hybrid Camera for Motion Deblurring and Depth Map Super-ResolutionA Hybrid Camera for Motion Deblurring and Depth Map Super-Resolution

Feng Li, et al.Feng Li, et al.• Geo-spatial Aerial Video Processing for Geo-spatial Aerial Video Processing for

Scene Understanding and Object TrackingScene Understanding and Object Tracking Jiangjian Xiao, et al.Jiangjian Xiao, et al.• Illumination and Camera Invariant Stereo MatchingIllumination and Camera Invariant Stereo Matching Yong Seok Heo, et al.Yong Seok Heo, et al.• Enhancing Photographs with Near Infrared Images Enhancing Photographs with Near Infrared Images Xiaopeng Zhang,et al.Xiaopeng Zhang,et al.• FuzzyMatte: A Computationally Efficient Scheme FuzzyMatte: A Computationally Efficient Scheme

for Interactive Mattingfor Interactive Matting Yuanjie Zheng, et al. Yuanjie Zheng, et al. • Accurate Multi-View Reconstruction Using Robust Accurate Multi-View Reconstruction Using Robust

Binocular Stereo and Surface MeshingBinocular Stereo and Surface Meshing Derek Bradley, et al.Derek Bradley, et al.• Demosaicing by Smoothing along 1D Features Demosaicing by Smoothing along 1D Features Boris Ajdin, et al. Boris Ajdin, et al.

AND MANY MORE…AND MANY MORE…

Is All the Work Finished?Is All the Work Finished?

• Nothing left to explore?Nothing left to explore?

No, No, No! No, No, No!

Bilateral filters raise deep, basic questions;Bilateral filters raise deep, basic questions;

Answers still vague and tentative..Answers still vague and tentative..

For example … For example …

What Is What Is ‘Noise’‘Noise’? What is ? What is ‘Texture’‘Texture’??

Wet Sand (Jay Sekora)

What Is What Is ‘Noise’‘Noise’? What is ? What is ‘Texture’‘Texture’??

Human Skin (Ken Perlin)Human Skin (Ken Perlin)

What Is What Is ‘texture’‘texture’? What is an ? What is an ‘Edge’‘Edge’??

Fine details don’t always match large structuresFine details don’t always match large structures

Wireframe VW Beetle [The Cellar: Image of the Day]Wireframe VW Beetle [The Cellar: Image of the Day]

What Is an What Is an ‘Edge’‘Edge’ in Human Vision? in Human Vision?

Illusory Contours (G. Kanizsa, 1955)Illusory Contours (G. Kanizsa, 1955)

What Is an What Is an ‘Edge’‘Edge’ in Human Vision? in Human Vision?

Do edges exist here?Do edges exist here?

Does Does absoluteabsolute intensity matter? intensity matter?

Often, Perceived Intensities Often, Perceived Intensities ≠≠ Pixel Values: Pixel Values:

Example: Example: ‘Simultaneous Contrast’ ‘Simultaneous Contrast’ Center strip has Center strip has CONSTANTCONSTANT intensity… intensity…

Does Does absoluteabsolute intensity matter? intensity matter?

Often, Perceived Intensities Often, Perceived Intensities ≠≠ Pixel Values: Pixel Values:

Example: Example: ‘Simultaneous Contrast’ ‘Simultaneous Contrast’ Center strip has Center strip has CONSTANTCONSTANT intensity… intensity…

What Is an ‘What Is an ‘Illumination Edge’Illumination Edge’??

Sometimes it isn’t a large Sometimes it isn’t a large intensityintensity change… change…

What Is a What Is a ‘Geometric Edge’‘Geometric Edge’ in Images? in Images?

3D ‘Peanut’ shape3D ‘Peanut’ shape

Some silhouettesSome silhouettes

are are SUGGESTEDSUGGESTED

by shape cuesby shape cues

What Is an ‘Edge’ at The What Is an ‘Edge’ at The Finest ScalesFinest Scales??

Scale Problems:Scale Problems:

Can’t resolve everyCan’t resolve everyhair and fiber in fur…hair and fiber in fur…

Long Black Whiskers:Long Black Whiskers:

• Visible on Bkgnd, butVisible on Bkgnd, but

• Visible against fur?Visible against fur?

Albrecht Durer (1502) “Young Hare”

For For anyany perceivable image? perceivable image?

Hmm.Hmm.

Edge?Edge?

Noise?Noise?

Regions?Regions?

Texture?Texture?

Silhouette?Silhouette?

……

http://mfrost.typepad.com/cute_overload/2007/02/backlit_extrava.html

Thank YouThank You For Attending! For Attending!

• For more in-depth info: links, and all docs:For more in-depth info: links, and all docs:http://people.csail.mit.edu/sparis/bf_coursehttp://people.csail.mit.edu/sparis/bf_course

Acknowlegements:Acknowlegements:• Sylvain Paris and Frédo Durand were supported in part by a Sylvain Paris and Frédo Durand were supported in part by a

National Science Foundation CAREER award 0447561, also by National Science Foundation CAREER award 0447561, also by NSF Grant No. 0429739 and by a grant from Royal Dutch/Shell NSF Grant No. 0429739 and by a grant from Royal Dutch/Shell Group. Group.

• Frédo Durand acknowledges a Microsoft Research New Faculty Frédo Durand acknowledges a Microsoft Research New Faculty Fellowship and a Sloan Fellowship. Fellowship and a Sloan Fellowship.

• Jack Tumblin's work was supported in part by the National Science Jack Tumblin's work was supported in part by the National Science Foundation through grants NSF-IIS 0535236 and NSF-SGER Foundation through grants NSF-IIS 0535236 and NSF-SGER 0645973, and also thanks Adobe Systems, Inc. for their support via 0645973, and also thanks Adobe Systems, Inc. for their support via two unrestricted gifts for computational photography research.two unrestricted gifts for computational photography research.

ConclusionsConclusions

•10 Minutes10 Minutes

•<14 slides<14 slides