light, color, and surface reflectance · intrinsic estimation benchmarking in complex scenes. ......

Light, Color, and Surface

Reflectance

Shida Beigpour

Overview

2

Introduction

Multi-illuminant Intrinsic Image Estimation

Multi-illuminant Scene Datasets

Multi-illuminant Color Constancy

Conclusions

Introduction

3

Modeling scene optics plays a crucial role in scene understanding.

¤ Interreflection

¤ Specularities

¤ Colorcast ¤ Colored shadows

¤ Colored illuminant

¤ Multi-illuminant

Introduction

4

1. Automatic White Balance (AWB)

AWB

Original Image Color corrected

Introduction

5


2. Changing the illuminant (Re-lighting)

Original Image Relighted

relight

Introduction

6



3. Segmentation

Original Image Segmentation

Introduction

7



3. Segmentation

4. Object Recoloring (physically plausible)

Original Image Recolored

Introduction

9



3. Segmentation

4. Object Recoloring

5. Photo forensics

Introduction

10



3. Segmentation

4. Object Recoloring

5. Photo forensics

6. Shadow removal

7. …

Original Image Shadow removed

Introduction

11

Context:

There are mainly two fields which investigate illumination and object modeling in

computer vision:

Intrinsic image estimation

Color constancy

Introduction

12

Context:

Many real-world scenes present complex reflectance under multiple-illuminants.

Most existing methods limit their domain by making unrealistic assumptions.

Interreflections

Multiple illuminants

Colored shadows

Introduction

13

Context:

Many real-world scenes present complex reflectance under multiple-illuminants.

Most existing methods limit their domain by making unrealistic assumptions.

Many fields in computer vision that rely on illuminant and reflectance models can

strongly benefit from more realistic models:

Object classification methods which use color cues

Segmentation algorithms

Overview

14

Introduction




Conclusions

Single-illuminant Object Reflectance Model

15

There are two main forms of reflectance: specular and diffuse.

Surface Particles

Object Surface

Surf

ace N

orm

al

Diffuse Reflection

Object

Image


16

• Lambertian Reflection Model

Models the interaction of light and matte (diffuse) object surfaces

• Dichromatic Reflection Model (DRM) [Shafer, JCRA 1985]

geometry

Surface albedo

Light Sensor Sensitivity

Coordinate Color channel

Pixel Value

Chromatic component

Diffuse component Specular component

Frensel reflectance

1

Multi-illuminant Object Reflectance Model

19

Image formation under multiple lights:

Surface Particles

Object Surface

Surf

ace N

orm

al

Diffuse Reflection

Object

Image

Diffuse Reflection

Multi-illuminant Object Reflectance Model *

22

We extend the DRM to account for multiple illuminants:

Multi-Illuminant Dichromatic Reflection (MIDR) model

Assumption: Single-colored object pre-segmented from its background.

As error metric we define Reconstruction error, the mean square error between the

estimated model and the object pixels.

DRM

MIDR

Geometry Chroma

* Sh. Beigpour and J. van de Weijer, Object Recoloring based on Intrinsic Image Estimation, Proceedings of IEEE

International Conference on Computer Vision (ICCV), November 2011

light Surface


An example of modeling:

Object is segmented from its background

23

Object Image

Diffuse

component

Specular

component

Illuminant

Color

Object

Color Object Image

DRM

Reconstruction


An example of algorithm performance:

First the object is modeled just by the DRM.

But in this case, the greenish inter-reflection is lost.

24

Object Image Reconstruction

Threshold

Secondary

Illuminant Mask Reconst. Error

2


25


The reconstruction error provides a mask for which the secondary illuminant is

dominant.

Illuminant Mask Body

Reflectance

Specular

Reflectance

Object Image Reflectance

Reflectance


26


Using the illuminant mask, two models are generated.

Object Image

Illuminant Mask

Reflectance

Body

Reflectance

Specular

Reflectance

Object Image

Reflectance


27


The models and corresponding masks are iteratively refined until convergence.

Object Image

28

An example of modeling:

Decomposition


1st Body

Reflectance

Specular

Reflectance

2nd Body

Reflectance

Object Color

Primary Light

Secondary Light

Object Image


29

Improving photo-fusion:

Here the green mug is replaced by a purple mug.

But now the greenish interreflection on the blue mug doesn’t match the

scene.

Photo-fusion


30


The purple color from the mug is assigned as the secondary illuminant.

1st Body

Reflectance

Specular

Reflectance

2nd Body

Reflectance

Object Color

Primary Light

Secondary Light


31


The mug is then constructed using the new color.

1st Body

Reflectance

Specular

Reflectance

2nd Body

Reflectance

Object Color

Primary Light

Secondary Light


32


The greenish inter-reflection on the blue mug is transformed to purple.

Photo-fusion


33

Recoloring a video

Original Video Recolored Video


35

Next step:

Multi-illuminant dataset with pixel-wise accurate ground-truth for the

purpose of quantitative evaluation.

Solving the model without the need of segmentation or assumptions

on the illuminant.

Overview

36

Introduction




Conclusions

Multi-illuminant Datasets

37

Observations:

Existing dataset for reflectance and illuminant estimation are mostly

limited to single-illuminant scenarios.

In addition intrinsic image datasets mainly consist of single-object scenes.

We propose two multi-illuminant datasets:

Synthetic intrinsic image dataset: Computer graphics generated dataset for

intrinsic estimation benchmarking in complex scenes.

Multi-Illuminant Multi-Object (MIMO) : Dataset for multi-illuminant color

constancy benchmarking.

Synthetic intrinsic image dataset * Our proposal is to use computer graphics generated data for this

purpose:

3D-Modeling : Blender

Photo-realistic rendering: Yafaray

Multi-spectral images:

Using 6-band color images instead of 3-band RGB to improve accuracy

We use recorded multi-spectral data for Munsell color patches for the

surfaces along with the multi-spectral Planckian and non-Planckian lights.

39

* Sh. Beigpour, M. Serra, J. van de Weijer, R. Benavente, M. Vanrell, O. Penacchio, and D. Samaras, Intrinsic Image

Evaluation On Synthetic Complex Scenes, Proceedings of IEEE International Conference on Image Processing

(ICIP), September 2013.

Intrinsic Image datasets

We propose the use of synthetic data for intrinsic image estimation:

Global lighting

Photo-realism

Direct lighting Photon-Mapping

Inter-reflections

Colored Shadows

40


We propose the use of synthetic data for intrinsic image estimation:

Global lighting

Photo-realism

Accurate pixel-wise ground-truth

Original Image Reflectance Illumination

41


An example of benchmarking for three intrinsic image methods:

42 42

Ground-truth

reflectance

Gehler, NIPS 2011

Barron ECCV 2012

Serra, CVPR2012

Reflectance

Estimation Results

Multi-illuminant Datasets

43

Observations:

Existing dataset for reflectance and illuminant estimation are mostly

limited to single-illuminant scenarios.

In addition intrinsic image datasets mainly consist of single-object scenes.

We propose two multi-illuminant datasets:

Synthetic intrinsic image dataset: Computer graphics generated dataset for

intrinsic estimation benchmarking in complex scenes.

Multi-Illuminant Multi-Object (MIMO) : Dataset for multi-illuminant color

constancy benchmarking.

Color constancy datasets

Ciurea & Funt

Parraga et. al.

Gehler et. al.

Barnard et. al.

Examples of some of the most popular single-illuminant color

constancy datasets.

44

Multi-Iluminant Multi-Object scene

dataset (MIMO)

45

We propose our multi-illuminant dataset MIMO

MIMO-Lab: Toy images captured under controlled lab conditions.

* Sh. Beigpour and C. Riess and J. van de Weijer, and E. Angelopoulou, Multi-illuminant estimation using

Conditional Random Field, IEEE Transactions on Image Processing (TIP), vol.23, no.1, pp.83,96, January 2014.

*


dataset (MIMO)

46



MIMO-Realworld: Indoor/outdoor images of some real-world scenes.


dataset (MIMO)

47



MIMO-Realworld: Indoor/outdoor images of some real-world scenes.


dataset (MIMO)

49

Ground-truth calculation:

We use the linearity of the images and illuminants :

DS Image

Macbeth

color

checker

Image of Illuminant a (fa) Both Illuminants (fab) Image of Illuminant b (fb)

Illuminant b

Illuminant a

Illuminant ab


dataset (MIMO)

50

Ground-truth calculation: The ratio of the each of the illuminants making up the multi-illuminant scene in the

green channel :

Using this ratio we can calculate the illuminant at each pixel of the multi-illuminant

image (iab) by a linear interpolation:

Illuminant b

The per pixel ra,g ratio Illuminant Chroma Map

(ground-truth)


dataset (MIMO)

51

Examples from the MIMO-lab dataset:

Multi-illuminant ratio Illumination ground-truth Original Image

Multi-Iluminant Multi-Object scene DS:

Expanding the framework

53

Examples from our new Multi-illuminant dataset in Norway:

6 scenes, 5 illumination conditions : 30 images

* Imtiaz Masud Ziko, Shida Beigpour, and Jon Yngve Hardeberg, 'Design and Creation of a Multi-Illuminant Scene

Image Dataset, International Conference on Image and Signal Processing (ICISP) 2014.

Blue, Orange, and ceiling lamp Scene with Macbeth Checkered

54

Blue Light Orange Light Ceiling Lamp

For each multi-illuminant scene we require the images of each light

separately:



59

In the color-coded ratio image, Red, Green and Blue stand for

Orange (from left), Ceiling lamp (from top), and Blue (from right).

Blue, Orange,

and ceiling lamp Pixel-wise

illumination Ratio



Pixel-wise

Groundtruth

Multi-Iluminant scene datasets

61

Conclusion:

We introduced a synthetic dataset for intrinsic image estimation for

multiple illuminants and complex scenes.

We created the first multi-illuminant complex scene dataset with pixel-

wise ground-truth accuracy.

Next step:

We use our multi-illuminant scene dataset for multi-illuminant color

constancy.

Overview

62

Introduction




Conclusions

Color constancy

63

Can you guess the color of the woman’s top?

Color constancy

64

Can you guess the color of the woman’s top?

Computation Color Constancy

65

Color constancy algorithm mainly consist of two steps:

Estimation of chromaticity of light source

Statistical methods

using the Minkowski framework:

Gray-world (e0,1,0 ) : The average reflectance is the illuminant color.

Max-RGB (e0,∞,0 ) : The maximum reflectance is the illuminant color.

Gray-edge (e1,m,σ ) : The average reflectance differences is the illuminant color.

Second-order gray-edge (e2,m,σ )

,,

1

)()()( mnc

mm

n

cn

ekdf

x

x

x

Minkowski norm


66



Statistical methods

Physics-based methods

Using R. Tan et. al. 2004, Inverse-Intensity Chromaticity (IIC) space for

color constancy:

1. Detection of possible specular regions

2. Illuminant estimation in the detected regions.

Using the linear relation between chromaticity and inverse intensity in the image:

Inverse intensity Image Chromaticity

Specularity chroma Diffuse chroma


67



Statistical methods

Physics-based methods

Gamut-mapping

Learning methods

Image color correction to canonical illumination

von Kries model

Diagonal-offset model

…


68

Motivation:

Single-illuminant color constancy methods are not capable of modeling the

complexity of illumination in the real-world scenarios.

The assumption of uniform illumination made by the single-illuminant

methods is unrealistic.

Objective:

The goal of Multi-illuminant Color Constancy is to provide accurate

pixel-wise illuminant estimates.


69

Multi-Illuminant Random Field (MIRF) method * :

Image is divided to patches using grid and illuminant is estimated locally.

Unary potentials: Statistical estimation

Physics-based estimation

Pair-wise potentials

Pair-wise Potential

Unary Potential

Smoothness

Factor




70






Pair-wise Potential

Unary Potential

Smoothness

Factor




71






Pair-wise Potential

Unary Potential

Smoothness

Factor



Multi-Illuminant Random Field (MIRF) method:


Unary potentials: Error metric:

The “angular error” is the angle between any two color vectors (i1 and i2) in RGB space.

We use the angular error between the label and estimate to define the energy of a graph

labeling.


72


73




f j = (fRj, fG

j, fBj) is the j-th pixel and P={p1,p2,…,pN} are the patches then the

local illuminant color for the i-th patch pi using Minkowski norm is:

Therefore we can write the unary potential as:

Statistical unary potential

Patch label Error function

Local estimate


76



Unary potentials: Physics-based estimation

Therefore:

where the binary weighting w is defined using sum of the intensity for possible

specular pixels ( ssp ) and a threshold ( tsp )

))(()|(

iiσ

i

ii

p xiρFx pr

p

p ,w Physics-based unary potential

Local illuminant estimate

Original Image Regions which w exists

Lehmann and Palm JOSA 2001



Unary potentials: Defining labels:

The labels are centers of the clustered formed by the patch-wise local illuminant

estimates in RGB space.

To improve the performance highly saturated values are discarded :


78

Each label

Label set dwi ili )( ,|L

T

wi )111(3

1,,White light

Physics-based estimates

Statistical estimates

Label-set K-means




79

Statistical estimates by Gray-word




80

Statistical estimates by Gray-word Results of the energy minimization

Multi-Illuminant Random Field (MIRF)

84

Experiment results:

We compare our method with the single-illuminant and the method of Gijsenij et. al.

We use the median angular error over each set as our measure.

MIMO-Lab (58 images):

Method Single-light Gijsenij et.al. MIRF

DN 10.5º N/A N/A

GW 2.9º 5.9º 2.8º (-3%)

WP 7.6º 4.2º 2.8º (-33%)

GE-1 2.8º 4.2º 2.6º (-7%)

GE-2 2.9º 5.7º 2.6º (-10%)

IEBV 8.3º N/A 3.0º (-64%)


86

Experiment results:

We compare our method with the single-illuminant and the method of Gijsenij et. al.

We use the median angular error over each set as our measure.

Gijsenij-outdoor (9 images):

Method Single-light Gijsenij et.al. MIRF

DN 3.6 N/A N/A

GW 13.8º 13.8º 10.1º (-27%)

WP 11.3º 8.4º 6.4º (-24%)

GE-1 10.1º 7.6º 4.7º (-38%)

GE-2 8.5º 7.4º 5.0º (-32%)

IEBV 7.3º N/A 7.3º

Original

Ground-truth

White-balance

3.1 2.9 10.6 6.8

Single illuminant

White-balance

91


Experiment results: Qualitative evaluation

2.7 6.6 8.3 4.5

Gijsenij

White-balance

2.0 2.6 8.7 3.7

Our method’s

White-balance

Conclusions

92

We proposed an intrinsic image estimation framework for single-colored

objects in multi-illuminant scenarios (applied to object recoloring and

color transfer).

We created two multi-illuminant scene datasets:

Synthetic intrinsic image dataset: Computer graphics generated dataset for intrinsic

estimation benchmarking in complex scenes.

Multi-Illuminant Multi-Object (MIMO) : Dataset for multi-illuminant color constancy

benchmarking.

We proposed a CRF-based color constancy method for multi-illuminant

scenes. State-of-the-art results are obtained on 3 datasets.

93

Thanks for your attention !

Publications

94

1. Sh. Beigpour and C. Riess and J. van de Weijer, and E. Angelopoulou, Multi-illuminant estimation using Conditional Random Field, IEEE Transactions on Image Processing (TIP). 2014.

2. Imtiaz Masud Ziko, Shida Beigpour, and Jon Yngve Hardeberg, Design and Creation of a Multi-Illuminant Scene Image Dataset, International Conference on Image and Signal Processing (ICISP) 2014

3. S. Beigpour, M. Serra, J. van de Weijer, R. Benavente, M. Vanrell, O. Penacchio, and D. Samaras, Intrinsic Image Evaluation On Synthetic Complex Scenes, International Conference on Image Processing (ICIP) 2013.

4. Sh. Beigpour and J. van de Weijer, Object Recoloring based on Intrinsic Image Estimation, Proceedings of IEEE International Conference on Computer Vision (ICCV), November 2011.

5. J. van de Weijer and Sh. Beigpour, The Dichromatic Reflection Model: Future Research Directions and Applications, invited paper at Int. Conf. on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), Portugal 2011.

6. M. Bleier, C. Riess, Sh. Beigpour, E. Eibenberger, E. Angelopoulou, T. Troeger, and A. Kaup. "Color Constancy and Non-Uniform Illumination: Can Existing Algorithms Work?" ICCV Workshop on Color and Photometry in Computer Vision (ICCV Workshop). November 2011.

7. Sh. Beigpour and J. van de Weijer, Photo-Realistic Color Alteration For Architecture And Design, Proceedings of Colour Research for European Advanced Technology Employment (CREATE) Conference, Jun 2010. (Chosen for oral presentation).

light, color, and surface reflectance · intrinsic estimation benchmarking in complex scenes. ......

Documents