color in image & video processing applications
Post on 03-Feb-2022
2 Views
Preview:
TRANSCRIPT
27/09/2010
1
Color in Image & VideoProcessing Applications
DAGM 2010Darmstadt
Joost van de Weijer
Universitat Autonoma de BarcelonaComputer Vision Center
Why use Color ?
photometric invariance discriminative power saliency detection
27/09/2010
2
Physics approach to color
R G B{ , , }
0 7
0.8
0.9
13-blue14-green
15-red16-yellow
17-magenta18
s
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
350 400 450 500 550 600 650 700 750
18-cyan
bc
e
e bc
s overview
THEORY
PART I: features• color spaces• color feature extraction• saliency detection.
APPLICATIONS
IMAGE CLASSIFICATION• combining color and shape
IMAGE SEGMENTATION• deviations from the
PART II: color constancy• bottom-up color constancy• top-down color constancy• color constant features
reflection model
OBJECT RECOLORING•complex reflection models
27/09/2010
3
Applications
segmentationclassification Object recoloring
human segmentation
Color Basics
2
- human vision- physics–based reflection model
27/09/2010
4
Human vision
Human vision
Humans have two types of photoreceptors: • rods: achromatic night time vision when light levels are low.• cones: color vision during day time when light levels are high.
27/09/2010
5
Human vision
Humans have two types of photoreceptors: • rods: achromatic night time vision when light levels are low.• cones: color vision during day time when light levels are high.
Normalized responsivity
S
M L
Normalized responsivity
opponent color system
The information from these receptors is combined in the retina as colour opponent signals through the optic nerve.
Retinal neurons appear to encode both achromatic contrast systemsRetinal neurons appear to encode both achromatic contrast systems and chromatic contrast systems.
2
g bR r
2
r bG g
2
r gB b
2 2
r gr gY b
RG R G BY B Y
Opponent mechanism [Itti PAMI 95]
((r,g,b) normalized RGB)
27/09/2010
6
opponent color system
The information from these receptors is combined in the retina as colour opponent signals through the optic nerve.
Retinal neurons appear to encode both achromatic contrast systemsRetinal neurons appear to encode both achromatic contrast systems and chromatic contrast systems.
+black-white
red-green
blue-yellow
+
+
-
Opponent mechanism Linear combination of RGB
B
G
R
O
O
O
111
211
011
3
2
1
red green
--
CIE 1931 standard observer
• computing the matching function for humans
• primary stimuli:
Image courtesy Bill Freeman
p yR=700nmG=546.1nmB=435.8nm
Projected illuminants at sub-intervals = 5nm
27/09/2010
7
CIE 1931 standard observer
Image courtesy Bill Freeman
CIE 1931 standard observer
Image courtesy Bill Freeman
27/09/2010
8
CIE 1931 standard observer
Image courtesy Bill Freeman
CIE 1931 standard observer
Image courtesy Bill Freeman
27/09/2010
9
CIE 1931 standard observer
Image courtesy Bill Freeman
CIE 1931 standard observer
Image courtesy Bill Freeman
27/09/2010
10
CIE 1931 standard observer
Image courtesy Bill Freeman
Experiment result(Commission International de l’Eclairage)
• Visuall field = 2 degrees• Interval = [380,780] • Sub-intervals = 5nm.
CIE 1931 standard observer
• primary stimuli:R=700nmG=546.1nmB=435.8nm
• Every position gives the mixing weights for the primary stimuli.• The resulting functions are• The resulting functions are the color matching functions
)(),(),( bgr
27/09/2010
11
Imaginary space: XYZ (Standard colorimètric Observer (CIE 1931))
Goal:• Avoid negative values in the color matching functions This simplifies
XYZ color space
Avoid negative values in the color matching functions. This simplifies the constructions of color measurement devices.• Correlate one of the functions to the intensity.
Chromaticity.exe
CIE diagram
ZYX
Yy
ZYX
Xx
ZYX
Zz
ZYX
31
31
31 ,,,,: zyxwhite
27/09/2010
12
• If two stimuli are presented, one of them at the center of the ellipse and
MacAdam Ellipses
them at the center of the ellipse, and the other one inside the ellipse, there is not a perceived difference.
• The space is not perceptually uniform (then the ellipses would be circles).
Perceptually uniform space
• CIELUV and CIELAB are examples of perceptually uniform color spaces.
kYY
if1611631
CIELUV
k
Y
Y
Y
Y
kY
Y
Y
Y
L
nn
nn
if 3.903
if 16116*
nn
ZY
Y
Yf
X
Xfa 500*
nn Z
Zf
Y
Yfb 200*
k xif 116
167.787x
k xif 3 xxf
008856.0k2*2*2** baLEab
27/09/2010
13
sRGB
• sRGB: standard RGB color space (HP, Microsoft, Pantone,Corel etc).
•Linear relation with XYZ space:
XR 498605372124063
• sRGB images are assumed to have a gamma=2.2.
• In case of unknown origin people often assume sRGB. For example in the case of data sets collected from the internet
Z
Y
X
B
G
R
linear
linear
linear
057.12040.00557.0
0415.08758.19689.0
4986.05372.12406.3
the case of data sets collected from the internet.
Device dependent color spaces
• RGB: output of majority of cameras and is dependent on the camera sensitivity response. • CMY(K): Is the standard generally applied for printing devices.
},,{0.43
},,{0.23
},,{3
2
2
2
BGRmaxBsiGR
BGRmaxGsiRB
BGRmaxRsiBG
H
HSI: human perception (others HSV, HSL)
BGRI 3
1
5.0
2
5.0
1
2
1
2
Lsi
LsiS
},,{},,{
},,{},,{
2
1
BGRminBGRmax
BGRminBGRmax
27/09/2010
14
Device dependent color spaces
• RGB: output of majority of cameras and is dependent on the camera sensitivity response. • CMY(K): Es the standard generally applied for printing devices.
HSI: often used in computer vision
B
G
R
O
O
O
3
1
3
1
3
16
2
6
1
6
1
02
1
2
1
3
2
1
Orthonormalopponent space:
O1
O3
G
B
f
f´
3
21
2
1arctan
22
OI
OOS
O
OH
333
S is not normalized with intensity !
O2RG
Projection of f on O3 is the intensity. Let f´ theprojection on the plane O1-O2, its length is thesatutation and its angle the hue.
f
Guidelines application of color spaces
• be aware that many color spaces are devise dependent (industrial applications).
• apply CIELAB or CIELUV only when perceptually uniformity of the color pp y y p p y yspace is relevant.
• when you want to decorrelate only luminance use opponent color space. HSI spaces introduces non-linearities.
Choose the color space which fits the photometric invariants your
trackingLearning color names
•Choose the color space which fits the photometric invariants your application requires.
27/09/2010
15
Reflection Models
2
Electromagnetic radiation spectrum
32101
))((log10 m
)(m )(nm610 1000
RadioTV
Radar123456789
10
RadarMicrowaves
Infrared
Visible lightUltraviolate
X rays10111213141516
710 100Gamma rays
27/09/2010
16
Pur
ple
gree
n
yello
w
oran
ge
visible light spectrum
400 450 500 550 600 650 700
P g y o
nanometers mnm 9101 Spectra.exe
http://askabiologist.asu.edu/research/seecolor/atable.htmlhttp://askabiologist.asu.edu/research/seecolor/atable.html slide credit: R. Baldrich
Surface reflectance
R G B{ , , }
0 7
0.8
0.9
13-blue14-green
15-red16-yellow
17-magenta18
s
?0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
350 400 450 500 550 600 650 700 750
18-cyan
bc
e
27/09/2010
17
Two types of reflection:
• Body reflection
• Surface reflection
Dichromatic reflection model
• Surface reflection
Body Reflectance
Normal
Metals:mainly specularreflection.
Colorant particles
Dielectric: specular and diffuse reflection. It includes materials that are coloured using pigments: paints, plastic, ceramic, fabric, paper, etc.
Reflecting materials
Body ReflectanceDiffuse reflection,
Surface ReflectanceSpecular reflection. The
reflection angle is similar toisotropic reflection. The spectral distribution
depends on colorants.
b b bf m c ( , ) ( ) ( )
reflection angle is similar to the incident angle. Its spectral distribution
depends on the illuminant.
s s s sf m c m h ( , ) ( ) ( ) ( )slide credit: R. Baldrich
27/09/2010
18
b sf f f ( , ) ( , ) ( , )
bf ( , ) : Reflected light by the object body. It depends on the pigments
Dichromatic reflection model:
bf ( , ) :
sf ( , ) : Reflected light from the surface. It has a SPD nearly the same as the incident light. (Specular or regular reflectance)
g y j y p p gused to colour the object and it’s the one that makes the object look coloured. (Diffuse reflectance)
: Angles that depend on light source position, observer and surface
The spectral and geometrical terms can be separated:
slide credit: R. Baldrich
ssbb cmcmf ,
sb ccf sb mm
Dichromatic Reflection Model
dichromatic model for matte surfaces:
bcf bm
RGB-histogram
27/09/2010
19
dichromatic model for specular surfaces:
Dichromatic Reflection Model
sb ccf sb mm
RGB-histogram
color spaces: normalized RGB
, , { , , }R G B
r g bR G B R G B R G B
• normalized RGB is given by:
• invariant for shadow and shading variations (matte surfaces):
bBb
bGb
bRb
bRb
cmcmcm
cm
BGR
Rr
bb
bg
br
br
ccc
c
normalized RGB
27/09/2010
20
color spaces: hue-saturation-intensity
3arctan
2
R Ghue
R G B
• defined as:
2 2 223sat R G B RG RB GB
3 R G Bi
sb
Bsb
Gsb
Rb
sbG
sbR
b
ccccccm
ccccmhue
22
3arctan
• hue is invariant for shading variations and specularities under white light:
hue
Take care of instabilities
• when working in different color spaces always take instabilities into account !• Error propagation is a convenient tool for instability evaluation:
q)(xq
xxx bestxx best
bestx
maxq
bestq
minq
q
q
it i tidithdth tS
22
q ...
:by defined isy uncertaint predicted The
).,...,(function compute to
,...,iesuncertaintingcorrespondwith measured are ,...,that Suppose
wu
wu
w
q
u
q
wuq
wu
27/09/2010
21
Take care of instabilities
• when working in different color spaces always take instabilities into account !
• Error propagation is a convenient tool for instability evaluation:Error propagation is a convenient tool for instability evaluation:
Ex. 1
3
arctan2
R Ghue
R G B
2
2 3arctan
2
R Ghue
R G B
2 2 2
2 2 2 2hue hue hueh
2 2 2 22
1(assuming )R R G B
sat
2 2 2 2hue hue huehue R G B
R G B
Take care of instabilities
• when working in different color spaces always take instabilities into account !
• Error analysis is a convenient tool for instability evaluation:
count
hue
The red bobsled is dominated by the blue sky and blue snow.
hue
hue
sat
saturation
27/09/2010
22
S.A. Shafer. Using color to separate reflection components. Color research and applications, 1985.
G.J. Klinker et al.. A physical approach to color image understanding. IJCV,
References: photometric invariants
1990.
M.J. Swain, D.H. Ballard . Color indexing. IJCV, 1991
T. Gevers, A.W.M. Smeulders. Color based object recognition. Pattern Recognition, 1999.
J.M. Geusebroek et al. Color Invariance. PAMI, 2001.
T. Gevers, H. Stokman. Robust histogram construction from colorinvariance for object recognition. PAMI, 2004.
J d W ij C S h id C l i l l f t t ti ECCV 2006 J. van de Weijer, C. Schmid. Coloring local feature extraction. ECCV, 2006
B.A. Maxwell et al. A Bi-illuminant Dichromatic Reflection Model for Understanding Images, CVPR, 2008.
T. Zickler et al. Color Subspaces as Photometric Invariants. IJCV, 2008.
10
Color Differential Structure
27/09/2010
23
differential-based computer vision
1. How do we combine the differential structure of the various color channels ?
2. How do we incorporate color invariance theory into the measurements of the differential structure while maintaining robustness ?
isoluminance
luminance gradient: isoluminant edges are not detected.
27/09/2010
24
Color Feature Detection
0-order structure 1-order structure
Color FeatureDetection
vector: xR xG+ 0=
from luminance to color
red green
+ =
+ =2‐channeltest‐image
22
22
yyyxyx
yxyxxx
GRGGRR
GGRRGR=tensor:
2
2
yyx
yxx
RRR
RRR
2
2
yyx
yxx
GGG
GGG+
DiZenzo. “Note on the Gradient of a Multi-Image”, Computer Vision, Graphics, and Image Processing, 1986.
27/09/2010
25
feature detection in oriented patterns
more tensor‐based features:
Harris corner points
symmetry points
oriented texturetraditional orientation estimation:
f f
(star and circle structures)
optical flow
orientation estimation
curvature estimation
...
arctan arctany y
x x
f f
f f
tensor‐based orientation estimation:
2 2 2 2
2 2arctan arctanx y x y
x y x y
f f f f
f f f f
differential-based computer vision
1. How do we combine the differential structure of the various color channels ?
2. How do we incorporate color invariance theory into the measurements of the differential structure while maintaining robustness ?
27/09/2010
26
• we differ between three types of edges1. material edge
Photometric Invariant Edge Detection
2. shadow/shading edge3. specular edge
• assumptions:1. white illumination2. neutral interface reflection3. shadows are not colored.
shadow
shading
material
specularity
Computation of quasi-invariance
IMAGE FULL FULL INVARIANT
nonlineartransformation
IMAGE FULL INVARIANT
FULL INVARIANT DERIVATIVE
DERIVATIVES
linear operationQUASI- INVARIANT DERIVATIVE
hue huex
27/09/2010
27
Dichromatic Model
• dichromatic model:bbCF ( )ssC
iim Cebbme CF ( )ssm C
body specular
colorant
bbm C
intensity illuminant
• first order photometric structure:bx
bxxxx mBGR CF },,{
( shadowmaterial shading ) specular
iixem C bb
xb
x emme C
Shadow-Shading-Specular Quasi-Invariant
B B B
R
G
opponent colors
R
G
spherical coordinates
R G
=
hue-saturation-intensity
specular variant
specular invariant
shading variant
shading invariant
shading-specular variant
shading-specular invariant
27/09/2010
28
spherical coordinates
• For matte surfaces :bbm cf B
shadow/shading directionR
G
• all shadow-shading variation is in the radial direction
0
uncertainty of cx
x
xrxr
x
xrxrspherical
xBxGxR
x
sin
0
0
0
sin
f
x
xx
sin
0
c
hue-saturation-intensity
• For specular surfaces : bbm cf ssm c
the hue direction
• there is no specular-shadow-shading variation in the hue-direction.
uncertainty of hx
0
0
0 xh
s
xixs
xixsxsh
hsi
xBxGxR
xf
0
0x
x
h
h
27/09/2010
29
invariant edge detection applications
Multi Image Applications • image retrieval
Colo Feature Extraction
Multi Image Applications
Single Image Applications
• image retrieval
• snakes
Color Feature Detection
• feature extraction
Instabilities
full quasiinvariant
test-image
h d h di i ishadow-shading invariance:
0},,{lim
BGR
specular-shadow-shading invariance:
}1,1,1{},,{lim
BGR
27/09/2010
30
Edge Detection
• experiments conducted on pantone colorset (1012) which is used to compose 500.000 edges.
• edge detection is based on the maximum response path of the derivative energy.
• edges are tested on- edge displacement.- percentage of missed edges.
• experiments conducted on pantone color set (1012) which is used to compose 500.000 edges.
shadow-shading:
f ll 0 21 2 0
Edge Detection
• edge detection is based on the maximum response path of the derivative energy.
• edges are tested on- edge displacement.- percentage of missed edges.
specular-shadow-shading:
full
quasi 0.043
0.21
0.99
2.0
full 0.85 9.8p g g
quasi 0.35 5.8
• Conclusion: Quasi invariants more than half the edge displacement, and have higher discriminative power.
27/09/2010
31
experiments : canny edge detection
luminance-gradient RGB-gradient
experiments : canny edge detection
shadow-shading quasi-invariant
shadow-shading-specular quasi-invariant
27/09/2010
32
Edge Classification
xFxF
xrF
shadow edges
xO3xr ,
Edge Classification
xF
specular edges
xO ,3F
27/09/2010
33
Edge Classification
red - object edgegreen-shading/shadow edgeBlue – specular edge
Photometric Invariant Corner Detection
• Harris corner detector combined with the quasi-invariants allows for photometric invariant corner detection
RGB specular-shadow-shading
shadow-shading
27/09/2010
34
experiments : Hough transform
RGB-gradient shadow-shading-specular quasi-invariant
references: color differential structure
S. DiZenzo. A note on the gradient of a multi-image. Computer Vision, Graphics, and Image Processing, 1986.
G. Sapiro and D. Ringach. Anisotropic diffusion of multivalued images with applications to color filtering. IEEE Image Processing, 1996.pp g g g,
J.M. Geusebroek et al. Color Invariance. IEEE Trans. Pattern Analysis and Machine Intelligence, 2001.
J. van de Weijer, Th. Gevers, J-M Geusebroek. Quasi-invariant edge and corner detection, IEEE Trans. Pattern Analysis and Machine Intelligence, 2006.
J. van de Weijer, Th. Gevers, A.W.M. Smeulders, Robust Photometrical Invariant Features from the Color Tensor, IEEE T. Image Processing, 2006.
27/09/2010
35
52
Color Salient Features
Saliency Detection
• Goal: direct our gaze rapidly towards objects of interest in ourenvironment.
• Visual attention is know to be driven by both bottom up (image y p ( gbased) and top-down (task based) cues.
• Bottom-up saliency uses simple visual attributes such as intensity, contrast, color opponency, orientation , direction and velocity of motion.
• What matters is feature contrast rather than absolute feature strength (as in center surround systems).
27/09/2010
36
overview approach
L.Itty, C. Koch “Computational Modelling of Visual Attention”, Nature Reviews Neurosciende, 2001.
Computational Modeling of Visual Attention
L.Itty, C. Koch “Computational Modelling of Visual Attention”, Nature Reviews Neurosciende, 2001.
27/09/2010
37
black-white focus of detectors
luminance-based points color-based points
color distinctiveness
• the information content of an event, v, is equal to :
)log(log pppvpvI fff )log(log yx pppvpvI fff
yyyxxx BGRBGRBGRv
yxH ff ,• equation differential-based salient point detectors :
xxxx ggpp '' ffff Color Boosting Saliency:
To change from strength to information content of edges:
27/09/2010
38
statistics of color images:
• The statistics of is computed by looking of the 40.000 images of the Corel database.
xf
RGB opponent colorspace
• Isosalient surfaces can be approximated by aligned ellipsoids in decorrelated color spaces.
statistics of color images:
xxxx ggpp '' ffff Color Boosting Saliency:
RGB opponent colorspace
27/09/2010
39
• The statistics of is computed by looking of the 40.000 images of the Corel database.
xf
color boosting:
color boosting:
J. van de Weijer, Th. Gevers, A. Bagdanov, Boosting color saliency in image feature detection, IEEE PAMI 2006.
decorrelation whitening
tU
Color boosting:
color boosting:
derivativesRGB color space
derivativesopponent color space
derivativescolor boosted space
J. van de Weijer, Th. Gevers, A. Bagdanov, Boosting color saliency in image feature detection, IEEE PAMI 2006.
27/09/2010
40
bottom-up color attention:
examples:
input image color edges color boosted edgesbottom‐up attention
saliency points
input car-image
RGB-based (first 20 points) saliency boosting (first 4 points)
27/09/2010
41
generality approach: global optimal regions
RGB gradient
color boosting
experiment: image retrieval
Quantitative evaluation of color boosting on a retrieval experiment.
• Nister database: around 10.000 images• detector: DoG (color boosted)• descriptor: SIFT+hue
27/09/2010
42
experiment: image retrieval
retrieval score
theoretical maximum
amount of boosting
• color boosting improves results between 5-10 percent• the obtained maximum score is ‘equal’ to the theoretical maximum.
experiment: edge detection
Berkeley Segmentation Dataset and Benchmar
27/09/2010
43
The do’s and dont’s of Color Features
1. Take care in combining different channels: Tensor-based features solve the opposing vector problem.
2. Look at what kind of photometric invariance your problem needs:
D t t k d i ti f i l l
Quasi-invariants are more stable for feature detection.
3. When working with invariance take instabilities into account.
Use error analysis to find certainty measures for your invariants.
Do not take derivatives of circular color spaces.
Compute first derivatives, then color space transform.
4. When considering photometric invariance always also take discriminativepower into account.p
5. From information theory an optimal color space for salient feature detectioncan be derived.
6. Color information is highly corrupted in compressed data. In compression(jpeg, mpeg) chrominance is subsampled.
Questions ?
top related