1

MOTION VECTOR PROCESSING USING THE

COLOR INFORMATION

Ai-Mei Huang and Truong NguyenImage Processing (ICIP), 2009 16th IEEE International Conference

on

2

CONTENTSINTRODUCTION

COLOR INFORMATION

MV PROCESSING FOR MCFI USING THE COLOR INFORMATION

SIMULATIONS

CONCLUSIONS

3

Introduction(1/2)

Color information has been shown to be effective in the object edges detection Due to its insensitivity on specular reflection Prevent false edge detection as compared to

luminance-based methods

Y1 Y2

Y3 Y4Cb Cr8

8

8

8

16

16

MB (Macro Block)

Chrominance(color information)Luminance(intensity)

4

Introduction(2/2)Color has sharper and more consistent

variations between object boundaries Applications often take the color information to assist

the image segmentation process.In our previous work [8]

The color information was found very useful for the unreliable MV detection

Especially in the areas where the luminance component tends to distribute uniformly.

In this paper, we would like to Examine the color information Analyze how the chrominance components can be

used to assist the MV processing in MCFI.[8] A.-M. Huang and T. Nguyen, “A novel multi-stage motion vectorprocessing method for motion compensated frame interpolation,”in Proc. ICIP’07, pp. 389–392, 2007.

5

Color Information(1/5)

The luminance components have stronger intensity distribution than the chrominance components Conventional motion estimation often ignore the color

information due to the complexity.

Y

Cb

Cr

6

Color Information(2/5)

Color characteristics are distinct from luminance Such as the insensitivity in highlight or shadow areas Used in preventing the false edge detection

The chrominance improves the edge identification for the static text, face features, the cap, and the shirts

7

Color Information(3/5)

If the moving objects have sharp edges, the ambiguous motions seem more unlikely to appear.

From Fig. 1(b), we can observe that the luminance has very smooth variations around the face and shirts areas. Since the motion is mainly determined using the

luminance difference, the motion can be easily wrong in these areas.

8

Color Information(4/5)

Many artifacts around the nose and the shirts in (a)MVs around the shirts can only be detected in (c)

The luminance have uniformly distribution so that the encoder always chooses the face motion. The color has strong gradients.

9

Color Information(5/5) Generally, chrominance residual distribution is similar to the luminancecomponents. The pavement and lawn have very similar intensity. The color difference will become relatively large.

10

Motion Vector AnalysisThe residual energy with color consideration

be represented as follows:

rY (i, j), rCb(i, j), and rCr(i, j) are the reconstructed residual signals of Y, Cb and Cr components of the 8×8 block, bm,n

α is the weight used to emphasize the degree of color difference. Empirically set α=8 for 4:2:0 YUV

The residues are embedded in the reconstructed signals during the decoding process.

11

MV classification process

Compare Em,n to a predefined threshold, ε1, based on the combined residual information.

The adjacent MBs will be merged as a group using the residual energy distribution.

16

16

MB

12

Motion Vector Correction using the Color Information

Minimizing the absolute bidirectional prediction difference (ABPD) between forward and backward predictions.

13

SIMULATIONSTwo video sequences, FOREMAN and FORMULA 1

CIF frame resolutionall encoded using H.263

with even frames skipped and the skipped frames are interpolated

14

Visual Comparisons(1/2)

Fig. 4(c), the artifacts around the nose and the eye are reduced. These artifacts are removed in Fig. 4(d).

Since the chrominance information sharpens the residual energy. Unreliable MVs around the shirts and face areas can be identified and be corrected accordingly.

15

Visual Comparisons(2/2)

The intensity between grass and pavement is very similar.

So the motion estimation easily fails on the white lines areas.

16

CONCLUSIONSWe present using color information for the

MV reliability classification.

Unreliable MVs with small luminance difference can be effectively detected.