1

Prepared by:

Precise Object Tracking under Deformation

Eng. Mohamed Hassan, EAEA

Supervised by: Prof. Dr. Hussien Konber, Al Azhar University

Prof. Dr. Mohamoud Ashour, EAEA

Dr. Ashraf Aboshosha, EAEA

Submitted to:Communication & Electronics Dept.,

Al Azhar University

2

Key subjects of this framework include: Motivation Visual tracking applications Block diagram of object tracking system Image deformation types Object extraction Morphological operations Geometrical Modeling and pose estimation Conclusion and Future Work

Outlines

3

Motivation

The main objectives of this research work are to:

Overcome the imprecision in object tracking caused by different deformation sources such as noise, change of illumination, blurring, scaling and rotation.

Developing a three dimensional (3D) geometrical model to determine the current pose of an object and predict its future location based on FIR model

Presenting a robust ranging technique to track a visual target instead of the traditional expensive ranging sensors.

4

The precise object tracking is an essential issue in several applications such as:

Robot vision Automated surveillance (civil and military) Medical applications Satellite and space systems Traffic systems Security etc.

Visual Tracking Applications

5

Block Diagram of Object Tracking System

Video Camera

USB Camera

USBBus

Frame grabber

PCImage

Acquisition

ImageProcessing

OutputTarget

6

Image Deformation Types

Noise. Scaling &Rotation. Blurring Change of illumination.

7

Definition: is considered to be any measurement that is not part of the phenomena of interest. Images are affected by different types of noise:

Gaussian noise

Salt and Pepper noise

Poisson Noise

Speckle Noise

Image Deformation: Noise

8

Image De-noising Techniques

The following digital filters have been employed for denoising

Linear filter (Average filter, Gaussian filter and unsharp filter)

Non linear filter (Median filter and Adaptive filter)

Coiflet Wavelets

Proposed filter

9

Spatial filtering term is the filtering operations that are performed directly on the pixels of an image.

The process consists simply of moving the filter mask from point to point in an image.

At each point (x,y) the response of the filter at that point is calculated using a predefined relationship.

Spatial Filters

10

f(x-1,y-1)f(x-1,y)f(x-1,y+1)

f(x,y-1)f(x,y)f(x,y+1)

f(x+1,y-1)f(x+1,y)f(x+1,y+1) w(-1,-1) w(-1,0) w(-1,1)

w(0,-1) w(0,0) w(0,1)

w(1,-1) w(1,0) w(1,1)

The result is the sum of products of the mask coefficients with the corresponding pixels directly under the mask

Pixels of image

Mask coefficients

w(-1,-1) w(-1,0) w(-1,1)

w(0,-1) w(0,0) w(0,1)

w(1,-1) w(1,0) w(1,1)

Linear Spatial Filters

11

Nonlinear spatial filters also operate on neighborhoods, and the mechanics of sliding a mask past an image are the same as was just outlined.

The filtering operation is based conditionally on the values of the pixels in the neighborhood under consideration.

Order-statistics filters are nonlinear spatial filters whose response is based on ordering (ranking)

Nonlinear Spatial Filters

12

The Wavelet transform is a multiresolution analysis tool which decomposes a signal into different frequency sub bands.

Wavelet transform, due to its excellent localization, has rapidly become an indispensable signal and image processing tool for a variety of applications.

Wavelet denoising attempts to remove the noise present in the signal while preserving the signal characteristics, regardless of its frequency content.

Wavelet Transform

13

Figure 1 The two-dimensional FWT - the analysis filter

Wavelet Transform

Figure 2 Two-scale of two-dimensional decomposition

14

The proposed filter is a cascaded spatial filter based on median fitter and Coiflet wavelets. Its edge-preserving nature makes it useful in cases where edge blurring is undesirable. It is very useful in real object tracking. This filter is the best one for removing all types of noise

Denoising Proposed Filter

I/p image Median filter Coiflet Wavelets O/p image

Figure 3 Cascaded spatial filter based on median fitter and Coiflet wavelets

15

Image Similarity Measure

To validate the efficiency of the previous digital filters the following similarity measures have been applied

2D Cross Correlation

Peak Signal-to-Noise Ratio (PSNR)dB

22

1 1 1

[ * ]i n i n i n

i x i y i x i yi i i

x m y m x m y m

1020 log IMaxPSNR

MSE

1 12

0 0

1, ( , )

m n

i j

MSE I i j k i jmn

P P

16

2D Cross Correlation

Unsharp filter

Average filter

Gaussian filter

Median filter

Adaptive filter

Proposed

filter

Salt and paper noise

0.9234 0.9890 0.6983 0.9809 0.7804 0.9984

Gaussian noise

0.5651 0.9861 0.9446 0.9701 0.9701 0.9876

Poisson noise

0.8270 0.9920 0.9900 0.9910 0.9913 0.9961

Speckle noise

0.6349 0.9879 0.7737 0.8341 0.8547 0.9871

Table 1. 2D cross correlation similarity measure

17

Peak Signal-to-Noise Ratio (PSNR)dB

Unsharp filter

Average filter

Gaussian filter

Median filter

Adaptive filter

Proposed

filter

Salt and paper noise

18.59 27.37 25.49 36.00 22.97 49.48

Gaussian noise

9.94 26.16 23.80 26.42 26.79 32.80

Poisson noise

14.74 28.71 30.21 31.92 32.80 43.16

Speckle noise

10.86 26.73 25.38 26.71 27.59 37.67

Table 2. PSNR similarity measure

18

Scaling & Rotation

Definition: Scaling & rotation is affine Transformation where Straight lines remain straight, and parallel lines remain parallel.

Scaling and Rotation: The linear transformation and radon transformation have been used to recover an image from a rotated and scaled origin.

19

Scaled image

Original image

Scaled &rotated image

Figure 4 Rotated and scaled image

Scaling & Rotation

20

Figure 5 Control point selection

Linear Transformation

21

Original image

Scaled & rotated image recovered image

Figure 6 Recovered by using linear transformation

Linear Transformation

22

Radon transform: This transform is able to transform two dimensional images with lines into a domain of possible line parameters, where each line in the image will give a peak positioned at the corresponding line parameters.Projections can be computed along any angle θ, by use general equation of the Radon transformation:

Radon Transformation

, cos sin

, . is the delta function

R x f x y x y x dydy

where

x' is the perpendicular distance of the beam from the origin and θ is the angle of incidence of the beams.

23

Original image

Figure7 Canny edge detection and edge linking

Edge detection Edge linking

Radon Transformation

24Figure 8 Radon transform projections along 180 degrees, from -90 to +89

Radon Transformation

25

Original image

Rotated image recovered image

Figure 9 Recovered by using radon transform

Radon Transformation

26

Blurring: degradation of an image can be caused by motion

There are two types of blurring

Known blurring: the length and the angle of blurring are known

Unknown blurring: the length and the angle of blurring are unknown

Blurring

27

Deblurring Techniques

Deblurring using Wiener filter Deblurring using a regularized filter Deblurring using Lucy-Richardson algorithm Deblurring using blind deconvolution algorithm

g = H f + n

A blurred or degraded image can be approximately described by this equation

28

Deblurring using the Blind Deconvolution Algorithm

Figure 10 Deblurring using the blind deconvolution algorithm

29Figure 11, Capability of object tracking under blurring (a, b) with known blur function and after deblurring (c, d

(a) Blurred image (b) Person detection under motion deformation

(c)Deblurred image (d) Person detection indeblurred image

Deblurring Techniques

30

Blurred image correlation with original one

Deblurred image using correct parameters correlation

Deblurring Techniques

31

Deblurred image using longer PSF correlation

Deblurred image using different angle correlation

Figure 12, 2D cross correlation with the deblurring form

Deblurring Techniques

32

Correlation Condition

blurred image with the original one 0.0614

deblurred image with the original one using correct parameters

0.3523

deblurred image with the original one using longer PSF

0.0558

deblurred image with the original one using different angle

0.1231

Table 3, 2D cross correlation with the deblurring form

Deblurring Techniques

33

Change of Illumination

Change of illumination

Color model deformation may happen due to the change in illumination

Proposed solution

Selecting an appropriate color model (RGB, HSV or ycbcr) to overcome the deformation problem

34

RGB Representation

The RGB color model

mapped to a cubeA Representation of additive color mixing

Weak points of the RGB color model

RGB color model is affected by the change of illumination

RGB is non uniform color model

35

HSV Representations

Hue, saturation and intensity are often plotted in cylindrical coordinates with hue the angle, saturation the radius, and intensity the axis.

HSV color wheel conical representation

of the HSV

The cylindrical representation of the HSV

36

Chrominance is defined as the difference between a color and a reference white at the same luminance.

YCbCr Color Model

The conversion from RGB to YCbCr

0.257 0.504 0.098 16

0.148 0.291 0.439 128

0.439 0.368 0.071 128b

r

Y R

C G

BC

The conversion from YCbCr to RGB

1.164 0.000 1.598 16

1.164 0.329 0.813 128

1.164 2.017 0.000 128

R Y

G Cb

B Cr

37

Advantage of YCbCr

The main advantages of this model are: The luminance component (Y) of YCbCr is independent of the

color The skin color cluster is more compact in YCbCr than in other

color space YCbCr has the smallest overlap between skin and non-skin data

in under various illumination conditions. YCbCr is broadly utilized in video compression standards

YCbCr is a family of color spaces used in video systems.

YCbCr is one of two primary color spaces used to represent digital component video (the other is RGB).

38

To track a visual target we have to relay on a segmentation technique such as:ThresholdingClusteringRegion growingEdge-basedPhysical model-basedFrame SubtractionFast block matchingThroughout this framework a color table thresholding segmentation technique has been applied to extract the visual target

Object Extraction

39

Original image

sample

Tracked object

Homogeneous Object Extraction

40

sample

Homogeneous Object Extraction

RGB YCbCrHSV

Figure 13, Comparison of homogeneous object extraction

Original image

41

Original image

Tracked object

sample

Inhomogeneous Object Extraction

42

Original image

RGB

sample

YCbCrHSV

Figure 14, Comparison of inhomogeneous object extraction

Inhomogeneous Object Extraction

43

The most basic morphological operations are dilation and erosion

Morphological operations

Dilation adds pixels to the boundaries of objects in an image. Expand/enlarge objects in the imageFill gaps or bays of insufficient widthFill small holes of sufficiently small sizeConnects objects separated by a distance less than the size of the window

Erosion removes pixels on object boundaries.to erode away the boundaries of regions of foreground pixels (i.e. white pixels, typically). Thus areas of foreground pixels shrink in size, and holes within those areas become larger

44

Opening and Closing are morphological operations which are based on dilation and erosion.

Opening smoothes the contours of objects, breaks narrow isthmuses and eliminates thin protrusions.

Closing also produces the smoothing of sections of contours but fuses narrow breaks, fills gaps in the contour and eliminates small holes.

Opening is basically erosion followed by dilation while closing is dilation followed by erosion.

Morphological operations

45

Binary object Binary after removing extra pixel

Binary object after dilation holes

Binary object after closing

Morphological operations

Figure 15, The effect of the morphological operation

46

Morphological operations

Figure 16, Center of gravity, ellipse fitting and bound box of an image

47

Geometrical Modeling

Figure 17 object tracking at different distance

48

bDN aeWhere, a = 30606.621b=-0.03410108

The relation between distance (D) and no of pixel (N)

Geometrical Modeling

Figure 18. The relation between range (D) and projection size (N)

49

The relation between the range and location of the object in 3D domain

Geometrical Modeling

Figure 19. The relation between the range and location of the object in 3D domain

50

1

nT

i

y t au t i e t a u t e t

Motion Estimation and Prediction based on FIR

Figure 19, FIR model structures

51

Motion Estimation and Prediction based on FIR

Figure 20, Models output w.r.t system output

52

Motion Estimation and Prediction based on FIR

Figure 21 Model output w.r.t system output

53

Motion Estimation and Prediction based on FIR

Figure 22 The capability of the model to predict the output if the system input is known

54

Throughout this framework the following academic tasks have been achieved

Developing a novel Universal filter for image denoising Selecting qualitative radon transformation for correction of the

rotation Intensive comparative study for dealing with kwon/unknown

bulrring Employing a color table thresholding segmentation technique on

YCbCr to extract the visual target 3D Geometrical modeling for estimation and prediction of target

pose As a future work, we are going to implement the applied

algorithm on an embedded system to develop a visual RADAR System

Conclusion and Future Work

55