Download - 5. Video Object Tracking and Processing
1
5. Video Object Tracking and Processing
To achieve augmented reality, computer generated graphics should be shown together with the live video
In addition, the generated graphics should be changed according to the live video contents
Hence we need to be able to understand the contents of the live video
References:1. Gonzalez, Rafael C., Digital image processing, Prentice-Hall, 2002. 2. The Mandelbrot DAZIBAO, “Lesson 5 – The HSV Colorspace”,
http://www.mandelbrot-dazibao.com/HSV/HSV.htm.3. Ogre Tutorials, http://www.ogre3d.org/wiki/index.php/Ogre_Tutorials
2
Project Requirements
In this project, a few color balls are randomly placed on the racing track
You are required to develop computer programs that can detect the color balls in the live video obtained from the camera installed on the robot car
Computer graphics should be generated to cover the areas in the video where the balls are found
The size and position of the graphics should be adjusted according to the distance and relative direction between the balls and the robot car (i.e. the camera)
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
3
Procedure for Detecting the Color Balls Grab a frame of the live video Convert the video data from RGB color space to HSV
color space HSV color space is more close to human description to colors
Select all pixels in the image that matches with the color of the ball Assume there are no other objects in the video that have the
same color and similar shape Remove the isolated pixels (optional)
Assume noise in the images may have similar color as the ball but exists in the form of isolated pixels
Locate the center of the remaining pixels Assume the remaining pixels all belong to the ball of interested
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
4
RGB Color Space For the current digital displays, each pixel is made of a Red, a
Green and a Blue sub-pixel Each sub-pixel is activated by a byte of data, i.e. a number
between 0 and 255 With these three bytes, the computer can generate 256x256x256
= 16,777,216 combinations If R = x, G = y and B = z, we can plot an RGB cube as follows:
While we have a huge amount of possibilities, it is also tricky to define a precise color by adjusting the three components: If R=G=B=0, we have black If R=G=B=255, we have white If R=G=255 and B=0, we have yellow But how about the others???
5
HSV Color Space HSV codification provides an
intuitive method for color selection. Closer to human perception: The blend of the three
components is defined by a single parameter called "Hue"
The "Saturation" parameters selects how grey or pure the color will be
The "Value" parameter defines the brightness of the color
The HSV Color Space can be visualized as a cone
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
6
RGB to HSV Conversion
r – value of Redg – value of Greenb – value of Bluemax – the greatest of (r,
g, b)min – the smallest of (r,
g, b)h [0, 360) is the hue ∈
angle in degrees, ands, v [0, 100] are the ∈
saturation and value
bifgr
gifrb
bgandrifbg
bgandrifbg
if
h
max,240minmax
60
max,120minmax
60
max,360minmax
60
max,0minmax
60
minmax0
otherwise
ifs
,100max
minmax
0max,0
100255
maxv
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
7
Why HSV? The color of an object as shown in the video can differ quite
significantly from what is seen using naked eyes Many environmental factors can affect the color of an object in
the video Exposure of the camera White balance setting Ambient lighting Reflection of the spot lights around
Theoretically, an object of red color should have the RGB value of (255,0,0)
In practice, one can never find such value although the object is really red in color when seeing it with naked eyes
Some tolerance should be allowed if we want to detect an object of red color in the video. But the problem is how to set this tolerance
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
8
Why HSV (cont)?
We should consider an object in the video as red in color if it is “rather red” to “very red”
Besides, a red object should be considered as red no matter it is pictured in a dark environment or a bright environment
How to set such a tolerance? By using the HSV codification
Hue (H) – Define a set of Red color Saturation (S) – Define how Red it is Value (V) – Define how bright it is
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
9
Why HSV (cont)?
Result of using the Paint.NET
• In RGB color space, for every R value, there can be a set of G and B values that will give you a red color
• Difficult to adjust if a set of Red colors are to be defined
• In HSV color space, H defines a set of Red colors
• S defines the saturation• V defines the brightness • A possible set of values for
Red can be• H < 25 & H > 335• S > 45• V > 60
• In HSV color space, H defines a set of Red colors
• S defines the saturation• V defines the brightness • A possible set of values for
Red can be• H < 25 & H > 335• S > 45• V > 60
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
10
Original Video Frame
A blue ball and a red ball are put in front of the camera
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
11
Select the color We set a tolerance as
follows:?? < H < ??; S > ??; V > ??
We compare the HSV value of every pixel of the video frame to see if it falls within the tolerance Yes – mark it as green No – no action
No matter how good is your estimation, there must be some pixels selected not belong to the object of interest They are considered as noise
Noise
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
12
Denoising
Noise on the image is often considered as isolated dots or islands
It can be reduced by using a simple denoising algorithm as follows
• For every 5x5 block, if most pixels have color falls within the tolerance, consider them as a part of the object
• For this block, only a few pixels are green. Hence they are noise
• For this 5x5 block, all of them have been marked as green. They are a part of the object
• Should carry out the above test for the whole image
• For this 5x5 block, all of them have been marked as green. They are a part of the object
• Should carry out the above test for the whole image
13
Denoising Result The above algorithm
should be carried out pixel-by-pixel for the whole image
As a result, most of the noise are removed
Depends on the actual environment, the denoising performance may not be as good as that
Need long time fine-tuning
Noise are removed
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
14
Locate the object
The location of the ball can be easily estimated by summing up all marked pixels row-by-row and column-by-column and look for their peaks The peak
For each column, if a marked pixel is found, add 1
For each row, if a marked pixel is found, add 1
…
…
15
Restriction of the algorithm
Certainly, the algorithm can only detect one blue ball If there are multiple blue balls, multiple peaks will
result Become very difficult to determine which peak
corresponds to which blue ball The difficulty also applies to the situation that there
are other large objects that are also blue in color Multiple balls with different color is not a problem
One can detect balls one color at a time
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
16
Replace the detected object with 3-D graphics
To achieve augmented reality, we need to combine graphics with live video
If the position of the object is detected, we can replace it with 3-D graphics, such as fire generated by Ogre particle system
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
17
Create a fire in Ogre// The particle "Examples/Smoke" is in the file // Example.particle// Assume mParticleSys and mParticleNode are two private // variables of EIE330projectAppmParticleSys = mSceneMgr->createParticleSystem("Smoke",
"Examples/Smoke");mParticleNode = mSceneMgr->getRootSceneNode() ->createChildSceneNode();mParticleNode->setVisible(false);
// Default - invisiblemParticleNode->setPosition(0, 0, -140);
// Set the position of the firemParticleNode->attachObject(mParticleSys);
:
mParticleSys->setDefaultDimensions(220, 220);// Set the size of the fire
18
Resize the fire
Big fire!
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
19
Changing the size and orientation of graphics For a real object on the
screen, its size and orientation will change with the change of the camera position
If a 3-D graphic is used to replace a real object on the screen, its size and orientation should also change with the change of the camera position
Camera is closer to the objectCamera is closer to the object
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
20
Changing the size and orientation of graphics
Camera is moved to the right hand side of the screen
Camera is moved to the right hand side of the screen
Camera is moved to the left hand side of the screen
Camera is moved to the left hand side of the screen
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
21
How do we know the relative direction and distance of camera?
By using the markers Each ball and the robot car will have a marker
associated with it In Lab 6, you are required to use the markers’ data
to estimate the relative distance and direction between the blue ball and the camera
Need to replace the blue ball with a 3-D object that will change in size and orientation according to the position of the camera
marker
marker
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
PC
Camera
22
Task for Lab 5
Modify the function TextureSystem::UpdateTexture()
to implement the following:
Change the color space Select the pixels with red and blue
color
Denoise the selected pixels Detect the position of the red ball Create a fire at the position of the red ball on the screen
23
Task for Lab 6 (1st part) Change the size and
orientation of an Ogre head with respect to the relative position of the camera, which is given in the first part of Lab 6
Procedure Create a Ogre head in EIE330ProjectApp::createScene() Implement the function
EIE330ProjectApp::placingObject(float distance, float direction);// change the size and orientation of the Ogre head// based on the input parameters
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun
24
Task for Lab 6 (2nd part) The blue ball will be replaced by a 3-D object of your
choice on the screen By making use of the markers’ data, estimate the
relative distance and angle between the blue ball and the camera
Modify the function placingObject() such that it will change the size and orientation of the 3-D object based on the estimated relative distance and angle
marker
marker
PC
Camera
25
More instructions
For Lab 5 and Lab 6, each team submits one lab report and does one demo for each lab
No more lecture for Lab 6 Once finishing Lab 5, students can start Lab 6
immediately. The lab sheets for Lab 5 and Lab 6 are available in WebCT
Both Lab 5 and Lab 6 do not need robot car and wireless camera
Use that time to fine-tune your robot car. Don’t wait until the last week!
Department of ELECTRONIC AND INFORMATION ENGINEERING
5. Video Object Tracking and Processing by Dr Daniel Lun