Company
LOGO
Mobile Motion Tracking using Onboard Camera
SupervisorProf. LYU, Rung Tsong Michael
Presented byLam Man Kit, Wong Yuk Man
Outline
1. Motivation & Objective1. Motivation & Objective
2. Motion Input 2. Motion Input
6. Experimental Result 6. Experimental Result
5. Applications 5. Applications
3. Previous Work3. Previous Work
4. Improvement & New Components4. Improvement & New Components
7. Demo7. Demo
Motivation
Rapid increase in the use of camera-phone.
Camera-phones’ processing power is increasing
Symbian OS makes programming on mobile phone possible
Objective
Develop motion input method on Symbian phone without requiring additional hardware
This acts as an innovative input method for applications such as: Camera mouse to control the cursor New input method for interactive games Gesture input
What is Motion Input?
Why Motion Input?
Motion input is quick and convenient Common way to play game on mobile
phone is to use joystick or keypad Limited freedom of movement Lack of maneuverability Difficult to press the small-sized keypad
Motion input doesn’t have these problems!
Motion Tracking
Motion tracking is a process to find the motion vector of the current frame from the reference frame
We use block matching algorithm to find the motion vector
Block Matching Algorithm
Comparison of blocks
(dx,dy)
The motion vector which corresponds to
the best match
New Position
Block Matching Algorithm
Evaluate the "goodness" of a match by Sum of Absolute Difference
Select the candidate block with the lowest error
N
i
N
j
jiYjiXyxSAD1 1
|),(),(|),(
Motion Tracking
Block matching algorithm Traditional block matching algorithm is slow Our solution: Hybrid-type fast block matching
algorithm That is “Adaptive Spiral SEA PPNM SSD” algorithm
Adaptive Search Window Spiral Scan Method Successive Elimination Algorithm Progressive Partial Norm Matching Sum of Squared Difference
Previous Work
In the previous semester, Developed a testing platform in both Window and
Symbian OS Developed our own block matching algorithm and
feature selection algorithm Built a real-time translational motion tracking engine Developed some applications using the engine
Improvement on Feature Selection
A good feature block: High variance -> complex block Not in a repeated pattern
If a good feature block is found, the performance of the motion tracking will be improved
We have made improvements to our existing algorithm
squares
Old Feature Selection
Divide the current frame into squares Apply Feature Selection Algorithm to each square However, the best feature block may appear between
two squares
Best feature block
New Feature Selection
More blocks are sampled Sample 26x26 squares with size 15x15 in a
54x54 window Old feature selection only samples 12x12
squares
Search in spiral way and stop searching when selection criteria are matched Prefer to find a feature block in the center Prevent out of bound problem (block appear
out of screen)
New Feature Selection
If a feature block is found on the edge, the following will happen:
The tracking algorithm will fail to find the exact match
Solution: Apply additional
constraint
New Feature selection
In order to find a feature block that is not on the edge, we apply a checking algorithm
Important difference in all directions -> interest point
New Feature Selection
Now our newest feature selection algorithm becomes: Find a block with a large variance which
indicates the complexity of the block Check if the block is on the edge or not by
calculating the SAD between the candidate block and its 4 neighbors
If either one of the SAD is small -> the block is on edge -> reject Else the block is not on edge
Rotation Tracking Engine
Observation and Motivation As we rotate the phone,
the object can still be tracked correctly (center of object always roughly equals center of green box)
First approach Track one block by
translational motion tracking engine, rotate the current block and then find which rotation of the current block best match with the previous block
Fail if tracking object is the same for different angle
One-block approach
Rotation Tracking Engine
Our approach If two blocks are
tracked simultaneously, angle of line connecting the two blocks indicates the rotation angle of the phone
Use only translational motion tracking
Two-block approach
Rotation Tracking Engine
Rotation Tracking Engine
Modification of adaptive search window method to suit rotation tracking
Phone’s motion can be both linear motion and circular motion
Linear Adaptive method used in translational motion tracking is not used in this engine
Why need adaptive search window method? To increase performance based on the fact that
during block matching, reaching the best candidate block earlier increases the elimination effect of SEA, PPNM, PDE
Rotation Tracking Engine
What to predict? Predict the coordinates of the blocks in the next frame Given the coordinates of the tracking blocks in the current frame
and the previous frames When to predict?
After each run of rotation tracking algorithm
θθ
Line L2
Line L1
Line L3
Rotation Tracking Engine
Solution Angle between L2 and L3 = θ
θ = Tan-1(slope(L2)) – Tan-1(slope(L3))
Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix
2
2
1
1
cossin
sincos
L
L
L
L
y
x
y
x
Rotation Tracking Engine
Solution The prediction of the positions of the next tracking blocks
should also take the translational movement into account Horizontal displacement = Tx
Tx = ( xL21 + xL22
– xL31 -
xL32
)/2
Vertical displacement = Ty Ty = ( yL21
+ yL22– yL31
- yL32
)/2
Coordinates of the next tracking block (xL1, yL1) is calculated by
1111
cossin
sincos
12
2
1
1
L
L
L
L
y
x
Ty
Tx
y
x
Rotation Tracking Engine
Reducing error by using level Increase/decrease one level only
when change is large To give less sensitive but more
desirable output for game e.g. skiing game: skier face only
to 7 directions Reduce difficulty of the game,
increase reliability of the engine
40°
80°
120°
- 40° 0°Level 1
Level 2
Level 3
Level 0
Level -1
Rotation Tracking Engine
Some screenshot showing how rotation tracking is carried out and how the result is displayed
Rotation Tracking Engine
A short video demo
Tailor-made background for both engines
Objective Max. performance measurement Increase usability
Condition Feature selection can always find a
good feature point Within certain distance
No repeat pattern Very distinct pattern
Pattern is nearly the same when rotated. E.g. Circle is good for rotation detection
Virtual Mouse
An application that make full use of the translational motion tracking engine
Remote control the mouse of PC by mobile phone using motion as input
Advantages: It allows input in many directions and provides high
levels of control Many buttons left for other purposes
E.g. Joystick can be used as course adjustment of mouse cursor
E.g. Some buttons can be used as shortcuts of applications
Virtual Mouse
Blu
etoo
th C
omm
unic
atio
n C
hann
el
Cable
Virtual Mouse
Server Configure Bluetooth device to provide
RFComm Service and regards the Bluetooth transmission port as Comm. Port
Server receives message from that Comm. Port
Call function in MouseAction.h to make the mouse move and trigger mouse click event
Virtual Mouse
Client Search for Bluetooth device nearby Connect to the selected Bluetooth device Every time motion tracking algorithm
finishes, results are sent to server (14 times/sec)
Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)
Car Racing Game
A “Car Racing Game” is developed using the motion tracking engine.
Game lags. It is because: The engine takes time to find the motion vector CPU speed of the Symbian phones is low
Solutions: Double Buffering Direct Screen Access
Skiing Game
We have developed a skiing game that makes use of the rotation tracking engine Rotate the phone to control the angle of the
skier
After using the rotation tracking engine: The game becomes more interactive The game has more degree of freedom for
rotating the skier
Skiing Game
Skiing Game
The process of using the rotation tracking engine is very simple Create an instance of the tracking engine Call the function of the engine to find the
motion vector Use the motion vector for the game logic
Skiing Game
Experimental Result on Symbian phone
Testing Environment Platform
Symbian Phone Nokia 6600 Algorithm
Our final (hybrid-type) algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method)
Algorithm parameter Block size = 17 x 17 (pixels) Search window size = 16 x 16 (pixels)
Number of block to track in each run of algorithm = 1 block ONLY
Result
Run time of translational motion tracking algorithms
0
10
20
30
40
50
60
background
Tim
e (
ms)
Full Search (ms)
Partial Sprial (ms)
Adaptive SEA (ms)
Experimental Result on Symbian phone
Testing Result (avg time to run) Final algorithm
7ms Partial Spiral algorithm
22ms Full Exhaustive Search algorithm (the most
simplest one) 55ms
Result
Accuracy of the rotational engine
505560657075808590
bg 1 bg 2 bg 3 bg 4 bg 5 bg 6 bg 7 bg 8
measured angle
Conclusion
We have developed: The real-time motion tracking algorithm and
feature selection method The translational and rotational motion
tracking engine on Symbian A series of applications/games that make full
use of our engines
Demo
Virtual Mouse
Demo
Act as an innovative input device for games
Q & A
Thank You !!
Experimental Result1 - Backgrounds
Experimental Result2 - Backgrounds
Stereo
If we want to track the movement in 3D, we have to: Use two cameras Know the focal length of the cameras Know the distance between two cameras
Depth can be recovered with two images Depth can be recovered with two images and triangulationand triangulation
Stereo
Therefore, stereo has two steps Finding matching points in the images Then using them to compute depth.
OO22
P’P’22
PP
OO11
P’P’11
Stereo
Suppose we have found the matching point P. Now the step to compute the depth is as follow:
f: focal length of the cameras
b: distance between two cameras
Note: y-axis is perpendicular to the slide
Stereo
If b is the distance between the two cameras, and f is the focal length of the cameras, we have
Use similar triangular, we can solve to get
z
bx
f
xl 2/'
z
bx
f
xr 2/'
z
y
z
y
f
y rl ''
''
'' 2/)(
rl
rl
xx
xxbx
''
'' 2/)(
rl
rl
yy
yyby
''
rl xx
fbz
Mono
In our project, only one camera is available in the Symbian phone
If we want to find the z-axis movement, we can find it by comparing the size of the capture image Getting closer -> larger image Getting farther -> smaller image
However, we have tried that using this algorithm is not accurate enough