carleton motion capture (cmc) matt kracum, will levine, coryn pavelsky, kohei shinkawa jack...
Post on 21-Dec-2015
215 views
TRANSCRIPT
Carleton Motion Capture(CMC)
Matt Kracum, Will Levine, Coryn Pavelsky, Kohei Shinkawa
Jack Goldfeather
Project Goal
• Record a person’s movements– Begin with simple gestures– Store gestures in a computer-readable way
• Display them as naturally as possible in 3D– User interface– Begin with 2D
Tasks for 2D
• Capturing data– Video camera– Mark body parts– Reduce other information
• Media formats– Video file split into frames
Tasks for 2D
• Tracking algorithm– Computer works frame by frame to identify and align points
• Animating and presenting data– Java Open Graphic Library (JOGL)– User interface
2D Marker Tracking
• Builds list of coordinates of each marker in each frame
• Built for colored sticky notes, repurposed for white LEDs
2D Scanning Algorithm
For each picture:For each expected marker:
For each pixel, starting where previous marker left off:If pixel above a threshold:
Mark it, update some data, recursively search neighbors
Save marker’s coordinates
Finding Marker Coordinates
Min X
Max X
Max Y
Min Y = (Average X, Average Y)
Determining center:
Save these values, run again from stopping point to find another marker
Processing Coordinates & Adding Predictions
• Huge mess of data• Must reorder coordinates• Predict missing points based on previous
frames
Reordering Coordinates
Picture 1 Head L. Shoulder R. Shoulder Foot
Picture 2 Head L. Shoulder R. Shoulder Foot
Picture 3 Head R. Shoulder L. Shoulder Foot
Double array of coordinates:
Reordering Coordinates
Picture 1 Head L. Shoulder R. Shoulder Foot
Picture 2 Head L. Shoulder R. Shoulder Foot
Picture 3 Head R. Shoulder L. Shoulder Foot
Double array of coordinates:
Reordering Coordinates
Picture 1 Head L. Shoulder R. Shoulder Foot
Picture 2 Head L. Shoulder R. Shoulder Foot
Picture 3 Head L. Shoulder R. Shoulder Foot
Double array of coordinates:
Predicting Coordinates
Picture 1 Head L. Shoulder R. Shoulder Foot
Picture 2 Head L. Shoulder R. Shoulder Foot
Picture 3 Head L. Shoulder Foot ????
Double array of coordinates:
Predicting Coordinates
Picture 1 Head L. Shoulder R. Shoulder Foot
Picture 2 Head L. Shoulder R. Shoulder Foot
Picture 3 Head L. Shoulder ???? Foot
Double array of coordinates:
Predicting Coordinates
Picture 1 Head L. Shoulder R. Shoulder Foot
Picture 2 Head L. Shoulder R. Shoulder Foot
Picture 3 Head L. Shoulder R. S. Guess Foot
Double array of coordinates:
R. S. Guess = R. Shoulder2 + (R. Shoulder2 – R. Shoulder1)
Send or Save Data
• Send double array to Animation Panel• Save data to a file
Out File
Double Array
Marker Tracking Animation Panel
OR
2D Visualization
• Run video & animation in parallel at 30 fps• Find proper Image format• Creating animation from data
Objectives
Display image at 30 FPS
• 30 FPS = 1/30 second per frame– Roughly the minimum rate human eyes can
not observe any flickering
• FPSAnimator – OpenGL Subclass can achieve a target
frames-per-second rate to avoid using all CPU time.
In 1/30 sec the program has to do…
• Animation– Figure where each point is from data file– Draw points and lines to show the corresponding stick man
• Video– Load a image file from directory– Render image to the panel
If even one of the processes costs more time, everything will be delayed
How do we run 2 threads in sync?
• SemaphoreAllow thread A to update its image only when semaphore is down. Likewise, thread B can only update its image when it’s down. Thread B after successful update increases the current position by 1.
Find proper Image format
• Two candidates for the image format– bitmap• Lossless compression• High quality
– jpg• Lossy compression• Low quality
For our image displaying, the quality of each image didn’t really matter.
Creating animation from data
• All the vertex information was stored in comma-separated value (CSV) file format
• Information is pre-ordered during the tracking phase
• Load all the information prior to drawing animation to reduce the I/O cost
Working with GUI
• Deploying Model-View-Control (MVC) model
•View: Two threads drawing images to panels. Check the model class every time it update the display.
•Model: Wrapper class shared among threads and controller class
•Controller: Directly listens to the GUI. Updates the model upon calls from the GUI
Challenges
• Recording & Processing– Marker changes– Thresholding– Camera limitations– Bounding, disabling guesses
• Animating & GUI
Tasks for 3D Upgrade
• Upgrade marker tracking– Use the ‘Z’ coordinates
• Implement math– Reconciling multiple camera views– Linear algebra
• Implement 3D animation
Tasks for 3D Upgrade
• Upgrade GUI– Change perspective in 3D model– Keyboard and mouse listeners, changing matrices
• Film for 3D– Synchronize cameras– Determine camera positions
3D Math
• Find world coordinates using 2 cameras• Find camera rotation matrix• Find camera translation vector
Rotation
We know that some rotation matrix R times V is in the plane of v. Thus we
get the equation:
n . (R*V) = 0unknown
Rotation
V is a vector parallel to the x axis, thus it can be represented as (s, 0, 0).
r11 r12 r13r21 r22 r33r31 r32 r33
( ( s00) =( r11
r21r31)R*V =
Rotation
n . = 0( r11r21r31)
a*r11 + b*r21 + c*r31 = 0
We have 3 equations and 3 unknowns, solve for the unknowns. Repeat for
each axis.
Translation
Since (T – P) is in the same plane as v and we know the normal to that
plane, we get the equation:
n . (R*(T - P)) = 0UnknownT = (t1, t2, t3)
Translation
This simplifies to:
a*t1 + b*t2 + c*t3 = 0X translate Y translate Z translate
We have 3 equations and 3 unknowns, solve for the unknowns.
Rotation
Calculate the rotation from world coordinates to camera coordinates using known vectors parallel to the
world coordinate system.
Translation
Calculate the translation from world points to camera points
using our calculated R and known points.
3D Coordinates
Find the 3D coordinates of a point seen through 2 or more cameras
using our translation and rotation.
3D Coordinates
The world point must be somewhere along W, the equation
for it is:
WC = T - s(R-1*c)Unknown
Scalar for WUnknown
World coordinate
Translationvector
Inverse rotationmatrix
Camera vector
3D Coordinates
We construct the equation:
(WC1 - WC2)2 = 0
(T1 - s1(R1-1*c1)) - (T2 - s2(R2-1*c2))2 = 0
OR
3D Coordinates
Take the derivative with respect to the 2 unknowns s1 and s2 which is:
a1*s1 + b1*s2 + c1 = 0
a2*s1 + b2*s2 + c2 = 0
Errors
• No solutions to equations for rotation and translation due to human error.
• Using least square sum we minimize instead of solving.
• More math, conceptually the same.
3D Marker Tracking
• Upgrade to 3D:– Input from three cameras– Output real world coordinates
• Fix problems with 2D:– Messy– Inaccurate– Inflexible
Upgrade to Mkr Objects
• Linked-list of these• Stores coordinates & other info• Allows easy prediction• Keeps count of guesses• Disable flag• Easily finds head and last known markers
Linked-List Based Algorithm:
Save marker’s coordinates into a Mkr object, find and link to previous marker’s Mkr object, and fill in missing markers
Linked-Lists of Coordinates
Head, picture 1:X, Y, Z coordinatesHead, picture 1:X, Y, Z coordinates
Head, picture 2:X, Y, Z coordinatesHead, picture 2:X, Y, Z coordinates
Head, picture 3:X, Y, Z coordinatesHead, picture 3:X, Y, Z coordinates
Left Shoulder, picture 3:X, Y, Z coordinatesLeft Shoulder, picture 3:X, Y, Z coordinates
Left Shoulder, picture 2:X, Y, Z coordinatesLeft Shoulder, picture 2:X, Y, Z coordinates
Left Shoulder, picture 1:X, Y, Z coordinatesLeft Shoulder, picture 1:X, Y, Z coordinates
Picture 2Coordinates
Camera 1Camera 1
hhhhh
Reordering & Missing Markers
HeadHead
HeadHead
HandHand
HeadHead
HandHand
FootFoot
FootFoot
FootFootPicture 3coordinates
Camera 1Camera 1
hhhhh
Reordering & Missing Markers
HeadHead
HeadHead
HandHand
HeadHead
HandHand
FootFoot
FootFoot
FootFootPicture 3coordinates
Just change pointers!
Camera 1Camera 1
hhhhh
Reordering & Missing Markers
HeadHead
HeadHead
HandHand
HandHand
FootFoot
FootFootPicture 3coordinates
Mkr objects keep track of levels, so easy to find what’s missing
HeadHead FootFoot
Camera 1Camera 1
Reordering & Missing Markers
HeadHead
HeadHead
HandHand
HandHand
FootFoot
FootFootPicture 3coordinates HeadHead FootFoot
HandHandHandHand
HandHand
Camera 2Camera 2 Camera 3Camera 3
World CoordinatesWorld Coordinates
HandHand
Rotate & translate back
Camera 1Camera 1
hhhhh
Reordering & Missing Markers
HeadHead
HeadHead
HandHand
HandHand
FootFoot
FootFoot
GuessGuessPicture 3coordinates
-Otherwise, predict from previous Mkr objects in list & increments guess count-If guess count too big, revert to last known coordinates
HeadHead FootFoot
Camera 1Camera 1
Creating World Coordinates
HeadHead HandHand FootFoot
HeadHead HandHand
HeadHead FootFootHandHandFrame 1
Camera 1Camera 1 Camera 3Camera 3
Camera 2Camera 2
HeadHeadFrame 1
World CoordinatesWorld Coordinates
FootFoot
Creating World Coordinates
HeadHead HandHand FootFoot
FootFootHandHand
HeadHead FootFootHandHandFrame 1
Camera 1Camera 1 Camera 3Camera 3
Camera 2Camera 2
Frame 1
World CoordinatesWorld Coordinates
HeadHead
HeadHead FootFootHandHand
Repeat for Next Frame
Frame 1
World CoordinatesWorld Coordinates
HeadHead FootFootHandHand
HeadHeadFrame 2
3D Visualization
• Constructing 3D world• Obtaining information from .obj files• Attaching models properly
Constructing 3D world
– Modelview • Modelview transformations are used to position the
items in the scene.
– Projection• Projection transformations defines how the 3D scene is
projected in the 2D screen.
Obtaining models from .obj file
• Dissect polygons into groups to form different body parts
• Assign different color coordination to each group
Attaching models properly
• Algorithm1. Draw models with size = 1 on (0,0,0) coordinates
aligned to z-axis2. Move model to sit on the lower point3. Adjust model’s coordinate to the vector formed
by the two points and its orthogonal vectors 4. Scale model to fit to the actual size
Mathematics for moving objectFirst Vector (PO)
Z.x = P.x - O.xZ.y = P.y - O.yZ.z = P.z - O.z
Second Vector (orthogonal to PO)Y.x = Z.yY.y = -Z.zY.z = 0
Third Vector (Cross product of first and second vector)X.x = Y.y * Z.z - Y.z * Z.yX.y = Y.z * Z.x - Y.x * Z.zX.z = Y.x * Z.y - Y.y * Z.x
Final MatrixX.x X.y X.z 0Y.x Y.y Y.z 0Z.x Z.y Z.z 00 0 0 1
For the Future
• Better equipment– Cameras– LEDs
• Improved tracking algorithm– Predictions– Bounding
For the Future
• Independence from Mathematica– Porting equation code to Java
• Wish: real time streaming!– Computer would show you as you move.