![Page 1: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/1.jpg)
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis Group
Automatic Performance Analysis in Trampoline from
RGB-Depth Data
Carlos Puig Toledo
Advisors: Sergio Escalera, Jordi González, Xavier Baró and Albert Clapés
![Page 2: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/2.jpg)
Human Pose Recovery and Behavior Analysis Group
Outline
• Introduction – The project
– The trampoline sport
– Kinect
• Method – Data acquisition
– The perspective problem
– Finding the landing points
• Results
• Conclusions
2
![Page 3: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/3.jpg)
Human Pose Recovery and Behavior Analysis Group
Project Introduction
• In collaboration with CAR Sant Cugat and ASCAMM
• Objectives: – Capture multi-modal RGB-Depth data
– Extract the landing points
• Transform perspective view
• Estimate jump location
– Evaluate
Introduction
3
![Page 4: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/4.jpg)
Human Pose Recovery and Behavior Analysis Group
The trampoline sport
Introduction
4
![Page 5: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/5.jpg)
Human Pose Recovery and Behavior Analysis Group
The trampoline sport
Chair of judges deductions
Touching the bed with 1 or both hands 0.4
Touching the bed with knees or hands & knees, falling to seat, front or back 0.6
Touching the springs, pads, frame or safety platform 0.6
Landing/falling on the springs, pads, frame or safety platform & spotter mat 0.8
Landing/falling outside the area of the trampoline 1.0
Introduction
5
![Page 6: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/6.jpg)
Human Pose Recovery and Behavior Analysis Group
What is Microsoft Kinect?
• Structured light scanner
• Packaged into a single unit: 1. Depth sensor:
‐ Infrared laser projector with monochrome CMOS sensor
‐ 11-bit VGA depth (2048 levels of sensitivity)
‐ Any ambient light condition
‐ 1-8 meters range
2. RGB camera
‐ 8-bit VGA resolution (640x480)
‐ 30 FPS
3. Motorized tilt
4. Multi-array microphone
Introduction
6
![Page 7: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/7.jpg)
Human Pose Recovery and Behavior Analysis Group
How it works?
• Speckle pattern of dots is projected by IR projector
• IR camera captures the pattern
• For each dot in reference pattern: – Find the dot in the observed pattern
– Compute the depth:
• Disparity
• Focal length
• Baseline between projector and camera
Introduction
7
![Page 8: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/8.jpg)
Human Pose Recovery and Behavior Analysis Group
Data Acquisition
• Setup the camera
• RGB-Depth Syncronization and Registration
Method
RGB Depth Depth registration 8
![Page 9: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/9.jpg)
Human Pose Recovery and Behavior Analysis Group
The perspective problem
• Mesh not occluded
• Segment blue mat – GMM in Hue channel
Method
0.58 0.6 0.62 0.64 0.66 0.680
5
10
15
20
25
30
35
9
![Page 10: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/10.jpg)
Human Pose Recovery and Behavior Analysis Group
The perspective problem
• Morphological operations – Opening
– Closing
• Flood-fill algorithm
Method
10
![Page 11: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/11.jpg)
Human Pose Recovery and Behavior Analysis Group
The perspective problem
• Edge parameters – Canny
– Hough transform
• Homography estimation
Method
11
![Page 12: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/12.jpg)
Human Pose Recovery and Behavior Analysis Group
Finding the landing points
• Construct point cloud from depth map
• Segment the blue mat
• Extract the mesh
12
Method
![Page 13: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/13.jpg)
Human Pose Recovery and Behavior Analysis Group
Finding the landing points
• Estimate plane parameteres with planar segmentation
• 3D transformations – Translate to the origin
– Rotate to match x-z plane
• Cropbox filter
13
Method
![Page 14: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/14.jpg)
Human Pose Recovery and Behavior Analysis Group
Finding the landing points
• Estimate the landing point – When?
– Where?
• Euclidean cluster extraction
14
Method
![Page 15: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/15.jpg)
Human Pose Recovery and Behavior Analysis Group
Perspective
15
Results
![Page 16: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/16.jpg)
Human Pose Recovery and Behavior Analysis Group
Landing points
16
Results
![Page 17: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/17.jpg)
Human Pose Recovery and Behavior Analysis Group
Landing points
17
Results
Difference
10 – 20 cm < 10 cm > 20 cm
![Page 18: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/18.jpg)
Human Pose Recovery and Behavior Analysis Group
Conclusions
• Landing points are estimated from multi-modal RGB-Depth data from a Kinect camera.
• Athletes can automatically know their performance.
• Robust system that accomplishes the requirements defined by athletes, trainers and judges.
• Cheap and easy to setup.
• Volunteer performance vs. Athlete performance.
18
Conclusion
![Page 19: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/19.jpg)
Human Pose Recovery and Behavior Analysis Group
Future work
• Perform more field tests.
• Detail some requirements.
• Make the system more robust to changes in the scenario.
• Increase the frame rate.
• More specifications could be added:
– Athlete position at landing.
– History of jumps with each landing location.
19
Conclusion
![Page 20: Automatic Performance Analysis in Trampoline from RGB-Depth …sergio/linked/masterthesis_presentation_carlospuig.pdf · Automatic Performance Analysis in Trampoline from RGB-Depth](https://reader033.vdocument.in/reader033/viewer/2022042107/5e862fdac8f49432dc0945ac/html5/thumbnails/20.jpg)
Human Pose Recovery and Behavior Analysis Group
20
Thanks for your attention!