speech and gesture corpus from designing to piloting
DESCRIPTION
Gheida Shahrour Supervised by Prof. Martin Russell Dr Neil Cooke. Electronic, Electrical and Computer Engineering University of Birmingham. Speech and Gesture Corpus From Designing to Piloting. Our research focuses on modelling human behaviour from body motion. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/1.jpg)
Speech and Gesture CorpusSpeech and Gesture CorpusFrom Designing to PilotingFrom Designing to Piloting
Gheida Shahrour
Supervised by
Prof. Martin Russell Dr Neil Cooke
Electronic, Electrical and Computer Engineering University of Birmingham
![Page 2: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/2.jpg)
MotivationMotivation
Our research focuses on modelling human behaviour from body motion.
No dataset which could serve our research focus.
![Page 3: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/3.jpg)
Dataset SpecificationDataset Specification
We need data that:
Contains the motion of people’s head, arms and hands
Captured from people come from different cultural backgrounds
Contains spontaneous speech
Captured using a marker-based tracking technique
![Page 4: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/4.jpg)
Why Marker-based Tracking Technique?Why Marker-based Tracking Technique?
Capturing people’s gestures is mainly based on computer vision techniques:• skin colour- people’s skin & light in images.• contour of people- objects may overlap/occluded• tracking from sequence of frames-may not be accurate• images are from 2D- accuracy issues.
To Avoid these problemsWe will capture gestures using marker-based
optical motion tracking:• data obtained from 3D coordinate system• less occlusion & recovered easily• tracking the object accurately- good calibration• tracking the light-reflective markers- accuracy.
![Page 5: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/5.jpg)
Qualisys Track Manager (QTM) Qualisys Track Manager (QTM) The Balance and Posture Laboratory in the School of Psychology
equipped with QTM system (http://www.qualisys.com):
1. 12 cameras with LED strobes which emits a beam of infrared light which is not visible to the naked eye.
2. QTM Software & Analogue Interface for recording speech
3. passive markers- different sizes
4. calibration Kit: axis L shape & wand T shape.
http://www.qualisys.com
![Page 6: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/6.jpg)
Camera & StrobeCamera & Strobe
http://www.qualisys.com
![Page 7: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/7.jpg)
How it works? How it works?
1.The spherical markers are coated with a material to amplify their brightness.
2.The strobes project light towards the markers and the markers reflect it back to the camera
3.Then the camera system measures a 2-dimensional position of the reflective target by combining the 2-D data from several cameras.
4.The camera uses the reflected data from multiple cameras to calculate the 3D position of the markers with high spatial resolution.
.
![Page 8: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/8.jpg)
How it works? How it works?
![Page 9: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/9.jpg)
The Process of Capturing dataThe Process of Capturing data Attach markers on the objects of interest- how? Define the measurement area where subjects will
stand Test the area Calibrate the area Capture your data Save your data
.
![Page 10: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/10.jpg)
Reprocess Data FilesReprocess Data Files Reprocess the files you captured to construct the 3D view-how?
![Page 11: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/11.jpg)
Labeling DataLabeling DataLabel your data – how?
1. Create a text file- Unique name
2. Unique colour
3. Upload the file
4. Drag & drop
5. Play the motion data
6. Play it again
7. Fill the gap
8. Play it again
9. Save the file
10.Export the data
![Page 12: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/12.jpg)
Experiments (1)_Methods & MaterialsExperiments (1)_Methods & Materials
2 volunteers each wears 36 7mm flat-based half spherical markers on:
- head(4) - elbows(2) - waist(4) - golf gloves(26).
12 cameras & measurement volume is not specified
frame rate: 200 frames per second
speech is not recorded.
![Page 13: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/13.jpg)
Experiments (1)3D ViewExperiments (1)3D View
![Page 14: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/14.jpg)
Experiments (1)Best ResultExperiments (1)Best Result
![Page 15: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/15.jpg)
Experiment (2)_MotivationExperiment (2)_Motivation To improve the quality of data.1. Quantity: number of unidentified markers’ trajectories
should be the same number of the markers used in the experiment.
2. Quality: No loss of markers, ghost markers The technique: the reduction both the number of
markers & the measurement volume
![Page 16: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/16.jpg)
Typical 3D Data & Cameras PositionTypical 3D Data & Cameras Position
![Page 17: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/17.jpg)
Low Vs High 3D Tracker ParametersLow Vs High 3D Tracker Parameters Prediction error Residual: the remaining of the trajectory set to low Filling gaps between frames
http://www.qualisys.com
![Page 18: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/18.jpg)
Markers’ Trajectories & Filling the Markers’ Trajectories & Filling the GapGap
![Page 19: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/19.jpg)
Missing DataMissing Data
![Page 20: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/20.jpg)
How to Fill these Gaps?How to Fill these Gaps?
![Page 21: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/21.jpg)
ExperimentsExperiments (2)_Methods & Materials (2)_Methods & Materials
3 volunteers each wears 28 7mm flat-based half spherical markers on:
- head(4) - elbows(2) - shoulders(2) - waist(4) - golf gloves(16)
![Page 22: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/22.jpg)
Experiments (2)_ Measurement VolumeExperiments (2)_ Measurement Volume
![Page 23: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/23.jpg)
Experiments (1)_Cameras PositionExperiments (1)_Cameras Position
![Page 24: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/24.jpg)
ExperimentsExperiments (2)_Cameras Position (2)_Cameras Position
![Page 25: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/25.jpg)
Experiments (2)_SessionsExperiments (2)_Sessions
![Page 26: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/26.jpg)
ExperimentsExperiments (2)_Result (2)_Result
![Page 27: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/27.jpg)
ExperimentsExperiments (2)_Result (2)_Result
![Page 28: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/28.jpg)
ConclusionConclusionWe will track motion of head, arms and handLeave 3 fingers out: middle, ring and pink.Occlusion of the markers on fingers is not only
due to the cameras set up, but also due to the degree of freedom of the hands
Finding unidentified trajectories of markers is laborious and time consuming.
Tracking all fingers is very useful for many applications such as Sign Language but this is not our focus.
![Page 29: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/29.jpg)
Data collection_ assignmentData collection_ assignmentEach volunteer will wear not less than 12mm passive markers on
head(4), elbows(2), waist(4), shoulder(3) and gloves(10)
![Page 30: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/30.jpg)
Group SetupGroup Setup Put yourselves into groups of 3. The members of each group should be from the same first
language, same gender & same country of birth
Each member in British group (country of birth is Britain & first language is English) will record 2 sessions. Each session will last 15 minutes captured in 5 stages. Each stage lasts for 3 minutes.
Each member in the cultural group (country of birth is not Britain & first language is not English) will record 4 sessions. 2 sessions in English as a Second Language and 2 in their first language. Each session will last 15 minutes captured in 5 stages. Each stage lasts for 3 minutes.
![Page 31: Speech and Gesture Corpus From Designing to Piloting](https://reader036.vdocument.in/reader036/viewer/2022062802/56814481550346895db118c9/html5/thumbnails/31.jpg)
Any Question?