d-bug turkish sign language recognition using microsoft kinect sponsored by innova
TRANSCRIPT
![Page 1: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/1.jpg)
D-BUGTurkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA
![Page 2: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/2.jpg)
GROUP MEMBERS
Duygu Aralıoğlu
Gülnur NevalErdem
Bedia Acar
Ulaş Şahin
![Page 3: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/3.jpg)
OUTLİNE Problem & Solution Project Details Work Done So Far Project Demo Video Future Work References
![Page 4: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/4.jpg)
OUTLİNE
Problem & Solution Project Details Work Done So Far Project Demo Video Future Work References
![Page 5: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/5.jpg)
PROBLEM & SOLUTİON
The main problem is communication problem between speech-impaired people and the others.
Being unable to express himself/herself can be frustrating.
![Page 6: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/6.jpg)
PROBLEM & SOLUTİON Our Solution:
Our goal: To help alleviate the frustration that speech-impaired people face by using assistive technology.
Final Product: Gets the sign language gestures and give the meaning of them in text format.
![Page 7: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/7.jpg)
OUTLİNE Problem & SolutionProject Details Work Done So Far Project Demo Video Future Work References
![Page 8: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/8.jpg)
PROJECT DETAİLS
Components of the System
Environmental Assumptions
Libraries & Tools
![Page 9: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/9.jpg)
COMPONENTS OF THE SYSTEM
InputHandler
InterfaceHandler
HMM
Recognizer
![Page 10: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/10.jpg)
COMPONENTS OF THE SYSTEM
HMM
InterfaceHandler
InputHandler
Recognizer
Jointpositions
Scaled hand
positions
Video
stream
Video stream
Mat
rix in
dice
s
Observation sequence
Probability
Gesture name
Figure 1: Interaction of the Components
![Page 11: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/11.jpg)
COMPONENTS OF THE SYSTEM InputHandler
Repeatedly arranges Kinect input
Calculates hand positions on the screen
Creates SpaceZoningMatrix
![Page 12: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/12.jpg)
COMPONENTS OF THE SYSTEM InputHandler
Figure 2: Space Zoning Matrix
![Page 13: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/13.jpg)
COMPONENTS OF THE SYSTEM InterfaceHandler
Manages the user interface
Moves hand cursors
Changes interface modes
![Page 14: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/14.jpg)
COMPONENTS OF THE SYSTEM What is an HMM?
HMM (Hidden Markov Model) is a probabilistic model created with a state set and an observation set. Model is based on the assumption that a visible observation sequence is triggered by hidden states of the model .
![Page 15: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/15.jpg)
COMPONENTS OF THE SYSTEM HMM
Figure 3: General Hidden Markov Model
Retrieved from http://www.mental.sk/temp/ai/csr.html
![Page 16: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/16.jpg)
COMPONENTS OF THE SYSTEM An HMM is defined with:
Initial state distribution matrix
State transition matrix
Observation matrix
![Page 17: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/17.jpg)
COMPONENTS OF THE SYSTEM Three problems:
Problem 1: Given a model, and a sequence of observations O find the probability of observing O in this model.
![Page 18: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/18.jpg)
COMPONENTS OF THE SYSTEM Three problems:
Problem 2: Given a model and an observation sequence O, determine the optimal state sequences that results in given observation sequence in the given model.
![Page 19: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/19.jpg)
COMPONENTS OF THE SYSTEM Three problems:
Problem 3: Given an observation sequence O and the dimensions N and M, determine the model which maximizes the probability of observing O.
![Page 20: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/20.jpg)
COMPONENTS OF THE SYSTEM
In TSL-Kinect
1. Use the solution to Problem 3 to train an HMM for each gesture.
2. Use the solution to Problem 1 to determinethe gesture is more likely to be “X” or “Y” or neither.
![Page 21: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/21.jpg)
COMPONENTS OF THE SYSTEM
Figure 4: Working principle of HMMs
Retrieved from http://lh5.ggpht.com
![Page 22: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/22.jpg)
COMPONENTS OF THE SYSTEM Recognizer
Creates observation sequence for both hand
Release sequence stacks
![Page 23: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/23.jpg)
ENVİRONMENTAL ASSUMPTİONS
Set up Kinect properly
Stand 1.8 - 2.4 meters away from the sensor
No moving objects around!
![Page 24: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/24.jpg)
LİBRARİES & TOOLS
The Microsoft official SDK for Kinect Microsoft Visual Studio 2010
NUI library Coding4Fun Kinect Toolkit
![Page 25: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/25.jpg)
OUTLİNE Problem & Solution Project DetailsWork Done So Far Project Demo Video Future Work References
![Page 26: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/26.jpg)
WORK DONE SO FAR
Basic user interface
User interface control
Simple gesture recognition
![Page 27: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/27.jpg)
OUTLİNE Problem & Solution Project Details Work Done So FarProject Demo Video Future Work References
![Page 28: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/28.jpg)
PROJECT DEMO VİDEO
![Page 29: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/29.jpg)
OUTLİNE Problem & Solution Project Details Work Done So Far Project Demo VideoFuture Work References
![Page 30: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/30.jpg)
FUTURE WORK
Improvement on user interface
HMM implementation
More gesture recognition
![Page 31: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/31.jpg)
OUTLİNE Problem & Solution Project Details Work Done So Far Project Demo Video Future WorkReferences
![Page 32: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/32.jpg)
REFERENCES
![Page 33: D-BUG Turkish Sign Language Recognition Using Microsoft Kinect Sponsored by INNOVA](https://reader035.vdocument.in/reader035/viewer/2022062423/5697c02a1a28abf838cd7af7/html5/thumbnails/33.jpg)
THANK YOU FOR YOUR ATTENTİON
ANY QUESTIONS