![Page 1: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/1.jpg)
INDIAN INSTITUTE OF TECHNOLOGY ROORKEE
Applications of Machine Learning using Sensors
Dept. of Computer Science and Engineering
Indian Institute of Technology, Roorkee, India
Dr. Partha Pratim Roy
Email: [email protected]
![Page 2: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/2.jpg)
2
Some Applications using Sensors
Sensor based Activity AnalysisGesture Recognition and Rendering Brain Computer Interface
![Page 3: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/3.jpg)
3
Leap Motion and its working area
IR Cameras
IR Blasters
![Page 4: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/4.jpg)
4
Sensor’s output
![Page 5: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/5.jpg)
5
Applications
5
Touchless navigation
Sign language interface
3D Air printing
Augmented Reality
Physical Rehabilitation
Consumer electronic interface
Some Applications of Gesture Recognition
![Page 6: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/6.jpg)
6
Recognition and Rendering of shapes on 2D display device
● Analysis of the motion of fingers in 3D space
● Recognition of natural gestures of 2D/3D geometric and non
geometric shapes using Leap Motion device.
● Rendering the shapes on 2D display device. MuPAD note
book was used for this purpose.
![Page 7: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/7.jpg)
7
Sample of Single-finger Gestures
![Page 8: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/8.jpg)
8
Sample of Multiple-finger Gestures
![Page 9: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/9.jpg)
9
Rendering of Shapes
Natural gestures and their corresponding shape rendering
➢ Single-finger gesture showing a pyramid
➢ Multiple-finger gesture showing a sphere
![Page 10: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/10.jpg)
10
Dataset Details
● Collected data for 36 distinct geometric and non-geometric shapes.● Collected data from 10 volunteer. Each volunteer plays each
gesture for 15 times.
![Page 11: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/11.jpg)
11
Feature Extraction (Extended Npen++ features)
Gesture direction: At every time instant t gesture direction is defined as:
![Page 12: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/12.jpg)
12
Feature extraction contd.
Aspect: The aspect of the trajectory characterizes the height-to-width ratio of the bounding box constituting the neighboring points of [x(t), y(t), z(t)] and is defined as:
![Page 13: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/13.jpg)
13
Feature extraction contd.
● Curliness: Curliness feature as denoted by C(t) measures the deviation from a straight line in the vicinity of point P(t).
● Lineness: Lineness L(t) is defined as the average square of distance between every point in the cubical box of P and the straight-line joining the first and last point
![Page 14: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/14.jpg)
14
Feature extraction contd.
Slope: In 3D, slope is defined as the direction ratios representedby l, m and n with respect to x, y and z dimensions of the straightline joining the start and endpoints within the bounding box.
![Page 15: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/15.jpg)
Classification by Hidden Markov Model (HMM)
15
Achieved 92.87% accuracy when evaluated using 5-fold cross-validation method.
![Page 16: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/16.jpg)
Variations in 3D shape rendering using parameters extracted from
gesture, e.g. “Cylinder”
16
![Page 17: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/17.jpg)
17
Sign Language Recognition
• A sign language is a visual language that uses a system of
manual and non-manual signs as the means of
communication.
• SLR aims to develop methods and algorithms that correctly
identify a given sequence and able to convert it into
text/speech.
• Manual Signs- it involves finger-spelling (air-writing), hand
shapes, orientation and movements of the hands and fingers
during signing.
• Non-manual Signs- it involves various facial expressions,
head tilting, shoulder raising and mouthing that are added
with manual signs.
![Page 18: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/18.jpg)
18
Continuous SLR
• It specify the recognition of continuous signs form the
sentences in the SLR.
• Recognition will take place as per the signs available in
vocabulary.
• A continuous SLR system has been proposed by modifying
the LSTM architecture.
A Mittal, P Kumar, PP Roy, B Raman, B. B. Chowdhuri, Continuous Sign Language Recognition using Leap motion, Pattern Recognition Letters,2018.
![Page 19: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/19.jpg)
19
Continuous SLR
![Page 20: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/20.jpg)
20
Manual Sign Implementation (air-writing)
1. Air-writing: It means writing in the 3D space without pen-paper.
Challenges:
a. No stroke information (No pen up-down information).
b. Continuous writing (words are attached with each other).
c. Large variation among same word written by the same writer.
d. No predefined plane or axis for writing.
e. Noisy signal or Jitter.
P Kumar, R Saini, PP Roy, DP Dogra. Study of text segmentation and recognition using leap motion sensor. IEEE Sensors Journal. 2016;17(5):1293-301.
![Page 21: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/21.jpg)
21
Manual Sign Implementation (air-writing)
![Page 22: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/22.jpg)
22
Air-Signature Authentication
"Analysis of 3D Signatures Recorded Using Leap Motion Sensor", Multimedia Tools and Applications, 2017.
![Page 23: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/23.jpg)
26
Non-manual Sign Recognition
• An example where the manual-signs for the two different
words are same but have different facial expressions.
“Independent Bayesian Classier Combination based Sign Language Recognition using Facial Expression”, Information Sciences,2017.
![Page 24: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/24.jpg)
27
Non-manual Sign Recognition
• Multimodal System set up
![Page 25: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/25.jpg)
28
SLR Learning Interface
• A SLR learning interface has been developed using Blender
tool and NLP techniques.
• The interface works by entering the isolated or complete
sentence to get corresponding signs.
“3D Avatar based Sign Language Learning System”, ACM TALLIP, 2018. (Submitted)
![Page 26: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/26.jpg)
29
Video Surveillance
![Page 27: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/27.jpg)
30
Video Surveillance – Moving Object Detection
“Moving object detection using modified temporal differencing and local fuzzy thresholding”. The Journal of Supercomputing 73(3): 1120-1139 (2017)
![Page 28: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/28.jpg)
31
Continuous Tracking
Ahmed et al., "Surveillance Scene Representation and Trajectory Anomaly Detection Using Aggregation of Multiple Concepts", Expert Systems with Applications, 2018.
![Page 29: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/29.jpg)
32
Human Activity Analysis
A novel framework of continuous human-activity recognition using Kinect. Neurocomputing311: 99-111 (2018)
![Page 30: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/30.jpg)
33
Human Activity Analysis using Depth Sensors
![Page 31: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/31.jpg)
34
Human Activity Analysis
"A novel framework of continuous human-activity recognition using Kinect. Neurocomputing311: 99-111 (2018)
![Page 32: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/32.jpg)
35
Human Interaction Monitoring System
"Kinect sensor-based interaction monitoring system using the BLSTM neural network in healthcare”. International Journal of Machine Learning and Cybernetics (2018).
![Page 33: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/33.jpg)
36
Human Interaction Monitoring System
"Kinect sensor-based interaction monitoring system using the BLSTM neural network in healthcare”. International Journal of Machine Learning and Cybernetics (2018).
![Page 34: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/34.jpg)
37
Electroencephalogram (EEG)
• The electroencephalogram (EEG) measures theactivity of large numbers (populations) of neurons.
• First recorded by Hans Berger in 1929.
• Electrodes measure voltage-differences at the scalpin the microvolt (μV) range.
• Voltage-traces are recorded with millisecondresolution – great advantage over brain imaging(fMRI, MEG).
![Page 35: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/35.jpg)
38
Beyond Clinical Significance of EEG
EEG is used by researchers for the development of various BCI applications:
a. Neuro-Marketing
b. Emotion detection
c. Gaming
d. Security and authentication
e. ERP (Event Related Potential)
f. Wheel Chair control, mouse movement etc.
![Page 36: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/36.jpg)
39
Methods for recording EEG
a. Invasive: recordings that are made with theelectrodes implanted within the depth of brainsurgically.
b. Non-Invasive: recordings that are obtainedfrom electrodes attached over the scalpsurface.
![Page 37: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/37.jpg)
40
EEG Devices
![Page 38: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/38.jpg)
41
EEG Acquisition
Standard placements of electrodes on the human scalp: A, auricle; C, central; F, frontal; Fp, frontal pole; O, occipital; P, parietal; T, temporal.
![Page 39: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/39.jpg)
42
BCI Applications using EEG
![Page 40: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/40.jpg)
43
Sample EEG recordings
![Page 41: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/41.jpg)
44
EEG Characteristics
▪ EEG potentials are good indicators of brain state.
▪ They often display rhythmic patterns at characteristic frequencies
![Page 42: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/42.jpg)
45
EEG Characteristics
• EEG rhythms correlate with patterns of behavior (level of attentiveness, sleeping,
waking, seizures, coma).
• Rhythms occur in distinct frequency ranges:
– Gamma: 20-60 Hz (“cognitive” frequency band)
– Beta: 14-20 Hz (activated cortex)
– Alpha: 8-13 Hz (quiet waking)
– Theta: 4-7 Hz (sleep stages)
– Delta: less than 4 Hz (sleep stages, especially “deep sleep”)
• Higher frequencies: active processing, relatively de-synchronized activity (alert
wakefulness, dream sleep).
• Lower frequencies: strongly synchronized activity (nondreaming sleep, coma).
![Page 43: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/43.jpg)
46
Extraction of Gamma Features (32-100 Hz)
Barjinder Kaur, Dinesh Singh, and Partha Pratim Roy. "A Novel framework of EEG-based user identification
by analyzing music-listening behavior." Multimedia Tools and Applications: 2017
![Page 44: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/44.jpg)
47
Features
• Root Mean Square (RMS)
• Standard Deviation (SD)
• Energy
• Mean (M)
![Page 45: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/45.jpg)
48
Neurophone: Assistive Framework
Pradeep Kumar, Rajkumar Saini, Pawan Sahu, Partha Pratim Roy, Debi Prosad Dogra, Balasubramaninan
Raman, ‘Neuro-phone: An assistive framework to operate Smartphone using EEG signals’, IEEE Tensymp
2017.
![Page 46: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/46.jpg)
49
Basic Applications
![Page 47: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/47.jpg)
50
Results for Neurophone
![Page 48: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/48.jpg)
51
Age and Gender Recognition by EEG
Barjinder Kaur, Partha Pratim Roy, Dinesh Singh. “Age and Gender Classification using BCI.“ Neural Computing & Applications, 2018.
![Page 49: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/49.jpg)
52
Deep Learning Architecture
![Page 50: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/50.jpg)
Results using Deep LSTM-BLSTM
![Page 51: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/51.jpg)
54
Bio-signal framework to secure Mobile devices
Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, A bio-signal based framework to secure mobiledevices, Journal of Network and Computer Applications, 2017
![Page 52: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/52.jpg)
55
BCI Applications using EEG
![Page 53: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/53.jpg)
56
Framework overview
Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, A bio-signal based framework to secure mobiledevices, Journal of Network and Computer Applications, 2017
![Page 54: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/54.jpg)
57
Attacking Scenario
Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, A bio-signal based framework to
secure mobile devices, Journal of Network and Computer Applications, 2017
![Page 55: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/55.jpg)
58
Customer choice prediction in Online Shopping
Mahendra Yadava, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, Analysis of
EEG signals and its application to neuromarketing, Multimedia Tools and Applications, 2017.
![Page 56: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/56.jpg)
59
Architecture
Mahendra Yadava, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, Analysis of
EEG signals and its application to neuromarketing, Multimedia Tools and Applications, 2017.
![Page 57: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/57.jpg)
60
Dataset Samples
Mahendra Yadava, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, Analysis of
EEG signals and its application to neuromarketing, Multimedia Tools and Applications, 2017.
![Page 58: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/58.jpg)
61
Choice Prediction Results
• Best results have been recorded in Theta band waves.
![Page 59: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/59.jpg)
62
Fusion of EEG Response and Sentiment Analysis of Products Review
![Page 60: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/60.jpg)
63
The main objectives of the paper are as follows:
➢ A multimodal scheme is framed to improve the overall rating
performance of a product by fusing EEG signals with
sentiment score.
➢ The multimodal optimization is performed using ABC
optimization algorithm to maximize the overall system rating.
![Page 61: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/61.jpg)
64
VADER Sentiment Analysis
➢ It is a lexicon of sentiment-related words and rule based
sentiment analysis
➢ It maps words to sentiment by building a lexicon or a
dictionary of sentiment.
➢ It produces three sentiment components.
Positive, Neutral, Negative.
➢ Other lexicon methods are TextBlob and NaiveBayes
Analyzer.
![Page 62: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/62.jpg)
65
Pre-processing steps on the customer reviews
![Page 63: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/63.jpg)
66
Comparative Analysis among VADER, TextBlob and NaiveBayes of customer’s reviews
![Page 64: Dr. Partha Pratim Roy - bdasp.iiita.ac.in · INDIAN INSTITUTE OF TECHNOLOGY ROORKEE Applications of Machine Learning using Sensors Dept. of Computer Science and Engineering Indian](https://reader030.vdocument.in/reader030/viewer/2022040607/5ebd8b4de3fb344d9c13d657/html5/thumbnails/64.jpg)
67
Acknowledgment