virtual interaction using myo and google cardboard (slides)

VIRTUAL INTERACTION USING MYO AND

GOOGLE CARDBOARD Name : Chong Wee Nee

ID : 1122702155

Supervisor : Dr Poo Kuan Hoong

Co-Supervisor : Ian Tan

Moderator : Dr Wong Ya Ping

Project Objectives

• To develop a Google Cardboard application and integrate the Myo Armband into the 3D environment.

• To enable user to control the 3D environment using the Myo Armband.

• To provide an immersive experience of learning Japanese characters

Project Scope

• Use only 5 gestures

• Positional tracking not included

• Capture hand movement and transform the data to display air drawn characters in the virtual world.

• The Japanese character will be limited to a few (Hiragana).

• The application will be able to determine if the Japanese character drawn is correct or wrong.

INTRODUCTION TO MYO ARMBAND AND

GOOGLE CARDBOARD

Myo Armband

• Reads muscle activity in forearm

• Electromyography technique

• EMG sensor & Motion sensor (9-axis IMU)

• Detect 5 gestures

• Connect to devices via Bluetooth

Google Cardboard

BACKGROUND STUDY

Virtual Reality Devices Oculus Rift HTC Vive Sony’s

Project Morpheus

Samsung Gear VR

Host Device Windows PC Windows PC Play Station 4 Galaxy Note 4, S6, S6 Edge

Field Of View 110°

Approx. 100° Approx. 100°

96°

Latency Less than 20ms

22ms Less than 18ms

Less than 20ms

Refresh rate 90Hz 90Hz 120Hz Mere 60Hz

Positional Tracking

Yes Yes (Lighthouse)

Yes No

Gesture Recognition Devices Device Method of Gesture Recognition

Kinect Depth Sensor

Play Station Move Camera, Orb on the controller

Wii Wii Remote, Sensor bar

Leap Motion 2 cameras, 3 infrared LED

Requirements

Hardware

• Android phone (Android 4.3 and up & Bluetooth LE 4.0)

• Google Cardboard

• Myo Armband

Software

• Android Studio 1.0 or higher

• Google Cardboard sdk

• MYO Android sdk

• Unity 5.0

• Blender

Flow Diagram

User

Myo Armband

Phone

Identify the Pose

hand pose

head movement

raw EMG data

gestural data

gestural data

gestural data

manipulated classroom

phone movement

updated orientation of VR world

Plugin

Identify the control assigned

to the gesture

Track Rotation & Track Position

FYP 1 Accomplishments

• 3D classroom environment

• Integrate Myo Armband into 3D environment

• Created plugin to connect Android-Myo, Unity

FYP 2 Accomplishments

• Capture hand movement and convert into drawing (gyroscope)

• Included two Japanese characters for users to trace.

• Added sound effect and background music.

• Identify whether the strokes are drawn correctly (compare points).

Future works

• Improve plugin (slow response)

• Add gestures

Screenshot

Screenshot

Screenshot

Screenshot

DEMONSTRATION

Q&A