design of hands-free interface for object...

Design of Hands-Free Interface for Object Relocation

Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar

GDMS Sr Engineer Mike DeMichele

User Robotic Aid Proposed design eliminates need for operator to perform

physical motion

DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION

Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 2

AGENDA

• Motivation

• Problem Statement and Stakeholders

• Technology Introduction

• Gap Analysis

• CONOPS

• Simulation

• Trade-off Analysis

• Business Case

2



PHYSICALLY DISABLED STATISTICS

3

• 5,596,000 people in the US, reported

some form of paralysis [1]

• 2,900,000 severely paralyzed persons in

America [1]

• Physical movement required by control

interfaces.

• Limiting factor in the system usability

and user workload.

[1] 2012 study performed by the Christopher and Donna Reeve Foundation



COMPLEXITY OF HUMAN MACHINE INTERACTION

4

Challenges of using human machine

interaction in a practical setting

• Systems requiring a huge amount

of user input and focus

• Time sensitive systems

• Systems requiring user to perform

small, precise movements

• Systems requiring frequent

operator turnover

[3] Robot FRIEND, Institute of Automation (IAT)

of University of Bremen

• Process driven by input from human user

• Relies on interface to continue data exchange



AGENDA

• Motivation



• Gap Analysis

• CONOPS

• Simulation


• Business Case

5



PROBLEM STATEMENT

6

The 2.9 million severely paralyzed individuals in the US require a technology

enable them to relocate objects without requiring motion. This design has the

potential to assist users in other daily activities that ordinarily rely on physical

input, such as navigating a robot to perform fetch and deliver tasks.



STAKEHOLDER ANALYSIS

Stakeholder Significance Tensions

Control Technology

Companies

- Lead on current development work for

alternatives

- Potential to modify control technologies to

best fit need

- Competition between companies

- Tension with government approvers

Severely Paralyzed - Benefit from Hands-Free Control System - Learning time

- May initially distrust system

Insurance Companies - Create new policies to include HFCS - Cost of insurance

- Additional costs to insurance

companies

HHS and FDA - Under Medical Device Act, aid would require

approval

- Tension if design is not accepted

In-home Care

Organizations

- Leading care for severely paralyzed

- Potential to integrate new robotic care device

- Concerns of being replaced/ losing

market dominance



STAKEHOLDER INTERACTIONS

8



AGENDA

• Motivation



• Gap Analysis

• CONOPS

• Simulation


• Business Case

9



ALTERNATIVES

• Eye Tracking Software

• Head Movement (Gyroscope)

• Voice Control

• Muscle Contraction Detection

• Brain-Computer Interface

10

TRADE SPACE

[4]RoboNurse, Computers Making Decisions- Standford

[5] Accompany Care-O-Bot, Fraunhofer Institute

Methods to Perform Hands-Free

Control



SELECTING ALTERNATIVES

• EEG will be evaluated as the BCI Alternative as it is the most

researched and cost effective BCI

• Muscle Contraction Detection and Gyroscope will not be evaluated as

these alternatives require physical input or movement

Alternatives to Evaluate:

- EEG

- Eye Tracking

- Voice Commands



VOICE COMMAND SUMMARY

• Unlimited Number of Commands

• Ready to Use Technology

• Study by School of Information Technology in China set up

a wheelchair to be driven via voice command • 92% average success rate

• 2.2s average delay



ELECTROENCEPHALOGRAPHY (EEG) SUMMARY

• Non-invasive BCI for detecting brain activity

• Uses electrodes to measure ERP pulses along the scalp

• Event related potentials (ERP) - Brain’s electrical response to a sensory, cognitive, or

motor event

• Key to utilizing EEG as a control method is to detect desired ERP from other ERPs (noise)

and map to a command

• Non-invasive BCI Interface for Device Operation

• 96-channel EEG system

• Cursor guidance commands

• User “mastery” within 5 sessions

• Patient initial success rate of 50-70% to 80-100% accuracy

13

https://drive.google.com/file/d/0BzLzei3_VBoAeFVCSFg0U2ljdDQ/view

*J. del R. Millán is with the IDIAP Research Institute, CH-1920 Martigny, Switzerland, and also with the Laboratory of Computational Neuroscience





EYE TRACKING SUMMARY

• Subject to false positives

• Calibration is key to success

• 2014 study published by Max Planck Institute for Informatics

in Germany • 40ms slower than using a mouse



AGENDA

• Motivation



• Gap Analysis

• CONOPS

• Simulation


• Business Case

15



GAP ANALYSIS

• The necessity of physical movement of the control interface is a highly limiting factor

is the system usability and user workload.

• For the 2,900,000 severely paralyzed persons in America, the necessity of physical

motion to operate a device interface renders the system unusable.

Win-Win - Create a hands-free control system interface which can match the

quality and cost of a traditional interface.

16



SCOPE

Before a new control interface can be selected for the robotic aid, we must examine

the maneuvers which the robot will need to perform.

This project will perform a simulation to collect and analyze the data crucial to

creating a hands-free control interface.



AGENDA

• Motivation



• Gap Analysis

• CONOPS

• Simulation


• Business Case

18



CONOPS

A paralyzed person cannot perform basic motor tasks without

assistance. A robot, controlled by the person, can cross the room and

retrieve an item (prepackaged meal, medicine, cell phone).

19

User with HFCS YouBot by Kuka [9] Youbot, Vertically-Integrated Projects, GT



Functional Diagram



USE CASE

21


Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 22 22

MISSION REQUIREMENTS

R.0 The HFCS shall be operable without tactile user input.

R.1 The HFCS shall not harm the user in any way.

R.2 The HFCS shall provide flexibility for use for a variety of functions.

R.3 The HFCS shall operate in real-time.

R.4 *The HFCS shall be capable of differentiating between [X] commands.

R.5 *The HFCS shall be capable of executing [Y] commands.

* where [X] and [Y] to come from simulation



COMMANDS FOR CONOPS

23

Loop 1: Correct direct

1. Stop motion

2. Pivot to correct direction

3. Resume forward motion

Assumption 1:

Hand contacting object will result in grasp.

(No slipping, etc)



VIRTUAL ROBOTIC SIMULATOR

• V-REP as a virtual robot simulator • Physics based collisions

• Large number of ready made models

• Free for educational use

• Matlab

• TRS and RTB - binds VREP and

Matlab

• YouBot will pick up an object and

move it to another location • Limited preprogrammed movement

YouBot, a popular multi-function robot



YouBot VIRTUAL ROBOT[9]

• Platform

• Omnidirectional wheels

• Zero turn radius pivoting

• Surface for placing multiple objects

• Arm

• 5 joints for 5 DoF movement (Arm is not capable

of full pitch motion)

• Each joint has a limited range of movement

• Gripper

• Open and close

• To simplify the YouBot movement, all the joint movement can be automated according to

the x-y-z position of the gripper in order to reduce user complexity



YouBot Dimensions

Platform

• Raise distance from ground .7m

• L x W x D: .58m x .38m x 14m

• Payload 20 kg

• Max Velocity .8m/s

Arm (Scale factor x1.4)

• Total reach .917m

• Payload .7kg

Gripper (Scale factor x1.4)

• Range .007m

Work Envelope .7182 m^2



AGENDA

• Motivation



• Gap Analysis

• CONOPS

• Simulation


• Business Case

27



Simulation Objectives

• For object relocation be domestic robot: • How many & what turn commands are needed

• How many & what travel commands are needed

• How many & what arm commands are needed

• How many & what gripper commands are needed

• What tasks are infeasible for robot?



Simulation Requirements

• The simulation shall model the path of a robot fetching objects in a domestic setting.

• The simulation shall model the arm movement of a robot fetching objects in a domestic

setting.

• The simulation shall model the gripper of a robot fetching objects in a domestic setting.

• The simulation shall collect platform travel statistics.

• The simulation shall collect platform turn statistics.

• The simulation shall collect arm movement statistics.

• The simulation shall collect gripper statistics.

• The simulation shall define range of robot.



SIMULATION OVERVIEW Fetch item - an item the virtual robot is instructed to pick up

Fetch point - the location of the fetch item, random variable

Task - a randomly generated trip from a start point, to the fetch point, picking up the fetch item and carrying it to an assigned finish point

Start location - initial (x,y) coordinates of the center of the robot’s platform, random variable

Finish location - final location of the center of the robot’s platform, random variable

Start Points (Green), End Point (Red) Fetch Points (Blue)



SIMULATION STATISTICS

Statistics to Record per Task

1. Start location

2. Fetch item location

3. End location

4. Total time on task

5. Time spent on fetch trip

6. Time spent on return trip

7. Time spent picking up object (frame

between fetch trip and return trip)

8. Total distance traveled

9. # of platform rotations

10. degrees rotated for each rotation

11. # lateral arm movements

12. Arm displacement per movement

(cm)



2D- Simulation Inputs

Room Map - Red marks obstacles

Start and fetch location

coordinates



2D- Simulation Modules

Name Description Output

GeneratePath.m Select start and fetch coordinates

then uses D* navigation algorithm

from Robotics Toolbox

Array of coordinates stepping through

shortest path to fetch location.

StatisticsScript.m Locates turns along shortest path.

Calculates angle of turn and

distance traveled on straight-aways

Matrix

Column 1,2 - (x,y)

Column 3 - angle of turn @ pivot point

Column 4 - distance traveled in previous

straight away

GraphicalData Displays frequency of maneuvers Bar graphs for frequency of travel distance

and angle



2D- Simulation Output for Single Run



2D- Simulation Output for 50 runs



3D Simulation

Depending on the position of the YouBot base, the arm joint positions are different even for moving

an object on the same height.



Alternative to Manual Control of Arm/Gripper

• Vision sensor to track objects to be picked up to simplify the

pick up action

• Option for user would be displayed an array of detected distinct

objects and be prompted to choose the distinct object to be

picked up (via voice command)

• Used as an aid to guide the arm movement



Vision Sensor for HFCS Display

VREP Vision sensor has an adjustable resolution, in the ranges that an object is detected, it will

return a depth value, color and intensity values for every point. From the angle of the vision

sensor, the relative coordinates can be calculated.



Simulation Results

-6 platform commands - turn left 18 degrees, turn left 2

degrees, turn right 18 degrees, turn right 2 degrees, start, stop

- Vision Sensor and active select algorithm

- Automated collision avoidance

- Return to user function



Simulation Impact

Completed Mission Requirements R.4 *The HFCS shall be capable of differentiating between 9 commands.

R.5 *The HFCS shall be capable of executing 9 commands.

Additional Requirement 1. Implement a vision sensor instead of manual control once the object is in “sight”

of the robot.



AGENDA

• Motivation



• Gap Analysis

• CONOPS

• Simulation


• Business Case

41



Decision Criteria



Weighting the Attributes

Goal

Category

Weight Measures Measure Weight Total Weight

Capabilities 0.5 Number of

Commands 0.7 0.35

Time Sensitivity 0.3 0.15

TRL 0.1 TRL 1 0.1

Usability 0.2 User Population 0.75 0.15

New User Time 0.25 0.05

Performance 0.2 Accuracy 0.4 0.08

Maintainability 0.2 0.04

Reliability 0.4 0.08



Utility Calculation



SENSITIVITY ANALYSIS



Cost vs Utility for 1 Year



AGENDA

• Motivation



• Gap Analysis

• CONOPS

• Simulation


• Business Case

47



● Eliminates need for 24/7 in-home caregiver

● Fill in-home caregiver shortage [4]

● Returns level of independence to physically disabled persons

VALUE

● Unit is defined as a Robotic Aid + Hands-Free Control System

● Sell units to Home Health Care Organizations

○ Open channel for improvement in future releases

○ Allow professionals to monitor ratio of robotic care vs human interaction

● Potential to form partnership

○ Leverage knowledge of in-home care agencies to further design

● Cost per Unit = $4,500

COST

49

● TOTAL 5 years $699k

● Nonrecurring $195K

● Recurring $504k

REVENUE ● Break even at 300 sales

Sell 100 per year Sell 200 for year 1 and 2, sell 100 after that



QUESTIONS?

Hand-held controller

Robotic Aid

Proposed design eliminates need for hand-held controller User



BACKUP SLIDES

51



References

[1] "Prevalence of Paralysis." Christopher & Dana Reeve Foundation. The Reeve Foundation Paralysis Resource Center, n.d. Web. 19 Nov. 2015.

[2] "Human Factors and Medical Devices." U.S. Food and Drug Administration. U.S. Food and Drug Administration, n.d. Web. 19 Nov. 2015.

[3] "Care-Providing Robot FRIEND." Institute of Automation. University of Bremen, n.d. Web. 19 Nov. 2015. [4] "Computers and Robots: Decision-Makers in an Automated World." Computers and Robots DecisionMakers in an Automated World. University of Stanford, n.d. Web. 19 Nov. 2015.

[5] "Care-O-bot 3." Fraunhofer Institute for Manufacturing Engineering and Automation. Fraunhofer Institute, n.d. Web. 19 Nov. 2015.

[6] "Brain Cells Chat, Even Without a Synapse." Science Magazine. AAAS, n.d. Web. 19 Nov. 2015.

[7]"Parts of Central Nervous System." , Control and Coordination, Science Help. Tutorvista, n.d. Web. 19 Nov. 2015.

[8] "A Closer Look at EEG." Epilepsy Society. Epilepsy Society, n.d. Web. 19 Nov. 2015. [9] "YouBot." YouBot Store. Kuka, n.d. Web. 19 Nov. 2015. [10] "Emotiv EPOC / EPOC+." Emotiv Epoc. Emotiv, n.d. Web. 19 Nov. 2015.

[11] "Quantitative EEG and Event-Related Potentials." Neuronetrix. COGNISION , n.d. Web. 19 Nov. 2015.

[12] T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor Analysis of EEG', Brain Topography, vol. 16, no. 1, pp. 19-27, 2003.

52

MARKET VALUE

According to a 2014 New York

Times article [4], ● 1.3 million new paid caregivers will

be needed to meet demand over

the next decade.

● By 2020 the direct care workforce

(5 million people) will become the

largest occupation in the United

States, surpassing the number of

retail salespeople.

● About 75 percent of services

provided by home care agencies

are paid by Medicaid and

Medicare.

53

Alternative Scores

EEG CONTEXT



EEG TERMINOLOGY

Signals - The electrical activity that travels through a user’s

brain

Synapses - Spaces between neurons that conduct the electrical

activity in the brain

Noise - General term for unwanted and, in general, unknown

modifications that a signal may suffer during capture,

storage, transmission, processing, or conversion

Event-Related Potential - electrophysiological response to a

stimulus

56

[6] AAAS, Brain and Behavior

User command - The instruction a user gives to perform an action (user signal to EEG)

Input Command - A collection of steps the end-device will perform in order to execute a

single user command (digital signals to end-device)



BRAIN-COMPUTER INTERFACE (BCI)

Direct communication pathway between the brain and a computing device

57

BCI Type Signal

Quality

Sensor Location Cost

Commitment

Other

1. Invasive Best In Brain Matter High Risk of scar tissue

2. Partially-

Invasive

Medium Under skull High

3. Non-invasive Poor On Scalp Low

EEG is the most studied non-invasive BCI method due to its low cost and ease of use. The data

quality is improving quickly.



Status of EEG as a Control Method Projects that follow a similar approach using EEG technology:

• Non-invasive BCI Interface for Device Operation

• 96-channel EEG system

• Cursor guidance commands

• User “mastery” within 5 sessions

• Patient initial success rate of 50-70% to 80-100% accuracy

• Brain Actuated Control of a Mobile Robot

• Combined with machine-learning techniques

• Six commands combined with “mental states”

• Worse than manual control by a slim factor of 1.5


*J. del R. Millán is with the IDIAP Research Institute, CH-1920 Martigny, Switzerland, and also with the Laboratory of Computational Neuroscience





PHYSIOLOGY OF THE BRAIN

Human Brain

- Comprised of

100,000,000,000 neurons

- Neurons connect via

synapses

- When thought occurs,

neurons generate spikes

of electrical activity

59

[7] Canadian Institutes of Health Research, Institutes of Neurological, Mental Health and Addiction



ELECTROENCEPHALOGRAHY (EEG)

• Offers a non-invasive method to detect electrical activity by using electrodes to

measure the electrical pulses along the scalp to ultimately graph these impulses

• Predominantly used in medical field for sleeping/brain abnormalities and

detection, stress evaluation, and prosthetics

• Event related potentials (ERP)

• Correspond to a brain response related to a sensory, cognitive or motor event

• The key to using EEG as a control method is to detect the desired ERP from

other ERPs (noise) and map it to a command

• EEG is not widely used as a control method due to poor signal quality from

being outside of the skull. Technological advances are improving quality.

60



How the EEG Hands-Free System Works

1) When a user thinks, electrical pulses fire within synapses in lobes of the brain specific to

that “thought” - user command

2) The EEG headset reads these signals as voltage and generates a graph indicating the

intensity and pattern of the electrical signals (The same stimuli or thought should produce

roughly the same EEG patterns when repeated)

3) These graphs will then be analyzed and chosen patterns will be stored in the database with

it’s corresponding thought or stimulus

4) These selected patterns will then be mapped to varying device actions through a system

that this project will create - input command

62 DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION


User concentrates

on a trained user

command

System filters out

noise and converts

signals to digital

format

System sends digital

signals to library to be

matched to a command

System outputs

corresponding

command to

simulator

14

Sensors

EEG raw data stream

EEG receives

signals and captures

them in the form of

graphs Raw signals are

stored in database and

inputted into the

system

[8] A Closer Look at EEG, Epilepsy Society



PATTERN LIBRARY BLOCK DIAGRAM

63



OPERATIONAL BLOCK DIAGRAM

64



EEG SIGNAL PROCESSING

• The EEG signal will be composed of 14 continuous and numerical data streams, one

from each of the EEG.

• General steps to preprocess the signal:

1. High pass filter to remove low frequency noise

2. Use a window function to group a variable length interval of data such as the Hann or

Hamming window function and eliminate edge artifacts

3. Fast Fourier Transform

4. Convert the imaginary part of any complex numbers into amplitude values

• Use a machine learning approach such as neural networks to build a learning model,

then continuously test in real time to measure accuracy

66



OPTIMIZING EEG Event-Related Potentials

Goal: Determine largest limiting factor in EEG Control System reliability

• Use simulation to examining the effects of different variables on

detection rate and reliability of ERPs such as: • Optimal training procedures or increasing training time

• Use of different machine learning approaches to improve detection

• Tentative input commands to be detected: • 6 directional commands for positive and negative displacement along each

of the x, y, z axes

• 1 toggle command for switching platform control vs arm control

• May need to simplify the number of different commands to reduce user

complexity

67



SIMULATION REQUIREMENTS

S.0.1 Virtual Robot shall accept input from HFCS.

S.0.1.1 Virtual Robot model shall contain a robot with 2D movement capability

S.0.1.2 Virtual Robot model shall contain a robot capable of grasping and

relocating

S.0.1.3 Virtual Robot model shall contain an object to be relocated

S.0.2 Simulation shall accept EEG ERPs from an input device

S.0.2.1 Simulation shall respond to a detected ERP in time < 2 seconds

S.0.2.2 Simulation shall map ERPs to input commands and interface with VREP

S.0.2.3 Simulation should detect and respond to 2+ simultaneous ERP

S.0.3 Virtual robot shall employ automated collision avoidance.

68



EPOC Cognitive Suite

1)Create user account

2)Train neurological signal to a basic motion

3)Repeat thought to perform signal

4)Map signal to another vehicle motion

69 Source: Emotiv EPOC



EEG Headset Alternatives

Standard EPOC

• 14 EEG channels and two references for accurate spatial resolution

• High performance wireless device

• iOS and Android compatible

EPOC+

• 9 axis-inertial motion sensors, Bluetooth capabilities, additional applications are

enabled

Raw Data Add-on

• Includes EEG firmware that allows the raw EEG data stream and marker

events in TestBench software

70



Brainwaves

Neural oscillations are rhythmic or repetitive neural activities in the central nervous system. Neural tissue generate oscillations through

neuron interactions.

A large number of neurons activating for a particular neural computation is called a neural ensemble.

Neural oscillations can be categorized into 5 main frequencies

• Beta (14-40Hz) - Waking consciousness

• Alpha (7.5-14Hz) - Conscious relaxation

• Theta (4-7.5Hz) - Sleeping and light meditation

• Delta (0.5-4Hz) – Deep sleep

• Gamma (above 40Hz) – Insight

• Only recently discovered

71



http://neuronetrix.com/technology-i-36.html



Age

• A study conducted by Thomas W. Pierce et. al suggested that

age has an effect on EEG

• Results showed that electrode groupings were higher in older

adults than younger adults

• Older adults had more electrode locations that did not load

than the younger adults

73


Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 74 74

[1]T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor

Analysis of EEG', Brain Topography, vol. 16, no. 1, pp. 19-27, 2003.



Gender

• A study by Corsi-Cabrera et. al suggested differences in the

brain waves between male and female

• Men showed higher beta power, women showed higher alpha

power

• The alpha waves of men decreased during analytic,spatial,

and mixed processing, while women decreased significantly

only in analytic and mixed processing

75



CURRENT TECHNOLOGIES UTILIZED BY PHYSICALLY

DISABLED PERSONS

76

• Prosthesis

• Wheelchair

• Voice Recognition [13] Johns Hopkins- Modular Prosthetic Limb

design of hands-free interface for object...

Documents