an assistive navigation paradigm for semi-autonomous wheelchairs using force feedback and goal...
TRANSCRIPT
An Assistive Navigation Paradigm for Semi-Autonomous
Wheelchairs Using Force Feedback and Goal Prediction
Master’s Thesis DefenseCandidate: John Staton
Advisor: Dr. Manfred HuberCommittee Members: Dr. David
Levine, Dr. Gergely Zaruba
John Staton 2008Computer Science & Engineering
Outline
• Introduction• Related Work• Concept Review• Design Methodology• Implementation• Experiments• Concluding Thoughts
John Staton 2008Computer Science & Engineering
Introduction - Motivation
• “49.7 million: Number of people age 5 and over with a disability, according to Census 2000; this is a ratio of nearly 1-in-5 U.S. residents, or 19 percent.”
• 25 million had difficulty walking a quarter mile or climbing a flight of 10 stairs, or used an ambulatory aid, such as a wheelchair (2.2 million) or a cane, crutches or a walker (6.4 million).
• “The rate of power wheelchair prescriptions increased 33 fold from 1994 to 2001”
John Staton 2008Computer Science & Engineering
Introduction – Early WorkSystem Name Sensors
CPWNS Vision, Dead ReckoningThe Intelligent
Wheelchair Vision, Infrared, Sonar
Intelligent Wheelchair System Vision, Sonar, Gesture Recognition
INRO GPS, Sonar, Drop-Off Detector
MAid Sonar, Infrared, Laser Range Finder, Dead Reckoning
OMNI Sonar, Infrared, Bump, Dead Reckoning
RobChair Sonar, Infrared, Bump
Rolland Vision, Sonar, Dead Reckoning, Infrared, Bump Sensors
SENARIO Dead Reckoning, Sonar
SIRIUS Sonar, Dead Reckoning
Smart Wheelchair Line Trackers, Bump Sensors
Smart Wheelchair Ultrasonic Beacons
TetraNauta Vision, Infrared, Sonar, Bump Sensors
VAHM Sonar, Infrared, Dead Reckoning
Wheelesely Vision, Infrared, Sonar
John Staton 2008Computer Science & Engineering
• Intelligent Wheelchair• Started in the 1990’s• Numerous universities
and labs• However
• Only two companies sell smart wheelchairs for research use
• Only one is commercially available, only in Europe
Introduction – Inspiration
John Staton 2008Computer Science & Engineering
“The majority of research and development activity in the field of control and automation applied to powered mobility for people with disabilities has
concerned sophisticated technology and techniques. … A more effective approach is to make use of the most flexible and adaptable
intelligence on the chair – the user. To accomplish this, researchers must design, build
and test their systems with real users and contexts in mind.” - Paul D. Nisbet
University of Edinburgh
Introduction – Thesis
John Staton 2008Computer Science & Engineering
• Something is disconnected– Smart wheelchair projects treat the wheelchair as
an “autonomous unit”– Adults prefer individual independence
• Solution– Semi-Autonomous, assistive wheelchair– Communicate with user– User still provides drive direction
Introduction – Thesis
John Staton 2008Computer Science & Engineering
• What communication technique to use?– Aural and Visual = distracting, both to user and to nearby
observers– Haptics!
• Subtle• Effective• Intuitive• Non-distracting• Already shown to be useful for various tasks (mobility aids,
steering tasks, bio-manipulation in virtual reality, surgical tasks…)
Related Work
• “Luoson III” – Lio, Hu, Chen, Lin• National Chung Cheng University, Taiwan• Specifics:
– Blind user– Ultrasonic sensors– Motion Prediction– MS FF Pro
John Staton 2008Computer Science & Engineering
Related Work
• Wheelchair University – Protho, LoPresti, Brienza
• University of Pittsburgh• Specifics
– Two design philosophies– Passive Assistance– Active Assistance– VR System
John Staton 2008Computer Science & Engineering
Related Work
• Metz University, France – Fattouh, Sahnoun, Bourhis
• Reminiscent of Luoson III– Distance sensors– Averaged feedback forces– VR System– MS FF 2
John Staton 2008Computer Science & Engineering
How Is This Thesis Different?
• Previous research– Obstacle avoidance
• This research project– Obstacle avoidance
&– Goal guidance
• Seeks to intuit the user’s intended goal• Guide the user to that goal & away from obstacles
– Active assistance
John Staton 2008Computer Science & Engineering
Concept Review – Force Feedback
• “Haptics” – anything related to or based on the sense of touch
• Force Feedback – Haptics applied to an I/O device
• Touch Sensations– Vibration– Robust effects
• Emulate the feeling of weight, friction, liquid, and more.
John Staton 2008Computer Science & Engineering
Concept Review – Force Feedback
John Staton 2008Computer Science & Engineering
Concept Review – Force Feedback
John Staton 2008Computer Science & Engineering
• Simple• Vertical grip that pivots around
a fixed end• Angle of the joystick
Or• Displacement from neutral
position• Intuitive• Effective
• Used in many applications• Flight control• Video games• Electric Powered
Wheelchairs• Therapy
Concept Review – Force Feedback
John Staton 2008Computer Science & Engineering
• “Effect” – The encapsulated force-response data sent to the FF device
• Categorized bythree distinctdimensions:
Static Dynamic
One-shot Open-ended
Interactive Time-based
Concept Review – Harmonic Functions
• Formal definition:• “Harmonic Function”
– Real function– Range in the real numbers– With continuous second partial derivatives– Satisfy Laplace’s Equation
• The sum of the second partial derivatives equal zero• No local maxima or minima• Smooth and differentiable
John Staton 2008Computer Science & Engineering
Concept Review – Harmonic Functions
John Staton 2008Computer Science & Engineering
Concept Review – Harmonic Functions
• Used repeatedly, and with great success• Path planning in a known
environment• Potential value =
probability of collision• 1.0 – obstacle, 0.0 – goal• Gradient = direction away
from obstacles and toward goals
John Staton 2008Computer Science & Engineering
Concept Review – Applying Harmonic Functions
John Staton 2008Computer Science & Engineering
Iterate through the entire grid, where for every grid[i,j]:
Re = 0.25 x (nb1+nb2+nb3+nb4) – grid[i,j];
grid[i,j] = grid[i,j] + Re;
nb1 = grid[i-1,j] x w;
nb2 = grid[i+1,j] x w;
nb3 = grid[i,j-1] x w;
nb4 = grid[i,j+1] x w;
The maximum Re is saved for each iteration through the grid.
As long as Re > 10-14 then the process repeats.
Successive Over-Relaxation (SOR)
Iterative, numerical method
Speed up convergence of the Gauss-Seidel method for solving linear systems of equations
Design Methodology
• Objectives– The ability to infer the user’s intention– The ability to help direct the user towards the intended
goal and away from obstacles• Design Methodology
– Two looping procedures– Outer loop
• Infers goal– Inner loop
• Directs user to goal
John Staton 2008Computer Science & Engineering
Design Methodology – Outer Loop
John Staton 2008Computer Science & Engineering
External User
Preference System
Goal Selection
Harmonic Function
Path Planning
Run-Time
System
Goals
Predicted Goal Grid
Location,Orientation,Past Behavior
Outer Loop
Design Methodology – Goal Selection
John Staton 2008Computer Science & Engineering
• Inputs– Series of goals
• Each goal is initially weighted• based on the knowledge of the external system of user preferences
– Recent User Behavior• Current Location• Current Orientation• Series of past user actions
• Output– Predicted user goal– Predicted likelihood for every goal
• Heuristic– “Confidence”– Modified (increased or decreased) based on the
similarity of the user actions to the actions that would lead to the goal(s).
External User
Preference System
Goal Selection
Goals
Predicted Goal
Location,Orientation,Past Behavior
Outer Loop
Design Methodology – Harmonic Function
John Staton 2008Computer Science & Engineering
• Inputs– Predicted Goal– Environmental Data Grid
• Output– Harmonic function as applied to grid
• Potential value for every location• Goal = 0.0• Obstacle = 1.0• All other locations 0.0 < grid[x, y] < 1.0
• Algorithm– Successive Over-Relaxation (SOR)
Harmonic Function
Path Planning
Run-Time
System
Predicted Goal Grid
Outer Loop
Design Methodology – Inner Loop
John Staton 2008Computer Science & Engineering
Run-Time Loop
ForceEffect
Generation
Force Effect
Playback (Joystick)
Wheelchair Location, Orientation
Motors
Risk, Direction
Force Vector Motor Command
Movement
External User
Preference System
Goal Selection
Harmonic Function
Path Planning
Run-Time
System
Goals
Predicted Goal Grid
Location,Orientation,Past Behavior
Outer Loop
Design Methodology – Force Vector Creation
John Staton 2008Computer Science & Engineering
• Inputs– Wheelchair Location– Wheelchair Orientation
• Output– Force Vector
• Heuristic– Force Vector =
direction, amount of force– Direction = away from obstacles,
toward goal– Amount of force = contingent upon the
“riskiness” of user action
Run-Time Loop
ForceEffect
Generation
Wheelchair Location,
Orientation
Risk, Direction
Force Vector
Movement
Design Methodology – Force Direction
John Staton 2008Computer Science & Engineering
• Force Direction– Direction of the harmonic function gradient (slope)
relative to the wheelchair’s current orientation– Compute the angular difference between the
gradient direction and the orientation of the wheelchair for use as the direction of theforce vector
Wheelchair Orientation
Gradient
Design Methodology – Risk
John Staton 2008Computer Science & Engineering
• Amount of force– “Risk” of current action
• Local Risk vs. Future Risk– Local:
• Wheelchair Velocity• Current Potential Value (from Harmonic Function)• Next Potential Value (from Harmonic Function)• Difference between Current and Next
– Future:• Current Potential Value (from Harmonic Function)• Potential Value some distance ahead, calculated based on current velocity• Difference between Current and Future
• Allows for locally “risky” behavior if future risk is minimized
Design Methodology – Risk
John Staton 2008Computer Science & Engineering
• More Formal– (V + Pc + (Pc-Pn)) = Local Risk
• Velocity = V, Pc = Current Potential, Pn = Next Potential
– (Pc - Pf) = Future Risk• Pc = Current Potential, Pf = Future Potential
– grid[i+x, j+y] = Future potential• X & Y are scaled based on V (and are dependant on orientation)
• How was this actually implemented?– “Levels” of risk
• For every risk factor that was present, a “level” of risk was added• Discrete force effects
Design Methodology – Force Feedback Playback
John Staton 2008Computer Science & Engineering
Run-Time Loop
Force Effect
Playback (Joystick)
Motors
Force Vector Motor Command
Movement
• Convert Force Vector to Force-Feedback effect– Thus communicating to the user:
• Severity of the situation/current action– The amount of force
• What action should be performed next– The direction of the force
• Send joystick position to motors as a motor command– Produces movement– Updates wheelchair’s position,
orientation & velocity– Repeat loop
Implementation
• Dell Dimension 8250– Pentium 4 2.66 Ghz– 512 MB RAM– Windows XP– Microsoft Sidewinder FF 2
• Microsoft Visual Studio ‘05– C# – Modifications to Microsoft Robotics Studio
John Staton 2008Computer Science & Engineering
Implementation – Microsoft Robotics Studio
John Staton 2008Computer Science & Engineering
Joystick Path PlanningConsole
Sensor Data
Implementation – Microsoft Robotics Studio
John Staton 2008Computer Science & Engineering
Experiments
• Two loops– Goal Selection/Harmonic Function Path Planning– Force Feedback/Simulation Environment
• Two major sets of experiments– Goal Selection– User Testing of Simulation Environment
• Quantifiable data (time, number of collisions)• Survey data
John Staton 2008Computer Science & Engineering
Experiments – Goal Selection• Experiments tested:
– Similar or clustered goals vs. semi-similar goals vs. one distinct goal
– User actions towards a goal, neutral actions, actions away from goal
– High, medium and low initial goal weight
– System predicts one goal and produces a prediction of it’s likelihood (100%, 50%, etc)
John Staton 2008Computer Science & Engineering
12 3
45
Experiments – Goal Selection Results/Analysis
John Staton 2008Computer Science & Engineering
• When goal is distinct• 100% accuracy (average predicted likelihood: 100%)
• When goals are semi-similar• 100% accuracy (average predicted likelihood: 80%)
• When goals are similar/clustered• Each of the three clustered goals averaged a predicted
likelihood of ~ 33%• The goal of the three with the highest weight was selected
by the system for each run• Predicts the goal properly when it is reasonable to expect so!
Experiments – User Testing• Each test subject was given time to
familiarize themselves with the simulation, both with force-feedback and without
• Six test runs were given, three with FF, three without– Time to complete course– Number of Collisions
• Post Test Survey– Helpful for avoiding obstacles, helpful
for approaching goal– Too forceful <-> Not forceful
John Staton 2008Computer Science & Engineering
Experiments – User Testing Results/Analysis
John Staton 2008Computer Science & Engineering
Without FF Collisions With FF Collisions
Subject 1 59.420 s 0 52.878 s 0
Subject 2 51.927 s 0 45.690 s 0
Subject 3 61.590 s 0.333 55.229 s 0
Subject 4 51.551 s 0 46.133 s 0
Subject 5 87.080 s 0.667 83.116 s 0.333
Subject 6 413.04 s 1 247.25 s 0.333
(average)
• Six Subjects– Three pairs (one male, one
female)– Ages:
• 20’s• 40-50’s• 70’s
• Subject 6 has hand tremors• Three runs without force-
feedback, three with, alternating• All subjects showed:
– Improved time to complete course with Force-Feedback
– Fewer collisions with Force-Feedback
Experiments – Survey Results
John Staton 2008Computer Science & Engineering
• First Question: “Were the force-feedback suggestions helpful in avoiding obstacles?”– All subjects answered “Yes”– Subject 6 (elderly gentleman with hand tremors) indicated that it didn’t necessarily help “at first”,
but he caught on to the idea of the system, and once that happened, it helped “greatly”.• Second Question: “Were the force-feedback suggestions helpful in approaching the goal?”
– 5 of the 6 subjects answered “Yes”– Subject 5, an elderly woman, answered “Somewhat”– When asked why, she indicated that, to her, what held the system back from a full “Yes” was the
strength of the force-feedback effects.• Final Question: “Between too forceful and not forceful enough, where would you rank the
force-feedback effects?”– Answers varied– Subjects 2, 4 and 6, all males, indicated that the force-feedback suggestions could have been more
forceful.– Subjects 1, 3, and 5, all females, indicated that the suggestions were too forceful, Subject 5 wrote
that the joystick was driving her, and not the other way around
Experiments – Survey Analysis
John Staton 2008Computer Science & Engineering
• What can we gather from this survey?– Force-feedback effect “forcefulness”
• Results indicate possible gender bias in response to effect force• More testing needed to expand on this pattern• Potential solutions:
– User-controlled amount of force– Training period
– Not “getting it” at first• Subject 6 needed a longer period of adjustment before truly understanding the
system• Potential solution:
– First –Time Walkthrough/Explanation/Example Training System
• Positive results!– All subjects found it helpful, and were genuinely excited at the potential of
the system
Concluding Thoughts
• Positive early results!– Faster times, fewer collisions– Positive survey answers and excited test subjects
• Future work– Training system, help user “get” the concept, and
determine user strength to adjust effect force– Real world issues: How to get environment data
and user position (GPS?), other issues that come with applying to a real chair
John Staton 2008Computer Science & Engineering