mcsig chi 2008
DESCRIPTION
A presentation of the McSig system for allowing a teacher to work with a visually impaired student when learning to write. Presented at CHI 2008.TRANSCRIPT
McSig - A Multimodal Collaborative Handwriting
Trainer for Visually-Impaired People
Beryl Plimmer, Rachel BlagojevicUniversity of AucklandAndrew Crossan, Stephen BrewsterUniversity of Glasgow
www.cs.auckland.ac.nz/~beryl www.dcs.gla.ac.uk/~stephen
Outline
Motivation Why is writing important for a visually impaired person Why is a difficult thing to train
How can we train users to draw a letter shape One solution – haptic trajectory playback How good are people at recalling a trajectory they’ve been
dragged through
McSig Iterative Design Evaluation
Conclusions
2
Why do Visually Impaired People Need to Write ?
Signature Difficult without visual
feedback Important for Job Applications,
Legal documents etc. One participant described her
signature as ‘resembling the meanderings of an inebriated fly’
More general problem with spatial data Presenting Creating
3
Accessible Computing
Magnifiers Need some sight
Screen Readers Synthesized speech
reads text from the screen
Not good for non-text information
Dynamic Braille Non-text or spatial data
is very difficult to present non-visually Maps, charts, graphs,
diagrams, web pages
4
Force Feedback Haptic – Relating to the sense of touch
Force feedback Tactile feedback
Force feedback Forces can be exerted through motors Can be arranged to form 3D objects Can be used to directly influence the movements of the
user We use a PHANTOM OMNI from SensAble
Provides 6 dof in, 3 dof out Produces rich feedback suitable for many general
purpose applications Max 3 Newton force so the user can overpower the
device5
Force Feedback as an Accessibility Aid Non-visual communication channel Can directly affect the user’s exploration
Potential for collaborative applications Research question
Can trajectory playback techniques be used to communicate shape and trajectory information to visually impaired people ?
6
Initial Studies Learning abstract trajectories through haptic
guidance drag a user through a set of pre-defined trajectories Initially significantly harder for visually impaired users &
congenitally blind participants found the task harder still But combining haptic guidance with a sonification of the
trajectory (pitch shift & panning a sinusoidal tone) significantly improved learning
7
Audio Pan
Pitch ShiftDrawing
Area
McSig: Multimodal, collaborative handwriting and signature training tool Teacher and student work
synchronously on shared representation sat next to each other Haptic guidance and audio feedback Teacher guides the student to learn
letter shapes using words and actions Student holds PHANTOM Omni
pen, teacher uses a stylus & touchscreen Teacher can move student’s pen
around the character shape Audio feedback as shape is drawn
Left/right movements: pan, up/down movements: pitch
8
McSig – system design Simulate standard school learning scenario Teacher can choose between three modes
Collaborative Playback mode: Student holds PHANTOM Omni pen student dragged through shape as it is drawn by teacher
Free Stencil mode: Teacher draws letter which is used as a virtual stencil which
thestudent can then explore Can reduce constraining forces as student gets more
experienced
Free drawing mode Student moves Omni pen freely within area. Movements echoed on teacher’s screen
9
First evaluation
1 blind adult Feedback key to learning
User tried to feel what they had drawn on the paper
Added Dutch drawing board – shape is raised on the paper
Can be felt with other hand
Stencil didn’t work very well
10
Usability testing 4 visually impaired adults (3 blind, 1 partially sighted)
Familiarization Letter set ‘o,c,a,d,e’ Started with playback mode and then moved on to stencil Finally drew the letter unsupported
Stencil mode hard to use Strengthened forces for shapes to give clearer path Still didn’t work very well
Audio feedback useful to some, teacher descriptions most useful Speech feedback error prone and reduced confidence of users Omni pen difficult to hold, plus pressing button whilst drawing
tricky Users not used to holding pens Gave some pen training before main study
11
Evaluation Could McSig improve handwriting performance?
Task designed with teachers Some children almost no handwriting skills, some have
good skills 5 characters chosen after discussion with teachers
o, c, a, d, e Participants
8 children 11-17 years old, read Braille, no other major disabilities
3 partially sighted, 5 blind 4 stage study
Familiarization with McSig, then for each letter: Pre-test McSig training Post-test
12
Familiarization Participants could feel setup,
PHANTOM, mat, PC Spatial orientation Drew circle, horizontal and vertical
lines Practised with the pen
Visually impaired people don’t commonly use pens
13
14
McSig: Pre-Test Training and Post-Test Pre-test: participants asked to draw each letter as
best they could Some unable to draw one or more of them
Training: teacher showed participant how to draw letter in Playback mode Experimenter wrote shape on screen, child felt it with
PHANTOM and scored line on tactile sheet Synchronous audio/haptic/tactile feedback Number of repeats based on child’s confidence
Post-test: participants draw character in free draw mode If participant could not draw it we trained and tested
again Time-out after 20 mins
Stopped earlier if all letters done
15
Results – partially-sighted children
Participants All could read enlarged print All had deteriorating sight but had learned to write
when sight was better Did not write now as sight too bad
Familiarized very quickly, could all do circle, horizontal and vertical lines, no problem
One participant did all of our letters in the pre-test
Both of the others started ‘d’ in wrong place
Pre Post16
Results – partially-sighted children
All letter correct except One did a normal ‘e’ in mirror image
Participants had eyes close to drawing surface but did not feel drawing surface with non-dominant hand Wanted to use their sight
All trained quickly and did all letters correctly in post-test Completed within 20 mins
Politely interested but not captivated
17
Results – blind children
Participants 5 totally blind One lost her sight at 3 years, others blind from
birth Familiarization took much longer
Pressure on pen – too much/too little Interacted with drawing space very differently
Non-dominant hand for orientation in space All but one could draw circle and lines Before and after examples
18
Before and after
19
20
Participant who had sight until age 3
21
Discussion - Technology
Training Student independent
Input - teacher Stylus pen Verbalization
Output Visualization on teacher’s
display Phantom Force feedback Sound pan and pitch Tactile trace Recognition to voice
output
Input Phantom pen position
Output Visualization on teacher’s
display
Sound pan and pitch Tactile trace Recognition to voice output
22
Discussion
Results suggest that McSig could help children to learn Especially blind children
Why didn’t stencil mode work? No tactile representation of the letter for non-
dominant hand Therefore an inconsistent interface
23
Future Work Cursive handwriting and signatures
Support move from single letters to cursive A signature can be created and then practised to
keep it consistent over time A self-teaching tool Wider context
Could be used in any application where the teacher wants to guide student Geometry, 2D and 3D shapes Charts and graphs
24
Conclusions Hard for visually-impaired people to learn to
handwrite Signatures difficult to learn and keep consistent Required for important aspects of life
McSig: a collaborative tool that allows a teacher to guide a student to handwrite letter shapes Dynamic haptic and audio feedback
Can improve handwriting in 20 minute session All blind students learned at least 2 new letters Enjoyed the experience
Now working on longer-term study to see how learning develops over time
25
McSig - A Multimodal Collaborative Handwriting
Trainer for Visually-Impaired People
Beryl Plimmer, Rachel Blagojevic
University of Auckland
Andrew Crossan, Stephen Brewster
University of Glasgow
www.cs.auckland.ac.nz/~beryl
www.dcs.gla.ac.uk/~stephen
Thanks to our participants, John Williamson and Malcolm Hall at Glasgow University and our sponsors University of Auckland and the EU FP6 MICOLE Project