eyes-free user interaction
DESCRIPTION
TRANSCRIPT
![Page 1: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/1.jpg)
Eyes-Free User Interaction
T. V. RamanGoogle
http://emacspeak.sf.net/raman
Charles L. ChenGoogle
www.clcworld.net
May 21, 2009
![Page 2: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/2.jpg)
OverviewAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 2 / 39
Android
Dialer
Contacts
TTS
Gestures
MusicBrowser
Overlays
Conclusion
![Page 3: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/3.jpg)
Android Programming
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
![Page 4: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/4.jpg)
Android: Engineer’s Dream!Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 4 / 39
■ Smart phones — computer in your pocket.■ Open platform — liberates innovation.■ New I/O — minimize user interaction.
Bend technology to your will!
![Page 5: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/5.jpg)
Innovative User InterfacesAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 5 / 39
Create innovative end-user solutions that:
■ Fit into the user’s mode of working,■ Avoid one-size fits all solutions,■ Design context-aware solutions,■ Tailor solutions to user’s needs.
![Page 6: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/6.jpg)
Eyes-Free InteractionAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 6 / 39
Not just for blind users!
■ Eyes-busy environments.■ One-handed use.■ Minimize explicit user interaction.
Rethink all assumptions about the user.
![Page 7: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/7.jpg)
Devices That SenseAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 7 / 39
Mobile phones make UI design exciting!
■ Can capture user intent — keyboard,. . ..■ Can see — camera.■ Can hear and speak — speech I/O.■ Can sense and stimulate touch — touch screen and vibration.■ Can sense motion — accelerometer, compass.■ Can sense position — GPS, cell towers.
■....
Our Android has many eyes and ears!
![Page 8: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/8.jpg)
Touch Screen Interaction, Eyes-Free!
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
![Page 9: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/9.jpg)
Using A Touch ScreenAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 9 / 39
These areatomic tasks:
■ Identifying the item to touch.■ Touching item to activate it.■ Eyes-free use requires help with the former.■ Activation should produce appropriate feedback.
![Page 10: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/10.jpg)
On-Screen TouchpadsAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 10 / 39
Eyes-Free use of an on-screen touchpad:
■ Need tosee the buttons to know where to touch.■ Because: buttons useabsolute positioning.■ Conclusion: (wrong) — you need to see to use a touch screen!■ Solution: Relax constraint of absolute positioning.
What if we usedrelative positioning instead?
![Page 11: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/11.jpg)
Phone KeypadAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 11 / 39
Eyes-free phone dialer:
■ Keypad centered where you touch — relative positioning.■ Users already know layout of a phone keypad.■ Canstroke from the center for a desired digit.
Enables eyes-free one-handed dialing!
![Page 12: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/12.jpg)
Dialer FeedbackAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 12 / 39
Enables one-handed, eyes-free use:
■ Buttonsvibrate when you move over them.■ Augmented by auditory feedback.■ Pressing buttons produces spoken and tactile feedback.■ Shake to erase input.
Synchronized auditory and haptic feedback is key.
![Page 13: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/13.jpg)
Text Input Via Touch Screen
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
![Page 14: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/14.jpg)
Contact ManagerAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 14 / 39
No one dials phone numbers any more!
■ Browsing contacts using touch screen.■ Finding contacts.■ Managing contacts.
![Page 15: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/15.jpg)
Entering Text InputAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 15 / 39
Extending stroke dialer for textual input:
■ Arrange letters in4 concentric circles.■ Strokesaccess distinct circles.■ Choose desired letter bytracing chosen circle.
Any letter is at most 3 steps away!
![Page 16: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/16.jpg)
Keypad LayoutAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 16 / 39
Gesture: Pick circle, and trace to desired character.
A I Q
⊙ Y
U M E
Letter Gesture
A Stroke up to the leftB Stroke A, trace rightE Stroke down to the right
![Page 17: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/17.jpg)
Managing ContactsAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 17 / 39
Contacts live in the cloud.
■ Contacts managed over the network.■ Updated automatically on the device.■ No more editing contacts with two fingers!
![Page 18: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/18.jpg)
Text To Speech On Android
Android Dialer ContactsTTS Gestures MusicBrowser Overlays Conclusion
![Page 19: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/19.jpg)
TTS LibraryAndroid Dialer ContactsTTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 19 / 39
■ Create speech-enabled apps.■ Developer compiles against the TTS library stub.■ Text To Speech is an Android service.■ ClassTTS manages communication with the service.■ Enables multiple apps to use TTS without code duplication.■ TTS library can be updated without a need to recompile clients.
![Page 20: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/20.jpg)
TTS Method SummaryAndroid Dialer ContactsTTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 20 / 39
■ speak(String text, int queueMode, String[] params)■ stop()■ isSpeaking()■ Also has methods for synthesizing to a file, specifying the language, etc.
![Page 21: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/21.jpg)
TTS Object PropertiesAndroid Dialer ContactsTTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 21 / 39
■ Can automatically prompt the user to install the TTS Service.■ When absent, fail silently (methods become no-ops)
See: http://eyes-free.googlecode.com/
![Page 22: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/22.jpg)
Gesture Library For Stroke Input
Android Dialer Contacts TTSGesturesMusicBrowser Overlays Conclusion
![Page 23: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/23.jpg)
Gesture OverlayAndroid Dialer Contacts TTSGesturesMusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 23 / 39
■ Overlay watches for touch events■ Returns identified gesture■ Exposes same UI as the stroke dialer■ Implemented as a custom transparentView with callbacks
![Page 24: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/24.jpg)
Gesture Overlay ImplementationAndroid Dialer Contacts TTSGesturesMusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 24 / 39
■ Implement InterfaceGestureListener ,■ Create a new TouchGestureControlOverlay with it,■ Set aFrameLayout as the baseView,■ Add theView with the content as a child to the FrameLayout,■ Add the TouchGestureControlOverlay as a child
Can easily enable/disable theTouchGestureControlOverlay
![Page 25: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/25.jpg)
A Simple Music File Browser
Android Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion
![Page 26: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/26.jpg)
Music File BrowserAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 26 / 39
Barebones version:
■ Uses a simpleListView■ Based on a FileBrowser tutorial on Anddev.org■ Browse directories and mp3 files on SDCard.■ Play music by clicking on the file.■ Directory path is the first entry.■ Click this to cycle through files in current level.■ Tactile feedback when scrolling through list items.
![Page 27: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/27.jpg)
Adding Spoken OutputAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 27 / 39
■ Create objectTTS in methodonCreate.■ Result: Runs methodTTS.onInit as part of setup.■ Can add application-specific code to methodTTS.onInit.■ Result: Application comes up speaking.
![Page 28: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/28.jpg)
Music Player TalksAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 28 / 39
private InitListener i = new InitListener() {public void onInit(int version) {tts.speak("Music File Browser started.", 0, null);
}};
![Page 29: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/29.jpg)
Speaking List ItemsAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 29 / 39
■ Add TTS call to speak list items.■ Can use event handler that produces tactile feedback.
![Page 30: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/30.jpg)
Speaking List ItemsAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 30 / 39
Augment event handler with a TTS call.
public boolean onTrackballEvent(MotionEvent event) {String filename = directoryEntries.get(selectedId);tts.speak(filename.substring(1), 0, null);
}
Self-voicing functionality added with just 13 additional lines of code!
![Page 31: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/31.jpg)
Adding Gesture Input To MusicBrowser
Android Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion
![Page 32: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/32.jpg)
Advantages Of An OverlayAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 32 / 39
■ GestureOverlay is a transparent overlay.■ Can be layered on an existingView.■ Does not interfere with visual appearance.
![Page 33: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/33.jpg)
Gestures For Music BrowserAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 33 / 39
■ Add GestureListener to handle user input.■ Gesture Tap: play/pause.■ Gesture Right: Browse to next track.
Add application-specific code in methodonGestureFinish.
![Page 34: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/34.jpg)
Steps In Adding Gesture InputAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 34 / 39
private GestureListener g = new GestureListener() {public void onGestureChange(Gesture arg0) {}public void onGestureFinish(Gesture arg0) {if (arg0 == Gesture.CENTER) {// play/pause music
} else if (arg0 == Gesture.RIGHT) {playNext();
}}public void onGestureStart(Gesture arg0) {}
};
![Page 35: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/35.jpg)
Adding Overlay To Music PlayerAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 35 / 39
myFrame = new FrameLayout(this);myFrame.addView(myList);myG = new TouchGestureControlOverlay(this,gestureListener);overlayActive = false;setContentView(myFrame);
![Page 36: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/36.jpg)
Toggling Gesture SupportAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 36 / 39
if (overlayActive) {myFrame.removeView(myG);overlayActive = false;tts.speak("Gestures disabled.", 0, null);
} else {myFrame.addView(myG);overlayActive = true;tts.speak("Gestures activated.", 0, null);
}
Gesture support enabled by adding about 40 lines of code.
![Page 37: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/37.jpg)
Conclusion
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
![Page 38: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/38.jpg)
SummaryAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 38 / 39
■ Programmable smart phones are an engineer’s dream!■ I/O peripherals open up new UI avenues.■ Provide desired information with minimal user interaction.
Profound impact on how we work and play!
![Page 39: Eyes-Free User Interaction](https://reader034.vdocument.in/reader034/viewer/2022051610/54825755b4af9faa0d8b475b/html5/thumbnails/39.jpg)
Watch Computing Take Off!Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free Interaction Google-IO May 2009 – 39 / 39