people, pens, and computers françois guimbretière hcil, university of maryland [email protected]...
Post on 21-Dec-2015
213 views
TRANSCRIPT
![Page 1: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/1.jpg)
People, Pens, and Computers
François GuimbretièreHCIL, University of Maryland
Ken HinckleyMSR
![Page 2: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/2.jpg)
New devices, old tasks
Middle picture from Sellen et al.
![Page 3: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/3.jpg)
People, Pens, and Tablet PC
The New Yorker
Illustration from Ken Hinckley presentation at Stanford
![Page 4: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/4.jpg)
Typical setting for today’s interface
• Fixed stable environment, with a keyboard,
• Indirect interaction,
• High precision pointing system
![Page 5: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/5.jpg)
Typical Tablet PC use
• Portable, unstable environment, without a keyboard
• Direct interaction,
• Low precision aiming
![Page 6: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/6.jpg)
Pen based interface for Tablet PC
Document areaHow to create a more fluid interface?
Interface frameworkHow to make it more pen-friendly?
![Page 7: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/7.jpg)
Pen based interface for Tablet PC
Interface frameworkHow to make it more pen-friendly?
![Page 8: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/8.jpg)
• Empirical foundations• Use of strokes to cross target is more pen friendly
• Crossing is as efficient as point-and-click [Accot & Zhai, 2002]
• The basic interactor
• How expressive is it?
CrossY: crossing-based GUI[Apitz & Guimbretière 04]
![Page 9: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/9.jpg)
CrossY
video
![Page 10: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/10.jpg)
Previous Work
• Theoretical basis• Steering Law, Trajectory-Based Tasks [Accot & Zhai 97-02]
• Limited scope examples• Toggle Map [Baudish 98]
• Lotus Notes: multiple e-mail selection
• Conceptual design• Visual Instruments: [Winograd & Guimbretière 98]
• Overloading• Gedrics: [Geißler 95]
![Page 11: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/11.jpg)
Command composition
• From stroke-by-stroke interaction• Borders are used to validate/cancel
![Page 12: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/12.jpg)
Command composition
• From stroke-by-stroke interaction• Borders are used to validate/cancel
• To multi-command stroke
![Page 13: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/13.jpg)
Scrolling
Line by line area
Page by page area
Absolute area
![Page 14: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/14.jpg)
CrossY scrollbar
• Overloading simplify interactions• Shorter distances to issue commands
• Not as much precision necessary
![Page 15: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/15.jpg)
CrossY scrollbar
• Overloading simplify interactions• Shorter distances to issue commands
• Not as much precision necessary
• Extending stroke for repeat• No need to wait for a timeout
![Page 16: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/16.jpg)
Cursor control
• Cross to jump to an absolute position
![Page 17: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/17.jpg)
Cursor control
• Cross to jump to an absolute position
![Page 18: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/18.jpg)
Cursor control
• Cross to jump to an absolute position
• Near drag for coarse adjustment
![Page 19: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/19.jpg)
Cursor control
• Cross to jump to an absolute position
• Near drag for coarse adjustment
![Page 20: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/20.jpg)
Cursor control
• Cross to jump to an absolute position
• Near drag for coarse adjustment
• Far drag for fine adjustment• Similar to FineSlider [Masui 95]
• But one single stroke
![Page 21: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/21.jpg)
Use of directionality
• Continuous find and replace
![Page 22: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/22.jpg)
Use of directionality
• Continuous find and replace• Reverse direction for undo
![Page 23: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/23.jpg)
What we learned
• Very well received by users• HCIL open house
• UIST
• Space requirements• Similar to point-and-click
• Trade-off with command combination due to sloppiness
• Overloading vs. easy discovery• Consistency helps with adoption
• Known in Windowing systems
![Page 24: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/24.jpg)
Pen based interface for Tablet PC
Document areaHow to create a more fluid interface?
![Page 25: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/25.jpg)
Scriboli and Stitching
Ken’s presentation
![Page 26: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/26.jpg)
My desk
![Page 27: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/27.jpg)
Affordances of paper documents[Sellen 01]
• Easy to navigate• Two-handed interactions and tactile feedback
• Reading across more than one document at once
• Easy to annotate• Directly on the document or on a nearby pad
• Well accepted during meetings• Socially accepted conventions
• Very difficult to modify• Printed documents are created and edited as digital documents
• Expensive to distribute and archive
![Page 28: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/28.jpg)
Bridging the gap: previous work
• Digital emulation• FreeStyle system [Wang 89]
• MATE [Hardock 93]
• XLibris [Schilit 98], [Golovchinsky 02]
• Tight coupling• DigitalDesk [Wellner 93], Ariel [Mackay 95]
• A-Book [Mackay 02]
• PaperLink [Arai 97]
• Intelligent Paper [Dymetman 98]
• Paper as input device• Xax [Johnson 93]
• Anoto
• Paper PDA [Heiner 99], [Avrahami 01]
![Page 29: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/29.jpg)
Cohabitation[Guimbretière 03]
![Page 30: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/30.jpg)
Stroke capture
From Anoto documentation
• Requirement• Stroke coordinates on the page
• Page ID
• Large address space
• Possible technologies• Anoto
• DataGlyphs [Hecht 94]
• MEMO pen [Nabeshima 95]
![Page 31: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/31.jpg)
System architecture
![Page 32: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/32.jpg)
My Desk with PADD
![Page 33: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/33.jpg)
PapierCraft: command system for PADD[Liao, Guimbretiere and Hinckley 05]
PADD document
Anoto pen
PADD notepad
![Page 34: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/34.jpg)
PapierCraft: using PADD as proxies
Copy Paste
Synchronization On your Tablet PC
![Page 35: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/35.jpg)
PapierCraft
• Gesture/Ink• Use the command button
• Scopes
• Command selection• Marking menu
• Writing down an unambiguous prefix of a command
• Batch mode processing
![Page 36: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/36.jpg)
Early feedback and future work
• Small scale evaluation• Commands with written words were very popular
• Higher recognition rate needed for strokes only feedback
• Contextual information might provide enough feedback during pastes
• Future work• More reliable recognition engine
• Pen-based feedback• Haptic feedback, LED
• Real time processing : streaming stroke
![Page 37: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/37.jpg)
Conclusions
• Four pen-based interfaces• CrossY explores crossing-based interfaces
• Scriboli explores high performance direct manipulation interfaces
• Stitching explores multi-devices interactions
• PapierCraft uses PADD as document proxies
• One goal• Bring the ease of use of pen and paper to digital interfaces
• Future• Seamless integration of both media
![Page 38: People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3fc26/html5/thumbnails/38.jpg)
Acknowledgments
• Collaborators• Jim Hollan
• Students• Georg Apitz, Nicholas Chen (CrossY/ScribolY)
• Kevin Conroy, Dave Levin, Chunyuan Liao (PADD/ProofRite)
• Sponsors• Microsoft
• HP
• Anoto, Logitech, Maxell
• Colleagues and friends• Corinna Löckenhoff
• Samrat (Bobby) Bhattacharjee
• Ben Bederson