satin, a toolkit for sketch-based applications at uist 2000
DESCRIPTION
Some research on a toolkit I developed to facilitate the construction of sketch-based interaction. Presented at UIST 2000. Software support for making effective pen-based applications is currently rudimentary. To facilitate the creation of such applications, we have developed SATIN, a Java-based toolkit designed to support the creation of applications that leverage the informal nature of pens. This support includes a scenegraph for manipulating and rendering objects; support for zooming and rotating objects, switching between multiple views of an object, integration of pen input with interpreters, libraries for manipulating ink strokes, widgets optimized for pens, and compatibility with Java’s Swing toolkit. SATIN includes a generalized architecture for handling pen input, consisting of recognizers, interpreters, and multi-interpreters. In this paper, we describe the functionality and architecture of SATIN, using two applications built with SATIN as examples.TRANSCRIPT
Nov 06 2000 1
Jason Hong and James LandayUniversity of California Berkeley
Group for User Interface Research
Nov 06 2000 2
Motivation
• Sketching is useful in many settings
Nov 06 2000 3
• Creative / Design• SILK• Music Notepad• Knight• Elec. Cocktail
Napkin• Teddy• PatchWork
Survey of Informal Ink Apps
• Capture• ZenPad• NotePals• Dynomite• FiloChat
• Others• Pegasus• XLibris• PerSketch• Kramer's
Patches
• Whiteboard• Tivoli• Flatland
Nov 06 2000 4
Example - Elec. Cocktail Napkin
Gross and Do (UIST96)
Nov 06 2000 5
Example - Flatland
Igarashi, Edwards, LaMarca, and Mynatt (AVI2000)
Nov 06 2000 6
Common Features
• Pen input as ink or gesture• Stroke is pen input from dragging• Ink is a stroke that stays on-screen• Gesture is a stroke that activates a command
• Recognize ink as objects• Transformation / clean-up of ink
• Immediate / deferred processing of ink
erase me
Nov 06 2000 7
Common Features
• Contain objects in addition to ink• Selecting and moving objects• Grouping of objects• Layering of objects
Nov 06 2000 8
Related Work
• Commercial Software Support– Newton OS– PenPoint– Microsoft Windows for Pen Computing– Windows CE– Palm OS
• Problems– Form-based and handwriting interfaces– Not easily extensible for building
informal ink apps
Nov 06 2000 9
Related Work
• Research Software Support– ArtKit (Henry et. al. 1990)– Garnet Pen Int. (Landay et. al. 1993)– Patches (Kramer, 1994)– OOPS (Mankoff et. al. 2000)
• Problems– Need more reusable libraries for
handling and processing strokes– Need more extensibility so lots of kinds
of informal ink apps can be built
Nov 06 2000 10
Problems
• Software infrastructure support for informal ink apps rudimentary
• Pen-based interaction techniques difficult to implement
• Similar to GUI development in 1980s– Lots of applications + interaction
techniques– Need cohesive frameworks + toolkits
Nov 06 2000 11
Goals of SATIN
• Toolkit support for informal ink apps– Provide reusable mechanisms for handling
+ processing strokes– Separate mechanism from policy– Be extensible for new kinds of apps
• Reusable widgets for pens• Distribute this toolkit for use
Nov 06 2000 12
Overview
Motivation Survey of Informal Ink Applications Two Applications Built with SATIN SATIN Architecture Overview Conclusions
Nov 06 2000 13
DENIM
Nov 06 2000 14
SketchySPICE
Nov 06 2000 15
SketchySPICE
Nov 06 2000 16
SketchySPICE
Nov 06 2000 17
Some Metrics
SATIN DENIM Sketchy SPICE
#source files 180 76 7
#methods 2192 642 63
#classes 220 131 32
#lines of code
20000 9000 1000
time 1 person18 months
3 people 3 months
1 person 3 days
Nov 06 2000 18
Overview
Motivation Survey of Informal Ink Applications Two Applications Built with SATIN SATIN Architecture Overview Conclusions
Nov 06 2000 19
Architecture Overview
Java Core Classes
Java Virtual Machine
Java2D
SATIN
Swing
Application
SATIN relies on Java2D and Swing
Applications use SATIN, can use Java2D and Swing
Nov 06 2000 20
Architecture Overview
SATIN can be broken into twelve concepts
Recognizers
Interpreters Scenegraph
StrokeLibraries
Widgets
Views
Rendering
Transitions
Clipboard
Notification
Command
Events
Nov 06 2000 21
Recognizers
• Problem - many recognition techniques• Recognizers let new recognition
technologies be plugged into SATIN• Given a stroke, return n-best list• No actions taken
• SATIN comes with Rubine's Recognizer (implemented by Chris Long)
Recognizer0.860.650.63
CutCopyUndo
Nov 06 2000 22
Interpreters
• Problem – many ways of handling strokes
• Interpreters let new ways of handling strokes be plugged into SATIN• e.g. straighten a line
• e.g. issue a command
Ink Interpreter
Gesture Interpreter erase me
Nov 06 2000 23
Interpreters and Recognizers
• Decoupled recognition (Recognizers) from actions (Interpreters)
• Interpreters can use recognizers
• Both are modules that can be plugged into apps built with SATIN
Interpreter Recognizer
Cut
0.860.650.63
CutCopyUndo
Nov 06 2000 24
Multi-Interpreters
• Problem – Hard to combine and extend existing interpreters
• Multi-Interpreters are a collection of interpreters + dispatch policy– e.g. dispatch to chain of interpreters
– e.g. disable some interpreters on context
DefaultMulti Interpreter
Intrp A Intrp Z…
Semantic ZoomMulti Interpreter
Intrp B
Intrp A Intrp Z…Intrp B
Nov 06 2000 25
Phrase(Patch)
Scenegraph
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
…
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Phrase(Patch)
Timed Stroke
Nov 06 2000 26
Stroke Dispatching
• Needed an approach that enabled:– Individual objects handle strokes on own– Priority to gestures over ink
• Default policy is top-down dispatch1. Process stroke with its gesture
interpreter2. Re-dispatch stroke to one of its children3. Process stroke with its ink interpreter4. Handle stroke in object itself
Nov 06 2000 27
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
Nov 06 2000 28
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
1. Process stroke with gesture interpreter
Sheet
DefaultMulti Interpreter
Hold Select Intrp
Standard Gesture Intrp
…
Nov 06 2000 29
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
Nov 06 2000 30
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
Sheet
DefaultMulti Interpreter
Hold Select Intrp
Standard Gesture Intrp
…
1. Process stroke with gesture interpreter
Nov 06 2000 31
Stroke Dispatching2. Re-dispatch stroke to children
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
Nov 06 2000 32
Stroke Dispatching1. Process stroke with gesture interpreter
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
Nov 06 2000 33
Stroke Dispatching2. Re-dispatch stroke to children
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
Nov 06 2000 34
Stroke Dispatching3. Process stroke with ink interpreter
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
Phrase Interpreter
Nov 06 2000 35
Stroke Dispatching4. Handle stroke in object
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
Nov 06 2000 36
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Timed Stroke
Nov 06 2000 37
Example - Cut Shallow
Nov 06 2000 38
Example - Cut Deep
Nov 06 2000 39
Reusable Stroke Libraries
Straighten
Merge
Nov 06 2000 40
Reusable Stroke Libraries
Split
Simplify
Nov 06 2000 41
Widgets
• Pie Menus• Pen PLAF
• Swing "Pluggable Look And Feel"• Larger and clickable sliders
• Single-click file opener
Nov 06 2000 42
Conclusions
• SATIN - Java toolkit for informal ink apps– Extensible thru Recognizers, Interpreters,
and Multi-Interpreters– Separates mechanism from policy for
strokes– Offers reusable libraries for manipulating
ink
• Reusable widgets for pens
Nov 06 2000 43
Download SATIN
http://guir.berkeley.edu/projects/satin
http://guir.berkeley.edu
Download SATIN at
Nov 06 2000 44
Extra Slides
Nov 06 2000 45
DENIM
Nov 06 2000 46
DENIM
Nov 06 2000 47
Survey of Informal Ink Apps
• Elec. Cocktail Napkin
• Flatland• Pegasus• FiloChat• Zenpad• PerSketch• Teddy• PatchWork
• XLibris• NotePals• Dynomite• SILK• Tivoli• Kramer's
Patches• Music Notepad• Knight
Nov 06 2000 48
Architecture Overview
• Recognizers• Interpreters• Scenegraph• Stroke Libraries• Widgets• Views
• Rendering• Transitions• Clipboard• Notifications• Commands• Events
SATIN can be broken into twelve concepts