1 the subarctic input system and extensions for handling inputs with ambiguity

1

The subArctic Input System

and

Extensions for Handling Inputs with Ambiguity

2

subArctic A Java-based GUI toolkit that I

(along with Ian Smith) built and distributed in 1996-97

Goal: highly extensible allowing support for lots of cool new interaction techniques– Emphasis on making new and strange

widgets / components / interactors easy to create

– “High ceiling”

3

Parties involved with a toolkit

Toolkit designer (me)

Interactor designer

Interface programmer

User

4



Interactor designer


User

Most toolkits target support

here

5



Interactor designer


User

By moving work up (into reusable library)

6



Interactor designer


User

But typically don’t help much here

(assume a fixed library)

7

subArctic


Interactor designer


User

SA tries to move work for many kinds of interactors into toolkit infrastructure

8

subArctic


Interactor designer


User

SA tries to move work for many kinds of interactors into toolkit infrastructure

Input system is a big part of that

9

Schema for pretty much all GUIs

init();

for (;;) {evt = wait_for_next_event();

dispatch(evt);

if ( damage_exists() ) redraw();

}

10

Schema of a GUI

init();


dispatch(evt);


}

Event Record – recording of the relevant facts about

some occurrence of interest (i.e., user has manipulated

an input device)

11

Schema of a GUI

init();


dispatch(evt);


}

Send (“dispatch”) the event to the object(s) that want it and/or know

how to respond to it(e.g., widget/component/interactor)

12

Event dispatch

All the work happens here Typically delegated to interactors

– E.g., buttons know how to respond to press and release like buttons should

– Each object keeps track of its own state

... but which interactor gets it Toolkit “event dispatch” process

13

Event dispatch policies

Two primary ways to decide which interactor gets an event

What are they?

14

Event dispatch policies

Two primary ways to decide which interactor gets an event– Positional dispatch

Based on where mouse is pointingExamples…

– Focus-based dispatchDesignated object always gets inputExamples…

15

Pop quiz

Should input for dragging be dispatched via positional or focus?

16

Pop quiz

Should input for dragging be dispatched via positional or focus?

Answer: No! (both)

17

subArctic input policies

subArctic encapsulates these “ways of dispatching inputs” in “dispatch policy objects”– Manages bookkeeping (e.g., picking)– Extensible set

Turns out there are other useful policies (e.g., for modal dialogs)

18

When interactors get events…

… they typically respond to them with the equivalent of a simple finite state machine

Press

Move

Release

19

subArctic has lib of common FSMs

Move a lot of input handling work typically done by interactor programmer up into the toolkit

One (highly parameterized) FSM for all – Brad’s “interactor” model (awful terminology :-)

Many customized FSM (extensible set)– subArctic input model

20

FSMs moved to toolkit object

“Dispatch agent”

Translates low level input into higher level terms

21

Dispatch agent example: move_drag

Translated to calls in input protocol:– drag_start(); – drag_feedback(); – drag_end();

With useful parameters (e.g. new pos)

Press

Move

Release

22

Dispatch agent example: move_drag

Translated to calls in input protocol:– drag_start(); – drag_feedback(); – drag_end();

With useful parameters (e.g. new pos)

Press

Move

Release

Defined by Java interface

23

Set of dispatch agents is extensible E.g., can subclass for

specialized kinds of drag such as “drag_within_box” or “snap_drag”– Can create custom for one interface– Once created can reuse

24

How it all goes together

Focus Policy

Positional Policy

Etc…Events

Press

Click

Rollover

Etc...

Etc...Text

Move drag

Grow drag

Etc...

25

How does interactor indicate it wants / can handle some type of input? “… implements input_protocol”

– Where “input_protocol” is interface with calls like drag_start(), etc.

For positional that’s it! For focus-based must also ask

for focus

26

Example: Hypertext for all

User (Ken Anderson) wanted to add hyperlinks to all objects– Hold down the control key and click– His external hyperlink database

would take over and map interactor id to hyperlink target

– But… how do you change every interactor to do this?

27

Example: Hypertext for all

In Swing, Motif, etc. this is essentially impossible

In SA, just insert a new subclass of the “click” dispatch agent that checks for the control key down– About 15 lines of code– Works for interactors written later!

28

Questions about the SA input system?

29

Providing Toolkit Level Support for Handling Ambiguity in Recognition-Based Input

See: http://doi.acm.org/10.1145/354401.354407and http://doi.acm.org/10.1145/332040.332459

Jennifer Mankoff, Gregory Abowd

Georgia Institute of Technology

Scott HudsonCarnegie Mellon University

http://doi.acm.org/10.1145/354401.354407

http://doi.acm.org/10.1145/332040.332459

30

Motivation

Recognition-based input offers the promise of naturalistic input modalities, BUT…

31

Motivation

Recognition-based input offers the promise of naturalistic input modalities, BUT…

Recognizers are imperfect– affects users– breaks current system models

New interfaces & mechanisms

32

Example Interaction

SILK

Hand-sketched interactors

33

Example Interaction

SILK

Interface developer can replace interactors with best recognition result

A button

34

Example Interaction

Correction Dialog (mediator)

test

36

Example Interaction

Problems with dialog – Not reusable or customizable– Hard to grow your own

Basically we don’t have toolkit support for recognition based UI

37

Motivation (cont.)

At much the same stage we were at for GUIs in 1983– No common model for input– No re-use

Infrastructure“widget library”

38

An alternative: Burlap

VIDEO

39

Goals of This Work

Robust, reusable infrastructure Reusable library Integrate with convent. toolkit

– Don’t throw out the baby with the bathwater

40

Talk Roadmap

Requirements for handling uncertain input

Extending toolkits to handle it Interaction techniques for

ambiguity Implementation

41

Invoking Application Actions

Action often done by callbacks– Direct procedure call to application

Hierarchical events are alternate approach– Delivered to app as well as toolkit

42

Hierarchical Events

Low-level events contribute to production of higher-level events

[Green TOG ‘86; Myers & Kosbie CHI ‘96]

User Inputcircle

stroke

down drag up• • • • • •

Corresponding Events

43

Implicit Assumption of Certainty

Implicit in all this is the assumption that the events really happened as reported

Problems arise when this isn’t true– E.g., brittle dialogs

44

Needed to Handle Uncertainty:

Allow for (and explicitly model) multiple alternatives– alternative higher level events– in recognition context: interpretations

Detect conflicting interpretations Mediation of conflicts

45

Needed to Handle Uncertainty:

Lexical feedback about uncertain events – split “feedback” from “action”

Library of mediators

46

How do we do this...

47

Extended Event Model

Uncertainty results in multiple interpretations

interpretation graph

Uncertain Input

circlebox

stroke


circle

stroke


Certain Input

48

Toolkit Extensions

Toolkit’s job is still to deliver events to objects– Now delivered to recognizers,

interactors, and application objects

Button

Checkbox Menu

Recog

49

Toolkit Extensions

Toolkit’s job is still to deliver events to objects– Objects initially only produce

(reversible) feedback, no actions

Button

Checkbox Menu

Recog

50

Another Change:

Interface Appearance Uncertain Event Hierarchy

circlebox

stroke


Events dispatched to all who might use it

51

Details: Arranging for Mediation

Identify any conflicts Look for a mediators

– Pluggable list of them in toolkit Mediator chosen by meta-

mediator Mediator can:

“Pass”, “Pause”, “Accept”

52

Doing Mediation

Example:User selects interpretation

circle

box

circle

53

Doing Mediation (cont.)

Mediator prunes interpretation graph to tree

– App informed of accept & reject

circlebox

stroke


circle

stroke


54

Mediation Strategies

Many mediation strategies– e.g., Automatic vs. user involvement

Toolkit is fully “pluggable” (!)– Library of mediators provided, but– Can extend/build new ones as needed

Research goal:Finding new ones

55

Providing a Library of Mediators

56

Providing a Library of mediators

Survey of existing techniques [Abowd & Mankoff GVU Tech Report 99]

Three major classes Explored additional techniques

57


Survey of existing techniques Three major classes

– Repetition

58



– Repetition– Choice:

Ripe for toolkit support Presentation form Instantiation time Contextual information Interaction Feedback

59



– Repetition– Choice– Automatic

ThresholdsConfusion matrixPlug in machine learning?

60


Survey of existing techniques Three major classes Explored additional techniques

61

Example: Target Ambiguity

Problem: There may be multiple targets of a user action

Example: clicking– Kabbash (95)– Worden (97)

62



Example: Clicking Solution: Give the user

a choice of all of the targets

QuickTime™ and a decompressor

are needed to see this picture.

63



Example: Clicking Solution: Give the user

a choice of all of the targets

Other applications:– Any interface

involving mouse press/release

– Requires separation of concerns

– Works with all interactors

64

Implementation

Added support for mediation; ambiguity to subArctic toolkit– Reusable– Fully “pluggable”– Full existing library still works as is (!)

Small library of mediators Also working on non-GUI toolkit (

http://dx.doi.org/10.1145/572003)

http://dx.doi.org/10.1145/572003

66

Conclusions

Reusable infrastructure to support ambiguous input– Reduces difficulty of creating UIs– Easier to explore new design space

Done by modifying a toolkit, not a separate mechanism– Integrated with conventional input – Other support from toolkit still useful

1 the subarctic input system and extensions for handling inputs with ambiguity

Documents

event n

toolkit n toolkit designer

n goal

events n

event dispatch n

subarctic n toolkit

event dispatch policies

pop quiz n