Building Whole Applications Using Only Programming-by-
Demonstration
Richard McDaniel
January 14, 1999
2
Purpose
Allow nonprogrammers to build applications with original, dynamic behavior Use programming-by-demonstration No written language at all - programs
created solely through demonstration Implement in system called Gamut
3
State of the Art
Full-fledged languages and libraries: Visual C++, Java
Simplified languages with syntax-supporting editor: Authorware, Agentsheets
Tools with macro languages: Director / LINGO, Excel / Visual Basic
4
Outline
Introduction Interaction Techniques Video Inferencing User Study Conclusion
5
Domain
“Board Games” Has background “board” “Pieces” above board show state Pieces follow rule-like behavior Autonomous behavior controlled by timers
Examples Tic-Tac-Toe, Pacman, Reader Rabbit
6
Example: Pacman
7
Innovations in Gamut
Interaction Techniques Allows developers to specify important
details Inferencing
Infer conditions from sources unaffected by the behavior
Infer chains of expressions
8
Inferencing - Conditions
Relationship can be formed between arbitrary objects
– Relating a toggle switch to decide which way to move
Run Away
?
9
GO
Expression Chain
Expressions form chain of relationships– In Monopoly, the position to move a piece is, “the square
which is the sum of the dice’s number of squares from the original position”
Red’s Turn
10
Interaction Techniques
Nudges Do Something, Stop That Hint highlighting
Guide Objects Temporal Ghosts Cards and Decks
11
Nudges
12
Negative Examples
Standard technique for demonstrating examples: Macro Recorder Requires special controls to make a
negative example Negative examples in Gamut provided
by Stop That No special set-up required
13
Guides, Ghosts, and Hints
14
Cards and Decks
15
Video
16
Stages of Inferencing
TranslationMatchingActions
CodeGeneration
17
Translation
Read events from undo history list Convert events to actions Eliminate redundant and
inconsequential events
18
Matching Actions
Matches actions in trace to original behavior
– Unordered– Typically not many (three or four)
Considerations– Degree to which actions match– Priority of actions in original behavior– Actions which create new objects
19
Recursive Difference
Propagates changes to descriptions Each description contains heuristics for
comparing values Builds an And/Or tree result
20
Code Generation
Resolves each value difference by creating a new description Search for previously defined description Use heuristics to generate description from
highlighted objects Assemble values into a table and use
Decision Tree learning
21
Decision Trees
Created at places where Gamut chooses between different behaviors
Generate attributes using heuristics and hint highlighting
Build table of examples that relate behaviors to attributes
Is Eaten
RunHome
Is Blue
ChasePacman
RunAway
Y N
Y N
Condition For Pacman Monster
22
User Study
Goal Determine if nonprogrammers can use
Gamut Perform an informal study
Participants Nonprogrammers, CMU community
Location HCI Institute’s usability lab
23
Tasks
Safari
G-bert
Pawn Race
24
Expertise with Techniques
techniques P1 P2 P3 P4
Do Something X X X
Stop That X X
Highlights Ghosts X
Guide Objects X X
Cards and Decks X X
Player MouseIcons
X
25
Problems
Reluctance to create guide objects Needed explicit instructions in tasks Once told, participants would create
appropriate objects Highlighting inappropriate objects
Some objects seem too obvious to highlight, like ghosts
26
Future Work
Apply to other domains Feedback for just inferred relationships Display for language Higher-level inferencing
27
Conclusion
Builds entire applications without a written programming language
Infers a broader range of behavior than prior systems
Is usable by nonprogrammers