Programming By Voice
Spoken Language Support for Software DevelopmentAndrew BegelAdvisor: Susan L. GrahamComputer Science Division, EECSUniversity of California, Berkeley
#1Broader research goal in building tools for programmers to improve productivity and access. Study programmers first and build tools second. and then build analyses to support tools
People spend a lot of time developing software. Theres a continuing demand for more people.Make people more productive. b. Enlarge the pool of developers.1b. Better education and helping people with disabilities1a. To serve community requires understanding of PL, SE and HCI.
MotivationProgrammers conventionally use keyboardLong hours at keyboard leads to higher risk of RSICan a programmer code using speech?Can a computer understand what the developer says?
while (counter < limit) { }#2Keyboard is unambiguous. Computers lacked processing power to do the workpeople are the precious resource. we need to make the computers resolve the ambiguitiesfor us. not we for them. addresses ambiguity and RSI. is a playpen to explore ambiguity. Mankoff Ambiguity mediators, bring amibugity throughout system.Programming by VoiceMy GoalFind out how developers use code verbally. Use this to develop a naturally verbalizable input form.Build development environment that supports verbal authoring, navigation, modification.Extend conventional compiler analyses to support ambiguities generated by speech.Learn how developers can use voice-based programming, and iterate design.
while counter is lessthan limit do ...#3rewritre to work with previous slide.push on technology for ambiguity resolution and see what its capable of.say human-centered again. methodology slide
ChallengesSpeech is inherently ambiguous.Programming tools were not designed for ambiguity.Speech tools are poorly suited for programming tasks.Programmers are not used to verbal software development.#4Not the best way to do things, but too hard to change, so instead well modify the input to the programming language to make it easier for people.i made progress on each of these. none of these is done yet.
Talk OutlineIntroduction and MotivationProgramming by VoiceProgram Analyses for Ambiguous InputsSPEech EDitor Programming EnvironmentSPEED User StudyConclusion#5target 5 minutesHow do Programmers Speak Code?10 programmers read Java code out loud (Begel 05)Graduate students in Computer ScienceFive knew Java, five did notFive were native English speakers, five were notFive were educated in U.S.A., five were notRead pre-written code into tape recorderAs if speaking to a sophomore-level CS undergrad who knows Java, but does not know the program
Most programmers spoke the same way#6
for int i equals zero i less than ten i plus plusfor (int i = 0; i < 10; i++ ) { }How do Programmers Speak Code?#7Didnt have to verbalize parens, semicolons, etc.
Spoken Words Can Be Hard To Write Down
2
How Do Programmers Speak Code?#8Think about how to say this
Spoken Words Can Be Hard To Write Down
2 2, two, to, too
How Do Programmers Speak Code?#9Think about how to say this
Spoken Words Can Be Hard To Write Down
2 2, two, to, too
How Do Programmers Speak Code?#10Think about how to say this
Spoken Words Can Be Hard To Write Down
2 2, two, to, too
print print, Print
How Do Programmers Speak Code?#11Think about how to say this
Spoken Words Can Be Hard To Write Down
2 2, two, to, too
print print, Print
drop stack process
How Do Programmers Speak Code?#12Think about how to say this
Spoken Words Can Be Hard To Write Down
2 2, two, to, too
print print, Print
drop stack process drop stack processdrop stackprocessdropstack processdropstackprocessHow Do Programmers Speak Code?#13Think about how to say this
Many Ways to Say the Same Thing
bar[i]
How Do Programmers Speak Code?#14Think about how to say this
Many Ways to Say the Same Thing
bar[i] bar sub i, bar of i, i from bar
How Do Programmers Speak Code?#15Think about how to say this
Many Ways to Say the Same Thing
bar[i] bar sub i, bar of i, i from bar
.
How Do Programmers Speak Code?#16Think about how to say this
Many Ways to Say the Same Thing
bar[i] bar sub i, bar of i, i from bar
. period, dot
How Do Programmers Speak Code?#17Think about how to say this
Many Ways to Say the Same Thing
bar[i] bar sub i, bar of i, i from bar
. period, dot
}
How Do Programmers Speak Code?#18Think about how to say this
Many Ways to Say the Same Thing
bar[i] bar sub i, bar of i, i from bar
. period, dot
} right brace, close the if, end method
How Do Programmers Speak Code?#19Think about how to say this
Many Ways to Say the Same Thing
bar[i] bar sub i, bar of i, i from bar
. period, dot
} right brace, close the if, end method
printlnHow Do Programmers Speak Code?#20Think about how to say this
Many Ways to Say the Same Thing
bar[i] bar sub i, bar of i, i from bar
. period, dot
} right brace, close the if, end method
println print line, print lin, print l nHow Do Programmers Speak Code?#21Think about how to say this
One Utterance May Mean Many Things
object stack
How Do Programmers Speak Code?#22Think about how to say this
One Utterance May Mean Many Things
object stack Object stack;object.stackobject(stack)object().stack()
How Do Programmers Speak Code?#23Think about how to say this
One Utterance May Mean Many Things
object stack Object stack;object.stackobject(stack)object().stack()
array sub i plus plus
How Do Programmers Speak Code?#24Think about how to say this
One Utterance May Mean Many Things
object stack Object stack;object.stackobject(stack)object().stack()
array sub i plus plus array[i]++array[i++]
How Do Programmers Speak Code?#25Think about how to say this
People Have Trouble Saying Some Things
System.out.println
How Do Programmers Speak Code?#26Think about how to say this
People Have Trouble Saying Some Things
System.out.println system out print linesystem dot out print linesystem dot out dot print line
How Do Programmers Speak Code?#27Think about how to say this
People Have Trouble Saying Some Things
System.out.println system out print linesystem dot out print linesystem dot out dot print line
(int)foo
How Do Programmers Speak Code?#28Think about how to say this
People Have Trouble Saying Some Things
System.out.println system out print linesystem dot out print linesystem dot out dot print line
(int)foo cast foo to integer int foocast something to integer. that something is foo.
How Do Programmers Speak Code?#29Think about how to say this
Sometimes They Describe the Code
And then theres a class.
How Do Programmers Speak Code?#30Think about how to say this
Sometimes They Describe the Code
And then theres a class.
Set all the fields of that object to null.
How Do Programmers Speak Code?#31Think about how to say this
Sometimes They Describe the Code
And then theres a class.
Set all the fields of that object to null.
All of these are just assignment operations.How Do Programmers Speak Code?#32Think about how to say this
Design TradeoffsCommandLanguage
Easy to analyze,but prescriptiveNaturalLanguage
Flexible,but ambiguousProgrammingby Voice#33Programming by Voice Related WorkHuman-CentricComputer-CentricMultipleTasksAuthoringOnlyArnold 00Snell 00Price 00 02Desilets 01 04Gray 03Begel 05#34My goal is to create a solution that is both complete and human-centriccomplete in that it enables authoring, navigation, and editing.Human centric in that the machine adapts to the programmer. Everyone knows this is important workbut no one has accomplished it yet. My workgoes the farthest
A More Natural Way to Codepublic class symbol implements serializablepublic class Symbol implements Serializable { }#35Didnt have to verbalize parens, semicolons, etc.
A More Natural Way to Codestatic hash map hash table gets new hash mappublic class Symbol implements Serializable { static HashMap hashtbl = new HashMap();}#36All punctuation has English equivalentsOpen Brace, End For LoopMost punctuation is optionalProvide verbalization for all abbreviationsRelaxed phrasing for better fit with English(int)foo cast foo to integerfoo = 6 set foo to 6foo[i]++ increment the ith element of array foo
A More Natural Way to Codeend the classpublic class Symbol implements Serializable { static HashMap hashtbl = new HashMap();
}#37All punctuation has English equivalentsOpen Brace, End For LoopMost punctuation is optionalProvide verbalization for all abbreviationsRelaxed phrasing for better fit with English(int)foo cast foo to integerfoo = 6 set foo to 6foo[i]++ increment the ith element of array foo
A More Natural Way to Codefor int i equals zero i less than ten i plus plusfor (int i = 0; i < 10; i++ ) { }#38Didnt have to verbalize parens, semicolons, etc.
Too Many Ambiguitiesfor (int i = 0; i < 10; i++ ) { }4 int eye equals 0 aye less then ten i plus plusKW or #?Spelling of ID?KW or ID?for int i equals zero i less than ten i plus plus#39A human could eyeball this.
Sometimes Its Non-Obviousfor (times = 8; file(2, load); times == one) {}for times equals 8 file 2 load times equals onefore *= 8; file.tooLode.times = won 4; times = ate(file).to(load).equals(1) #40Spoken JavaSemantically identical to JavaSyntactically easier to say than JavaMethodology generalizable to any computer language
All punctuation has English equivalentsOpen Brace, End For LoopMost punctuation is optionalProvide verbalization for all abbreviationsRelaxed phrasing for better fit with English(int)foo cast foo to integerfoo = 6 set foo to 6foo[i]++ increment the ith element of array foo#41SPEED: Speech EditorBuild an editor that supports naturally verbalized programsSPEED: SPEech EDitorBased on IBM ViaVoice, Eclipse IDE, Harmonia
Spoken Java Language for CompositionSpoken Command language for Navigation, Editing, Template instantiation, Refactorings, SearchAudible and visual feedbackSimilar to JavaSpeak (Smith 2000)#42Harmonia Analysis FrameworkFramework to support interactive editorsLanguage-based, programmer-oriented toolsIncremental analysesLexing (Wagner 97), GLR Parsing (Wagner 97, Begel 04), Static Semantics (Garrison 87, Begel, Jamison)C, Java, Titanium, Cool, Flex, BisonAlso, languages where indentation and CRs are significantInteractive Program Transformations (Boshernitsan)CodeLink (Toomim et. al. 04)Shorthand Editing
#43Compile ErrorsStructural Navigation,v Browsing, Search, Code FoldingSemantic Info Display, Indentation, Syntax Highlighting
Talk OutlineIntroduction and MotivationProgramming by VoiceProgram Analyses for Ambiguous Inputs SPEech EDitor Programming EnvironmentSPEED User StudyConclusion#44Spoken input requires new compiler analysesExisting analyses are not built for ambiguityTarget 20 minutes.Traditional Compiler AnalysesLexicalAnalysisFORIParsingSemanticAnalysis
for (i = 0; i < 10; i++ ) { }Programming languages are designed to be unambiguousFor LoopFORAssign ExprI=0iLocalVarint#45Ambiguity-Aware Analyses
for i equals zero ...Handles input stream, syntactic and semantic ambiguitiesLexicalAnalysisFORIAmbiguousParsingSemanticAmbiguityResolutionFor LoopFORAssign ExprI=0iLocalVarintfoureyeLocalVar?4EYEFOUREYEAssign Expr=0Ambig Stmt#46Insight here is to generalize GLR algorithm for lexical ambiguitiestotally changes the algorithm.Scan Input Stream
CommercialSpeechRecognizerHomophoneDictionary
LexicalAnalysis
#47Homophones Cause Ambiguitiesforiequalsfor i =4 i equalsforeeye ==fore ayeequalsfoureyeequalsforiequalsConcatenated words cause them too4forefoureyeaye===#48Capitalization, abbreviationAmbiguity-Aware Analyses
for i equals zero ...LexicalAnalysisFORIXGLRAmbiguousParsingSemanticAmbiguityResolutionFor LoopFORAssign ExprI=0iLocalVarintfoureyeLocalVar?4EYEFOUREYEAssign Expr=0Ambig Stmt#49XGLR Parsing [Begel 04]