SmartSynth: Synthesizing Smartphone Automation

Scripts from Natural Language

Vu Le (UC Davis)Sumit Gulwani (MSR Redmond)

Zhendong Su (UC Davis)

motivation

programming

App Inventor Tasker

natural languageWhen I receive a new SMS, if the phone is connected to my car’s bluetooth, it reads the message content and replies the sender “I’m driving.”

scriptwhen (number, content) := MessageReceived()

if (IsConnectedToBTDevice(Car_BT) then

Speak(content);SendMessage(number, "I'm driving");

SmartScriptScript P ::= I E T C MParameter I ::= input (i1, …, in) | εEvent E ::= (r1, …, rn) := when Event() | εConversions T ::= F1; …; Fn; Condition C ::= if (Π1 ∧ … ∧ Πn) then | εBody M ::= Stmt1; … ; Stmtn;Conversion F ::= x := Convert(a)Predicate Π ::= Predicate (a1, …, an) Statement Stmt ::= S | foreach x in a do S1; … ; Sn; odAtomic Stmt. S ::= A | FAction A ::= (r1, …, rn) := Action (a1, …, an) Argument a ::= x | i | r | l

approachScript = Components +

Relations

approachScript = Components +

Relations

approachScript = Components +

Relations

approachScript = Components +

Relations

definitiono Component

• API: MessageReceived• Entity• Literal: “I’m driving”• API return value: MessageReceived.TextO

o Relation = <Entity, API parameter>• E.g., < “I’m driving”, SendMessage.TextI

>

component discovery

Description ComponentWhen I receive a new SMS

MessageReceived

if the phone is connected to

IsConnectedToBTDevice

my car’s bluetooth Car_BTreads Speakthe message content MessageReceived.TextO

replies SendMessagethe sender MessageReceived.Num

berO

“I’m driving” "I'm driving"

mapping featureso Regular expressionso Bag of wordso Phrase lengtho Punctuationo Parse tree (Stanford NLP parser)

relation discoveryo Rule-based relation detection

• Relative locations of APIs and entities

Return Value or Literal API ParameterCar_BT IsConnectedToBTDevice.T

extI

MessageReceived.TextO Speak.TextIMessageReceived.NumberO

SendMessage.NumberI

“I’m driving” SendMessage.TextI

script generationwhen (number, content) := MessageReceived()

if (IsConnectedToBTDevice(Car_BT) then

Speak(content);

SendMessage(number, "I'm driving");

ambiguityDescription Possible Component

MappingsWhen I receive a new SMS

MessageReceived

if the phone is connected to

IsConnectedToBTDeviceIsConnectedToWifiNetworkIsConnectedToDataService

my car’s bluetooth Car_BTreads Speakthe message content MessageReceived.TextO

repliesSendMessageSendEmail

“I’m driving” "I'm driving"

incompletenesswhen (number, content) := MessageReceived()

if (IsConnectedToBTDevice(Car_BT) then

Speak(content);

SendMessage(number, "I'm driving");

insightNLP + Program

Synthesis

ambiguity{ MessagedReceived, IsConnectedToBTDevice, Car_BT,

Speak, MessageReceived.TextO, (SendMessage OR SendEmail), "I'm driving“ }

Ranking

Script 1 (SendMessage)

SendMessage

SendEmails

Script 1

Script 2

Synthesizer

incompletenesso Search for most likely missing

relations

q & ao Distinguishing multiple choice

questions• Question: API parameter• Answers: Equally-likely related entities

What do you want the phone to speak?A. The received message contentB. “I’m driving”

architecture

Component Discovery

ScriptDiscovery

Natural LanguageDescription

MappingComponents

IntendedScript

Natural LanguageQ&A (Optional)

1

2

3

5

User

Feedback onDescription (Optional)

6

Feedback onComp. Mapping

4

evaluationo 50 scripts collected from user forumso User study: give NL descriptions from

scripts• 11 students participated• Each student was given 25 problems• 640 correct descriptions (725 total)

component mapping

[1] [1] [2] [1] [2] [3] [1][2][3][4] SmartSynth0

10

20

30

40

50

60

70

80

90

100

Series1[1] Regex + Bags-of-Words [2] Phrase length[3] Punctuation [4] Parse tree

relation detection

1 2 3 4 7 80

1

2

3

4

5

# Relations

# D

etec

ted

Rela

tions

completing ratio

0 1 2 3 4 7 80

50

100

150

200

250

Detected by NLP Engine Completed by Synthesis Engine

# Relations

# De

scrip

tions

synthesis time

2 3 4 5 6 7 8 10 11 120

1

2

3

4

5

6

7

Parsing time Total time

# Components

Tim

e (s

)

related worko General purpose programming using

natural languageo Natural language interfaceso Specification extraction from natural

language

thank youScript = Components +

Relations

NLP

Program Synthesis

Identify

Detect

Resolve

Complete

v.s. SiriConversation

Single phone API

Composition of APIs

Siri

SmartSynth

v.s. Tasker

TaskerSmartSynth

Time

Prod

uctiv

ity