semeval-2017 task 11: end-user development using natural language
TRANSCRIPT
NLP & Semantic Computing Group
N L P
SemEval-2017 Task 11: End-User Development using Natural Language
Semeval, August 2017
Juliano Efson Sales, Siegfried Handschuh, Andre Freitas
NLP & Semantic Computing Group
Programming is a complex task
User effort associated with finding, understanding and integrating multiple
interfaces within a software development task.
NLP & Semantic Computing Group
https://medium.com/@keskkyla/https-medium-com-keskkyla-what-is-this-github-you-speak-of-dd457a29771
six thousand NEW people pushing code to github every day
NLP & Semantic Computing Group
KB of APIs Sem
an
tic
Pa
rsin
g
From this:
To this:
“Get the latest financial news about Google,
translate it to German and send it by email to
all stakeholders.”
NLP & Semantic Computing Group
Applications
• End-user programming user natural language.
• Semantic matching of software APIs and services.
Smart IDEs
Programming automation
Web service orchestration
NLP & Semantic Computing Group
Similar initiatives
• Robot commands (Pane and Myers, 2006), (Stenmark and Nugues, 2013), (Dukes, 2014)
• Operating system’s functions (Manaris and Dominick, 1993)
• Web services orchestration (Englmeier et al., 2006)
• Mobile programming by voice (Amos Azaria, 2016)
• Domain-specific natural language programming (Pane and Myers, 2006)
NLP & Semantic Computing Group
In comparison
• Open domain
• Large vocabulary
• Large KB of APIs (real world Web APIs)
• High lexical and semantic heterogeneity
NLP & Semantic Computing Group
From NL to n-ary predicate-argument structures
if (excitation_frequency > 20 & excitation_frequency < 20000) then simulation = valid
NLP & Semantic Computing Group
“Please convert US$ 475 to the Japanese currency and send this value to John Smith by SMS.”
SEQUENCE { ACTION: [ convert currency ] PARAMS: [US$ 475] - [(to) Japanese currency] ACTION: [ send sms ] PARAMS: [this value] - [(to) John Smith]
}
ACTION ENDPOINT: [action id] PARAMS:
from: “USD” to: “JPY” from amount: 475
NLP & Semantic Computing Group
Action KB • 3800 micro-services from Mashape (mashape.com)
• 1900 actions and triggers from the ifttt.com platform.
NLP & Semantic Computing Group
Test Collection Creation
• High-level task descriptions:
“You are arranging a meeting with some people in Andre’s office. Adamantios is coming for that meeting, but he does not know how to drive in Passau. Additionally, you do not know where the office is.”
1
NLP & Semantic Computing Group
Test Collection Creation
• Convert into NL commands using crowd-sourcing:
Ask Andre for the address of his office;
Make a map from the university to it;
Send the map to Adamantios including driving directions.
2
NLP & Semantic Computing Group
Test Collection Creation
• Data Curation
Many data quality issues
Correction of misspellings
3
NLP & Semantic Computing Group
Interpretation Challenges
• Lexical/conceptual approximation
“If someone reports a problem in GitHub, send the problem’s headline by Skype to John.”
any new issue (…)
issue title (…)
NLP & Semantic Computing Group
Syntactic and Ordering Variation
“If Manchester United wins, call me.”
“Get ready to call me in the case of victory of Manchester United.”
NLP & Semantic Computing Group
Co-reference and metonymy resolution
“If an issue is created, send its content to the Tech Manager.”
its -> return value of the issue
Tech Manager -> [email protected] according to the user KB
NLP & Semantic Computing Group
Evaluation
• 1. Resolved individual actions ignoring parameter values;
• 2. Resolved individual actions considering parameter values;
• 3. Resolved scenarios ignoring parameter values;
• 4. Resolved scenarios considering parameter values.
precision, recall
number of resolved scenarios
NLP & Semantic Computing Group
Challenging examples
• Description of commands split into two sentences.
“Get the price of the book The Intelligent Investor. If it costs less than 25 Euros, buy it.”
where “25 Euros” is the parameter value of the action defined in the first sentence.
NLP & Semantic Computing Group
Challenging examples
• Capturing actions with more specific/ fine-grained semantics.
“Once I have bet my running distance target of the week, set my current weight as 100 Kg in Fitbit.”
Selection of “Daily step goal achieved” instead of “Weekly distance goal reached”
NLP & Semantic Computing Group
Challenging examples
“Pay the invoice that I will receive from [email protected] with my credit card”
Implicit trigger (receive email) into the "pay" action.
“Send to my mother‘s email the music Bachianas N.5 of Villa-Lobos from Youtube.” Implicit "search video on YouTube" action.