cosql: a conversational text-to-sql challenge for natural ...myasu/files/cosql_slides.pdfcosql: a...
TRANSCRIPT
![Page 1: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/1.jpg)
CoSQL: A Conversational Text-to-SQL Challenge for Natural Language
Interfaces to Databases
EMNLP 2019
Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, Dragomir Radev
Yale University, Univ. of Michigan, Salesforce Research, Cornell University
![Page 2: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/2.jpg)
Outline
● What is NL2SQL and why Conversational NL2SQL?
● Task formulation
● Data collection
● Experiments
![Page 3: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/3.jpg)
NL2SQL Task
Interface system that allows humans to use natural language to query database (DB)
![Page 4: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/4.jpg)
NL2SQL Task
Interface system that allows humans to use natural language to query database (DB)
![Page 5: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/5.jpg)
NL2SQL Task
Interface system that allows humans to use natural language to query database (DB)
NL2SQL
![Page 6: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/6.jpg)
NL2SQL Task
ExecuteGermany
Interface system that allows humans to use natural language to query database (DB)
NL2SQL
![Page 7: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/7.jpg)
NL2SQL datasets
Great datasets
![Page 8: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/8.jpg)
NL2SQL datasets
Great datasets
● ATIS [Price, 1990; Hemphill et al. 1990; Dahl et al., 1994]
● GeoQuery [Zelle and Mooney, 1996]
● Academic [Li and Jagadish, 2014]
● Scholar [Iyer et al., 2017]
● WikiSQL [Zhong et al., 2017]
● Spider [Yu et al., 2018]
ATIS Geo
Academic Spider (multi-domain)
![Page 9: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/9.jpg)
NL2SQL datasets
Great datasets
● ATIS [Price, 1990; Hemphill et al. 1990; Dahl et al., 1994]
● GeoQuery [Zelle and Mooney, 1996]
● Academic [Li and Jagadish, 2014]
● Scholar [Iyer et al., 2017]
● WikiSQL [Zhong et al., 2017]
● Spider [Yu et al., 2018]
○ Large & multi-domain (10,000+ questions over 200 DBs)○ Complex questions (covers most SQL keywords, multiple tables, etc.)
ATIS Geo
Academic Spider (multi-domain)
![Page 10: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/10.jpg)
NL2SQL datasets
Great datasets
● ATIS [Price, 1990; Hemphill et al. 1990; Dahl et al., 1994]
● GeoQuery [Zelle and Mooney, 1996]
● Academic [Li and Jagadish, 2014]
● Scholar [Iyer et al., 2017]
● WikiSQL [Zhong et al., 2017]
● Spider [Yu et al., 2018]
○ Large & multi-domain (10,000+ questions over 200 DBs)○ Complex questions (covers most SQL keywords, multiple tables, etc.)
… but assume all utterances are single-sentence, well-formed questions
ATIS Geo
Academic Spider (multi-domain)
![Page 11: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/11.jpg)
In practice ...
Complex requests are processed through interactions rather than a single utterance
![Page 12: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/12.jpg)
In practice ...
Complex requests are processed through interactions rather than a single utterance● ?? “Show me Delta flights from Seattle to Boston next Monday after 7pm”
![Page 13: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/13.jpg)
In practice ...
Complex requests are processed through interactions rather than a single utterance● ?? “Show me Delta flights from Seattle to Boston next Monday after 7pm” ● Instead, users would explore DB interactively
Image from [Suhr et al., 2018]
![Page 14: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/14.jpg)
How about context-dependent NL2SQL?
ATIS [Price, 1990; Hemphill et al. 1990; Dahl et al., 1994]
![Page 15: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/15.jpg)
How about context-dependent NL2SQL?
ATIS [Price, 1990; Hemphill et al. 1990; Dahl et al., 1994]● All user questions can be
mapped into SQL queries
![Page 16: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/16.jpg)
How about context-dependent NL2SQL?
ATIS [Price, 1990; Hemphill et al. 1990; Dahl et al., 1994]● All user questions can be
mapped into SQL queries⇒ cannot handle ambiguous/unanswerable/unrelated questions real users may ask
![Page 17: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/17.jpg)
How about context-dependent NL2SQL?
ATIS [Price, 1990; Hemphill et al. 1990; Dahl et al., 1994]● All user questions can be
mapped into SQL queries⇒ cannot handle ambiguous/unanswerable/unrelated questions real users may ask
● System only returns exec result.
![Page 18: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/18.jpg)
How about context-dependent NL2SQL?
ATIS [Price, 1990; Hemphill et al. 1990; Dahl et al., 1994]● All user questions can be
mapped into SQL queries⇒ cannot handle ambiguous/unanswerable/unrelated questions real users may ask
● System only returns exec result. ⇒ Hard for user to check if their questions are interpreted correctly.
![Page 19: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/19.jpg)
How about context-dependent NL2SQL?
ATIS [Price, 1990; Hemphill et al. 1990; Dahl et al., 1994]● All user questions can be
mapped into SQL queries⇒ cannot handle ambiguous/unanswerable/unrelated questions real users may ask
● System only returns exec result. ⇒ Hard for user to check if their questions are interpreted correctly. Want system response explaining SQL interpretation & exec result
![Page 20: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/20.jpg)
Most previous work expects user query to be well-formed and in a single sentence
But in practice,● User may prefer to explore DB through interactive exchanges
Our goal: Conversational NL2SQL (CoSQL)
![Page 21: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/21.jpg)
Most previous work expects user query to be well-formed and in a single sentence
But in practice,● User may prefer to explore DB through interactive exchanges
● User may ask ambiguous/unanswerable/unrelated questions, and system needs to detect and inform user (e.g. ask for clarification)
Our goal: Conversational NL2SQL (CoSQL)
![Page 22: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/22.jpg)
Most previous work expects user query to be well-formed and in a single sentence
But in practice,● User may prefer to explore DB through interactive exchanges
● User may ask ambiguous/unanswerable/unrelated questions, and system needs to detect and inform user (e.g. ask for clarification)
● More user-friendly and interpretable if system explains the SQL interpretation and execution result of user questions
Our goal: Conversational NL2SQL (CoSQL)
![Page 23: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/23.jpg)
Most previous work expects user query to be well-formed and in a single sentence
But in practice,● User may prefer to explore DB through interactive exchanges
● User may ask ambiguous/unanswerable/unrelated questions, and system needs to detect and inform user (e.g. ask for clarification)
● More user-friendly and interpretable if system explains the SQL interpretation and execution result of user questions
⇒ We collect real-world dialog data on Mechanical Turk
Our goal: Conversational NL2SQL (CoSQL)
![Page 24: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/24.jpg)
The system needs to:
I. Understand users’ questions, and determine whether the questions can be answered by SQL
CoSQL (Desiderata & Tasks)
![Page 25: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/25.jpg)
The system needs to:
I. Understand users’ questions, and determine whether the questions can be answered by SQL
CoSQL (Desiderata & Tasks)
Dialog act management
![Page 26: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/26.jpg)
Task 1. Dialog Act Management
CoSQL (Desiderata & Tasks)
![Page 27: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/27.jpg)
Task 1. Dialog Act Management
● Given user’s question, the system classify it into an utterance type:
Context-dependent user question Dialog act type
CoSQL (Desiderata & Tasks)
![Page 28: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/28.jpg)
Task 1. Dialog Act Management
● Given user’s question, the system classify it into an utterance type:“INFORM_SQL” (answerable with SQL), “AMBIGUOUS”, “GREETING”, “CANNOT_ANSWER”, “NOT_RELATED”, etc.
Context-dependent user question Dialog act type
CoSQL (Desiderata & Tasks)
![Page 29: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/29.jpg)
Some interesting dialog action types with examples:
1. INFER_SQL: if the user’s question must be answered by SQL+human inference. SQL cannot directly return the answer, but we can infer the answer based on the SQL results.Examples: users’ questions are “are they..?” (yes/no question) or “the 3rd oldest...”. Are there more female or male students overall? Can you list all the events that happened within the last 5 years?
2. AMBIGUOUS: the user’s question is ambiguous, the system needs to double check the user's intent (e.g. what/did you mean by…?) or ask for which columns to return.
3. AFFIRM: affirm something said by the system (user says yes/agree)
4. NEGATE: negate something said by the system (user says no/deny)
5. NOT_RELATED: the user’s question is not related to the database, the system reminds the user
6. CANNOT_UNDERSTAND: the user’s question cannot be understood by the system, the system asks the user to rephrase or paraphrase question.
7. CANNOT_ANSWER: the user’s question cannot be easily answered by SQL, the system tells the user its limitation.Example: What is the average population across the counties? Which county has a population closest to that?
Task 1: Dialog Act Management
![Page 30: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/30.jpg)
The system needs to:
I. Understand users’ questions, and determine whether the questions can be answered by SQL
CoSQL (Desiderata & Tasks)
Dialog act management
![Page 31: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/31.jpg)
The system needs to:
I. Understand users’ questions, and determine whether the questions can be answered by SQL
II. If the questions can be answered by SQL, translate them into SQL
CoSQL (Desiderata & Tasks)
Dialog act management
![Page 32: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/32.jpg)
The system needs to:
I. Understand users’ questions, and determine whether the questions can be answered by SQL
II. If the questions can be answered by SQL, translate them into SQL
III. If the questions cannot be answered by SQL, inform user
CoSQL (Desiderata & Tasks)
Dialog act management
![Page 33: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/33.jpg)
The system needs to:
I. Understand users’ questions, and determine whether the questions can be answered by SQL
II. If the questions can be answered by SQL, translate them into SQL
III. If the questions cannot be answered by SQL, inform user
CoSQL (Desiderata & Tasks)
Dialog act management
Dialog SQL state tracking
![Page 34: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/34.jpg)
Task 2. Dialog SQL State Tracking
CoSQL (Desiderata & Tasks)
![Page 35: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/35.jpg)
Task 2. Dialog SQL State Tracking
● Given a NL question in dialog → map it to SQL if possible.
CoSQL (Desiderata & Tasks)
Context-dependent user question SQL query
![Page 36: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/36.jpg)
Task 2. Dialog SQL State Tracking
● Given a NL question in dialog → map it to SQL if possible.● Context-dependent semantic parsing
CoSQL (Desiderata & Tasks)
Context-dependent user question SQL query
![Page 37: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/37.jpg)
The system needs to:
I. Understand users’ questions, and determine whether the questions can be answered by SQL
II. If the questions can be answered by SQL, translate them into SQL
III. If the questions cannot be answered by SQL, inform user
CoSQL (Desiderata & Tasks)
Dialog act management
Dialog SQL state tracking
![Page 38: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/38.jpg)
The system needs to:
I. Understand users’ questions, and determine whether the questions can be answered by SQL
II. If the questions can be answered by SQL, translate them into SQL
III. If the questions cannot be answered by SQL, inform user
IV. Show results to usersV. Describe how it got the result and what the result means
CoSQL (Desiderata & Tasks)
Dialog act management
Dialog SQL state tracking
![Page 39: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/39.jpg)
The system needs to:
I. Understand users’ questions, and determine whether the questions can be answered by SQL
II. If the questions can be answered by SQL, translate them into SQL
III. If the questions cannot be answered by SQL, inform user
IV. Show results to usersV. Describe how it got the result and what the result means
CoSQL (Desiderata & Tasks)
Dialog act management
Dialog SQL state tracking
System response generation
![Page 40: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/40.jpg)
Task 3. System response generation
CoSQL (Desiderata & Tasks)
![Page 41: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/41.jpg)
Task 3. System response generation
● SQL query and execution results → generate a NL response that describes the query and returned results
CoSQL (Desiderata & Tasks)
SQL query & execution result System response (NL description)
![Page 42: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/42.jpg)
Task 3. System response generation
● SQL query and execution results → generate a NL response that describes the query and returned results
● Allow users to verify if the result answers their questions.
CoSQL (Desiderata & Tasks)
SQL query & execution result System response (NL description)
![Page 43: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/43.jpg)
CoSQL (Example)Database
![Page 44: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/44.jpg)
CoSQL (Example)
User Question
Database
![Page 45: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/45.jpg)
CoSQL (Example)
Interface System
User Question
Database
![Page 46: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/46.jpg)
CoSQL (Example)
![Page 47: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/47.jpg)
CoSQL (Example)
![Page 48: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/48.jpg)
CoSQL (Example)
![Page 49: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/49.jpg)
CoSQL (Example)
![Page 50: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/50.jpg)
CoSQL (Example)
![Page 51: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/51.jpg)
CoSQL (Example)
![Page 52: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/52.jpg)
Examples1. If the returned result can be combined with the SQL description, combine
them together to generate the response. For example: Given SQL: SELECT avg(salary) FROM instructor Result Returned: 200k
System Response Generation (example)
![Page 53: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/53.jpg)
Examples1. If the returned result can be combined with the SQL description, combine
them together to generate the response. For example: Given SQL: SELECT avg(salary) FROM instructor Result Returned: 200k Response: The average salary of all instructors is 200k.
System Response Generation (example)
![Page 54: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/54.jpg)
Examples1.
2. If the returned result is too large and cannot be combined with the SQL description, describe them separately. For example:
Given SQL: SELECT avg(T1.salary), T1.department_id FROM instructor as T1 JOIN department
as T2 ON T1.department_id = T2.id GROUP BY T1.department_id
Result Returned: a long table
System Response Generation (example)
![Page 55: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/55.jpg)
Examples1.
2. If the returned result is too large and cannot be combined with the SQL description, describe them separately. For example:
Given SQL: SELECT avg(T1.salary), T1.department_id FROM instructor as T1 JOIN department
as T2 ON T1.department_id = T2.id GROUP BY T1.department_id
Result Returned: a long table Response: Here is the result table that shows the average salary in
each department. For example, the average of CS professors is 250k.
System Response Generation (example)
![Page 56: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/56.jpg)
Task-oriented dialog: dialog state-tracking with pre-defined slots and values
Task-oriented dialog task v.s. CoSQL
Topic:'food' Cuisine:'Chinese'Neighborhood:'Palo Alto'
![Page 57: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/57.jpg)
Task-oriented dialog: dialog state-tracking with pre-defined slots and values
CoSQL: cover more diverse & complex semantics of practical user questions with SQL
Task-oriented dialog task v.s. CoSQL
SELECT T2.name, T2.budgetFROM instructor as T1 JOIN department as T2 ON T1.department_id = T2.idGROUP BY T1.department_idHAVING avg(T1.salary) > (SELECT avg(salary) FROM instructor)
Topic:'food' Cuisine:'Chinese'Neighborhood:'Palo Alto'
![Page 58: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/58.jpg)
Task-oriented dialog: dialog state-tracking with pre-defined slots and values
CoSQL: cover more diverse & complex semantics of practical user questions with SQL
Task-oriented dialog task v.s. CoSQL
SELECT T2.name, T2.budgetFROM instructor as T1 JOIN department as T2 ON T1.department_id = T2.idGROUP BY T1.department_idHAVING avg(T1.salary) > (SELECT avg(salary) FROM instructor)
Topic:'food' Cuisine:'Chinese'Neighborhood:'Palo Alto'
![Page 59: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/59.jpg)
Task-oriented dialog: dialog state-tracking with pre-defined slots and values
CoSQL: cover more diverse & complex semantics of practical user questions with SQL
Task-oriented dialog task v.s. CoSQL
SELECT T2.name, T2.budgetFROM instructor as T1 JOIN department as T2 ON T1.department_id = T2.idGROUP BY T1.department_idHAVING avg(T1.salary) > (SELECT avg(salary) FROM instructor)
Topic:'food' Cuisine:'Chinese'Neighborhood:'Palo Alto'
![Page 60: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/60.jpg)
Task-oriented dialog: dialog state-tracking with pre-defined slots and values
CoSQL: cover more diverse & complex semantics of practical user questions with SQL
Task-oriented dialog task v.s. CoSQL
SELECT T2.name, T2.budgetFROM instructor as T1 JOIN department as T2 ON T1.department_id = T2.idGROUP BY T1.department_idHAVING avg(T1.salary) > (SELECT avg(salary) FROM instructor)
Topic:'food' Cuisine:'Chinese'Neighborhood:'Palo Alto'
![Page 61: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/61.jpg)
Data Collection
● Wizard of Oz method: Real-time dialog between two players (user & system)
![Page 62: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/62.jpg)
Interface System: CS students with SQL skill
Data Collection
DB User: Crowd Workers
● Wizard of Oz method: Real-time dialog between two players (user & system)
![Page 63: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/63.jpg)
Interface System: CS students with SQL skill
Data Collection
DB User: Crowd Workers
● Wizard of Oz method: Real-time dialog between two players (user & system)● Take each question from Spider as user’s dialog goal.
![Page 64: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/64.jpg)
Interface System: CS students with SQL skill
Data Collection
DB User: Crowd Workers1. Explore DB2. Query DB3. Verify the response
from system
● Wizard of Oz method: Real-time dialog between two players (user & system)● Take each question from Spider as user’s dialog goal.
![Page 65: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/65.jpg)
Interface System: CS students with SQL skill1. Dialog act management2. SQL translation3. Response generation4. Inform of
ambiguous/unanswerable questions
Data Collection
DB User: Crowd Workers1. Explore DB2. Query DB3. Verify the response
from system
● Wizard of Oz method: Real-time dialog between two players (user & system)● Take each question from Spider as user’s dialog goal.
![Page 66: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/66.jpg)
Data Collection (MTurk Process)
Step1 User tutorial & quality control
![Page 67: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/67.jpg)
Data Collection (MTurk Process)
Step1 User tutorial & quality control
Step2 Real-time pairing of
user & DB expert
![Page 68: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/68.jpg)
Data Collection (MTurk Process)
Step1 User tutorial & quality control
Step2 Real-time pairing of
user & DB expert
Step3User & expert check DB
tables and task-goal
![Page 69: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/69.jpg)
Data Collection (MTurk Process)
Step1 User tutorial & quality control
Step2 Real-time pairing of
user & DB expert
Step3User & expert check DB
tables and task-goal
![Page 70: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/70.jpg)
Data Collection (MTurk Process)
Step1 User tutorial & quality control
Step2 Real-time pairing of
user & DB expert
Step3User & expert check DB
tables and task-goal
Step4Real-time
chat
User side System side
![Page 71: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/71.jpg)
Data Collection (MTurk Process)
Step1 User tutorial & quality control
Step2 Real-time pairing of
user & DB expert
Step3User & expert check DB
tables and task-goal
Step4Real-time
chat
User side System side
![Page 72: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/72.jpg)
Dataset Statistics
CoSQL v.s. context-dependent NL2SQL
![Page 73: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/73.jpg)
Dataset Statistics
CoSQL v.s. context-dependent NL2SQL ⇒ More practically useful & interpretable
![Page 74: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/74.jpg)
Dataset Statistics
CoSQL v.s. task-oriented dialogue datasets
![Page 75: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/75.jpg)
Dataset Statistics
CoSQL v.s. task-oriented dialogue datasets ⇒ Semantically diverse
![Page 76: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/76.jpg)
inform_sql
thank_you
ambiguous
Dialog act distribution
cannot_answer
![Page 77: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/77.jpg)
Semantic change by turns
![Page 78: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/78.jpg)
User follow-up question characteristics
Coreference Change constraint
Different attribute of same topic
Same attribute for different topic
Question about the answer
Previous Q How many female students are in the class?
Follow up Q
![Page 79: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/79.jpg)
User follow-up question characteristics
Coreference Change constraint
Different attribute of same topic
Same attribute for different topic
Question about the answer
Previous Q How many female students are in the class?
Follow up Q What are their names?
![Page 80: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/80.jpg)
User follow-up question characteristics
Coreference Change constraint
Different attribute of same topic
Same attribute for different topic
Question about the answer
Previous Q How many female students are in the class?
Show me courses taught in the winter
Follow up Q What are their names?
How about in the summer?
![Page 81: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/81.jpg)
User follow-up question characteristics
Coreference Change constraint
Different attribute of same topic
Same attribute for different topic
Question about the answer
Previous Q How many female students are in the class?
Show me courses taught in the winter
Where is the location of the conference?
Follow up Q What are their names?
How about in the summer?
What is the time?
![Page 82: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/82.jpg)
User follow-up question characteristics
Coreference Change constraint
Different attribute of same topic
Same attribute for different topic
Question about the answer
Previous Q How many female students are in the class?
Show me courses taught in the winter
Where is the location of the conference?
What is the price of a bagel?
Follow up Q What are their names?
How about in the summer?
What is the time?
And a swiss roll?
![Page 83: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/83.jpg)
User follow-up question characteristics
Coreference Change constraint
Different attribute of same topic
Same attribute for different topic
Question about the answer
Previous Q How many female students are in the class?
Show me courses taught in the winter
Where is the location of the conference?
What is the price of a bagel?
What is the highest rated college in CT?Answer: Yale
Follow up Q What are their names?
How about in the summer?
What is the time?
And a swiss roll?
![Page 84: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/84.jpg)
User follow-up question characteristics
Coreference Change constraint
Different attribute of same topic
Same attribute for different topic
Question about the answer
Previous Q How many female students are in the class?
Show me courses taught in the winter
Where is the location of the conference?
What is the price of a bagel?
What is the highest rated college in CT?Answer: Yale
Follow up Q What are their names?
How about in the summer?
What is the time?
And a swiss roll? How many departments does Yale have?
![Page 85: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/85.jpg)
Context-dependent user question Dialog act type
Tasks & Experiments
![Page 86: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/86.jpg)
Context-dependent user question SQL query
Context-dependent user question Dialog act type
Tasks & Experiments
![Page 87: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/87.jpg)
Context-dependent user question SQL query
SQL query & execution result System response (NL description)
Context-dependent user question Dialog act type
Tasks & Experiments
Dialog act management
Dialog SQL state tracking
System response generation
![Page 88: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/88.jpg)
Context-dependent user question Dialog act type
Task 1: User Dialogue Act Prediction
![Page 89: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/89.jpg)
Models● Majority -- always predict “INFORM_SQL”● TBCNN [Mouet al., 2016]
Context-dependent user question Dialog act type
Task 1: User Dialogue Act Prediction
![Page 90: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/90.jpg)
Models● Majority -- always predict “INFORM_SQL”● TBCNN [Mouet al., 2016]
Accuracy (%)
Context-dependent user question Dialog act type
Task 1: User Dialogue Act Prediction
![Page 91: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/91.jpg)
Models● Majority -- always predict “INFORM_SQL”● TBCNN [Mouet al., 2016]
Accuracy (%)
Context-dependent user question Dialog act type
✓ common types (“inform_sql”, “thank_you”, “greetings”)
✘ other types (“ambiguous”, “cannot_answer”, etc.)
Task 1: User Dialogue Act Prediction
![Page 92: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/92.jpg)
Context-dependent user question SQL query
Task 2: SQL state tracking (context-dependent NL2SQL)
![Page 93: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/93.jpg)
Models● Context-dependent Seq2Seq [Suhr et al., 2018]
● Context-dependent SyntaxSQLNet [Yu et al., 2019]
Context-dependent user question SQL query
Task 2: SQL state tracking (context-dependent NL2SQL)
![Page 94: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/94.jpg)
Models● Context-dependent Seq2Seq [Suhr et al., 2018]
● Context-dependent SyntaxSQLNet [Yu et al., 2019]
Context-dependent user question SQL query
Task 2: SQL state tracking (context-dependent NL2SQL)
![Page 95: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/95.jpg)
Models● Context-dependent Seq2Seq [Suhr et al., 2018]
● Context-dependent SyntaxSQLNet [Yu et al., 2019]
On Spider (single turn):
21%
Context-dependent user question SQL query
Task 2: SQL state tracking (context-dependent NL2SQL)
![Page 96: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/96.jpg)
SQL query & execution result System response (NL description)
Task 3: System Response Generation
![Page 97: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/97.jpg)
Models● Template -- prepared from (SQL, response) pairs in training data● Seq2seq● Pointer-generator [See et al., 2017]
SQL query & execution result System response (NL description)
Task 3: System Response Generation
![Page 98: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/98.jpg)
Models● Template -- prepared from (SQL, response) pairs in training data● Seq2seq● Pointer-generator [See et al., 2017]
SQL query & execution result System response (NL description)
Task 3: System Response Generation
![Page 99: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/99.jpg)
Models● Template -- prepared from (SQL, response) pairs in training data● Seq2seq● Pointer-generator [See et al., 2017]
Human evaluation
SQL query & execution result System response (NL description)
Task 3: System Response Generation
Semantic correctness
![Page 100: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/100.jpg)
● CoSQL, first truly conversational NL2SQL corpus○ Include ambiguous/unanswerable questions real users ask○ Include system response to improve interpretability & user experience
Summary
![Page 101: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/101.jpg)
● CoSQL, first truly conversational NL2SQL corpus○ Include ambiguous/unanswerable questions real users ask○ Include system response to improve interpretability & user experience
● More diversity and complexity in semantics and discourse (e.g. ambiguous questions, multi-domain), compared to related datasets
Summary
![Page 102: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/102.jpg)
● CoSQL, first truly conversational NL2SQL corpus○ Include ambiguous/unanswerable questions real users ask○ Include system response to improve interpretability & user experience
● More diversity and complexity in semantics and discourse (e.g. ambiguous questions, multi-domain), compared to related datasets
● Experiments on the three tasks show a large room for future research
Summary
![Page 103: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/103.jpg)
● CoSQL, first truly conversational NL2SQL corpus○ Include ambiguous/unanswerable questions real users ask○ Include system response to improve interpretability & user experience
● More diversity and complexity in semantics and discourse (e.g. ambiguous questions, multi-domain), compared to related datasets
● Experiments on the three tasks show a large room for future research
● CoSQL project page: https://yale-lily.github.io/cosql
Summary
![Page 104: CoSQL: A Conversational Text-to-SQL Challenge for Natural ...myasu/files/CoSQL_slides.pdfCoSQL: A Conversational Text-to-SQL Challenge for Natural Language Interfaces to Databases](https://reader034.vdocument.in/reader034/viewer/2022050404/5f810be90e5b2767c468cee3/html5/thumbnails/104.jpg)
Thank you!Michihiro Yasunaga