dialogue - cs.princeton.edu

70
Dialogue Fall 2019 COS 484: Natural Language Processing

Upload: others

Post on 01-Dec-2021

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Dialogue - cs.princeton.edu

Dialogue

Fall 2019

COS 484: Natural Language Processing

Page 2: Dialogue - cs.princeton.edu

Overview

• What’s a dialogue system? • Properties of Human Conversation • Chatbots v.s. Task-oriented dialogues systems • Rule-based v.s. Data-driven • Remaining Challenges

Dialogue Systems

Page 3: Dialogue - cs.princeton.edu

Overview

• What’s a dialogue system? • Properties of Human Conversation • Chatbots v.s. Task-oriented dialog systems • Rule-based v.s. Data-driven • Remaining Challenges

Dialogue Systems

Page 4: Dialogue - cs.princeton.edu

What’s a Dialogue System?

Dialog Systems are HOT 🔥. — Did you use it?

Page 5: Dialogue - cs.princeton.edu

Desktop

Dialog Systems are HOT 🔥. — Preferable user interface.

Smart Mobile Embedded Devices

keyboard & mouse

“turn off the light.”

language

What’s a Dialogue System?

Page 6: Dialogue - cs.princeton.edu

What’s a Dialogue System?

Google Duplex: Can you distinguish human and AI?

Dialog Systems are HOT 🔥. — Killer apps for NLP.

Page 7: Dialogue - cs.princeton.edu

What’s a Dialogue System?

Google Duplex: Can you distinguish human and AI?

(https://techeology.com/what-is-google-duplex/)

Page 8: Dialogue - cs.princeton.edu

What’s a Dialogue System?

Dialog Systems are HOT 🔥. — Killer apps for NLP.

• give travel directions • control home appliances • find restaurants • help make phone calls • customer services • …

They can

Page 9: Dialogue - cs.princeton.edu

Overview

• What’s a dialog system? • Properties of Human Conversation • Chatbots v.s. Task-oriented dialog systems • Rule-based v.s. Data-driven • Remaining Challenges

Dialogue Systems

Page 10: Dialogue - cs.princeton.edu

Properties of Human Conversation

A: travel agent C: human client

Page 11: Dialogue - cs.princeton.edu

Properties of Human Conversation

Turn structure: (C-A-C-A-C…)

Page 12: Dialogue - cs.princeton.edu

Properties of Human Conversation

Turn structure: (C-A-C-A-C…)

Spoken DS: endpoint detection (know when to start talking)

Page 13: Dialogue - cs.princeton.edu

Properties of Human Conversation

#: overlap

Page 14: Dialogue - cs.princeton.edu

Properties of Human Conversation

asking

Page 15: Dialogue - cs.princeton.edu

Properties of Human Conversation

answering

Page 16: Dialogue - cs.princeton.edu

Properties of Human Conversation

asking

Page 17: Dialogue - cs.princeton.edu

Properties of Human Conversation

answering

Page 18: Dialogue - cs.princeton.edu

Properties of Human Conversation

answering

Page 19: Dialogue - cs.princeton.edu

Properties of Human Conversation

A taxonomy of dialog acts (Bach and Harnish, 1979)

Page 20: Dialogue - cs.princeton.edu

Properties of Human Conversation

grounding: acknowledge that the listener has understood the speaker

Page 21: Dialogue - cs.princeton.edu

Properties of Human Conversation

correction subdialogue

local structure between dialog acts

Page 22: Dialogue - cs.princeton.edu

Properties of Human Conversation

mixed initiative: conversation is not controlled by one participant

Page 23: Dialogue - cs.princeton.edu

Properties of Human Conversation

mixed initiative: conversation is not controlled by one participant

user-initiative v.s.

system-initiative

Page 24: Dialogue - cs.princeton.edu

Properties of Human Conversation

conversational implicature

Page 25: Dialogue - cs.princeton.edu

Properties of Human Conversation

• Turn structure • Dialog acts • Grounding • Dialogue structure • Initiative • Implicature

These subtle characteristics of human conversations are among the reasons it is difficult to build dialogue systems

that can carry on natural conversations with humans.

Page 26: Dialogue - cs.princeton.edu

Overview

• What’s a dialog system? • Properties of Human Conversation • Chatbots v.s. Task-oriented dialog systems • Rule-based v.s. Data-driven • Remaining Challenges

Dialogue Systems

Page 27: Dialogue - cs.princeton.edu

Chatbot

Chatbots: A conversation between a user and Cleverbot, an IR-based chatbot (Carpenter, 2017).

GOAL mimicking the unstructured

conversational characteristic of human-human interaction

Page 28: Dialogue - cs.princeton.edu

Chatbot

Rule-based method: A simplified sketch of the ELIZA algorithm. The power of the algorithm comes from the particular transforms associated with each keyword.

Page 29: Dialogue - cs.princeton.edu

Chatbot

Corpus-based method (Information Retrieval): Return the response to the most similar turn.

(Jafarpour et al. 2009, Leuski and Traum 2011)

Given user query q and a conversational corpus C. Find the turn t in C that is most similar to q, and

return the following turn.

Page 30: Dialogue - cs.princeton.edu

Chatbot

Corpus-based method (Seq2Seq): An encoder decoder model for neural response

generation in dialogue.

Page 31: Dialogue - cs.princeton.edu

Chatbot

Corpus-based method (Seq2Seq): Sample responses generated by a Seq2Seq model trained either with a

maximum likelihood objective, or adversarially trained to produce sentences that are hard for an adversary to distinguish from human

sentences (Li et al., 2017).

Page 32: Dialogue - cs.princeton.edu

Chatbot

• Goal: • mimicking the unstructured conversational characteristic of

human-human interaction • Methods:

• Rule-based • Corpus-based (IR, Seq2Seq)

• Evaluation: • Chatbots are generally evaluated by humans • Adversarial evaluation: train a “Turing-like” evaluator classifier

to distinguish between human-generated responses and machine-generated responses.

Page 33: Dialogue - cs.princeton.edu

Task-Oriented Dialog System (Travel): A transcript of an actual dialog with the GUS system of Bobrow et al. (1977) P.S.A. and Air California were airlines of that period.

GOAL get information from the

user to help complete the specific task.

Task-Oriented Dialogue System

Page 34: Dialogue - cs.princeton.edu

Domain-Specific Knowledge: Ontology / Frame / Slot / Value

Task-Oriented Dialogue System

How to incorporate task related knowledge?

Page 35: Dialogue - cs.princeton.edu

Domain-Specific Knowledge: Ontology / Frame / Slot / Value

a knowledge structure representing the kinds of intentions the system can extract from user sentences.

Task-Oriented Dialogue System

How to incorporate task related knowledge?

Page 36: Dialogue - cs.princeton.edu

Domain-Specific Knowledge: Ontology / Frame / Slot / Value

a knowledge structure representing the kinds of intentions the system can extract from user sentences.

contains one or more frames.

Task-Oriented Dialogue System

How to incorporate task related knowledge?

Page 37: Dialogue - cs.princeton.edu

Domain-Specific Knowledge: Ontology / Frame / Slot / Value

a collection of slots

Slot1……

Slot2……

Slot3……

Slot4 ……

Task-Oriented Dialogue System

How to incorporate task related knowledge?

Page 38: Dialogue - cs.princeton.edu

Domain-Specific Knowledge: Ontology / Frame / Slot / Value

Also defines the values that each slot can take.

Slot1……

Slot2……

Slot3……

Slot4 ……

Slot1Value1 Slot1Value2…

Task-Oriented Dialogue System

How to incorporate task related knowledge?

Page 39: Dialogue - cs.princeton.edu

How to incorporate task related knowledge?

Slot1……

Slot2……

Slot3……

Slot4 ……

Slot1Value1 Slot1Value2…

DATE MONTH NAME DAY (BOUNDED-INTEGER 1 31) YEAR INTEGER

Domain-Specific Knowledge: Ontology / Frame / Slot / Value

Task-Oriented Dialogue System

Page 40: Dialogue - cs.princeton.edu

“Show me morning flights from Boston to San Francisco on Tuesday”

Task-Oriented Dialogue System

How to incorporate task related knowledge?

Page 41: Dialogue - cs.princeton.edu

Step#1: domain classification

“Show me morning flights from Boston to San Francisco on Tuesday”

Task-Oriented Dialogue System

How to incorporate task related knowledge?

DOMAIN: AIR-TRAVEL

Page 42: Dialogue - cs.princeton.edu

Step#1: domain classification

Step#2: intent determination

“Show me morning flights from Boston to San Francisco on Tuesday”

Task-Oriented Dialogue System

How to incorporate task related knowledge?

DOMAIN: AIR-TRAVEL

INTENT: SHOW-FLIGHTS

Page 43: Dialogue - cs.princeton.edu

Step#1: domain classification

Step#2: intent determination

Step#3: slot filling

“Show me morning flights from Boston to San Francisco on Tuesday”

DOMAIN: AIR-TRAVEL

INTENT: SHOW-FLIGHTS

ORIGIN-CITY: Boston ORIGIN-DATE: Tuesday ORIGIN-TIME: morning DEST-CITY: San Francisco

Task-Oriented Dialogue System

How to incorporate task related knowledge?

Page 44: Dialogue - cs.princeton.edu

Task-Oriented Dialogue System

How to incorporate task related knowledge?

A sample dialogue from the HIS System of Young et al. (2010) using the dialog acts

Page 45: Dialogue - cs.princeton.edu

Architecture of Task-Oriented SDS

Page 46: Dialogue - cs.princeton.edu

Architecture of Task-Oriented SDS

NLU component: to extract slot fillers from the user’s utterance

Page 47: Dialogue - cs.princeton.edu

Architecture of Task-Oriented SDS

Dialogue state tracker: maintains the current state of

the dialogue (most recent dialogue act + agenda)

Page 48: Dialogue - cs.princeton.edu

Architecture of Task-Oriented SDS

Dialogue policy: decides what the system should do or say The

topic of next (intent-level)

Page 49: Dialogue - cs.princeton.edu

Task-Oriented Dialogue System

• Goal: • get information from the user to help complete the specific

task. • Domain-Specific Knowledge:

• Ontology / Frame / Slot / Value • Slot Filling and Dialogue State Tracking

• Architecture: • ASR / SLU / DST / Dialogue Policy / NLG / TTS

• Evaluation: • Task completion success (slot error rate / task error rate) • Efficiency cost (#turns) • Quality cost (more comprehensive)

Page 50: Dialogue - cs.princeton.edu

Group Discussion!

Information Retrieval

Question Answering

ChatbotTask-Oriented Dialog System

What are their differences?

Chatbot v.s. Task-Oriented Dialog System

Page 51: Dialogue - cs.princeton.edu

Group Discussion!

structured

Information Retrieval

Question Answering

ChatbotTask-Oriented Dialog System

Input unstructured unstructured unstructured

Chatbot v.s. Task-Oriented Dialog System

Page 52: Dialogue - cs.princeton.edu

Group Discussion!

structured

Information Retrieval

Question Answering

ChatbotTask-Oriented Dialog System

Input unstructured unstructured unstructured

single-round Interaction single-round multi-round multi-round

Chatbot v.s. Task-Oriented Dialog System

Page 53: Dialogue - cs.princeton.edu

Group Discussion!

structured

Information Retrieval

Question Answering

ChatbotTask-Oriented Dialog System

Input unstructured unstructured unstructured

single-round Interaction single-round multi-round multi-round

availablesupervision available sparse, delayed sparse, delayed

Chatbot v.s. Task-Oriented Dialog System

Page 54: Dialogue - cs.princeton.edu

Group Discussion!

structured

Information Retrieval

Question Answering

ChatbotTask-Oriented Dialog System

Input unstructured unstructured unstructured

single-round Interaction single-round multi-round multi-round

availablesupervision available sparse, delayed sparse, delayed

dataset synthesis, collected collected collected wizard-of-oz

Chatbot v.s. Task-Oriented Dialog System

Page 55: Dialogue - cs.princeton.edu

Group Discussion!

structured

Information Retrieval

Question Answering

ChatbotTask-Oriented Dialog System

Input unstructured unstructured unstructured

single-round Interaction single-round multi-round multi-round

availablesupervision available sparse, delayed sparse, delayed

dataset synthesis, collected collected collected wizard-of-oz

Chatbot v.s. Task-Oriented Dialog System

Page 56: Dialogue - cs.princeton.edu

Group Discussion!

structured

Information Retrieval

Question Answering

ChatbotTask-Oriented Dialog System

Input unstructured unstructured unstructured

single-round Interaction single-round multi-round multi-round

availablesupervision available sparse, delayed sparse, delayed

dataset synthesis, collected collected collected wizard-or-oz

Chatbot v.s. Task-Oriented Dialog System

Page 57: Dialogue - cs.princeton.edu

Overview

• What’s a dialogue system? • Properties of Human Conversation • Chatbots v.s. Task-oriented dialogues systems • Rule-based v.s. Data-driven • Remaining Challenges

Dialogue Systems

Page 58: Dialogue - cs.princeton.edu

Rule-based system v.s. Data-driven system

How to build a task-oriented dialog system?

Rule-Based v.s. Data-Driven

Page 59: Dialogue - cs.princeton.edu

How to build a task-oriented dialog system?

Semantic grammars can be parsed by any Context-Free Grammar parsing algorithm.

Rule-based system (SLU/DST)

Rule-Based v.s. Data-Driven

Page 60: Dialogue - cs.princeton.edu

How to build a task-oriented dialog system?

A simple finite-state automaton architecture for frame-based dialog.

Rule-based system (Dialog Policy)

Rule-Based v.s. Data-Driven

Page 61: Dialogue - cs.princeton.edu

Data-driven system (SLU/DST)

How to build a task-oriented dialog system?

An LSTM architecture for slot filling, mapping the words in the input to a series of IOB tags plus a final state consisting of a domain concatenated with an intent.

Rule-Based v.s. Data-Driven

Page 62: Dialogue - cs.princeton.edu

Data-driven system (Dialog Policy)

How to build a task-oriented dialog system?

Rule-Based v.s. Data-Driven

Page 63: Dialogue - cs.princeton.edu

Data-driven system (Dialog Policy)

How to build a task-oriented dialog system?

simulator?

Rule-Based v.s. Data-Driven

Page 64: Dialogue - cs.princeton.edu

Rule-based v.s. Data-driven Pros & cons?

How to build a task-oriented dialog system?

Rule-Based v.s. Data-Driven

Page 65: Dialogue - cs.princeton.edu

Rule-based v.s. Data-driven Pros & cons?

How to build a task-oriented dialog system?

Rule-Based Methods

- hand-craft rules, “safe” but not “flexible”. - cheap in terms of dataset. - expensive in terms of engineering.

Data-Driven Methods

- learn from interactions, dialogue manager is evolvable. - uncontrolled behavior in unseen situation. - cheap in terms of engineering, but expensive in terms of data/interaction

Rule-Based v.s. Data-Driven

Page 66: Dialogue - cs.princeton.edu

Overview

• What’s a dialogue system? • Properties of Human Conversation • Chatbots v.s. Task-oriented dialogues systems • Rule-based v.s. Data-driven • Remaining Challenges

Dialogue Systems

Page 67: Dialogue - cs.princeton.edu

Challenges

Understanding the Context

Two sets of interactions with Siri in 2014.

Page 68: Dialogue - cs.princeton.edu

Challenges

The same follow-up questions that Siri couldn’t answer in 2014 receive appropriate responses when posed to Siri in 2017.

Understanding the Context

Page 69: Dialogue - cs.princeton.edu

Understanding the Context

Challenges

Uncertainty / Ambiguity

Data/Interaction Scarcity

Domain Adaptation

Reward Design

Knowledge Embedding

Page 70: Dialogue - cs.princeton.edu

Next Tuesday

Putting together the threads of conversational AI

Maybe we don't have enough threads yet to knit together the whole, but let's try anyway! We present our view of what is necessary for conversational AI, and the pieces we have worked on so far to get there. In particular: software (ParlAI, a unified platform for dialogue research), various neural architectures for memory, reasoning, retrieval and generation, and interactive learning, tasks for employing personality (Persona-Chat), knowledge (Wizard of Wikipedia) and perception (Image-Chat), evaluation studies & techniques (Acute-Eval), and a recent competition (ConvAI2). We'll try not to be too pessimistic (!), but we'll also discuss how far we still have to go.

Jason Weston’s guest lecture: