Unified Pragmatic Models for Generating and Following Instructions

Daniel Fried, Jacob Andreas, and Dan KleinUC Berkeley

1

Interpreting instructions

Instruction walk along the blue carpet and you pass two objects

2

Interpreting instructions

Instruction walk along the blue carpet and you pass two objects

3

✔

Interpreting instructions

Instruction walk along the blue carpet and you pass two objects

4

✗

Interpreting instructions

Instruction walk along the blue carpet and you pass two objects

5

✗

Interpreting instructions

Instruction walk along the blue carpet and you pass two objects

6

✔

Listener: reasoning about routes

7

walk along the blue carpet and you pass

two objects

?walk along the blue carpet and you pass

two objects

Listener: reasoning about routes

8

walk along the blue carpet and you pass

two objects

walk along the blue carpet and you pass

two objects

Listener: reasoning about routes

9

walk along the blue carpet and you pass

two objects

walk along the blue carpet and you pass

two objects

Listener: reasoning about routes

walk along the blue carpet and you pass

two objects 10

walk along the blue carpet and you pass

two objects

Generating instructions

11

✔

GeneratedInstruction:

go forward four segments to the intersection with the bare concrete hall

Speaker: reasoning about interpretation

12

?

Speaker: reasoning about interpretation

13

go forward four segments to the

intersection with the bare concrete hall

Explicit pragmatic reasoning

14

Interpretation

Frank and Goodman, 2012; Goodman and Stühlmuller, 2013;

Wang et al., 2016

Generation

Golland et al., 2010; Monroe and Potts, 2015; Andreas and Klein, 2016;

Mao et al., 2016; …

“blue”

Monroe et al. 2017Our work

Both

“The sun is in the sky”

Explicit pragmatic reasoning

14

Interpretation

Frank and Goodman, 2012; Goodman and Stühlmuller, 2013;

Wang et al., 2016

Generation

Golland et al., 2010; Monroe and Potts, 2015; Andreas and Klein, 2016;

Mao et al., 2016; …

“blue”

Monroe et al. 2017Our work

BothLarge, structured

domains

“The sun is in the sky”

Learned models of listeners and speakers

BaseListener

walk along the blue carpet and

you pass two objects

Both models trained independently on action sequences annotated with human instructions

15

BaseSpeaker

walk along the blue carpet and

you pass two objects

16

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

Pragmatic Listener

16

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

Pragmatic Listener

?walk along the blue carpet and you pass

two objects

16

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

Pragmatic Listener

proposes

BaseListener ?

walk along the blue carpet and you pass

two objects

16

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener ?

walk along the blue carpet and you pass

two objects

16

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

walk along the blue carpet and you pass

two objects

16

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

rescores

walk along the blue carpet and you pass

two objects

16

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

rescores

0.1 walk along the blue carpet and you pass

two objects

16

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

rescores

0.1

0.1 walk along the blue carpet and you pass

two objects

17

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

rescores

0.1

walk along the blue carpet and you pass

two objects

17

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

rescores

0.1

walk along the blue carpet and you pass

two objects

17

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

rescores

0.1

0.2 walk along the blue carpet and you pass

two objects

17

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

rescores

0.1

0.2

0.2

walk along the blue carpet and you pass

two objects

18

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

rescores

0.1

0.2

walk along the blue carpet and you pass

two objects

18

walk along the blue carpet and you pass

two objects

Building a pragmatic listener

BaseSpeaker

proposes

BaseListener

rescores

0.1

0.2

walk along the blue carpet and you pass

two objects

Unified pragmatic inference

Pragmatic Listener

19

BaseSpeaker

BaseListener ?

Pragmatic Speaker

BaseSpeaker

BaseListener?

actions

instructions

walk along the blue carpet …

walk along the blue carpet …

proposes

Building a pragmatic speaker

BaseSpeaker

20

proposes

Building a pragmatic speaker

BaseSpeaker

walk forward past the stool

go forward four segments to the intersection with the bare concrete

hall

20

proposes rescores

Building a pragmatic speaker

BaseSpeaker

BaseListener

walk forward past the stool

go forward four segments to the intersection with the bare concrete

hall

20

proposes rescores

Building a pragmatic speaker

BaseSpeaker

BaseListener

walk forward past the stool

go forward four segments to the intersection with the bare concrete

hall

20

0.4

proposes rescores

Building a pragmatic speaker

BaseSpeaker

BaseListener

walk forward past the stool

go forward four segments to the intersection with the bare concrete

hall

0.4

20

0.4

proposes rescores

Building a pragmatic speaker

BaseSpeaker

BaseListener

walk forward past the stool

go forward four segments to the intersection with the bare concrete

hall

0.4

21

0.8

proposes rescores

Building a pragmatic speaker

BaseSpeaker

BaseListener

walk forward past the stool

go forward four segments to the intersection with the bare concrete

hall

0.4

21

0.8

0.8

proposes rescores

Building a pragmatic speaker

BaseSpeaker

BaseListener

walk forward past the stool

go forward four segments to the intersection with the bare concrete

hall

0.4

22

0.8

Base model implementations

……

+

LSTM Encoder

BaseListener

[Mei et al., 2016]

walk along the blue carpet …

……

LSTM Decoder with Attention

23

Base model implementationsBase

Speaker

24

+

LSTM Encoder……

LSTM Decoder with Attention……

walk

walk along the blue

along the blue

…

…

Listener tasks

Listenerwalk along the blue carpet and

you pass two objects

SAIL navigation [MacMahon et al., 2006; Chen and Mooney, 2011]

✔

Sequential Context-dependent Execution (SCONE) [Long et al. 2016]

1. a red guy appears on the far left2. then to orange’s other side

Listener

✔

25

Listener results, SAIL

59.6

64.4 65.3

50

60

70Model accuracy at following human instructions

Artzi & Zettlemoyer

26

Base listener

Pragmatic listener

Instruction walk along the blue carpet and you pass two objects

BaseListener ✗

Pragmatic Listener ✔

Listener example, SAIL

27

Listener results, SCONE

69.7 70.9 69.672 72.7

69.6

50

60

70

80

Alchemy Scene Tangrams

Model accuracy at following human instructionsBase listener Pragmatic listener

28

Listener example, SCONE

a red guy appears on the far leftthen to orange’s other side

Instruction

Pragmatic Listener

✔Base

Listener

✗

29

Speaker task and evaluation

Speakerwalk along the

wood path to the chair

walk along the wood path to the

chair

Human directionfollowers (MTurk)

Humans try to interpret it

Speaker produces an instruction

30

Speaker results

62.8

29.3 31.3

60.0

75.2 75.369.3

88.0

0

25

50

75

100

SAIL Alchemy Scene Tangrams

Human accuracy at following instructions from:

31

83.378.0

66.073.2

Base speaker Pragmatic speakerOther humans

Speaker example, SCONE

BaseSpeaker

remove the last figureadd it back

✗

Pragmatic Speaker

remove the last figureadd it back in the 3rd position

✔

Humantake away the last itemundo the last step

✗32

Real-world navigation

human description:walk through the kitchen. go right into the living room and stop by the rug.

base speaker:walk past the dining room table and chairs and wait there .

pragmatic speaker:walk past the dining room table and chairs and take a right into the living room . stop once you are on the rug .

Matterport3D vision-and-language navigation dataset [Anderson et al., 2018]

33

Conclusions

Unified inference for sequential interpretation and generation

Reasoning counterfactually, and about likely interpretations

Pragmatics helps for complex tasks in structured domains

34

Speaker

Listener

Thanks!http://github.com/dpfried/pragmatic-instructions

35