learning plans from english procedure...

Dustin [email protected]

MIT Media Lab

Learning Plansfrom English

Procedure Descriptions

mailto:[email protected]

mailto:[email protected]

Machine Reading

- How can we get knowledge into computers?

- Machine reading: processing and understanding natural language-- accumulating relevant knowledge through inferences about text.

2

Approach 1: Read Children’s Stories

3

Jack was having a birthday party. Mother baked a cake.


- What is the relationship between Jack and Mother?

- Who will eat the cake?

- Where was Mother when she was baking the cake?

- Where was Jack?

- ... 3


Eugene Charniak’s Ph.D. [1] “the chief concern motivating the model discussed here is relating a large body of knowledge to a particular story”

By preschool, children can represent and remember event sequences [2].

By the time they are reading, children have acquired a lot of world knowledge already!

4

[1] E. Charniak “Toward a Model of Children’s Story Comprehension.” 1972.[2] J. Wenner, P.J. Bauer. Bringing order to the arbitrary. 1999.


Big QuestionCharacterizing the problem:

How can we acquire knowledge by reading, when reading itself requires knowledge?

5

Big QuestionCharacterizing the problem:

How can we acquire knowledge by reading, when reading itself requires knowledge?

5

- Bootstrap solution: Gradually compose more new (more complex) representations out of existing (simpler) representations.

6

Approach 2: Use Commonsense KB


is a gameis a children’s gameis a nickname for John.is a boy’s nameis a childhood game.is a lifting device.is kind of nickname.

(40)

Use OpenMind Common Sense knowledge base [1].

[1] http://commons.media.mit.edu

http://commons.media.mit.edu

http://commons.media.mit.edu

7




(40)

bake a cake because you want toballoon used forlikely to find at toy balloonlikely to find at helium balloonbuying presents is forare fun

(17)

8




(40)


(17)

can care for a childis a womantake care of their childrenloves her childis part of my family

(260)

9




(40)


(17)


(260)

you should have an ovena birthday may make you want tobecause you want to celebrate a birthdayfirst thing add flour

(19)

10




(40)

bake cake because you want balloon used forlikely to find at toy balloonlikely to find at helium balloonbuying presents is forare fun

(17)


(260)

Problem: Retrieving only the relevant knowledge

you should have an ovena birthday may make you want tobecause you want to celebrate a birthdayfirst thing add flour

(19)

11

Plans: a solution?

- are also called: plans, event chains, stories, scripts, narratives, procedures, task networks, sequential decision processes, ...

- Unite procedural and declarative semantic knowledge.

- Structure what is problematic, what questions are worth asking; embedding knowledge in a goal-driven problem solving context.

"Questions arise from a point of view–from something that helps to structure what is problematical, what is worth asking, and what constitutes an answer (or progress). It is not that the view determines reality, only what we accept from reality and how we structure it...”

-- Alan Newell in Artificial Intelligence and the Concept of Mind

12

Lexical-semantics of Events

- Verbs and their arguments reference lexical-semantic frames, or event structures [1].

- Properties of verbs (TimeBank)

- Tense: {none, present, past, future}

- Grammatical Aspect: {none, progressive, perfect, progressive perfect}

- Modality: {none, to, would, should, could, can, might}

- Polarity: {positive, negative}

- Event Type: { ?? }

[1] Levin B. and Hovav, Argument Realization 2005.

12

Lexical-semantics of Events

- Verbs and their arguments reference lexical-semantic frames, or event structures [1].

- Properties of verbs (TimeBank)

- Tense: {none, present, past, future}

- Grammatical Aspect: {none, progressive, perfect, progressive perfect}

- Modality: {none, to, would, should, could, can, might}

- Polarity: {positive, negative}

- Event Type: { ?? } - Need a semantic theory!

[1] Levin B. and Hovav, Argument Realization 2005.

13

Lexical-semantics of Events- Some compelling evidence of this relationship from

cognitive science:

- Verb aspect influences retrieval of event knowledge:

- imperfective form (was verb-ing) versus perfect form (had verb-ed) in a statement (e.g. “I was cooking/ I had cooked”) changes the retrieval of the location “kitchen” [1].

- When observing the same event, speakers of different languages attend to the event features relevant to organizing their languages specific verb lexicon, e.g. path versus manner [2].

- When reading about actions, people’s corresponding brain motor regions are activated [3].

[1] T. R. Ferretti, M. Kutas, and K. Mcrae. Verb aspect and the activation of event knowledge. 2007.[2] A. Papafragou, J. Hulbert, and J. Trueswell. Does language guide event perception? ... 2008. [3] R. A. Zwaan and L. J. Taylor. Seeing, acting, understanding: motor resonance in lang. comp. 2006.

14

Acquiring Plans the Open Mind Way

- Honda’s OMICS project [1], a clone of the Media Lab effort. Collecting knowledge exclusively about indoor common sense. High quality, manually reviewed.

- Has a parallel corpus of stories about ways to accomplish 174 different goals.

[1] http://openmind.hri-us.com/

15

access the internetact as a security guardanswer the doorbellanswer the phoneapply band aidassist person standing upassist someone in walkingboil the milkbuy from vending machinecall 911calm an infantchange a baby diaperchange a bulbchange bed sheetscharge a cell phonecheck for intruderscheck for weathercheck if a store is openchop vegetablesclean a spillclean the dishesclean the floorclean the showerclean the tableclean upclean up toysclose the blindsclose the curtainscook fishcook noodlecook pastacook ricedance with the childrendo laundrydraw the curtains

dry clothesdust an objectempty the kitchen sinkempty the trashentertain childrenerase the whiteboardfeed a childfeed a pet catfeed a pet dogfeed infantfeed the fishfetch a cold drinkfetch a ladderfetch an objectfill water in containerfind a personfind an objectfind out more informationfind the timefold clothesfollow someone aroundgather all scattered toysget food from refrigeratorget mailget the newspapergive a medicinegive a messagegive a message on phonego outsidegreet a visitorguard the househandle toxic materialshang clothesheat food in microwaveheat food on kitchen gas

help someone carry thingiron clotheskeep the dog awaykick a ballload the dishwasherlock up the houselock windowsmail a lettermake a bedmake a dinner reservationmake a flight reservationmake a listmake a presentationmake a shopping listmake a tossed saladmake baby sleepmake breakfastmake coffeemake fresh orange juicemake hot dogmake soupmake sure children fedmake teamake toasted breadmaking omelettemove furnituremow the lawnopen a web pageopen packageopen the garageopen the mailpack a mailing boxpack a suitcasepaint a wallpay bills

perform research on specphotocopy a paperpick up dishesplace ladder near wallplay a game on the compplay a movieplay a songplay pianoplug an electric appliance in plug battery into chargerpour beer into a glassprint documentpush someone in a wheelpush somethingput away groceriesput object awayput up a paintingraise the blindsread a story to a childrecharge batteriesremove and replace garbagereplace a refrigerator filterreplace a water tap filterreplace batteries in the treplace heater filterretrieve a toolsecure all exitssecure all windowssecure the perimeter of send a faxsend party invitationsserve a drinkserve a mealset a wake up alarmset the dining table

“sweep floor”

Problems with the corpus

“sweep floor”

P1. many ways to say the same thing!


“sweep floor”

P2. Temporal Abstraction (nested events)


“sweep floor”

P3. Global Alignment(problem of context)


“sweep floor”

P4. Causal discontinuation (OR)


1. Parse English sentences: clean, parse, extract predicate-argument structures.

2. Find global alignment of stories: Read in stories one at a time, align sequences, use structure to either:

a) detect missing nodes / context (alignment)

b) detect disjunctions (abstraction / is-a)

c) detect nested sequences (composition / part-of)

3. Infer corresponding state descriptions: Construct corresponding situation models for each step.

- Evaluation: the narrative cloze.

The Approach

22

Four example parsed narratives (of 36)

23

... and related background knowledge

1. Parse English sentences: clean, parse, extract predicate-argument structures.

2. Find global alignment of stories: Read in stories one at a time, align sequences, use structure to either:

a) detect missing nodes / context (alignment)

b) detect disjunctions (abstraction / is-a)

c) detect nested sequences (composition / part-of)

3. Infer corresponding state descriptions: Construct corresponding situation models for each step.

- Evaluation: the narrative cloze.

The Approach

25

Sequence Alignment- Sequential modeling work from NLP on language

models, and bioinformatics on modeling nucleic acids in DNA/amino acids in proteins.

- Needleman and Wunsch sequence alignment algorithm. Given two strings, A and B, populate a Score matrix and use back-pointers to retrieve the best alignment.

- Gap penalty:

- Beyond 2 strings?

- Time Complexity for n=2 seq:

!OpenCost! (len(gap)! 1)GapExtensionCost

max (|A|, |B|)n

425 ! 1014

26

n=1, pre-

27

n=1, post-

28

n=2, pre-

29

n=2, post-

30

n=3discarded

31

n=4, pre-

32

n=4, post-

33

n=5 pre

34

n=5 post

35

n=7 pre

36

n=7 post

37

Needleman-Wunsch Results

38

Beyond string matching- Match(Ai, Bj) = {1,0}

- But, how well do these match?

- get(mail) get(letter)

- open(box) open(mailbox)

- close(door) shut(door)

- return(home) go(inside)

39

Beyond string matching- Match(Ai, Bj) = {1,0}

- But, how well do these match?

- get(mail) get(letter)

- open(box) open(mailbox)

- close(door) shut(door)

- return(home) go(inside)

[1] Jiang, J and Conrath, D: Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy. 1997

- Jiang & Conrath generalization similarity metric [1]

40

A more general description

L. De Raedt and J. Ramon. Deriving distance metrics from generality relations. 2008.

41

n=4, pre-

42

n=4, post-

JC similaritytext similarity

43

n=5 pre

44

n=5 post

JC similaritytext similarity

45

Needleman-Wunsch (Relational) Results

46

What about inferring hidden state?

go to the mailbox

open the mailbox

extract letter

in(mailbox, mail) at(home, Dustin) sealed(letter) closed(mailbox)

in(mailbox, mail) at(mailbox, Dustin) sealed(letter) open(mailbox)

in(hand,mail) at(mailbox, Dustin) sealed(letter) open(mailbox)

- Learning a Left-Right Hidden Markov Model or Context Free Grammar (searching a model space, inducing a model w/ some MDL type constraint)

- Good for dealing with sequential, incomplete data. Not good for relational rich data. Hard to augment background knowledge.

- Relational Markov Models? Logical Hidden Markov Models (LoHMMs)?

47

Possible Solutions

Relational Markov Models

C. Anderson, P. Domingos, and D.S. Weld. Relational Markov Models and their Application to Adaptive Web Naviation. 2002.

A hidden Markov model

and a semantic abstraction

hierarchy (a taxonomy).

Relational Markov Models

C. Anderson, P. Domingos, and D.S. Weld. Relational Markov Models and their Application to Adaptive Web Naviation. 2002.

A graph rank-based heuristic

on the state transitions is used

to generalize arguments,

in favor of a compact representation.

Problems Representational Solutions

P1. Lexical Ambiguity

P2. Temporal Abstraction

P3. Global Alignment

P4. Causal discontinuation

1. Taxonomic (Is-A) hierarchy

2. Compositional (Part-of) hierarchy

3. Sequential

An ideal representation?

Representational Abstraction

51

h3 = [shape(circle) ∧ taste(sweet) ∧ color(green)]

h0

h2h1

h5h4h3

generalization lattice

<- more general m

ore specific ->

lattice = a set, a partial ordering, and greatest upper/lower bound operations.

h0 = [shape() ∧ taste() ∧ color()]

A hypothesis space of abstractions (removing details).

Representational Composition- Bootstrap solution: Gradually compose more

new (more complex) representations out of existing (simpler) representations.

52

composition

appletaste(sweet)

color(light-green)

shape(circle) composition

Structural versus Assertional

Various encodings of the statement “John loves tacos:”

LOVES: SUBJECT: John OBJECT: tacos

Loves(John,tacos)

tacos

loves

John

Different structures, same assertions!

Structural knowledge is the internal organization of the knowledge while assertional knowledge is the set of claims it makes about the world.

Relational Sequence Alignment

[1] Much related work by Kristian Kersting and Luc De Raedt.

- Relational (in first order logic), instead of propositional, descriptions of states.

Predicates/arity: vi/2, cd/1, ls/0, pdfview/2

Ground atoms -- predicates with non-variable terms:

vi(ch2,tex)

Ground clauses:

Generalized Clauses (with variables):

cd(X), vi(Y, tex), latex(Y, tex)

Sequential

Relational Sequence Alignment

[1] Much related work by Kristian Kersting and Luc De Raedt.

learning plans from english procedure...

Documents