neural symbolic machines - ni laoneural symbolic machines neural program induction for semantic...

Neural Symbolic MachinesNeural Program Induction for Semantic Parsing

Chen Liang, Ni LaoCollaborate: Jonathan Berant, Quoc Le, Kenneth Forbus

Overview● Semantic parsing: (updated) WebQuestions dataset

● Neural program induction

● Manager-Programmer-Computer (MPC) framework

● Neural Symbolic Machine

● Experiments and analysis

Semantic parsing

Semantic Parsing

Knowledge Base

Natural language

Logical form /Program

Desired behavior / Answer

[Berant, et al 2013][Liang 2013]

● Question answering with structured data (KG / tables / personal data)

● Voice to action● Personal assistant

Full supervision (hard to collect)

Weak supervision (easy to collect)

Large-scale Knowledge BaseProperties of Hundreds of millions of entitiesPlus relations among themE.g. Freebase, 26k predicates, 200M entities, 3B triples

Question Answering“What are the names of Obama’s daughters?”x.parent(Obama,x)∩gender(x,Female)

Question Answering with Knowledge Base

FreebaseDBpedia

YAGONELL

OpenIE/ReVerbMicrosoft Satori

Slides adapted from [Yih+ 2016]

WebQuestions Dataset • What character did Natalie Portman play in Star Wars? ⇒ Padme Amidala• What currency do you use in Costa Rica? ⇒ Costa Rican colon• What did Obama study in school? ⇒ political science• What do Michelle Obama do for a living? ⇒ writer, lawyer• What killed Sammy Davis Jr? ⇒ throat cancer

• 5,810 questions crawled from Google Suggest API and answered using Amazon MTurk• 3,778 training, 2,032 testing• A question may have multiple answers using Avg. F1 as main evaluation metric

[Examples from Berant]

Slides adapted from [Yih+ 2016]

[Berant+ 13]

WebQuestionsSP Dataset • Remove invalid QA pairs• Add logical form annotations• Result in 3,098 training and

1,639 testing questions

Available at http://aka.ms/WebQSP

"Version": "1.0", "FreebaseVersion": "2015-08-09", "Questions": [ { "RawQuestion": "what does jamaican people speak?", "Parses": [ { "ParseId": "WebQTest-0.P0", "Sparql": " ... ", "TopicEntityName": "Jamaica", "TopicEntityMid": "m.03_r3", "InferentialChain": ["location.country.languages_spoken" ], "Answers": [ { "AnswerArgument": "m.01428y", }, { "AnswerArgument": "m.04ygk0", } ] }, { "ParseId": "WebQTest-0.P1", "Sparql": "...", ... "InferentialChain": ["location.country.official_language" ], ....

[Yih+ 16]

Generic Semantic Parsing

Knowledge Base

Who is Justin Bieber’s sister?

semantic parsing

querymatching

Jazmyn Bieber

[Kwiatkowski+ 13]

Slides adapted from [Yih+ 2016]

KB-Specific Semantic Parsing

Knowledge Base

Who is Justin Bieber’s sister?

semantic parsingquery

Jazmyn Bieber

[Berant+ 13]

Slides adapted from [Yih+ 2016]

• Language mismatch• Lots of ways to ask the same question

“What was the date that Minnesota became a state?”“When was the state Minnesota created?”

• Need to map them to the predicate defined in KBlocation.dated_location.date_founded

• Compositionality• The semantics of a question may involve multiple predicates and entities

• Large search space• Some Freebase entities have >160,000 immediate neighbors• 26k predicates in Freebase

Key Challenges

Slides from [Yih+ 2016]

Previous State-of-the-Art

topic entity core inferential chain

constraints

[Yih+ 2016]

Staged Query Graph Generation

[Yih, et al 2016]

Previous State-of-the-art

● 6 rules to determine if a constraint or aggregation should be added to cvt nodes

● 3 extra rules for answer nodes● 7 features for cvt nodes● 3 features for answer nodes● 3 CNN matching scores

[Yih, et al 2016]

Hand crafted rules and features

Pure seq2seq approaches to semantic parsing● Require strong / full supervision

○ Only applied on domains with tiny schemas/vocabs and known semantic annotations: JOBS, GEO, ATIS, IFTTT

● Lack of integration with the executer○ Only after the whole program is written can any feedback be generated from the machine○ No way to search over the exponentially large hypothesis space given a large schema

(e.g. Freebase has 26k predicates, 100M entities)○ The entire burden is on the neural network

[Jia & Liang, 2016; Dong & Lapata, 2016]

knowledge base

answer

execution

Overview● Semantic parsing: (updated) WebQuestions dataset

● Neural program induction

● Manager-Programmer-Computer (MPC) framework

● Neural Symbolic Machine

● Experiments and analysis

Program induction & Memory● Differentiable function => computable function

○ Use differentiable memories for backprop training

● Tasks○ Mostly simple tasks like addition or sorting○ Hard to scale to large knowledge bases

● A lot of recent progress○ Neural Turing machine [Graves+ 2014]○ Memory network [Sukhbaatar+ 2015]○ Dynamic memory network [Kumar+ 2015]○ Stack-augmented RNN [Joulin&Mikolov 2015]○ Queue & stack LSTM [Grefenstette+ 2015]○ Neural Programmer Interpreter [Reed&Freitas 2015]○ Neural programmer [Neelakantan+ 2015]○ Differentiable neural computer [Graves+ 2016]○ Dynamic neural module network [Andreas+ 2016]○ Dynamic Neural Turing Machine [Gulcehre+ 2016]

How about giving NN regular operations and memory?

[Reed&Freitas 2015]

● The operations learned are not as scalable and reliable.

● Why not leverage existing modules which are scalable and reliable?

● Impressive example to demonstrate the power but...

[Zaremba&Sutskever 2016]

Reinforcement Learning Neural Turing Machines● Interact with a discrete Interfaces

○ a memory Tape, an input Tape, and an output Tape

● Use Reinforcement Learning algorithm to train ● Solve simple algorithmic tasks

○ E.g., reversing a string

[Zaremba&Sutskever 2016]

Need higher level programming language

for semantic parsing

Overview● Semantic parsing: (updated) WebQuestions dataset

● Neural program induction

● Manager-Programmer-Computer (MPC) framework

● Neural Symbolic Machine

● Experiments and analysis

Symbolic Machines in Brains

● 2014 Nobel Prize in Physiology or Medicine awarded for ‘inner GPS’ research

● Positions are represented as discrete numbers in animals' brains, which enable accurate and autonomous calculations

[Stensola+ 2012]

Brain Symbolic ModulesEnvironment

Mean grid spacing for all modules (M1–M4) in all animals (colour-coded)

Programmer ComputerManager

program

code assist

question & reward

answer built-in functions

The MPC FrameworkWeak

supervisionNeural Symbolic

knowledge base(built-in data)

Non-differentiableAbstractScalablePrecise

Neural Computer Interface

Overview● Semantic parsing: (updated) WebQuestions dataset

● Neural program induction

● Manager-Programmer-Computer (MPC) framework

● Neural Symbolic Machine

● Experiments and analysis

Knowledge Base & Semantic Parsing

● Knowledge graph○ Let E denote a set of entities (e.g., ABELINCOLN), and ○ let P denote a set of relations (or properties, e.g., PLACEOFBIRTH)○ A knowledge base K is a set of assertions or triples (e1, p, e2) ∈ E × P × E

e.g., (ABELINCOLN, PLACEOFBIRTH, HODGENVILLE)

● Semantic parsing○ given a knowledge base K, and a question q = (w1 , w2 , ..., wk ), ○ produce a program or logical form z that

when executed against K generates the right answer y

● Predefined functions, equivalent to a subset of λ-calculus○ A program C is a list of expressions (c1...cl)○ An expression is either a special token "Return" or a list "( F A0 ... Ak )"○ F is one of the functions

○ An argument Ai can be either a relation p ∈ P or a variable v○ A variable v is a special token (e.g. "R1") representing a list of entities

Lisp: High-level Language with Uniform Syntax

Entities

US

Obama

…...

Functions

Hop

ArgMax

…...

Relations

CityInCountry

BeersFrom

…...

US

Hop

!CityIn Argmax

Population

ReturnOutput: NYC

Question: Largest city in US

(define v0 US)(define v1 (Hop v0 ?CityIn))(define v2 (Argmax v1 Population))(return v2)

Lisp Code

Program as a sequence of tokens

Knowledge Graph

Seq2Seq

Hop......

CityInCountry…...

R1…...

GO…...

wholargest…...

Merged Decoder Vocab

Functions

WordsPredicates

Variables

Encoder Vocab

Specials

Semantic Parsing with NSM

Key Variable

v1 R1(m.USA)Execute( Argmax R2 Population )

ExecuteReturn

m.NYCKey Variable

... ...

v3 R3(m.NYC)

Key Variable

v1 R1(m.USA)

v2 R2(list of US cities)

Execute( Hop R1 !CityIn )

Hop R1 !CityIn( )

Largest city ( Hop R1in US GO !CityIn Argmax R2( )Population)

R2 Population ReturnArgmax )(

Entity Resolver

● Add a key-variable memory to Seq2Seq model for compositionality

● The 'keys' are the output of GRUs● The 'variables' are just symbols

referencing results in computer: 'R1', 'R2'

Largest city GO

…...

Hop R1 !CityIn

Hop R1 !CityIn Argmax

Key-Variable Memory

Comments Variable

Entity extracted from the word after “city in”

R1(m.USA)

Generated by querying v1 with !CityIn

R2(a list of US cities)

● Human use names/comments to index intermediate results

Embeddings Variable

[0.1, -0.2, 0.3, …] R1(m.USA)

[0.8, 0.5, -0.3, …] R2(a list of US cities)

● The memory is 'symbolic'○ Variables are symbols referencing intermediate results in computer○ No need to have embeddings for hundreds of millions of entities in KG○ Keys are differentiable, but variables are not

● NN use embeddings (outputs of GRUs) to index results

Model Architecture

● Small model: 15k+30k+15k*2+5k = 80k params● Dot product attention● Pretrained embeddings● Dropout (a lot)

Encoder

a0 a1 an

SoftMax

q0 q1 qm

dot product

Linear Projection 300*50=15k

Linear Projection 600*50=30k

GRU 2*50*3*50=15k

Attention

Decoder

GRU 2*50*3*50=15k

Linear Projection 100*50=5k

dropout

dropout

dropout

dropout

dropout

GloVe GloVe

Give the NN a better coding interface

Code Assist

Syntax check:Only a variable can follow ‘Hop’

Semantic check:only relations that are connected to R1 can be used ~ 20k => ~ 100

● A Strong IDE / Interpreter helps reduce the search space○ Exclude the invalid choices that will cause syntax and semantic error

Largest city

GO

…...

Hop R1

Hop R1 !CityIn

Neural Computer Interface

Implemented as a changing mask on

decoding vocabulary

Non-differentiable => REINFORCE Training

● Optimizing expected F1

● Use baseline B(q) to reduces variance without changing the optima

● Gradient computation is approximated by beam search instead of sampling

Sampling v.s. Beam search

● Decoding uses beam search○ Use top k in beam ( normalized probabilities) to compute gradients○ Reduce variance and estimate the baseline better

● The coding environment is deterministic. Closer to a maze than Atari game.

Stochastic Deterministic

Problem with REINFORCE

Training is slow and get stuck on local optimum● Large search space

○ model probability of good programs with non-zero F1 is very small

● Large beam size○ Normalized probability small○ Decoding and training is slow because larger

number of sequences● Small beam size

○ Good programs might fall off the beam

Solution:Add some gold programs into the beam with reasonably large probability... but we don’t have gold programs, only weak supervision

● Ideally we want to do supervised pretraining for REINFORCE, but we only have weak supervision

● Use an iterative process interleaving decoding with large beam and maximum likelihood training

● Training objective:

● Training is fast and has a bootstrap effect

Finding Approximate Gold Programs

Drawbacks of the ML objective● Not directly optimizing expected F1

● The best program for a question could be a spurious program that accidentally produced the correct answer, and thus does not generalize to other questions

○ e.g., answering PLACEOFBIRTH with PLACEOFDEATH

● Because training lacks explicit negative examples, the model fails to distinguish between tokens that are related to one another

○ e.g., PARENTSOF vs. SIBLINGSOF vs. CHILDRENOF

Augmented REINFORCE● Add the approximate gold program into the final beam with

probability α, and the probabilities of the original programs in the beam are normalized to be (1 − α).

● The rest of the process is the same as in standard REINFORCE

Top k in beam

Approximate gold programs

(1 − α)

α

REINFORCE

Overview● Semantic parsing: (updated) WebQuestions dataset

● Neural program induction

● Manager-Programmer-Computer (MPC) framework

● Neural Symbolic Machine

● Experiments and analysis

Freebase Preprocessing● Remove predicates which are not related to world knowledge

○ Those starting with "/common/", "/type/", "/freebase/"

● Remove all text valued predicates○ They are almost never the answer of questions

● Result in a graph which is small enough to fit in memory○ #Relations=23K○ #Nodes=82M○ #Edges=417M

KG server 1

KG server m

…...

Decoder 1

Decoder 2

Decoder n

…...Trainer

QA pairs 1

QA pairs 2

QA pairs n

Solutions 1

Solutions 2

Solutions n

…...…...

Model checkpoint

System Architecture● 200 decoders, 50 KG servers, 1 trainer, 251 machines in total● Since the network is small, we didn’t see much speedup from GPU

Compare to State-of-the-Art

● First end-to-end neural network to achieve state-of-the-art performance on semantic parsing with weak supervision over large knowledge base

● The performance is approaching state-of-the-art result with full supervision

Augmented REINFORCE

● REINFORCE get stuck at local maxima● Iterative ML training is not directly optimizing the F1 measure● Augmented REINFORCE obtains the best performances

● Gradually increasing the program complexity during ML training○ First run iterative ML training with only the "Hop" function and the

maximum number of expressions is 2○ Then run iterative ML training again with all functions, and the maximum

number of expressions is 3. The relations used by the "Hop" function are restricted to those that appeared in the best programs from in first one

● A lot of search failures without curriculum learning

Curriculum LearningInspired by STAGG [Yih, et al 2016]

Reduce Overfitting

● With all these techniques the model is still overfitting○ Training F1@1 = 83.0%○ Validation F1@1 = 67.2%

Future work

● Better performance with more training data

● Actions to add knowledge into KG and create new schema

● Language to action

Programmer ComputerManager

inputs&code

outputs

question

answer

knowledge base

(built-in data)

Short-term

Long-term

Programmer Computer

New functions

built-in functions

New knowledge

Acknowledgement

● Thanks for discussions and helps from Arvind, Mohammad, Tom, Eugene, Lukasz, Thomas, Yonghui, Zhifeng, Alexandre, John

● Thanks for MSR researchers, who made WebQuestionSP data set available