neural text generation from structured data with ...€¦ · neural text generation from structured...
TRANSCRIPT
![Page 1: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/1.jpg)
Neural Text Generation from Structured Data with Application to the Biography Domain
Remi Lebret ´ David Grangier Michael Auli
EPFL, Switzerland Facebook AI Research Facebook AI Research
(EMNLP)http://aclweb.org/anthology/D/D16/D16-1128.pdf
Presenter : Abhinav Kohar (aa18)March 29, 2018
![Page 2: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/2.jpg)
Outline
•Task•Approach / Model•Evaluation•Conclusion
![Page 3: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/3.jpg)
Task: Biography Generation (Concept-to-text Generation)• Input (Fact table/Infobox) Output (Biography)
![Page 4: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/4.jpg)
Task: Biography Generation (Concept-to-text Generation)• Input (Fact table / Infobox) Output (Biography)
• Characteristics of the work:• Using word and field embeddings along with NLM• Scale to large # of words and fields (350 words -> 400k words)• Flexibility (does not restrict relations between field and generated
text)
![Page 5: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/5.jpg)
Table conditioned language model
•Local and global conditioning•Copy actions
![Page 6: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/6.jpg)
Table conditioned language model
![Page 7: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/7.jpg)
Table conditioned language model
![Page 8: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/8.jpg)
![Page 9: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/9.jpg)
![Page 10: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/10.jpg)
![Page 11: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/11.jpg)
![Page 12: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/12.jpg)
Motivation Zct -Allows model to encode field specific regularityeg: Number of date field is followed by month , Last token of name field followed by “(” or “was born”
![Page 13: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/13.jpg)
Why Gf, Gw: fields impacts structure of generation eg: politician/athleteActual token helps distinguish eg: hockey player/basketball player
![Page 14: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/14.jpg)
Local conditioning : context dependentGlobal conditioning : context independent
![Page 15: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/15.jpg)
Copy ActionsModel can copy infobox’s actual words to the outputW: Vocabulary words , Q: All tokens in tableEg: If “Doe” is not in W, Doe will be included in Q as “name_2”
![Page 16: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/16.jpg)
Model
• Table conditioned language model• Local conditioning• Global conditioning• Copy actions
![Page 17: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/17.jpg)
![Page 18: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/18.jpg)
![Page 19: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/19.jpg)
![Page 20: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/20.jpg)
Training
• The neural language model is trained to minimize the negative log-likelihood of a training sentence s with stochastic gradient descent (SGD; LeCun et al. 2012) :
![Page 21: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/21.jpg)
Evaluation
•Dataset and baseline•Result•Quantitative Analysis
![Page 22: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/22.jpg)
Dataset and Baseline• Biography Dataset : WIKIBIO
• 728,321 articles from English Wikipedia• Extract first “biography” sentence from each article + article infobox
• Baseline• Interpolated Kneser-Ney (KN) model
• Replace word occurring in both table/sent with special tokens• Decoder emits words from regular vocab or special tokens (replace special tokens with
corresponding words from table)
![Page 23: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/23.jpg)
Template KN model
• The introduction section of the table in input (shown earlier):
• “name 1 name 2 ( birthdate 1 birthdate 2 birthdate 3 – deathdate 1 deathdate 2 deathdate 3 ) was an english linguist , fields 3 pathologist , fields 10 scientist , mathematician , mystic and mycologist .”
![Page 24: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/24.jpg)
Experimental results: Metrics
![Page 25: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/25.jpg)
Experimental results: Attention mechanism
![Page 26: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/26.jpg)
Quantitative analysis
• Local only cannot predict right occupation• Global (field) helps to understand he was a scientist• Global (field,word) can infer the correct occupation
• Date issue?
![Page 27: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/27.jpg)
• Conclusion: • Generate fluent descriptions of arbitrary people based on
structured data• Local and Global conditioning improves model by large margin• Model outperforms KN language model by 15 BLEU• Order of magnitude more data and bigger vocab
• Thoughts:• Generation of longer biographies• Improving encoding of field values/embeddings • Better loss function • Better strategy for evaluation of factual accuracy
![Page 28: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/28.jpg)
References:
• http://aclweb.org/anthology/D/D16/D16-1128.pdf• http://ofir.io/Neural-Language-Modeling-From-Scratch/• http://www.wildml.com/2016/01/attention-and-memory-in-deep-
learning-and-nlp/• https://github.com/odashi/mteval• http://cs.brown.edu/courses/cs146/assets/files/langmod.pdf• https://cs.stanford.edu/~angeli/papers/2010-emnlp-generation.pdf
![Page 29: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/29.jpg)
Questions?
![Page 30: Neural Text Generation from Structured Data with ...€¦ · Neural Text Generation from Structured Data with Application to the Biography Domain Remi Lebret ´ David Grangier Michael](https://reader033.vdocument.in/reader033/viewer/2022052012/6028006de59d60492b17aa49/html5/thumbnails/30.jpg)
Performance : Sentence decoding