nlp applications using deep learning - github pages · as a result of their deep understanding,...
TRANSCRIPT
![Page 1: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/1.jpg)
NLP Applications using Deep Learning
Giri Iyengar
Cornell University
Feb 28, 2018
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 1 / 30
![Page 2: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/2.jpg)
Agenda for the day
EntailmentQuestion AnsweringNamed Entity Recognition
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 2 / 30
![Page 3: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/3.jpg)
Overview
1 Textual Entailment
2 Question Answering
3 Named Entity Recognition
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 3 / 30
![Page 4: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/4.jpg)
Deep Understanding
What is Deep Understanding?Students develop deep understanding when they grasp the relativelycomplex relationships between the central concepts of a topic or discipline.Instead of being able to recite only fragmented pieces of information, theyunderstand the topic in a relatively systematic, integrated or holistic way.As a result of their deep understanding, they can produce new knowledgeby discovering relationships, solving problems, constructing explanationsand drawing conclusions. – Dept. of Education, Queensland
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 4 / 30
![Page 5: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/5.jpg)
Deep Understanding
What is Deep Understanding?Students develop deep understanding when they grasp the relativelycomplex relationships between the central concepts of a topic or discipline.Instead of being able to recite only fragmented pieces of information, theyunderstand the topic in a relatively systematic, integrated or holistic way.As a result of their deep understanding, they can produce new knowledgeby discovering relationships, solving problems, constructing explanationsand drawing conclusions. – Dept. of Education, Queensland
That is, Deep Understanding involves Knowledge, Reasoning, Learning,and Action
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 5 / 30
![Page 6: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/6.jpg)
Textual Entailment
An example of a positive TE (text entails hypothesis) is:text: If you help the needy, God will reward you.hypothesis: Giving money to a poor man has good consequences.
An example of a negative TE (text contradicts hypothesis) is:text: If you help the needy, God will reward you.hypothesis: Giving money to a poor man has no consequences.
An example of a non-TE (text does not entail nor contradict) is:text: If you help the needy, God will reward you.hypothesis: Giving money to a poor man will make you a better person.
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 6 / 30
![Page 7: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/7.jpg)
Textual Entailment is required for many applications
Question AnsweringInformation ExtractionCreation of Knowledge Bases
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 7 / 30
![Page 8: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/8.jpg)
Textual Entailment Approaches
Build a classifier that is input [(T, H), L] sentence pairs and labels
Construct a seq2seq model to convert T to H
Construct Knowledge Bases to capture semantic information (manual,not scalable)Try to learn a latent knowledge representation
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 8 / 30
![Page 9: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/9.jpg)
Textual Entailment Approaches
Build a classifier that is input [(T, H), L] sentence pairs and labelsConstruct a seq2seq model to convert T to H
Construct Knowledge Bases to capture semantic information (manual,not scalable)Try to learn a latent knowledge representation
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 8 / 30
![Page 10: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/10.jpg)
Textual Entailment Approaches
Build a classifier that is input [(T, H), L] sentence pairs and labelsConstruct a seq2seq model to convert T to H
Construct Knowledge Bases to capture semantic information (manual,not scalable)
Try to learn a latent knowledge representation
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 8 / 30
![Page 11: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/11.jpg)
Textual Entailment Approaches
Build a classifier that is input [(T, H), L] sentence pairs and labelsConstruct a seq2seq model to convert T to H
Construct Knowledge Bases to capture semantic information (manual,not scalable)Try to learn a latent knowledge representation
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 8 / 30
![Page 12: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/12.jpg)
Textual Entailment Recognition using RBM
Parse each sentence into a parse treeRepresent each sentence by a composite representation similar toRecursive Tree ModelUse a Restricted Boltzmann Machine to jointly learn a latentrepresentation on top of these (T, H) representationsGiven a sentence pair, look at the reconstruction error and classify ifthey are entailed or not
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 9 / 30
![Page 13: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/13.jpg)
Textual Entailment Recognition using RBM
Figure: Image Source: Lyu, ICTAI 2015
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 10 / 30
![Page 14: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/14.jpg)
Textual Entailment Recognition using RBM
Figure: Image Source: Lyu, ICTAI 2015
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 11 / 30
![Page 15: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/15.jpg)
Textual Entailment Recognition using RBM
Figure: Image Source: DeepLearning4J
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 12 / 30
![Page 16: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/16.jpg)
Overview
1 Textual Entailment
2 Question Answering
3 Named Entity Recognition
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 13 / 30
![Page 17: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/17.jpg)
IBM Watson wins Jeopardy
YouTube
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 14 / 30
![Page 18: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/18.jpg)
Application of QA Systems
Dialog SystemsChatbotsIntelligent Assistants
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 15 / 30
![Page 19: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/19.jpg)
Type of QA Systems
Open - Includes General knowledge in addition to questions, whoseanswers are in the textClosed - The answers can be found completely using the Contextprovided in the text and the question
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 16 / 30
![Page 20: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/20.jpg)
Conventional NLP Approaches to QA
ParsingPart of speech taggingNamed Entity extractionEncode rules. E.g. Jeopardy category, Daily Double
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 17 / 30
![Page 21: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/21.jpg)
Deep Learning approaches to closed QA
Closed QA taskI: Jane went to the hallway.I: Mary walked to the bathroom.I: Sandra went to the garden.I: Daniel went back to the garden.I: Sandra took the milk there.Q: Where is the milk?A: gardenI: It started boring, but then it got interesting.Q: What’s the sentiment?A: positive
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 18 / 30
![Page 22: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/22.jpg)
SQuAD: Stanford Question Answering Dataset
Figure: Source - Rajpurkar 2016
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 19 / 30
![Page 23: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/23.jpg)
GRU for QA
Figure: Source - Stroh, Mathur cs224d Report
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 20 / 30
![Page 24: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/24.jpg)
Seq2Seq for QA
Figure: Source - Stroh, Mathur cs224d Report
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 21 / 30
![Page 25: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/25.jpg)
Dynamic Memory Networks for QA
Figure: Source - Kumar et. al 2016
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 22 / 30
![Page 26: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/26.jpg)
Match-LSTM for QA
Figure: Source - Wang, Jiang ICLR 2017
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 23 / 30
![Page 27: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/27.jpg)
Match-LSTM for QA
Figure: Source - Wang, Jiang ICLR 2017
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 24 / 30
![Page 28: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/28.jpg)
Overview
1 Textual Entailment
2 Question Answering
3 Named Entity Recognition
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 25 / 30
![Page 29: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/29.jpg)
Named Entity Recognition
Names (e.g. John Smith, New York Times)Places (e.g. Boston, Seattle, Sarajevo)Titles (e.g. Dr., PhD, Justice)Dates (e.g. Sept 11th, Veterans Day, Memorial Day)
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 26 / 30
![Page 30: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/30.jpg)
State of the Art conventional NER
Hand-crafted featuresDomain-specific knowledgeGazetteers for each domain, language etcCapitalization patterns
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 27 / 30
![Page 31: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/31.jpg)
biLSTM+CRF for NER
Figure: Source - Lample et al, 2016
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 28 / 30
![Page 32: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/32.jpg)
biLSTM + CRF
Start with GloVe / word2vec embeddingsCapture both left and right contexts for each word using LSTMsImpose adjacency constraints using CRF that learns a transitionmatrix between adjacent states
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 29 / 30
![Page 33: NLP Applications using Deep Learning - GitHub Pages · As a result of their deep understanding, they can produce new knowledge by discovering relationships, solving problems, constructing](https://reader030.vdocument.in/reader030/viewer/2022040613/5f07efb17e708231d41f80dd/html5/thumbnails/33.jpg)
Giri Iyengar (Cornell Tech) NLP Applications Feb 28, 2018 30 / 30