Upload File

language processing deep learning methods for natural

  • Home
  • Documents
  • Language Processing Deep Learning Methods for Natural
91
Deep Learning Methods for Natural Language Processing Garrett Hoffman Director of Data Science @ StockTwits

Upload: others

Post on 05-Apr-2022

6 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Language Processing Deep Learning Methods for Natural

Deep Learning Methods for Natural Language ProcessingGarrett HoffmanDirector of Data Science @ StockTwits

Page 2: Language Processing Deep Learning Methods for Natural

Talk Overview

https://github.com/GarrettHoffman/AI_Conf_2019_DL_4_NLP
Page 3: Language Processing Deep Learning Methods for Natural

Learning Distributed Representations of Words with Word2Vec

3

Page 4: Language Processing Deep Learning Methods for Natural

Sparse Representation

Page 5: Language Processing Deep Learning Methods for Natural

Sparse Representation

Page 6: Language Processing Deep Learning Methods for Natural

Sparse Representation

Page 7: Language Processing Deep Learning Methods for Natural

Sparse Representation

Page 8: Language Processing Deep Learning Methods for Natural

Sparse Representation

Page 9: Language Processing Deep Learning Methods for Natural

Sparse Representation Drawbacks

Page 10: Language Processing Deep Learning Methods for Natural

Sparse Representation Drawbacks

Page 11: Language Processing Deep Learning Methods for Natural

Sparse Representation Drawbacks

□

Page 12: Language Processing Deep Learning Methods for Natural

Distributed Representation

Page 13: Language Processing Deep Learning Methods for Natural

Distributed Representation

Page 14: Language Processing Deep Learning Methods for Natural

Distributed Representation

Page 15: Language Processing Deep Learning Methods for Natural

Distributed Representation

Page 16: Language Processing Deep Learning Methods for Natural

Word2Vec

“Distributed Representations of Words and Phrases and their Compositionality”, Mikolov et al. (2013)

https://arxiv.org/abs/1310.4546
Page 17: Language Processing Deep Learning Methods for Natural
Page 18: Language Processing Deep Learning Methods for Natural

Word2Vec - Generating Data

McCormick, C. (2016, April 19). Word2Vec Tutorial - The Skip-Gram Model.

http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
Page 19: Language Processing Deep Learning Methods for Natural

Word2Vec - Skip-gram Network Architecture

McCormick, C. (2016, April 19). Word2Vec Tutorial - The Skip-Gram Model.

http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
Page 20: Language Processing Deep Learning Methods for Natural

Word2Vec - Skip-gram Network Architecture

McCormick, C. (2016, April 19). Word2Vec Tutorial - The Skip-Gram Model.

http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
Page 21: Language Processing Deep Learning Methods for Natural

Word2Vec - Embedding Layer

McCormick, C. (2016, April 19). Word2Vec Tutorial - The Skip-Gram Model.

http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
Page 22: Language Processing Deep Learning Methods for Natural

Word2Vec - Embedding Layer

McCormick, C. (2016, April 19). Word2Vec Tutorial - The Skip-Gram Model.

http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
Page 23: Language Processing Deep Learning Methods for Natural

Word2Vec - Skip-gram Network Architecture

McCormick, C. (2016, April 19). Word2Vec Tutorial - The Skip-Gram Model.

http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
Page 24: Language Processing Deep Learning Methods for Natural

Word2Vec - Output Layer

McCormick, C. (2016, April 19). Word2Vec Tutorial - The Skip-Gram Model.

http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
Page 25: Language Processing Deep Learning Methods for Natural

Word2Vec - Intuition

McCormick, C. (2017, January 11). Word2Vec Tutorial Part 2 - Negative Sampling.

http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/
Page 26: Language Processing Deep Learning Methods for Natural

Word2Vec - Negative Sampling

McCormick, C. (2017, January 11). Word2Vec Tutorial Part 2 - Negative Sampling.

http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/
Page 27: Language Processing Deep Learning Methods for Natural

Word2Vec - Negative Sampling

McCormick, C. (2017, January 11). Word2Vec Tutorial Part 2 - Negative Sampling.

http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/
Page 28: Language Processing Deep Learning Methods for Natural

https://www.tensorflow.org/tutorials/word2vec

Word2Vec - Results

https://www.tensorflow.org/tutorials/word2vec
Page 29: Language Processing Deep Learning Methods for Natural

Pre-Trained Word Embedding

https://github.com/Hironsan/awesome-embedding-models
Page 30: Language Processing Deep Learning Methods for Natural

Distributed Representations of Sentences and Documents

Doc2Vec

https://cs.stanford.edu/~quocle/paragraph_vector.pdf
Page 31: Language Processing Deep Learning Methods for Natural

Recurrent Neural Networks and their Variants

31

Page 32: Language Processing Deep Learning Methods for Natural

Sequence Models

Page 33: Language Processing Deep Learning Methods for Natural

Recurrent Neural Networks (RNNs)

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 34: Language Processing Deep Learning Methods for Natural

Recurrent Neural Networks (RNNs)

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 35: Language Processing Deep Learning Methods for Natural

Recurrent Neural Networks (RNNs)

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 36: Language Processing Deep Learning Methods for Natural

Long Term Dependency Problem

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 37: Language Processing Deep Learning Methods for Natural

Long Short Term Memory (LSTMs)

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 38: Language Processing Deep Learning Methods for Natural

Long Short Term Memory (LSTMs)

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 39: Language Processing Deep Learning Methods for Natural

Long Short Term Memory (LSTMs)

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 40: Language Processing Deep Learning Methods for Natural

LSTM - Forget Gate

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 41: Language Processing Deep Learning Methods for Natural

LSTM - Learn Gate

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 42: Language Processing Deep Learning Methods for Natural

LSTM - Update Gate

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 43: Language Processing Deep Learning Methods for Natural

LSTM - Output Gate

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 44: Language Processing Deep Learning Methods for Natural

Gated Recurrent Unit (GRU)

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Page 45: Language Processing Deep Learning Methods for Natural

Types of RNNs

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Page 46: Language Processing Deep Learning Methods for Natural

Types of RNNs

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Page 47: Language Processing Deep Learning Methods for Natural

LSTM Network Architecture

Page 48: Language Processing Deep Learning Methods for Natural

Learning Embeddings End-to-End

Page 49: Language Processing Deep Learning Methods for Natural

Dropout

Page 50: Language Processing Deep Learning Methods for Natural

Bidirectional LSTM

http://colah.github.io/posts/2015-09-NN-Types-FP/

http://colah.github.io/posts/2015-09-NN-Types-FP/
Page 51: Language Processing Deep Learning Methods for Natural

Convolutional Neural Networks for Language Tasks

51

Page 52: Language Processing Deep Learning Methods for Natural

Computer Vision Models

Page 53: Language Processing Deep Learning Methods for Natural

Convolutional Neural Networks (CNNs)

Page 54: Language Processing Deep Learning Methods for Natural

Convolutional Neural Networks (CNNs)

http://colah.github.io/posts/2014-07-Conv-Nets-Modular/

http://colah.github.io/posts/2014-07-Conv-Nets-Modular/
Page 55: Language Processing Deep Learning Methods for Natural

CNNs - Convolution Function

0 0 0 0 0 0

0 1 2 1 1 2

0 1 1 1 1 1

1 0 0 0 0 0

0 0 1 1 1 0

0 1 1 1 1 1

0 0 2

1 2 0

1 2 2

Input Vector Kernel / Filter

Page 56: Language Processing Deep Learning Methods for Natural

CNNs - Convolution Function

0 0 0 0 0 0

0 1 2 1 1 2

0 1 1 1 1 1

1 0 0 0 0 0

0 0 1 1 1 0

0 1 1 1 1 1

0 0 2

1 2 0

1 2 2

Input Vector Kernel / Filter

Page 57: Language Processing Deep Learning Methods for Natural

CNNs - Convolution Function

0 0 0 0 0 0

0 1 2 1 1 2

0 1 1 1 1 1

1 0 0 0 0 0

0 0 1 1 1 0

0 1 1 1 1 1

0 0 0

1 0 0

0 2 0

Input Vector Kernel / Filter

2

Output Vector

Page 58: Language Processing Deep Learning Methods for Natural

CNNs - Convolution Function

0 0 0 0 0 0

0 1 2 1 1 2

0 1 1 1 1 1

1 0 0 0 0 0

0 0 1 1 1 0

0 1 1 1 1 1

0 0 0

1 0 0

0 2 0

Input Vector Kernel / Filter

2 3

Output Vector

Page 59: Language Processing Deep Learning Methods for Natural

CNNs - Convolution Function

0 0 0 0 0 0

0 1 2 1 1 2

0 1 1 1 1 1

1 0 0 0 0 0

0 0 1 1 1 0

0 1 1 1 1 1

0 0 0

1 0 0

0 2 0

Input Vector Kernel / Filter

2 3 4

Output Vector

Page 60: Language Processing Deep Learning Methods for Natural

CNNs - Convolution Function

0 0 0 0 0 0

0 1 2 1 1 2

0 1 1 1 1 1

1 0 0 0 0 0

0 0 1 1 1 0

0 1 1 1 1 1

0 0 0

1 0 0

0 2 0

Input Vector Kernel / Filter

2 3 4 3

Output Vector

Page 61: Language Processing Deep Learning Methods for Natural

CNNs - Convolution Function

0 0 0 0 0 0

0 1 2 1 1 2

0 1 1 1 1 1

1 0 0 0 0 0

0 0 1 1 1 0

0 1 1 1 1 1

0 0 0

1 0 0

0 2 0

Input Vector Kernel / Filter

2 3 4 3

0

Output Vector

Page 62: Language Processing Deep Learning Methods for Natural

CNNs - Convolution Function

0 0 0 0 0 0

0 1 2 1 1 2

0 1 1 1 1 1

1 0 0 0 0 0

0 0 1 1 1 0

0 1 1 1 1 1

0 0 0

1 0 0

0 2 0

Input Vector Kernel / Filter

2 3 4 3

0 1

Output Vector

Page 63: Language Processing Deep Learning Methods for Natural

CNNs - Convolution Function

0 0 0 0 0 0

0 1 2 1 1 2

0 1 1 1 1 1

1 0 0 0 0 0

0 0 1 1 1 0

0 1 1 1 1 1

0 0 0

1 0 0

0 2 0

Input Vector Kernel / Filter

2 3 4 3

0 1 1 1

1 2 2 2

2 2 3 3

Output Vector

Page 64: Language Processing Deep Learning Methods for Natural

CNNs - Max Pooling Function

3

Input Vector Output Vector

2 3 4 3

0 1 1 1

1 2 2 2

2 2 3 3

Page 65: Language Processing Deep Learning Methods for Natural

CNNs - Max Pooling Function

3 4

Input Vector Output Vector

2 3 4 3

0 1 1 1

1 2 2 2

2 2 3 3

Page 66: Language Processing Deep Learning Methods for Natural

CNNs - Max Pooling Function

3 4

2

Input Vector Output Vector

2 3 4 3

0 1 1 1

1 2 2 2

2 2 3 3

Page 67: Language Processing Deep Learning Methods for Natural

CNNs - Max Pooling Function

3 4

2 3

Input Vector Output Vector

2 3 4 3

0 1 1 1

1 2 2 2

2 2 3 3

Page 68: Language Processing Deep Learning Methods for Natural

Convolutional Neural Networks (CNNs)

Page 69: Language Processing Deep Learning Methods for Natural

CNN Architecture for Text

Page 70: Language Processing Deep Learning Methods for Natural

State of the Art in NLP - Generalized Language Models

70

Page 71: Language Processing Deep Learning Methods for Natural

Generalized Language Modeling

Page 72: Language Processing Deep Learning Methods for Natural

Types of RNNs

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Page 73: Language Processing Deep Learning Methods for Natural

P(wn|w1,…wn−

1)

Generalized Language Modeling

Page 74: Language Processing Deep Learning Methods for Natural

Current SOTA

Page 75: Language Processing Deep Learning Methods for Natural

ULMFiT

http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html

http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html
Page 76: Language Processing Deep Learning Methods for Natural

ULMFiT

http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html

http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html
Page 77: Language Processing Deep Learning Methods for Natural

ULMFiT - GLM Pre Training

AWD-LSTM

Page 78: Language Processing Deep Learning Methods for Natural

ULMFiT

http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html

http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html
Page 79: Language Processing Deep Learning Methods for Natural

ULMFiT - Refine GLM for Target Task

Discriminative Fine-Tuning

Slanted Triangular Learning Rates (STLR)

Page 80: Language Processing Deep Learning Methods for Natural

ULMFiT

http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html

http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html
Page 81: Language Processing Deep Learning Methods for Natural

ULMFiT - Target Task Classification Training

Concat Pooling

Gradual Unfreeze

Page 82: Language Processing Deep Learning Methods for Natural

BERT / GPT-2 - Transformer Model

Transformer Model

Page 83: Language Processing Deep Learning Methods for Natural

Attention Mechanism

http://www.abigailsee.com/2017/04/16/taming-rnns-for-better-summarization.html

http://www.abigailsee.com/2017/04/16/taming-rnns-for-better-summarization.html
Page 84: Language Processing Deep Learning Methods for Natural

Transformer Model

Attention Is All You Need

https://arxiv.org/pdf/1706.03762.pdf
Page 85: Language Processing Deep Learning Methods for Natural

Transformer Model

Attention Is All You Need

https://arxiv.org/pdf/1706.03762.pdf
Page 86: Language Processing Deep Learning Methods for Natural

Transformer Model

http://mlexplained.com/2017/12/29/attention-is-all-you-need-explained/

http://mlexplained.com/2017/12/29/attention-is-all-you-need-explained/
Page 87: Language Processing Deep Learning Methods for Natural

Practical Considerations for Modeling with Your Data

87

Page 88: Language Processing Deep Learning Methods for Natural

Practical Considerations

Page 89: Language Processing Deep Learning Methods for Natural

Practical Considerations

Page 90: Language Processing Deep Learning Methods for Natural

Practical Considerations

Page 91: Language Processing Deep Learning Methods for Natural

Thanks!Any questions?

https://github.com/GarrettHoffman/AI_Conf_2019_DL_4_NLP
http://www.oreilly.com/people/d3807-garrett-hoffman

Gas Processing Methods

Quantitative Methods and Analysisferent stocks. Deep theories from various disciplines like physics, statistics, control theory, graph theory, game theory, signal processing, probability,

Deep Learning for Natural Language Processing - …docs.huihoo.com/.../Deep-Learning-for-Natural-Language-Processing... · Xipeng Qiu (Fudan University) Deep Learning for Natural

Design Methods for Deep Foundations

Innovative deep-mixing methods for oil & gas applications deep mixing methods for oil... · Innovative deep-mixing methods for oil ... This paper presents two case histories in which

P05 deep boltzmann machines cvpr2012 deep learning methods for vision

Natural Language Processing with Deep Learning …

Deep Learning: Methods and Applications

Processing images with Deep Learning

Deep processing triangle directions

Symposium on Sparse Signal Processing and Deep … on Sparse Signal Processing and Deep ... The Sparse Signal Processing and Deep Learning symposium will explore deep connection

Processing of Deep and Surface Anaphors

Deep Water Pre-Processing: East Coast India · 2019-06-04 · Deep Water Pre-Processing: Smith, et al, 2008 Deep Water Pre-Processing: East Coast India Phil Smith1, Ian F. Jones1,

SciBorg: Deep Processing and Chemical Informatics

Deep Representation Learning in Speech Processing

Deep Moveout Processing

Applications of Deep Learning 25-04-2019...Deep learning methods employ multiple processing layers to learn hierarchical representations of data and have produced state-of-the-art

Introduction to Deep Processing Techniques for NLP Deep Processing Techniques for Natural Language Processing Ling 571 January 3, 2011 Gina-Anne Levow

Efficient Methods for Deep Learning

Deep Learning & Natural Language processing

Deep Learning for Natural Language Processing

Deep Foundation Design Methods

Symposium on Sparse Signal Processing and Deep Learning ... · Symposium on Sparse Signal Processing and Deep Learning ... and spare implementation of deep networks using novel hardware

Comparing Deep Reinforcement Learning Methods for ...und...Reinforcement Learning methods (Deep Q-Learning, Deep Deterministic Policy Gra-dient and Distributed Proximal Policy Optimization);

Processing Methods for Composites

Deep Learning Architectures for Sequence Processing

Empirical Methods in Natural Language Processing · Empirical Methods in Natural Language Processing ... Welcome to the 2009 Conference on Empirical Methods in Natural Language Processing!

Beyond Deep Learning: Combining Neural Processing and ... · Deep Learning for Natural Language Processing • Major tools: word embedding, deep networks (recurrent neural networks,

Deep Learning for Human Language Processing

Deep Seismic Imaging and Velocity Estimation in …...seismic data processing steps. Special data processing methods like surface consistent deconvolution, Kirchoff migration and 102

Pork Meat Deep Processing Insight

Improving Language Understanding by Generative Pre-Training€¦ · learning in natural language processing (NLP). Most deep learning methods require substantial amounts of manually

Signal Processing for Deep Learning and Machine Learning · Signal Processing for Deep Learning and Machine Learning Kirthi Devleker, Sr. Product Manager, Signal Processing and Wavelets

Innovative processing methods with thermoplastic PET …cpscindia.com/pdf/fINISHINGoPTION/06 Papers/Innovative processing... · Innovative processing methods with thermoplastic PET-foam

Deep Water Processing