bbm495-lecture8burcucan/bbm495-lecture8-4pp.pdf · exp(uwvc) dot product compares similarity of o...
Post on 27-Jul-2020
2 Views
Preview:
TRANSCRIPT
4/29/19
1
BBM 495WORD EMBEDDINGS
2018-2019 SPRING
§ How similar is pizza to pasta?§ How related is pizza to Italy?
§ Representing words as vectors allows easy computation of similarity
4/29/19
2
§ Increase in size with vocabulary§ Very high dimensional: require a lot of storage§ Subsequent classification models have sparsity issues§ Models are less robust
4/29/19
3
4/29/19
4
4/29/19
5
4/29/19
6
4/29/19
7
Predict!
4/29/19
8
§ Remember: two vectors are similar if they have a high dot product§ Cosine is just a normalized dot product
§ So: § Similarity (o,c) ∝ uo · Vc
§ Wewill need to normalize to get a probability
§ We use softmax to turn into probabilities:
4/29/19
9
4/29/19
10
4/29/19
11
all§ Take gradients at each window
§ Go through gradient for each center vector v in a window§ In each window, we will compute updates for all parameters that
are being used in that window. For example:
4/29/19
12
4/29/19
13
4/29/19
14
4/29/19
15
§ Dense Vectors, Dan Jurafsky§ Representation for Language: from Word Embeddings to
Sentence Meanings, Christopher Manning, Stanford University, 2017
§ Natural Language Processing with Deep Learning, Richard Socher, Stanford University
§ More Word Vectors, Richard Socher, Stanford University§ Improving Distributional Similarity with Lessons Learned from
Word Embeddings, Omer Levy,
top related