something old, something new - yow! conferences · 2019-09-23 · @evanahari, yow! australia 2016...
TRANSCRIPT
@EVANAHARI, YOW! AUSTRALIA 2016
Something Old, Something New
A Talk about NLP for the Curious
Jabberwocky
– Lewis Carrollfrom Through the Looking-Glass and What Alice Found There, 1871
“`Twas brillig, and the slithy toves Did gyre and gimble in the wabe:
All mimsy were the borogoves, And the mome raths outgrabe.”
Why are these monkeys following
me?
Arrfff!
LOL
Challenges• Mistakes
• Slang & sparse words
• Ambiguity types • Lexical • Syntax level • Referential
Human Language• The cortical speech center unique to humans • Evolution over hundred thousands of years
• Vocabulary • Grammar • Speed
• An advanced processing unit • Sounds • Meaning of words • Grammar constructs • Match against a knowledge base • Understanding context and humor!
Human Language ProcessingPhonology − organization of sounds
Morphology − construction of words
Syntax − creation of valid sentences/phrases and identifying the structural roles of words in them
Semantics − finding meaning of words/phrases/sentences
Pragmatics − Situational meaning of sentences
Discourse − order of sentences affecting interpretation
World knowledge − mapping to general world knowledge
Context awareness - the hardest part…?
Natural Language Processing
• Computers generating language
• Computers understanding human language
Lexical analysis
Syntactic analysis
Semantic analysis
Discourse Integration
Pragmatic Analysis
– J. R. Firth, 1957
“You should know a word by the company it keeps.”
Language Models• Represent language in a mathematical way
A language model is a function that captures the statistical characteristics of the word-sequence distribution in a language
• Dimensionality challenge
10-word sequence from a 100 000 word vocabulary —> 10^50 possible sequences
• Large sample set vs processing time & cost vs accuracy
Bag-of-words
• Not suited for huge vocabulary • Semantics are not considered • Order of words are lost
= [111100]= [111100]= [110011] = [111100]
= [443311]
Vocabulary:Happy birthday to you dear “name”
= [100000]= [010000] = [001000]= [000100] = [000010] = [000001]
Sample text:Happy birthday to you Happy birthday to you Happy birthday dear “name”Happy birthday to you
Term frequency
n-grams“Hello everyone who is eager to learn NLP!”
• “gram”: a unit, e.g. letter, phoneme, word, …
• uni-gram: Hello, everyone, who, is, …
• bi-gram: Hello-everyone, everyone-who, who-is, …
• n-gram: n-length sequences of units
• k-skip-gram: skip k units
• bi-skip-tri-gram: Hello-is-learn, everyone-eager-NLP
n-gram Probabilistic Model• Given a sequence of words what is the likelihood of the next?
• Using counts of n-grams extracted from a training data set we can predict the next word x based on probabilities
• Simple; only n-1 words determines the probability
• Difficult to handle infrequent words and expressions
• Smoothening (e.g. Good-Turing, Katz-Back-off model, etc)
• Use additional sampling (bi-grams, tri-grams, skip-grams)
P(xi | xi-(n-1),… ,xi-1) = count(xi-(n-1),… ,xi-1)
count (xi-(n-1),… ,xi-1, xi)
Example use: Named Entity Extraction (NER) Examples:
• Grammar based: “…live in <city>”
• Co-occurrence based: “new+york”, “san+francisco”, …
Common pattern: Inference of applying various models
AppleRound
Red
HasLeaf
+
+
0
Naive Bayes Probabilistic Model
Example Use: Text Classification
Sample Data AppleRed No
Green YesYellow YesRed YesRed Yes
Green YesYellow NoYellow NoRed Yes
Yellow YesRed No
Green YesGreen YesYellow No
Feature No YesGreen 4 4/14 0.29Yellow 3 2 5/14 0.36Red 2 3 5/14 0.36
Grand Total 5 95/14 9/140.36 0.64
Incoming fruit text says “red” - is it about an apple?P(Yes | Red) = P( Red | Yes) * P(Yes) / P (Red)
P (Red |Yes) = 3/9 = 0.33 P(Yes)= 9/14 = 0.64 P(Red) 0.36
P (Yes | Red) = 0.33 * 0.64 / 0.36 = 0.60
60% chance it’s about an apple!
Naive BayesThings to Consider:
• Easy and fast, good for multi-class, better than most
• Does not handle unknown categories well, needs smoothing
• Needs less training data, but well representative
• Assuming attributes to be truly independent
Combining Models
Things to Consider:
• How many models can you afford?
• How good are your models (i.e. training data)?
• Latency vs accuracy?
Bag of Words
0 0 0 1 =
= 0 1 0 0
=
=
Continuous Bag of Words (Embeddings)
2 3 8 1
7 5 6 2
Distributed Representation• A word is a dot in a multi-dimensional vector
space, where each dimension is features of a word
• Decide features?
• HUMAN: decides features; gender, plurality, semantic characteristics
• COMPUTER: learn the features; continuous values
Neural Net Language Model• A model based on the capabilities of NN is an
NNLM
• Rely on the NN to discover the features of a distributed representation
• Extrapolations makes it possible to keep a dense model - even for very large data sets
Mikolow et al’s CBOW vs Continuous Skip-gram
• CBOW - predict a term based on context (near-terms)
• w-2, w-1, w+1, w+2 —> w
• fast to train
• higher accuracy for frequent words
• conditioning on context needs larger data sets
• Continuous Skip-gram - predict context (near-terms) based on a word
• w —> w-2, w-1, w+1, w+2
• k-skip-n-gram: k and n determines complexity (training time vs accuracy)
• helps create more samples from a smaller data set (data sparsity, rare terms)
Diagram borrowed from Mikolow et al’s paper
1. Probability of next term, i.e. Bayes TheoremApproximate t with n - to gain simplicity of n-grams
2. d-dimensional feature vector Cwt-i (column wt-i of parameter matrix C):Ck contains learned features for word k
3. Use standard NN for probabilistic classification (Softmax):where
NN-based Probabilistic Prediction Model
P(w1, w2,… ,wt-1, wt) = P(w1)P(w2|w1)P(w3|w1,w2)…P(w1, w2,… ,wt-1)
x = (Cwt-n+1, 1, …, Cwt-n+1, d, Cwt-n+2, 1, …, Cwt-2, d, Cwt-1, 1, …, Cwt-1, d)
SUM(i=1 to N) eai eak P(wt = k|wt-n+1, … ,wt-1) =
ak = bk + SUM(i=1 to h) Wki tanh(ci + SUM(j=1 to (n-1)d) Vijxj)
Diagram borrowed from Bengio et al’s paper
NLP is not New…
ABBYY, Angoss, Attensity, AUTINDEX, Autonomy, Averbis, Basis Technology, Clarabridge, Complete Discovery Source, Endeca Technologies, Expert
System S.p.A., FICO Score, General Sentiment, IBM LanguageWare, IBM SPSS, Insight, LanguageWare,
Language Computer Corporation, Lexalytics, LexisNexis, Luminoso, Mathematica,
MeaningCloud, Medallia, Megaputer Intelligence, NetOwl, RapidMiner, SAS Text Miner and
Teragram;, Semantria , Smartlogic, StatSoft, Sysomos, WordStat, Xpresso, ….
…but Getting Hot (Again)• Big text data sets available
• Distributed processing tech & capacity cheaper
• ML-based training economically possible (and more accurate)
• Open source movement
• Large upswing potential…
No animals were harmed during this photo shoot
Cheat Sheet• openNLP-Java,Apache,familiar,easier,older• coreNLP-Java,Stanford,popular,goodtoolspan• NLTK-python,richinresources,easiest• spaCy-upandcoming,python,promising..• FasCext-nothingnew..?• Spark-“MLframework”,customimplementaKon,largescale
• Deeplearning4j-word2vec(java,scala)• Tensorflow(SyntaxNet)-separatedopKmizaKon&moretuningnobs,beCersyntaxparsingmodel,veryrecentlylargescaletoo
• Language key to our species’ success • Our multi-step process is complex and our brains
forgiving • A language models represents word sequence
distributions within a language • Bag-of-words, n-grams are common representations • Naive bayes common for probabilistic models • Distributed representations are dense and powerful • NNLM based on learned word-features • Positive NLP trends:
More open source tools and frameworks and generated
distributed representations available to all
Summary and Questions
@EVANAHARI, YOW! AUSTRALIA 2016
Jabberwocky
Vote!