latent dirichlet allocation (nicolas loeff)

8/3/2019 Latent Dirichlet Allocation (Nicolas Loeff)

http://slidepdf.com/reader/full/latent-dirichlet-allocation-nicolas-loeff 1/39

Latent DirichletAllocationD. Blei, A. Ng, M. Jordan

Includes some slides adapted from J.Ramos at Rutgers, M.Steyvers and M. Rozen-Zvi at UCI, L. Fei Fei at UIUC.

Overview

What is so special about text?Classification methodsLSIUnigram / Mixture of UnigramProbabilistic LSI (Aspect Model)LDA modelGeometric interpretation

What is so special about text?

No obvious relation between featuresHigh dimensionality, (often larger vocabulary, V, than the number of features!)Importance of speed

The need for dimensionality

reductionRepresentation:

Presenting documents as vectors in thewords space - ‘bag of words’ representationIt is a sparse representation, V>>|D|

A need to define conceptual closeness

Bag of wordsOf all the sensory impressions proceeding to

the brain, the visual experiences are thedominant ones. Our perception of the worldaround us is based essentially on themessages that reach the brain from our eyes.For a long time it was thought that the retinalimage was transmitted point by point to visualcenters in the brain; the cerebral cortex wasa movie screen, so to speak, upon which the

image in the eye was projected. Through thediscoveries of Hubel and Wiesel we nowknow that behind the origin of the visualperception in the brain there is a considerablymore complicated course of events. Byfollowing the visual impulses along their pathto the various cell layers of the optical cortex,Hubel and Wiesel have been able todemonstrate that the message about theimage falling on the retina undergoes a step-wise analysis in a system of nerve cellsstored in columns. In this system each cell has its specific function and is responsible for a specific detail in the pattern of the retinal image.

sensory, brain,visual, perception,

retinal, cerebral cortex,eye, cell, opticalnerve, imageHubel, Wiesel

China is forecasting a trade surplus of $90bn(£51bn) to $100bn this year, a threefoldincrease on 2004's $32bn. The CommerceMinistry said the surplus would be created bya predicted 30% jump in exports to $750bn,compared with a 18% rise in imports to$660bn. The figures are likely to further annoy the US, which has long argued thatChina's exports are unfairly helped by adeliberately undervalued yuan. Beijingagrees the surplus is too high, but says theyuan is only one factor. Bank of Chinagovernor Zhou Xiaochuan said the countryalso needed to do more to boost domesticdemand so more goods stayed within thecountry. China increased the value of theyuan against the dollar by 2.1% in July and

permitted it to trade within a narrow band, butthe US wants the yuan to be allowed to tradefreely. However, Beijing has made it clear that it will take its time and tread carefullybefore allowing the yuan to rise further invalue.

China, trade,surplus, commerce,

exports, imports, US,yuan, bank, domestic,foreign, increase,

trade, value

Bag of words

Order of words in document can beignored, only count matters.Probability theory: Exchangability(includes IID) (Aldous, 1985).Exchangable RVs have a representationas mixture distribution (de Finetti, 1990).

What does this have to do with

Vision?ObjectObject Bag of Bag of

‘words’‘words’

TF-IDF Weighing Scheme (Salton

and McGill, 1983)Given corpus D, word w , document d ,calculate w d = f w, d · log (| D|/f w, D )

Many varieties of basic scheme.

Search procedure:Scan each d , compute each w

i, d , return set

D’ that maximizes Σ i w i, d

A Spatial Representation: Latent

Semantic Analysis (Deerwester, 1990)

Document/Term count matrix

SCIENC E …

6190RESEAR CH

2012SOUL

3034LOVE

Doc3 …Doc2Doc1

High dimensional space,not as high as |V|

RESEARCH

SCIENCE

• Each word is a single point in semantic space (dimensionality reduction)• Similarity measured by cosine of angle between word vectors

Feature Vector representation

From: Modeling the Internet and the Web: Probabilistic methodsand Algorithms , Pierre Baldi,Paolo Frasconi, Padhraic Smyth

Classification: assigning words to

topicsDifferent models for data:

Discrete Classifier,modeling the boundariesbetween different classes of

the data

Prediction of

Categoricaloutput e.g., SVM

Density Estimator: modeling thedistribution of the data pointsthemselves

GenerativeModels e.g. NB

Generative Models – Latent

semantic structure

Latent Structure

∑= ),()(ww P P

Distribution over words

)()()|()|(

P P P =

Inferring latent structure

Topic Models

Unsupervised learning of topics (“gist”) of documents:

articles/chaptersconversationsemails.... any verbal context

Topics are useful latent structures to explainsemantic association

Probabilistic Generative Model

Each document is a probability distributionover topics

Each topic is a probability distribution over words

Generative Process

l o a n

TOPIC 1

m o n e y

l o a n

b a n k

m o n e

r i v e r

TOPIC 2

r i v e r

s t r e a m

b a n k

s t r e a m

b a n k

l o a n

DOCUMENT 2: river 2 stream 2 bank 2 stream 2 bank 2 money 1 loan 1 river 2 stream 2 loan 1 bank 2 river 2 bank 2 bank 1 stream 2

river 2 loan 1 bank 2 stream 2 bank 2 money 1 loan 1 river 2 stream 2

bank 2 stream 2 bank 2 money 1 river 2 stream 2 loan 1 bank 2 river 2bank 2 money 1 bank 1 stream 2 river 2 bank 2 stream 2 bank 2

money 1

DOCUMENT 1: money 1 bank 1 bank 1 loan 1 river 2 stream 2

bank 1 money 1 river 2 bank 1 money 1 bank 1 loan 1 money 1

stream 2 bank 1 money 1 bank 1 bank 1 loan 1 river 2 stream 2 bank 1

money 1 river 2 bank 1 money 1 bank 1 loan 1 bank 1 money 1

stream 2

Mixturecomponents

Mixtureweights

Bayesian approach: use priorsMixture weights ~ Dirichlet( α )Mixture components ~ Dirichlet( β )

Vision: Topic = Object categories

Simple Model: Unigram

Words of document are drawn IID from asingle multinomial distribution:

Unigram Mixture Model

First choose topic z , then generate wordsconditionally independent given topic.

Probabilistic Latent Semantic

Indexing (Hoffman, 1999)Document d in training set, and word w n areconditionally independent given topic.

Not truly generative (dummy r.v. d ). Number of parameters grows with size of corpus(overfitting).Document may contain several topics.

Vision app.: Sivic et al., 2005

“face”

Vision app.: Fei Fei Li, 2005

“beach”

Example: Word density distribution

A geometric interpretation

Topics sampled repeatedly in eachDocument (like pLSI).

But, number of parameters does not growwith size of corpus.Problem: Inference.

LDA - Inference

Coupling between Dirchlet distribuionsmakes inference intractable.

Blei, 2001: Variational Approximation

LDA - Inference

Other procedures:Monte Carlo Markov Chin (Griffith et al.,

2002)Expectation Propagation (Minka et al., 2002)

Experiments

Perplexity: Inverse of geometric mean per-wordlikelihood (monotonically decreasing function of

likelihood):

Idea: Lower Perplexity implies better generalization.

Experiments – Nematode corpus

Experiments – AP corpus

Polysemy

PRINTINGPAPER PRINT

PRINTED

TYPEPROCESS

INK PRESSIMAGE

PRINTER PRINTS

PRINTERSCOPY

COPIESFORM

OFFSETGRAPHICSURFACE

PRODUCEDCHARACTERS

PLAYSSTAGE

AUDIENCETHEATER ACTORSDRAMA

SHAKESPEAREACTOR

THEATREPLAYWRIGHT

PERFORMANCEDRAMATICCOSTUMES

COMEDYTRAGEDY

CHARACTERS

SCENESOPERA

PERFORMED

TEAMGAME

BASKETBALLPLAYERSPLAYER

PLAYINGSOCCER PLAYED

BALLTEAMS

BASKETFOOTBALL

SCORECOURT

GAMESTRY

COACHGYMSHOT

JUDGETRIAL

CASEJURY

ACCUSEDGUILTY

DEFENDANTJUSTICE

EVIDENCE

WITNESSESCRIME

LAWYER WITNESS

ATTORNEYHEARING

INNOCENTDEFENSECHARGE

CRIMINAL

HYPOTHESISEXPERIMENT

SCIENTIFICOBSERVATIONS

SCIENTISTSEXPERIMENTS

SCIENTISTEXPERIMENTAL

METHODHYPOTHESES

TESTEDEVIDENCE

BASEDOBSERVATION

SCIENCEFACTSDATA

RESULTSEXPLANATION

STUDYTEST

STUDYINGHOMEWORK

NEEDCLASSMATH

TRYTEACHER

WRITEPLAN

ARITHMETICASSIGNMENT

PLACESTUDIED

CAREFULLYDECIDE

IMPORTANT NOTEBOOK

REVIEW

Choosing number of topicsSubjective interpretability

Bayesian model selectionGriffiths & Steyvers (2004)

Generalization test

Non-parametric Bayesian statisticsInfinite models; models that grow with size of data

Teh, Jordan, Teal, & Blei (2004)Blei, Griffiths, Jordan, Tenenbaum (2004)

latent dirichlet allocation (nicolas loeff)

Documents

using hierarchical latent dirichlet allocation to...

autoencoding variational bayes for latent dirichlet...

latent dirichlet allocation - stanford artificial...

latent dirichlet allocation€¦ · 3. latent dirichlet...

clustering images using the latent dirichlet allocation...

measuring topic quality in latent dirichlet...

1 an introduction to latent dirichlet allocation (lda)

lia$at$trec$2012$web$track: unsupervised$search$concepts...

blei2003-latent dirichlet allocation

latent dirichlet allocation - stanford...

vssml16 l4. association discovery and latent dirichlet...

latent dirichlet allocation

latent dirichlet markov allocation for sentiment...

flda: latent dirichlet allocation based unsteady flow...

latent dirichlet allocation (lda) and topic modeling:...

latent dirichlet allocation - university of...

latent dirichlet allocation - neural information...

latent dirichlet allocation: towards a deeper...

class-speciﬁc simplex-latent dirichlet allocation c...

max-margin latent dirichlet allocation for image...