p robabilistic g raphic m odel &lda yilun wang chu-kochen honors college, zhejiang university
TRANSCRIPT
![Page 1: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/1.jpg)
PROBABILISTIC GRAPHIC MODEL&LDA
Yilun Wang
Chu-kochen Honors College, Zhejiang University
![Page 2: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/2.jpg)
OUTLINE
![Page 3: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/3.jpg)
WHAT DOES A PROBABILISTIC MODEL DO? What are mechanisms underlying gene expression
data? Colon Cancer Research.
How to predict prices of stocks and bonds from historical data? Hedge fund dynamics.
Given a list of movies that a particular user likes, what other movies would she like? Netflix Prize.
How to identify aspects of a patient’s health that are indicative of disease? Heart Disease Classification.
Which documents from a collection are relevant to a search query? Google Research.
![Page 4: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/4.jpg)
HOW
Setps:1. Formulating questions about data.
明确要干什么,要求解什么,有哪些参数2. Design an appropriate joint distribution.
建模,确定数据的结构,隐变量,共轭先验(确定图模型)3. Cast our questions on the computation on the joint.
将要求解的概率通过积分,条件独立,拆成多个可计算的部分4. Develop efficient algorithms to perform or
approximate the computations on the joint.利用吉布斯采样或者变分推理等方法求解
![Page 5: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/5.jpg)
PROBABILITY REVIEW
R1. Joint Distributions
R2. Marginal Probabilities
R3. Conditional Probabilities (R1+R2) Joint/Marginal
R4. Independence
![Page 6: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/6.jpg)
PROBABILITY REVIEW
Bayes' rule
( ) ( ) ( )|A
P B A P A dA P B=ò
“Bayesian Inference with Tears”
posterior
likelihoodprior
evidence
Probability Estimation
( ) ( ) ( )| |P A B P B A P Aµ
(R2+R3)
![Page 7: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/7.jpg)
GRAPHICAL MODELS
A family of probability distributions defined in terms of a directed (DGM/DAG/Bayesian Network) or/and(chain) undirected (Markov Networks) graph
![Page 8: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/8.jpg)
GRAPHICAL MODELS
A more economic representation of the joint图模型是表示随机变量之间的关系的图,图中的节点表示随机变量, (缺少 )边表示条件独立假设。因此可以对联合分布提供一种紧致表示
Advantages of GM allow us to articulate structural assumptions
about collections of random variables. provide general algorithms to compute
conditionals, marginals, expectations and independencies, etc.
provide control over the complexity of these operations.
decouple the factorization of the joint from its particular function form.
![Page 9: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/9.jpg)
CONDITIONAL INDEPENDENCE
Independence:
Conditional Independence
![Page 10: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/10.jpg)
CONDITIONAL INDEPENDENCE
Take graphic model of LDA as an example:
![Page 11: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/11.jpg)
CONDITIONAL INDEPENDENCE
Sometime we want to evaluate the following CI:
?
![Page 12: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/12.jpg)
PROBABILISTIC GRAPHIC MODEL
Graphical model is the study of probabilistic models Just because there are nodes and edges doesn’t mean
it’s a graphical model These are not graphical models:
Xiaojin Zhu, Tutorial on Graphic Models at KDD-2012http://pages.cs.wisc.edu/~jerryzhu/
![Page 13: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/13.jpg)
DIRECTED GRAPHIC MODELS
![Page 14: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/14.jpg)
Binary varibles
EXAMPLE: ALARM
求 P(B, ~E, A, J, ~M)
![Page 15: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/15.jpg)
Used extensively in natural language
processing Plate representation on the right
EXAMPLE: NAÏVE BAYES
![Page 16: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/16.jpg)
EXAMPLE: PROBABILISTIC LSI
Eric Xing, Topic Models, Latent Space Models, Sparse Coding , and All That
![Page 17: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/17.jpg)
EXAMPLE: LATENT DIRICHLET ALLOCATION
![Page 18: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/18.jpg)
Generative model Models each word in a document as a sample
from a mixture model. Each word is generated from a single topic,
different words in the document may be generated from different topics.
A topic is characterized by a distribution over words.
Each document is represented as a list of admixing proportions for the components (i.e. topic vector).
The topic vectors and the word rates each follows a Dirichlet prior --- essentially a Bayesian pLSI
EXAMPLE: LATENT DIRICHLET ALLOCATION
![Page 19: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/19.jpg)
EXAMPLE: LATENT DIRICHLET ALLOCATION
![Page 20: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/20.jpg)
EXAMPLE: LATENT DIRICHLET ALLOCATION
![Page 21: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/21.jpg)
CONDITIONAL INDEPENDENCE
![Page 22: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/22.jpg)
D-SEPARATION CASE 1: TAIL-TO-TAIL
![Page 23: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/23.jpg)
D-SEPARATION CASE 2: HEAD-TO-TAIL
![Page 24: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/24.jpg)
D-SEPARATION CASE 3: HEAD-TO-HEAD
![Page 25: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/25.jpg)
D-SEPARATION
![Page 26: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/26.jpg)
UNDIRECTED GRAPHICAL MODELS
![Page 27: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/27.jpg)
FACTOR GRAPH
![Page 28: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/28.jpg)
WHERE DOES COMPLICATED MODEL SUCH AS LDA COME FROM?
![Page 29: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/29.jpg)
THE ORIGIN OF LDA
Dice Model Is Dice Model a
generative model?
Unigram Model
xiDN
Language Model
wφN
DProbability
Vocabulary
CorpusTopic
Dice Model
![Page 30: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/30.jpg)
THE EVOLUTION PROCESS
E1: Add a conjugate prior Why Conjugate
prior?
E2: Sampling with repeated choice of dice
xiDN
α xiDN
Bayesian (completed) Dice Model
wφN
D
α wφN
D
α
Language Model
![Page 31: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/31.jpg)
THE EVOLUTION PROCESS
E3: Turn DM-E2 into a Bayesian mixture model
Mixture of unigrams
xiDN
B
2ψ
α
β
D
Π
K
α
β
Nwdizd
ψzd
![Page 32: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/32.jpg)
THE EVOLUTION PROCESS
Mixture of unigrams
D
Π
K
α
β
Nwdizd
ψzd
Corpus
Topic 1
Topic 2
Topic 3
![Page 33: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/33.jpg)
D
Π
D
α
β
Nwdizd
ψzd
THE EVOLUTION PROCESS
Finally: we reach the pLSA/LDA
Corpus
Topic 1
Topic 2
Topic 3
![Page 34: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/34.jpg)
LDA VARIATIONS
![Page 35: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/35.jpg)
REVISITING K-MEANS: NEW ALGORITHMS VIA BAYESIAN NONPARAMETRICS
Bayesian nonparametrics can be used for modeling infinite mixtures, and hierarchical Bayesian models can be utilized for sharing clusters across multiple data sets.
Revisiting the k-means clustering algorithm from a Bayesian nonparametric viewpoint
![Page 36: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/36.jpg)
RECALL
Mixture Gaussian
![Page 37: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/37.jpg)
RECALL
Hjort, N., Holmes, C., Mueller, P., and Walker, S. Bayesian Nonparametrics: Principles and Practice. Cambridge University Press, Cambridge, UK, 2010.
Dirichlet process mixture: infinite mixture
![Page 38: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/38.jpg)
DP-MEAN
![Page 39: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/39.jpg)
THE CONTEXTUAL FOCUSED TOPIC MODEL(CFTM)
cFTM infers a sparse (“focused”) set of topics for each document, while also leveraging contextual information about the author(s) and document venue.
hierarchical beta process
Xu Chen, Mingyuan Zhou, Lawrence Carin, Duke University, The Contextual Focused Topic Model
![Page 40: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/40.jpg)
![Page 41: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/41.jpg)
LDA
cFTM+HBP
![Page 42: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/42.jpg)
PROS
(1) It automatically infers the number of topics by combining properties from the Dirichlet process and hierarchical beta process, allowing an unbounded number of topics for the entire corpus, while inferring a focused (sparse) set of topics for each individual document.
![Page 43: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/43.jpg)
PROS
(2) The cFTM nonparametrically clusters then authors and venues, thereby increasing statistical strength while also inferring useful relational information.
(3) Instead of pre-specifying the importance of author/venue information (as was done in [6]), the cFTM automatically infers the document-dependent, probabilistic importance of the author/venue information on word assignment.
Data: DBLP+NSF
![Page 44: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/44.jpg)
TM-LDA: EFFICIENT ONLINE MODELING OF LATENT TOPIC TRANSITIONS IN SOCIAL MEDIA
Much of the textual content on the web, and especially social media, is temporally sequenced, and comes in short fragments, including microblog posts on sites such as Twitter and Weibo, status updates on social networking sites such as Facebook and LinkedIn, or comments on content sharing sites such as YouTube
Yu Wang, Eugene Agichtein, Michele Benzi, Emory University, TM-LDA: Efficient Online Modeling of Latent Topic Transitions in Social Media
![Page 45: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/45.jpg)
Efficiently mining text streams such as a sequence of posts from the same author, by modeling the topic transitions that naturally arise in these data.
TM-LDA learns the transition parameters among topics by minimizing the prediction error on topic distribution in subsequent postings. After training, TM-LDA is thus able to accurately predict the expected topic distribution in future posts.
![Page 46: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/46.jpg)
Space of topic distributions
Given the topic distribution vector of a historical document x, the estimated topic distribution of a new document ˆy is given by ˆy = f(x)
![Page 47: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/47.jpg)
EXPERIMENT
![Page 48: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/48.jpg)
EXPERIMENT
![Page 49: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/49.jpg)
COMSOC: ADAPTIVE TRANSFER OF USER BEHAVIORS OVER COMPOSITE SOCIAL NETWORK
Accurate prediction of user behaviors is important for many social media applications, including social marketing, personalization and recommendation, etc.
1. alleviate the data sparsity problem 2. enhance the predictive performance of
user modeling
Erheng Zhong, Wei Fan, Junwei Wang, Lei Xiao, and Yong Li, HKUST, IBM Research Center, Tencent, ComSoc: Adaptive Transfer of User Behaviors overComposite Social Network
![Page 50: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/50.jpg)
![Page 51: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/51.jpg)
Comsoc
![Page 52: P ROBABILISTIC G RAPHIC M ODEL &LDA Yilun Wang Chu-kochen Honors College, Zhejiang University](https://reader036.vdocument.in/reader036/viewer/2022062300/56649cf85503460f949c9376/html5/thumbnails/52.jpg)
Thank you!