Grounding Topic Models with Knowledge Bases

Zhiting Hu,‡⇤ Gang Luo,§ Mrinmaya Sachan,‡ Eric Xing,‡ Zaiqing Nie††Microsoft Research, Beijing, China §Microsoft, California, USA

‡School of Computer Science, Carnegie Mellon University{zhitingh,mrinmays,epxing}@cs.cmu.com, {gluo,znie}@microsoft.com

AbstractTopic models represent latent topics as probabilitydistributions over words which can be hard to inter-pret due to the lack of grounded semantics. In thispaper, we propose a structured topic representationbased on an entity taxonomy from a knowledgebase. A probabilistic model is developed to inferboth hidden topics and entities from text corpora.Each topic is equipped with a random walk over theentity hierarchy to extract semantically groundedand coherent themes. Accurate entity modeling isachieved by leveraging rich textual features fromthe knowledge base. Experiments show signifi-cant superiority of our approach in topic perplexityand key entity identification, indicating potentialsof the grounded modeling for semantic extractionand language understanding applications.

1 IntroductionProbabilistic topic models [Blei et al., 2003] have been oneof the most popular statistical frameworks to identify latentsemantics from large text corpora. The extracted topics arewidely used for human exploration [Chaney and Blei, 2012],information retrieval [Wei and Croft, 2006], machine transla-tion [Mimno et al., 2009], and so forth.

Despite their popularity, topic models are weak models ofnatural language semantics. The extracted topics are diffi-cult to interpret due to incoherence [Chang et al., 2009] andlack of background context [Wang et al., 2007]. Furthermore,it is hard to grasp semantics merely as topics formulated asword distributions without any grounded semantics [Song etal., 2011; Gabrilovich and Markovitch, 2009]. Though recentresearch has attempted to exploit various knowledge sourcesto improve topic modeling, they either bear the key weaknessof representing topics merely as distribution over words orphrases [Mei et al., 2014; Boyd-Graber et al., 2007; Newmanet al., 2006] or sacrifice the flexibility of topic models by im-posing a one-to-one binding of topics to pre-defined knowl-edge base (KB) entities [Gabrilovich and Markovitch, 2009;Chemudugunta et al., 2008].

⇤This work was done when the first two authors were at Mi-crosoft Research, Beijing.

This paper aims to bridge the gap, by proposing a newstructured representation of latent topics based on entity tax-onomies from KBs. Figure 1 illustrates an example topicextracted from a news corpus. Entities organized in the hi-erarchical structure carry salient context for human and ma-chine interpretation. For example, the relatively high weightof entity Amy Winehouse can be attributed to the fact thatWinehouse and Houston were both prominent singers whohave passed from drug-related causes. In addition, the vary-ing weights associated with taxonomy nodes ensure flexibil-ity to express the gist of diverse corpora. The new mod-eling scheme poses challenges for inference as both topicsand entities are hidden from observed text and the topics areregularized by hierarchical knowledge. We develop LatentGrounded Semantic Analysis (LGSA), a probabilistic genera-tive model, to infer both topics and entities from text corpora.Each topic is equipped with a random walk over the taxon-omy which naturally integrates the structure to ground thesemantics as well as leverages the highly-organized knowl-edge to capture entity correlations. For accurate entity mod-eling, we augment bag-of-word documents with entity men-tions and incorporate rich textual features of entities fromKBs. To keep inference over large corpora and KBs practical,we use ontology pruning and dynamic programming.

Extensive experiments validate the effectiveness of our ap-proach. LGSA improves topic quality in terms of perplex-ity significantly. We apply the model to identity key enti-ties of documents (e.g., the dominant figures of a news arti-cle). LGSA achieves 10% improvement (precision@1 from80% to 90%) over the best performing competitors, showingstrong potential in semantic search and knowledge acquisi-tion. To our knowledge, this is the first work to combine sta-tistical topic representation with structural entity taxonomy.Our probabilistic model that incorporates rich world knowl-edge provides a potentially useful scheme to accurately in-duce grounded semantics from natural language data.

2 Related WorkTopic modeling Probabilistic topic models such asLDA [Blei et al., 2003] identify latent topics purely based onobserved data. However, it is well known that topic modelsare only a weak model of semantics. Hence, a large amountof recent work has attempted to incorporate domain knowl-edge [Foulds et al., 2015; Yang et al., 2015; Mei et al., 2014;

Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16)

1578

Root Category

Arts

Music Awards

World Music Awards Winners

Britney Spears

Amy Winehouse

Whitney Houston

Death

Death Dead People

Deaths by Drowning

Causes of Death

Drowning Myocardial Infarction

Dead Body Death

Customs

Funeral Burial R.I.P.

Culture

Condolences New Jersey Tourism

Hotel Drug-related Deaths

category

entity

Figure 1: Structured representation of topic “Death of Whitney Houston” from our model. Entities (leaf rectangular nodes) andcategories (internal elliptical nodes) with highest probabilities are shown. The thickness of each node’s outline is proportionalto the weight of that node. The dashed links denote multiple edges that have been consolidated.

Chen and Liu, 2014; Andrzejewski et al., 2011] or relationalinformation [Chang and Blei, 2009; Hu et al., 2015b] asregularization in topic modeling. Yet, a clear shortcomingin these models is that topics are simply modeled as worddistributions without grounded meanings. LGSA mitigatesthis by grounding topics to KB entities. Another line ofresearch attempts to explicitly map document semantics tohuman-defined concepts [Gabrilovich and Markovitch, 2009;Chemudugunta et al., 2008]. These methods assume one-to-one correspondence between topics and a small set of onto-logical concepts. Though enjoying clear meaning, this worksacrifices expressiveness and compactness of latent semanticmodels. LGSA ensures both interpretability and flexibilityby extracting latent yet grounded topics. Topic models havealso been used to model KB entities for the entity linkingtask [Han and Sun, 2012; Kataria et al., 2011] which has adifferent focus from ours.

Using hierarchical knowledge Semantic hierarchies arekey knowledge sources [Resnik, 1995; Hu et al., 2015a].Few generative models have been developed for specific taskswhich integrate hierarchical structures through random walks[Kataria et al., 2011; Hu et al., 2014; Boyd-Graber et al.,2007]. E.g., [Boyd-Graber et al., 2007] exploits WordNet-Walk [Abney and Light, 1999] for word sense disambigua-tion. Our work is distinct in that we use entity taxonomyto construct a representation of topics; moreover, we inferhidden entities from text, leading to unique inference com-plexity. We propose an efficient approach to tackle this issue.Note that our work also differs from hierarchical topic mod-els [Griffiths and Tenenbaum, 2004; Movshovitz-Attias andCohen, 2015] which aim to infer latent hierarchies from datarather than ground latent semantics to existing KBs.

3 Latent Grounded Semantic AnalysisModel Overview: LGSA is an unsupervised probabilisticmodel that goes beyond the conventional word-based topicmodeling, and represents latent topics based on the highly-organized KB entity taxonomies. We first augment the con-ventional bag-of-word documents with entity mentions in or-der to capture salient semantics (§3.1). An entity is modeledas distributions over both mentions and words. Here we lever-

Symbol Description

E,M,V the set of entities, mentions, and wordsD,K,E,M, V #docs, #topics, vocabulary sizes of entities/mentions/words

Md, Vd # of mention and word occurrences in doc dmdj the jth mention in doc dwdl the lth word in doc dzdj the topic associated with mdj

edj the entity associated with mdj

rdj the path in taxonomy associated with mdj

ydl the entity index associated with wdl

✓d, ✓0d multinomial distribution over topics and entities of doc d

⇤k random walk transition distributions of topic k�k multinomial distribution over entities of topic k⌧k probabilities over category nodes of topic k

⌘e, ⇣e multinomial distribution over mentions and words of entity ep⌘e ,p

⇣e base measures of priors over ⌘e and ⇣e extracted from KBs

�⌘e ,�

⇣e concentration parameters (prior strengths)

Table 1: Notations used in this paper.

age entities’ rich textual features from KBs for accurate en-tity modeling (§3.3). Finally, each topic is associated with aroot-to-leaf random walk over the entity taxonomy. This en-dows the topic with a semantic structure, as well as capturesthe valuable entity correlation knowledge from the taxonomy(§3.2). A well-defined generative process combines the abovecomponents in a joint framework (§3.4). Next, we present thedetails of each component (Table 1 lists key notations; Fig-ure 2 illustrates a running example of LGSA; and the graphi-cal model representation of LGSA is shown in Figure 3).

3.1 Document ModelingTopic models usually represent a document as a bag of words.However, language has rich structure and different wordclasses perform different functions in cognitive understand-ing of text. The noun class (e.g., entity mentions in a newsarticle) is an important building block and carries much ofthe salient semantics in a document. Semanticists have of-ten debated the cognitive economy of the noun class [Kempand Regier, 2012]. If every mention had a unique name, thiswould eliminate ambiguities. However, our memory is con-strained and it is impossible to remember all mention names.Hence, ambiguity is essential. Our work grounds mentions toKBs, leading to consistent interpretation of entities.

As the first step, our model augments the bag-of-word rep-resentation with mentions. Each document d 2 {1, . . . , D} is

1579

…

…

Gates, the co-founder of Microsoft, was the wealthiest man in the world.

Bill GatesMicrosoft

Inc.

0.9 0.1

0.6 0.4IT

Root

Sports

Basketball

KobeBryant …

Document Topic based on KB taxonomy

Figure 2: Model overview. Mentions Gates and Microsoft(highlighted) in the document refer to entities in the KB tax-onomy. Words co-founder and wealthiest (underlined) are de-scribing entity Bill Gates. A topic random walk is parameter-ized by the parent-to-child transition probabilities.

↵ ✓d

� ⇤k

zdj rdj, edj mdj

ydl wdl

⌘e

⇣e

�⌘

�⇣

p⇣e

p⌘e

DVd

Md

E

K

Figure 3: Graphical representation of LGSA. The entity edjis the leaf of the path rdj . Document’s entity distribution ✓0

dis derived from other variables and thus does not participatein the generative process directly.

now represented by a set of words wd = {wdl}Ndl=1 as well as

a set of mentions md = {mdj}Mdj=1 occurring in d. Mentions

can be automatically identified using existing mention detec-tion tools. E.g., the document in Figure 2 contains mentions{Gates, Microsoft} and words {co-founder, . . .}.

Each document d is associated with a topic distribution✓d = {✓dk}Kk=1 and an entity distribution ✓0

d = {✓0de}Ee=1,based on which the entity groundings can be identified.LGSA simulates a generative process in which the entitymentions are determined first and the content words comelater to describe the entities’ attributes and actions (e.g., inFigure 2, wealthiest characterizes Gates). This leads to thedifferential treatment of mentions and words in the genera-tive procedure: each mention mdj is associated with a topiczdj and an entity edj (drawn from zdj as described next), whileeach word wdl is associated with an index ydl indicating wdl

is describing the ydl-th mentioned entity (i.e., edydl ).

3.2 Topic Random Walk on Entity TaxonomyWe now present the taxonomy-based modeling of latent top-ics, from which the underlying entities {edj} of the mentions{mdj} are drawn. A KB entity taxonomy is a hierarchicalstructure that encodes rich knowledge of entity correlations,e.g., nearby entities tend to be relevant to the same topics. Tocapture this useful information through a generative proce-dure, we model each topic as a root-to-leaf random walk overthe entity taxonomy. Let E be the set of entities from KB andH be the hierarchical taxonomy where entities are leaf nodes

assigned to one or more categories; categories are further or-ganized into a hierarchical structure in a generic-to-specificmanner. For each category node c, we denote the set of itsimmediate children (subcategories or leaf entities) as C(c).

The topic random walk over H (denoted as ⇤k) for topick is parameterized by a set of parent-to-child transitions, i.e.,⇤k = {⇤k,c}c2H where ⇤k,c = {⇤k,cc0}c02C(c) is the tran-sition distribution from c to its children. Starting from the rootcategory c0, a child is selected according to ⇤kc0 . The pro-cess continues until a leaf entity node is reached. Hence therandom walk assigns each generated entity edj a root-to-leafpath rdj . A desirable property of the random walk is that enti-ties with common ancestors in the hierarchy share sub-pathsstarting at the root and thus tend to have similar generatingprobabilities. This effectively encourages clustering highly-correlated entities and produces semantically coherent topics.For example, entities Bill Gates and Microsoft Inc. in Figure 2share the sub-path from root to category IT, which carries atransition probability of 0.9. Thus the two entities are likely toboth have high generating probabilities in the specific topic,while the less relevant Kobe Bryant will have a low proba-bility. Based on ⇤k, we can compute the probability of therandom walk reaching each of the entities, and hence obtaina distribution over entities, �k. Similarly, for each categorynode c we can compute a probability ⌧kc indicating the pos-sibility of c being included in a random walk path. The set ofparameters {⇤k,�k, ⌧k} together forms a structured repre-sentation of the latent topic k, which has grounded meaning.

3.3 Entity Modeling on Mentions and WordsAs described before, we learn entity representations in bothmention and word spaces. Moreover, since the rich textualfeatures of entities in KBs encode relevance between entitiesand mentions/words, we leverage them to construct informa-tive priors for accurate entity modeling.

Specifically, each entity e 2 E has a distribution overmention vocabulary M, denoted as ⌘e, along with a distri-bution over word vocabulary V , denoted as ⇣e. Intuitively,⌘e captures the relatedness between e and other entities, e.g.,mention Gates tends to have high probability in entity Mi-crosoft Inc.’s mention distribution; while ⇣e characterizes theattributes of entity e, e.g., word wealthiest for entity BillGates. The informative priors over ⌘e and ⇣e are derived fromthe frequency of mentions and words in entity e’s Wikipediapage. Let p⌘

e be the prior mention distribution over ⌘e, witheach dimension p⌘em proportional to the frequency of men-tion m in e’s page. The prior word distribution p⇣

e over ⇣eis built in a similar manner. To reflect the confidence of theprior knowledge, we introduce scaling factors �⌘ and �⇣ witha larger value indicating a greater emphasis on the prior.

Note that in LGSA the mention distribution of an entity(e.g. Microsoft Inc.) can put mass on not only its referringmentions (e.g. Microsoft), but also other related mentions(e.g. Gates). This captures the intuition that, for instance,the observation of Gates can promote the probability of thedocument being about Microsoft Inc.. This differs from pre-vious entity linking methods and improves the detection ofdocument’s key entities, as shown in our empirical studies.

1580

3.4 Generative ProcessWe summarize the generative process of LGSA in Algo-rithm 1 that combines all the above components. Given thementions md and words wd of a document d, each mentionis first assigned a topic according to the topic distribution ✓d.The topics in turn generate entities for each mention throughthe random walks. For each word, one of the above entities isuniformly selected.

Algorithm 1 Generative Process for LGSA• For each topic k = 1, 2, . . . ,K,

1. For each category c 2 H, sample the transition probabili-ties, ⇤kc|� ⇠ Dir(�).

• For each entity e = 1, 2, . . . , E,1. Sample the mention distribution: ⌘e|�⌘,p⌘

e ⇠ Dir(�⌘p⌘e).

2. Sample the word distribution: ⇣e|�⇣ ,p⇣e ⇠ Dir(�⇣p⇣

e).• For each document d = 1, 2, . . . , D,

1. Sample the topic distribution: ✓d|↵ ⇠ Dir(↵).2. For each mention mdj 2 md,

(a) Sample a topic indicator: zdj |✓d ⇠ Multi(✓d).(b) Initialize path rdj = {c0}, and h = 0.(c) While leaf not reached

i. Sample the next node: ch+1 ⇠ Multi(⇤zdj ,ch).ii. If ch+1 is a leaf node, then the corresponding entity

edj = ch+1; otherwise, h = h+ 1.(d) Sample a mention mdj |⌘edj ⇠ Multi(⌘edj ).

3. For each word wdl 2 wd,(a) Sample an index ydl ⇠ Unif(1, . . . ,Md).(b) e0dl := edydl(c) Sample a word wdl|⇣e0dl ⇠ Multi(⇣e0dl).

4 Model InferenceExact inference for LGSA is intractable due to the couplingbetween hidden variables. We exploit collapsed Gibbs sam-pling [Griffiths and Steyvers, 2004] for approximate infer-ence. As a widely used Markov chain Monte Carlo algo-rithm, Gibbs sampling iteratively samples latent variables({z, r, e,y} in LGSA) from a Markov chain whose station-ary distribution is the posterior. The samples are then used toestimate the distributions of interest: {✓,✓0,⇤,�, ⌧ ,⌘, ⇣}.We directly give the sampling formulas and provide the de-tailed derivations in the supplementary materials.

Sampling topic zdj for mention mdj according to:

p(zdj = z|edj = e, r�dj , .)

/ (n(z)d + ↵) ·

Xr(e2r)

p(r|r�dj , zdj = z, .),(1)

where n(z)d denotes the number of mentions in document d

that are associated with topic z. Marginal counts are repre-sented with dots; e.g., n(·)

d is obtained by marginalizing n(z)d

over z. The second term of Eq.(1) is the sum over the prob-abilities of all paths that could have generated entity e, con-ditioned on topic z. Here the probability of a path r is theproduct of the topic-specific transition probabilities along the

path from root c0 to leaf c|r|�1 (i.e. entity e):

p(r|r�dj , zdj = z, .) =

|r|�2Y

h=0

n(z)ch,ch+1 + �

n(z)ch,· + |C(ch)|�

, (2)

where n(z)ch,ch+1 is the number of paths in topic z that go from

ch to ch+1. All the above counters are calculated with themention mdj excluded.

Sampling path rdj and entity edj for mention mdj as:

p(rdj = r, edj = e|zdj = z,mdj = m, .)

/ p(r|r�dj , zdj = z, .) · n(m)e + �⌘p⌘em

n(·)e + �⌘

· (n(e)d + 1

n(e)d

)q(e)d ,

(3)

where n(m)e is the number of times that mention m is gener-

ated by entity e; n(e)d is the number of mentions in d that are

associated with e; and q(e)d =P

l 1(eydl = e) is the numberof words in d that are associated with e. All the counters arecalculated with the mention mdj excluded.

Sampling index ydl for word wdl according to:

p(ydl = y|edy = e, wdl = w, .) / n(w)e + �⇣p⇣ew

n(·)e + �⇣

, (4)

where n(w)e is the number of times that word w is generated

by entity e, and is calculated with wdl excluded.The Dirichlet hyperparameters are set as fixed values: ↵ =

50/K, � = 0.01, a common setting in topic modeling. Weinvestigate the effects of �⌘ and �⇣ in our empirical studies.

Efficient inference in practice: The inference on largetext corpora and KBs can be complicated. To ensure effi-ciency in practice, we use ontology pruning, dynamic pro-gramming, and careful initialization: (a) The total number ofall entities’ paths can be very large, rendering the computa-tion of Eq.(3) for all paths prohibitive. We make the observa-tion that in general only a few entities in E are relevant to adocument, and these are typically ones with their name men-tions occurring in the document [Kataria et al., 2011]. Hence,we select candidate entities for each document using a name-to-entity dictionary [Han and Sun, 2011], and only the pathsof these entities are considered when sampling. Our experi-ments show the approximation has negligible impact on mod-eling performance, while dramatically reducing the samplingcomplexity, making the inference practical. (b) We furtherreduce the hierarchy depth by pruning low-level concrete cat-egory nodes (whose shortest root-to-node path lengths exceeda threshold). We found that such a “coarse” entity ontologyis sufficient to provide strong performance. (c) To computethe probabilities of paths (Eq.(2)) we use dynamic program-ming to avoid redundant computation. (d) We initialize theentity and path assignments to ensure a good starting point.The entity assignment of a mention is sampled from the priorentity-mention distributions p⌘; based on the assignments, apath leading to the respective entity is then sampled accordingto an initializing transition distribution where the probabilityof transitioning from a category c to its child c0 is proportionalto the total frequency of descendant entities of c0.

1581

5 ExperimentsWe evaluate LGSA’s modeling performance on two news cor-pora. We observe that LGSA reduces topic perplexity signif-icantly. In the task of key entity identification, LGSA im-proves over competitors by 10% in precision@1. We also ex-plore the effects of entity textual priors.

DatasetText corpus Wikipedia KB (pruned)

#doc #word #mention #entity #category #layer

TMZ 3.2K 150K(4.6K) 71K(15K) 72K 102K 11NYT 0.3M 130M(169K) 13M(71K) 100K 7.1K 4

Table 2: Statistics of two datasets. The numbers in parenthe-ses are the vocabulary sizes. The average #path to each entityin TMZ and NYT KBs are 300 and 25, respectively.

Datasets: We evaluate on two news corpora (Table 2): (a)TMZ news is collected from TMZ.com, a popular celebritygossip website. Each news article is tagged with one or morecelebrities which serve as ground truth in the task of keyentity identification; (b) NYT news is a widely-used largecorpus from LDC1. For both datasets, we extract the men-tions of each article using a mention annotation tool The WikiMachine2. We use the Wikipedia snapshot of 04/02/2014 asour KB. In Wikipedia, entities correspond to Wikipedia pageswhich are organized as leaf nodes of a category hierarchy. Wepruned irrelevant entities and categories for each dataset.

Baselines: We compare the proposed LGSA with the fol-lowing competitors (Table 3 lists their differences):(a) ConceptTM (CnptTM) [Chemudugunta et al., 2008]employs ontological knowledge by assuming one-to-one cor-respondence between human-defined entities and latent top-ics. Thus each topic has identifiable transparent semantics.(b) Entity-Topic Model (ETM) [Newman et al., 2006] mod-els both words and mentions of documents by word topicand mention topic, respectively. No external knowledgeis incorporated in ETM. (c) Latent Dirichlet Allocation(LDA) [Blei et al., 2003] is a bag-of-words model and rep-resents each latent topic as a word distribution. Following[Gabrilovich and Markovitch, 2009], LDA can be used foridentifying key entities by measuring the similarity betweenthe document’s and the entity Wikipedia page’s topic distri-butions. (d) Explicit Semantic Analysis (ESA) [Gabrilovichand Markovitch, 2009] is a popular Wikipedia-based methodaimed at finding relevant entities as semantics of text. Fea-tures including content words and Wikipedia link structuresare used to measure the relatedness between documents andentities. (e) Mention Annotation & Counting (MA-C). Wemap each mention to its referent entity, and rank the enti-ties by the frequency they are mentioned. The priority ofoccurrence is further incorporated to break the tie. We useThe Wiki Machine in the mention-annotation step. (f) LGSAwithout Hierarchy (LGSA-NH). To directly measure advan-tage of structured topic representation, we design the intrinsiccompetitor that models latent topic as a distribution over en-tities without incorporating the entity hierarchical structure.

1https://www.ldc.upenn.edu2http://thewikimachine.fbk.eu

Topic Perplexity: We evaluate the quality of extracted top-ics by topic perplexity [Blei et al., 2003]. As a widely usedmetric in text modeling, perplexity measures the predictivepower of a model in terms of predicting words in unseenheld-out documents [Chemudugunta et al., 2008]. A lowerperplexity means better generalization performance.

MethodFeatures Tasks

word mention structuredknowledge

topicextraction

key entityidentification

CnptTMp p p p

ETMp p p

LDAp p p

ESAp p p

MA-Cp p p p

LGSA-NHp p p p

LGSAp p p p p

Table 3: Feature and task comparison of different methods

We use 5-fold cross validation testing. Figure 4a and 4bshow the perplexity values on the TMZ and NYT corporarespectively using different number of topics. We see thatLGSA consistently yields the lowest perplexity, indicatingthe highest predictive quality of extracted topics. We fur-ther observe that: (a) ETM and LDA perform inferior toCnptTM and LGSA, showing that without the guidance ofhuman knowledge, purely data-driven method is incapable ofaccurately modeling text latent semantics. (b) Compared toLGSA, CnptTM has an inferior performance in that it boundseach topic with one pre-defined concept, which is not flex-ible enough to represent diverse corpus semantics. LGSAavoids the pitfall by associating an entity distribution witheach topic, which is both expressive and interpretable. (c)Comparing LGSA and LGSA-NH further reveals the advan-tage of the structured topic representation—LGSA reducesperplexity by 6.5% on average. (d) Even without taxonomystructure, LGSA-NH still outperforms the baselines. This isbecause our model goes beyond the bag-of-words assumptionand accounts for the mentions and underlying entities, whichcaptures salient text semantics. (e) On the NYT dataset, LDAand ETM perform best at K = 400, where our method yields17.9% and 5.03% lower perplexity, respectively. This againvalidates the benefits of incorporating world knowledge.

Key Entity Identification: Identifying key entities in doc-uments (e.g., the persons that a news article is mainly about)serves to reveal fine-grained semantics as well as map docu-ments to structured ontologies, which in turn facilitates down-stream applications such as semantic search and documentcategorization. Our next evaluation tries to measure the pre-cision of LGSA in key entity identification. We test on theTMZ dataset since the ground truth (usually a celebrity) isavailable. Given a document d, LGSA infers its entity distri-bution ✓0

d and ranks entities accordingly.Figure 4c shows the Precision@R (proportion of test in-

stances where a correct key entity is included in the top-R pre-dictions) based on 5-fold cross validation. Here, both LGSAand LDA achieves their best performance by setting #topicK = 30. From the figure, we can see that LGSA consis-tently outperforms all other methods, and achieves 90% pre-

1582

(a) (b) (c)

Figure 4: (a) Perplexity on TMZ dataset, (b) Perplexity on NYT, and (c) Precision@R of key entity identification on TMZ.

Games

Game American Football

NBA

SportsSports

Television

ESPNSportBall

Games

Basketball

BasketballKobe

Bryant

Baseball

BaseballL.A.

Dodgers

LeBron James

L.A. Lakers

NBA

Kobe Bryant Absolved in Church Assault Case

Kobe Bryant in the Gym with Manny Pacquiao

San Diego Church: Kobe's Innocent!

LeBron James Just Jumped over a Guy!LeBron Alleged Mishegas at Jewish Basketball Game

Root

Kardashian family

Kris Humphries

Kim Kardashian

Marriage

Divorce

Annulment

Fraud

Marriage

Practice of law

Lawsuit

Lawyer

Kris Has Lawyered up for DivorceThe Annulment Documents

Kim: Kris' Parents Hated MeKim: No Reconciliation

Minnesota

Minnesota

Root

(a) (b)

Figure 5: Topics (a) “Sports” and (b) “Kardashian and Humphries’ Divorce”, showing top entities (by entity distributions �)and categories (by the probabilities of reaching category nodes through the random walks ⇤). Titles of several news are attachedto their top-1 key entities.

cision at rank-1. The results reveal that: (a) MA-C has aninferior performance than LGSA, which can be attributed tothe improper decoupling of candidate selection (i.e., mentionannotation) and ranking (i.e., counting). In particular, for in-stance, though the observation of mention Gates may help tocorrectly annotate mention MS as referring to entity MicrosoftInc. it cannot directly promote the weight of Microsoft Inc.in the document. In contrast, LGSA captures this useful sig-nal by allowing each entity to associate weights with all rel-evant mentions (Sec.3.3). (b) Our proposed model also out-performs ESA and LDA. Indeed, LGSA essentially combinesthese two lines of work (i.e., the explicit and latent semanticrepresentations), by stacking the latent topic layer over the ex-plicit entity knowledge. This ensures the best of both worlds:the flexibility of latent modeling and the interpretability ofexplicit modeling. (c) LGSA-NH is superior over previousmethods while falling behind the full model. This confirmsthe effect of incorporating grounded hierarchical knowledge.

Qualitative Analysis: We now qualitatively investigatethe extracted topics, illustrating the benefits of semantically-grounded modeling as well as revealing potential directionsfor future improvements. Figure 5 shows two example topicsfrom the TMZ corpus. We can see the top-ranked entities andcategories are semantically coherent and the highly-organizedstructure provides rich context and relations between the topentities (e.g., Kobe Bryant and LeBron James are both fromNBA), helping topic interpretation. More importantly, the ex-tracted topics show the benefits of entity grounding in la-

Figure 6: Effect of entity prior strength � on TMZ dataset.

tent semantic modeling. Figure 5 also demonstrates exam-ple news titles and their key entities inferred by LGSA. Thisnaturally links documents to KBs, showing strong potentialin semantic search and automatic knowledge acquisition. Itis also noticeable that there exists no single entity or cate-gory in Wikipedia that directly corresponds to the topic ofKardashian and Humphries’ divorce. In contrast, the fullmeaning is constituted through the combination of a pri-ori unrelated ones. This validates the superior expressive-ness of LGSA compared to CnptTM and ESA which relyon pre-defined concepts. The analysis also reveals some po-tential improvement space of our work. E.g., the actions ofKardashian and Humphries are captured in entity Divorce,while incorporating action representations (e.g., verbs withgrounded meaning) would help to characterize the full se-mantics more directly. We consider this as a future work.

Impact of Entity Prior Strengths: LGSA leverages men-

1583

tion/word frequency of entities in KBs to construct informa-tive priors over mention/word distributions. Here we studythe effect of these entities priors by showing performancevariation with different prior strengths. Figure 6 shows theresults where we have set �⌘ = �⇣ = � for simplicity. Wecan see that LGSA performs best with an modest � value (i.e.10.0) in both tasks. The improvement of performance as �increases in a proper range validates that the textual featuresfrom KBs can improve modeling; while improperly strongpriors can prevent the model from flexibly fitting to the data.

6 Conclusion and Future WorkWe proposed a structured representation of latent topics basedon an entity taxonomy from KB. A probabilistic model,LGSA, was developed to infer both hidden topics and entitiesfrom text corpora. The model integrates structural and textualknowledge from KB, grounding entity mentions to KB. Thisleads to improvements in topic modeling and entity identifi-cation. The grounded topics can be useful in various languageunderstanding tasks, which we plan to explore in the future.

Acknowledgments: Zhiting Hu and Mrinmaya Sachan aresupported by NSF IIS1218282, NSF IIS1447676, AFOSRFA95501010247, and Air Force FA8721-05-C-0003.

References[Abney and Light, 1999] Steven Abney and Marc Light. Hiding a

semantic hierarchy in a markov model. In the Workshop on Un-supervised Learning in NLP, ACL, pages 1–8, 1999.

[Andrzejewski et al., 2011] David Andrzejewski, Xiaojin Zhu,Mark Craven, and Benjamin Recht. A framework for incorpo-rating general domain knowledge into latent dirichlet allocationusing first-order logic. In IJCAI, pages 1171–1177, 2011.

[Blei et al., 2003] David M Blei, Andrew Y Ng, and Michael I Jor-dan. Latent dirichlet allocation. JMLR, 3:993–1022, 2003.

[Boyd-Graber et al., 2007] Jordan L Boyd-Graber, David M Blei,and Xiaojin Zhu. A topic model for word sense disambiguation.In EMNLP-CoNLL, pages 1024–1033, 2007.

[Chaney and Blei, 2012] Allison June-Barlow Chaney andDavid M Blei. Visualizing topic models. In ICWSM, 2012.

[Chang and Blei, 2009] Jonathan Chang and David M Blei. Rela-tional topic models for document networks. In AISTATS, pages81–88, 2009.

[Chang et al., 2009] Jonathan Chang, Sean Gerrish, Chong Wang,Jordan L Boyd-graber, and David M Blei. Reading tea leaves:How humans interpret topic models. In NIPS, pages 288–296,2009.

[Chemudugunta et al., 2008] Chaitanya Chemudugunta, AmericaHolloway, Padhraic Smyth, and Mark Steyvers. Modeling docu-ments by combining semantic concepts with unsupervised statis-tical learning. In The Semantic Web-ISWC 2008, pages 229–244.Springer, 2008.

[Chen and Liu, 2014] Zhiyuan Chen and Bing Liu. Topic modelingusing topics from many domains, lifelong learning and big data.In ICML, 2014.

[Foulds et al., 2015] James Foulds, Shachi Kumar, and Lise Getoor.Latent topic networks: A versatile probabilistic programmingframework for topic models. In ICML, pages 777–786, 2015.

[Gabrilovich and Markovitch, 2009] Evgeniy Gabrilovich andShaul Markovitch. Wikipedia-based semantic interpretation fornatural language processing. JAIR, 34(2):443, 2009.

[Griffiths and Steyvers, 2004] Thomas L Griffiths and MarkSteyvers. Finding scientific topics. PNAS, 101(Suppl 1):5228–5235, 2004.

[Griffiths and Tenenbaum, 2004] DMBTL Griffiths and MIJJBTenenbaum. Hierarchical topic models and the nested Chineserestaurant process. NIPS, 16:17, 2004.

[Han and Sun, 2011] Xianpei Han and Le Sun. A generative entity-mention model for linking entities with knowledge base. In ACL,pages 945–954. ACL, 2011.

[Han and Sun, 2012] Xianpei Han and Le Sun. An entity-topicmodel for entity linking. In EMNLP, pages 105–115. ACL, 2012.

[Hu et al., 2014] Yuening Hu, Ke Zhai, Vladimir Eidelman, andJordan Boyd-Graber. Polylingual tree-based topic models fortranslation domain adaptation. In ACL, 2014.

[Hu et al., 2015a] Zhiting Hu, Poyao Huang, Yuntian Deng,Yingkai Gao, and Eric P Xing. Entity hierarchy embedding. InACL, pages 1292–1300, 2015.

[Hu et al., 2015b] Zhiting Hu, Junjie Yao, Bin Cui, and Eric Xing.Community level diffusion extraction. In SIGMOD, pages 1555–1569. ACM, 2015.

[Kataria et al., 2011] Saurabh S Kataria, Krishnan S Kumar, Ra-jeev R Rastogi, Prithviraj Sen, and Srinivasan H Sengamedu.Entity disambiguation with hierarchical topic models. In KDD,pages 1037–1045. ACM, 2011.

[Kemp and Regier, 2012] Charles Kemp and Terry Regier. Kinshipcategories across languages reflect general communicative prin-ciples. Science, 336(6084):1049–1054, 2012.

[Mei et al., 2014] Shike Mei, Jun Zhu, and Jerry Zhu. Robust reg-bayes: Selectively incorporating first-order logic domain knowl-edge into Bayesian models. In ICML, pages 253–261, 2014.

[Mimno et al., 2009] David Mimno, Hanna M Wallach, JasonNaradowsky, David A Smith, and Andrew McCallum. Polylin-gual topic models. In EMNLP, pages 880–889. ACL, 2009.

[Movshovitz-Attias and Cohen, 2015] Dana Movshovitz-Attiasand William W Cohen. KB-LDA: Jointly learning a knowledgebase of hierarchy, relations, and facts. In ACL, pages 1449–1459,2015.

[Newman et al., 2006] David Newman, Chaitanya Chemudugunta,and Padhraic Smyth. Statistical entity-topic models. In KDD,pages 680–686. ACM, 2006.

[Resnik, 1995] Philip Resnik. Using information content to evalu-ate semantic similarity in a taxonomy. In IJCAI, pages 448–453,1995.

[Song et al., 2011] Yangqiu Song, Haixun Wang, ZhongyuanWang, Hongsong Li, and Weizhu Chen. Short text conceptu-alization using a probabilistic knowledgebase. In IJCAI, pages2330–2336. AAAI Press, 2011.

[Wang et al., 2007] Xuerui Wang, Andrew McCallum, and XingWei. Topical n-grams: Phrase and topic discovery, with an appli-cation to information retrieval. In ICDM, pages 697–702, 2007.

[Wei and Croft, 2006] Xing Wei and W Bruce Croft. LDA-baseddocument models for ad-hoc retrieval. In SIGIR, pages 178–185.ACM, 2006.

[Yang et al., 2015] Yi Yang, Doug Downey, Jordan Boyd-Graber,and Jordan Boyd Graber. Efficient methods for incorporatingknowledge into topic models. In EMNLP, 2015.

1584