Using Entity Information from a Knowledge Base to Improve RelationExtraction

Lan Du1∗, Anish Kumar2, Mark Johnson2 and Massimiliano Ciaramita3

1Faculty of Information Technology, Monash University Clayton, VIC 3800, Australia2Department of Computing, Macquarie University Sydney, NSW 2109, Australia

3Google Research, Zurich, Switzerland

Abstract

Relation extraction is the task of ex-tracting predicate-argument relationshipsbetween entities from natural languagetext. This paper investigates whether back-ground information about entities avail-able in knowledge bases such as FreeBasecan be used to improve the accuracy ofa state-of-the-art relation extraction sys-tem. We describe a simple and effectiveway of incorporating FreeBase’s notabletypes into a state-of-the-art relation extrac-tion system (Riedel et al., 2013). Experi-mental results show that our notable type-based system achieves an average 7.5%weighted MAP score improvement. Tounderstand where the notable type infor-mation contributes the most, we performa series of ablation experiments. Resultsshow that the notable type information im-proves relation extraction more than NERlabels alone across a wide range of entitytypes and relations.

1 Introduction

The goal of relation extraction is to extract rela-tional information about entities from a large textcollection. For example, given the text “MichaelBay, the director of Transformers, visited Parisyesterday,” a relation extraction system might ex-tract the relationship film director(Michael Bay,Transformers). These tuples can be then used toextend a knowledge base. With the increase in theamount of textual data available on the web, rela-tion extraction has gained wide applications in in-formation extraction from both general newswiretexts and specialised document collections such asbiomedical texts (Liu et al., 2007).

∗This work was partially done while Lan Du was at Mac-quarie University.

A typical relation extraction system functions asa pipeline, first performing named entity recog-nition (NER) and entity disambiguation to linkthe entity mentions found in sentences to theirdatabase entries (e.g., “Michael Bay” and “Trans-formers” would both be linked to their respectivedatabase ids). Then the context in which these en-tity mentions co-occur is used to predict the re-lationship between the entities. For example, thepath in a syntactic parse between two mentions ina sentence can be used as a feature to predict therelation holding between the two entities. Contin-uing our example, the text pattern feature X-the-director-of-Y (or a corresponding parse subtreefragment) might be used to predict the databaserelation film director(X,Y). In such a pipeline ar-chitecture, information about the entities from thedatabase is available and can be used to help de-termine the most appropriate relationship betweenthe entities. The goal of this paper is to identifywhether that information is useful in a relation ex-traction task, and study such information about theentities with a set of ablation experiments.

We hypothesise that information from databaseentries can play the role of background knowl-edge in human sentence comprehension. There isstrong evidence that humans use world knowledgeand contextual information in both syntactic andsemantic interpretation (Spivey-Knowlton and Se-divy, 1995), so it is reasonable to expect a machinemight benefit from it as well. Continuing with ourexample, if our database contained the informationthat one particular entity with the name MichaelBay had died a decade before the movie Trans-formers was released, then it might be reason-able to conclude that this particular individual wasunlikely to have directed Transformers. Clearly,modelling all the ways in which such backgroundinformation about entities might be used would beextremely complex. This paper explores a simpleway of using some of the background information

Lan Du, Anish Kumar, Mark Johnson and Massimiliano Ciaramita. 2015. Using Entity Information from aKnowledge Base to Improve Relation Extraction . In Proceedings of Australasian Language TechnologyAssociation Workshop, pages 31−38.

about entities available in FreeBase (Bollacker etal., 2008).

Here we focus on one particular kind of back-ground information about entities — the informa-tion encoded in FreeBase’s notable types. Free-Base’s notable types are simple atomic labelsgiven to entities that indicate what the entity isnotable for, and so serve as a useful informa-tion source that should be relatively easy to ex-ploit. For example, the search results for “JimJones” given by FreeBase contains several dif-ferent entities. Although they all have the samename entity (NE) category PERSON, their no-table types are different. The notable typesfor the top 4 “Jim Jones” results are organiza-tion/organization founder, music/composer, base-ball/baseball player and government/politician. Itis clear that the notable type information providesmuch finer-grained information about “Jim Jones”than just the NE category. It is reasonable to ex-pect that notable types would be useful for relationextraction; e.g., the politician Jim Jones is likelyto stand for election, while the baseball player islikely to be involved in sport activities.

We extend one state-of-the-art relation extrac-tion system of Riedel et al. (2013) to exploit thisnotable type information. Our notable type ex-tensions significantly improve the mean averagedprecision (MAP) by 7.5% and the weighted MAPby 6% over a strong state-of-the-art baseline. Witha set of ablation experiments we further evaluatehow and where the notable type information con-tributes to relation extraction.The rest of this paperis structured as follows. The next section describesrelated work on relation extraction. Section 3 de-scribes how a state-of-the-art relation extractionsystem can be extended to exploit the notable typeinformation available in FreeBase. Section 4 spec-ifies the inference procedures used to identify thevalues of the model parameters, while section 5explains how we evaluate our models and presentsa systematic experimental comparison of the mod-els by ablating the notable type in different waysbased on entities’ NE categories. Section 6 con-cludes the paper and discusses future work.

2 Related work

Most approaches to relation extraction are eithersupervised or semi-supervised. Supervised ap-proaches require a large set of manually annotatedtext as training data (Culotta and Sorensen, 2004),

but creating these annotations is both expensiveand error-prone. Semi-supervised approaches, bycontrast, rely on correlations between relationsand other large data sources.

In relation extraction, most semi-supervised ap-proaches use distant supervision, which alignsfacts from a large database, e.g., Freebase, to un-labelled text by assuming some systematic rela-tionship between the documents and the database(Bunescu and Mooney, 2007; Mintz et al., 2009;Riedel et al., 2010; Yao et al., 2010). Typically,we assume that (a) an entity linker can reliablyidentify entity mentions in the text and map themto the corresponding database entries, and (b) forall tuples of entities that appear in a relation inthe database, if we observe that entity tuple co-occurring in a suitable linguistic construction (e.g.,a sentence) then that construction expresses thedatabase relationship about those entities. Pre-vious work (Weston et al., 2013; Riedel et al.,2013; Bordes et al., 2013; Chang et al., 2014)has shown that models leveraging rich informationfrom database often yield improved performance.

In this work we are particularly interested in ex-ploring entity type information in relation extrac-tion, as semantic relations often have selectionalpreference over entity types. Yao et al. (2010),Singh et al. (2013), Yao et al. (2013), Koch et al.(2014) and Chang et al. (2014) have shown that theuse of type information, e.g., NE categories, sig-nificantly improves relation extraction. Our workhere is similar except that we rely on Freebase’snotable types, which provide much finer-grainedinformation about entities. One of the challengesin relation extraction, particularly when attempt-ing to extract a large number of relations, is to gen-eralise appropriately over both entities and rela-tions. Techniques for inducing distributed vector-space representations can learn embeddings ofboth entities and relations in a high-dimensionalvector space, providing a natural notion of simi-larity (Socher et al., 2013) that can be exploited inthe relation extraction task (Weston et al., 2013).Instead of treating notable types as features Lingand Weld (2012), here we learn distributed vector-space representations for notable types as well asentities, entity tuples and relations.

3 Relation extraction as matrixcompletion

Riedel et al. (2013) formulated the relation extrac-

32

tion task as a matrix completion problem. In thissection we extend this formulation to exploit no-table types in a simple and effective way. Specif-ically, we follow Riedel et al. (2013) in assum-ing that our data O consists of pairs 〈r, t〉, wherer ∈ R is a relation and t ∈ T is a tuple of en-tities. The tuples are divided into training andtest depending on which documents they are ex-tracted from. In this paper, the tuples in T arealways pairs of entities, but nothing depends onthis. There are two kinds of relations inR: syntac-tic patterns found in the document collection, andthose appearing in the database (including targetrelations for extraction). For our notable type ex-tension we assume we have a function n that mapsan entity e to its FreeBase notable type n(e).

For example, given text “Michael Bay, the di-rector of Transformers, visited Paris yesterday”we extract the pair 〈r, t〉 where t = 〈Michael Bay,Transformers〉 and r = X-the-director-of-Y (actu-ally, the path in a dependency parse between thenamed entities). From FreeBase we extract thepair 〈r′, t〉 where r′ = film/director. FreeBasealso tells us that n(Michael Bay) = Person andn(Transformers) = Film. Our goal is to learn amatrix Θ whose rows are indexed by entity tuplesin T and whose columns are indexed by relationsin R . The entry θt,r is the log odds of relationr ∈ R holding of tuple t ∈ T , or, equivalently,the probability that relation r holds of tuple t isgiven by σ(θt,r), where σ is the logistic function:σ(x) = (1 + e−x)−1.

Riedel et al. (2013) assume that Θ is the sum ofthree submodels: Θ = ΘN + ΘF + ΘE, whereΘN is the neighbourhood model, ΘF is the latentfeature model and ΘE is the entity model (thesewill be defined below). Here we extend these sub-models using FreeBase’s notable types.

3.1 A notable type extension to theneighbourhood model

The neighbourhood model ΘN captures depen-dencies between the syntactic relations extractedfrom the text documents and the database relationsextracted from FreeBase. This is given by:

θNr,t =∑

〈r′,t〉∈O\{〈r,t〉}

wr,r′ ,

whereO is the set of relation/tuple pairs in the dataandO\{〈r, t〉} isO with the tuple 〈r, t〉 removed.w is a matrix of parameters, where wr,r′ is a real-valued weight with which relation r′ “primes” re-

lation r that will be learnt from the training data.The neighbourhood model can be regarded as pre-dicting an entry θr,t by using entries along thesame row. It functions as a logistic regression clas-sifier predicting the log odds of a FreeBase rela-tion r applying to the entity tuple t using as fea-tures the syntactic relations r′ that hold of t.

Our notable type extension to the neigh-bourhood model enriches the syntactic pat-terns in the training data O with notabletype information. For example, if there isa syntactic pattern for X-director-of-Y inour training data (say, as part of the tuple〈X-director-of-Y, 〈Michael Bay, Transformers〉〉),then we add a new syntactic pattern〈Person(X)-director-of-Film(Y)〉, where Per-son and Film are notable types and addthe tuple 〈Person(X)-director-of-Film(Y),〈Michael Bay, Transformers〉〉 to our data O.Each new relation corresponds to a new col-umn in our matrix completion formulation.More precisely, the new relations are mem-bers of the set N = {〈r, n(t)〉 : 〈r, t〉 ∈ O},where n(t) is the tuple of notable types cor-responding to the entity tuple t. For example,if t = 〈Michael Bay, Transformers〉 thenn(t) = 〈Person, Film〉. Then the notable typeextension of the neighbourhood model is:

θN′

r,t =∑

〈r′,t〉∈O\{〈r,t〉}

wr,r′ + w′r,〈r′,n(t)〉

where w′ is a matrix of weights relating the rela-tions N to the target FreeBase relation r.

3.2 A notable type extension to the latentfeature model

The latent feature model generalises over relationsand entity tuples by associating each of them witha 100-dimensional real-valued vector. Intuitively,these vectors organise the relations and entity tu-ples into clusters where conceptually similar rela-tions and entity tuples are “close,” while those thatare dissimilar are far apart. In more detail, eachrelation r ∈ R is associated with a latent featurevector ar of size K = 100. Similarly, each entitytuple t ∈ T is also associated with a latent featurevector vt of size K as well. Then the latent fea-ture score for an entity tuple t and relation r is justthe dot product of the corresponding relation andentity tuple vectors, i.e.: θFr,t = ar · vt.

We extend the latent feature model by associat-ing a new latent feature vector with each notable

33

type sequence observed in the training data, anduse this vector to enrich the vector-space repre-sentations of the entity tuples. Specifically, letT ′ = {n(t) : t ∈ T } be the set of notable typetuples for all of the tuples in T , where n(t) is thetuple of notable types corresponding to the tupleof entities t as before. We associate each tuple ofnotable types t′ ∈ T ′ with a latent feature vectorv′t′ of dimensionality K. Then we define the no-table type extension to the latent feature model as:

θF′

r,t = ar · (vt + v′n(t)) .

This can be understood as associating each entitytuple t ∈ T with a pair of latent feature vectors vtand vn(t). The vector vn(t) is based on the notabletypes of the entities, so it can capture generalisa-tions over those notable types. The L2 regularisa-tion employed during inference prefers latent fea-ture vectors in which vt and v′n(t) are small, thusencouraging generalisations which can be stated interms of notable types to be captured by v′n(t).

3.3 A notable type extension of the entitymodel

The entity model represents an entity e with a K-dimensional (K = 100) feature vector ue. Sim-ilarly, the ith argument position of a relation r isalso represented by a K-dimensional feature vec-tor dr,i. The entity model associates a score θEr,twith a relation r ∈ R and entity tuple t ∈ Tas follows: θEr,t =

∑|t|i=1 dr,i · uti , where |t| is

the arity of (i.e., number of elements in the entitytuple t), ti is the ith entity in the entity tuple t,and dr,i and uti are K-dimensional vectors asso-ciated with the ith argument slot of relation r andthe entity ti respectively. The intuition is that thelatent feature vectors of co-occurring entities andargument slots should be close to each other in theK-dimensional latent feature space, while entitiesand argument slots that do not co-occur should befar apart.

Our notable type extension of the entity modelis similar to our notable type extension of the la-tent feature model. We associate each notable typem with a K-dimensional feature vector u′m, anduse those vectors to define the entity model score.Specifically, the entity model score is defined as:

θE′

r,t =

|t|∑i=1

dr,i ·(uti + u′n(ti)

),

where n(e) is the notable type for entity e and |t| isthe length of tuple t. The L2 regularisation againshould encourage generalisations that can be ex-

pressed in terms of notable types to be encoded inthe u′n(ti) latent feature vectors.

4 Inference for model parameters

The goal of inference is to identify the valuesof the model’s parameters, i.e., w,a,v,d and uin the case of the Riedel et al model, and theseplus w′,v′ and u′ in the case of the notabletype extensions. The inference procedure is in-spired by Bayesian Personalised Ranking (Rendleet al., 2009). Specifically, while the true value ofθr,t is unknown, it’s reasonable to assume that if〈r, t+〉 ∈ O (i.e., is observed in the training data)then θr,t+ > θr,t− for all 〈r, t−〉 6∈ O (i.e., notobserved in the training data). Thus the trainingobjective is to maximise

` =∑

〈r,t+〉∈O

∑〈r,t−〉6∈O

`〈r,t+〉,〈r,t−〉

where: `〈r,t+〉,〈r,t−〉 = log σ(θr,t+ − θr,t−), andθr,t = θNr,t + θFr,t + θEr,t or θr,t = θN

′r,t + θF

′r,t + θE

′r,t,

depending on whether the submodels with notabletype extensions are used. The objective function` is then maximised by using stochastic gradientascent. The stochastic gradient procedure sweepsthrough the training data, and, for each observedtuple 〈r, t+〉 ∈ O, samples a negative evidence tu-ple 〈r, t−〉 6∈ O not in the training data, adjustingweights to prefer the observed tuple.

In our experiments below we ran stochastic gra-dient ascent with a step size of 0.05 and an L2regulariser constant of 0.1 for the neighbourhoodmodel and 0.01 for the latent feature and entitymodels (we used the same regulariser constants formodels both with and without the notable type ex-tensions). We ran 2,000 sweeps of stochastic gra-dient ascent.

5 Experimental evaluation

We used a set of controlled experiments to seeto what extent the notable type information im-proves the state-of-the-art relation extraction sys-tem. We used the New York Times corpus (Sand-haus, 2008) in our experiments, assigning articlesfrom the year 2000 as the training corpus and thearticles from 1990 to 1999 for testing. The entitytuples T were extracted from the New York Timescorpus (tuples that did not appear at least 10 timesand also appear in one of the FreeBase relationswere discarded). The relations R are either syn-tactic patterns found in the New York Times cor-pus, FreeBase relations, or (in our extension) no-

34

table types extracted from FreeBase. Our evalua-tion focuses on 19 FreeBase relations, as in Riedelet al. (2013).

5.1 Notable type identificationOur extension requires a FreeBase notable typefor every entity mention, which in turn requiresa Freebase entity id because a notable type is aproperty associated with entities in FreeBase. Wefound the entity id for each named entity as fol-lows. We used the FreeBase API to search for thenotable type for each named entity mentioned inthe training or test data. In cases where several en-tities were returned, we used the notable type ofthe first entity returned by the API. For example,the FreeBase API returns two entities for the string“Canada:” a country and a wine (in that order), sowe use the notable type “country” for “Canada” inour experiments. This heuristic is similar to themethod of choosing the most likely entity id for astring, which provides a competitive baseline forentity linking (Hoffart et al., 2011).

5.2 Evaluation procedureAfter the training procedure is complete and wehave estimates for the model’s parameters, we canuse these to compute estimates for the log odds θr,tfor the test data. These values quantify how likelyit is that the FreeBase relation r holds of an entitytuple t from the test set, according to the trainedmodel.

In evaluation we follow Riedel et al. (2013) andtreat each of the 19 relations r as a query, andevaluate the ranking of the entity tuples t returnedaccording to θr,t. For each relation r we poolthe highest-ranked 100 tuples produced by eachof the models and manually evaluate their accu-racy (e.g., by inspecting the original document ifnecessary). This gives a set of results that can beused to calculate a precision-recall curve. Aver-aged precision (AP) is a measure of the area underthat curve (higher is better), and mean average pre-cision (MAP) is average precision averaged overall of the relations we evaluate on. Weighted MAPis a version of MAP that weights each relation bythe true number of entity tuples for that relation(so more frequent relations count more).

An unusual property of this evaluation is thatincreasing the number of models being evaluatedgenerally decreases their MAP scores: as we eval-uate more models, the pool of “true” entity tuplesfor each relation grows in size and diversity (recall

Relation # NF NFT NFE NFET

person/company 131 0.83 0.89 0.83 0.86location/containedby 88 0.68 0.69 0.68 0.69person/nationality 51 0.11 0.55 0.15 0.45author/works written 38 0.51 0.53 0.57 0.53person/parents 34 0.14 0.31 0.11 0.28parent/child 31 0.48 0.58 0.49 0.58person/place of birth 30 0.51 0.48 0.56 0.57person/place of death 22 0.75 0.77 0.75 0.77neighbourhood/neighbourhood of 17 0.48 0.55 0.52 0.54broadcast/area served 8 0.21 0.41 0.26 0.30company/founders 7 0.46 0.27 0.40 0.28team owner/teams owned 6 0.21 0.25 0.25 0.25team/arena stadium 5 0.06 0.07 0.06 0.09film/directed by 5 0.21 0.25 0.24 0.35person/religion 5 0.20 0.28 0.21 0.23composer/compositions 4 0.42 0.44 0.06 0.42sports team/league 4 0.70 0.62 0.63 0.64film/produced by 3 0.17 0.30 0.12 0.26structure/architect 2 1.00 1.00 1.00 1.00MAP 0.43 0.49 0.42 0.48Weighted MAP 0.55 0.64 0.56 0.62

Table 1: Averaged precision and mean aver-age precision results. The rows correspond toFreeBase relations, and the columns indicate thecombination of sub-models (N = neighbourhoodmodel, F = latent feature model, E = entity model).The superscript “T ” indicates the combined mod-els that incorporate the notable type extensions,and the # column gives the number of true facts.

0.5 Recall

0 0.1 0.2 0.3 0.4 0.6 0.7 0.8 0.9 1

Precision

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1NFNFTNFENFET

Figure 1: Averaged 11-point precision-recallcurve for the four models shown in Table 1.

that this pool is manually constructed by manuallyannotating the highest-scoring tuples returned byeach model). Thus in general the recall scores ofthe existing models are lowered as the number ofmodels increases.

5.3 Experiments with Notable Types

We found we obtained best performance from themodel that incorporates all submodels (which wecall NFET ) and from the model that only incorpo-rates the Neighbourhood and Latent Feature sub-models (which we call NFT ), so we concentrateon them here. Table 1 presents the MAP andweighted MAP scores for these models on the 19FreeBase relations in the testing set.

The MAP scores are 6% higher for both NFT

and NFET , and the weighted MAP scores are 9%and 6% higher for NFT and NFET respectively.

35

Relation # NE

NFE

T

NE

+P

NE

+L

NE

+O

NE

+M

person/place of birth 30 0.52 0.57 0.54 0.50 0.50 0.54author/works written 38 0.57 0.53 0.61 0.56 0.57 0.49team/arena stadium 5 0.08 0.09 0.10 0.09 0.07 0.09composer/compositions 4 0.35 0.42 0.51 0.37 0.35 0.45person/company 131 0.81 0.86 0.84 0.82 0.83 0.86film/directed by 5 0.30 0.35 0.41 0.27 0.27 0.41neighbourhood/neighbourhood of 17 0.59 0.54 0.59 0.49 0.59 0.62film/produced by 3 0.20 0.26 0.29 0.18 0.19 0.40person/religion 5 0.22 0.23 0.21 0.22 0.28 0.53location/containedby 88 0.66 0.69 0.68 0.64 0.64 0.70sports team/league 4 0.53 0.64 0.54 0.52 0.75 0.75person/parents 34 0.33 0.28 0.30 0.32 0.35 0.34parent/child 31 0.55 0.58 0.56 0.55 0.59 0.56person/place of death 22 0.71 0.77 0.74 0.74 0.78 0.72company/founders 7 0.22 0.28 0.28 0.21 0.29 0.22team owner/teams owned 6 0.34 0.25 0.27 0.34 0.36 0.35person/nationality 51 0.19 0.45 0.23 0.50 0.20 0.21broadcast/area served 8 0.32 0.30 0.33 0.38 0.31 0.29structure/architect 2 1.00 1.00 1.00 1.00 1.00 1.00MAP 0.45 0.48 0.48 0.46 0.47 0.50Weighted MAP 0.57 0.62 0.59 0.60 0.58 0.60

Table 2: Results of ablation experiments on theNFET model. The columns correspond to experi-ments, and the column labels are explained in Ta-ble 3.

A sign test shows that the difference between themodels with notable types and those without thenotable types is statistically significant (p < 0.05).Clearly, the notable type extensions significantlyimprove the accuracy of the existing relation ex-traction models. Figure 1 shows an averaged 11-point precision-recall curve for these four mod-els. This makes clear that across the range ofprecision-recall trade-offs, the models with no-table types offer the best performance.

5.4 Ablation Experiments

We performed a set of ablation experiments to de-termine exactly how and where the notable typeinformation improves relation extraction. In theseexperiments entities are divided into 4 “named en-tity” (NE) classes, and we examine the effect ofjust providing notable type information for the en-tities of a single NE class. The 4 NE classes weused were PERSON, LOCATION, ORGANISA-TION, and MISC (miscellaneous). We classifiedall entities into these four categories using theirFreeBase types, which provide a more coarse-grained classification than notable types. For ex-ample, if an entity has a FreeBase “people/person”type, then we assigned it to the NE class PER-SON; if an entity has a “location/location” type,then its NE class is LOCATION; and if an en-tity has a “organisation/organisation” type, thenits NE class is ORGANISATION. All entities notclassified as PERSON, LOCATION, or ORGAN-ISATION were labelled MISC.

We ran a set of ablation experiments as fol-

Ablation setting DescriptionNE All entities are labelled with their NE class instead of

their notable type.NE+P Only PERSON entities have notable type information;

the notable type of other entities is replaced with theirNE class.

NE+L Only LOCATION entities have notable type informa-tion; the notable type of other entities is replaced withtheir NE class.

NE+O Only ORGANISATION entities have notable type in-formation; the notable type of other entities is replacedwith their NE class.

NE+M Only MISC entities have notable type information; thenotable type of other entities is replaced with their NEclass.

Table 3: Descriptions of the ablation experimentsin Table 2.

lows. For each NE class c in turn, we replacedthe notable type information for entities not clas-sified as c with their NE class. For example, whenc = PERSON, only entities with the NE labelPERSON had notable type information, and thenotable types of all other entities was replacedwith their NE labels. Table 3 lists the different ab-lation experiments. The ablation experiments aredesigned to study which NE classes the notabletypes help most on. The results are reported in Ta-ble 2. The results clearly indicate that different re-lations benefit from the different kinds of notabletype information about entities.

Column “NE+P” shows that relationssuch as “author/works written”, “com-poser/compositions” and ”film/directed by”benefit the most from notable type informationabout PERSONs. We noticed that there are about43K entities classified as PERSON, which in-cludes 8,888 book authors, 802 music composers,1212 film directors, etc. These entities have 214distinct notable types. Our results show that it ishelpful to distinguish the PERSON entities withtheir notable types for relations involving profes-sions. For example, not all people are authors, soknowing that a person is an author increases theaccuracy of extracting “author/works written”.Similarly, Column “NE+L” shows that “per-son/nationality” and “broadcast/area served”gain the most from the notable type informationabout locations. There are about 8.5K entitiesclassified as LOCATION, which includes 4807city towns, 301 countries, and so on. Thereare 170 distinct notable types for LOCATIONentities.

Column “NE+O” shows that the notable typeinformation about ORGANISATION entities im-proves the accuracy of extracting relations involv-ing organisations. Indeed, there are more than

36

3K business companies and 200 football teams.Notable type information about organisations im-proves extraction of the “parent/child” relation be-cause this relation involves entities such as com-panies. For example, in our corpus the sentence“CNN, a unit of the Turner Broadcasting, saysthat 7000 schools have signed up for The News-room” expresses the parent/child(Turner Broad-casting, CNN) relation.

The ablation results in Column “NE+M” showthat information about MISC entities is most use-ful of all, as this ablation experiment yielded thehighest overall MAP score. There are about 13.5Kentities labelled MISC. The most frequent no-table types for entities in the MISC NE class are“film/film” and “book/book”. Therefore it is rea-sonable that notable type information for MISCentities would improve AP scores for relationssuch as “film/directed by” and “person/religion”.For example, “George Bush reached a turningpoint in his life and became a born-again Chris-tian” is an example of the “person/religion” rela-tion, and it’s clear that it is useful to know that“born-again Christian” belongs to the religion no-table type. The “sports team/league” relation isinteresting because it performs best with notabletype information for entities in the ORGANISA-TION or MISC NE classes. It turns out thatroughly half the sports teams are classified as OR-GANISATIONs and half are classified as MISC.The sports teams that are classified as MISC aremissing the “organisation/organisation” type intheir FreeBase entries, otherwise they would beclassified as ORGANISATIONs.

In summary, the ablation results show that thecontribution of notable type information dependson the relation being extracted. The result demon-strates that relations involving organisations bene-fits from the notable type information about theseorganisations. It also demonstrates that certain re-lations benefit more from notable type informationthan others. Further research is needed understandsome of the ablation experiment results (e.g., whydoes person/place of death perform best with no-table type information about ORGANISATIONs?)

6 Conclusion and future work

In this paper we investigated the hypothesis thatbackground information about entities present in alarge database such as FreeBase can be useful forrelation extraction. We modified a state-of-the-art

relation extraction system (Riedel et al., 2013) byextending each of its submodels to exploit the “no-table type” information about entities available inFreeBase. We demonstrated that these extensionsimprove the MAP score by 6% and the weightedMAP score by 7.5%, which is a significant im-provement over a strong baseline. Our ablationexperiments showed that the notable type informa-tion improves relation extraction more than NERtags across a wide range of entity types and rela-tions.

In future work we would like to develop meth-ods for exploiting other information available inFreeBase to improve a broad range of naturallanguage processing and information extractiontasks. We would like to explore ways of exploit-ing entity information beyond (distant) supervi-sion approaches, for example, in the direction ofOpenIE (Wu and Weld, 2010; Fader et al., 2011;Mausam et al., 2012). The temporal informationin a large database like FreeBase might be es-pecially useful for named entity linking and re-lation extraction: e.g., someone that has died isless likely to release a hit single. In summary, webelieve that there are a large number of ways inwhich the rich and diverse information present inFreeBase might be leveraged to improve naturallanguage processing and information retrieval, andexploiting notable types is just one of many possi-ble approaches.

Acknowledgments

This research was supported by a Google awardthrough the Natural Language UnderstandingFocused Program, and under the AustralianResearch Council’s Discovery Projects fund-ing scheme (project numbers DP110102506 andDP110102593).

ReferencesKurt Bollacker, Colin Evans, Praveen Paritosh, Tim

Sturge, and Jamie Taylor. 2008. Freebase: a col-laboratively created graph database for structuringhuman knowledge. In Proceedings of the ACM SIG-MOD international conference on Management ofdata, pages 1247–1250.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.2013. Translating embeddings for modeling multi-relational data. In Advances in Neural InformationProcessing Systems 26, pages 2787–2795.

37

Razvan Bunescu and Raymond Mooney. 2007. Learn-ing to extract relations from the web using mini-mal supervision. In Proceedings of the 45th An-nual Meeting of the Association of ComputationalLinguistics, pages 576–583.

Kai-Wei Chang, Wen-tau Yih, Bishan Yang, andChristopher Meek. 2014. Typed tensor decom-position of knowledge bases for relation extrac-tion. In Proceedings of the 2014 Conference onEmpirical Methods in Natural Language Processing(EMNLP), pages 1568–1579.

Aron Culotta and Jeffrey Sorensen. 2004. Dependencytree kernels for relation extraction. In Proceedingsof the 42nd Meeting of the Association for Compu-tational Linguistics (ACL’04), pages 423–429.

Anthony Fader, Stephen Soderland, and Oren Etzioni.2011. Identifying relations for open information ex-traction. In Proceedings of the Conference of Em-pirical Methods in Natural Language Processing(EMNLP ’11), pages 1535–1545.

Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bor-dino, Hagen Furstenau, Manfred Pinkal, Marc Span-iol, Bilyana Taneva, Stefan Thater, and GerhardWeikum. 2011. Robust disambiguation of namedentities in text. In Proceedings of the 2011 Con-ference on Empirical Methods in Natural LanguageProcessing (EMNLP) , pages 782–792.

Mitchell Koch, John Gilmer, Stephen Soderland, andDaniel S. Weld. 2014. Type-aware distantly su-pervised relation extraction with linked arguments.In Proceedings of the 2014 Conference on Em-pirical Methods in Natural Language Processing(EMNLP), pages 1891–1901.

Xiao Ling and Daniel S Weld. 2012. Fine-grained en-tity recognition. In AAAI.

Yudong Liu, Zhongmin Shi, and Anoop Sarkar. 2007.Exploiting rich syntactic information for relationextraction from biomedical articles. In HumanLanguage Technologies 2007: The Conference ofthe North American Chapter of the Associationfor Computational Linguistics, NAACL-Short’07,pages 97–100.

Mausam, Michael Schmitz, Stephen Soderland, RobertBart, and Oren Etzioni. 2012. Open language learn-ing for information extraction. In Proceedings ofthe 2012 Joint Conference on Empirical Methodsin Natural Language Processing and ComputationalNatural Language Learning, pages 523–534.

Mike Mintz, Steven Bills, Rion Snow, and Daniel Ju-rafsky. 2009. Distant supervision for relation ex-traction without labeled data. In Proceedings of theJoint Conference of the 47th Annual Meeting of theACL and the 4th International Joint Conference onNatural Language Processing of the AFNLP, pages1003–1011.

Steffen Rendle, Christoph Freudenthaler, Zeno Gant-ner, and Lars Schmidt-Thieme. 2009. Bpr:Bayesian personalized ranking from implicit feed-back. In Proceedings of the Twenty-Fifth Confer-ence on Uncertainty in Artificial Intelligence, pages452–461.

Sebastian Riedel, Limin Yao, and Andrew McCal-lum. 2010. Modeling relations and their men-tions without labeled text. In Proceedings of the2010 European Conference on Machine Learningand Knowledge Discovery in Databases: Part III,ECML PKDD’10, pages 148–163.

Sebastian Riedel, Limin Yao, Andrew McCallum, andBenjamin M. Marlin. 2013. Relation extractionwith matrix factorization and universal schemas. InProceedings of the 2013 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 74–84.

Evan Sandhaus. 2008. The new york times annotatedcorpus. Linguistic Data Consortium, Philadelphia.

Sameer Singh, Sebastian Riedel, Brian Martin, Jiap-ing Zheng, and Andrew McCallum. 2013. Jointinference of entities, relations, and coreference. InProceedings of the 2013 Workshop on AutomatedKnowledge Base Construction, pages 1–6.

Richard Socher, Danqi Chen, Christopher D. Manning,and Andrew Ng. 2013. Reasoning with neural ten-sor networks for knowledge base completion. InAdvances in Neural Information Processing Systems(NIPS 26), pages 926–934.

Michael Spivey-Knowlton and Julie Sedivy. 1995. Re-solving attachment ambiguities with multiple con-straints. Cognition, 55:227–267.

Jason Weston, Antoine Bordes, Oksana Yakhnenko,and Nicolas Usunier. 2013. Connecting languageand knowledge bases with embedding models for re-lation extraction. In Proceedings of the 2013 Con-ference on Empirical Methods in Natural LanguageProcessing, pages 1366–1371.

Fei Wu and Daniel S. Weld. 2010. Open informationextraction using wikipedia. In Proceedings of the48th Annual Meeting of the Association for Compu-tational Linguistics, ACL’10, pages 118–127.

Limin Yao, Sebastian Riedel, and Andrew McCallum.2010. Collective cross-document relation extractionwithout labelled data. In Proceedings of the 2010Conference on Empirical Methods in Natural Lan-guage Processing, pages 1013–1023.

Limin Yao, Sebastian Riedel, and Andrew McCallum.2013. Universal schema for entity type prediction.In Proceedings of the 2013 Workshop on AutomatedKnowledge Base Construction, pages 79–84.

38