Hot Stuff at Cold Start: HLTCOE participation at TAC 2014

Tim FininUniversity of Maryland,

Baltimore County

Paul McNameeJohns Hopkins University

HLTCOE

Dawn LawrieLoyola University

Maryland

James MayfieldJohns Hopkins University

HLTCOE

Craig HarmanJohns Hopkins University

HLTCOE

AbstractThe JHU HLTCOE participated in the ColdStart task in this year’s Text Analysis Con-ference Knowledge Base Population evalu-ation. This is our third year of participa-tion in the task, and we continued our re-search with the KELVIN system. We sub-mitted experimental variants that explore useof forward-chaining inference, slightly moreaggressive entity clustering, refined multi-ple within-document conference, and prior-itization of relations extracted from newssources.

1 Introduction

The JHU Human Language Technology Center ofExcellence has participated in the TAC KnowledgeBase Population exercise since its inception in 2009.Our focus over the past year was on the Cold Starttask. We attempted to enhance our KELVIN sys-tem (McNamee et al., 2012; McNamee et al., 2013;Mayfield et al., 2014) by application of forward-chaining inference rules, improved cross-documententity coreference, refined within-document corefer-ence, biasing extraction towards relations identifiedin news, and through a variety of software engineer-ing and architectural modifications

In the rest of the paper we present our system,which is architecturally very similar to our 2013submission, and briefly discuss our experimental re-sults.

2 Cold Start KB Construction

The TAC-KBP Cold Start task is a complex taskthat requires application of multiple layers of NLP

software. The most significant tool that we use isa NIST ACE entity/relation/event detection system,the BBN SERIF system. SERIF provides a substratethat includes entity recognition, relation extraction,and within-document coreference analysis. In addi-tion to SERIF, significant components which we re-lied on include: a maximum entropy trained modelfor extracting personal attributes (FACETS, also aBBN tool); cross-document entity coreference (theHLTCOE Kripke system); and a procedurally imple-mented rule system.

The system is organized as a pipeline with threestages: (i) document level processing done in par-allel on small batches of documents, (ii) cross-document co-reference resolution to produce an ini-tial KB, and (iii) knowledge-base enhancement andrefinement through inference and relation analysis.The next section describes the major steps in thesestages.

3 System Description

KELVIN runs from two Unix shell scripts1 that exe-cute a pipeline of operations. The input to the systemis a file listing the source documents to be processed;the files are presumed to be plain UTF-8 encodedtext, possibly containing light SGML markup. Dur-ing processing, the system produces a series of tab-separated files, which capture the intermediate stateof the growing knowledge base. At the end of thepipeline the resulting file is compliant with the ColdStart guidelines.

Our processing consists of the following steps,which are described in detail below:

1Named Margaret and Fanny after Lord Kelvin’s wives.

DRAFT: Tim Finin, Paul McNamee, Dawn Lawrie, James Mayfield and Craig Harman, Hot Stuff at Cold Start: HLTCOE participation at TAC 2014, 7th Text Analysis Conference, National Institute of Standards and Technology, Nov. 2014.

1. Document-level processing2. Curating intra-document coreference3. Cross-document entity coreference4. KB cleanup and slot value consolidation5. Applying inference rules to posit additional as-

sertions6. KB cleanup and slot value consolidation7. Selecting the best provenance metadata8. Post-processing

The Margaret script performs the document-levelprocessing in parallel on our Sun Grid Engine com-puting cluster. Fanny executes the balance of thepipeline, and many of these steps are executed as asingle process.

3.1 Document-Level ProcessingBBN’s SERIF tool2 (Boschee et al., 2005) providesa considerable suite of document annotations thatare an excellent basis for building a knowledge base.The functions SERIF can provide are based largelyon the NIST ACE specification,3 and include:

• identifying named-entities and classifyingthem by type and subtype;

• performing intra-document coreference anal-ysis, including named mentions, as well ascoreferential nominal and pronominal men-tions;

• parsing sentences and extracting intra-sentential relations between entities; and,

• detecting certain types of events.

We run each document through SERIF, and ex-tract its annotations.4 Additionally we run anothermodule named FACETS, described below, whichadds attributes about person entities. For each en-tity with at least one named mention, we collect itsmentions, the relations, and events in which it par-ticipates. Entities comprised solely of nominal orpronominal mentions are ignored for the Cold Starttask, per the task guidelines.

2Statistical Entity & Relation Information Finding3http://www.itl.nist.gov/iad/mig/tests/

ace/2008/doc/ace08-evalplan.v1.2d.pdf4We used an in-house version of SERIF, not the annotations

available from LDC.

FACETS is an add-on package that takes SERIF’sanalyses and produces role and argument annota-tions about person noun phrases. FACETS is im-plemented using a conditional-exponential learnertrained on broadcast news. The attributes FACETScan recognize include general attributes like religionand age (which anyone might have), as well as somerole-specific attributes, such as employer for some-one who has a job, (medical) specialty for physi-cians, or (academic) affiliation for someone associ-ated with an educational institution.

3.2 Intra-Document Coreference

Within document conference is one of the areas thatleads to numerous errors in downstream applicationssuch as the Cold Start task. For example, we haveobserved cases where family members or politicalrivals are mistakenly combined into a single entitycluster. This creates problems in knowledge basepopulation where correct facts from distinct individ-uals can end up being combined into the same entity.For example, if Bill and Hillary Clinton are men-tioned in a document that also mentions that she wasborn in the state of Illinois, a conjoined cluster mightresult in a knowledge base incorrectly asserting thatBill Clinton was born in Illinois.5 Therefore in ourprior submission and in this submission, this issuehas received specific attention.

In last year’s submission, we built a classifier todetect such instances and then remove document en-tities identified as problematic from the KB. Ourclassifier uses name variants from the AmericanEnglish Nickname Collection6 and lightweight per-sonal name parsing to identify acceptable variants(e.g., Francis Albert Sinatra and Frank Sinatra). Ifour rules for name equivalence are not satisfied,then string edit distance is computed using a dy-namic time warping approach to identify the leastcost match; two entity mentions that fail to meet acloseness threshold by this measure are deemed tobe mistakenly conflated. Organizations and GPEsare handled similarly. Name variants for GPEs in-clude capital cites and nationalities for known coun-tries. In addition, both are permitted to match withacronyms.

5He was born in Arkansas.6LDC2012T11

In this year’s submission, we decided to merge theopinions of two separate intra document coreferencesystems, SERIF and the Stanford CoreNLP tools.The hypothesis was that the two systems might failin different ways so a combined system might out-perform either independent tool.

In order to accomplish this task, the first step is toalign the two separate mention chains. This is doneby aligning each mention independently, whether itis a named, nominal, or pronominal reference. Oneof the main challenges with alignment is that the dif-ferent systems could identify different entities fromwithin the same span of text (e.g “Christian” and“the Christian church”). We use a rule-based ap-proach to identify the one best alignment for eachmention.

Once all mentions are aligned, new entities arecreated that represent merged entities that were iden-tified by both systems. In addition there are othermentions only identified by one of the systems. Ini-tially, these form separate entities. At this point eachentity is given a score by the classifier developedlast year. These scored entities are considered forremerging based not the classifier scores.

If merging two entities that were unified by oneof the systems does not negatively impact the classi-fier score, then the merging will occur. After entitymerging, mentions only identified by one of the sys-tems are considered for merging into a merged en-tity. Merging of named mentions occurs if it doesnot negatively impact the merged score. Nominalmentions are also considered for merging; however,a different classifier was developed based purelyon the string edit distance. Nominal mentions aremerged if the classifier does not report a negativelyimpact to this classifier’s score. With this process,new entities are produced for cross-document coref-erence.

3.3 Cross-document entity coreferenceIn 2013 we developed a tool for cross-documentcoreference named Kripke. Our motivation for anew tool were that we wanted an easy-to-run, ef-ficient, and precision-focused clusterer; previously(i.e., in 2012) we had used string-matching alone,or a Wikipedia-based entity linker. We producedruns that ran Kripke with standard settings, and alsoa variant with somewhat more aggressive clustering

which we thought might improve (entity-clustering)recall.

Kripke is an unsupervised, procedural clustererbased on two principles: (a) to combine two clus-ters each must have good matching of both namesand contextual features; (b) a small set of discrimi-nating contextual features is generally sufficient fordisambiguation. Additional details can be found inSection 4.

3.4 KB cleanup and slot value consolidation

This step, which is repeated several times in thepipeline ensures that all relations have their inversesin the KB, culls relations that violate type or valueconstraints, and reduces the number of values tomatch expectations for each type of slot.

Inverses. Producing inverses is an entirely de-terministic process that simply generates Y in-verse X in Doc D from an assertion of X slotY in Doc D. For example, inverse relations likeper:parent and per:children, or per:schools attendedand org:students. While straightforward, this is animportant step, as relations are often extracted inonly one direction during document-level analysis,yet we want both assertions to be explicitly presentin our KB to aid with downstream reasoning.

Predicate Constraints. Some assertions ex-tracted from SERIF or FACETS can be quicklyvetted for plausibility. For example, the ob-ject of a predicate expecting a country (e.g.,per:countries of residence) must match a small,enumerable list of country names; Massachusetts isnot a reasonable response.7 Similarly, 250 is an un-likely value for a person’s age. We have proceduresto check certain slots to enforce that values mustcome from a accepted list of responses (e.g., coun-tries, religions), or cannot include responses from alist of known incorrect responses (e.g., a girlfriendis not allowed as a slot fill for per:other family).

Consolidating Slot Values. Extracting values forslots is a noisy process and errors are more likely forsome slots than for others. The likelihood of findingincorrect values also depends the popularity of boththe entity and slot. For example, in processing a col-lection of 26K articles from the Washington Post,we observed more than fifty entities who had 14 or

7And in 2014 neither is Texas.

Figure 1: Kelvin initially extracted 121 distinctvalues for Barack Obama’s employer from 26,000Washington Post articles. The number of attestingdocuments for each followed a power law, with ninedocuments for the most popular value only one forthe majority.

more employers. One entity was reported as havinghad 122 employers (per:employee of)!

Slot value consolidation involves selecting thebest value in the case of a single valued slot (e.g.,per:city of birth) and the best set of values for slotsthat can have more than one value (e.g, per:parents).In both cases, we use the number of attesting docu-ments to rank candidate values, with greater weightgiven to values that were explicitly attested ratherthan implicitly attested via inference rules. See Fig-ure 1 for the number of attesting documents for eachof the values for the entity that have 122 distinct val-ues for employer.

For slots that admit only a single value, we se-lect the highest ranked candidate. However, for list-valued slots, it is difficult to know how many, andwhich values to allow for an entity. We made thepragmatic choice to limit list-values responses in apredicate-sensitive fashion, preferring frequently at-tested values. We associate two thresholds for se-lected list-valued predicates on the number of val-ues that are reasonable – the first represents a num-ber that is suspiciously large and the second is anabsolute limit on the number of values reported. Ta-ble 1 shows the thresholds we used for some pred-icates. For predicates in our table, we accepted thenth value on the candidate list if n did not exceedthe first threshold and rejected it if n exceeded thesecond. For n between the thresholds, a value is ac-cepted only if it has more than one attesting docu-ment.

relation many maximumper:children 8 10per:countries of residence 5 7per:employee of 8 10per:member of 10 12per:parents 5 5per:religion 2 3per:schools attended 4 7per:siblings 9 12per:spouse 3 8

Table 1: The number of values for some multi-valued slots were limited by a heuristic process thatinvolved the number of attesting documents for eachvalue and two thresholds.

3.5 Inference

We apply a number of forward chaining inferencerules to increase the number of assertions in our KB.To facilitate inference of assertions in the Cold Startschema, we introduce some unofficial slots into ourKB, which are subsequently removed prior to sub-mission. For example, we add slots for the sexof a person, and geographical subsumption (e.g.,Gaithersburg is part-of Maryland). The most pro-lific inferred relations were based on rules for fam-ily relationships, corporate management, and geo-political containment.

Many of the rules are logically sound and followdirectly from the meaning of the relations. For ex-ample, two people are siblings if they have a parentin common and two people have an “other family”relation if they one is a grandparent of the other. Ourknowledge of geographic subsumption produced alarge number of additional relations, e.g., know-ing that a person’s city of birth is Gaithersburg andthat it is part of Maryland and that Maryland is astate supports the inference that the person’s state-orprovince of birth is Maryland.

For TAC 2014, we only used inference rules weconsidered to be sound and did not use heuristicrules. For example, in 2013 we we inferred that ifa person attended a school S, and S has headquartersin location L, then the person has been a resident ofL. In general, we do not add an inferred fact that isalready in our knowledge base. Some of the heuris-tic rules are default rules in that they only add a valuefor a slot for which we have no values. For example,we know that person P1 is the spouse of person P2and that the sex of P1 is male and we have no value

for the sex of P2, we infer that P2 is female. In thiscase, the rule is both a default rule and one whoseconclusion is very often, but not always, true.

The 2013 TAC-KBP guidelines stipulated that re-lations must be attested in a single document, whichconstrained severely the number of inferred asser-tions allowed, and required removing relations notevidenced entirely in a single document. For exam-ple, consider learning that Lisa is Homer’s child inone document and that Bart is Homer’s child in an-other. Assuming that the two Homer mentions co-refer, it follows that Lisa and Bart are siblings. Thisyear the 2014 guidelines do allow such inferences,but required systems to keep a set of provenance de-scriptions (document and offsets) for each inferredrelation. Table 6 shows the distribution of the ad-ditional relations that we inferred using only soundrules from the facts extracted in the evaluation cor-pus.

We ran the inference step over the entire knowl-edge base which had been loaded into memory, sincein general, a rule might have any number of an-tecedent relations. However, we realized that manyof our inference rules do not require arbitrary joinsand could be run in parallel on subsets of the knowl-edge base if we ensure that all facts about any en-tity are in the same subset. The fraction of rules forwhich this is true can be increased by refactoringthem. For example, the rule for per:sibling mightnormally be written as

X per:parent P ∧ Y per:parent P → X per:siblings Y

but can also be expressed asP per:child X ∧ P per:child Y → X per:siblings Y

assuming that we materialize inverse relations in theknowledge base (e.g, asserting a child relation forevery parent relation and vice versa). A preliminaryanalysis of our inference rules shows that all couldbe run in at most three parallelizable inference stepsusing a Map/Reduce pattern.

3.6 Selecting provenance metadata

This step selects the provenance strings to supporteach relation for the final submission. The 2014evaluation rules allow for up to four provenancestrings to support a relation, none of which can ex-ceed 150 characters. For simple attested values, ourinitial provenance strings are spans selected from the

sentence from which we extracted the relation, e.g.,“Homer is 37 years old” for a per:age relation. In-ferred relations can have more than one provenancestring which can come from the different documents,e.g., “His daughter Lisa attends Springfield Elemen-tary” and “Maggie’s father is Homer Simpson” fora per:siblings relation.

An initial step is to minimize the length of anyoverly-long provenance strings is to select a sub-string that spans both the subject and object. Can-didate provenance strings whose length exceedsthe maximum allowed after minimization are dis-carded8. If there are multiple provenance candi-dates, a simple greedy bin packing algorithm is usedto include as many as possible into the four slotsavailable. Preference is given for attested valuesover inferred values and provenance sources withhigher certainty over a those with lower.

3.7 Post-processingThe final steps in our pipeline ensure compliancewith the task guidelines. We normalize temporalexpressions, ensure that all entities have mentions,insist that relations are consistent with the types oftheir subjects and objects, confirm that logical in-verses are asserted, and check that entities have men-tions in the provenance documents. so forth.

4 Kripke

The Kripke system9 takes a set of document-levelentities and performs agglomerative clustering onthem to produce cross-document entity clusters. Thetool is written in approximately 2000 lines of Javasource code. The intent is for the system to havea precision bias, which we feel is appropriate forknowledge base population.

The principles on which Kripke operations are:

• Coreferential clusters should match well intheir names.

• Coreferential clusters should share contextualfeatures.

• Only a few, discriminating contextual featuresshould be required to disambiguate entities.

8This could result in a relation being discarded if it has nolegal provenance strings after minimization

9Named after Princeton philosopher Saul Kripke, who wrotea book on naming entities in the 1970s.

Size PER ORG GPE1 75660 44481 165062 15485 7750 30363 6319 3009 12224 3384 1603 6735 2137 969 4246 1570 690 3147 1086 517 2238 853 351 2149 692 272 140

10-20 2839 1367 65021-30 814 463 25631-40 344 240 16441-50 186 156 7651-60 101 104 7161-70 76 83 4871-80 46 77 3881-90 42 59 3091-100 13 43 25

101-200 65 153 136201-300 9 48 83301-400 3 15 41401-500 2 3 24501-600 3 4 17601-700 0 4 14701-800 2 4 10801-900 0 2 6901-1000 0 1 3

1001-2000 0 4 26>2000 1 2 6

Table 2: A histogram of cluster sizes for each entitytype for run hltcoe3 shows that Kripke is effective atproducing large clusters but that most entity clustersare small.

To avoid the customary quadratic-time complex-ity required for brute-force pairwise comparisons,Kripke maintains an inverted index of names usedfor each entity. Only entities matching by full name,or some shared words or character n-grams are con-sidered as potentially coreferential.10 Related in-dexing techniques are variously known as blocking(Whang et al., 2009) or canopies (McCallum et al.,2000).

At present, contextual matching is accomplishedsolely by comparing named entities that co-occur inthe same document. Between candidate clusters, thesets of all names occurring in any document formingeach cluster are intersected. Each name is weightedby normalized Inverse Document Frequency, so thatrare, or discriminating names have a weight closerto 1. The top-k (i.e., k=10) weighted names were

10Support for orthographically dissimilar name variants (i.e.,aliases) was planned, but not implemented in time for this year.

used, and if the sum of those weights exceeds a cut-off, then the contextual similarity is deemed ade-quate. Such a technique should be able to tease apartGeorge Bush (41st president) and his son (43rd pres-ident) through co-occurring names (e.g., Al Gore,Barbara Bush, Kennebunkport, James Baker versusthe entities Dick Cheney, Laura Bush, Crawford,Condolezza Rice).

The system runs by executing a cascade of clus-tering passes, where in each subsequent pass con-ditions are relaxed in the requirements for goodname and contextual matching. The hope is thathigher precision matches are made in earlier phasesof the cascade, and these will facilitate more difficultmatches later on.

An example of Kripke’s performance on the 2014evaluation corpus is given in Table 2, which showresults from the hltcoe3 run. The number of clustersby size shows a power law distribution, with largercluster being more common for entities of type GPEand ORG. The largest clusters for each type were thePER Barack Obama with 2641 document entities,the ORG Associated Press with 2341 and the GPEUnited States with 6719. Associated with each ofthese were many smaller entity clusters with one ortwo entities that had not been merged.

5 Development Tools

Evaluation is an essential step in the process of de-veloping and debugging a knowledge base popula-tion system. We briefly describe several of the eval-uation tools used by the KELVIN system. Two wereaimed at comparing the system’s output from twodifferent versions: entity-match, which focuses ondifferences in entities found and linked; and kbdiff,which identifies differences in relations among thoseentities. Together, these tools support assessment ofrelative KB accuracy by sampling the parts of twoKBs that disagree (Lawrie et al., 2013). Tac2Rdfproduces an RDF representation of a TAC KB sup-ported by an OWL ontology and loads it into a stan-dard triple store, making it available for browsing,inference and querying using standard RDF tools.KB Annotator allows developers to browse the sys-tem output and annotate entities and relations as ei-ther supported or not by the document text providedas provenance.

Figure 2: The RDF KB encoding can be queried viaSPARQL (here using the Yasgui interface) to findanomalies or collect data for analysis or training.

Entity-match defines a KB entity as the set of itsmentions. From the perspective of an entity in oneKB, its mentions might be found within a single en-tity in the other KB, spread among multiple entities,or missing altogether from the other KB. In the firstcase there is agreement on what makes up the en-tity. In the second case, there is evidence either thatmultiple entities have been conflated in the first KB,or that a single entity has been incorrectly split inthe second. In the third case, the entity has goneundetected. The tool reports shows the entities andcases into which they fall. If there is disagreementbetween the KBs, it reports each corresponding en-tity in the second KB and the number of mentionsthat map to it.

Kbdiff identifies assertions in one KB that do notappear in the other. The challenge here is to identifywhich entities are held in common between the twoKBs. Provenance is again useful; relations from dif-ferent KBs are aligned if they have the same pred-icates and the provenance of their subjects and ob-jects match. The algorithm works by first reading allthe assertions in both KBs and matching them basedon provenance and type. The output includes asser-tions in the first KB lacking a match in the second(prefixed by <) and those in the second but not thefirst (prefixed by >.)

Figure 3: Pubby provides a simple way to browsethe RDF version of the extracted knowledge as agraph via a Web browser.

Tac2Rdf translates a KB in TAC format toRDF using an OWL ontology (available athttp://tackbp.org/static/tackbp.ttl) that encodesknowledge about the concepts and relations, bothexplicit and implicit. For example, the Cold Startdomain has an explicit type for geo-political entities(GPEs), but implicitly introduces disjoint GPE sub-types for cities, states or provinces, and countriesthrough predicates like city of birth. Applyingan OWL reasoner to this form of the KB detectsvarious logical problems, e.g., an entity is beingused as both a city and a country. The RDF KBresults are also loaded into a triple store, permittingaccess by an integrated set of standard RDF toolsincluding Fuseki for SPARQL (Prud’Hommeauxand Seaborne, 2008) querying, Pubby for browsing,and the Yasgui SPARQL GUI.

Figure 2, for example, shows the results of anad hoc SPARQL query for GPEs with the string“baltimore” in their canonical mention along withthe number of documents in which they were men-tioned and their subtype. Such queries are usefulin identifying possible cross-document coreferencemistakes (e.g., GPEs with mentions matching both“X County” X) and likely extraction errors (e.g.,pairs of people connected by more than one rela-tion in the set {spouse, parent and other-family}).

Figure 4: Overview of a KB including counts on en-tity type and relations.

Figure 5: Yhe results of a relation search providesaccess to relations and their associated entities.

Clicking on the second entity in the table of resultsopens the entity in the Pubby linked data browser, asshown in Figure 3.

The Interactive KB Browser system loads triplesfrom a TAC submission into a KB and provides aWeb-based interface allowing one to view a docu-ment’s text, see the entities, entity mentions and re-lations found in it, and give feedback on their cor-rectness. Figure 4 details an overview of the KB,by listing entities by their type with numeric countsoverall and for each type. It also allows search-ing with subjects and objects found using stringmatches. Either all relations can be included or aparticular relation. Search results are presented in atable as in shown in Figure 5. The light blue text

Figure 6: The entity view shows the docments, men-tions and facts associated with an entity.

Figure 7: A document view where hovering over anentity mention reveals strings that are co-referent.

represents links to entities and relationships. In thisexample, the object of the relation is a string ratherthan an entity, so the object is not a link. While arelation is viewed as a list of facts, an entity consistsof a type, canonical mentions, mention strings, doc-uments in which the entity is mentioned, and factspertaining to the entity. Figure 6 show an exampleentity and 7 shows document text with entity men-tions identified in a contrasting color as well as theextracted facts. Mousing over a mention highlightsit and any co-referential mentions.

Whenever a fact is encountered, the developer orassessor has the option of providing an annotationor judgment. As shown in Figure 8, the judgment issolicited by stating the relationship as a fact, show-

Figure 8: For each fact extracted from a document, adeveloper or assessor can express a judgment on itscorrectness.

Name Inference Clustering InDoc NewsBiashltcoe1 Standardhltcoe2 Yes Standardhltcoe3 Aggressivehltcoe4 Standard Yeshltcoe5 Standard Yes

Table 3: Experimental variables for submitted runs.

ing the sentence with relevance parts of the text incontrasting colors, and a simple “yes”, “no”, “un-sure” feedback on the fact. This allows an asses-sor to annotate errors in named entity resolution, in-document coreference, and relation detection withrespect to particular facts. An annotator may alwayannotate all the facts concerning a particular entity,wich can then be used for error analysis and as train-ing data.

6 Submissions and results

We submitted the maximum of five experimentalconditions that started with a simplistic baselinepipeline, and which added individually: inferencerules, more aggressive entity coreference, within-document mention-chain purification, and a filterthat required relations to be attested in newswiresources. KELVIN does not access the Internet dur-ing processing. Table 3 summarizes the various con-ditions and Tables 4 and 5 give the key performancemetrics. The number of times each slot was assertedfor run hltcoe3 is given in Table 6.

Table 7 lists the number of entities of each typewhich are included in each of our runs. Note thatas entities having no asserted relations cannot im-prove scores in the ColdStart task, we did not in-clude such “mention only” entities in our submis-

Slot Read Inferred Queriedper:employee or member of 56107 0 yesorg:employees or members 43660 0 yesorg:alternate names 38368 0 yesper:title 28378 0 yesper:countries of residence 12641 1500 yesgpe:residents of country 12641 1500 yesgpe:employees or members 12447 0 yesper:origin 8714 0 yesorg:parents 5888 0 yesorg:city of headquarters 5338 0 yesgpe:headquarters in city 5338 0 yesper:cities of residence 4826 0 yesgpe:residents of city 4826 0 yesgpe:headquarters in country 4756 1749 yesorg:country of headquarters 4756 1729 yesorg:subsidiaries 3494 0 yesgpe:subsidiaries 2394 0 noper:spouse 2345 0 yesper:date of death 1933 0 yesper:age 1916 0 yesgpe:residents of stateorprovince 1676 1994 yesper:statesorprovinces of residence 1676 1994 yesper:country of birth 1366 78 yesgpe:births in country 1366 78 yesper:alternate names 1251 0 yesper:children 1194 0 yesper:parents 1194 0 yesgpe:headquarters in stateorprovince 1161 2449 yesorg:stateorprovince of headquarters 1161 2449 yesorg:founded by 1080 0 yesper:schools attended 1018 0 yesorg:students 1018 0 yesper:organizations founded 980 0 yesper:siblings 838 426 yesper:top member employee of 715 7911 yesorg:top members employees 715 7911 yesper:date of birth 497 0 yesorg:date founded 452 0 yesper:charges 429 0 yesgpe:deaths in city 353 0 yesper:city of death 353 355 yesper:other family 345 355 yesorg:members 302 0 yesper:city of birth 280 0 yesgpe:births in city 280 0 yesorg:member of 195 0 noper:religion 178 0 yesgpe:member of 107 0 yesgpe:deaths in country 102 85 noper:country of death 102 85 noorg:organizations founded 87 0 noorg:date dissolved 67 0 yesorg:shareholders 47 0 nogpe:deaths in stateorprovince 46 182 yesper:stateorprovince of death 46 182 yesper:holds shares in 43 0 yesgpe:births in stateorprovince 39 139 yesper:stateorprovince of birth 39 139 yesgpe:organizations founded 13 0 noorg:holds shares in 4 0 yesorg:website 0 0 yesorg:number of employees members 0 0 yesper:employee of 0 1209 noper:cause of death 0 0 yes

Table 6: This table shows the number of assertionsfor each slot that were read or inferred for run hlt-coe3 and whether or not the slot was used in anyevaluation queries. Slots not listed were neither as-serted nor queried.

0-hop 1-hop All-hopRun GT R W D GT R W D GT R W D

1 1073 136 133 46 1364 64 57 11 2437 200 190 572 1073 121 113 37 1364 66 71 10 2437 187 184 473 1073 156 112 33 1364 71 61 7 2437 227 173 404 1073 84 82 29 1364 40 30 7 2437 124 112 365 1073 107 113 40 1364 53 43 7 2437 160 156 47

Table 4: Ground-truth, right, wrong and duplicate answers for our submitted 2014 runs.

0-hop 1-hop All-hopRun P R F1 P R F1 P R F1

1 0.5056 0.1267 0.2027 0.5289 0.0469 0.0862 0.5128 0.0821 0.14152 0.5171 0.1128 0.1852 0.4818 0.0484 0.0879 0.5040 0.0767 0.13323 0.5821 0.1454 0.2327 0.5379 0.0521 0.0949 0.5675 0.0931 0.16004 0.5060 0.0783 0.1356 0.5714 0.0293 0.0558 0.5254 0.0509 0.09285 0.4864 0.0997 0.1655 0.5521 0.0389 0.0726 0.5063 0.0657 0.1162

Table 5: Micro precision, recall and F1 scores for our submitted 2014 runs.

run entities PER ORG GPE facts1 340412 187056 110506 42850 2883582 340420 187109 110445 42866 3110193 306499 171951 98234 36314 2835824 372509 203779 120045 48685 2365505 340593 187134 110569 42890 244283

Table 7: Number of entities mentions and facts iden-tified in the evaluation corpus for each run.

sions. The number of reported entities is gener-ally similar in each run, with differences likely at-tributable to changes in cross-document entity coref-erence.

6.1 Discussion

Comparing our various experimental conditions, wemake the following observations.

It appears that more aggressive cross-documentcoreference does improve recall, as was hoped for;0-hop recall rises from 0.127 in hltcoe1 to 0.145 inhltcoe3. Precision also improves.

Use of inference rules (contrast hltcoe2 to hlt-coe1) does not appear to improve performance. Todate we have not been able to conduct an analysis ofthe reasons for this.

Requiring posited relations to be supported fromnews documents (i.e., hltcoe5 vs. hltcoe1 baseline)seems to have lowered recall, and therefore F1.

7 Conclusion

The JHU Human Language Technology Center ofExcellence has participated in the TAC Knowl-edge Base Population exercise since its inceptionin 2009 and in Cold Start task since 2012. Wemodified the KELVIN system used in the 2012 and2013 Cold Start task by enhancing inference, cross-document entity coreference, and application of in-ference rules.

ReferencesE. Boschee, R. Weischedel, and A. Zamanian. 2005. Au-

tomatic information extraction. In Proceedings of the2005 International Conference on Intelligence Analy-sis, McLean, VA, pages 2–4.

Dawn Lawrie, Tim Finin, James Mayfield, and Paul Mc-Namee. 2013. Comparing and Evaluating Seman-tic Data Automatically Extracted from Text. In AAAI2013 Fall Symposium on Semantics for Big Data.AAAI Press, November.

James Mayfield, Paul McNamee, Craig Harmon, TimFinin, and Dawn Lawrie. 2014. KELVIN: Extract-ing Knowledge from Large Text Collections. In AAAIFall Symposium on Natural Language Access to BigData. AAAI Press, November.

Andrew McCallum, Kamal Nigam, and Lyle Ungar.2000. Efficient clustering of high-dimensional datasets with application to reference matching. In Knowl-edge Discovery and Data Mining (KDD).

Paul McNamee, Veselin Stoyanov, James Mayfield, TimFinin, Tim Oates, Tan Xu, Douglas W. Oard, andDawn Lawrie. 2012. HLTCOE participation at TAC2012: Entity linking and cold start knowledge baseconstruction. In Text Analysis Conference (TAC),Gaithersburg, Maryland, November.

Paul McNamee, James Mayfield, Tim Finin, Tim Oates,Baltimore County, Dawn Lawrie, Tan Xu, and Dou-glas W Oard. 2013. KELVIN: a tool for automatedknowledge base construction. In Human LanguageTechnologies: The 2013 Annual Conference of theNorth American Chapter of the Association for Com-putational Linguistics, volume 10, page 32.

E Prud’Hommeaux and A. Seaborne. 2008. SPARQLquery language for RDF. Technical report, WorldWide Web Consortium, January.

Steven Euijong Whang, David Menestrina, GeorgiaKoutrika, Martin Theobald, and Hector Garcia-Molina. 2009. Entity resolution with iterative block-ing. In SIGMOD 2009, pages 219–232. ACM.