Text Analysis ConferenceKnowledge Base Population
2013
Hoa Trang DangNational Institute of Standards and Technology
Sponsored by:
TAC KBP Goals• Goal: Populate a knowledge base (KB) with information about
entities as found in a collection of source documents, following a specified schema for the KB
• KBP 2009-2011: Focus on augmenting an existing KB. Decompose KBP into two tasks▫ Entity-Linking: link each given named entity mention to a node in
reference KB (or create new node)▫ Slot-Filling: Learn attributes about target entities from the source
documents and add new information about the entity to the reference KB• KBP 2012: Combine entity-linking and slot-filling to build a KB from
scratch -> Cold Start• KBP 2013:
▫ Conversational, informal data (discussion fora)▫ Temporal constraints for Slot Filling (2011 pilot)▫ Sentiment analysis for Slot Filling
TAC KBP 2013 Track Participants• Track coordinators
▫ Hoa Dang (Slot Filler Validation)▫ Jim Mayfield (Entity Linking, Cold Start KBP)▫ Margaret Mitchell (Sentiment Slot Filling)▫ Mihai Surdeanu (English Slot Filling and Temporal Slot
Filling)• LDC linguistic resource providers: Joe Ellis, Jeremy Getman,
Justin Mott, Xuansong Li, Kira Griffitt, Stephanie M. Strassel, Jonathan Wright
• Coordinators emeritus: Ralph Grishman, Heng Ji• Advisor: Boyan Onyshkevych• 45 Teams
▫ 14 countries (21 USA, 9 China, 3 Spain, 2 Germany,….)
6 (8) TAC KBP 2013 Tracks• Entity-Linking
▫ English▫ Chinese▫ Spanish
• Slot-Filling (English)▫ Regular▫ Sentiment▫ Temporal▫ Slot Filler Validation Task
• Cold Start (English)
Entity Linking and Slot Filling Tracks• Goal: Augment a reference knowledge base (KB) with info
about query entities (PER, ORG, GPE) as found in a diverse collection of documents
• Reference KB: Oct 2008 Wikipedia snapshot. Each KB node corresponds to a Wikipedia page and contains:▫ Infobox▫ Wiki_text (free text not in infobox)
• English source documents:▫ 1M News docs▫ 1M Web docs▫ 99K Discussion Forum docs (threads)
• Chinese source documents: 2M news, 800K Web• Spanish source documents: 900K news
Entity-Linking Evaluation Results• English
▫ Participants: 26 teams▫ Highest F1: 0.721 (0.730 in 2012)▫ Median F1: 0.583 (0.536 in 2012)
• Chinese▫ Participants: 4 teams▫ Highest F1: 0.622 (0.740 in 2012)▫ Median F1: 0.619 (0.617 in 2012)
• Spanish▫ Participants 3 teams▫ Highest F1: 0.709 (0.641 in 2012)▫ Median F1: 0.651 (0.612 in 2012)
Regular Slot Filling Evaluation Results
•Participants: 18 teams•Human F1: 0.685 (0.814 in 2012)•Highest System F1: 0.373 (0.517 in 2012)•2nd Highest System F1: 0.339 (0.296 in 2012)•Median System F1: 0.150 (0.099 in 2012)
Sentiment Slot Filling Track
• Sentiment analysis for KBP:▫Holder (PER, ORG, GPE)▫Target (PER, ORG, GPE)▫Polarity (positive, negative)
• Implemented as regular slot filling, with different set of slots▫{per,org,gpe}:positive-towards▫{per,org,gpe}:negative-towards▫{per,org,gpe}:positive-from▫{per,org,gpe}:negative-from
• Participants: 3 teams• Evaluation results:
▫Human F1: 0.727▫Highest System F1: 0.132▫Median System F1: 0.014
Temporal Slot Filling Track• Find tightest temporal constraints [T1 T2 T3 T4] on a
given relation▫ Relation is true for a period beginning between T1 and
T2▫ Relation is true for a period ending between T3 and T4
• Participants: 5 teams• Evaluation results:
▫ Human Accuracy: 0.688▫ Highest System Accuracy: 0.331▫ Median System Accuracy: 0.148
Slot Filler Validation Track (SFV)
• Task: Determine whether or not a candidate slot filler is correct
• Objective: improve precision without excessive reduction of recall
• Participants: 5 teams• Some SFV runs had overwhelmingly positive impact
on individual SF runs!
Cold Start KBP Track • Goal: Build a KB from scratch, containing all targeted info about all
entities as found in a relatively closed domain corpus of documents• KB schema: same entity types and slots as regular slot-filling task• Source document collection:
▫ 50K Web pages from small-town publications (from TREC KBA document stream)
• Required capabilities:▫ Entity-linking: Grounding all named entity mentions in docs to KB
nodes▫ Slot-filling: Learning attributes about all named entities
• Post-submission evaluation queries traverse KB starting from a single entity node (entity mention):▫ 0-hop: Find all children of Michael Jordan▫ 1-hop: Find date of birth of each of the children of Michael Jordan
Cold Start Evaluation Results (Preliminary)• Participants: 3 teams• 0-hop queries:
▫ Highest F1 0.384 (0.497 in 2012)• 1-hop queries:
▫ Highest F1 0.145 (0.255 in 2012)• Combined 0-hop and 1-hop F1
▫ Highest F1: 0.278 (~0.352 in 2012)
TAC KBP Discussion/Planning Sessions• Monday, November 18 (2:15-3:10pm):
▫ English Slot Filling▫ Slot Filler Validation▫ Temporal Slot Filling?▫ +Spanish Slot Filling?▫ +Event identification and argument extraction?
• Tuesday, November 19 (3:00-4:00pm):▫ Cold Start▫ English Entity Linking (as queries in Cold Start
framework?)▫ Cross-Lingual Spanish and Chinese Entity Linking
+ Discussion forum