0

WelcometotheOPERA

1

AIDAin2019…achallenge

Nomoretrainingdata,onlyexamplesthatillustratetheevaluationIncreasinglydata-intensiveneurallearnersWhatdowedo???

2

Arangeofresponses…

•  Justmakemachinelearningwork!

•  Learning,augmentedwithexternaldata

•  Half-half

•  Include(some)learningbutonlyifit’seasy

•  Forgetmachinelearning!3

5

4

3

2

1

Overview

1.  Systemoverview2.  TA1Englishentityandrelationprocessing3.  TA1Rus/Ukrentityandeventprocessing4.  TA1/2KBconstructionandvalidation5.  TA3Hypotheses

4

SYSTEMOVERVIEWZaidSheikh,AnkitDangi,EduardHovy

5

OPERAarchitecture

TA1extractionengines

TA2corefengine

TA3hypothesisformation

Ontology CSR

PowerLoomdatabase

textspeechimagesvideo

6

OPERAframework

Coref

Mini-KBcreation/AIFvalidation

TA1Mini-KB

Input

Mini-KBcreation/AIFvalidation

Hypothesisformation

BeliefGraphconstruction

TA2Mini-KB

Queries

Englishentities

Mini-KBcreation/AIFvalidation

ImagesSpeech

Rus/Ukrentities

Text

Englishevents

Rus/Ukrevents

7

TA1frameworkInput

Mini-KBcreation/AIFvalidation

Domainfilter&languagedetection

Ru/Ukpipeline

MT:Ru/Uk–>

Eng

Ru/UkEntityandEventdetection

EventframeassemblyII

CSRCombination

Imagepipeline

Entitydetection

PersonandGeoID

Englishentity

detection

Englishevent

detection

Englishentitylinking

Engentityrelations

EventframeassemblyI

Englishpipeline

Englishargumentdetection

Englishcoref

Speechpipeline

8

OPERATA2+TA3framework

Coref

Mini-KBcreation/AIFvalidation

TA1Mini-KB

Mini-KBcreation/AIFvalidation

Hypothesisformation

BeliefGraphconstruction

TA2Mini-KB

Queryinput

9

KBsandnotations

•  AllresultswritteninOPERA-internalframenotation(json)andstoredinCSR(BlazeGraph)

•  Input/outputconvertersfrom/toAIDAAIF

•  TwoseparateKBcreationandvalidationprocedures,fortwoparallelKBs(givesinsurance,coverage,andbackup):– Chalupsky:usesPowerLoomandChameleonreasoner

– Chaudhary:usesspecializedrules10

5

4

Internaldryruns

•  Internaldryrunmini-evalsusingthepracticeannotationsreleasedbyLDC

•  Evaluatedresultsmanually

•  Resultslookpromising,BUT…hardtocalculateP/R/F1forvariouspartsoftheTA1pipelinebecauseLDCdoesnotlabelallmentionsofevents,relationsandentities,justthe“salient”or“informative”ones(sowehavetojudgethemourselves…laboriousandnotguaranteed)

11

TA1TEXT:ENGLISHENTITIESANDRELATIONS

XiangKong,XianyangChen,EduardHovy

12

OPERATA1framework

Input

Mini-KBcreation/AIFvalidation

Domainfilter&languagedetection

Ru/Ukpipeline

MT:Ru/Uk–>

Eng

Ru/UkEntityandEventdetection

EventframeassemblyII

CSRCombination

Imagepipeline

Entitydetection

PersonandGeoID

Englishentity

detection

Englishevent

detection

Englishentitylinking

Engentityrelations

EventframeassemblyI

Englishpipeline

Englishargumentdetection

Englishcoref

Speechpipeline

13

1.Entitydetection:Type-basedNERdata

•  Multi-levellearning:– Trainseparatedetectorsfortype,subtype,andsubsubtype-leveltypeclassification

– Addressesdataimbalance– Mayintroducelayer-inconsistenttypes!

•  Type-levelfromLDContology:– Trainingdata:KBPNERdataandasmallamountofself-annotateddata

•  Sub(sub)type-level:– Trainingdata:YAGOknowledgebase(350k+entitytypes)obtainedfromHengJi—thanks!

14

2

3

2.Entitylinking

•  Task:GivenNERoutputmentions,linkthemtothereferenceKB

•  Challenges:Over-largeKB,noisyGeonames–  PreprocessKB:Removeduplicatedandunimportantentries(i.e.,notlocatedinRussiaorUkraine,ornoWikipediapage)

•  Approach,givenanentity:– UseLucenetofindallcandidatesinKB–  Filterspuriousmatches–  Buildconnectednessgraph,withPageRanklinkstrengthscores

–  Prune(densify)graphtodisambiguateentity15

2

3.Entityrelationextraction

•  Task:Extractentitypropertiesandeventparticipants•  Four-stepapproach:

1.  BERTwordembeddingsforfeatures2.  Convolution:extractandmergealllocalfeaturesforasentence3.  Piecewisemaxpooling:splitinputintothreesegments(byposition)

andreturnmaxvalueineachsegment,for2entities+1relation4.  Softmaxclassifiertocomputeconfidenceofeachrelation

16

1

Englishentity/relationdiscussion

•  Challengesandproblems– Subsubtypeissuperfine-grained;ourNERengineisstillnotrobustenough

– Wereturnbothtypeandsubsubtypelabels,butintheevalNISTwilljudgeonlyoneofthem

•  Mostlylearned,butsomemanualassistance

17

321 2

TA1RUSSIANANDUKRAINIANMariiaRyskina,Yu-HsuanWang,AnatoleGershman

18

OPERATA1framework

Input

Mini-KBcreation/AIFvalidation

Domainfilter&languagedetection

Ru/Ukpipeline

MT:Ru/Uk–>

Eng

Ru/UkEntityandEventdetection

EventframeassemblyII

CSRCombination

Imagepipeline

Entitydetection

PersonandGeoID

Englishentity

detection

Englishevent

detection

Englishentitylinking

Engentityrelations

EventframeassemblyI

Englishpipeline

Englishargumentdetection

Englishcoref

Speechpipeline

19

Goalsandchallenges

•  Goal:ExtractentityandeventmentionsfromRussianandUkrainiantext,andbuildframes

•  Challenges:– Lackofpretrainedoff-the-shelfextractors– Lackofannotateddatatotrainsystems– Highlyspecificontology

•  Twopipelines:1.  RusandUkrsourcetext2.  MTintoEnglish

20

ExampleinputandoutputInput:Про-российскиесепаратистыатаковалиКраматорскийаэропорт.Translation:Pro-RussianseparatistsattackedKramatorskairport.Output:mn0:eventConflict.Attack, text:атаковали

Attacker:mn1,Target:mn3mn5:relationGeneralAffiliation.MemberOriginReligionEthnicity

Person:mn1,EntityOrFiller:mn2, text:Про-российскиесепаратистыmn6:relationPhysical.LocatedNear, text:Краматорскийаэропорт

EntityOrFiller:mn3,Place:mn4

mn1:entityORG, text:Про-российскиесепаратистыmn2:entityGPE.Country.Country, text:Про-российскиеmn3:entityFAC.Installation.Airport, text:Краматорскийаэропортmn4:entityGPE.UrbanArea.City, text:Краматорский

21

Approach1:ProcessinginRus/Ukr

UniversalDependencyParsing

ConceptualMentionExtraction(COMEX)

StanfordNLP UDPipe

Ontology Lexicon

•  OurontologyisasupersetoftheNIST/LDContology•  Lexiconsare(semi-)manuallycreatedfromthetrainingdata•  Conceptualextractionusing(manual)rule-basedinference•  Focusisonhighprecision

22

5

Parsing/tagging/chunkingpipeline

•  Syntaxpipeline:– UDPipe1.2(Straka&Strakova2017)–  Extractheadnounsanddependents– Notallentitiesandeventsneeded

•  Eventframeconstruction:COMEX– OurontologyisasupersetoftheAIDAontology–  Triggertermsmanuallymappedtoontology:

•  Directmatching—manuallycuratedlistoftriggerwords•  Englishtriggers—translationorWordNet/dictionarylookup

– Analysisguidedbyannotation:•  LDCannotationsfromseedlingcorpus•  Ownmanualannotationaswell

23

5

5

COMEXontology

*fighter-planeLDC_ent_146

*airplaneLDC_ent_142

*vehicleLDC_ent_140

*weaponLDC_ent_160

*physical-entity

*entity

*mil-vehicleLDC_ent_145

*MiG-29

•  Multipleinheritance•  Greatercoverage

24

5

COMEXlexicons

•  Connectwordstoontologyconceptsviawordsenses•  Providerulesforconnectingconceptsintoamentiongraph•  Semanticrequirementsforslotfillersarespecifiedintheontology

W,атаковать,WS:attack-physical,WS:attack-verbalS,WS:attack-physical,*attack-physical,VERBA,WS:attack-physical, Attacker=Pull:active-subj;Pull:passive-subjA,WS:attack-physical, Target=Pull:active-dir-obj;Pull:passive-dir-objA,WS:attack-physical, Instr=Pull:active-subjA,WS:attack-physical, Place=Pull:obl-in#R,Pull:active-subj, nsubj, Trigger->Voice=ActR,Pull:passive-subj, obl, Trigger->Voice=Pass,Target->Case=Ins

Whilethelexiconscontainhundredsofwords,thenumberofrulesissmall25

Lexiconconstruction•  InitialvocabularyandthecorrespondingconceptsfromtheavailableLDCannotations

•  VocabularyenrichmentbyextractingallnamedandnominalentitiesfromtheseedlingcorpusfilesthatcontainatleastoneLDCannotation

•  EventtriggerenrichmentusingWordNet•  Cross-languagevocabularyenrichmentusingMTandalignment•  Manualcurationoftheresultingvocabulary•  Manualadditionofattributerules•  Iterativeimprovementprocess:

1.  Extractmentionsfromanewfile2.  Scoreresults3.  Addvocabulary,fixrulesanddocross-languagetransfer

26

5

SampleCOMEXperformance

English Russian UkrainianPrecision 0.91–1.0 0.93–1.0 1.0Recall 0.22–0.56 0.11–0.70 0.07–0.42F1 0.35–0.70 0.20–0.62 0.13–0.59Vocabulary 178 1483 1430*Rules 33 30 13

COMEXisthemost‘manual’ofOPERA’sTA1extractionmodules

27

(Thisworkcontinues;thenumberschangeeveryday)

Approach2:Rus/Ukr—>English•  Pipeline:

– MTRus/Ukr–>EnglishusingMSAzure–  RunOPERATA1extractors–  AlignsourcetexttoextractedmentionsinEng

•  Back-translatefromEng,includingXML-likeentity/eventtags

•  Outputisgenerallygood(espwhennoXMLtags)

•  Problemsinback-translation:–  SometimesmessesuptheXMLtags– Mayswitcheventarguments– Maymessuppropernames(e.g.Slavyansk–>Slavska,Slavovsk,Slavic

–  ThingsliketyposoruncommonwordsgettranslatedincorrectlyintoEng,butmaybeeasytofixinthesourceusingfuzzymatching 28

2

MT

Approachescomplementary

•  Rus/Ukr:moreprecise–  Lessnoise,betterentitytyping

•  MT:moregeneral–  Betteratnames,time/numbers,eventtyping

•  Overlapsanddifferences:–  Entityoverlap:84%ofRus/Ukr=44%ofMToutput–  Eventoverlap:58%ofRus/Ukr=49%ofMToutput–  Typeagreement:87%ofoverlap–  Remainingmentions:65–70%correctoneachside– Differencesinspans,eventvs.entitychoices

29

Rus/Ukrentity/relationdiscussion

•  Challengesandproblems– Slowmanualrulebuilding,limitedcoverage(buthighprecision)

– COMEX<—>AIDAontologyalignment– Noiseintranslation

•  Mostlymanual

30

52 5 5

TA1/2KBCONSTRUCTIONANDVALIDATION

HansChalupsky

31

OPERATA1framework

Input

Mini-KBcreation/AIFvalidation

Domainfilter&languagedetection

Ru/Ukpipeline

MT:Ru/Uk–>

Eng

Ru/UkEntityandEventdetection

EventframeassemblyII

CSRCombination

Imagepipeline

Entitydetection

PersonandGeoID

Englishentity

detection

Englishevent

detection

Englishentitylinking

Engentityrelations

EventframeassemblyI

Englishpipeline

Englishargumentdetection

Englishcoref

Speechpipeline

32

CSR:PowerLoom-basedCommonsemanticrepository

•  ContainsallKEs–  Containsdiscretetermpropositions,[structured]distributionalvectors/tensors,continuousembeddings

–  Eachwithvectorofscores(e.g.,TA1extractionconfidence,sourcetrustworthiness,reasoningimplicationconfidence,cross-KEcompatibility,hypothesis-basedlikelihoods,etc.)

•  RepresentedinPowerLoom(Chalupskyetal.2010)–  Predicate-logic-basedrepresentationbasedonKIFthatisasupportedsyntaxofCommonLogic

–  Dynamic,scalable,multi-contextualsystemtostore,manageandreasonwithinformation

–  Blazegraphdatabasetechforscalabilityandintegration–  Representhypothesesandprobabilitiesviareification

•  IntheCSReverythingisahypothesis33

5

4

M9Approach:3-stepdecouplingforKBconstructionandvalidation

Multi-mediaAnnotations

NEREntityCorefEDLDBPediaEventsEventCorefRelationsAudioImage,Video…..

Extractors×N Annotation Ontology

Domain Ontology

AIDASeedling

34

KnowledgeAggregation

HypothesisIntegration,Evaluation,Inference

Domain Ontology Domain

Ontology Export

Ontologies

AIDAFull

…..

BackgroundKBBlazegraphTriplestore

Lotsoftypeheterogeneity

ReuseTACEVICdomainmodel

Exporttodifferenttargets

M18Approach:SingleaugmentedontologyforKBconstructionandvalidation

Multi-mediaAnnotations

NEREntityCorefEDLDBPediaEventsEventCorefRelationsAudioImage,Video…..

Extractors×N Domain Ontology

35

KnowledgeAggregation

HypothesisIntegration,Evaluation,Inference

AIDAFull

BackgroundKBBlazegraphTriplestore

Supportsometypeheterogeneity

Domainadditions,APItocode

Annotation Augmentations

Domain Augmentations

5

2

Incrementalcycleofhypothesisrepresentation,evaluation,refinement

Createhypothesis

representation

NERDoc.CorefEDLEventsEventCorefRelationsAudioImage,Video…..

Extractors×NDomain

Ontology

Select Import

Infer,EvalFix

Rules, Constraints

KB

•  Cycle:–  Usecorefsandotheridentitytoconnectannotations(mentionoverlap,

namelinks,EDL,within-doccoref,eventcoref)–  Applyinferences,evaluateconstraints,detectconflicts,doattribution–  Fixconflicts“ViktorYanukovych”?=“ViktorViktorovychYanukovych”

—no:irreflexive(parent)36

52

TA1/2KBintegrationchallenges•  Challenges:Ontological

–  Multipletypesystems:NERtypes,relationtypes,eventtypes,KBschemas,targetschemas…

–  Missingtypes,conflictingtypesoncethingsarelinked–  Types,eveniffine-grained,primarilyusefulasconstraints,notas

equalitysignal–“Humvee17generally-not-equal-toHumvee42”–  Inferencerequirements:“Donechyna”and“Ukraine”arecompatible

locationsofaneventbutnotwithrespecttohaving“Donetsk”astheircapital

–  Ontological“fluidity”—thingschangeuntillateinthegame

•  Challenges:Datasparsityandnoise–  Multi-lingualnamesandcross-lingualmatching–  Language-specificnamingschemes(e.g.,patronyms)–  Cross-lingualuseofcontextvectors–  Nofine-graineddocument,textormediacontextallowedacross

documents–  Linkingdecisionsaggregatesupportandontologicalconflictwhich

propagates 37

5

4

5

TA1scores

TA1Classqueries TA1Graphqueries

BestMAP

WorstMAP

TRECMAP

0.4843 0.4737 0.4773

0.4527 0.3697 0.4020

0.4379 0.2816 0.3278

0.4243 0.1470 0.1957

0.2290 0.0892 0.1244

Prec Recall F10.4715 0.2163 0.29660.4944 0.1328 0.20940.3605 0.0533 0.09290.0398 0.0312 0.03500.0138 0.0040 0.0062

Run:TA1a_OPERA_TA1a_aditi_V2

OPERA

TA3HYPOTHESISCONSTRUCTIONAditiChaudhary,AnatoleGershman,JaimeCarbonell

39

OPERATA2+TA3framework

Coref

Mini-KBcreation/AIFvalidation

TA1Mini-KB

Mini-KBcreation/AIFvalidation

Hypothesisconstruction

BeliefGraphconstruction

TA2Mini-KB

Queryinput

40

Candidatehypothesisgeneration

1.  (HavecompletedBeliefGraphandbeliefscorepropagationthroughout)

2.  RetrieveKEscorrespondingtoentrypoints

3.  RetrieveeventsEandrelationsRthatmatchconstraints.If:–  Zero-hop:theretrievedeventisonehypothesis–  One-hop:obtainaneventforeveryrole.Prioritizeevents/relationswithmaximumoverlapwiththeroles—thismaygivemanypermutations

4.  GeneratehypothesiscandidatesetH=h1,h2...hnfromtheretrievedEandR

5

Approach

Mention

matcher

BG

EvidenceKEs

Information Need

statement

EntrypointsarematchedtoEvidenceNodes

HypothesisGraph

candidates

HypothesisGraphselector

Role

inference

Augmented

HypothesisGraphs

Hypothesis

rankingand

filtering Hypotheses

42

Candidatehypothesisgeneration

Hypothesisranking

•  GiveninformationneedIandcandidatesetH=h1,h2...hn,weneedtorankHbasedonrelevanceanddiversity

•  Maximalmarginalrelevance:MMR=

– Sim(hi,I)=similarityscorebetweenhypothesishiandtheinformationneedI—givesrelevance

– Sim(hi,hj)=similarityscorebetweenhypotheseshiandhj—givesdiversity

λargmaxhi,E,HSim(hi,I)-(1-λ)argmaxhj,E,HSim(hi,hj)

Relevanceanddiversity

•  Sim(hi,I)=Measuringrelevance… sumof:–  PercentageofframescoveredinI–  Percentageofeventssatisfyingtheeventframes–  Percentageofrelationssatisfyingtherelationframes–  Numberofrole-entityexactmatchconstraints

•  Sim(hi,hj)=Measuringdiversity(=inversesimilarity)betweentwohypotheses…sumof:–  Numberofoverlappingevents–  Numberofoverlappingrelations–  Numberofoverlappingentities–  Numberofoverlappingargumentsfortheaskedframes

TA3M18evaluation

•  Thiswasanextremelycomplextask•  Wereceivedalotofnumbers•  We’restillanalyzingthem

•  Mostofthenumbersarenothelpfulforus•  Manythingsconfuseus•  Wewishformoredetailaboutcertainaspects

46

Task3a(usingown/anyTA2KB)

Hypossubmit-ted

Theoriesmatched

Correct-ness

Edgecohe-rence

KEcohe-rence

Relstrict

Rellenient Coverage

24 6 0.4393 0.4834 0.6655 0.2832 0.6554 0.0320

42 4 0.2607 0.2894 0.423 0.1343 0.4192 0.0127

7 1 1.0000 1.0000 1.0000 1.0000 1.0000 0.0035

2 1 0.4167 0.4167 1.0000 0 1.0000 0.0032

42 1 0.3864 0.4475 0.5851 0.3295 0.4836 0.0032

OPERA

Task3a(usingotherTA2teams’KBs)

Hypossubmit-ted

Theoriesmatched

Correct-ness

Edgecohe-rence

KEcohe-rence

Relstrict

Rellenient

Coverage

34 2 0.1042 0.2178 0.3711 0.1002 0.3512 0.0079

20 2 0.5107 0.6072 0.8145 0.3823 0.7973 0.0061

7 1 1.0000 1.0000 1.0000 1.0000 1.0000 0.0035

2 1 0.4167 0.4167 1.0000 0 1.0000 0.0032

42 1 0.3864 0.4475 0.5851 0.3295 0.4836 0.0032

Task3b(usingLDCKB)

Hypossubmit-ted

Theoriesmatched

Correct-ness

Edgecohe-rence

KEcohe-rence

Relstrict

Rellenient

Coverage

42 2 0.5961 0.6249 0.839 0.3829 0.839 0.0238

45 6 0.8065 0.8069 0.9589 0.6263 0.9589 0.0153

42 4 0.8266 0.8612 0.8979 0.8312 0.9034 0.0100

24 2 0.8207 0.8406 1.0000 0.5905 1.0000 0.0060

29 1 0.8925 0.9063 1.0000 0.7421 1.0000 0.0051

Hypothesisassessmentprocedure

•  Assessmentprocedure:– Firstassesseachedgeascorrect/incorrect– Foronlycorrectones,matchagainstthegoldprevailingtheory

•  Howassess/match?Decisions:– TypeandinformativementionofEdge– TypeandinformativementionofLeftside– TypeandinformativementionofRightside

50

Definedinontology.Smalldifferences.

Undefined.Manydifferencesofopinion

Confusion#1:Initialedgefiltering

•  Differenceinassessedhypothesisscoresonsamegold-standardLDCKBinput:

•  Whythediscrepancies?OurSIN-drivenhypothesiscreationwasdifferent.ButwhydoesLDC’sownPrecisionnotgetupto.80? 51

Edgescorrect EdgessubmittedOPERA 59.6% 2545BBN 80.6% 908GAIA 82.1% 571UTexas 82.6% 1079PNNL 89.3% 394LDConTA2 0.59Precision

Confusion#2:•  WhydidGAIAdoalotbetteronGAIA’sownKBsthanonLDC’sKBs?

52

0.0320 _version2_QueryTypeBthroughE_GAIA_1.GAIA_2.GAIA_2_v2 GAIA2_v2runningonGAIAKBs

0.0248 _version4_QueryTypeBthroughE_GAIA_1.GAIA_2p.GAIA_2 GAIA2runningonGAIAKBs

0.0238 _version2_QueryTypeBthroughE_LDC_2.LDC_2.OPERA_TA3b_2 OPERArunningonLDCKBs

0.0153 _version4_QueryTypeBthroughE_LDC_2.LDC_2.BBN_TA3_v2a BBNrunningonLDCKBs

0.0127 _version2_QueryTypeBthroughE_OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.OPERA_TA3a_2 OPERArunningonOPERAKBs

0.0100 _version3_QueryTypeBthroughE_LDC_2.LDC_2.UTexas_3 UTexasrunningonLDCKBs

0.0079 _version3_QueryTypeBthroughE_GAIA_1.GAIA_2.OPERA_TA3a_1 OPERArunningonGAIAKBs

0.0061 _version2_QueryTypeBthroughE_BBN_1.BBN_TA2_v2.GAIA_2 GAIArunningonBBNKBs

0.0060 _version2_QueryTypeBthroughE_LDC_2.LDC_2.GAIA_2 GAIArunningonLDCKBs

0.0051 _version3_QueryTypeBthroughE_LDC_2.LDC_2.PNNL_sheafbox_10 PNNLrunningonLDCKBs

coverage

Confusion#3:Informativementions

53

evt/rel:data:relation-instance-HYP-E102-3-r201907150216-23type:ldcOnt:Physical.LocatedNearhandle:womanofOdessaedgeprov:HC000Q7MI:(5700-0)-(5714-0)womanofOdessa-------------------------edge:ldcOnt:Physical.LocatedNear_EntityOrFillerarg:data:entity-instance-HYP-E102-3-r201907150216-0handle:thestrangledwomanofOdessa,whoforpro-Russianshas

becomeasymboloftheWestspartialityintheUkrainiancrisisassessed:CORRECT-------------------------edge:ldcOnt:Physical.LocatedNear_Placearg:data:entity-instance-HYP-E102-3-r201907150216-24handle:Odessaassessed:WRONG

Arg1:thewoman

54

edge:ldcOnt:Physical.LocatedNear_EntityOrFillerarg:data:entity-instance-HYP-E102-3-r201907150216-0handle:thestrangledwomanofOdessa,whoforpro-Russianshasbecomea

symboloftheWestspartialityintheUkrainiancrisisconf:1.000000assessed:CORRECTbestPT:E102Theory4argprov:HC000Q7MI:(5686-0)-(5804-0)thestrangledwomanofOdessa,whoforpro-Russianshasbecomeasymbol

oftheWestspartialityintheUkrainiancrisiscontext:xactlywhathappened.Thisscarcityofinformationexplainswhysomany

rumourshaveemergedaround>>thestrangledwomanofOdessa,whoforpro-RussianshasbecomeasymboloftheWestspartialityintheUkrainiancrisis<<.Thisphotowasoriginallypostedhere.

Arg2:Odessa

•  Whyisitwrong?Sometheories:– OdessaisnotLocatedNearOdessa,itISOdessa–  ThewomanwasborninOdessabutnowlivesinKiev–  Thewomanwasonlyrumoredtohavebeenstrangled– …andmore… 55

edge:ldcOnt:Physical.LocatedNear_Placearg:data:entity-instance-HYP-E102-3-r201907150216-24handle:Odessaconf:1.000000assessed:WRONGargprov:HC000Q7MI:(5709-0)-(5714-0)Odessacontext:hisscarcityofinformationexplainswhysomanyrumourshaveemerged

aroundthestrangledwomanof>>Odessa<<,whoforpro-RussianshasbecomeasymboloftheWestspartialityintheUkrainiancrisis.Thisp

Challenges

•  Hypothesesgraphsaretooextensive,aseventsareconnectedbyatleastonecommonargument—needtoaddrestrictions

•  Backgroundknowledgesometimesrequiredtolinkentities(e.g.,SU25==militaryjet)—perhapspre-populateKBwithbackgroundknowledge?

56

FINALE

57

Finale

•  OPERAisanend-to-endsystem–  Successfulcombinationofmachinelearningandmanualcomponentsandapproaches

–  Task3b(usingLDC’sKB)submissionhadthehighestcoverage

– Managedwithlimiteddataandchangingontology•  Absorbed&processedGAIATA1&TA2outputs

– GottopTA2graphqueryF1(1a)scoreusingGAIAKBs•  Currentfocus:

–  (Ofcourse)improvementseverywhere– Newdomainandontology–  Seriousintegrationofcomponent-levelscores 58

Somediscussionpoints

1.  Multipleinheritanceintheontology

2.  Mainroleofannotateddataisasexamples:continuousteam–LDCinteractiontogtsystemfeedback?

3.  Itishardtoreconstructwhatexactlyassessorssaw.Ifthespecifictextualcontextthatanassessorlookedatforadecisionisrecorded,thenwecan–  seethetexttheybasedtheirjudgmenton–  maybealsogetsomefinerclassificationoftheerrortype

4.  TA1bbias:Component-basedre-processingofsameresultswhengivennewhypotheses

59

60

60

543

2

1