tto3 semantic annotation analysis bonnie dorr september 9, 2008
Post on 18-Dec-2015
220 Views
Preview:
TRANSCRIPT
TTO3 Semantic Annotation Analysis
Bonnie Dorr
September 9, 2008
Analysis of Annotation Study
Interannotator AgreementMulti-translation
EquivalenceMulti-language
Equivalence
Modality 95% 75% equivalence NA (no Arabic)
Temporal Type 95% 100% equivalence 62% equivalence
Inherent Time 96% 93% equivalence 85% equivalence
Temporal Parent 85% 100% equivalence 92% equivalence
Temporal Relations 97% 100% equivalence 75% equivalence
Belief NA (not doubly annotated) 100% equivalence 90.5% equivalence
Dialog Acts NA (not doubly annotated) NA (no multi-trans) NA (no parallel docs)
Event Coref NA (not doubly annotated) 100% equivalence NA (no parallel docs)
Annotation Agreement: UMBC Modality
*Loose = Within .2
Modality Item Strict Loose* Example
Belief (13) 96% 46% 100% Expect = .9
Epistemic (8) 94% 63% 88% May = .8
Epiteuctic (5) 91% 34% 95% Get = 1
Obligative (62) 99% 34% 98% Should = .7
Permissive (15) 91% 60% 100% Allow = .8
Potential (31) 98% 74% 87% Can = .8
Volitive (19) 95% 37% 95% Prefer = .7
Evaluative (11) 100% 100% 100% Best = 1
OVERALL (164) 97% 52% 95%
Labeling Equivalence: UMBC Modality
Modality Item Strict Loose* Example
Potential/Permissive (1) 100% 0% 0% Can/Allow = .8
Volitive (2) 100% 50% 100% Want = .7
Evaluative (1) 100% 100% 100% Best = 1
OVERALL (4) 100% 50% 75%
*Loose = Within .2
Modality Example 1Sheikh Mohamed, who is also the Defense Minister of the United Arab Emirates, announced at the inauguration ceremony that "we want to make Dubai a new trading center." INFORM-100(announce)
HUM-92(Shiekh Mohammed)CEREMONY-102(ceremony)MODALITY-103(want)[TYPE=VOLITIVE,VALUE=1]
CHANGE-3(make)GEO-POL-ENT-106(Dubai)LOCATION-OF-5()
COMMERCE-EVENT-108(trade)
Sheikh Mohammed, who is also the UAE Defence Minister, said during the launching ceremony "we wish to make Dubai a center of modern economy". ASSERTIVE-ACT-177(say)
HUM-172(Shiekh Mohamed)CEREMONY-179(ceremony)MODALITY-180(wish)[TYPE=VOLITIVE,VALUE=1]
EVENT-182(make)GEO-POL-ENT-184(Dubai)CENTER-POINT-186(center)
ECONOMY-187(economy)
Equivalence: Strict = 100%, Loose = 100%
Modality Example 2The minister, {who has his own website}, also said: "I want Dubai to be the best place in the world for state-of-the-art technology companies."
ASSERTIVE-ACT-70(say)MINISTER-67(minister)MODALITY-71(want)[TYPE=VOLITIVE,VALUE=.8]
MODALITY-6()[TYPE=EVALUATIVE,VALUE=1]GEO-POL-ENT-74(Dubai)FOR-PROF-CORP-79(company)
TECHNOLOGY-78(technology)
The minister {who has a personal website on the internet}, further said that he wanted Dubai to become the best place in the world for the advanced (hitech) technological companies.ASSERTIVE-ACT-941(say)
MINISTER-936(minister)DISCOURSE-940(further)MODALITY-942(want)[TYPE=VOLITIVE,VALUE=1]
CHANGE-EVENT-944(become)MODALITY-2()[TYPE=EVALUATIVE,VALUE=1]
GEO-POL-ENT-943(Dubai)FOR-PROF-CORP-949(company)
Equivalence: Strict = 0%, 100%, Loose = 100%, 100%
Note: {} units omitted for simplificity
Modality Example 3He said companies engaged in e-commerce would be able to set up offices, employ staff and own equipment in the open zone, including fully-owned foreign companies.ASSERTIVE-ACT-157(say)
HUMAN-156(he)MODALITY-170(would be able)[TYPE=PERMISSIVE,VALUE=.8]
SET-168() CAUSE-BY-169(set up)[FOR-PROFIT-158, OFFICE-161]HIRE-162(employ) [SET-163(staff)]OWN-164(own) [DEVICE-165(equipment) [ZONE-167(zone)]]
He said that it would he possible for the companies working in the electronic trade sector to establish offices here and provide employment and own machinery in the zone opened to companies fully owned by the foreigners.ASSERTIVE-ACT-598(say)
HUMAN-596(he)MODALITY-599(would be possible)[TYPE=POTENTIAL,VALUE=.6]
OWN-604(establish)[FOR-PROFIT-600, OFFICE-605]HIRE-606(employment)OWN-607(own) [SET-608(machinery) [PLACE-609(zone)]]
Equivalence: Strict = 0%, Loose = 0%
Annotation Agreement: BBN’s Temporal Annotations
Matches Clashes AgreementTemporal Type 104 6 94.5%Inherent 106 4 96.4%Parent Pointers 93 17 84.5%
(The following values are measured over agreed parent pointers.)
Exact Match Partial Match Clash Exact Agree Part AgreeTemp Relations 81 9 3 87.1% 96.8%
Labeling Equivalence: BBN’s Temporal Annotations
Temporal Type Matches Clashes EquivalenceMulti-Trans: Dubai (E1/E2) 14 0 100.0%Multilingual: Dubai (English/Arabic) 7 6 53.8%Total 21 6 77.8%
Inherent Matches Clashes EquivalenceMulti-Trans: Dubai (E1/E2) 13 1 92.9%Multilingual: Dubai (English/Arabic) 11 2 84.6%Total 24 3 88.9%
Parent Pointers Matches Clashes EquivalenceMulti-Trans: Dubai (E1/E2) 14 0 100.0%Multilingual: Dubai (English/Arabic) 10 3 76.9%Total 24 3 88.9%
(The following value is measured over agreed parent pointers.)
Temp Relations Exact Match Partial Mat Clash Exact Equiv Part EquivMulti-Trans: Dubai (E1/E2) 14 0 0 100.0% 100.0%Multilingual: Dubai (English/Arabic) 7 2 1 70.0% 90.0%Total 21 2 1 87.5% 95.8%
E1: Sheikh Mohammed, who is also the Defense Minister(وزير) announced(واعلن) at the inauguration ceremony(التدشين) that we want(نريد) to make(نجعل) Dubai a new trading center.
E2: Sheikh Mohamed, who is also the Defense Minister(وزير) said(واعلن) during the launching ceremony(التدشين) we wish(نريد) to make(نجعل) Dubai a new center of modern economy
TimeUnit Type Relation Parentminister(الوزير)* Present.State After/Before/Conc announcedannounced(واعلن) Past.Say* Before <writer>ceremony(التدشين) Unspec.Event After/Before/Conc announced*want(نريد) Present.State* After/Before/Conc announced
make(نجعل) Unspec.State* After* want
ANNOUNCE(واعلن) [Past.Say, Before(<writer>)] MINISTER(الوزير) [Present.State, After/Before/Conc(ANNOUNCED)] CEREMONY(التدشين) [Unspec.Event, After/Before/Conc(ANNOUNCED)] WANT(نريد) [Present.State, After/Before/Conc(ANNOUNCED)] MAKE(نجعل) [Unspec.State, After(WANT)]
Temporal Relation Example 1: Multi-translation and Multilingual
<writer>
announced
ceremony
minister
TIME
want
make
Equivalence Arabic-English English-English
Type Match 25% 100%
Parent Match 75% 100%
Relation Match 67% 100%
E1: The minister who has(يملك) his own website also said(واضاف) I want(اريد) Dubai to be the best place(تصبح) in the world for companies.
E2: The minister who has(يملك) a personal website further said(واضاف) he wanted(اريد) Dubai to become the best place(تصبح) in the world for companies.
TimeUnit Type Relation Parenthas(يملك) Present.State* After/Before/Concurrent saidsaid(واضاف) Past.Say* After <announced>(prev sent)*want(اريد) Present.State*+ After/Before/Concurrent saidplace(تصبح) Unspec.State* After want
SAID(واضاف) [Past.Say, After(<announced>)] HAS(يملك) [Present.State, After(SAID)] WANT(اريد) [Present.State, After/Before/Concurrent(SAID)] PLACE(تصبح) [Unspec.State, After(WANT)]
Temporal Relation Example 2: Multi-translation and Multilingual
Equivalence Arabic-English English-English
Type Match 0% 75%
Parent Match 75% 100%
Relation Match 100% 100%
<announced>
said
want
has
TIME
place
E1: He said(وقال) companies engaged in commerce would be able(سيكون) to set up(وتقيم) offices, employ(وتستخدم) staff and own(وتمتلك) equipment in the open zone.
E2: He said(وقال) that it would be possible(سيكون) for companies working in the electronic trade sector to establish(وتقيم) offices here and provide(وتستخدم) employment and own(وتمتلك) machinery in the zone.
TimeUnit Type Relation Parentsaid(وقال) Past.Say Before* <writer>*possible(سيكون) Future.State After saidestablish(وتقيم) Unspec.Event After possibleprovide(وتستخدم) Unspec.Event Concurrent establishown(وتمتلك) Unspec.Event Concurrent provide
SAID(وقال) [Past.Say, Before(<writer>)] POSSIBLE(سيكون) [Future.State, After(SAID)] ESTABLISH(وتقيم) [Unspec.Event, After(POSSIBLE)] PROVIDE(وتستخدم) [Unspec.Event, Concurrent(ESTABLISH)] OWN(وتمتلك)[Unspec.Event, Concurrent(PROVIDE)]
Temporal Relation Example 3: Multi-translation and Multilingual
<writer>
said
establish
possible
TIME
provide
own
Equivalence Arabic-English English-English
Type Match 100% 100%
Parent Match 80% 100%
Relation Match 100% 100%
Labeling Equivalence: CMU and Columbia’s Committed Belief
Belief Type Labeling Labeled Text 1 only Text 2 only Rec Prec F-ScoreMultilingual: Dubai (English/Arabic) 21 18 2 53.8% 91.3% 67.7%Multi-Trans: Dubai (E1/E2) 28 0 11 100.0% 71.8% 83.6%Total 49 18 13 73.1% 79.0% 76.0%
Belief Type Equivalence Assigned Match EquivalenceMultilingual: Dubai (English/Arabic) 21 19 90.5%Multi-Trans: Dubai (E1/E2) 28 28 100.0%Total 49 47 95.9%
Committed Belief Example 1E1: Sheikh Mohamed, who is also the Defense Minister of the United Arab Emirates, announced at the inauguration ceremony that "we want to make Dubai a new trading center."
E2: Sheikh Mohammed, who is also the UAE Defence Minister, said during the launching ceremony "we wish to make Dubai a center of modern economy".
[WILL INSERT EXAMPLE SOON]
Committed Belief Example 2E1: The minister, who has his own website, also said: "I want Dubai to be the best place in the world for state-of-the-art technology companies."
E2: The minister who has a personal website on the internet, further said that he wanted Dubai to become the best place in the world for the advanced (hitech) technological companies.
[WILL INSERT EXAMPLE SOON]
Committed Belief Example 3E1: He said companies engaged in e-commerce would be able to set up offices, employ staff and own equipment in the open zone, including fully-owned foreign companies.
E2: He said that it would he possible for the companies working in the electronic trade sector to establish offices here and provide employment and own machinery in the zone opened to companies fully owned by the foreigners.
[WILL INSERT EXAMPLE SOON]
Labeling Equivalence: CMU Event Coref
Event Coref Equivalence Assigned Strict Match Loose Match Equivalence EquivalenceMulti-Trans: Dubai (E1/E2) 10 7 10 70.0% 100.0%Total 10 7 10 70.0% 100.0%
*Loose = arguments unordered and/or extra unused arguments
Event Coref Labeling Labeled Text 1 only Text 2 only Rec Prec F-ScoreMulti-Trans: Dubai (E1/E2) 10 0 3 100.0% 76.9% 87.0%Total 10 0 3 100.0% 76.9% 87.0%
Event Coref Example 1E1: Sheikh Mohamed, who is also the Defense Minister of the United Arab Emirates, announced at the inauguration ceremony that "we want to make Dubai a new trading center."
E2: Sheikh Mohammed, who is also the UAE Defence Minister, said during the launching ceremony "we wish to make Dubai a center of modern economy".
[WILL INSERT EXAMPLE SOON]
Event Coref Example 2E1: The minister, who has his own website, also said: "I want Dubai to be the best place in the world for state-of-the-art technology companies."
E2: The minister who has a personal website on the internet, further said that he wanted Dubai to become the best place in the world for the advanced (hitech) technological companies.
[WILL INSERT EXAMPLE SOON]
Event Coref Example 3E1: He said companies engaged in e-commerce would be able to set up offices, employ staff and own equipment in the open zone, including fully-owned foreign companies.
E2: He said that it would he possible for the companies working in the electronic trade sector to establish offices here and provide employment and own machinery in the zone opened to companies fully owned by the foreigners.
[WILL INSERT EXAMPLE SOON]
Remaining Analyses
• Additional multilingual and multi-translation sets
• Non-parallel documents about the same event
• Additional genre: Informal texts
• Additional phenomena: Coreference
top related