harnessing disagreement for event semantics

Harnessing Disagreement for Event Semantics

Lora Aroyo Chris Welty

Objects vs. Events

events perdure = their parts exist at different time points objects endure = they have all their parts at all points in time

objects are wholly present at any point in time, events unfold over time

Flickr: vanilllaph

Event Extraction in NLP

•  Find Events mentioned in text •  Type them (communication, bombing, …) •  Find the role fillers (location, date,

participants, …)

•  State of the art is extremely low: .10 F

Measuring •  Define the task for human annotators •  Annotate by multiple people •  Measure agreement •  While k<.6 refine definition and repeat

... for events this process is long, disagreement

is high, agreement becomes forced

Position

Human disagreement about events are part of their semantics

Flickr: elkabong

Events are Vague Humans have no clear notion of what

events are

“event is a significant "happening" or gathering of people. I would define a "happening" as an event if the group of people gathered were united in one common goal.”

We Asked the Crowd What an EVENT is Flickr: massimo vitali

We Asked the Crowd What an EVENT is

Event is a happening, which can be scheduled or unscheduled. An earthquake or fire happens (unscheduled). A wedding or birthday party (scheduled). It is an occasion that is unusual and tends to be memorable.

“An event would be any occurrence where physical action has taken place. It may be a single, momentary instance (I sneezed), or it may span a period of time (the festival ran for four hours). An event may also be made up of a number of smaller events, such as a day at school is an event, but each individual class is also an event itself. Basically an event must have a physical action over any delimited time span.”

We Asked the Crowd What an EVENT is

“A planned public or social get together or occasion.”

“an event is an incident that's very important or monumental”

“An event is something occurring at a specific time and/or date to celebrate or recognize a particular occurrence.”

“a location where something like a function is held. you could tell if something is an event if there people gathering for a purpose.”

“Event can refer to many things such as: An observable occurrence, phenomenon or an extraordinary occurrence.”

We Asked the Crowd What an EVENT is

What do Experts think an EVENT is?

“an event is the exemplification of a property by a substance at a given time” Jaegwon Kim, 1966

“events are changes that physical objects undergo” Lawrence Lombard, 1981

“events are properties of spatiotemporal regions”, David Lewis, 1986

under30ceo.com

What do Experts think an EVENT is?

“an event is the exemplification of a property by a substance at a given time” Jaegwon Kim, 1966

“events are changes that physical objects undergo” Lawrence Lombard, 1981

“events are properties of spatiotemporal regions”, David Lewis, 1986

under30ceo.com

nothing everything

Why is event semantics hard?

the World is Open

1. events have multiple dimensions 2. each dimension has levels of granularity

3. people have different views on both

all this leads to very complex semantics

and our goal is ...

1. not to enforce agreement 2. to capture different view points

3. to teach machines to reason in the disagreement space

Flickr: elkabong

Hypothesis Artificially restricting humans does not help machines to learn.

Machines will learn from diversity

Flickr: elkabong

What do People Disagree on?

are sub-events always mere parts? are “mentions” meaningful for events?

are events coreferential across documents? (e.g. perspectives, observations)

“the bombing targeted a housing development in Baghdad, killing 3 and injuring 13”

indistinguishable by people, confusable: is bombing part of killing, or killing part of bombing?

What about targeting?

“merelogically extensional” (i.e arbitrary): container bursting into fragments as a result of explosion

some events don’t exist:

an action by military forces prevented the bombing.

Disagreement Framework

•  ontology: disagreements on the basic status of events themselves as referents of linguistic utterances, e.g. are people events or do events exist at all.

•  granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc.

•  interpretation: disagreements that result from (non-granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September.

Granularity Disagreement

•  spatial, temporal, participants

•  compositional, classificational

Event Participants Disagreement

Prime minister Benjamin

Netanyahu

Benjamin Netanyahu

Israeli Prime minister

Cabinet

Benjamin Netanyahu’s

Cabinet

Israeli Cabinet

his Cabinet

Israeli Government

{TOLD}

50%

35%

15%

10%

15%

5%

45%

15%

Temporal Disagreement

Spring 1998

March 1, 1998

March 1998

Sunday Prime minister

Benjamin Netanyahu

Benjamin Netanyahu

Israeli Prime minister

{TOLD}

50%

35%

15%

25%

15%

50%

5%

Spatial Disagreement

Lebanon

Israel Southern Lebanon

Israel's Northern Frontier

Middle East

{WILLING TO WITHDRAW}

35%

45%

10%

30%

65%

Approach Principles

1.  tolerate, capture & exploit disagreement 2.  understand the range of disagreements

by creating a space of possibilities with frequencies & similarities

3.  score machine output based on where it falls in this space

4.  adaptable to new annotation tasks

Flickr: auroille

Position

Human disagreement about events are part of their semantics

Flickr: elkabong

Conclusion Artificially restricting humans does not help machines to learn.

Machines will learn from diversity

Flickr: elkabong