21 August 2003 CLEF 2003

iCLEF-2003 at MarylandHeadline Generation and

Interactive Query Formulation

Douglas W. Oardand the Maryland/BBN iCLEF team

Outline

• Document selection– First 40 translated words– Automatically generated translated “headlines”

• Application: news stories from speech recognition

• Iterative refinement– User-assisted query translation– First 40 words + access to full translation

Berne, 25 May (sda) the Upper House of Parliament supports the objectives that national campaign against racism. It is ready, all possibilities to examine, like the Council of Europe the action of the working group that YouthIn the small urban villa at the Museum in Amsterdam took 11 1987 one of the largest German tax affairs their beginning. Here **time-out** hid the company Sunpark sport B.V. Steffis income before that German treasury. Nut/mother ofIslamabad, 26th Dec (sda/afp) the authorities of Pakistan have that on Monday Indian Consulate General in Karachi closed. To the reason that said a minister of foreign affairs of Pakistan Najmuddin Sheikh, India is into unrests in

Berne, 31 January (sda) eight months before the popular vote has itself on Monday in Berne the Swiss federal committee “to the anti racism law” presented. Nearly all parties are, likewise involved employers and Trade unions as well as

Berne, 7 September (sda) the Upper House of Parliament endorses an active participation that Switzerland at the European youth campaign against racism and intolerance. It is ready, so rapidly as possible the necessary decisions to

Geneva, 9 March (sda) the UN human right commission has on Wednesday for the first time a resolution against the racism generally as well as the antisemitism, the racism against Arabs and against black ones discharges. Before one

Zurich, 26 October (sda) the boy LDU does not want Scientologist in its Rows. Executive committee and presidency have one in the statutes incompatability rule imported, like JLDU vice-president Roland Tschaeppeler on Wednesday

St. Gallen, 26 August (sda) of members of FDP of the city pc, Gallen have that Party exclusion of the pc, Galler of physician and opponent of the Anti racism of law walter Fischbacher requests. Like it in one Report of Friday is called.Black humans must legitimize themselves thus again once. The psychological and social effects S.14 of racism however, like among other things chance inequality, bad education and poverty, “incomprehensible-prove” too such IQ

Since their aunt died 1990 in Zurich, George Hug lead and its two brothers and sisters world-wide a fight against physician, those medical allowance with financial interests mix. “gradually”, Barbara Hug reports, well-known

Upper house of Parliament supports objectives that national campaign against racism

Eight months before popular vote has itself in Berne Swiss federal committee presented

Authorities of Pakistan have that on Monday Indian consulate general closed

Swiss federal committee to anti-racism law presented

Upper house of parliament endorses active participation that Switzerland at European youth campaign

UN human right commission has on for first time resolution as well as anti-semitism

Boy LDU does not want scientologists in rows

Since aunt died in Zurich George Hug lead and two brothers and sisters worldwide fight

Of members pc. Gallen have that Party exclusion Galler physician and opponent walter Fischbacher requests

Black humans must legitimize themselves thus again once

Headline Generation

Types of Headlines

• “Eye catcher”:‘Under God’ Under Fire

• Indicative:Pledge of Allegiance; Constitutionality

• Informative:U.S. Court Decides Pledge of Allegiance

Unconstitutional

After months of debate following the Sept. 11 terrorist hijackings, the Transportation Department has decided that airline pilots will not be allowed to have guns in the cockpits.

HMM HedgeForm headlines by selecting words from story, in order, allowing morphological variation

Humans were able to do this on all prose stories

After months of debate following the Sept. 11 terrorist hijackings, the Transportation Department has decided that airline pilots will not be allowed to have guns in the cockpits.

After months of debate following the Sept. 11 terrorist hijackings, the Transportation Department has decides that airline pilots will not be allowed to have guns in the cockpits.

Human-Generated Headlines

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5 6 7 8 9 10

Sentence Number

Hedge TrimmerForm headlines by removing constituents from a parse of the first sentence.

Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields.

28 wordsS

NP VP

TimeNNP VP V PP Adv

S

NNP

NP

VP

VP

AP PPV NP

Hedge TrimmerForm headlines by removing constituents from a parse of the first sentence.

Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields.

24 wordsS

NP VP

TimeNNP VP V PP Adv

S

NNP

NP

VP

VP

AP PPV NP

Hedge TrimmerForm headlines by removing constituents from a parse of the first sentence.

Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields.

20 wordsS

NP VP

TimeNNP VP V PP Adv

S

NNP

NP

VP

VP

AP PPV NP

Hedge TrimmerForm headlines by removing constituents from a parse of the first sentence.

Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields.

18 wordsS

NP VP

TimeNNP VP V PP Adv

S

NNP

NP

VP

VP

AP PPV NP

Hedge TrimmerForm headlines by removing constituents from a parse of the first sentence.

Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields.

14 wordsS

NP VP

TimeNNP VP V PP Adv

S

NNP

NP

VP

VP

AP PPV NP

Hedge TrimmerForm headlines by removing constituents from a parse of the first sentence.

Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields.

12 wordsS

NP VP

TimeNNP VP V PP Adv

S

NNP

NP

VP

VP

AP PPV NP

Human-Generated Headlines

Phenomenon

preambles

Sentence2.7%

Headline0.0%

conjoined S 4% 0.5%

conjoined VP 27% 3%

relative clauses 3.5% 0.3%

determiners 25% 3%

time expressions 24% 1.5%

trailing prepositional phrases

58% 52%

trailing SBARs 16% 8%

Named Entity Preservation• Syntactic structure alone is not sufficient for

deciding which constituents to remove• Examples:

– Original: According to local authorities the death toll is expected to reach 1,500 with 7000 wounded in Algeria’s earthquake. (18)

– without NEP: According to local authorities the death toll is expected to reach 1,500 with 7000 wounded in Algeria’s earthquake. (10)

– with NEP: According to local authorities the death toll is expected to reach 1,500 with 7000 wounded in Algeria’s earthquake. (10)

Intrinsic Evaluation: BLEU

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

1gram 2gram 3gram 4gram

BBN/UMD StatBBN/UMD Stat2BBN/UMD TrimBBN/UMD TopicISI GenISI Gen10ISI Gen15ISI Tran

BBN/UMD Trim

Red Scores

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

RED1Avg RED2Avg RED3Avg RED4Avg

BBN/UMD Stat

BBN/UMD Stat2

BBN/UMD Trim

BBN/UMD Topic

ISI Gen

ISI Gen10

ISI Gen15

ISI Tran

BBN/UMD Trim

Intrinsic Evaluation: RED

Making Translated Headlines

Systran: TranslateFirst Sentence

Hedge Trimmer

“Headline”

Document

Manual Intrinsic Evaluation

Surprise Language Summarization Scores

0

0.5

1

1.5

2

2.5

3

3.5

Q0 Q1 Q0 Q1 Q0 Q1 Q0 Q1

Subject 0 Subject 1 Subject 2 Average

HMM

UTD

Trimmer

Q0: Can you tell what this story is about?

Q1: Were you (was it) correct?

iCLEF Results

• Search effectiveness– First40: Fα = 0.473

– Trimmer: Fα = 0.379

• Search speed– First40: 16.75 seconds/doc– Trimmer: 12.35 seconds/doc– Significant with p<0.007

Topic Questionnaires

Q1: Were you familiar with this topic before the search?

Q2: Was it easy for you to guess what the document

was about based on the surrogate?

Q3: Was it easy to make relevance judgments?

Q4: Do you have confidence in your judgments?

First40 Trimmer

Question 1 2.09 1.97

Question 2 3.65 2.91 p<0.02 Question 3 3.75 3.28

Question 4 3.78 3.13 p<0.03

Focus: Query Formulation

Choose Document-Language

Terms

Query-DocumentMatching

GeneratePossible

Translations

Select Document-Language

Terms

Document

Author

Query

Choose Document-Language

Terms

MonolingualSearcher

Choose Query-Language

Terms

Cross-LanguageSearcher

2003 End-to-End Experiments

• 64 trials (8 participants)– Half with automatic query translation– Half with user-assisted query translation

• Spanish documents– Systran translations into English

• Experienced searchers– 6 of 8 were librarians, average age 33– 7 of 8 had little or no experience with MT– 8 of 8 reported poor or no Spanish skills

Examples from Parallel Text

Bilingual Term List

Sentence aligned Corpus

English Word te Spanish Word tgtranslation

Passage containing te Passage containing tgaligned with

English Text + Translation Lexicon

• Create a pool of candidate examples– Find possible synonyms w/round-trip translation– Select all translations with any “back translation”

• Find terms that characterize the pool– Rank non-stopwords by sentence-based tf*idf– Select top 10 terms as a query

• Choose one example of usage– Rank candidate pool sentences using the query– Select the top-ranked sentence

2002 ResultsF

α=

0.8

20 minutes per topic

2003 Results

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1 2 3 4 5 6 7 8 9

Topic

Automatic

User-Assisted

Fα

=0.

8

10 minutes per topic