Kim Harris with Aljoscha Burchardt (DFKI), Hans Uszkoreit (DFKI), Arle Lommel (CSA)

BEYOND AUTOMATED QUALITY SCORES

From BLEU to professional error annotation in MT quality estimation and improvement

Kim Harris • TAUS Roundtable Vienna 2016

• The closer a machine translation is to a professional human translation, the better it is

• Relatively high correlation with human judgements• One of the most popular automated and inexpensive

metrics.• Automated quality scores based on comparisons with sets

of HT references• Can be useful for certain estimation tasks but not for improvement• No ability to assess why scores improve or worsen• Focus on the score and not the actual quality

BLEU: Status Quo

Kim Harris • TAUS Roundtable Vienna 2016

• MQM/DQF error annotation for HT and MT• Analysis of quality based on real issues• Ranking/estimation properties• Use results to improve output

Error Annotation for MT Improvement

Kim Harris • TAUS Roundtable Vienna 2016

Annotation: Humans in the HQMT loop

Kim Harris • TAUS Roundtable Vienna 2016

Error profiles based on MQM annotation

By languages By system types

Kim Harris • TAUS Roundtable Vienna 2016

Error profiles

Kim Harris • TAUS Roundtable Vienna 2016

Error and source barrier analysis• Moving away from completely automatic • Analyse MQM errors, linguistic phenomena in target MT• Compare to source phenomena• Test suite analysis

• Basis for improved quality translation in MT thanks to categorization and markup of translation barriers in source language

• Mapping (almost) all linguistic phenomena for one language

• determine possible relationships between phenomena in the source and errors in the target

• can be used to test different MT systems and domains

New paradigm in HQMT

Kim Harris • TAUS Roundtable Vienna 2016

Enter: The Test Suite

Kim Harris • TAUS Roundtable Vienna 2016

Structure of Barrier Categories

Kim Harris • TAUS Roundtable Vienna 2016

Beyond BLEU

Kim Harris • TAUS Roundtable Vienna 2016

The Bigger Vision

Quality Translation 21 (QT21) has received funding from the EU’s Horizon 2020 research and innovation programme under grant no. 645452. META-QT has received funding from the EU’s Horizon 2020 research and innovation programme through the contract CRACKER (grant agreement no.: 645357). Formerly co-funded by FP7 and ICT PSP through the contracts T4ME (grant agreement no.: 249119), CESAR (grant agreement no.: 271022), METANET4U (grant agreement no.: 270893) and META-NORD (grant agreement no.: 270899).

Thank you!