machine translation as a tool - ku leuven · 2015-01-25 · machine translation as a tool prof. dr....

Machine Translation as a Tool

Prof. dr. Frieda Steurs KU Leuven, Faculty of Arts

Agenda

Introduction Predictions MT usage & the market impact Translators’ profession Quality Productivity Compensation

TM/MT integration Small-scale experiment at KU Leuven

Portsmouth Conference 2014

About the presenter

Postgraduate EMT in Specialised Translation Research Assistant for the SCATE project at KU Leuven Experience with training translators on CAT tools No special expertise in MT

[email protected]


mailto:[email protected]

Smart computer-aided translation environment (SCATE) in a nutshell

Time-frame: 01/03/2014 - 28/02/2018 Consortium: KU Leuven, 6 university teams, 10 companies Funding: Flemish government, industry, IWT Primary goal:

Develop techniques to improve translation efficiency and consistency!

• How? Better exploitation of resources, e.g. translation memories, terminology… Syntax-based fuzzy matching Automatic bilingual term extraction from comparable corpora

Stronger integration of speech recognition models with machine translation

Personalised user interfaces


SCATE Participants

Project Consortium

SCATE Participants

Industrial Advisory Committee

In 6 years from now… We can expect that by 2020 automated translation tools will be embedded into every device, application and website.

TAUS

13 years ago…. A decent MT machine is just around the corner…A computer will never understand, but it can translate, at least to some extent. And, since translation without understanding is meaningless, the future of the human translator is proof- sensing what a machine has pre-translated. Champollion (2001)

About Machine Translation


Present

Is the human translator proof-reading what a machine has pre-translated?

YES


MT for Big Data

Big Language

Many languages

Increasing content volumes


…in the professional market, human translation still rules, though a quiet but pronounced shift to machine translation with post‐editing is unquestionably underway in the agencies, some of whom are also actively developing their own engines using open‐source software.

LT-Innovate Report 2013

What about the LSP’s?

42% of the LSPs use MT

Source: CSA


MT for Translation Rule-based

• Linguistic rules • Specialised dictionaries per

domain, customer, product line!

• Suitable for software documentation, technical manuals, E-learning courseware, UI

• Consistent and predictable quality

• Out-of-domain translation quality

• Lack of fluency • Hard to handle exceptions to

rules • It does not cover all

languages! • Expensive to develop and

customize

Statistical

• Large bilingual corpora • Terminology choices hard to

manage! • Suitable for user-generated

content • Unpredictable translation

quality • Poor out-of domain quality • Does not know grammar • High CPU and disk space

required • Good fluency • Good for catching expeptions

to rules • It covers all languages! • Rapid and cost-effective

development

Hybrid

• Rules +Statistics • Better output • Solves ambiguity • More expensive to manage

two systems!


Systran, Apertium, Babelfish, Prompt…

Google, Moses, Language Weaver, Asia Online…

Systran, Asia Online, Prompt

What is the market impact?

In a nutshell: • Impact on the translation profession

o Deprofessionalization of the translator? o New jobs, e.g. Post-Editor, MT quality evaluator… o How to maintain creativity?

• Quality o MT Process vs. quality of the MT output o What is the final expected quality?

• Productivity o Raw MT output, Light post-editing vs full post-editing

• Compensation o Per word, per hour or both?


Benefits of MT for…

End-clients o Reduce costs

o Reduce time to market o Increase productivity o Translate new content

Translators • New opportunities • Increase productivity • Higher income • More consistency


What do translators say?

…how to protect the language skills and mental function of those routinely exposed to toxic machine language. (Translator’s blog)

I fear it might be the post-MT crash (of our industry, in terms of

rates and quality). (Translator, Proz ) “post-processing of machine translations”, is likely to be much

more deadly to the translating community, because this tool was designed by our beloved “translation industry leaders” to simply kill off the translating profession and replace it by a new type of workers called post-processors of machine translations. (Translator’s blog)


What do translators say? My proposal is that you start investing time and money to “update the skills” of your human base, current freelance translators and interpreters, so that you may “flip it” around....I urge industry leaders to invite us, everyday translators and interpreters, to become an integral part (not a side story) of this equation. We have been severed from the most important conversations about our own future. Many of us are afraid of the new technologies because there is yet no clear answer to the question “what’s in it for me?”. The “industry” as a whole may well benefit from inviting us to become part of the equation going forward. If translators and interpreters do not learn –quickly and swiftly– to use 21st century technologies, we may not survive as a viable profession. Unless all the parties are on the same page, the “industry” as a whole may not evolve the way industry leaders might have anticipated. We need each other.

Claudia Brauer (2014), blog post Will you help me flip it?


What do translators say?

…it is safe to say that the number of professional translators who are using MT has risen in the past couple of years. I suspect that you would find large differences of usage among different language groups, type of projects, experience levels, and even in the way it is used. Significantly, MT seems to have transitioned into one of many productivity tools that are useful to some but not all, and I think the stigma it used to have among translators is gradually going away.

ATA Chronicle (Feb. 2013) Conversation: Charlotte Brasler and Jost Zetzsche


What do you say?


Translation vs post-editing

Translation process of a translator (TAUS 2010)

Translation process of a post-editor (TAUS 2010)


Translators vs Post-editors Skills?

Translators • Fluent at least in two languages

o Excellent knowledge of SL o Perfect command of TL

• Excellent written and verbal communication skills

• Understand the cultural differences • Specialised domain knowledge • Word-processing skills • Knowledge of language

technologies • …

Post-editors • Excellent word-processing and editing skills

• Ability to work and make corrections directly on-screen

• Strong knowledge of general MT and its shortcomings

• Specific knowledge of the approach of the MT systems in use, especially their weaknesses and the reason for producing the encountered errors

• Knowledge of the SL, TL and the audience

• Ability to make quick decisions and reduce errors as needed

• Ability to balance post-editing speed and effort with respect to the required quality level;

• Knowledge of CAT environments and of the differences between post-editing MT output as opposed to editing the past human translations present in fuzzy matches.

Source: Doherty & Gaspari 2014, GALA blog


Source: http://www.multilingual-tech.com/2014/05/08/language-translations-can-be-funny-right/

Quality


Quality

Quality is relative…The evaluation of the translation quality- whether human or computerized – has to take into account the following intra-linguistic and inter-linguistic factors: morphology, syntax, content, terminology, style, conformity. (Bruderer 1978) Typically, there are many “perfect” translations of a given source sentence. These translations may vary in word choice or in word order even when they use the same words. And yet humans can clearly distinguish a good translations from a bad one. (Papineni et al., 2002)


Quality

• EN-15038:

…simple to control!


Lucija Biel, 2006

Quality

Source: Asia Online

…more difficult to control!


Quality: MT Evaluation methods

Human • No standardised method • Word Error Rate (WER) • Subjective Sentence Error

Rate (SSER) • Metrics:

o Fluency and Adequacy (TAUS DQF, COSTA MT-Evaluation, Appraise etc.)

• Post-editing effort

Automatic • Metrics:

o BLEU o NIST o WER o METEOR o LEPOR


Productivity

Post-editing MT output can significantly increase the productivity of human translators!

Guerberof 2009; Plitt and Masselot 2010; Federico et al. 2012; Green et al. 2013; Läubli et

al. 2013


Productivity

TEP • Translation: 250 words per

hour • Editing: 1000 words per

hour • Proofing: 4000 words per

hour

MT Post-editing • 350-800 words per hour • …


Productivity

Source: Asia Online, Custom MT engines with Language StudioTM

With Customised MT


Productivity

Autodesk 2010


PE Compensation

• Various experiments, no standard • Hourly rates or price per word?

o 65%-85% of new word rate (Lionbridge) o 40%-85% of new word rate (CSA report) o $45/hr (Asia Online)


PE Compensation

• It depends on certain factors: o Quality of the MT output o Translator’s productivity (speed) o Language pair o Content type


PE Compensation

Good MT output

Less PE effort

More words per hour

More earnings?


MT becomes even better when integrated in the translators TM environment! (O’Brien & Moorkens 2014)

A number of CAT tools connect to general and customized MT engines via API:

TM/MT integration


SDL Trados Studio 2014

memoQ 2014


OmegaT

In the OmegaT user manual it is stated the following: If you do not wish to send your source segments to Google to get translated, untick the Google Translate menu entry in Options. Note that nothing but your source segment is sent to the MT service. The online version of Google Translate allows the user to correct the suggestion and send the corrected segment in. This feature, however, is not implemented in OmegaT.


Edit distance memoQ 2014

Integrated PE analysis features


Post-edit Modifications Analysis in Trados Studio 2014


MateCat


MemSource

Post-editing Analysis report


Post-editing analysis & Compensation


Signs and Simptoms of Translation- Blog by Emma Goldsmith

Small-scale experiment at University of Leuven

• Goal:

o To test the TM/MT integration o To investigate the impact of MT on productivity and the final translation quality

• Participants and language combinations:

o 21 MA/Postgraduate students with EN > NL (Productivity test) o Translation Technology class

• Source text: o 5 short English texts, technical documentation, same level of difficulty

• Tools: o Trados Studio 2014

• Resources: o TM with output from Google


What did we measure?

• Productivity (time): human translation vs post-editing MT • Quality: human-translations vs post-edited translations • The usefulness of MT

• Translators’ expectations


How?

1. Translation from scratch, no TM/MT provided. 2. Translation with TM. 3. Translation with TM/MT. 4. Track the translation time of the whole project. 5. Answer few questions about the usefulness of MT


Language combination used for the assignment:


Did you study English in the previous years?


Source text

EN NL


Results

Productivity Total average time per text for EN-NL


Nr of words per text varied between 140-200

Results

Quality Evaluation: Rate the translations on a scale from 1-4 in terms of:

o Clarity: How easily can you understand the translation?

o Accuracy: How much of the meaning in the source text is also expressed in the translation?

o Style: Is the language used in the translation

appropriate for this content type?

Fiederer & O’Brien (2009)


CLARITY: How easily can you understand the translation?


In general

CLARITY: How easily can you understand the translation?


Human Translation vs. Post-editing

ACCURACY: How much of the meaning in the source text is also expressed in the

translation?


In general

ACCURACY: How much of the meaning in the source text is also expressed in the translation?



STYLE: Is the language used in the translation appropriate for a software product

user manual?


In general

STYLE: Is the language used in the translation appropriate for a software product user manual?



Usefulness of MT General

Final quality

Terms

Collocations


Did you find the machine translation output useful?


Nr of students

Terminology? Collocations?

0

5

10

15

20

25

Useful forterminology

Useful forcollocations

Very uselessUselessUsefulVery useful


Nr o

f stu

dent

s

Do you think the machine translation output had an impact on the quality of your final translation?


Is MT a disruptive technology? … or is it simply ….


What did students think of MT?


Limitations

• Students had to record the time themselves • Not all of them had a good knowledge of English • Knowledge of the technical domain • Starter-knowledge of Trados Studio 2014


The debate that puts machine translation and professional translation on the opposite sides is misleading. The question isn’t about choosing ‘either- or’ because those two solve totally different problems. Professional translation is the best choice when your goal is to get someone to understand your message. MT is a risky choice in those kinds of situations but is a more suitable tool when you want to understand someone else’s message.

Multilizer, Translation Blog


Thank you!

[email protected] [email protected]

References • Thicke, Lori. The industrial process for quality machine translation. Retrieved from

http://www.jostrans.org/issue19/art_thicke.php on 5 November 2014. • Guerberof Arenas, Ana (2009). Productivity and Quality in the Post–editing of Outputs

from Translation Memories and Machine Translation. Localisation Focus 7(1): 11-21http://isg.urv.es/library/papers 2009_Ana_Guerberof_Vol_7-11.pdf

• Guerberof Arenas, Ana (2013) What do professional translators think about post-editing?. The Journal of Specialised Translation 19: 75-95. http://www.jostrans.org/issue19/art_guerberof.pdf

• O’Brien, Sharon (2002) Teaching Post-editing: A Proposal for Course Content. Proceedings of the 6th EAMT Workshop on “Teaching Machine Translation”. EAMT/BCS, UMIST, Manchester, UK. 99-106. http://mt-archive.info/EAMT-2002-OBrien.pdf

• Language Technology at Autodesk. Translation and Post-editing Productivity (2011).http://langtech.autodesk.com/productivity.html

• Doherty Stephen & Gaspari Federico , QT LaunchPad Project (2014) Effective Post-Editing in Human and Machine Translation Workflows. http://www.gala-global.org/blog/2014/effective-post-editing-in-human-and-machine-translation-workflows/


http://www.jostrans.org/issue19/art_thicke.php

http://www.jostrans.org/issue19/art_guerberof.pdf

http://mt-archive.info/EAMT-2002-OBrien.pdf

http://mt-archive.info/EAMT-2002-OBrien.pdf

References • O'Brien, Sharon and Moorkens, Joss (2014). Towards intelligent post-editing interfaces.

In: FIT XXth World Congress 2014, 4-6 Aug 2014, Berlin, Germany. Retrieved from http://doras.dcu.ie/20136/ on 5 November 2014.

• MateCat: http://www.matecat.com • SDL: http://www.sdl.com/products/sdl-trados-studio/ • Kilgray Technologies: http://kilgray.com • MemSource: http://www.memsource.com • eMpTy Pages Blog: Improving the MT Technology through Translator Dialogue. http://kv-

emptypages.blogspot.nl/2014/05/improving-mt-technology-to-translator.html • Patenttranslator blog: There Are Many Ways to Commit Suicide, But Post-Processing of

Machine Translations Would Be a Horrible Way To Go (2014) :http://patenttranslator.wordpress.com/2014/10/22/there-are-many-ways-to-commit-suicide-but-post-processing-of-machine-translations-would-be-a-horrible-way-to-go

• Translationtribulations.com


http://doras.dcu.ie/20136/

http://www.matecat.com

http://www.sdl.com/products/sdl-trados-studio/

http://kilgray.com

http://www.memsource.com

http://kv-emptypages.blogspot.nl/2014/05/improving-mt-technology-to-translator.html

http://kv-emptypages.blogspot.nl/2014/05/improving-mt-technology-to-translator.html

References • Plitt.M, Masselot F. A productivity test of statistical machine translation post-editing in a typical

localisation context. Retrieved on October 21 from http://ufal.mff.cuni.cz/pbml/93/art-plitt-masselot.pdf • De Palma, Don. Post-editing in practice: http://www.tcworld.info/e-magazine/translation-and-

localization/article/post-editing-in-practice/ • Asia Online, Fair-compensation of Post-editing MT:

http://www.asiaonline.net/EN/Resources/Articles/FairCompensationForMTPostEditing.aspx. Retrieved on 27 October 2014

• Smets, Martine. Microsoft and the need for agile translations: http://cdt.europa.eu/EN/news/Pages/default.aspx?CDTID=266

• Deliverables D5.1 Baseline Definition Results on KPI 1: Post-editing Effort: http://www.matecat.com/deliverables/d5-1-baseline-definition/results-on-kpi-1-post-editing-effort/

• Apertium: http://www.apertium.org/index.eng.html#translation • ATA Chronicle (February 2013):http://www.atanet.org/chronicle/feature_article_february2013.php • Example of a Translation Integrated Workflow:

http://www.asiaonline.net/EN/MachineTranslation/default.aspx?QID=21


http://ufal.mff.cuni.cz/pbml/93/art-plitt-masselot.pdf

http://www.asiaonline.net/EN/Resources/Articles/FairCompensationForMTPostEditing.aspx

http://cdt.europa.eu/EN/news/Pages/default.aspx?CDTID=266

http://www.matecat.com/deliverables/issue/d5-1-baseline-definition/

http://www.matecat.com/deliverables/d5-1-baseline-definition/results-on-kpi-1-post-editing-effort/

http://www.apertium.org/index.eng.html%23translation

machine translation as a tool - ku leuven · 2015-01-25 · machine translation as a tool prof. dr....

Documents