asia online - webinar - a 6 step plan to create an …...title microsoft powerpoint - asia online -...

Copyright © 2014, Asia Online Pte LtdCopyright © 2014, Asia Online Pte Ltd

A 6 Step Plan to Create an

Optimal MT Post-Editing Strategy

Dion Wiggins

Chief Executive Officer

[email protected]

Kirti Vashee

VP Enterprise Translation Sales

[email protected]

Copyright © 2014, Asia Online Pte Ltd

6 separate, but related, sub-strategies are required that address the key areas relating to post-editing MT.

1. Optimal Workflow Strategy

2. Quality and Measurement Strategy

3. Human Resources Training Strategy

4. Custom Engine Improvement and Feedback Strategy

5. Post-Editor Compensation Strategy

6. Post-Editor Recruitment and Retention Strategy


• Translation Workflow– It is important to understand that TM and MT are

different, but complimentary to each other

– Human workflow is different also

– TMS tools are the best way

to automate workflow, but

have some boundaries

– How do you determine

if MT should be used?


– What is the boundary between TM and MT?

– How transparent should MT be in the workflow?

– Should different editing tools be used?

– Can human only workflow be

applied to MT workflow?

– At what stage should MT be

applied to the workflow?• Pre-Translate Document

• Segment Level


• Who is responsible for managing / executing the

MT task?

– Translators / Editors• Should be focused on translating / editing

• Should not technology and auxiliary tasks that are best defined at a

project level by the client or project manager

– Project Managers• Which MT engine to use

• What terminology and glossary to leverage

• What translation memories to leverage

• Incorporation of post-editor feedback

• Improvement of custom MT engine

• Rules to manage quality, formatting and reduce effort


Translation Memory (TM) Match

Machine Translate

Human TranslateCorrect TM Match

Matched Sentences Non-Matched Sentences

Edit

Proof

Translate1. TM match. A high fuzzy score

(usually 85%-90%) is used as

raw MT will often be better

than a lower scoring fuzzy

match.

2. Send unmatched segments to

MT.

3. MT output can be corrected by

a translator or in the case of

high quality MT, often bypass

this step and go directly to an

editor.

1. Different projects have

different budgets and quality

requirements.

2. The high quality segments for a

given project should be sent to

Asia Online to improve the

quality of future translations.

This is usually just prior to

publication.

Translate via your preferred

TMS

Edit and Proof using your

preferred editing tools.

Pre-Translate

Correct MT Output


ProofEditTranslate

• Tried and proven method / approach

• Well understood by both LSP industry and end

clients

• This approach needs adaptation for use with MT

Human Translation ProofEdit


ProofEditTranslate

• Same person is responsible for both post editing and re-translating non-editable segments

• There is no edit stage after the re-translate, so a key quality check is missing

• Simply replacing the Human Translation with Machine Translation is insufficient

Machine Translation ProofPost Edit

or

Re-translate Bad Segments


ProofEditTranslate

Machine Translation

Quality Check

Is MT Segment

Editable?

ProofHuman Translation

Edit

Post Edit

No Yes

Note: Human Translator and Post Editor can be the same

person, but Human Translator and Editor must not

be the same person.

Copyright © 2014, Asia Online Pte LtdAsia Online Automated

Client Role

Incremental Improvement

Select Language Pair

and Top Level Domain

Create Sub-Domain

Engine Project

Upload Data

Linguist Review

Create Training /

Quality Improvement Plan

Pre-Customization

Consultation with Client

Clean Data

Data Manufacturing

Client Data Validation

Training

Evaluate Quality

Fine Tune with Rules

Release to Production

Release to

Diagnostic Review

Machine Translate /

Post Edit

Create New

Project Version

Initial Customization

More Client

Data Needed?

Acceptable

Quality?

Evaluation & Release

Yes

No

Yes No

Asia Online Specialist

Key


Original

Source File

Machine

Translate

Raw Machine

Translations

Human

Post Editing

Post Edited

Translations

Send Raw MT and

Post Edited Translations

back to Asia Online

2

1

2

1

Data Analysis and

Manufacturing

Incremental

Quality

Improvement Supported File Types


• Why are you Measuring?– Value to business– Quality – Individuals– Productivity

• Define Objectives and Benchmarks– Every project is different

• Budgets• Quality targets• Purpose

– Frequency

• Define methodologies– BLEU and other automated metrics– Human metrics– Hybrid human / automated metrics

• Define Quality– Very simple: Whatever the customer says it is!!!– Understand what the customer thinks is quality


• Start with understanding human levels of

productivity and quality

• Progress to measuring between each notable

body of work – i.e. each project

• Focus on productivity, not the

number of errors


• Don’t get hung up on the details– Measuring MT the way you measure a human is wrong– MT and humans make different kinds of errors

• Productivity – How many words per day per human resource

• Margin– What is the increase/loss in profit margin

• End Customer Priorities– Consistency – Writing style and terminology

• Expert Customized MT + Human delivers higher quality than a human only approach

– Speed– Versatility

• New Business Generation– Will existing customers to spend more with you?– Will new customers select you over other vendors?– What value can you offer that others do not?


Automated Metrics

(Good indicators, but not absolute)

• BLEU (Bilingual Evaluation Understudy)

• NIST

• F-Measure (F1 Score or F-Score)

• METEOR (Metric for Evaluation of Translation with Explicit ORdering)

Manual Quality Metrics

(Most not designed for MT, more for HT)

• Edit Distance (Does not take into account complexity of edit)

• SAE-J2450 (Industry specific)


• MT errors are easy to see and easy to fix (i.e.

simple grammar).

• MT provides more accurate and consistent

terminology than human translators, especially

when more than 1 human works on a project.

• Human errors may be fewer, but harder to see

and harder to fix.Time

Margin


Machine Translation Post Edit

Proof

++ =

Human Translation

+

Total Time

Total Cost

More AccurateMinimum

Human Translation Edit Proof

+ + = Total Time

Total Cost

Edit

Compare


• Project Managers– Understand which MT engine is used – Understand the capabilities and the

limitations of the engine– Understand the means to guide and

improve the engine– The correct bar to set for TM vs. MT use– Collect and apply feedback to improve MT– Assign appropriate team members for

improvement roles• Train them!!!!• Talk to your vendors and seek third party training

– Manage engine updates for MT systems that can be managed– Post-Editor selection

• Choose people who can understand and respond to your specific feedback on MT output. There are unknown factors in many MT situations so mutual trust is also important.


– Manage post editors and set expectations

for the amount of editing

– Train post-editors on the specifics of each

engine

– Provide examples of error types to look

out for and report

– Provide examples of when to edit and

when to not edit

– Provide examples on when to reject a

segment and what to do when you do reject it


� Learn How to Give Corrective Feedback to MT� Better MT Output = Higher Productivity = Higher Pay� Learn to communicate with MT developers

� Understand Error Patterns� Provide corrective strategies

� Learn to Use Power Tools To Improve Productivity� Some MT solutions have run time tools to improve output � Add Glossaries, REGEX Rules, DNT Lists

� Improve and Clean Before Editing� Fix Dumb Errors early and quickly to minimize repetition� Use Run Time Tools� Retrain the engine frequently so edits get easier

� Integrate Into Your Preferred Editing Tools� Can be as simple as importing TMX files� Many popular TMS tools are already integrated with MT


• Learn how to work with and what to expect from each engine – Different vendors have different capabilities

• Understand your own throughput with and without MT

• Get examples from the LSP of MT quality before accepting work

• Be wary of “light post-editing” scenarios if it is not clearly spelled out with examples or guidelines

• How and when do you provide feedback?– Can you provide feedback? – Will it be listened to and acted on?

• What do you edit and what do you leave alone?

• What do you reject?– How long do you spend on a segment to determine if it should be rejected?

• What happens to a rejected segment?– Full human re-translation– Send back for someone else to human re-translate

• Develop rapid error detection and correction skills


• Most Do-It-Yourself (DIY) initiatives are done by people who don’t know how to it = Upload & Pray

• Choose people who can understand and respond to your specific feedback on MT output. There are unknown factors in many MT situations so mutual trust is also important.

• Ask questions about the specific MT system you are working with to understand possibilities

• Develop rapid error detection and correction skills

• Be wary of “light post-editing” scenarios if it is not clearly spelled out with examples or guidelines


Higher Productivity and Throughput = Lower Rates OK

Lower Productivity and Throughput = Higher Rate or Full

Rate Required

POST EDITORS: The job is ONLY worth doing if you make as much

or more than doing traditional translation work.

Low Word Rates combined with low quality MT

output

Lower Word Rates combined with high quality MT

output


• Project Planning – Engines must be managed like any other translation task

• Workflow and post-editing cycles

• MT Approach – How much data is required?– Clean Data SMT - <1% of total training data

– Dirty Data SMT – 20% of total training data

• Quality Improvement Plan– If you don’t have one, the benefits are crippled!!


Human OnlyHuman Only

• Terminology Definition

• Non-Translatable Terms

• Historical Translations

• Style Guide

• Quality Requirements

• Translate

• Edit

• Proof

• Project Management

MT + HumanMT + Human




• Style Guide**


• Customize MT

• Translate

• Edit

• Proof


Almost identical information and data is required in order to customize a high

quality machine translation system

When preparing for a high quality human translation project, many core steps are

performed in order to ensure that the writing style and vocabulary are designed for a

customers target audience.

MT OnlyMT Only




• Style Guide**


• Customize MT

• Translate

• Edit

• Proof



Manage translation projects

while generating corrective

data for quality improvement.

Create a new custom engine

using foundation data and

your own language assets

2. MeasureMeasure the quality of the

engine for rating and future

improvement comparisons

Provide corrective feedback

removing potential for

translation errors.

Customize Measure2

Manage Improve3

1

4


Successful DIY MT

Generic MT

Google & Bing Translate

Most DIY MT Low

<--

Qu

ali

ty o

f T

ran

sla

tio

n O

utp

ut

�� H

igh

Spanish, Portuguese, French

Hungarian, Turkish, Korean, Hindi, Zulu, Swahili

Expert Customized MT


• Project planning and tracking

• Data gathering and acquisition

• Data manufacturing

• Data preparation and cleaning

• Gap analysis

• Terminology management

• Normalization

• Pre-translation analysis

• Post-translation analysis

• …


• Establish rates per project

• The best and most fair means is to set rate based

on actual effort– Hourly rate

– Words per hour set by a measurement from trusted

translator(s)

http://www.asiaonline.net/EN/Resources/Articles/FairCompensationForMTPostEditing.aspx


• If post-editing MT is 2 times faster than a traditional

translation approach there is justification for reducing

rates to 50% of the regular rate. However, since this is a

small sample it would be wiser to adjust this upwards to

a level that may accommodate more variance in the MT

output e.g. 60% or higher.

• The rate setting process should be as transparent as

possible to avoid translator resistance.

• Develop win-win scenarios where the

LSP reduces costs, has faster

turnaround and the translator/

post-editor earns a fair or higher wage.


• Post-Editor Recruitment – Communication of business opportunities

• Do not force translators to be post-editors– Some great post-editors are average translators

– Some great translators are average post-editors

• Provide Training

• Explain how rates are set

• Show examples of MT output

• Exclude people who are hostile

• PAY THEM PROPERLY!!!!


� Understand the Specific Quality of MT� Be aware of both Automatic & Human Assessment Metrics� Look at samples of representative output (100 segments+)

� Understand Your Specific Productivity� Words / Hour

� Establish a Threshold Limit for Compensation� Based on a productivity calculation on actual output from MT

� Understand if MT Output can be Improved� Fix Dumb Errors early and quickly to minimize repetition� Cycle quickly through initial MT development cycles� Retrain the engine frequently so edits get easier

� Determine MT Competence of Agency� Most DIY efforts are unlikely to be successful� Err on the side of overcompensation than under

� Ask for Post Editing Examples� Avoid over correction to maximize productivity� Understand Workflow


• PEMT will become more common so learn to determine when it makes sense for you

• There are opportunities similar to early days of TM for those who learn to build PEMT leverage

• Find Agencies that are

competent and that you trust

• Develop new skills that allow

you to build PEMT leverage


Questions?

Dion Wiggins

Chief Executive Officer

[email protected]

Kirti Vashee

VP Enterprise Translation Sales

[email protected]

asia online - webinar - a 6 step plan to create an …...title microsoft powerpoint - asia online -...

Documents