translation quality measurement2

Translation Quality Measurement

Biographical Notes on the Authors

Riccardo SchiaffinoRiccardo Schiaffino worked as translator, translation manager and special software translation project lead for a major software company. As a translation manager, Riccardo worked on the improvement of translation quality and on translation quality metrics and tools. He holds an MA degree in Translation, and has been working in translations for over 18 years, first in Italy and then in the U.S. Riccardo is ATA accredited. Contact: [email protected].

Franco Pietro ZearoFranco Pietro Zearo is a project manager with Lionbridge Technologies in Boulder, Colorado. He holds a degree in translation from the Advanced School of Modern Languages for Translators and Interpreters at the University of Trieste, Italy, and earned an MBA from the University of Phoenix. Before joining Lionbridge in 1996, he worked as a freelance technical translator in Italian, English, and Russian. At Lionbridge, he has held positions in translation, localization analysis, presales, and cultural and globalization consulting. He has been responsible for translation quality on numerous projects for many Fortune 500 clients. In his previous role as senior technical translator, he helped define best practices for the translation department. Contact: [email protected].

mailto:[email protected]

mailto:[email protected]

Overview

• Technical translation and quality• Translation quality initiatives• Quality Control vs. Quality Assurance

• Our proposal for quality assurance• Checklists• Sampling techniques

• Conclusions• Importance of cost/benefit factors

Overview

Measuring QualityTranslation Quality AssessmentQuality Assurance FormsError CategoriesSamplingTranslation Quality IndexQuestions and Answers

Overview

• Why Is Quality Measurement Important?

• How to Set Up a Quality Measurement System

• Demo of a Translation Quality Measurement Tool Prototype

• Practical Recommendations

• Questions & Answers

Our Definition of Quality

Functional approach to quality

Different views of translation lead to:

Different concepts of quality

Different assessments

Quality is defined as meeting the needs and expectations of the customer or user.

Our Definition of Quality

Functional approach to quality

Quality is defined as

consistently meeting

the needs and expectations

of the customer or user

Correct Translation

A correct translation is a translation with no errors or where total error points result in a Translation Quality Index above the desired threshold

Conformance to specificationsCustomer’s vs. One’s own

Fitness for useHow well the translation performs its intended purpose

Value ( = quality & price) How well the translation performs its intended purpose at a price customers are willing to pay

SupportE.g.: Printing, testing

Psychological impressions E.g.: In-country translators; certification

Customer-driven Considerations

Customer-driven Considerations

Price Time

Quality

Importance of Quality

Cost Time

Quality

Quality as a Competitive Weapon Good Quality

Higher Profits

Good quality of translation (product) and service (process) can pay off in higher profits

Improving on quality can reduce costs and speed up time-to-market

Why is Quality Measurement Important?

You can’t manage what you can’t measure

It is difficult to improve something if you cannot measure it.

Such measurement should be repeatable and objective.

Different persons should arrive at similar assessment for the same piece of translation.


It is difficult to improve something if you cannot measure it.

Such measurement should be repeatable and objective.

Different evaluators should arrive at similar assessment for the same piece of translation.


It is difficult to improve something if you cannot measure it

Metrics provide: A way to objectively quantify a process A means to reduce the cost of poor quality A means to increase customer satisfaction An opportunity for benchmarking Competitive advantages

“You cannot measure quality”

This is not true:There are certain components of translation

quality that will always remain subjective.

However, There are other elements that can be

objectively measured.By concentrating of these, we believe we

can achieve a satisfactory measurement of translation quality.

Who Benefits from Reliable Translation Quality

Measurement? Professional Translators

Translation Companies andIn-House Translation Departments

Translation Customers and Users

Why Do We Make Errors?

The reasons behind the errors are separate from the measurement of the errors: Studying why errors happen is important, but it pertains more to quality control and improvement than to quality assurance

E.g., capitalization errors due to the "Autocorrect" (mis)feature of MS Word (e.g., HBsAg "corrected" to HbsAg)

QC vs QA

Quality Control (QC)Quality verification over the whole text.

Example: editing.

Quality Assurance (QA)Sampling techniques, control of quality over a

(statistically significant) sample of the whole text.

Example: quality measurement.

QC vs QA

Quality Control (QC)Quality verification over the whole text.

Example: Editing.

Quality Assurance (QA)Sampling techniques, control of quality over a

(statistically significant) sample of the whole text.

Appropriate use: Quality measurement.

Translation Quality Factors

Usability Accuracy

Legal Marketing

Inspection Points

Key Principle: Reject “defective material” at its lowest value

Proof

Edit

Translation

SLContent Development(GIGO)

$ V

alue

of

Serv

ice

Stages of Production

Cost/Benefit Analysis

0

10

20

30

40

50

60

70

80

90

1 2 3 4 5

Quality Level

Time o

r Mon

ey

Investment

Value Added

Quality measurements are a tool to determine the optimal level ofquality.

They could help us identify a cut-off point.

Ideas from other disciplines

Software project management techniquesW. Edwards Deming and other quality

assurance experts

When we study translation quality, we can focus on different things:

The translation process(the “process”)

Aim:

Inputs Outcomes

Boundary Boundary

Feedback

Conversion Process

Process Process Process Process Process

The translator

The translated text(the “product”)

Product & Process Assessment

Translation quality assessment must apply to both:

The translated text(the “product”)

The translation process(the “process”)

Aim:

Inputs Outcomes

Boundary Boundary

Feedback

Conversion Process


Translation Quality Initiatives

The translation process

The translator

The translated text

SAE J2450LISA QA

DIN 2345ISO 900xUNI EN 10754 EUATCASTM

ATA and other translators’ certification initiatives

ISO 9002EUATC Quality StandardDIN 2345ASTM Standard for Language Translation

SAE J2450LISA QA Model

Academic translation theories and studiesPrivate sector methodologies

Translation Quality Initiatives

Quality Measurement: Our Proposal

What Can Other Disciplines Teach Us? Use checklists to collect the data

Identify types of errors, issues or problems Determine relative importance of issues (may be

different for different languages; e.g., spelling errors in English, French or Italian)

Use sampling techniques to assess your quality level Determine percent thresholds for various levels

of quality Determine whether you have achieved your

target quality or not

Criteria for Successful Quality Measurements

Translation quality measurements should be:

Repeatable (two assessments of the same sample yield similar results)

Reproducible (different evaluators should arrive at a similar assessment for the same piece of translation

Objective (void of subjective bias)

Classification of Errors

Error?

Meaning orform?

Mistranslationor Addition/Omissions?

Grammar orStyle?

Usage or "StyleProper"?

Misinterpretation of SLMiswriting of TL

etc.

AdditionOmission

etc (?)

SyntaxMorphology

SpellingPunctuation

etc.

Registeretc.

Yes

Meaning Form

GrammarStyle

Style

Mistranslation Addition/Omission

Usage

InconsistencyNon-adherence to

guidelinesetc.

May be detectedby other users

(includingtranslators)

Best detected bytranslators

End No

StartAn example of of an errorclassification process

(not complete)

Measurement through Circumstantial Evidence

Errors are circumstantial evidence of quality

We believe that precise error measurement provides sufficient indication of good and bad translations

A good translation is a translation with very few errors or none at all

Definition of Errors

Deal with errors only when they violate agreed upon protocols of engagement whether implicit or explicit

Examples of explicit and implicit criteria: Non-compliance errors (e.g. not following

instructions) Violations of generally accepted language

conventions

Summary: Error Categorization

Select a (small) set of categories

CTQ: Critical-To-Quality categories

Provide clear definitions

Set tolerance limits

Min / Max # of errors per X words

Assign a weight

Critical, Major, Minor

Summary: Error Categorization

Select a (small) set of categories

CTQ: Critical-To-Quality categories

Provide clear definitions

Assign a weight

Critical, Major, Minor

Real Life Examples

Development of translation quality measurement at J.D. Edwards

Use of sampling techniques for quality assurance at Lionbridge

The J.D. Edwards’ QA FormLanguage Customization

Weighting the major categories

Accuracy 50%Style 15%Grammar 30%Formatting 5%

100%

1 - Give appropriate weight to the four following categories (total must add up to 100%)

Language Setup

Total

Categories


Weighting the items within the major categories

Incorrect meaning 40%Non-standard terminolgy 20%Inconsistent terminolgy 20%Untranslated SL 20%

100%

Wrong register 40%Inappropriate anglicisms 30%Inappropriate use of passive/active voice 30%

100%

Spelling errors 20%Typos 15%Grammar errors 35%Syntax errors 25%Punctuation errors 5%

100%

Layout errors 50%Font errors 40%Double spaces 10%

100%

2 - Within the Accuracy category, give appropriate weight to the four following items (total must add up to 100%)

3 - Within the Style category, give appropriate weight to the three following items (total must add up to 100%)

4 - Within the Grammar category, give appropriate weight to the five following items (total must add up to 100%)

Total

Total

Accuracy

Style

Total

Formatting

Total

5 - Within the Formatting category, give appropriate weight to the five following items (total must add up to 100%)

Grammar


Weighting the items within the major categories (detail)


100%Total

Accuracy

How We Worked to Develop Our Spreadsheet Determine type of errors, issues or problems

Determine relative importance of issues (may be different for different languages; e.g., spelling errors in English, French or Italian)

Determine which are the responsibility of translation

Determine tolerance limits for various levels of quality

Translation Quality Measurement Tool

The Translation Quality Measurement tool helps to measure process quality

It is NOT an editing tool, but it serves to measure whether a process is effective

Use of the Tool

Use the tool to measure the effectiveness of quality control process

Analyze the results obtained through the tool (control charts)

If the process is NOT in statistical control Discover special causes and deal with them

appropriately Remove them if they are negative Incorporate them in process if they are positive

Improve the process when it is in statistical control

A TQI Tool Prototype

ATA Implementation

SAE Implementation (Modified)

TQI LogError Category EP Remarks Bookmark Path File Grader Date

Is Formal 2irregular capitalization; should be is, not Is

x2_is

C:\Documents and Settings\RS1643403\Desktop\Quality Measurement

CoffeMakerTest.doc

RS1643403

11/1/2003

aluminium Formal 1British spelling; American English should be aluminum, not aluminium

x3_aluminium


CoffeMakerTest.doc

RS1643403

11/1/2003

food Meaning 2The container is not made to cook food, it is made to brew a beverage.

x5_food


CoffeMakerTest.doc

RS1643403

11/1/2003

right for the gas cooker, the electric plate and the pyroceram

Formal 1A better phase might be: "acceptable for use on gas and electric stoves."

x6_right_for


CoffeMakerTest.doc

RS1643403

11/1/2003

pyroceram Meaning 4The word "pyroceram" is unknown to most English speakers.

x7_pyroceram


CoffeMakerTest.doc

RS1643403

11/1/2003

wash Meaning 2

In English, the word "wash" typically means water and soap. The instructions specify only using water, so a better word choice would be "rinse."

x8_wash


CoffeMakerTest.doc

RS1643403

11/1/2003

trow Meaning 2The word "trow" is a misspelling of "throw."

x9_trow


CoffeMakerTest.doc

RS1643403

11/1/2003

total 14

N. of words 42

TQI 67%

Use of Checklists

There are several quality assessment methodologies that rely on the use of checklists – among these the LISA methodology.

Language: Reviewer: Date: Result: Pass Comments:

Client NameProject NameProject NumberProject Manager

Critical max. error points + 1Number of words 0 Major 5 pointsMax error points allowed 0 Minor 1 point

Error Category Minor Major Critical total max. allowed0 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 0

Total 0 0More elaborate descriptions of the error criteria can be found in the LISA QA model version 1.0 Reference Manual.

Quality Assurance Form

Consistency

MistranslationAccuracyTerminologyLanguageStyleCountry

Use of Checklists There are several quality assessment methodologies

that rely on the use of checklists – among these the LISA methodology.

We would like, however, to advocate the use not of “universal” checklists, but of checklist specifically tailored to each language.

Checklists for evaluating translation companies Checklists and tests for evaluating translators Checklists for evaluating translations

Limitations of universal checklists Language specific checklists (example, different

weight of spelling correctness for different languages)

Development of Translation Quality Measurement at

J.D. Edwards From the concept of checklists to a spreadsheet of

measurements

Checklists are appropriate to control whether a certain action has been performed or not (e.g., spell check done or not – as opposed to a measurement of how many spelling mistakes were found)

Based on LISA model (www.lisa.org)

Flexibility (different settings for different languages)

http://www.lisa.org/



Use of Quality Assurance Forms

The LISA Quality Assurance Form







Consistency


Purposes of sampling according to LISA

To determine whether something has been done or not.

To accept / reject the batch of product at hand.

To determine if the process that produced the product at hand was within acceptable limits.

Guidelines for Sampling

Select a sampleSelection criteria (e.g. random, systematic)Size considerations Cost considerations

Evaluate the sample

Repeatable, reproducible, objective

Investigate the outcome / causes

Correct / Improve

Statistical Methods

Defect Counts

Statistics on Effort Per Defect

Defect Density Prediction

Defect Pooling

Defect Seeding

Defect Counts

Useful to obtain a quantitative measurement of how much QC work to do.

Ratio of new defects to defects solved.

Statistics on Effort Per Defect

In order to estimate the scope of the defect correction work, it is necessary to have good data on the time necessary to fix the various types of defects

Defect Density Prediction

One way to judge whether the QC work on a translation project is complete is to measure its defect density (the number of defects per page, per 1,000 words or per screen).

Defect Pooling

Defect pooling is a simple defect prediction technique that separates the defects found in a translation sample into two pools.

Depending on the number of defects found in either of the two pools (but not in both) it is then possible to estimate the defects that have not been found in the sample.

This number can then be used to estimate the number of defects in the entire project.

Defect Seeding

Defect seeding is a statistical technique in which a sample of a population is extracted and used to estimate the total population.

The technique works by deliberately inserting (“seeding”) defects in a complete translation that will be QCed.

The ratio of the seeded defects found compared to the total number of defects seeded provides a rough estimate of the total number of translation defects yet to be found.

A common problem with this type of technique is forgetting to remove the errors deliberately inserted.

Calibration and Error Seeding

One of the things one can do to calibrate a translation quality measurement tool (or process) is to use error seeding: Not only to be able to estimate what percentage of errors is not discovered, but also in order to estimate how much variance there is in assessing the errors that do get discovered.

Suggested process: calibration of a (generic) translation quality measurement tool

Have the sample translations (a suitable number of them) scored "by hand" by expert translators, so as to obtain a suitable range of evaluated samples, from very good to very bad.

Importance of tightly defining the pool of reviewers Importance of instructions for reviewers Have other expert translators score the same tests, but using the

tool On the basis of the results of the previous two steps, adjust the

weights, types of errors, etc. in the tool until you are satisfied it is going to help in assessing translation quality - that is, until you are confident that trained evaluators are going to obtain with the tool consistent and reliable scores

In doing this remember to remove from the kind of errors that can be assessed those that are controversial, i.e., those that lead to differences of opinion whether they are errors or not

Finally adjust the tool so that it produces the range of error scores that is useful for your organization (e.g., if you want "0" or 100% as your perfect score)

Translation Quality Index (TQI)

The TQI is a number—obtained by the rigorous application of a QA process—that indicates the quality of a given translated text

The concept of a “Translation Quality Index” Translation Quality Index (TQI)

A number—obtained by the rigorous application of a QA form—that is indicative of the quality of a given translation

Delusions of Accuracy

“Averages can be calculated to nineteen places of decimal with astonishing ease.

When the job is done, it looks very accurate.

It is an easy and fatal step to think that the accuracy of our arithmetic is equivalent to the accuracy of our knowledge about the problem in hand.”

M.J. Moroney, Facts from Figures

Index / Indices

Depending on one’s purpose, there may be more than a single TQI.

E.g., a TQI may be developed for external purposes (to standardize the work obtained from outsourcing).

Another TQI may be primarily for internal purposes (to measure the quality of a given special process).

An Example of a “Translation Quality Index” (1)

LISA QA Model ver. 1.0 (1995)

3,000 words (12 pages @ 250 words)30 error points30 error pts / 3,000 words = 1.0%10,000 error pts out of 1 million wordsDPMO = 99.0% = TQI


Microsoft Quality Standards for Printver. 1.0 (1998)

10,000 words (40 pages @ 250 words) 0 major errors 15 minor errors15 errors / 10,000 words = 0.15%1,500 errors out of 1 million wordsDPMO = 99.85% = TQI


2,000 words (8 pages @ 250 words)1 critical error2 major errors 3 minor errors6 errors / 2,000 words = 0.3%3,000 errors out of 1 million wordsDPMO = 99.7% = TQI

Let’s Calculate Two TQIs

Implicit TQI = 93.2%Implicit TQI = 99.0%

250 words (estimate)

17 error points

17 error pts / 250 words = 0.068

3,000 words (12 pages @ 250 words)

30 error points

30 error pts / 3,000 words = 0.01

ATA Framework for Standard Error Marking

LISA QA Model ver. 1.0 (1995)

Control Charts

Concept of “statistical control”

98.50

98.75

99.00

99.25

TQI Control Chart

C2

C3

C4

C5

C6

C7

C8

C9

C10

C11

C12

C13

UCL LCL Mean Data Point

Process Flow Diagram Aim:

Inputs Outcomes

Boundary Boundary

Feedback

Conversion Process


Example of Process for Accepting or Rejecting a Translation Process

1) Determine and describe what your process actually is (NOT what you think it is or what the process should be)

2) Measure the quality you have now3) Determine if you have special cases, and if so,

eliminate them (what the special cases are can be seen through the use of control charts)

4) Once the process is in statistical control (i.e., any quality variance is not due to special cases)

5) Change the process to improve quality6) Measure the new level of quality to determine the

effectiveness of the changes to the process

Very Important

Improvements made to the overall process should result in improvements to the product (the translation)

Measurements of the product quality should indicate if there have been actual improvements to the process

Therefore, means to measure product quality must be in place

How to Apply Statistical Methods for Quality Improvement

1. Define error categories and tolerances

2. Create a QA form

3. Obtain a TQI index

4. Use the TQI index to improve the translation

process

How to Set Up a Quality Measurement System – Stage 1, Preparation

1. Collect examples of good and bad translations

2. Analyze the examples to separate controversial issues from agreed upon errors

3. Decide what to measure (error categorization)

4. Define what to measure in as many details as necessary (error definition)

How to Set Up a Quality Measurement System – Stage 2, Calibration

5. Assign a weight to various types of errors

6. Determine critical errors (if necessary)

7. Repeat 3, 4, 5, and 6 until the system works in an objective, repeatable, and reproducible way

Quality Assurance Forms and Tools







Consistency



100%

Wrong register 40%Inappropriate anglicisms 30%Inappropriate use of passive/active voice 30%

100%

Spelling errors 20%Typos 15%Grammar errors 35%Syntax errors 25%Punctuation errors 5%

100%

Layout errors 50%Font errors 40%Double spaces 10%

100%

2 - Within the Accuracy category, give appropriate weight to the four following items (total must add up to 100%)

3 - Within the Style category, give appropriate weight to the three following items (total must add up to 100%)

4 - Within the Grammar category, give appropriate weight to the five following items (total must add up to 100%)

Total

Total

Accuracy

Style

Total

Formatting

Total

5 - Within the Formatting category, give appropriate weight to the five following items (total must add up to 100%)

Grammar

Create a QA form (or a tool) to help graders give objective scores

How to Set Up a Quality Measurement System – Stage 3, Sampling Sampling

Selection criteria (e.g. random, systematic) Size considerations (the greater the sample,

the more accurate the results) Select confidence intervals, margins of error Cost considerations (find the point of

diminishing returns)

How to Set Up a Quality Measurement System – Stage 4, Measurement Measurement

Evaluation must be repeatable, reproducible, objective

Use of independent auditors

Calculation of a Translation Quality Index (TQI)

How to Set Up a Quality Measurement System – Stage 5, Statistical Analysis

Investigate the Outcome

• At this stage there shouldn’t be any special causes (use of control charts)

98.50

98.75

99.00

99.25

TQI Control Chart

C2

C3

C4

C5

C6

C7

C8

C9

C10

C11

C12

C13

UCL LCL Mean Data Point

How to Set Up a Quality Measurement System – Stage 6, Process Improvement Take corrective actions (process

improvement)

Compare the TQI values before and after a process change to check for actual process improvement

How to Set Up a Quality Measurement System – Summary

1. Preparation

2. Calibration

3. Sampling

4. Measurement

5. Statistical Analysis

6. Process Improvement

Practical Recommendations

Importance of

Glossaries (for terminology)

Style Guides (for syntax)

Translation Instructions (for special cases)

Protocols of Engagement (regulating the treatment of errors/defects and defining the acceptance/rejection criteria)

Translation Guide for Customers (including a detailed customer checklist to specify what is important and what is not)

Conclusions

Desirability of common standards (see GAAP - Generally Accepted Accounting Principles) It is not possible to directly compare different

quality initiatives A common standard would still permit

assigning different weights to different categories but in a much more transparent and comparable way

Translation Quality Scale

Quality Continuum

A EDCB

Translation Quality Scale

Quality Grades

90 607080 50100

TQI

Select Bibliography •Brue, G. : Six Sigma for Managers, New York, McGraw Hill , 2000

•Deming, W. Edwards: Out of the Crisis, Cambridge (Mass), MIT Press, 2000

•Eckersley, H.: “Systems for Evaluating Translation Quality”, in Multilingual Computing & Technology, #47 Volume 13 Issue 3, April/May 2002

•Grove, A.: High Output Management, 2nd ed., New York, Vintage Press, 1995

•Hönig, H. : “Positions, Power and Practice: Functionalist Approaches and Translation Quality Assessment”, in Schäffner, C. (ed.) Translation and Quality. Clevendon, Multilingual Matters, 1998

•Language International: “Engineering Language Quality – A word with quality-standards consultant John Gagliardi”, in Language International Vol. 12 No. 3, June 2000

•Lauscher S.: “Concepts of Translation Quality and Quality Assessment”, in Proceedings of the 39th Annual Conference of the American Translators Association, 1998

•Ling Koo, S., and Kinds, H.: “A Quality-Assurance Model for Large Projects”, in Sprung, R. (ed.) Translating into Success. Cutting-edge strategies for going multilingual in a global age. Amsterdam/Philadelphia, John Benjamins Publishing Company, 2000

•LISA: “Microsoft Quality Standards”, in Case Studies and Client Requirements, 1998

•McConnell, S.: Software Project Survival Guide, Redmond, Microsoft Press, 1998

•Moroney, M.J.: “Facts from Figures”, Harmondsworth, Penguins, 1951, 1956(3rd) ,

•Reiss, Katharina: Translation Criticism - The Potential & Limitations. Categories and Criteria for Translation Quality Assessment. Translated by Erroll F. Rhodes. St. Jerome Publishing 2000

•Schäffner, C. (ed.): Translation and Quality, Clevendon, Multilingual Matters, 1998

•Shewhart, W. : Statistical Method from the Viewpoint of Quality Control. 1939. New York: Dover Publications. Reprint, 1986. (Originally published: Washington, D.C.: Graduate School of the Department of Agriculture, 1939.)

•Spurr W., and Bonini C. : Statistical Analysis for Business Decisions, Homewood, IL: Richard D. Irwin, Inc., 1967

•Sturz, W.: “DIN 2345 Hits the Language Industry” in Language International Vol. 10 No. 5, May 1998

•Vogel, S.; Nießen, S.; Hermann, N.: “Automatic Extrapolation of Human Assessment of Translation Quality” http://www‑i6.informatik.rwth‑aachen.de/PostScript/InterneArbeiten/Paper_2000/Vogel_Evaluation_LREC_2000-corrected.ps.gz , 2000

•Woyde, R.: “Introduction to the SAE J2450 Translation Quality Metric”, in Language International Vol. 13 No. 2, April 2001

translation quality measurement2

Education

importance of quality

translation quality

control of quality

quality price

quality measurement

translation manager

correct translation

piece of translation