7/28/2019 TAUS Adequacy/Fluency Guidelines

1/6

TAUSBestPracticeGuidelinesQualityEvaluationusingAdequacyand/orFluency

Approaches

May2013

WhyareTAUSindustryguidelinesneeded?

Adequacyand/orFluencyevaluationsareregularlyemployedforassessingthe

qualityofmachinetranslation.However,theyarealsousefulforevaluationof

humanand/orcomputerassistedtranslationincertaincontexts.Thesemethodsare

lesscostlyandtimeconsumingtoimplementthananerrortypologyapproachand

canhelptofocusonassessingqualityattributesthataremostrelevantforspecific

contenttypesandpurposes.

Providingguidelinesforbestpracticeswillenabletheindustryto:

Adoptstandardapproaches,ensuringasharedlanguageandunderstandingbetweentranslationbuyers,suppliersandevaluators

Bettertrackandcompareperformanceacrossprojects,languagesandvendors

ReducethecostofqualityassuranceAdequacy/FluencyBestPracticeGuidelines

Establishcleardefinitions

Adequacy

Howmuchofthemeaningexpressedinthegold-standardtranslationorthesource

isalsoexpressedinthetargettranslation(LinguisticDataConsortium).

Fluency

Towhatextentthetranslationisonethatiswell-formedgrammatically,contains

correctspellings,adherestocommonuseofterms,titlesandnames,isintuitively

acceptableandcanbesensiblyinterpretedbyanativespeaker(LinguisticData

Consortium).

Clearlydefineevaluationcriteriaandratingscales,usingexamplestoensure

clarity.

Adequacy

Howmuchofthemeaningexpressedinthegold-standardtranslationorthesource

isalsoexpressedinthetargettranslation?Ona4-pointscaleratehowmuchofthemeaningisrepresentedinthetranslation:

7/28/2019 TAUS Adequacy/Fluency Guidelines

2/6

o Everythingo Mosto Littleo None

Fluency

Towhatextentisatargetsidetranslationgrammaticallywellinformed,without

spellingerrorsandexperiencedasusingnatural/intuitivelanguagebyanative

speaker?

Rateona4-pointscaletheextenttowhichthetranslationiswell-formed

grammatically,containscorrectspellings,adherestocommonuseofterms,titles

andnames,isintuitivelyacceptableandcanbesensiblyinterpretedbyanative

speaker:

o Flawlesso Goodo Dis-fluento Incomprehensible

Datasegmentsshouldbeatleastsentencelength.

Thechosenevaluationdatasetmustberepresentativeofentiredataset/content

ForMToutput,ataminimumtwohundredsegmentsmustbereviewed ForMToutput,theorderinwhichthedata/contentorderispresented

shouldberandomizedtoeliminatebias

HumanevaluatorteamsarebestsuitedtoprovidefeedbackforAdequacyand/or

Fluency.

Forperiodicreviewsthereshouldbeatleastfourevaluatorsperteam Thelevelofagreementbetweenevaluatorsandconfidenceintervalsmustbe

measured

EvaluatorsmustratethesamedataToensureconsistencyqualityhumanevaluatorsmustmeetminimum

requirements.

Ensureminimumrequirementsaremetbydevelopingtrainingmaterials,screeningtests,andguidelineswithexamples

Evaluatorsshouldbenativeornearnativespeakers,familiarwiththedomainofthedata

Evaluatorsshouldideallybeavailabletoperformoneevaluationpasswithoutinterruption

7/28/2019 TAUS Adequacy/Fluency Guidelines

3/6

Foradequacyevaluationevaluatorswillneedtobeabletounderstandthesource

language.Thisrequirementcanbeovercomeifevaluatorsaregivenagoldreference

segmentforeachtranslatedsegment.Notethatthequalityofthegoldreference

shouldbevalidatedinadvance.

Determinewhenyourevaluationsaresuitedforbenchmarking,bymakingsure

resultsarerepeatable.

Definetestsandtestsetsforeachmodelanddetermineminimalrequirementsforinter-rateragreements.

Trainandretainevaluatorteams Establishscalableandrepeatableprocessesbyusingtoolsandautomated

processesfordatapreparation,evaluationsetupandanalysis

Captureevaluationresultsautomaticallytoenablecomparisonsacrosstime,

projects,vendors.

Usecolor-codingforcomparingperformanceovertime,e.g.greenformeetingorexceedingexpectations,ambertosignalareductioninquality,

redforproblemsthatneedaddressing.

ImplementaCAPA(CorrectiveActionPreventiveAction)process.

Bestpracticeisfortheretobeaprocessinplacetodealwithqualityissues-correctiveactionprocessesalongwithpreventiveactionprocesses.Examples

mightincludetheprovisionoftrainingortheimprovementofterminologymanagementprocesses.

Adequacy/Fluencyreviewmaynotidentifyrootcauses.Ifreviewscoreshighlightmajorissues,moredetailedanalysismayberequired,forexample

usingErrorTypologyreview.

Ifevaluationsfollowtheserecommendationsyouwillbeabletoachievereliable

andstatisticallysignificantresultswithmeasurableconfidencescores.

Furtherresources:

ForTAUSmembers:Forinformationonwhentouseadequacyand/orfluency

approaches,conditionsforsuccess,step-by-stepprocessguides,readytouse

templatesandguidanceontrainingevaluators,pleaserefertotheTAUSDynamic

QualityFrameworkKnowledge.

Ourthanksto:

KarinBerghoefer(AppenButlerHill)fordraftingtheseguidelines.

ThefollowingorganizationsforreviewingandrefiningtheGuidelinesattheTAUS

QualityEvaluationSummit15March2013,Dublin:

7/28/2019 TAUS Adequacy/Fluency Guidelines

4/6

ABBYYLanguageServices,CapitaTranslationandInterpreting,Crestec,Intel,Jensen

Localization,JonckersTranslation&Engineerings.r.o.,Lingo24,LogrusInternational,

Microsoft,PalexLanguages&Software,Symantec,andTekom.

ConsultationandPublication

Apublicconsultationwasundertakenbetween11and24April2013.Theguidelines

werepublishedon2May2013.

Feedback

Togivefeedbackonhowtoimprovetheguidelines,pleasewriteto

dqf@tauslabs.com.

7/28/2019 TAUS Adequacy/Fluency Guidelines

5/6

TAUSMembersasofMay2013

ABBYY

Acclaro

AcrolinxAdobe

AEDIT

Amesto

AppenButlerHill

AttachedLanguage

Solutions

Autodesk

AVBVertalingen

BBNRaytheon

Beo

CATechnologies

Capita

CelerSoluciones

CharlesUniversity

Cisco

Citrix

ClayTablet

Cloudwords

CLSCommunication

CNGL

Concorde

CPSL

Crestec

CrossLang

DELL

DigitalLinguistics

E4NET

eBay/Paypal

EMC

EuropeanPatentOffice

FBINVTCGlobalTextware

Google

HarleyDavidson

HP

Pactera

HonyakuCenterHunnect

iDISC

Informatica

Inpokulis

Intel

JensenLocalization

JohnDeere

JohnHopkins

University

KantanMT

Kilgray

L&L

LanguageIntelligence

LanguageService

Associates

LDSChurch

LexWorks

Lingo24

LingoSailTechnology

LingsoftOy

LinguisticSystems

LinguaCustodia

Lionbridge

Logrus

LucySoftware

Manpower

Medilingua

Medtronic

MemSource

Microsoft

MoraviaWorldwideMorphologic

MultiCorporaR&DInc

NICT

Omniage

OmniLingua

OraclePangeanic

Philips

PROMT

PTC

RRDonnelley

Safaba

SDL

Semantix

Siemens

SimpleShift

Skrivanek

Smartling

SpeakLike

SpokenTranslation

STPNordic

StrakerSoftware

Symantec

Systran

Tauyou

Tekom

Tembua

Tilde

Translated.net

TripAdvisor

TrustedTranslation

UniversityofHelsinki

UrbanTranslation

Service

Welocalize

WinandWinnow

XTMInternationalYahoo!

YamagataEurope

7/28/2019 TAUS Adequacy/Fluency Guidelines

6/6

TAUSMissionStatement

TAUSisaninnovationthinktankandplatformforindustry-sharedservices,

resourcesandresearchforthetranslationsectorglobally.

Weenvisiontranslationasastandardfeature,aubiquitousservice.LiketheInternet,

electricity,andwater,translationisoneofthebasicneedsofhumancivilization.

Ourmissionistoincreasethesizeandsignificanceofthetranslationindustrytohelp

theworldcommunicatebetter.

Wesupportentrepreneursandprincipalsinthetranslationindustrytoshareand

definenewstrategiesthroughacomprehensiverangeofevents,publicationsand

knowledgetools.

Tofindouthowwetranslateourmissionintoservices,pleasewriteto

memberservices@translationautomation.comtoscheduleanintroductorycall.