taus mt showcase, the dqf tools, rahzeb choudhury and maxim khalilov, taus, 12 june 2013
DESCRIPTION
This presentation is a part of the MosesCore project that encourages the development and usage of open source machine translation tools, notably the Moses statistical MT toolkit. MosesCore is supported by the European Commission Grant Number 288487 under the 7th Framework Programme. For the latest updates, follow us on Twitter - #MosesCoreTRANSCRIPT
TAUS MACHINE TRANSLATION SHOWCASE
TAUS Dynamic Quality Framework Tools 11:30 – 11:50 Wednesday, 12 June 2013 Rahzeb Choudhury and Maxim Khalilov TAUS
2
Clients
MLV’s
In country offices/partners
Distributed translators/authors
4 to 30 vendors
10 to 40 languages
100’ to 1000’s translators/authors
Vendor Management Project Management
Quality Assurance TranslaOon Memory
Account Management
Resources Management
Quality Assurance
Project Management TranslaOon Memory
Resources Management
Quality Assurance
Project Management
TranslaOon Memory
Quality Assurance TranslaOon Memory
Cascaded Supply Chain
This slide may not be used or copied without permission from TAUS
Dynamic Quality Framework • Need to go up and down in quality and the translaPon
industry’s use of quality evaluaOon models and metrics needed to reflect this,
• Need industry benchmarking to strengthen and improve the credibility of quality assurance processes,
• Need to lower cost of quality assurance at buyers and providers.
" We aim to achieve this through an industry shared knowledge base, guidance on
QE model selecOon and shared tools for benchmarking.
This slide may not be used or copied without permission from TAUS
Project OrganizaPon
• Ongoing iniPaPve since January 2011
• Bimonthly project management calls, primarily with TAUS members
• TAUS DQF team – Nora Aranberri, Maxim Khalilov,
Sharon O’Brien, Jaap van der Meer, Rahzeb Choudhury
• Development outsourced
This slide may not be used or copied without permission from TAUS
DYNAMIC QUALITY FRAMEWORK THEORY
This slide may not be used or copied without permission from TAUS
This slide may not be used or copied without permission from TAUS
This slide may not be used or copied without permission from TAUS
DYNAMIC QUALITY FRAMEWORK IN PRACTICE
DQF Content Profiling – Since July 2012
This slide may not be used or copied without permission from TAUS
DQF Content Profiling
• DelineaPng quality requirements
• Shared approach to idenPfy best matching QE model(s) for content type and operaPng model
• Greater credibility to decisions
• Based on feedback during QE Summit Dublin (March 2013), working on version 2
• Available for scruPny QE Summit San Jose, October 9
This slide may not be used or copied without permission from TAUS
DQF Knowledgebase – Since July 2012
This slide may not be used or copied without permission from TAUS
DQF Knowledgebase
This slide may not be used or copied without permission from TAUS
DQF Knowledgebase
This slide may not be used or copied without permission from TAUS
DQF Knowledgebase
This slide may not be used or copied without permission from TAUS
DQF Knowledgebase
This slide may not be used or copied without permission from TAUS
Acer DQF KB
• Leverage industry best pracPces to create/refine quality programs
• Central resource that can be systemaPcally reviewed and improved to benefit all
• Work together to standardize approaches, definiPons and metrics
• Following QE Summit Dublin held consultaPons and published publicly available best pracPces guidelines
This slide may not be used or copied without permission from TAUS
DQF Best PracPce Guidelines – Since May 2013
This slide may not be used or copied without permission from TAUS
DQF Best PracPce Guidelines
This slide may not be used or copied without permission from TAUS
DQF Tools
This slide may not be used or copied without permission from TAUS
Metrics: Repor-ng
ü See how you compare against peers ü Drill down to see the impact of individuals ü Assess correlaPon between automated metrics and
human evaluaPon ü Shared learning to feed fundamental research Apply objec-ve criteria and raise the credibility of transla-on quality assurance func-on across the industry
This slide may not be used or copied without permission from TAUS
Quality evaluaPon
ProducPvity tesPng
System comparison
-‐ Adequacy -‐ Fluency -‐ Error typology
Post-‐edited MT / translaPon from scratch
-‐ Ranking -‐ Quick comparison
DQF tools
This slide may not be used or copied without permission from TAUS
Metrics: Tools
This slide may not be used or copied without permission from TAUS
MT EvaluaPon Tool: new project
This slide may not be used or copied without permission from TAUS
MT EvaluaPon Tool: new project
This slide may not be used or copied without permission from TAUS
MT EvaluaPon Tool: new project
This slide may not be used or copied without permission from TAUS
MT EvaluaPon Tool: new project
This slide may not be used or copied without permission from TAUS
MT EvaluaPon Tool: new project
This slide may not be used or copied without permission from TAUS
MT EvaluaPon Tool: quality evaluaPon project
This slide may not be used or copied without permission from TAUS
MT EvaluaPon Tool: quality evaluaPon project
This slide may not be used or copied without permission from TAUS
MT EvaluaPon Tool: quality evaluaPon project
This slide may not be used or copied without permission from TAUS
Metrics: Repor-ng
This slide may not be used or copied without permission from TAUS
Metrics: Repor-ng
This slide may not be used or copied without permission from TAUS
Post-‐editor 1
Metrics: Repor-ng
This slide may not be used or copied without permission from TAUS
Upcoming DQF
• ConsutlaPons (pricing MT post-‐ediPng)
• TranslaPon Tools Adopt DQF Models
• DQF Tool v2 -‐ Go live August
• LocalizaPon InsPtute Course translaPon quaity evaluaPon; will be based largely on the DQF, starts September 2013
• QE Summit, San Jose, 9 October 2013, hosted by Adobe
This slide may not be used or copied without permission from TAUS
QE Summit San Jose
This slide may not be used or copied without permission from TAUS