i

Crowdsourced Translation Practices from the Process Flow Perspective

A Thesis Submitted for the Degree of

Doctor of Philosophy

By

Aram Morera Mesa

Department of Computer Science and Information Systems,

University of Limerick

Supervisors: J.J. Collins, Dr. David Filip

Co-Supervisor: Reinhard Schler

Submitted to the University of Limerick, October 2014

ii

Table of Contents Abstract .................................................................................................................................... vii

Declaration ............................................................................................................................. viii

Acknowledgments..................................................................................................................... ix

Publications and Presentations from this Research Project ....................................................... x List of Figures ........................................................................................................................... xi

List of Tables ........................................................................................................................... xii

Chapter 1 Introduction ............................................................................................................... 1

1.1 Overview .......................................................................................................................... 1

1.2 Research Question and Objectives................................................................................... 4 1.3 Methodology .................................................................................................................... 7

1.4 Scope ................................................................................................................................ 8

1.5 Thesis Structure ............................................................................................................... 8

Chapter 2 Literature Review .................................................................................................... 10

2.1 Introduction .................................................................................................................... 10

2.2 Localisation .................................................................................................................... 10

2.2.1 Criticism of Localisation......................................................................................... 13

2.2.2 The localisation process .......................................................................................... 13

2.2.3 Localisation technologies........................................................................................ 18

2.2.4 Localisation Levels ................................................................................................. 23

2.2.5 Summary of the literature review for localisation .................................................. 25

2.3 Crowdsourcing ............................................................................................................... 25

2.3.1 Introduction to Crowdsourcing ............................................................................... 25

2.3.2 Crowdsourcing in Action ........................................................................................ 29

2.3.3 Crowdsourcing Classifications ............................................................................... 32

2.3.4 Matching the Taxonomy to the Definition.............................................................. 39

2.3.5 Crowdsourcing in Localisation ............................................................................... 40

iii

2.3.6 Taxonomies within Localisation ............................................................................. 43

2.3.7 Benefits of Crowdsourcing in the Context of Localisation .................................... 46

2.3.8 Criticism in the Context of Localisation ................................................................. 48

2.3.9 Other Collections of Practices ................................................................................ 51

2.3.10 Summary of the Literature Review for Crowdsourcing ....................................... 51

2.4 Workflows...................................................................................................................... 51

2.4.1 Workflow Models ................................................................................................... 52

2.4.2 Workflow Patterns .................................................................................................. 53

2.4.3 Workflows in the Language Industry ..................................................................... 54

2.4.4 Industry workflows for crowdsourced translation .................................................. 55

2.4.5 Models of crowdsourced translation workflows ..................................................... 56

2.4.6 Modelling practices ................................................................................................. 57

2.4.7 Summary of the Literature Review for Workflows ................................................ 67

Chapter 3 Taxonomy of crowdsourcing .................................................................................. 69

3.1 Data Collection .............................................................................................................. 70

3.1.1 Data from Models ................................................................................................... 70

3.1.2 Online Questionnaire and Survey considerations ................................................... 71

3.1.3 Survey Administration ............................................................................................ 72

3.1.4 Survey Design ......................................................................................................... 73

3.2 Clustering ....................................................................................................................... 77

3.2. 1 TwoStep Clustering ............................................................................................... 77

3.2.2 Other Approaches to Clustering.............................................................................. 83

3.2.2.2 Hierarchical clustering ........................................................................................ 90

3.3 Conclusions .................................................................................................................... 91

Chapter 4 Workflow Models ................................................................................................... 93

4.1 Static models vs simulable models ................................................................................ 93

4.2 The models ..................................................................................................................... 93

iv

4.2.1 Crowdin................................................................................................................... 94

4.2.2 Asia Online ............................................................................................................. 98

4.2.3 Facebook ................................................................................................................. 99

4.2.4 Pootle .................................................................................................................... 101

4.2.5 Launchpads Translations ..................................................................................... 104

4.2.6 DotSub .................................................................................................................. 107

4.2.7 Amara .................................................................................................................... 110

4.2.8 Kiva ....................................................................................................................... 112

4.3 The practices ................................................................................................................ 114

4.4 Summary ...................................................................................................................... 118

Chapter 5 Refinement of the Practices ................................................................................... 119

5.1 Introduction .................................................................................................................. 119

5.2 The Choice of Semi Structured Interview ................................................................... 119

5.3 The Selection of Interviewees ...................................................................................... 119

5.4 The interviews .............................................................................................................. 121

5.5 The questions ............................................................................................................... 122

5.5.1 Question Sequence ................................................................................................ 123

5.5.2 Question list .......................................................................................................... 123

5.6 Approach to Data Analysis .......................................................................................... 129

5.7 Analysis Outcome ........................................................................................................ 133

Practice 1: Content Selection ......................................................................................... 133

Practice 2: TU Granularity Selection ............................................................................. 142

Practice 3: Leveraging Translation Memory ................................................................. 149

Practice 4: Leveraging MT ............................................................................................ 152

Practice 5: Leveraging Terminology ............................................................................. 155

Practice 6: Translation without Redundancy ................................................................. 157

Practice 7: Open Alternative Translations ..................................................................... 160

v

Practice 8: Hidden Alternative Translations .................................................................. 163

Practice 9: Super Iterative Translation .......................................................................... 166

Practice 10: Freeze ......................................................................................................... 171

Practice 11: Version Rollback ........................................................................................ 175

Practice 12: Deadlines ................................................................................................... 176

Practice 13: Open Assessment ........................................................................................ 180

Practice 14: Hidden Assessment..................................................................................... 185

Practice 15: Expert Selection and Edition ..................................................................... 186

Practice 16: Metadata Based Selection .......................................................................... 192

5.8 Discussion of Practices ................................................................................................ 196

5.9 Summary ...................................................................................................................... 207

Chapter 6 Practices and scenarios .......................................................................................... 208

6.1 Scenario 1 Translation for Engagement ....................................................................... 208

6.1.1 Practices to Support Translators ........................................................................... 209

6.1.2 Practices to Enable Higher Engagement ............................................................... 210

6.1.3. Practices that give the Crowd Ownership ............................................................ 211

6.1.4 Other Practices for Translation for Engagement .................................................. 211

6.1.5 Discussion of the Translation for Engagement Scenario ...................................... 212

6.2 Scenario 2 Crowd TEP................................................................................................. 213

6.2.1 Volunteer translator .............................................................................................. 213

6.2.2 Crowd Post-Edition ............................................................................................... 216

6.3 Scenario 3 Colony translation ...................................................................................... 217

6.4 Scenario 4 Wiki Style Translation ............................................................................... 219

6.5 Long tail scenario variations ........................................................................................ 221

6.6 Summary ...................................................................................................................... 222

7 Conclusions ......................................................................................................................... 223

7.1. Summary of results ..................................................................................................... 223

vi

7.3. Impact of the Research Contributions. ........................................................................ 225

7.4. Limitations and Future Research ................................................................................ 226

7.5 Summary ...................................................................................................................... 232

Bibliography .......................................................................................................................... 233

Appendix 1 Survey Questionnaire ......................................................................................... 245

Appendix 2 Email Template .................................................................................................. 247

Appendix 3 Ethical Clearance Application Form for Survey ................................................ 248

Appendix 4 Ethical Clearance Application Form for Interviews .......................................... 252

Appendix 5 Survey Responses............................................................................................... 256

Appendix 6 Failed Mind Map for Pattern Language Development ...................................... 258

vii

Abstract This thesis explores a series of crowdsourced translation platforms that includes Asia Onlines Wikipedia Translation Project, DotSub, Amara, Kiva and others. It then creates a taxonomy of them based on a previously existing one for general crowdsourced processes. The taxonomy resulted in four approaches to crowdsourced translation: translation for engagement, colony translation, crowd TEP and wiki style translation.

Having created the taxonomy, the thesis focuses on the different processes enacted by the platforms and presents workflow models for a selection of the platforms. Through analysis of the workflow models, the thesis identifies fourteen crowdsourced translation practices. Some of these, such as the leverage of MT and TM, are common in mainstream localisation processes, while others, such as the collection of redundant alternative translations and redundantly iterative translations, appeared only with the emergence of crowdsourcing.

In order to define the practices in a way that emulates design patterns for software, eight interviews with experts in crowdsourced translation processes were carried out. These resulted in over 64,000 words discussing the advantages and disadvantages, pre-requisites and other features of the practices. This information was used to refine the definition of the practices. The refined practices definitions have been organized in a pattern catalogue that describes the relationships between the different patterns and different scenarios. This makes them helpful for organisations interested in implementing crowdsourced translation.

viii

Declaration I hereby declare that this thesis is entirely my own work, and that it has not been submitted as an exercise for a degree at any other university.

ix

Acknowledgments This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (www.cngl.ie) at University of Limerick and the Armando Morera Fumero Foundation.

This thesis started as something completely different and over the almost five years it took to write it many people had an impact on it. I would like to thank Lamine Aouad and Eoin Conchir who were briefly my supervisors and during that time they made valuable contributions that got me closer to finishing. Reinhard Schler set the scene for this research and without his initial push, this thesis would never have started. J.J. Collins and David Filip were critical to the completion of this thesis not only because of their guidance, but because of the great work they did in finding flaws in my research, helping me solve them and encouraging my work until I finished.

I would also like to thank Karl Kelly and Geraldine Harrahill, who with their mastery of the internal processes of the University of Limerick solved numerous issues for me, allowing me to focus on my research. Without them, I would have been more stressed and whined more about the universitys inner workings.

Whining there was, though, and it was Asanka Wasala, Luca Morado, Naoto Nishio, Rajat Gupta and Solomon Gizaw, my office colleagues, who had to listen to half of it. Without the supportive atmosphere that they created, I would have likely given up and that makes their support as valuable a contribution as any.

The other half of the whining was enjoyed by my parents, friends and partner. Five years is very long time and I have to thank them for not disinheriting, ostracising or dumping me. Writing a thesis affects both your professional and private life, and having such kind people around me in the private life was also fundamental for the eventual completion of this research.

Lastly, I would like to thank the experts and researchers who contributed their knowledge to the interviews and the survey in this thesis.

x

Publications and Presentations from this Research Project Papers

Aouad, L., OKeeffe, I., Collins, J.J., Wasala, A., Nishio, N., Morera, A., Morado, L., Ryan, L., Gupta, R., Schaler, R. (2011) A View of Future Technologies and Challenges for the Automation of Localisation Processes: Visions and Scenarios, in Lee, G., Howard, D. and lzak, D., eds., Convergence and Hybrid Information Technology, Communications in Computer and Information Science, Springer Berlin Heidelberg, 371382, available: http://dx.doi.org/10.1007/978-3-642-24106-2_48.

Morera, A., Aouad, L., Collins, J. (2012) Assessing Support for Community Workflows in Localisation, Presented at the Business Process Management Workshops, Springer, 195206.

Morera-Mesa, A., Collins, J.J., Filip, D. (2013) Selected Crowdsourced Translation Practices, Presented at the Translating and the Computer Conference 35, London.

Presentations

Morera, A., Auouad, L. (2011) Towards an intelligent localisation workflow management system, in II Simposio Internacional de Jvenes Investigadores En Traduccin, Interpretacin y Estudios Interculturales, Barcelona.

Morera, A., Auouad, L., Collins J.J. (2011) Assessing Enterprise Support for Community Workflows in Localization, in SIMPDA 2011, Campione dItalia.

Morera, A., Auouad, L., Collins J.J. (2011) Integrating the Community in the Workflow: Mapping Project Attributes to Patterns, in SIMPDA 2011, Campione dItalia.

Morera, A., Auouad, L., Collins J.J. (2011) Elevator Pitch for Crowdsourced Translation Workflows, in SFI Digital Content Workshop, Dublin.

Morera, A., Auouad, L., Collins J.J. (2011) Assessing Support for Community Workflows in Localisation, in BPMS2, Clermont-Ferrand.

Reviews Mesa, A.M. (2013) Keiran J. Dunne and Elena S. Dunne (eds.): Translation and localization

project management: the art of the possible, Machine Translation, 18.

xi

List of Figures Figure 1 Industrial TEP process. Adapted from Ray and Kelly (2011) ..................................... 2 Figure 2 Percentage of Internet users that add content according to Eurostat ......................... 31 Figure 3 An industrial localisation process model on WorldServer ........................................ 54 Figure 4 High level representation of a crowdsourced localisation timeline. Adapted from DePalma and Kelly (2011) ....................................................................................................... 55 Figure 5 Representation of a crowdsourced process. Adapted from Vashee (2009) ............... 56 Figure 6 A BPMN model of a suggested process for bi-text managemet in crowdsourcing scenarios. Used with permission (Filip and Conchir 2011) ............................................... 57 Figure 7 Petri nets elements used in this thesis........................................................................ 59 Figure 8 The same behaviour modelled in Petri nets and BPMN 2.0...................................... 60 Figure 9 Summary of TwoStep clustering with five clusters .................................................. 80 Figure 10 Summary of TwoStep clustering with six clusters .................................................. 83 Figure 11 Crowdin Process Model at the Locale Level ........................................................... 94 Figure 12 Crowdin MT and TM leveraging subworkflow ...................................................... 96 Figure 13 Crowdin translate and vote subworkflow ................................................................ 97 Figure 14 Model for Asia Onlines Wikipedia translation project .......................................... 98 Figure 15 Facebooks process at the locale level ................................................................... 100 Figure 16 Facebooks process at the string level. .................................................................. 101 Figure 17 Model of a Pootle Process at the locale level ........................................................ 102 Figure 18 Model of a Pootle process at the string level ......................................................... 103 Figure 19 Model of Launchpad's translation platform at the locale level ............................. 105 Figure 20 A model of the Launchpad Translation process at the string level ....................... 106 Figure 21 A suggested model for Launchpad Translation process at the string level ........... 107 Figure 22 A model of DotSub's process at the video level .................................................... 108 Figure 23 A model of the Amara process at the video level .................................................. 110 Figure 24 A model of Amara's process at the subtitle level .................................................. 111 Figure 25 A suggested model for Amara at the video level .................................................. 112 Figure 26 A model of Kiva's process from the point of view of a volunteer ......................... 113 Figure 27 References for each practice .................................................................................. 133

xii

List of Tables Table 1 Comparison of the localisation process as described by Esselink (2000), Pym (2004) and Schler (2008a) ................................................................................................................. 15 Table 2 Localisation process stages and related practices ....................................................... 17 Table 3 Levels of localisation adapted from Carey (1998)...................................................... 24 Table 4 Google Scholar hits per year for crowdsourcing and some crowdsourcing related terms ......................................................................................................................................... 26 Table 5 Preselection of Contributors from Geiger, Seedorf et al (2011) ................................. 38 Table 6 Aggregation of Contributions from Geiger, Seedorf et al (2011)............................... 38 Table 7 Remuneration of Contributions from Geiger, Seedorf et al (2011) ............................ 38 Table 8 Accessibility of peer contributions from Geiger, Seedorf et al (2011) ....................... 38 Table 9 Characteristics of the platforms obtained through the modelling process .................. 70 Table 10 Additional data for the taxonomy ............................................................................. 78 Table 11 Original cluster distribution and distribution after data expansion........................... 82 Table 12 Cluster membership with the same weight assigned to all dimensions .................... 86 Table 13 Conversion values for the characteristics of each dimension ................................... 87 Table 14 Cluster membership after triplicating the weight of the Aggregation dimension..... 87 Table 15 Cluster membership as per hierarchical clustering ................................................... 91 Table 16 Date and duration of interviews for each subject ................................................... 121

1

Chapter 1 Introduction

1.1 Overview The objective of this thesis is to create a collection of practices that facilitates the implementation of crowdsourced translation processes. This chapter introduces the concept of localisation and the challenge posed by the increased demand for localised content and services. An overview is presented on crowdsourcing and its connection to localisation; and on workflows and their relationship to crowdsourcing in localisation. The chapter then discusses the research questions addressed by this work, the expected contribution to knowledge, and presents a number of potential answers to the questions posed. The methodology is presented next in which the methodologies used to explore the research questions is described, and is followed by the scope and finally, the structure of the thesis.

1.1.1 Localisation Localisation was defined by LISA (Localization Industry Standards Association) as "the process of modifying products or services to account for differences in distinct markets" (Lommel 2003). Localisation is a complex process that involves "project management, engineering, quality assurance, and human-computer interface design issues" (Schler 2008a). This complexity results in localisation involving the collaboration of many different professionals such as software developers, localisation engineers, project managers, translators, etc that are often geographically dispersed throughout different time zones.

Localisation was initially carried out by internal teams in multinational corporations, but as the market grew, specialized companies known as Language Service Provides (LSP) appeared (Esselink 2000).

The interest in localising content is a consequence of globalisation, that is the process of increasing interdependence and global enmeshment which occurs as money, people, images, values, and ideas flow ever more swiftly and smoothly across national boundaries (Hurrell and Woods 1995). This increase in globalization also has resulted in an increased demand for localized content that has become a challenge for the industry both because of volume and diversity of locales (Ryan et al 2009, p.17).

2

1.1.2 Crowdsourcing One of the recent developments in localisation is the introduction of crowdsourcing, which is the practice of leveraging communities to carry out tasks that were traditionally carried out by professionals under contract (Howe 2006a). The idea of using crowdsourcing to deal with the increasing volume of content and locales has gained traction within the industry to the point that it has been argued that with a crowd of motivated, tech-savvy users, crowdsourcing is actually the best way of localising a product (Kelly 2009; Rickard 2009). It has already been proven that for some tasks, a crowd may deliver better quality than an expert (Sakamoto et al. 2011). In language related tasks, including translation, the outcome has been of quality comparable to that of professionals for long tail languages (Callison-Burch 2009; Bloodgood and Callison-Burch 2010; Zaidan and Callison-Burch 2011).

The potential gain from adoption of this paradigm was illustrated by Facebook that in 2009 had sixty five languages available and an additional forty in production (DePalma and Kelly 2011), in some cases with a quality that is higher than what could be expected of professionals (Jimnez-Crespo 2013) Although this success has lead industry experts to state that crowdsourcing will become integrated in the content supply chain, most LSPs have not been able to find a way of integrating crowdsourcing into their processes and see it as a threat to the traditional industry model that follows a Translate Edit Publish (TEP) process similar to the one illustrated in Figure 1 (Ray and Kelly 2011).

Figure 1 Industrial TEP process. Adapted from Ray and Kelly (2011)

In this context, not many organisations have methods that allow them to use crowdsourcing in their processes, leaving them ill prepared to take advantage of the possibilities that crowdsourcing brings. The fact that research into best practices for working with swarms of volunteers in crowdsourcing has only begun underlines this problem (ibid.). Furthermore, as noticed by Zhao and Zhu (2012), the fact that crowdsourcing is such a new phenomenon is observable in the many publications focused on conceptualization, i.e. defining

3

crowdsourcing and how it relates to other similar phenomena, instead of focusing on systems or applications of crowdsourcing.

This thesis explores the crowdsourcing processes used by Non-Governmental Organisations (NGO), open source projects like Ubuntu (Mackenzie 2006), LibreOffice and Firefox (Dalvit et al. 2008), commercial companies like Facebook (Losse 2008; Mesipuu 2010) and Asia Online (Morera et al. 2012) that have been able to leverage generally unpaid volunteer translators. Companies like Amazon with Amazon Mechanical Turk (Kittur et al. 2008), TextEagle (Eagle 2009) and Crowdflower (Munro et al. 2010) that have been able to offer translation by tapping into generally paid crowds, are also included. All of these organisations have successfully leveraged the crowd and the bespoke processes that they use or enable illustrate the current crowdsourced translation practices.

Practices in this thesis are paths of action that are followed in given contexts. These practices are diverse and depend on the main purpose of the organisation's crowdsourcing effort; be this increasing customer engagement, lowering cost or some other purpose that has not been discussed in the literature. These practices contrast with LSP approaches to translation that generally focus on leveraging technologies such as Machine Translation (MT), Translation Memories (TM), Translation Management Systems (TMS) and Terminology Databases (TD) to produce an output with volume and quality that would otherwise require many more human resources or time (Somers 2003; Bowker 2005; Plitt and Masselot 2010; Straub and Schmitz 2010).

1.1.3 Workflows According to the Workflow Management Coalition a workflow is The automation of a business process, in whole or part, during which documents, information or tasks are passed from one participant to another for action, according to a set of procedural rules (WfMC 1999).

Business process models are often used to represent workflows. These models are similar to, but simpler than that which they represent (Maria 1997), in this case, workflows since they do not represent information that is not relevant to the purpose of the model. This reduction of complexity, among other things, makes the models useful for understanding and communicating the processes they represent (Giaglis 2001). There are a variety of business process modelling languages like Business Process Execution Language (BPEL) Yet Another Workflow Language (YAWL), XML Process Definition Language (XPDL) and many more

4

(van der Aalst 2004) that can be used to define workflows and a number of systems able to enact them including jBoss, Windows Workflow Foundation, WebSphere Process Server among others (Louridas 2008). These business process modeling and execution languages are not frequently used by LSPs, which instead use Translation Management Systems (TMS). TMSs often integrate workflow solutions suitable for the Translation, Editing, and Proofreading (TEP) process (Rinsche and Portera-Zanotti 2009). Although there are organisations that claim to use crowdsourced translation in their processes, the literature underlines once more shows the immaturity of the field by having very few examples of process models as it will be shown in Chapter 2.

This thesis uses coloured Petri nets, which have been extensively used in the workflow literature, to create process models that facilitate human understanding of the process, one of the functions of process models according to Curtis et al (1992).

1.2 Research Question and Objectives The objective of this research is to create a collection of practices that facilitates the implementation of crowdsourced translation processes. These practices are based on the patterns of other disciplines. In this context, a pattern is formal way of documenting a common solution to a common problem in a particular field of expertise (Dsilets and van der Meer 2011). According to Buschmann et al (2007), the optimal way to present patterns to facilitate their implementation is in the form of a pattern language, that is, a collection that

makes explicit the interdependencies, synergies and conflicts that exist between the patterns. However, organizing the practices in a manner similar to that of a pattern language falls out

of the scope of this thesis and instead a series of recommendations of practice combinations matching existing scenarios were developed. In order to produce these recommendations a series of questions had to be answered.

Q1. What are the existing kinds of crowdsourced translation processes?

Several classifications for crowdsourcing processes in general have been developed and at least two specific to crowdsourced translation. The existing classifications specific to crowdsourced translation discussed in Chapter 2 do not meet the requirement of having a set of dimensions specific aspects of the objects being classified with characteristics values for each dimension that are mutually exclusive and collectively exhaustive (Bailey 1994;

5

Nickerson et al. 2009), necessary for a taxonomy to be useful. Therefore a new taxonomy is hereby proposed.

Hypothesis 1A: All crowdsourced translation processes fit in the same group and the cluster has a good fit. This would indicate that the processes are too similar to be divided in taxa. If this is true, a number of random platforms can be selected to review the processes that they enact and it is likely that most practices appear across all the platforms.

Hypothesis 1B: All crowdsourced translation processes fit in one or more clusters but the cluster fit is poor. This would indicate that the processes are too dissimilar to form distinct groups or if they do, the within-group commonalities are limited. This could be true or the result of insufficient data. If this is true, a number of random platforms can be selected to review the processes that they enact, but it is unlikely that any common practices will emerge. If common practices emerged, it would indicate that the clustering process needs additional data to work correctly.

Hypothesis 1C: There are different types of crowdsourcing processes and the fit is good. If this is true at least a representative platform for each class will be selected to review the processes that it enables. Some practices will appear only within a specific group, and others will appear across multiple groups.

Q2. What practices appear in the different types of crowdsourced translation processes?

One of the tools used to understand how a process works is the process model. Current process representations for crowdsourced translation, such as the one presented by DePalma and Kelly (2011), are very high level and lack sufficient detail necessary to support an understanding of crowdsourcing in action. To obtain a deeper understanding of the processes a series of workflow models were created. These models facilitate the comparison of processes in order to find their similarities and differences which will be useful when identifying the practices that are common among different types of crowdsourced translation processes and the practices that fit only in one crowdsourced translation scenario. By analysing the models and considering practices described in the literature, it is possible to identify suitable candidates because they appear repeatedly.

Hypothesis 2A: If the answer to Q1 is H1A (one cluster, good fit) and the models are very similar, there will be similarities that will be a starting point for practices. In this

6

case the practices extracted from them will be generally applicable in crowdsourced translation processes.

Hypothesis 2B: If the answer to Q1 is H1B (one or more clusters, poor fit) and the process models are diverse, it would indicate that no practices can be extracted from the data in this thesis. This does not imply that there are no useful practices for crowdsourced translation, since it is possible that by adding more platforms and more models, clusters with a better fit and similarities among models emerge.

Hypothesis 2C: If the answer to Q1 is H1C (multiple clusters, good fit) and the process models are diverse across different classes but similar within a class, there will be practices that appear repeatedly across classes and are applicable to most crowdsourced translation processes; these would be generally applicable practices. There will also be practices that appear only within a given class and are applicable mainly in those processes; these would be practices applicable only within specific classes of crowdsourced translation.

Q3. What are the forces that shape the candidate practices?

Refining practices to make them more useful for those who may want to apply them requires

a good understanding of the forces, i.e. requirements, consequences and constraints that shape them. Some of these forces will be evident - e.g. if you only collect one suggested translation per person, you need more than one person involved in order to have Open Alternative Translations, the practice of openly collecting multiple translations for a single source TU. Other forces may not be as transparent and can only be explicated through systematic research. To identify the latter, a group of experts were interviewed. This is an exploratory endeavour and too many hypothesis specific to each practice exist to list them here. Those hypotheses that the researcher considered but were not confirmed by the interviewees appear in the discussion of each practice in Chapter 5.

Q4. How can the different practices be combined?

Buschmann et als (2007) talk about patterns, a formal way of defining practices (Dsilets and van der Meer 2011), and how in order to make their implementation less challenging, they should be organized in a pattern language. They suggested that a pattern language can be created by organizing the patterns according to the order in which their problem part appears in the processes. This type of pattern organisation illustrates their relationships of

7

interdependence, synergies and conflicts. Ideally the practices should have been organized in a manner analogous to that of a pattern language, but in order to create a pattern language a more holistic approach to the collection of practices would have been necessary. For example, resourcing and data management practices would have had to be considered, but these fell out of scope. For this reason, instead of organizing the practices in a manner similar to a pattern language, Chapter 6 contains a series of combinations of practices based on existing scenarios.

1.3 Methodology A survey was used to develop a taxonomy that allows for better understanding of the different approaches used in crowdsourced translation. The survey had seven questions of which four were used to collect the attributes used by the taxonomy and three to identify crowdsourcing platform, organisation and rationale for crowdsourcing. Given the low number of responses, the researcher selected other platforms and familiarized himself with them in order to be able to add them to the taxonomy. With this data at hand the researcher used the k-means algorithm to find out the different groups within the data. This was done according to the three level model proposed by Bailey (1994). More details on how the classification was carried out can be read in Chapter 3.

From the taxonomy a number of platforms were selected for a deeper analysis resulting in the creation of workflow models. A standard methodology for the creation of workflow models has yet to be proposed. Van der Aalst et al (2004) observe that modelling a workflow is a non-trivial task that requires knowledge of the workflow language and lengthy discussions with the workers and management involved. Curtis et al (1992) stated that In practice, most process descriptions have employed narrative text and simple diagrams to express process. The models in this thesis were created either from the narrative presented by representatives of companies in conferences, or by creating several user accounts in different services in order to take the role of different stakeholders and directly modelling each step.

The analysis of the workflow models allowed for the identification of a number of practices that were later on refined using interviews with experts in order to make them easier to implement by making them more similar to the design patterns proposed by Alexander el al (1977).

8

Lastly, combinations of practices were attached to different existing scenarios in order to make their implementation easier. These collections take cues from pattern languages, and it is the case with the latter, they emerge from praxis, will evolve with it and there is not any established academic methodology for their development.

1.4 Scope This research focuses on crowdsourced translation and deals only with the practices that directly affect payload data (the translation themselves) or metadata for the payload data generated by the crowd (votes and comments).

There are crowdsourcing practices related to resourcing and motivation that are also important for a successful crowdsourcing process, but these are outside the scope of this thesis.

1.5 Thesis Structure Chapter 1 has provided background information about localisation, its relationship with crowdsourcing and pointed out the high level nature of the research on processes done so far.

Chapter 2 has three main sections. Section one discusses localisation in more depth, including more background information and discussion of the tools and technologies used in mainstream processes. Section two is dedicated to crowdsourcing. It discusses some of the existing taxonomies for general crowdsourcing and specifically for crowdsourced translation. Besides that, it addresses the benefits that crowdsourcing offers and criticisms that have been raised against it. Section three discusses workflows, the role of workflow models, their presence in the industry and issues related to their creation.

Chapter 3 discusses the methodology for the creation of a new taxonomy specific for crowdsourced translation processes, presents the data used to create the taxonomy and present the new taxonomy.

Chapter 4 discusses the approach used for the development of the workflow models, presents the workflow models for eight different crowdsourcing platforms and outlines the practices that will be refined and become part of the collection.

9

Chapter 5 presents the methodology used to refine the proposed practices through interviews; presents the practices after their being refined with the outcome of the interviews together with a discussion of features that did not emerge during the interviews.

Chapter 6 presents a series of scenarios and suggestions of practices based on real scenarios.

Chapter 7 summarises the content of the thesis, discusses how the research questions have been answered, limitations and future research.

10

Chapter 2 Literature Review

2.1 Introduction This chapter reviews the relevant literature on localisation, crowdsourcing, and workflows; the three fields that are the focus of this thesis. The localisation section examines the paradigm, the processes involved and their evolution, and identifies crowdsourced translation as a gap in the literature. The section on crowdsourcing defines the concept, uses illustrative examples to further elaborate upon it, and discusses classifications. A critique is offered in the context of localisation and the manner in which some current practices minimise some of the liabilities referred to in the literature. The following section introduces workflows and the role of process models in this thesis. Support for workflows in the localisation process is reviewed as well as models for crowdsourced translation processes. Finally, a review is presented of existing guidelines proposed to achieve good quality process models and their influence in the development of the models in Chapter 4 is discussed.

2.2 Localisation This section presents the results of a literature review for localisation in general. This

includes the localisation process, aspects of which have been affected by crowdsourcing and the practices in this thesis. When one of those aspects of the process is discussed, the connection to the relevant practice is pointed out. The literature included was identified via a search for localisation translation in Google Scholar and the candidate list of articles was refined by analysing the abstracts to determine relevance. If a work was added to the selection, both the works in its bibliography and the works that cited it where identified and submitted to the same selection process. Furthermore, the researcher included papers and reports that he became aware of through conferences and industry meetings, some of which were not indexed by Google Scholar at the time.

As observed in Chapter 1, localisation is a consequence of the phenomenon of globalisation, the increased flow of ideas, money and people across national boundaries (Hurrell and Woods 1995). In the 1980s this process led to multinational corporations starting their localisation efforts with internal teams in order to expand their reach in foreign markets (Schler 2008a). As the foreign markets grew in importance and, as a result, the need for

11

localisation increased, specialized vendors known as Language Service Provides (LSP) appeared (Esselink 2000).

Within the localisation industry, there is also a specialised meaning for globalisation, that is the action of addressing the business issues associated with taking a product global and includes considerations regarding the integration of localisation across a company, product design, marketing sales and support (Lommel 2003) and having international websites (Esselink 2000). One of the aspects of the globalization of a product or service that is of great importance to localisation is internationalisation.

Internationalisation, often referred to as i18n, has been described as the process of generalizing a product so that it can handle multiple languages and cultural conventions without the need for redesign (Lommel 2003), which will ensure that the product is functional and accepted in international markets and localizable (Esselink 2000). Pym (2004) noted that some authors use the term globalization to refer to internationalisation, and cites Brooks (2000) as an example of author that uses globalization when talking about internationalisation. That DiFranco (2006) still uses the term globalization to refer to internationalisation shows that the distinction between both was not clear until recently.

The internationalisation process is carried out during the development of the product (Esselink 2000) and beyond the technical aspects such as separation of text and code, it has been argued that it is attained by taking culture specific features out of the object to be localised (Pym 2004). Internationalisation practices include:

The separation of text and code, to avoid translators having to deal with source code (Esselink 2000).

The usage of abstractions for dates and times that can then formatted according to the locale (Pym 2004).

The usage of style guides to create controlled languages, or writing for a global audience (Esselink 2000; Pym 2004).

Preparation of a monolingual glossary for the project (Pym 2004). The usage of a text encoding that supports international characters (Esselink 2000;

Pym 2004)

Poor or no internationalisation contributes to increased cost of localisation (Sprung and Jaroniec 2000), resulting in that better internationalised projects are cheaper to localise

12

(Schler 2008a). Furthermore, following internationalisation best practices can lead to localisation becoming focused only on the translation process (Schmitt 1999). This observation of translation being the focus of a localisation process that follows best practice is one of the justifications for this thesis focusing on crowdsourced translation processes instead of crowdsourced localisation processes. Furthermore, the practices discussed in this thesis are applicable in translation processes that do not have to be part of a localisation effort.

Localisation was defined by LISA (Localisation Industry Standards Association) as "the process of modifying products or services to account for differences in distinct markets" (Lommel 2003). Pym (2004) defined it as the adaptation and translation of a text (like a software program) to suit a particular reception situation which is referred as a locale, that is a group of coinciding linguistic and cultural options including language, currencies, date formats, number separators, sorting order and more. To illustrate this, Microsoft windows has been localized to 20 varieties of Spanish, that is one language and 20 locales (Pym 2004). In an ideal case, localisation will make a product seem like it has been developed in the local market. However, this is a challenging goal given that it is a complex process involving many processes such as "project management, engineering, quality assurance, and human-computer interface design issues" (Schler 2008b). Furthermore a complete localisation effort includes the provision of services and technologies for the management of multilingualism across the digital global information flow (Schler 2008a). It has also been noted that the core characteristic of localisation is its relation to digital content (ibid.) that requires devices that are able to interpret it in order to represent it (Ryan et al. 2009). Localisation also is often discussed in relation to translation, in this context the bigger focus of localisation on tools and technologies stands out as a significant difference (Esselink 2000).

Although translation within the localisation industry is seen mainly as the replacement of original strings in a language with strings in the target language, Pym (2004) is critical of this attitude because it addresses translation as a linguistic task instead of a communicative task. Dunne (2011), who approaches translation from a pragmatic point of view that strongly considers the impact of translation in the success of a product, also considers that approaching translation from a linguistic point of view in localisation results in poor quality.

13

2.2.1 Criticism of Localisation

One of the criticisms raised against localisation is that it serves rich countries since the decisions regarding the markets for which localisation is carried out are based on the GDP, something that results in many products being localised into Danish with five million speaker and much fewer localised into Bengali with a hundred million speakers (Schler 2008a). Crowdsourcing has occasionally been used to alleviate this unbalance when it has been used to provide translations for underserved languages (Munro 2010; Scannel 2012).

It has been observed that 90% of localisation is done by American corporations to extend their reach to other markets (Collins 2002) and a part of a trend away from native language content that should at least concern people in the localisation industry (ibid.). However, localisation and the work required to support it: encoding, fonts, spell checkers, etc. (Hall and Schler 2005) has also enabled the creation of local content in languages that are not of interest for multinational companies, creating room for these languages to have a digital presence that will contribute to their conservation (Schler 2008b). An example of a community taking advantage of these localisation enabling technologies can be seen in the work of communities that have created extensions in order to be able to use Facebook in languages that are not supported by the company (Scannel 2012).

Pym (2004) also observed that localised products by aiming to betray no provenance eliminate otherness and in long term do not contribute to the enrichment of intercultural communication. Considering research carried out by Jimnez-Crespo (2013) this risk of eliminating the otherness is specially real with crowdsourced translations because they come closer to texts originally written in the target language than translations done using the TEP approach.

2.2.2 The localisation process

Pym (2004) and Schler (2008a) refer to Esselink (2000) when they describe the localisation process, the differences in how they describe the process are mainly in granularity as visible in Table 1. The blank cells in the table indicate:

14

A stage that is not discussed. For example Esselink includes in the process stages like Pre-sales and kick off meetings, while Pym and Schler start at the analysis stage and do not discuss previous related tasks.

A stage that is considered within a bigger, umbrella stage. For example, Schler talks only about a translation stage, without addressing its substages, while Esselink and Pym break it down to translation of the software and translation of the help and the documentation.

In spite of the changes in technology, more current references that do not cite Esselink as the source of the project defined, describe processes that are almost the same. Three examples of this are the process analysed by Carla DiFranco (2006) to evaluate cost saving opportunities; the project for which Zouncourides-Lull (2011) presents a work breakdown structure and the process described by Yahaya (2008) when discussing the management of geographically distributed teams working for the UN. Given that the stages of the process have remained constant in a span of over ten years, the changes that have happened to the process during that time must have happened within those stages, ie: all the processes described include a translation stage, but the methods adopted in the translation stage around 2000 are different from those used in 2008. As an example, in 2000 the translation would not have been undertaken using online tools, something that was possible in 2008.

The practices that were identified in this thesis have different impacts across the stages. In order to keep a baseline understanding of what those stages entail, a definition for each of them together with the practices that affect them follows and Table 2 offers an overview of the new practices affecting each stage.

-Analysis: The original material is analysed in order to find out how amenable to localisation it is, if there are changes required for specific markets, if the software supports the right encoding, if all the necessary materials (strings, animations,etc) are available to the localizers, what tools will be needed for the project, what is the effort taking on account word count, images, UI design issues. This stage may include pseudo translation, the process of replacing original strings with strings that have the traits (special characters, script, extension) of target languages, which can be very useful to identify internationalisation issues (Schler 2008a).

At this stage, an organisation planning to use crowdsourced translation will have to select the contents that are suitable for the types of crowdsourcing that they can use. This would mean

15

applying the Content Selection practice discussed in chapters four and five of this thesis or the similar Identify Compatible Content pattern suggested by Dsilets (2011a).

Table 1 Comparison of the localisation process as described by Esselink (2000), Pym (2004) and Schler (2008a)

Author

Stage n. Esselink (2000) Pym (2004) Schler (2008a)

1

Pre-sales (sending and receiving Requests for

Quotations) - -

2

Kick-off meeting

(introduction of the team to the project)

- -

3 Analysis of source

material

Analysis of received

Material Analysis

4 Scheduling and budgeting Scheduling and Budgeting -

5 Terminology setup Glossary Translation or

Terminology Setup -

6 Preparation of source

material

Preparation of Localisation

Kit Preparation

7 Translation of software Translation of the software Translation

8 Translation of online help

and documentation Translation of help and

documentation -

9 - Processing Updates -

10 Engineering and testing of

software Testing of Software Engineering/Testing

11 Screen Captures - -

12 Help engineering and DTP

of documentation

Testing of Help and

Publishing of Documentation

-

13 Processing updates - -

16

14 Product QA and delivery Product QA and Delivery -

15 Project closure Post-mortem with Client Project Review

-Preparation: The localisation kit contains the materials necessary to localize the product and can include: the material, tools, style guides, fonts, protocols for reporting issues, contact details, translations memories, terminology

Guidelines for translation are used by Facebook (Losse 2008; Lenihan 2014), but they are not discussed in depth in this thesis. Terminology and TM are used in several of the platforms discussed in Chapter 4 and special considerations regarding these in the context of crowdsourced translation are discussed in Chapter 5.

-Translation: This stage is carried out by translators, but does not only consist of translating strings. The digital nature of localisation means that during the translation stage translators may have to carry out file management and may use Computer Assisted Translation (CAT) tools that can leverage Machine Translation (MT) systems, Translation Memories (TM) and Terminology Databases (TDB). Although there are localisation tools like Alchemy and Passolo (Reynolds 2009) that allow the translators to see the translations in context, it is common to work with strings out of context.

As stated before, this thesis will focus on how crowdsourcing has affected this particular stage. The Super Iterative Translation the practice letting contributors edit the existing translations in order to improve them or make the language more up to date, Open Alternative Translations, Hidden Alternative Translations the practice of collecting multiple translations for a single source TU without letting contributors see the translations suggested by others, and Translation without Redundancy the practice of collecting a single translation for each TU all belong to this stage.

-Testing of Software: Although digital content undergoes testing in its original language, the localisation process can introduce new issues, some affect only the appearance like clipped strings because of text expansion or special characters not being correctly represented, others are functional, like keyboard shortcuts no longer working or having been mapped to the wrong keys. Determining these issues requires the creation of a localisation specific test plan.

It could be argued that with many systems that use crowdsourcing, the content is in a

17

perpetual beta state (Kazman and Chen 2009) and that in this state, the testing of the localised versions is always crowdsourced. Some aspects of the Super Iterative Translation practice that is discussed in this thesis and the similar Publish then Revise pattern (Dsilets 2011b) have an impact on this and the product QA or Engineering stage.

-Product QA or Engineering: When tests raise issues, localisation engineers are then charged with solving them. Freeze the practice of preventing the changes in the translation of a TU, be this editions or additions of new alternative translations, Open Assessment the practice of openly collecting assessments for the different translations for a TU; often done in the form of votes, Hidden Assessmentthe practice of collecting assessments for the different translations for a TU without letting contributors see how others have voted, Expert Selection and Edition the practice of having an expert, often a professional translation or a vetted member of the community, select and edit the translations that will be published and the different approaches to Metadata Based Selection the practice of using metadata as a guide to let a computer automatically select the translation that will be published are QA processes. Super Iterative Translation and Version Rollback the practice of reverting changes in order to solve a quality loss caused by those changes help solving the issues raised during linguistic testing and also fit in this stage.

-Delivery: The localized files are sent to the client so that a localized build can be produced.

Some of the crowdsourced translation platforms analysed in this thesis allow for the download of translated files that can be then used outside the platform. But platforms like Amara and Facebook display the translations directly online without the inclusion of a stage that works like the traditional deliver stage. This is again linked to the Super Iterative Translation practice, where delivery happens automatically any time a translation is updated.

Table 2 Localisation process stages and related practices

Stage New practices directly

affecting the stage

Analysis Content Selection

Translation Super Iterative Translation, Open Alternative Translations,

18

Hidden Alternative Translation

Testing Super Iterative Translation

Product QA or Engineering

Open Assessment, Hidden Assessment, Expert Selection and Edition, Metadata Based Selection Freeze and Rollback

Delivery Super Iterative Translation

2.2.3 Localisation technologies

One of the core features of localisation is the fact that it deals with digital objects (Ryan et al. 2009) and as a consequence of this, a series of specific tools and technologies have been developed responding to the needs of practitioners. A comprehensive list of tools used in localisation is out of scope, since it would have to include general tools and technologies that are not exclusive of the field, such as spell checkers, word processors, email, ftp clients, general electronic dictionaries, email, etc, but a list of those that are most relevant to this thesis follows.

2.2.2.1 Translation Memory (TM) tools: A translation memory is a database of existing translations stored as bitext (bilingual texts) that are divided in segments, usually sentences, that can be automatically queried by CAT (Computer Assisted Translation) tools that are also referred to as Translation Environment Tools (TEnT). When confronted with a new sentence in a text in the original language, CAT tools retrieve translations by using a matching algorithm and present them to the translator. In the case where a match does not exist, once the sentence is translated, the new sentence pair is stored in the TM (Bowker 2002; Somers 2003).

In the context of crowdsourced translations, TM is used in several of the platforms analysed, like Pootle, Launchpad and Crowdin. Leveraging TM has therefore been added to the list of practices. Besides leveraging previously existing translations, industrial CAT tools offer other advanced features that are often not available in crowdsourcing platforms, at least in the cases those analysed in Chapter 4. Some of the features of industrial CAT tools are listed below. The feature lest was collected from Esselink (2000) and Somers (2003):

Concordance search: Search for one term within a whole TM.

19

Filters: Tools that convert file formats that the CAT tool cannot manipulate natively

in to formats that can be manipulated by the CAT tool.

Customisable segmentation: There is a higher chance of finding matches if the segmentations rules used by the CAT tool are the same that were used for the creation of the TM.

Alignment: In cases where there is not a TM but there are previously existing translations, the existing original text and translation can be aligned to create a TM.

Document statistics: Compares a text to the TM and produces a report on matches, which can be used to estimate budget and deadlines.

Machine translation (MT) system or interface: Obtains an automatically generated translation if there is not a TM match.

Project management module: Simplifies some project management tasks like reporting to the client or Key Performance Index.

Quality control module: Can include checkers for spelling, grammar, style guide compliance, terminology compliance and completeness.

Terminology tools: See below.

Term extractor: Analyses the text and suggest term candidates.

The literature indicates that the use of CAT tools results in shorter time for translation, reduced costs and increased consistency in the language (Esselink 2000; Webb 2000; Somers 2003; Brki et al 2009; Reynolds 2009; Rinsche and Portera-Zanotti 2009). Bowkers (2005) article about pilot study carried out to determine the impact of CAT tools noticed that other researchers have seen increases of productivity between 10-70% with 30% being a reasonable expectation. Yamada (2011) also saw increases in productivity, but there were also decreases in productivity when the translations stored in the TM did not stay close to the source. According to Esselink (2000) CAT tools are popular in localisation because software is updated regularly and most of the text from one release will match the text from previous releases and because software documentation tends to be repetitive, which means that the potential for leveraging previously existing translation is very high.

There are also disadvantages to the use of TM, like the lack of visual context, file management overhead (Esselink 2000), issues of TM ownership, existence of a learning curve, potential lower remuneration for translators (Bowker 2002), and the spread of mistakes that are contained in the TM (Bowker 2005; Moorkens 2011). The issue of the lack of visual context has been addressed by the tools that Reynolds (2009) calls localisation resource

20

editors. Those tools can work with binaries and represent the visual context (Esselink 2000; Fernndez Costales 2010).

Chapter 5 presents an evaluation of this technology in the context of crowdsourced translation using a qualitative technique, and discusses how the advantages and downsides of TM affect work in crowdsourcing scenarios.

2.2.3.2 Machine Translation

Machine translation has been defined as a methodology and technology used to automate language translations from one human language to another, using terminology, glossaries and advanced grammatical, syntactic, and semantic analysis techniques (Esselink 2000) or more simply, a software that takes inputs as sentences in one natural language and outputs the corresponding sentences in another natural language. It has been said that TM is used to support human translators and MT aims to replace them (Esselink 2000; Levitina 2011). It is common for lower quality solutions to present themselves as alternatives to human translators

while higher quality systems present themselves as productivity tools (Esselink 2000).

As noted above, some TM systems include an MT system and others retrieve MT translations from web services (ibid.).

By 2002 there was very limited usage of MT (DePalma 2006) and even by 2009 MT was not widely implemented by translation agencies (Rinsche and Portera-Zanotti 2009). This lack of penetration may be caused by pure MT output having been proven to be only practical when used to translate very controlled input (Esselink 2000). However, as of 2007 and depending on the language only between 3% and 30% of Microsoft knowledge base had been translated by humans. If an article had not been translated, an MT version was available to the users and these translations were generally welcomed (Levitina 2011). This signals that MT has reached a state where its unedited output can be suitable in contexts.

Within the localisation industry MT is mostly used combined with TM and post edition by human translators, with the usual process being the leverage of the TM, application of MT to the segments without matches or with low matches and post edition by human (ibid.).

2.2.3.2.1 Types of MT

There are three commonly used types of MT: rule based, corpus based and Hybrid.

21

Rule based MT: Use dictionaries and syntactic rules to translate the content. Since these rules change between languages new rules must be created for each language pair. Its output is usually grammatically correct, but often requires a lot of post editing when it is used without customer specific glossaries (Levitina 2011). These systems were dominant in the 1980s (Hutchins 2005).

Corpus based MT: The two main corpus based MT approaches are statistical MT and Example Based MT (EBMT). Through the analysis of bilingual corpora the system creates statistical models of equivalences that are used to translate. Thanks to the large collections of text available in the internet and the increase in computing power these techniques have become more viable in recent years (Levitina 2011).

The core difference between both approaches is the way in which their models are created; while EBMT searches for analogous examples to create the translation, statistical MT uses

statistical correlations. Statistical MT systems can be improved using corrective feedback, this has been done for example by Asia Online (Baer and Moreno 2009; Vashee 2009), one of the organisations whose process for a crowdsourced translation project is analysed in Chapter 4.

Hybrid MT: These systems either apply rules to a statistical engine or apply statistics to correct the output of a rule based model.

There is also a fourth approach, called interlingua that uses an abstract language model as a proxy between languages so that it is only necessary to create a conversion from and to that language (Hiroshi and Meiying 1993), but there is no evidence in the literature of this approach ever having been used in the localisation industry.

Several of the platforms and projects reviewed for this thesis use MT and as a result MT leverage was added to the list of practices. The advantages of MT faster throughput, no untranslated strings and potentially higher consistency if the system has been trained with consistent TMs still apply, but a new set of issues linked to crowdsourcing appears and these be discussed in Chapter 5.

2.2.3.3 Terminology tools

22

Identifying equivalents for specialized terms is an important part of every localisation project. These terms may come from the field of knowledge to which the content belongs or be dictated by the client like in the case of trademarks (Bowker 2002; Muegge 2007). As localisation projects grow, the need for a unified terminology that can be used by different translators becomes apparent (Rinsche and Portera-Zanotti 2009). Having a terminology setup does not only help the quality of language, reduction of cost and increase the efficiency in the localisation project, but also during the development and writing stages (Muegge 2007; Rinsche and Portera-Zanotti 2009; Straub and Schmitz 2010).

The first step in setting up a terminology is the term extraction (Bowker 2002), this can be done manually or with automated tools. Automated term extraction tools can use linguistic and statistical methods. The linguistic method identifies certain part-of-speech patterns, like noun+noun, while the statistical one looks for repeated sequences (Rinsche and Portera-Zanotti 2009). Term extractors can be standalone or part of a CAT tool.

Once the terms have been extracted, these must be stored (Bowker 2002). Although some organisation store their terminology in general purpose databases, spread sheets, websites and text documents, in the long term this causes issues, such as low efficiency and interoperability problems (Muegge 2007; Schmitz 2001 cited by Bowker 2002). Besides the standalone terminology tools, CAT tools and TMS can also integrate a terminology management module (Bowker 2002; Rinsche and Portera-Zanotti 2009). These tools store the terms in a diversity of data models, from custom databases to the XML based TermBase eXchange standard (Melby 2008). The retrieval of terms can be done via manual query of the database or automatically by an integrated tool.

Again, terminology related technology is used in the context of crowdsourcing. Facebook for example has two stages in their translation, first the community translates the most important

terminology and then the rest of the site (Losse 2008; Mesipuu 2010; Lenihan 2014). Besides the increased consistency that is derived through the use of terminology tools, the experts interviewed for Chapter 5 noticed other benefits that are discussed in Chapter 5.

2.2.3.4 Translation Management Systems Translation Management Systems (TMS), also referred to as Globalization Management Systems, are sever based tools that partially automate the orchestration of business functions, project tasks, process workflows and language technologies characteristic of large translation

23

projects (Sargent and DePalma 2008; Reynolds 2009; Rinsche and Portera-Zanotti 2009; Freij 2010). This automation helps to reduce the overhead caused by working with global teams. These tools can include features such as integration with content and documentation management systems, TM storage, an online translation environment, a quoting and invoicing module, a workflow system and a terminology management module (Reynolds 2009; Levitina 2011).

It has been observed that TMSs are difficult to implement and expensive, and that most agencies probably need a simpler more agile solution (Reynolds 2009). Since these systems have been developed to respond to the needs of big vendors and publishers, they do not cater for the needs of most organisations that intend to use some types of crowdsourcing (Morera et al. 2012). However, it could be argued that the systems such as Pootle or Crowdin that are used for crowdsourced translation and discussed in Chapter 4 of this thesis are precisely specialized TMS. Although they do not have quoting and invoicing modules, and workflow systems, they do integrate many of the translation tools discussed before and automate much of the process.

2.2.4 Localisation Levels

Several authors have proposed or agreed with others proposals on the levels of localisation (Carey 1998; Brooks 2000; Pym 2004; Thayer and Kolko 2004; Zhou 2011). Pym (2004) quotes the levels proposed by Brooks that are:

1) Complete Localisation or adapted, which entails content and examples from the new locale

2) Partial Localisation, which entails only part of the product being localized; for example, the UI, but not the documentation.

3) Enabled software, which entails the software being able to handle new locales (by supporting encoding and fonts).

Zhou deals specifically with video games and embraces the levels proposed by Thayer and Kolko (2004):

1) Basic: Original GUI and icons and only the text is translated.

24

2) Complex: The GUI and icons have been adapted.

3) Blending: Story and graphics are adapted.

There are some similarities between the different rankings of levels of localisations and Thayer and Kolkos (2004) observation that Theres no plot behind Microsoft Word indicates the main reason for the differences between them. If we assume that the documentation has been localised, their Complex and Blending levels would meet the Complete level in Brooks ranking and their basic level would match Brooks partial localisation level.

Careys (1998) ranking depicted in Table 3 contains seven different levels including a level of no adaptation at the lowest end. Levels two and four would fit into Brooks partial localisation and levels five to seven would fit into Brooks complete localisation level. Of these three models of levels of localisation, this thesis favours Brooks because of its being applicable to content in general, without requiring a narrative like in Thayer and Kolko (2004) and because all levels are clear, unlike in Careys (1998) that contains an ambiguous stage enable code that is not explained in their paper.

Table 3 Levels of localisation adapted from Carey (1998)

Careys levels of localisation

One No localisation effort made.

Two Translate documentation and packaging only.

Three Enable code.

Four Translate software menus and dialogs.

Five Translate online help, tutorials, and sample and readme files.

Six Add support for locale-specific hardware.

Seven Customize features for locale.

25

Independently of the ranking system used to measure the levels of localisation, Carey, Brook and Zhou agree that the level of localisation is chosen according to the expected return on investment (ROI). Pym (2005) adds that tolerance to English in certain niches, such as software development, and tolerance to English in certain locales, such as French speaking countries, is also a factor that affects the level of localisation of a product.

In the context of the projects and platforms analysed in this thesis, content translated via crowdsourcing would generally fit Thayer and Kolkos (2004) basic level and Brooks (2000) third localisation level, i.e. complete localisation; but it is worth noticing that, at least when it comes to the localisation of a social network platform, crowdsourced translation may feel more like text originally written in the target language than like a translation (Jimenez-Crespo 2013). From a cultural point of view, this is indicative of a more successful localisation effort if the purpose is that of conveying the impression of a locally created product.

2.2.5 Summary of the literature review for localisation

In this section we have discussed the concept of localisation, the enterprise process which includes translation that is the aspect of localisation on which this thesis focuses, the tools involved in the process, the different localisation levels and how they all relate to the practices in the crowdsourced translation paradigm.

2.3 Crowdsourcing This section presents the results of a literature review for crowdsourcing in general and specifically in localisation. The papers included were identified via a search for crowdsourcing, localisation and translation in Google Scholar. Relevant candidates where selected by title and the selection was further refined after reading their summaries. If a paper was added to the selection, both the papers in its bibliography and the papers that cited it were identified and submitted to the same selection process. Furthermore, the researcher included papers and reports that he became aware of through conferences and industry meetings, some of which were not indexed by Google Scholar at the time.

2.3.1 Introduction to Crowdsourcing

The term crowdsourcing is subject to multiple interpretations because of its being a relatively new concept that is still evolving. This is visible in how much of the literature is dedicated to conceptualize it (Zhao and Zhu 2012), how different authors use various names for it, such as

26

peer production, user-powered systems, community systems and mass collaboration among others (Doan et al. 2011) and how other authors discuss it without defining it (Alonso et al. 2008; Kittur et al. 2008; Huberman et al. 2009).

Table 4 Google Scholar hits per year for crowdsourcing and some crowdsourcing related terms

PLATFORM

YEAR Facebook twitter Wikipedia Open Source YouTube Amazon

Mechanical Turk

Ebay Digg Innocentive

2006 93 86 29 22 45 4 12 11 7 2007 257 210 145 127 252 8 62 59 45 2008 364 260 248 236 357 20 90 100 66 2009 533 376 461 411 448 55 130 118 99 2010 965 869 868 771 702 213 225 175 177 2011 1590 1450 1300 1170 925 418 288 207 261 2012 2400 2018 1750 1540 1380 655 315 252 296

The popularity of crowdsourcing as a research topic has continually increased since the publication of Howes seminal article and that is illustrated by Table 4 that shows the number of hits per year that Google Scholar produces for eigh of the most popular platforms brought up as examples of crowdsourcing and the open source approach to software development that is also frequently discussed as a type of crowdsourcing. The table is the result of searching for the keywords intext:[[PlatformName]] intext:crowdsourcing, which enforces both keywords to appear in the text of the documents retrieved. Although Google searches are not trustworthy data for information critical research, the table illustrates the rise in popularity of crowdsourcing and several platforms as a research topic.

2.3.1.1 A Definition of Crowdsourcing As stated before, much research on crowdsourcing is dedicated to conceptualize it and part of the conceptualization is defining it. The most popular definition for crowdsourcing is the one given by Howe in 2006 in an article for Wired magazine (Howe 2006b) that refers to it as the practice of leveraging communities to carry out tasks that were traditionally carried out by professionals. Across the literature, this definition has been paraphrased (Brabham 2008a; Kleemann et al. 2008; Oprea et al. 2009; Horton and Chilton 2010), and modified it slightly, for example The use of an Internet-scale community to outsource a task (Yang et al. 2008).

27

However, in contrast with these general definitions, other authors have focused on specific aspects of the tasks for their definitions. For example Heer and Bostoc (2010) focused on the size of the task and defined crowdsourcing as a process where web workers complete one or more small tasks, often for micro-payments on the order of $0.01 to $0.10 per task.; Brabhams (2008b) definition, which according to Estells-Arolas and Gonzlez-Ladrn-de-Guevara (2012) is part of the most cited academic paper about crowdsourcing, focuses on the problem solving aspect of crowdsourcing, by stating that crowdsourcing is an online, distributed problem-solving and production model. The impact of this definition can be seen in authors such as Doan et al (2011), from whose paper the following definition can be collated: crowdsourcing is a general-purpose problem-solving method that enlists a crowd of humans to help solve a given problem.

That crowdsourcing is a general purpose practice is underlined by the diversity of companies that have used it or are using it and the fields in which their activities fit. In his article Howe (2006) talks about how crowdsourcing was being used in fields such as photography (iStockPhoto), software development (open source software), video (Web Junk 20), problem solving (InnoCentive), knowledge collection (Wikipedia) and micro-task execution (Amazons mechanical turk). It has also been used in data analysis (Humangrid), map creation (OpenStreetMap), to complement Optical Character Recognition (OCR) (ReCaptcha), in design (Wilogo) (Schenk and Guittard 2009) and funding (Kickstarter) (Howe 2008).

However, if problem solving is understood as the process of finding solutions, or as defined by DZurilla and Goldfried (1971) a process that makes available effective responses to deal with problematic situations and increases the probability of selecting the most effective response to those situations; crowdfunding would not fit Brabhams definition. This is because in crowdfunding the problem is always insufficient funding and the solution is always more funding. In this context the crowd is not finding a solution, but enabling it.

With the intention to characterize crowdsourcing in order to define it, it has been argued that its core characteristic is the open call, meaning that it is not limited to preselected candidates (Howe 2006b; Schenk and Guittard 2009). However, there are organisations like Kiva, who use volunteers that have undergone training, or IBM, who used their employees (DePalma and Kelly 2011), that refer to what they do as crowdsourcing. In fact, Geiger et al (Geiger,

28

Seedorf, et al. 2011) turned this preselection (or lack thereof) of the contributors into one of the dimensions of their taxonomy.

Schenk and Guittard (2009) did, however, find three elements that always appear in crowdsourcing independently of the type activity, those three elements are the first explicit collection of characteristics of crowdsourcing found in this literature review and they are collaborators, who carry out the tasks; organisations, that benefit from the activity of the collaborator, and platforms that enable the collaborators. Later, Estells-Arolas and Gonzlez-Ladrn-de-Guevara (2012) identified eight defining characteristics of crowdsourcing that must be addressed in a general definition for crowdsourcing. To identify them, they collected forty definitions from 209 papers from different fields. The characteristics are as follows:

1) The people who form the crowd. 2) What the people have to do. 3) What the people get from the process. 4) The init