perceptions of ielts in cambodia: a case study of test

!

IELTS Research Reports Volume 13 © www.ielts.org 1

Perceptions of IELTS in Cambodia: A case study of test impact in a small developing country Authors Stephen Moore, Macquarie University, Australia Richmond Stroupe, Soka University, Japan Paul Mahony, IDP Cambodia

Grant awarded Round 15, 2009

This study aims to evaluate the impact of the IELTS Test on Cambodian education and society by means of a critical language testing case study. It finds that the Test is the major gate-keeping mechanism for Cambodians to access overseas scholarships and study at English-medium institutions, particularly in postgraduate programs.

Click here to read the Introduction to this volume which includes an appraisal of this research, its context and impact.

ABSTRACT This study investigated the impact of the IELTS Test across different stakeholders in contemporary Cambodia. The major stakeholders included test-takers, examiners, learners, teachers, parents/guardians of test-takers, and local users of test results (eg, scholarship officers and employers).

The study followed a mixed-methods approach in collecting data (including multiple methods and multiple sources) within an interpretive research paradigm (ie, in which qualitative approaches predominate). The overall project has been framed as a critical language testing (CLT) case study.

The research was guided by 15 specific research questions. Among the main findings were:

1. the IELTS Test is the major gate-keeping mechanism for Cambodians to access the majority of scholarships available to them for overseas study

2. the IELTS Test is generally trusted as an accurate measurement tool for academic English proficiency

3. several different stakeholder groups indicated that IELTS was too Eurocentric and that Cambodian candidates had to acquire more ‘world knowledge’ in addition to developing their English language proficiency skills to achieve a satisfactory band score

4. the IELTS Test was widely considered to be expensive for Cambodian test-takers.

Overall, the IELTS Test in Cambodia was found to have a minor impact on education and society in Cambodia in general terms, but a significant impact on facilitating opportunities for Cambodians to pursue higher education overseas at English-medium institutions (especially for postgraduate studies). The full impact of IELTS’s role in contributing to the decisions as to which Cambodians can (and cannot) pursue overseas study is ultimately unknowable but will, nevertheless, play out over the coming decades.

http://www.ielts.org/PDF/vol13_introduction.pdf

Stephen Moore, Richmond Stroupe and Paul Mahony


AUTHOR BIODATA STEPHEN MOORE Stephen Moore is a Senior Lecturer in the Department of Linguistics at Macquarie University in Australia where he convenes the postgraduate program in applied linguistics. His teaching, supervision, research and publications extend across the fields of discourse analysis, language for specific purposes, TESOL teacher education and language assessment. His publications include articles in Discourse & Society, Text & Talk, Critical Discourse Studies, World Englishes, TESOL Quarterly, English for Specific Purposes, and Prospect. Stephen taught in Phnom Penh in 1994–5 and from 1998 to 2000, and he has maintained strong links to ELT in Cambodia since then, principally through research, including supervision of Cambodian ELT researchers, and editorial board memberships. He was an IELTS examiner from 1998 to 2008.

RICHMOND STROUPE Richmond Stroupe has worked with university and professional language-learners from Asia since 1989. He is currently a Professor and Program Chair for the Master in International Language Education: TESOL at Soka University, Tokyo, Japan. He is active in a number of professional organisations, including the Japan Association for Language Teaching (JALT), where he is the Chair of the International Affairs Committee, and TESOL International (United States), where he is the Chair of the Standing Committee on Standards. Regionally, Richard continues to be an international mentor for the CamTESOL Conference series, and is also editor-in-chief of the related academic publication, Language Education in Asia, supported by IDP Education. He regularly presents and publishes scholarly work related to his academic interests, including curriculum development, teacher training, and international comparative education.

PAUL MAHONY Paul Mahony has been Country Director for IDP Education in Cambodia since 1995. He was the IELTS Administrator for Cambodia from 1995 to 1997. Paul has focused on improving standards in IDP’s school in Cambodia through empirical decision-making focused on an active testing regime, including the IELTS Test. He introduced IELTS as a capstone test for the General English Program at IDP’s school in 2006. He was the founder and has been the annual Convenor of the CamTESOL Conference series since 2005.

IELTS RESEARCH REPORTS VOLUME 13, 2012 Published by: IDP: IELTS Australia and British Council Editor: Jenny Osborne, IDP: IELTS Australia Editorial consultant: Petronella McGovern, IDP: IELTS Australia Acknowledgements: Dr Lynda Taylor, University of Cambridge ESOL Examinations IDP: IELTS Australia Pty Limited British Council ABN 84 008 664 766 Bridgewater House Level 8, 535 Bourke St 58 Whitworth St Melbourne VIC 3000, Australia Manchester, M1 6BB, UK Email [email protected] Email [email protected] Web www.ielts.org Web www.ielts.org © IDP: IELTS Australia Pty Limited 2012 © British Council 2012 This publication is copyright. Apart from any fair dealing for the purposes of: private study, research, criticism or review, as permitted under the Copyright Act, no part may be reproduced or copied in any form or by any means (graphic, electronic or mechanical, including recording, taping or information retrieval systems) by any process without the written permission of the publishers. Enquiries should be made to the publisher. The research and opinions expressed in this volume are of individual researchers and do not represent the views of IDP: IELTS Australia Pty Limited. The publishers do not accept responsibility for any of the claims made in the research. National Library of Australia, cataloguing-in-publication data. 2012 edition, IELTS Research Reports Volume 13. ISBN: 978-0-9872378-1-1

Perceptions of IELTS in Cambodia: A case study of test impact in a small developing country


REPORT 6: CONTENTS

!

1 Introduction ................................................................................................................................... 5 1.1 Aims......................................................................................................................................... 5 1.2 Context .................................................................................................................................... 5 1.3 Rationale.................................................................................................................................. 5

2 Brief review of the literature......................................................................................................... 6

3 Research questions ...................................................................................................................... 8

4 Methodology .................................................................................................................................. 9 4.1 Overview.................................................................................................................................. 9 4.2 Synopsis of data collection and analysis ................................................................................. 9 4.3 Data collection instruments (by stakeholder) ......................................................................... 11

4.3.1 Questionnaire for test-takers.......................................................................................... 13 4.3.2 Questionnaire for test preparation course students ....................................................... 13 4.3.3 Questionnaire for test preparation course teachers ....................................................... 14 4.3.4 Focus group of IELTS examiners................................................................................... 14 4.3.5 Focus group of parents/guardians ................................................................................. 14 4.3.6 Interviews of embassy officials....................................................................................... 15 4.3.7 Interviews of employers ................................................................................................. 15 4.3.8 Interviews of returned graduates from overseas............................................................ 15 4.3.9 IELTS preparation course classroom observations ....................................................... 15

5 Results by data collection instrument and stakeholder.......................................................... 16 5.1 Test-taker............................................................................................................................... 16 5.2 Test preparation course student ............................................................................................ 19 5.3 Test preparation course teachers .......................................................................................... 29 5.4 Focus group of IELTS examiners .......................................................................................... 35 5.5 Focus group of parents/guardians ......................................................................................... 37 5.6 Interviews of embassy officials .............................................................................................. 38 5.7 Interviews of employers ......................................................................................................... 39 5.8 Interviews of returned graduates from overseas ................................................................... 40 5.9 IELTS preparation course classroom observations ............................................................... 41

6 Research questions answered................................................................................................... 44 6.1 Research question 1.............................................................................................................. 44 6.2 Research question 2.............................................................................................................. 44 6.3 Research question 3.............................................................................................................. 45 6.4 Research question 4.............................................................................................................. 47 6.5 Research question 5.............................................................................................................. 47 6.6 Research question 6.............................................................................................................. 48 6.7 Research question 7.............................................................................................................. 48 6.8 Research question 8.............................................................................................................. 49 6.9 Research question 9.............................................................................................................. 49 6.10 Research question 10............................................................................................................ 50 6.11 Research question 11............................................................................................................ 51 6.12 Research question 12............................................................................................................ 52 6.13 Research question 13............................................................................................................ 52 6.14 Research question 14............................................................................................................ 53 6.15 Research question 15............................................................................................................ 53



7 Discussion ...................................................................................................................................54 7.1 Part 1: Questionnaire-derived results ....................................................................................54 7.2 Part 2: All results ....................................................................................................................56

7.2.1 Critical perspective characteristic 1................................................................................58 7.2.2 Critical perspective characteristic 2................................................................................59 7.2.3 Critical perspective characteristic 3................................................................................60 7.2.4 Critical perspective characteristic 4................................................................................61

7.3 Conclusion of discussion .......................................................................................................62

8 Summary of key findings............................................................................................................62

9 Conclusions, limitations and further research .........................................................................63 9.1 Conclusions ...........................................................................................................................63 9.2 Limitations..............................................................................................................................63 9.3 Further research ....................................................................................................................64

Acknowledgements.............................................................................................................................64

References...........................................................................................................................................65

List of abbreviations and acronyms..................................................................................................68

Appendix 1: Test-taker (TT) questionnaire .......................................................................................69

Appendix 2: Test preparation (TP) course student questionnaire .................................................77

Appendix 3: IELTS Test preparation course teacher questionnaire ..............................................83

Appendix 4: Materials evaluation questionnaire..............................................................................90

Appendix 5: IELTS examiner focus group discussion points ........................................................99

Appendix 6: Parent/guardians focus group discussion points ....................................................100

Appendix 7: Embassy officials interview questions......................................................................101

Appendix 8: Employers interview questions..................................................................................102

Appendix 9: Returned graduates interview questions ..................................................................103

Appendix 10: Lesson observation protocol ...................................................................................104

Appendix 11: Group statistics: Learning and study habits ..........................................................105

Appendix 12: Group statistics: Test-taking approaches...............................................................108



1 INTRODUCTION English language teaching (ELT) is a booming educational sector in contemporary Cambodia, and the English language is seen by many Cambodians as a ticket to a better job and better educational opportunities both within Cambodia and abroad. Indeed, as noted by Ross (2008, p 6), in the context of language testing in Asia more generally, there is an “association of English language proficiency with conceptions of scholastic and professional merit”. IELTS is the dominant international test used in Cambodia for various decision-making purposes, such as study abroad opportunities and scholarships. However, despite its well-established usage in the Cambodian educational context, there have been no studies of the impact of IELTS in Cambodia.

1.1 Aims The project has several aims, with the broadest being to evaluate the impact of the IELTS Test on Cambodian education and society. To meet this aim, the project was broadly conceptualised as a test impact study within a critical language testing framework (Shohamy, 2001). As a test impact study, the project aims to contribute to the ongoing validation process of IELTS, but from a country-specific perspective. The project aims to focus on establishing the perceptions of major stakeholders such as test-takers, examiners, learners, teachers, parents/guardians of test-takers, and local users of test results (eg, scholarship officers on-site in Cambodia; employers in Cambodia). The project also aims to contribute to the development of a ‘research culture’ in ELT in Cambodia through involving Cambodian nationals as research assistants.

1.2 Context Hawkey’s (2006) global study of IELTS impact has provided a strong foundation on which this IELTS impact study in Cambodia builds. For example, Hawkey’s questionnaires have been slightly modified and/or reused and complemented by other research instruments. The project has been conceived as a case study (Duff, 2008) within which a qualitative-dominant mixed methods approach (Teddlie and Tashakkori, 2009) was proposed as being most suitable to explore the complexity of test impact (Wall and Alderson, 1993; Lynch, 2001), and to capture sufficient and adequate data to answer the various research questions. Where appropriate, new research instruments were created or adapted following best practice principles as provided by Dornyei (2003) and Brown (2001) for questionnaire design; Stewart, Shamdasani and Rook (2007) for focus group discussions; and Richards (2003) for interviews and classroom observations.

Regarding the specific context of Cambodia, the IELTS Test was first established there in 1992 and has been administered since then by IDP Education (Cambodia). There were 11,378 IELTS Tests taken between 1992–2010, with an almost exponential growth rate in registrations each year. Ninety-six percent of IELTS test-takers in Cambodia were Cambodian nationals. There are approximately 10 IELTS examiners responsible for all IELTS testing in Cambodia. Apart from quarterly testing in the northern provincial city of Siem Reap, all other IELTS testing is undertaken at least fortnightly at IDP’s test centre in Phnom Penh (KH001). This IELTS impact study in Cambodia investigated IELTS-related activities (ie, test preparation; use of test results) from February to September 2010.

1.3 Rationale As a country with a relatively small population and a language that is not widely spoken elsewhere, Cambodia is increasingly reliant on the English language development of its human resources as a means of lifting itself out of poverty and engaging with the outside world. IELTS plays a very important gate-keeping role in this context, and a critical evaluation of its impact on various stakeholders is warranted. This study seeks to determine what sort of profile the IELTS Test has



amongst various stakeholders and the broader Cambodian community, what Cambodians actually know about the Test, and how it is regarded in the country. The study also seeks to determine the nature of the IELTS Test operating as a ‘high-stakes’ test in the Cambodian context and what, if any, are the actual consequences of this.

There are other more practical reasons for conducting this particular impact study. For example, because IELTS usage in Cambodia is highly centralised in the capital, Phnom Penh, it is relatively easy to investigate and provide a reasonably comprehensive picture of IELTS perceptions in the country as a whole, and this could be valuable for comparable case studies of the impact of IELTS in other countries. Also, very important to this particular study is the fact that IDP Education (Cambodia) has been fully supportive of this research proposal, and all three principal researchers have longstanding interests in ELT development and research in Cambodia.

2 BRIEF REVIEW OF THE LITERATURE This project is philosophically aligned with critical language testing. In particular, it is concerned with the power of English as an international language (Pennycook ,1994); the power of language tests (McNamara, 2007); the consequences of language tests (Messick, 1989 and 1996); and issues of language test washback and impact (Hamp-Lyons, 1997 and 2000). [Note that in this study we follow the distinction made in Hawkey 2006 (p 8) to use the term washback “to cover influences of language tests or programs on language learners and teachers, language learning and teaching processes (including materials) and outcomes”, and the term impact “to cover influences of language tests and programs on stakeholders beyond language learners, teachers, except when it is the influences of a test or program on learners and teachers outside their learning or teaching roles”.] These scholars all point to the need for language testers to take seriously their responsibility to ensure that language tests do not unfairly impact test-takers and that the voices of other stakeholders, who might also be impacted by the test, are also heard.

In discussing the increased interest in social aspects of language testing, McNamara (1998) notes how Messick (1989) legitimated research on the impact of language tests and the political character of tests (pp 304-5). Elana Shohamy, as McNamara (1998, pp 315-316) also points out, has played a pioneering role in research developments in the field of language test impact, coining the term (in Shohamy 1997) ‘critical language testing’ to “describe [both] a perspective and a research agenda”. Lynch (2001, p 367) elaborates this description by arguing that CLT must “consider…research paradigms beyond the dominant, post-positivist-influenced one”, implying that a mixed-methods approach could provide the most suitable theoretical footing for CLT research. This IELTS impact study in Cambodia essentially employs a mixed-methods approach to formulate a research agenda for providing a perspective on the impact of IELTS in a small developing country. It is thus well placed to offer insights as an original CLT case study.

To understand exactly what ‘CLT’ means in language education and testing contexts, it is worth noting Lynch’s (2001, p 362) perceptive comment that Shohamy’s own use of the word ‘testing’ in CLT is actually synonymous with the broader term ‘assessment’ (ie, it goes beyond post-positivist approaches of psychometrics to include alternative assessments, such as portfolios). That being the case, ‘communicative language assessment’ would seem to be the more appropriate label, but CLT is now a well-established term. One must nevertheless always bear in mind that the term ‘CLT’ is wider than its name suggests. Thus, in the present study, we use the term CLT to embrace the wider notion of any evidence (measured or collected) that informs an evaluation of language proficiency.



Although CLT broke onto the language testing scene with some force in the late 1990s (see, for example, Shohamy, 1997; 2001; McNamara, 1998; and Lynch, 2001), more than a decade later one is hard-pressed to find a published case study that claims to be an example of CLT in practice. In this sense, the present study is possibly breaking new ground. However, conversely, given the lack of published CLT studies available, our study has necessarily been undertaken without the benefit of other ‘worked examples’ to draw from for comparative purposes. In these circumstances, we believe that the framework posited by Lynch 2001, in which Shohamy’s principles of CLT are melded with Pennycook’s notions of critical applied linguistics, is a most suitable mechanism for exploring the true impact of IELTS in Cambodia.

Lynch’s 2001 framework combines ideas about critical applied linguistics drawn from Pennycook (1999; 2001) and ideas about CLT drawn from Shohamy (2001). Lynch (2001, p 357) suggests there are four characteristics for a critical approach to applied linguistics:

1. an interest in particular domains, such as gender, class, ethnicity, and the ways in which language and language-related issues (like all human relations and activities) are interconnected with them

2. the notion that our research needs to consider paradigms beyond the dominant, post-positivist one

3. a concern for changing the human and social world, not just describing it, ie, the ‘transformative agenda’, with the related and motivational concern for social justice and equality

4. the requirement that critical applied linguistics be self-reflexive.

Lynch uses these four characteristics of critical applied linguistics as categories within which to position the 15 principles that Shohamy (2001, p 10) claims underlie CLT. (Lynch’s framework is reproduced in Table 17). To sum up, using a CLT paradigm in the present study is appropriate because the IELTS Test is both a ‘market leader’ in English language proficiency testing and a high-stakes test with the power to make a significant social impact.

Various studies of test washback have been undertaken since Alderson and Wall’s (1993) seminal paper set out a research agenda in this area. For example, Cheng, Watanabe and Curtis (2004) provide a number of washback studies from a variety of contexts around the world, including Saville and Hawkey’s (2004) account of the washback of IELTS on teaching materials. Another notable example is Green (2007), which is an impact study of IELTS and preparation for academic writing. Indeed, there have been many other IELTS-related research projects over the past decade and a half that dealt with some aspect of test impact. These include: Merrifield (2008), which looks into the use of IELTS Test scores by professional associations; Ingram and Bayliss (2007), which investigates predictive validity of IELTS in a university context; O’Loughlin and Arkoudis (2009), which explores the rate and nature of improvement in IELTS Test scores; Coleman, Starfield and Hagan (2003), which deals with stakeholder attitudes towards IELTS; and Mickan and Motteram (2008), which investigates test preparation for the IELTS exam. However, no impact study thus far appears to have dealt exclusively and thoroughly with the IELTS Test’s overall impact in one specific country.

In the case of Cambodia, no published language test impact studies of any kind have explored Cambodia as a site of research. CamTESOL, an IDP-run TESOL conference in Cambodia, has encouraged language testing and assessment research presentations through a dedicated stream at its annual conference, yet most presenters in that stream have talked about non-Cambodian contexts and no Cambodian-specific papers have been published in the selected conference proceedings to date. (Hogan (2009) was a CamTESOL presentation and paper that did centre on the IELTS Test, but not specifically in relation to Cambodia). As Blackhurst (2004) shows, Cambodia does not feature as a major IELTS test-taking population in terms of the number of test-takers, nor does Khmer feature as a



major language of IELTS test-takers. One could say, therefore, that Cambodia does not have a high profile in the context of IELTS; but IELTS does have a high profile in the context of Cambodia. This gap in the literature is where the IELTS impact study in Cambodia fits.

3 RESEARCH QUESTIONS 1. Research Question 1: What impact does IELTS have on education and society in

Cambodia?

2. Research Question 2: What are the profiles of the candidates taking the IELTS test?

3. Research Question 3: What is the washback of the IELTS Test on courses preparing candidates to take it?

4. Research Question 4: What are the profiles of the participants who have already taken the IELTS Test?

5. Research Question 5: What is the impact of IELTS on the participants who have taken the test (including Cambodian graduates returned from overseas study)?

6. Research Question 6: What are the profiles of IELTS examiners in Cambodia?

7. Research Question 7: What is the impact of IELTS on the examiners in Cambodia?

8. Research Question 8: What are the profiles of the teachers preparing candidates to take the IELTS Test?

9. Research Question 9: What is the washback of the IELTS Test on the teachers preparing candidates to take the test?

10. Research Question 10: Which textbooks and other materials are used on IELTS preparation courses?

11. Research Question 11: What is the washback of IELTS on these preparation course materials?

12. Research Question 12: What do IELTS-preparation lesson observations and analyses indicate about the washback of the IELTS Test on the lessons?

13. Research Question 13: What are the profiles of the main local users of IELTS Test results in Cambodia?

14. Research Question 14: What is the impact of the IELTS Test on these users?

15. Research Question 15: What is (or has been) the impact of the IELTS Test on other stakeholders (eg parents of test-takers)?



4 METHODOLOGY 4.1 Overview The IELTS impact study in Cambodia adopted a mixed methods approach. The study was framed as a critical language testing project within an interpretive paradigm involving both qualitative data (ie, interviews, focus group discussions, and classroom observations), and quantitative data (ie, questionnaires). To evaluate the impact of IELTS in Cambodia, a wide range of stakeholders was consulted (per Taylor, 1999, cited in Hawkey, 2006, p 15), including IELTS test-takers; IELTS preparation course learners and teachers; IELTS examiners; parents/guardians of test-takers; embassy staff and staff of foreign agencies that process scholarship awards; local employers; and Cambodian graduates returned from studies abroad. The study was led by two researchers very familiar with the Cambodian context, but not residing in Cambodia during 2010. Stephen Moore was based in Sydney, while Richmond Stroupe was based in Tokyo. IDP Education (Cambodia) provided both administrative and technical support on-site in Cambodia. The local team, led by (Mr) Heang Chan Veasna, assisted with facilitating participant recruitment; administering questionnaires; arranging interviews; providing database support, including the full historical records of IELTS results in Cambodia since 1992; providing support with data analysis of questionnaires (eg, using SPSS software); and translation and interpreting support. Specialist advice on testing, primarily concerning quantitative data analysis and interpretation, was provided by Associate Professor Mehdi Riazi, a language testing expert in the Linguistics Department at Macquarie University.

4.2 Synopsis of data collection and analysis The collection of data for this project was complex, involving the recruitment of a variety of stakeholders and the use of a range of different instruments. While the qualitative data was collectable during two relatively short periods mid-year, the quantitative data required ongoing collection for an extended period (from March to September 2010) to ensure as many test-taking participants as possible. Data analysis and interpretation for this project began in June 2010 and continued through December. The final report was written in January and February 2011.

At the proposal stage, the project’s documentation set out certain methods, participant numbers, and a timeline. After the project had been accepted for funding by IELTS Australia, some small changes to methods, participant numbers and the timeline became necessary. Some of these changes were conditioned by the rigorous ethics approval process at Macquarie University, which required more detail to be specified about the methodology and participant recruitment. Other modifications became necessary to meet the practical challenges of time and resource management, as well as accommodating to the local culture in collecting the data in Phnom Penh. Table 1 shows the methodological changes that occurred between the original proposal and what actually eventuated.



Instrument Per proposal

Per ethics application

Actual Comments

Questionnaires: Test-takers 200 200 208 IELTS-preparation students

100 Not quantified (see comments)

109 In ethics application, this category was conflated with “200 test-takers” from previous instrument

IELTS-preparation teachers (i) For preparing students for IELTS (ii) Analysis of textbook materials

20 10 (i) 11 (ii) 10

There were only 18 IELTS-preparation course teachers at ACE in 2010, including all those teaching GEP Levels 11A, 11B, and 12

Returned graduates from overseas

30 Not included - Would have required creating and validating new instrument. Experiences of returned graduates were thought to be better accessed by interview.

Focus Groups: IELTS-preparation students

Not quantified

Not included - Questionnaires and classroom observations were thought to be sufficient sources of information for this category of stakeholder

IELTS-preparation teachers

Not quantified

Not included - Questionnaires and classroom observations were thought to be sufficient sources of information for this category of stakeholder

Parents/guardians Not quantified

Approximately 5 participants in one focus group

3 participants

Difficult to arrange one meeting time for such a group

IELTS examiners Not included

Approximately 5 participants in one focus group

6 participants

Interviews: Embassy staff Not

quantified 10 6 Unable to arrange more than 6

interviews Employers Not

quantified 10 7 Unable to arrange more than 7

interviews Returned graduates from overseas

Not included

Interviews with approximately 5 graduates

7 Tried to match numbers with previous two categories, therefore exceeded target slightly

Sponsors Not quantified

10 - Not considered a feasible category on its own as it overlapped embassy and employer categories

Pre/post interviews with test-takers

20-30 Not included - Not considered feasible as very difficult to recruit participants

Classroom observations:

Advanced General English Program (Level 12) class

Not quantified

Up to 3. 1 Difficult to recruit teachers

IELTS-preparation class

Not quantified

Up to 3. 1 Difficult to recruit teachers

Table 1: Summary of methods’ adjustments from original proposal



4.3 Data collection instruments (by stakeholder) To better understand the nature of an IELTS impact study whose focus is ultimately a whole country (ie, Cambodia), it is important to establish who the key stakeholders are, and what the most appropriate means would be to find out their particular views about the IELTS Test. Our proposal did not have the budget to scientifically survey the whole population of Cambodia (currently approximately 14 million people). Indeed, with about 80% of the population being rural subsistence farmers, we can safely assume that they would have no knowledge of and, therefore, no opinion about IELTS. To maximise the use of our limited budget and time resources, we decided to conceptualise the stakeholders from the ‘inside out’. In other words, we identified the IELTS test-takers as the stakeholders most central to the IELTS enterprise in Cambodia, and then placed other stakeholders in terms of their relationship to the test-takers. Thus, the key stakeholders we identified for the IELTS impact study in Cambodia are set out diagrammatically in Figure 1.

Parent/Guardian

Returned Graduate from Overseas

Employer

Successful Candidate

Embassy Staff

IELTS Test-taker

IELTS Examiner

Parent/Guardian

IELTS Preparation Course Student

IELTS Preparation

Course Teacher

Figure 1: Key stakeholders in IELTS testing in Cambodia



This diagram construes IELTS in terms of the pathway an IELTS test-taker in Cambodia typically follows and the main stakeholders associated with this pathway (and on whom the IELTS Test can be seen to make an impact). Parents/guardians are depicted twice: first, in terms of the support they must provide when their children are preparing for the IELTS Test; and second, in terms of being beneficiaries of children who have successfully completed degree programs overseas and have returned to make careers in Cambodia. In the pre-test learning stage, IELTS preparation course teachers are a key stakeholder, while during the actual IELTS exam, IELTS examiners who assess spoken and written skills are key stakeholders. Embassy staff has an important role to play in processing IELTS scores to facilitate the entry of successful IELTS test-takers into educational programs in overseas countries. Returned graduates themselves constitute an important stakeholder group as they can report on their experiences of IELTS and how well it prepared them for their specific overseas study program. Last, but not least, the employers of well-educated, English proficient Cambodians are also a key stakeholder group as they are gatekeepers to many of the best jobs and careers in the country.

While we acknowledge that there are other plausible stakeholders in Cambodia that we have not included in this study (eg, education policy makers), we feel that we have captured an appropriate balance of a variety of stakeholders that also fits the scale and scope of what is possible to achieve with the resources available and project timeline constraints. In adhering to Cambodian cultural practices and expectations, we paid a small participation fee for Cambodian test-takers (US$5); for expatriate teacher questionnaire participants (US$20); and IELTS examiner focus group participants (US$20). Without such payments the study would not have been possible. No other participants were paid. All participants were recruited through the use of an Expression of Interest form and signed an Information and Consent form.

The various stakeholders and the respective data collection instruments used for each are summarised in Table 2. What follows in the rest of this section is a brief summary of the rationale for including each stakeholder and useful information about the research methods adopted for the questionnaire instruments. The results are reported in the Results section (Section 5) which, for reading convenience, follows the same order in sequencing each of the nine instruments.

Instrument Stakeholder Targeted number of participants 1 Questionnaire Test-takers 200

2 Questionnaire Test preparation students 100

3 Questionnaire Test preparation teachers 10

4 Focus group IELTS examiners 5 5 Focus group Parent/guardians of test-takers 5

6 Interview Embassy officials 10

7 Interview Employers 10

8 Interview Returned graduates from overseas 10

9 Observation IELTS preparation class 3

Table 2: Summary of data collection instruments and targeted recruitment (by stakeholder)



4.3.1 Questionnaire for test-takers 4.3.1a Rationale

The most important stakeholder in the IELTS enterprise in Cambodia is undoubtedly the individual test-taker. For her or him, IELTS is the gateway to overseas study and, therefore, the key to achieving their career potential and related opportunities. For sponsored candidates, IELTS is a very high-stakes test, with perhaps no opportunity to re-sit the exam if the required band scores are not achieved. For other candidates, the IELTS Test is a hurdle that can be attempted on several occasions, if necessary. Either way, the IELTS Test stands between them and the prospect of a relatively secure and prosperous life in Cambodia.

4.3.1b Methods All questionnaire instruments used in the IELTS impact study in Cambodia were slight adaptations of questionnaires used and validated by Hawkey (2006). Different sections of the original test-taker’s questionnaire were used to prepare separate questionnaires for two crucial groups of stakeholder, namely test-takers and test preparation course students.

The original test-taker’s questionnaire comprised five sections: Personal information (name, age, etc); (Part 1) English language background; (Part 2) learning, study and text-taking approaches; (Part 3) questions regarding IELTS preparation courses; and (Part 4) questions regarding the IELTS Test (to be completed after taking the exam). For the purposes of the IELTS impact study in Cambodia, we excluded Part 3 from the test-taker’s questionnaire (but used it as the core for our test preparation course questionnaire). Thus, the test-takers (TT) group completed a questionnaire (see Appendix 1) comprising four sections as set out below.

1. Personal information (name, age, etc)

2. Part 1: English language background

3. Part 2: Learning, study and test-taking approaches

4. Part 4: Questions regarding the IELTS Test (to be completed after taking the exam)

4.3.2 Questionnaire for test preparation course students 4.3.2a Rationale Potential test-takers stand to benefit considerably from taking a test preparation course prior to sitting a test. In the case of IELTS, there is anecdotal evidence that a candidate who has taken such a course can achieve band scores half a band above what they might have otherwise achieved. Test preparation courses help students understand the structure of the Test, the types of tasks that are required, and the issue of efficient time management. Because of the close relationship between the Australian Centre for Education (ACE) which offers test preparation programs, and IDP which administers IELTS in Cambodia, our project was able to easily target IELTS preparation course students as a separate category of stakeholder in Cambodia.

4.3.2b Methods

As noted in Section 4.3.1b, the questionnaire instrument used to collect data from IELTS Test preparation course students was adapted from the instrument used and validated by Hawkey (2006). Our questionnaire for test preparation course students (TP) contained the same two sections on personal information and English language background as in our TT questionnaire, but also included questions specific to ACE’s IELTS preparation courses. Thus, the test preparation course (TP) group completed a questionnaire (see Appendix 2) comprised of three sections as set out below.




2. Part 1: English language background

3. Part 3: Questions regarding IELTS preparation classes

4.3.3 Questionnaire for test preparation course teachers 4.3.3a Rationale

As in the case of test preparation course students outlined above in Section 4.3.2a, their teachers are also key stakeholders in the IELTS enterprise in Cambodia. The teachers may or may not also be certificated IELTS examiners, but their work essentially involves ‘teaching to a test’, with the very clear goal of maximising student scores on the IELTS exam. Thus, these teachers will have insights into the challenges Cambodian students face in preparing for the IELTS Test. They are also in a position to be able to shed light on how IELTS impacts on the lives of both teachers and students.

4.3.3b Methods

Making further use of Hawkey’s (2006) questionnaires, the IELTS impact study in Cambodia used two questionnaire instruments to survey IELTS course preparation teachers. The first questionnaire concerned the preparation of students for IELTS, while the second questionnaire focused on analysing and evaluating textbook materials.

Thus, the test preparation course teachers completed a questionnaire (Appendix 3) with four sections, as set out below, and a separate materials evaluation questionnaire (Appendix 4).


2. Information about the teachers’ students

3. Teachers’ perception of the IELTS Test

4. Teachers’ IELTS preparation classes

4.3.4 Focus group of IELTS examiners 4.3.4a Rationale IELTS examiners are important stakeholders in the IELTS enterprise in three ways. First, they have undergone specialist training in writing and speaking assessment, and have achieved certification in standardised testing of the writing and speaking components of the IELTS Test. Thus, they have a vested interest in maintaining the international standards set for the IELTS Test. Second, they have a direct impact on the results of candidates with whom they have contact, either directly in face-to-face interviews, or indirectly through assessing written scripts or double-marking recorded interviews. Third, for the IELTS impact study in Cambodia, IELTS examiners can shed light on their collective experience of Cambodian candidates, including candidate strengths and weaknesses.

4.3.5 Focus group of parents/guardians 4.3.5a Rationale The parents or guardians (NB, many young Cambodians are cared for by extended family members, rather than their biological parents) of young Cambodians play a very important support role in guiding their children towards obtaining the best education possible and the best educational opportunities available. Due to the relatively poor quality of the education sector in Cambodia, many parents and guardians have a cherished goal of seeing their children eventually study overseas. Upon returning to Cambodia these young Cambodians are equipped not only with solid English skills but



also with a qualification that will open doors to a prosperous career in Cambodia (therefore, providing further security to aging parents and guardians). Thus, parents and guardians have an important stake in their children’s success at undertaking the IELTS examination given that it plays a gate-keeping role in providing access to universities in Australia, New Zealand, the UK, the USA, and elsewhere.

4.3.6 Interviews of embassy officials 4.3.6a Rationale

Embassy officials can play an important role in the IELTS enterprise in Cambodia in two ways. First, Cambodians who wish to study overseas must have their visa applications processed by embassy officials who, among other tasks, look for appropriate evidence of language proficiency. Since most overseas-bound students plan to study in English-medium universities, the IELTS Test is commonly requested by host institutions. (TOEFL exam results are also widely accepted by host institutions in western countries). Thus, embassy officials in Phnom Penh who handle such visa applications must have an awareness of the IELTS Test and the meaning of IELTS scores. Second, another role for many embassies in Cambodia is to manage scholarship programs for overseas study (eg, in Australia, New Zealand, UK, USA, Canada, and the European Union). Eligibility for such programs requires a proper evaluation of English language proficiency, and again, IELTS is a widely accepted instrument for this purpose. Embassy officials in Phnom Penh therefore have an important role to play in the process of an IELTS test-taker securing the visas and scholarships necessary for studying overseas. For this reason they are included as stakeholders in the IELTS impact study in Cambodia.

4.3.7 Interviews of employers 4.3.7a Rationale

Competence in English is an important skill expected of employees hoping to work for the top commercial and development organisations in Cambodia. In their job advertisements, companies and non-government organisations (NGOs) often specify proficiency in English as a requirement. However, rarely, if ever, do these advertisements indicate a proficiency test score such as, for example, IELTS Band 6 or TOEFL 550. Given the importance placed on English by many employers in Cambodia, and the availability of the IELTS standardised exam in Phnom Penh, we felt that employers constituted a relevant stakeholder group for inclusion in this study.

4.3.8 Interviews of returned graduates from overseas 4.3.8a Rationale

In this IELTS impact study in Cambodia, we have used the label ‘returned graduates from overseas’ to refer to Cambodians who successfully sat an IELTS Test and then completed an overseas degree, before returning to work again in Cambodia. They have been able to benefit from the intended purpose of the IELTS Academic module in that they were able to use their IELTS score to enable entry to a postgraduate degree program in an English-speaking country. The project team thought that it would be useful to learn about their experiences of how well they felt the IELTS Test had evaluated their English proficiency, and the extent to which their English proficiency level enabled them to study their specialist degree programs overseas.

4.3.9 IELTS preparation course classroom observations 4.3.9a Rationale

To provide more direct evidence of actual IELTS-related practices (rather than just relying on stakeholder self-reported accounts), the IELTS impact study in Cambodia planned for lesson observations of up to three IELTS-preparation type lessons. ACE in Phnom Penh runs essentially three types of IELTS preparation: (i) 10-weeks (45 hours) IELTS preparation course; (ii) 1-5 weeks (15 hours) IELTS orientation course; and (iii) General English Program (GEP) Level 12, which concludes with students sitting the IELTS academic module examination. The first and third categories of preparation course were targeted in the present study.



5 RESULTS BY DATA COLLECTION INSTRUMENT AND STAKEHOLDER Table 3 provides an overview of the data collection instruments used in this project and indicates the actual number of participants recruited for each category.

Instrument Stakeholder Actual number of participants 1 Questionnaire Test-takers 208 (including 102 prep’n students) 2 Questionnaire Test preparation students 109

3 Questionnaire Test preparation teachers 11 (10 for materials evaluation)

4 Focus group IELTS examiners 6

5 Focus group Parent/guardians of test-takers 3

6 Interview Embassy officials 6

7 Interview Employers 7

8 Interview Returned graduates from overseas 7

9 Observation IELTS preparation class 2 classes (~30 students)

Table 3: Summary of data collection instruments and actual recruitment (by stakeholder)

5.1 Test-taker 5.1.1 Background to results Due to the overlap between the stakeholder categories of ‘test preparation course student’ and ‘test-taker’, some of our data analyses were performed on the combined data set. For this reason, in this section dealing with the background to TT results, there is also some reference to the background to TP results.

The TT and TP questionnaires were first piloted with a few respondents representative of each group. During this piloting stage, four respondents in each group were asked to complete their respective questionnaire and to indicate any questions that might be unclear or confusing. The respondents were timed when they answered the questionnaires to determine the amount of time needed to complete each questionnaire (N=8; Age X! =25.13 [range was 17 to 32]; Time X! =33.13 mins.). After completing the questionnaires, each respondent was interviewed individually (in English with the availability of Khmer translation if necessary). Based on the results of this piloting process, minimal changes were made to the questionnaires. Those changes that were deemed necessary included: 1) modifying vocabulary and respondent choices appropriate for high school students; 2) nominal changes to reflect the course titles at ACE; and, 3) simplification, emphasis or examples added to questions which were unclear to respondents (see Appendix 2 for Part 3, questions 3, 5, 8, 9, 11, 13; and Appendix 1 for Part 4, question 1.2).

The TT group comprised two-sub-groups: those TT who had completed test preparation courses at ACE and subsequently sat the IELTS Test, and those who had registered for the IELTS Test independent of any ACE course. When test-takers registered for the IELTS exam, they were asked whether they were also a student in Group 1 category (ie, a potential TP who would subsequently sit the IELTS exam and, therefore, later be categorised as TT respondents as well), or Group 2 category (ie, TT only). The composition of TT and TP respondents is set out below.



Group 1 (TP to TT respondents)

! ACE General English Program (GEP) Level 11B ! ACE GEP Level 12 ! IELTS Preparation course ! IELTS Overview course ! Australian Development Scholarship (ADS) Pre-Departure program ! New Zealand Development Scholarship (NZDS) Pre-Departure program

Group 2 (TT only)

! New Zealand's International Aid and Development Agency: English Language Training for Officials (NZAID ELTO) Candidate

! NZDS 2010 Candidate ! ADS 2010 Candidate ! Self-funded

Those test-takers who completed an Expression of Interest form and subsequently signed an Information and Consent form were provided with a TT questionnaire. If participants were enrolled in an IELTS preparation course at ACE, they received the TP questionnaire. Thus, all participant test-takers, whether enrolled at ACE or not, received the TT questionnaire. All questionnaires were collected when the test-takers returned to IDP’s Phnom Penh office to receive their IELTS scores. Some test-takers who did not return their questionnaires were followed up by telephone calls to remind them to return the forms (and receive their payment).

Data were collected from 208 TT, including 106 respondents who only sat the IELTS exam and 102 TP who also enrolled in a test preparation course (see Table 4). TT were categorised according to their purpose for taking the IELTS exam (ie, seeking eligibility for scholarships or advanced study, or, if self-funded, for similar or other purposes). Explanations of each categorisation for both groups are summarised in Table 5.

Test Preparation Course Respondents

Test Preparation Course Respondents who took the IELTS exam

Test-taker (only) Respondents

Category N Category N Category N

ADS Pre-Departure 11 ADS Pre-Departure 11 ADS 2010 Candidate 37

GEP Level 11B 7 GEP Level 11B 5 NZAID ELTO Candidate 3

GEP Level 12 60 GEP Level 12 60 NZDS 2010 Candidate 10

IELTS Overview 10 IELTS Overview 9 NZDS Pre-Departure 5

IELTS Preparation 21 IELTS Preparation 17 Self-funded 50 IELTS Preparation* 1

Total 109 Total 102 Total 106

Total Test Preparation Respondents 109

Total Test-taker Respondents (102 + 106) 208

* One IELTS Preparation student completed the TT questionnaire only

Table 4: Respondent categorisation



• ADS 2010 Candidate: AusAID-sponsored candidates who applied for Australian Development Scholarships (ADS) during the 2010 selection process. Candidates from the government sector and female candidates are required to have a minimum overall band score of 5.0. Male candidates from the private sector are required to have a minimum overall band score of 5.5.

• ADS Pre-Departure: AusAID-sponsored candidates who took the IELTS exam after 1000 hours of Pre-Departure training at ACE. After being successfully selected for ADS, they are required to achieve an IELTS minimum requirement, generally an overall band score of 6.5 in order to undertake postgraduate study in Australia.

• GEP Level 11B: ACE students who completed the Preparation Questionnaire when they attended GEP Level 11B in March 2010. Subsequently, these respondents completed the test-taker (TT) questionnaire when they completed GEP Level 12 and sat the IELTS exam in June 2010.

• GEP Level 12: ACE students who sat the IELTS Test as a requirement to graduate from the ACE General English Program. The program completion requirement is a minimum overall band score of 5.0 with no macro skill less than 4.0.

• IELTS Overview: ACE students who attended a 15 hour IELTS Preparation course. • IELTS Preparation: ACE students who attended a 45 hour IELTS Preparation course. • NZAID ELTO Candidate: NZAID-sponsored candidates who applied for the English Language

Training for Officials (ELTO) scholarship program to undertake English Training in New Zealand for 22 weeks. The IELTS requirement is an overall band score between 4.5 and 5.5.

• NZDS 2010 Candidate: NZAID-sponsored candidates who applied for New Zealand Development Scholarships (NZDS) during the 2010 selection. The IELTS requirement is a minimum overall band score of 5.5.

• NZDS Pre-Departure: NZAID-sponsored candidates who sat the IELTS Test after 750 hours of Pre-Departure training at ACE. After being successfully selected for NZDS, they need to achieve a minimum IELTS requirement, generally an overall band score of 6.5, to undertake postgraduate study in New Zealand.

• Self-funded: Private test-takers who sat the IELTS Test without any of the financial assistance provided to other categories of test-taker.

Table 5: Respondent category explanations

5.1.2 Results for test-takers In order to be able to evaluate the impact of IELTS in Cambodia, the study collected data from two major but different stakeholder groups which also happened to overlap: test-takers (TT) and test preparation course students (TP). As noted in Sections 4.3.1b and 4.3.2b, the original validated test-taker questionnaire from Hawkey (2006) was converted into two separate questionnaires: one for the TT group and one for the TP group. A total of 208 completed TT questionnaires were received, including 102 TP respondents out of 109 who, subsequent to their test preparation course, also took the Test and completed all sections of the TT questionnaire. There were an additional 106 respondents who only sat the IELTS exam without enrolling in a test preparation course.

Due to the overlapping of TT and TP categories and their data collection instruments, the detailed results of the TT category are presented with those of the TP category in Section 5.2.2. This combined approach facilitates comparisons of these two major categories of stakeholder.



5.2 Test preparation course student 5.2.1 Background to results Potential TP participants were identified by their enrolments in IELTS preparation courses at ACE and expressions of interest were sought from them. Upon signing an Information and Consent Form, questionnaires for TP respondents were distributed in their respective preparation course classes, completed at home, and returned to IDP’s office. The courses in which these respondents were enrolled included: GEP Levels 11B and 12, and IELTS Preparation and Overview courses. The following are descriptions for each of the courses at ACE involving IELTS test preparation (IDP Education, 2010a):

! General English Program (GEP): ACE's General English Program is a part-time program designed for students wishing to achieve an upper-intermediate level of English. For students who start at the beginner level, the full course of study lasts approximately four years ranging from Beginner 1 to Level 12.

o GEP Levels 11B: Students who have successfully completed Level 11B can communicate with confidence in most social situations and in work situations which are relevant to their own needs and experience. The textbook is IELTS Foundation. The program takes 45 hours to complete this level. The next level from GEP Level 11B is GEP Level 12.

o GEP Level 12: Students complete Level 12 by taking an IELTS Test. The General English Program Level 12 Certificate is awarded on successful completion of this course and an overall band score of 5.0 on the International English Language Testing System (IELTS) with no macro-skill (Listening, Speaking, Reading, Writing) less than 4.0 on the Academic Module. The textbook is Action Plan for IELTS. It takes 45 hours to complete this level. Graduates can continue their study to English for Academic Purposes Program.

! IELTS Preparation Course (45 hours)

! IELTS Overview Course (15 hours): To improve candidates’ scores in IELTS Tests, ACE offers a 45-hour IELTS Preparation Course and a very popular 15-hour Orientation Course to familiarise candidates with the IELTS Test.

During the data analysis process, a small number of respondents were removed from the sample: nine participants were younger than 16 years old, and, therefore, not eligible to participate in the study; one respondent enrolled but did not take a test preparation course; and one respondent applied for, but subsequently did not sit, the IELTS Test. In addition, seven TP respondents, who also subsequently took the IELTS Test, completed only the TP questionnaire and not the TT questionnaire – the data from these respondents’ TP questionnaires were retained in the database. Data entry was cross-checked between the respondent records of the then current ACE course enrolment records and IELTS Test application data to ensure an accurate account of our TP data collection.

5.2.2 Results As noted in Section 5.1.2, the results of TT respondents will be reported here alongside the results of TP respondents. The reasons for this concerned database efficiency and the overlap between TT and TP participants. By presenting the results together, comparisons between the two categories are easily made and highlighted. It is important to note here that differences between the TT and TP samples are at least partly due to sampling methods: our source for TP respondents was exclusively ACE; and our TT sample of 106 respondents represented less than a quarter of all IELTS test-takers in the period from March through September 2010 (however, it did not include any TP who went on to sit the



IELTS Test, to avoid double counting the same respondents). Nevertheless, we think that the personal characteristic differences between TP and TT respondents as summarised in Table 6 are, to some extent, a function of the different wider TP and TT populations in Cambodia. As shown in Table 6, all but two respondents among the two groups were Cambodian. The balance between the number of male and female respondents was more equal among TP (Male, N=56 [55%]; Female, N=46, [45%]) than for TT (Male, N=65 [61%]; Female, N=41 [39%]). More TT were employed, compared to the higher percentage of TP who were still in school. These data reveal an overall higher level of maturity, employment experience and age of the TT (only group) compared to the TP. The differences highlighted in Table 6 are consistent with our own familiarity with the ELT sector in Cambodia. For example, there is a (shrinking) gender bias that still favours males; most ACE students are younger and more likely to be students than employees; and more TT are likely to be employed and have opportunities to use English in their workplace and to apply for scholarships overseas.

Test preparation (TP) Respondents Test-taker (TT) Respondents* N Percent N Percent Home country Cambodia 102 100% Cambodia 104 98% Korea 1 1%

Philippines 1 1%

Total 102 100% Total 106 100% Nationality Cambodian 102 100% Cambodian 104 98%

Korean 1 1%

Filipino 1 1%

Total 102 100% Total 106 100% Sex Male 56 55% 65 61% Female 46 45% 41 39%

Total 102 100% Total 106 100% Age (years) 16-19 43 41% 16-19 13 12%

20-24 23 23% 20-24 12 11%

25-29 20 20% 25-29 51 49%

30-46 16 16% 30-46 30 28%

Total 102 100% Total 106 100% Employment / education status

Employed 41 41% Employed 78 73%

University student

32 31% University student

21 20%

High school student

29 28% High school student

7 7%

Total 102 100% Total 106 100% *Note: Data for TTs in this table does not include that of TPs who went on to take the IELTS Test

Table 6: Summary of respondent personal characteristics



Table 7 summarises the personal language use experience of both TP and TT respondents. As might be expected, the overwhelming majority of respondents were Cambodian nationals living in Phnom Penh, 87% (TP) to 86% (TT), and they reported using Khmer as the primary language at home. However, a small minority of respondents reported using several languages on a daily basis. Approximately 10% of the respondents in both categories (TP, N=12, 12%; TT, N=9, 8%) reported having lived with a native speaker of English as a child.

Few respondents in either group experienced English language instruction at an early age (TP, N=6, 6%; TT, N=6, 6%); however, the numbers of respondents in each group increasingly experienced English language instruction as they progressed through their formal educational experience. It is interesting to note that the TT respondents, an older cohort, reported a higher percentage of English language instruction at the tertiary level than TP, a younger cohort (TP, N=71, 70%; TT, N=88, 83%), whose experiences are based on more recent teaching and learning environments in Cambodia. Additionally, while reporting a lower incidence of English language instruction at the tertiary level, this same group also reported slightly higher involvement in ‘extra’ or private English language classes (TP, N=84, 82%; TT, N=83, 78%). Of all English language courses, approximately one third of TP respondents indicated that over half (TP, N=32, 31%) of such lessons were led or assisted by native English speakers. This percentage was nearly equal to the 30% for TT (N=32). Nevertheless, over half of TP (N=54, 54%) reported that their English teachers used English as a medium of instruction all the time.

The responses for TT regarding teachers’ language use was more mixed, but over 67% indicated that their teachers used English either all the time (N=33, 31%) or more than half the time (N=39, 37%). No respondents indicated that they had been enrolled in classes where Khmer was exclusively used to teach English.



Language spoken at home TP TT f Percent f Percent Khmer 89 86% 91 85% Khmer and English 4 4% 4 4% Khmer and Chinese 2 3% 1 1% English 2 2% 2 2% Chinese 0 0% 1 1% Khmer, English and Chinese

1 1% 0 0%

Tagalog and English 0 0% 1 1% Korean 0 0% 1 1% No answer 4 4% 5 5%

Total 102 100% 106 100% Lived with a native speaker of English as a child TP TT f Percent f Percent Yes 12 12% 9 8% No 88 86% 97 92% No answer 2 2% 0 0%

Total 102 100% 106 100% Did you study English in . . . TP TT Yes No Yes No f Percent f Percent f Percent f Percent Kindergarten 6 6% 96 94% 6 6% 100 94% Primary school 36 35% 66 65% 24 23% 82 77% Secondary school 85 83% 17 17% 75 71% 31 29% College / university 71 70% 31 30% 88 83% 18 17% Extra language classes

84 82% 18 18% 83 78% 23 22%

Number of English lessons taught or assisted by native speakers of English TP TT f Percent f Percent More than half 32 30% 32 30% About half 25 25% 15 14% Less than half 17 17% 20 19% A few 21 21% 32 30% None 7 7% 7 7% Total 102 100% 106 100% Amount of time teachers of English speak to you in English TP TT f Percent F Percent All the time 54 53% 33 31% More than half 23 22% 39 37% About half 8 8% 14 13% Less than half 17 17% 20 19% Never 0 0% 0 0% Total 102 100% 106 100%

Table 7: Personal experience of language use



The statistical means for responses on questions related to daily current use of English are presented in Table 8, (where 1 = often, 2 = sometimes, 3 = rarely, and 4 = never). Questions in this section asked whether respondents used English for socialising, communicating with friends from overseas, through media, in the workplace, through reading English texts in specialised subjects, or by writing English daily. Not surprisingly, based on the sample population in the current study (students and professionals preparing for, or having recently taken, the IELTS Test), for all questions, the mean for responses indicated ‘Sometimes’ or, more frequently, ‘Often’. While few differences can be seen between the two groups, TT used English more often for work (as this group reported a higher percentage of employment) and for reading specialised texts in English in their subject areas.

Often Sometimes Rarely Never No Answer

X! f % f % f % f % f %

TP 1.96 20 20% 62 61% 19 19% 0 0% 0 0% Use English for socialising TT 2.01 28 26% 51 48% 25 24% 2 2% 0 0%

TP 2.22 27 26% 37 36% 25 25% 13 13% 0 0% English contact with friends overseas

TT 2.04 37 35% 39 37% 19 18% 11 10% 0 0%

TP 2.08 29 28% 47 46% 9 9% 13 13% 3 3% Exposed to English in the media

TT 2.08 32 30% 41 39% 22 21% 8 8% 2 2%

TP 2.10 27 26% 39 38% 14 14% 9 9% 13 13% Use English for work TT 1.65 58 55% 31 29% 13 12% 4 4% 0 0%

TP 1.60 52 51% 34 33% 11 11% 1 1% 4 4% Read texts in English in specialist subject

TT 1.33 73 69% 29 27% 3 3% 0 0% 1 1%

TP 1.73 42 41% 43 42% 17 17% 0 0% 0 0% Write in English in a day TT 1.67 51 48% 39 37% 15 14% 1 1% 0 0%

Table 8: Current use of English

Part 2 of the questionnaire presented questions related to learning and studying, and test-taking approaches employed by both TP and TT respondents (see Appendices 11 and 12, respectively, where 1 = strongly disagree, 2 = moderately disagree, 3 = slightly disagree, 4 = slightly agree, 5 = moderately agree, and 6 = strongly agree). SPSS PASW Statistics 18 software was used to determine, using independent t Tests, whether significant differences exist between responses from TP and TT respondents on each question. Regarding learning and study habits, both TP and TT reported slight to moderate agreement on most items, indicating an overall directed and organised approach to studying. This result would be consistent with the expectations of a comparatively educated sample in a developing country from 1) a relatively elite social class, with the goal, potential and possible ability to pursue study abroad; or 2) professionals in upper status employment positions that would include those using English in the workplace.

Related to learning and study habits, there were significant differences between the groups. TT respondents reported a higher level of awareness of their shortcomings when using English than did TP respondents. TT respondents (M = 4.71, SD = 1.18) reported that they are more likely than TP respondents (M = 4.15, SD = 1.20), t(205) = -3.43, p = .001, two-tailed, to know when they are making grammar mistakes while speaking or writing. Likewise, TT respondents (M = 4.80, SD = 1.07) are also more often conscious of making pronunciation mistakes than TP respondents (M = 4.35, SD = 1.23), t(203) = -2.80, p = .006, two-tailed, when they are speaking. These results may be due in part to a



higher likelihood of TT respondents reflecting on their communication experiences: TT respondents (M = 5.19, SD = .86) reported a significantly higher frequency of reflecting on how to better improve their communicative ability after having conversations in English than did TP respondents (M = 4.85, SD = 1.20), t(183) = -2.31, p = .022, two-tailed.

Regarding test-taking approaches, one significant difference was found between TP and TT respondents. TP respondents (M = 3.12, SD = .74) reported a significantly higher likelihood to predict questions when listening to a passage in this section of the exam compared to TT respondents (M = 2.70, SD = .84), t(204) = 3.83, p = .000, two-tailed. These results seem to indicate that TP respondents may have obtained this specific and concrete test-taking strategy during their test preparation courses.

For the respondents categorised as TP, 70% (N=76) were currently enrolled in an IELTS preparation course at the time of completing the questionnaire. All who responded indicated that they were enrolled in the course at ACE (N=62, 82%); however, a comparison of responses and ACE enrolment records seems to indicate that there may have been some confusion on the part of respondents regarding this question (not all respondents answered the question, but all were enrolled in an ACE IELTS preparation course; some respondents gave multiple answers based on previous and current enrolment, others answered based solely on current enrolment). There was similar inconsistency with regard to the type and length of preparation course in which the respondents were enrolled.

TP respondents were also asked, in general, to indicate the proportion of class time and usefulness of that time allotted for each language skill (reading, writing, listening and speaking) and other language focus (grammar and vocabulary) The results are presented in Table 9. The proportion of time was relatively balanced among all four skills areas, with reading, writing and listening (approximately 21%, 20.9% and 19.4%, respectively) reported as being allotted slightly more time in class, with speaking (15.8%) being allotted comparatively less time. Vocabulary and grammar associated activities were reported to occur during approximately 10% of class time (10.8% and 9.5%, respectively). Additional activities with less frequency were reported, including reviewing practice tests and past papers, and pronunciation. Most TP respondents indicated that activities in all skill areas were very or quite useful; only a few respondents indicated that one or another was not very useful.

Very useful Quite useful Not very useful Blank Percentage of time in class f Percent f Percent f Percent f Percent Reading 21.00% 97 89% 11 10% 1 1% 0 0% Writing 20.94% 86 79% 20 18% 3 3% 0 0% Listening 19.38% 85 78% 23 21% 1 1% 0 0% Speaking 15.79% 76 69% 30 28% 2 2% 1 1% Vocabulary 10.82% 62 57% 44 40% 2 2% 1 1% Grammar 9.48% 52 47% 48 44% 6 6% 3 3% Other 2.17% 9 8% 20 18% 2 2% 78 72%

Table 9: Macro-skill time proportion and usefulness

TP respondents were also asked about the occurrences of specific activities during class related to each skill area (Table 10). Activities related to questions associated with listening passages were most common within this skills area (ie, reading the questions and predicting what listening passages would be about, 91%; reading questions and guessing the type of answer required, 92%). Listening to and taking part in seminar or workshop activities were less common (34%). Identifying main ideas and increasing reading speed (90% and 86%, respectively) were important reading skills that were included in class activities. It is also interesting to note that a surprising number of TP respondents



indicated that they were not sure whether the activities included in this part of the questionnaire related to reading skills that occurred in their preparation classes (eg, analysing text structure and organisation (24%), interpreting statistics, graphs and diagrams (27%), reading subject-specific texts (24%), and using English only dictionaries (39%)). Conversely, with regard to writing, over 95% of TP respondents indicated that describing statistics, graphs and diagrams, learning how to organise essays, and making use of transition words were included in their classes. Other aspects of writing (eg, learning how to write in different styles (79%); planning written answers to test questions (62%); copying out good paragraphs and model answers (61%)) were less common. Practising making a point and providing supporting examples (85%) and using words/phrases to organise speech (80%) were the most commonly reported activities related to speaking skills, while engaging in group discussions or debates (75%) and using filler words to cover silences in speech (71%) were also frequently reported.

Yes No Not sure LISTENING f Percent f Percent f Percent L.1. Reading the questions and predicting what listening passages would be about.

99 91% 5 5% 5 5%

L.2. Listening to live, taped or video talks / lectures and taking notes.

79 72% 13 12% 17 16%

L.3. Listening and taking part in seminar / workshop activities.

37 34% 55 50% 16 15%

L.4. Using information from a lecture or talk to write reports. 43 39% 47 43% 18 17% L.5. Reading questions and guessing the type of answer required.

100 92% 5 5% 3 3%

L.6. Practice in recognising previous information repeated in different words.

75 69% 6 6% 28 26%

READING R.1. Analysing text structure and organisation. 67 61% 15 14% 26 24% R.2. Interpreting statistics / graphs / diagrams. 52 48% 28 26% 29 27% R.3. Reading texts to predict test questions and tasks. 75 69% 20 18% 14 13% R.4. Learning quick and efficient ways of reading texts in English.

92 84% 9 8% 8 7%

R.5. Reading articles, reports, books in your specialist subject area.

61 56% 21 19% 26 24%

R.6. Using English-only dictionaries to complete reading tasks.

43 39% 53 49% 43 39%

R.7. Reading quickly to get the main idea of a text. 98 90% 7 6% 4 4% WRITING W.1. Copying out good paragraphs and model answers. 67 61% 23 21% 19 17% W.2. Describing a graph / a process / statistical data. 105 96% 0 0% 4 4% W.3. Learning how to organise essays. 104 95% 2 2% 3 3% W.4. Practising using words or phrases to organise a written text (eg firstly, furthermore, secondly, therefore).

104 95% 1 1% 4 4%

W.5. Learning how to write in different styles. 86 79% 6 6% 17 16% W.6. Short report writing. 44 40% 40 37% 25 23% W.7. Planning written answers to test questions. 68 62% 13 12% 27 25% W.8. Editing written work. 59 54% 26 24% 24 22% W.9. Writing parts of test answers. 52 48% 21 19% 35 32% W.10. Writing long essays, reports (ie, over 1000 words). 15 14% 87 80% 7 6% SPEAKING S.1. Practising making a point and providing supporting examples.

93 85% 7 6% 9 8%

S.2. Planning and delivering oral presentations. 58 53% 28 26% 22 20% S.3. Group discussions / debates. 82 75% 20 18% 7 6% S.4. Practising using filler words to cover silences in your speech (eg well...you see).

77 71% 21 19% 11 10%

S.5. Practising using words or phrases to organise your speech (eg firstly, furthermore, secondly, I have two points).

87 80% 10 9% 12 11%

Table 10: Occurrence of skill activities in IELTS preparation classes



Regarding class activities specifically related to the IELTS exam, the largest number of TP respondents (31.3%) reported using practice tests in their preparation classes. Techniques for taking the exam (18.1%), information about the exam (17.2%), and looking at past papers (16.7%) were also indicated. Numerous specific test-taking techniques were also included in preparation classes (eg, time management, skimming/scanning, predicting answers, and understanding exam instructions). A number of respondents (N=13, 33%) indicated making use of the Guided Individual Learning (GIL) Centre (IDP, 2010b), a self-access centre at ACE.

Students most commonly reported using Action Plan for IELTS and Focus on IELTS as course textbooks. Most TP respondents reported that the textbooks included language skill (reading, writing, listening, and speaking) activities, practice tests and test-taking strategies. These activities were reflected in the TP respondents answers related to positive aspects of the textbook: the most often selected response (N=43, 46%) indicated that the textbooks provided useful advice and strategies, as well as practice tests (N=18, 19%) and test-taking techniques (N=10, 11%). Shortcomings of the textbook reported were an insufficient number of practice tests (N=17, 18%), lack of CDs (N=17, 18%), the complicated or complex nature of the texts (N=16, 17%), and the cost of the textbook (N=11, 12%).

TP respondents noted that the most common supplementary material utilised by their teachers were handouts (N=60, 55%), but also reported that teachers used additional practice tests (N=18, 17%), practice readings (N=17, 16%), and internet resources (N=13, 12%), among others. Most TP respondents felt that those who were successful on the IELTS Test were those who practiced (N=71, 65%), prepared in a general sense (N=13, 12%), and attended a preparation course (N=12, 11%). A slight majority of the TP respondents in the current study indicated that they felt that they had been successful in their course (N=56, 51%), yet a considerable number (N=29, 27%) felt they had not been successful. Seventeen percent of the TP respondents (N=18) were unsure whether they had been successful or not. Those students who felt they had been successful based their view on the belief that they had had enough experience (N=20, 36%), had gained confidence (N=12, 21%), and had improved their skills (N=12, 21%). Conversely, students who felt they were not successful in the course noted that they had not had enough experience (N=18, 62%) and had not sufficiently improved their skills (N=4, 14%).

A majority of TP respondents (N=61, 56%) determined that even if they had not been required to take the IELTS Test, they felt that similar preparation would have been helpful, particularly for students who wished to study abroad (N=20, 33%) or who wanted to improve their English abilities (N=11, 18%). In contrast, those TP respondents who would not have chosen to prepare in the same way if the IELTS Test were not required, would have preferred a different approach to the course because they were not planning to study abroad (N=15, 33%), and felt that the course, based only on the IELTS Test, was too limiting. For those TP already engaged in an English medium environment, an overwhelming majority (N=92, 84%) stated that their communicative skills and knowledge had improved, compared with general skills (N=34, 37%), specific writing skills (N=11, 12%), and academic skills (N=10, 11%). Most TP respondents also felt that such an IELTS preparation course would be sufficient for those not attending university (N=71, 65%), for future employment (N=9, 13%) and for improving language skills (N=9, 13%). Again, those TP respondents who thought a similar class would not provide sufficient general preparation suggested that such a course does not improve communication skills (N=6, 23%) and is too narrowly focused on the IELTS exam (N=6, 23%).



As previously mentioned, after completing the IELTS preparation course the vast majority of TP students sat the IELTS Test, and then subsequently completed the TT questionnaire. Those TT respondents who did not complete a preparation course also filled out this same questionnaire. Of those TP respondents who completed the TP questionnaire and sat the exam, 40% (N=41) were employed or not currently students (working, recently graduated, or waiting to enter graduate school), compared to 74% (N=78) of TT respondents. Twenty-one (20%) of the TT respondents were university students and seven (7%) were high school students; 32 (31%) of the TP respondents were university students, and 29 (28%) were high school students. Both TP and TT respondents sat the IELTS Test between 1 March and 30 September 2010. Although most had only taken the exam one time (TP, N=88, 86%; TT, N=71, 67%), a small portion of each group had taken the exam two or more times (TP, N=14, 14%; TT, N=35, 33%).

Members of both groups were taking the IELTS Test in order to:

! study abroad (TP, N=32, 31%; TT, N=50, 48%) ! secure a scholarship (TP, N=22, 22%; TT, N=44, 42%) ! determine their English language ability (TP, N=14, 14%; TT, N=6, 6%).

While most had not taken another exam (TP, N=93, 91%; TT, N=78, 74%), some of each group had also taken other language proficiency exams, namely the TOEFL exam (TP, N=7, 7%; TT, N=26, 25%).

Only three TT respondents took the IELTS General Training (GT) Module, all others in both groups took the Academic (A) Module. Most TP respondents received band scores between 5.0 and 6.0 inclusively (5.0, N=21, 21%; 5.5, N=23, 23%; 6.0, N=34, 33%). With consideration to their expectations, 30% of TP respondents achieved the score they expected to receive on the exam, and 38% reached their overall band scale requirement (for admission, employment or scholarship). The most commonly required band scores were:

! 5.0 (N=21, 21%) ! 6.0 (N=17, 17%) ! 6.5 (N=39, 38%).

Likewise, most TT respondents scored between 5.0 and 6.5 (5.0, N=26, 25%; 5.5, N=25, 24%; 6.0, N=23, 22%; 6.5 N=22, 21%). Forty-four (43%) of TT respondents achieved the band score they expected, and 33% (N=34) achieved their required score.

The questionnaire asked respondents to consider the IELTS Test experience (Table 11). PASW Statistics 18 software was used to determine whether significant differences exist between TP and TT responses to each question, using independent t tests. Both TP and TT respondents worried about taking the IELTS Test. Both groups ranked the Reading sub-test as the most difficult on the exam, followed by the Listening, Writing and Speaking sub-tests. Two significant differences were found: TP respondents ranked both Writing (M = 2.06, SD = .88) and Speaking (M = 3.13, SD = .94) as more difficult than the TT respondents’ ranking of Writing (M = 2.45, SD = .87), t(206) = -3.27, p = .001, two-tailed, and Speaking (M = 3.41, SD = .84), t(206) = -2.27, p = .025, two-tailed. Additionally, perhaps indicating a benefit of timed practice experienced during preparation courses, TT respondents (M = 1.55, SD = .76) indicated that time pressure was a more influential factor that affected their performance as compared to TP respondents (M = 1.81, SD = .90), t(194) = 2.27, p = .024, two-tailed. TT respondents also struggled with the familiarity of topics and their general fear of tests more than TP respondents. Both groups indicated that difficulty with the language and questions was problematic for them on the exam. In general, however, more TP respondents (N=65, 64%) than TT respondents (N=56, 53%) felt that they had done their best on the exam.



N Mean Std. Deviation

Std. Error Mean

Did you worry about taking the IELTS Test? (1 = very much, 4 = very little)

TP 102 1.81 .853 .084 TT 106 1.75 .926 .090

Rank the sections of the IELTS exam that you felt were most difficult. (1 = the most difficult, 2 = second most difficult, 3 = third most difficult, 4 = easiest)

TP 102 2.16 .952 .094 • Listening TT 106 2.28 1.002 .097 TP 102 1.91 .857 .085 • Reading TT 106 1.74 .887 .086 TP 102 2.06* .877 .087 • Writing TT 106 2.45* .863 .084 TP 102 3.13** .941 .093 • Speaking TT 106 3.41** .837 .081

What affected your performance? (1 = a lot, 2 = quite a lot, 3 = not a lot, 4 = not at all)

TP 102 2.19 .741 .073 • Difficulty of language TT 106 2.38 .951 .092 TP 102 2.18 .737 .073 • Difficulty of questions TT 106 2.37 .760 .074 TP 101 2.21 .864 .086 • Unfamiliarity of topics TT 106 2.02 .873 .085 TP 100 1.81** .895 .090 • Time pressure TT 106 1.55** .758 .074 TP 100 2.38 .885 .089 • My fear of tests TT 106 2.52 .968 .094

* Significant at the .01 level. ** Significant at the .05 level.

Table 11: IELTS exam experience

In general, 85% (N=87) of TP respondents and 75% (N=80) of TT respondents felt that the exam was fair, based on the ability of the exam to cover all skill areas and accurately reflect the test-takers’ actual language proficiency. The international reputation of the Test was also mentioned as a positive attribute. Yet others felt the exam was not completely fair because of the fact that test-takers needed to have broad general knowledge (which is not necessarily related to language ability although it is inextricably linked with language), and that the exam results can be biased based on time constraints during the Test and the amount of practice prior to the Test. Indeed, in addition to language skills, nearly 50% of all respondents (TP, N=52, 50%; TT, N=52, 49%) stated that general knowledge was also important in order to gain a higher score on the exam. Test-taking strategies were also seen as valuable.

Most respondents felt the Test was most appropriate for undergraduate (TP, N=95, 93%; TT, N=97, 92%) and postgraduate (TP, N=84, 82%; TT, N=93, 88%) students. Both groups also indicated that the exam was, at least, at times appropriate for all students over the age of 15 (TP, N=61, 60%; TT, N=50, 47%), including professionals (TP, N=59, 58%; TT, N=71, 67%).



A majority of both groups also felt the exam was appropriate for individuals from all countries and cultures (TP, N=64, 63%; TT, N=64, 60%) and for students in all subject areas (TP, N=74, 73%; TT, N=80, 75%). Overall, the Test was viewed as covering all skills adequately, fairly, and clearly, although some respondents disliked the time constraints, level of difficulty, and some sections which dealt with specific skill areas. Practice (TP, N=47, 46%; TT, N=41, 39%) was the most common response when asked what advice the respondents would give others who were preparing for the IELTS Test. Other advice included being prepared for the exam (including taking a preparation course), becoming familiar with the Test, and being confident and relaxed.

5.3 Test preparation course teachers 5.3.1 Background to results Test preparation course teachers were invited to complete two questionnaires: one was concerned with their experience of teaching IELTS Test preparation courses (Appendix 3); the other was concerned with evaluating the textbook they used on such courses (Appendix 4). As was the case for the questionnaires used with TT and TP respondents, the teacher questionnaires were piloted before their use in the study. One ACE teacher was asked to complete these questionnaires and to make comments on any questions that might be unclear or inappropriate to the Cambodian context. The teacher had over 15 years of teaching experience, seven of which were at ACE, and had specific experience with the test preparation courses offered at that institution. When she had completed the questionnaires, she was interviewed by Stroupe regarding any recommended changes. Only slight editorial adjustments were suggested, including nominal changes to reflect the titles of courses offered at ACE.

Expression of Interest flyers were distributed to relevant teaching staff at ACE, and after signing an Information and Consent form, the two questionnaires for the course preparation teachers were distributed. Participating teachers were asked to complete and return the questionnaires within two weeks.

5.3.2 Results 5.3.2a Results for test preparation course teachers Data were collected from 11 teachers who taught IELTS test preparation courses. All teachers (three female, eight male) who completed questionnaires were instructors at ACE (two part-time instructors; 9 full-time instructors) and ranged in age from 31 (31-40, N=3; 41-50, N=6) to over 50 years old (51-60, N=2). Each had between seven and 33 years of teaching experience (X! =15.36 yrs.). All but one indicated they had completed a BA-level degree, two had earned Masters degrees, and nine had completed certification or diplomas in Teaching English as a Foreign Language, or similar qualifications. In addition, five indicated that they had received training as an IELTS examiner (or for examining some other international language test), and three reported being currently certificated IELTS examiners. Three also indicated that they had experienced training specific to IELTS preparation (professional development or conference workshops). Ten of the 11 teachers reported having specific experience teaching IELTS preparation courses (or similar courses) in the past, ranging from previously teaching only one course to teaching multiple types of courses, over a number of years, at various institutions.

In addition, a number of teachers had experience instructing test preparation classes for other English language examinations, including (in order of frequency) Test of English as a Foreign Language (TOEFL), Test of English for International Communication (TOEIC), Cambridge Preliminary English Test (PET), Cambridge First Certificate in English (FCE), Cambridge Certificate in Advanced English (CAE), and the Trinity College Graded Examinations in Spoken English (GESE).



When asked to discuss each of the tests with which they had had experience, most comments were related to the IELTS Test. While some teachers suggested that the Test was reliable, challenging and appropriate, certain aspects were criticised (eg, the scoring of the Speaking sub-test can be subjective; the reading sub-test can be confusing based on the use of yes/no and negative questions; and the Writing sub-test’s Task 1 is irrelevant). Regarding the TOEFL, the recent changes to the TOEFL internet-based Test (iBT) were viewed as positive, particularly relating to the integrated tasks. Nevertheless, the TOEFL was criticised as being US-centric, heavily grammar-focused, and inauthentic (eg, its listening sections). The TOIEC was similarly criticised with respect to its grammar focus and authenticity. It was also pointed out that the PET reflects ‘everyday English’ and the CAE tests everyday English skills (although the lack of authenticity of listening sections was noted in the case of each of these tests); and the GESE was potentially less stressful for test-takers as they were able to pre-select their speaking topic.

The teacher respondents taught students ranging in age from 14 years to over 18, the majority of whom were Cambodian. Teacher responses regarding the national origin of students were: Cambodia, N=9; China, Korea, N=1. Students’ level(s) of education ranged from secondary (up to 16 years) to postgraduate, with the highest number at the upper secondary (17-19 years) or equivalent level. Students in the teachers’ preparation courses were preparing for, or applying to, programs in Australia (37%), New Zealand (18%), Japan, the United States (Hawaii) (" 1% for each), or were undecided (27%). Most teachers (N=9, 82%) indicated that their students were preparing to take the Academic module of the IELTS Test, while four (37%) indicated their students would be taking at least some parts of the General Training module. This indicates an overlap among teachers’ students, some of whom would be taking the Reading or Writing sub-tests of either module. Over half of the teachers (55%) were working with students who had never taken the IELTS Test before, while two teachers indicated that all or most of their students had already sat the exam at least once.

Those teachers who were teaching students who had already experienced the IELTS Test were asked to compare their students’ test results with their own assessment of the students’ language abilities. Students’ writing and listening abilities were rated as most similar to the respective students’ IELTS Test results, while reading skills were assessed to be lower than the corresponding results, and speaking skills were assessed as slightly higher.

Regarding the appropriateness of the use of the IELTS Test for different purposes, there was broad agreement among the teachers that the exam is suitable for use at the undergraduate and postgraduate levels. There were mixed opinions when suitability of the exam was considered at pre-university and vocational levels, as well as for professional work and for immigration purposes, although there seems to be more support for use of the exam in these two latter instances.

All teachers indicated that the IELTS Test motivates their students, although there were mixed opinions as to whether the exam causes unhelpful amounts of stress for students. Teachers also seem to recognise at least the same amount of success in their IELTS preparation courses as in other language courses, due in part to the motivation of the students and the clear goals of such courses.

Almost all teachers (N=10) indicated that the exam influences their choice of content in their classes, including focusing on students’ needs (as related to their performance on the exam), their consideration of topics for and types of writing (as related to what students may encounter on the exam). Additionally, the framework of the Test also influences the teachers’ choice of style of question, content, and type of exercises, all of which are chosen in order to mirror those found on the exam to better prepare students. While a four skills approach is still utilised by some teachers, a majority (N=7) indicated that the exam also influences their teaching methodology. Teachers reported



using the exam marking criteria and a systematic approach to teaching writing in their classes. While some encouraged students to be independent learners, other teachers suggested that their courses had become less communicative and had taken on more of a workshop-type environment as task-specific skills were emphasised.

For the teachers, the positive aspects of teaching an IELTS preparation course include having clear goals and objectives (and associated tasks) in the class; being able to observe clear progress in their students (either during the class or reflected in later exam scores); and working with students who are highly motivated. On the other hand, teachers felt that the structured nature of the IELTS preparation courses stifled creativity and became repetitive and boring at times (with an emphasis on practice [tests]). Students also seem to have unrealistic expectations of the gain that could be expected from taking such a course. In order for students to achieve their goals on the Test, teachers suggested that vocabulary and general knowledge related to global issues are important. This may explain, in part, the factors that guide their selection of content for their classes. On the other hand, most teachers also indicated that a certain level of critical thinking skills was useful when taking the exam. This may be contradictory to the perception that IELTS preparation courses can, at times, be repetitive, focusing on rote learning and practice, and limited in the development of creativity. Nevertheless, when making suggestions to other teachers related to IELTS preparation courses, the teacher respondents in the current study indicated that understanding the IELTS Test (format, scoring criteria, structure, etc) was very important, along with understanding students’ needs related to the exam. While some teachers also encouraged others to help students develop effective learning skills/strategies and a sense of autonomous learning, one teacher also suggested bringing non-IELTS oriented activities into the classroom to break the potential monotony of more structured classroom instruction.

Teachers showed a common understanding of the structure and requirements of the IELTS Test: the exam does not include a dedicated grammar section; candidates do not have to ask questions in the Speaking module; candidates have only one opportunity to hear the passages in the Listening modules; candidates have to write at least 150 words for the first Writing task; candidates cannot refer back to the reading texts when they complete the Reading sub-test (the Reading sub-test contains three sections); and, during the Listening sub-test, candidates may be required to label a diagram. When comparing the different components of the exam, teachers rated the Reading sub-test as the most challenging for their students, followed by the Writing, Listening and Speaking sub-tests. Likewise, teachers indicated that they spent the most class time on writing (20-40%) and reading (20-35%) skills; somewhat less time on listening (15-30%) and speaking (10-20%); and the least amount of time on vocabulary building and grammar (5-15% each). Some teachers also reported giving some attention to study and time management skills.

There was broad agreement among the teacher respondents concerning the activities they used to prepare their students for the different skills necessary for the Test. While there were some differences in the form of listening used in practice activities (live or recorded talks or lectures while taking notes), most teachers indicated that they teach students to read questions and anticipate the topic of an upcoming listening passage and teach the types of answers required. While teachers also drew the students’ attention to repeated words in a listening passage, teachers did not seem to make use of authentic situations where listening skills are necessary, such as taking part in a seminar discussion or using information from a listening passage (lecture) in a writing assignment. When developing reading skills, teachers cited the following as goals in their preparation classes: analysing text organisation; interpreting graphic information (statistics, graphs, diagrams, etc); learning how to read efficiently and effectively; and identifying the main idea of a text quickly. The use of monolingual dictionaries and reading general texts was not encouraged. The organisational structure of writing tasks (describing graphs and diagrams, essay organisation, and components of the writing process [planning, drafting, editing, revision, etc]) were also emphasised.



While delivering presentations was not seen as very useful, using supporting examples, engaging in group discussions, organising ideas (with specific transitional vocabulary), and increasing fluency (including using filler words to fill in gaps of silence) were all seen as necessary speaking skills. When considering the introduction of the specific mechanics, organisation and assessment criteria of the IELTS Test, teachers placed different emphasis on different aspects of the Test (information about content and format, looking at past papers or taking practice tests, test-taking strategies and using assessment criteria to provide feedback in class), and there was no discernable trend apparent in their responses.

All the teachers reported using one of three textbooks:

1. Roberts, R, Preshous, A, and Gakonga, J, 2004, IELTS Foundation, Macmillan Education in association with The Open University, Oxford

2. Jakeman, V, and McDowell, C, 2006, Action Plan for IELTS, Cambridge University Press, Cambridge

3. O’Connell, S, 2002, Focus on IELTS, Longman, New York.

On the one hand, this choice reflects the fact that all the respondents to the teachers’ questionnaire were employed, at the time of the study, in the same school (ACE), with its own set curriculum. On the other hand, while the situation in Cambodia has improved in recent years particularly in Phnom Penh, the availability of a large selection of English language textbooks, for IELTS or otherwise, has been significantly limited. In general, the teachers were pleased with the textbook they used (ie, textbooks covered information about the format of the exam, question types, relevant topics and comparable reading texts, activities and exercises). On the other hand, the perceived shortcomings of their textbooks echoed those views earlier expressed about the preparation courses in general: activities are non-communicative and can become repetitive and boring. In addition, the textbooks can be seen as ‘Eurocentric’ and at times rather confusing (eg, as in lacking sufficient detail or instructions) or limited (ie, simplistic and ‘thin on content’).

In the opinion of the teacher respondents, successful learners in IELTS preparation courses:

! understand and utilise study skills ! have higher all-round English skills ! need to practice and study longer than the average student ! need to develop or have already achieved an extensive vocabulary ! are knowledgeable about world events ! take responsibility for their own learning.

If, however, the IELTS Test were not required, the majority of teacher respondents (N=8, 73%) would not have prepared students in the same way for their future studies, but rather would have focused on more traditional English for Academic Purposes (EAP) study skills (ie, research skills, summary writing, critical thinking skills, note taking etc). On the other hand, a few teachers thought that the IELTS preparation courses were sufficient as they already stressed academic skills, and in fact, would be useful for students who were planning to attend university, but not necessarily planning to take the IELTS Test.



5.3.2b Results for materials evaluation For the second questionnaire that IELTS preparation course teachers were invited to complete (Appendix 4), teachers (N=10) were asked to evaluate the materials (course books) that they were currently using in their courses. Two course books were evaluated: IELTS Foundation (N=7), and Action Plan for IELTS (N=3). IELTS Foundation was being used with GEP 11A and IELTS Preparation students (ie, upper intermediate as defined by ACE) whereas Action Plan for IELTS was being used with GEP 12 and IELTS Preparation students at a slightly higher level at the time of the current study. Teacher respondents’ evaluations of each textbook are set out in the following sub-sections.

Textbook 1: IELTS Foundation

Teachers characterised this textbook as being “a language teaching book and an international test preparation book combined” (with the international test being IELTS). There was strong agreement among the teachers that the textbook was organised on the basis of topics, themes, notions and functions, rather than language skills, grammatical structures, tests or tasks. The themes included in the textbook were seen as related to those found on the exam, and the predictability of each chapter (as each was organised in a similar way) would be easy to follow and reassuring to some students, while others might find the same organisation monotonous (depending on the personality of each student). The step-by-step approach to skills development apparent in each chapter was seen as a defining characteristic of the textbook. Although numerous language features were presented in the textbook (recognition of sounds, pronunciation of sounds, stress and intonation, collocations and pronunciation), the teachers seemed to indicate that more emphasis was placed on grammar, linking words and expressions, and word formation, while less was placed on notions and functions, connotations and idioms. Teachers also felt that reading, dictionary, and study skills were represented to some extent in the textbook, as were question types, organisation structures for writing, and parts of speech. The organisation of the textbook was described as comprehensive and appropriately scaffolded (eg, early units presenting foundation knowledge and skills on which later more challenging units were based). The study skills section was identified as quite useful, while it was also pointed out that some of the grammar explanations could be clearer.

There was broad agreement among the teacher respondents that the enabling skills listed were covered in the textbook (including items 4.1–4.12, see Appendix 4). There were mixed responses regarding the coverage of making inferences (4.7), evaluating evidence (4.8) and recognising roles (4.10). Additional skills that were identified as being developed in the textbook included guessing meaning from context, skimming/scanning and study skills. Regarding question and tasking techniques, teachers indicated that skills related to multiple/dual choice, true/false, gap filling/completion, sequencing, paraphrasing, and summarising question types were often presented in the textbook. The results were mixed concerning substitution, open-ended, note-taking, and correcting/editing question or task types. Less emphasis was placed on conversion and matching type questions/tasks. Classification, and again, guessing meaning from context and skimming/scanning, were question or task types that were not included explicitly on the questionnaire but which teachers identified as being present in the textbook. Considering both skills and question or task types covered by the book, teachers indicated that the items included were relevant to, and supportive of, the skills necessary for the IELTS Test. From a communicate perspective, tasks in the textbook provided opportunities for students to engage in pair communication, and report and essay writing. However, teachers indicated that activities such as games and puzzles, surveys or project work, creative writing and reading for pleasure were not included.



The text types most often identified by the teachers reflected the academic or professional nature of the textbook (lecture/talks, textbook/journal article, discussion, face-to-face conversation, manual/brochure, advertising, maps/charts/tables/graphs, and interviews). Other text types more closely associated with ‘general daily activities’ (fiction, radio/TV reports, telephone, and email) were reported to be less frequent in the textbook. Likewise, the topics of texts in the materials were similar to those found in academic or professional settings: health, physical environment, education training, science, travel, world of work, social environment and moral issues. Topics more commonly found in general communication course books (daily routines, shopping, food and drink, customs) were present but not emphasised.

Regarding the authenticity of the listening and reading texts and tasks, the teachers recognised and identified a mixture of scripted (listening)/adapted (reading) and authentic texts in the materials. It seems that the teachers recognised and accepted that such a mixture was necessary in order to facilitate the types of tasks needed to provide learning experiences related to target skills (ie, involving a balance between complexity of the task and difficulty of the textbook), and to provide texts/tasks along a continuum of difficulty to meet the needs of all students. Most teachers seemed to indicate that an appropriately balanced mixture of prepared and authentic texts had been achieved. In general, the listening and reading components in the textbook were viewed as adequate and relevant to the IELTS Test, although at times, these areas needed to be supplemented to avoid becoming monotonous and, in the case of the listening, some respondents felt that it might be too focused on British culture.

Concerning the writing skills and associated tasks presented in the textbook, here again teachers indicated the shortcomings of how the materials addressed the Writing sub-test’s Task 1, in that they were limited to ‘foundation’ (basic) exercises. Some teachers indicated that Task 2 activities were more appropriate for advanced learners, while one teacher indicated that it was better in general to avoid these tasks altogether. There was also criticism of the topics selected (ie, they were considered Eurocentric and not sufficiently international or local), and it was indicated that the writing tasks placed much more emphasis on the process of writing rather than on the completion of a piece of writing. Some students found this frustrating since they wanted to experience the sense of accomplishment gained by completing an essay. The Eurocentric criticism was also aimed at the speaking component of the textbook, although in general, these sections were very positively appraised based on their relevance to the IELTS Test and their balance of different tasks. As with teachers anywhere, those who responded to the questionnaire would have liked the textbook to include additional supplementary materials and activities.

Overall, teachers indicated that IELTS Foundation was a well balanced, academically oriented textbook, targeted at the intermediate level (IELTS band score 4.5–6.0), and included a broad range of relevant activities which correspond to the question types common to and skills necessary for the IELTS Test. Nevertheless, British/Euro-centrism, monotony and dated texts were common criticisms of the materials. Teachers did feel that the textbook makes significant progress towards familiarising students with the IELTS exam (although it was suggested that this could be done more explicitly), develops necessary skills, and provides effective activities and tasks.

Textbook 2: Action Plan for IELTS

As only three teachers evaluated Action Plan for IELTS, it is difficult to identify trends in their responses with any degree of reliability. However, some generalisations can be made about the textbook based on their responses and comments. Action Plan for IELTS can be characterised as a self-guided, ‘exam-driven’ textbook. While not always suited to classroom use, the book does provide a broad introduction to the IELTS Test, including the skills required and types of questions/tasks that make up the exam. To this end, the textbook focuses on grammar, punctuation, and linking words/expressions rather than idioms, pronunciation or functions. The book stresses the basic skills



that students need to increase their scores on the exam, such as identifying main points and overall meaning, predicting information, distinguishing fact from opinion, drawing conclusions and evaluating evidence. The question/tasking techniques covered in the textbook mirror those that comprise the exam: matching, paraphrasing, and gap filling/completion. In the opinion of the teachers who evaluated this textbook, the materials provide a sufficient, albeit simplified, overview of the questions/tasks that make up the actual IELTS Test. The book is quite straightforward, and, therefore, allows less opportunity for creative communicative activities, while writing (reports and essays), on the other hand, is emphasised. The textbook’s text types are appropriate to academic and professional contexts, and include discussions, maps/charts/tables/graphs, textbook/journal articles, lecture/talks and interviews. Likewise, text topics are also academically and professionally oriented: health, physical environment, science, social environment, and the arts. The topics are presented in a well-balanced mixture of authentic materials and scripted (listening)/adapted (reading) versions. Overall, the textbook addresses all skills (listening, reading, writing, and speaking) as they relate to the IELTS Test in terms of question/task types and accompanying skills. However, a common criticism across the skill areas is the brevity of the tasks, and, therefore, their lack of depth. In general, the textbook is seen as a sufficiently broad introduction to the IELTS Test, while more advanced learners would need to search beyond this book for more authentic materials to remain sufficiently challenged.

5.4 Focus group of IELTS examiners 5.4.1 Background to results The focus group discussion, held at IDP’s office in Phnom Penh in early June 2010, was audio-recorded and lasted 52 minutes. (See Appendix 5 for question prompts.) Moore explained the nature of a focus group with the key goal being to generate discussion amongst the participants. Six IELTS examiners participated: four were Australian, one was British and one was Burmese. They ranged in age from 40 to 60 years old, with an average of 55; and their years of experience as an IELTS examiner ranged from four to 15 years, with an average of 10.8 years. The audio-recording was initially transcribed by a professional transcriber; this transcription was then fine-tuned by Moore against the original recording.

5.4.2 Results All six of the focus group participants had been examining IELTS in Cambodia for many years, and two participants had also examined IELTS in Australia prior to their work in Cambodia. They reported differences in the mix of candidates (ie, in Cambodia around 95% of candidates are Cambodian nationals, whereas in Australia the candidate profile is much more varied); the age spread (ie, in Cambodia the typical age range of a candidate is between 16 and 25 years); in the proportion of Academic versus General Module candidates (ie, in Cambodia almost all candidates sit the Academic module); and in the ability range (ie, in Cambodia, the band range is much narrower than in Australia, where some candidates are very strong and others are very weak).

In discussing the impact of IELTS in Cambodia, the point was made by one participant that, because all of the participants worked at ACE, the view was a bit like ‘a frog in a well’. In other words, although they were well aware of IELTS as it impacted on their own workplace, they were perhaps not very aware of the impact of IELTS more generally in the wider world of Cambodia. Another participant noted that at least half of the IELTS test-takers each year in Cambodia were actually ACE students completing GEP Level 12 (for which sitting the IELTS Test is an integrated component). Thus, IELTS and ACE have a mutually significant impact on one another.

One focus group participant stated the view that IELTS had very little impact in Cambodia because so few people took it in proportion to the country’s total population (of about 14 million), and because most Cambodians were not even aware of the IELTS Test.



When asked for comments about the notion of IELTS being a ‘high-stakes’ test in Cambodia, there was a consensus that, for some candidates who are government officials sponsored by their ministries to undertake pre-departure EAP training for overseas degree programs, the stakes are indeed high. If an appropriate band score is not achieved, then these candidates must return to their ministries and their old jobs, rather than head overseas for postgraduate study. This entails a huge loss of face for those unfortunate candidates. The focus group participants suggested that this was a rare event, perhaps occurring only once every two or three years.

One participant mentioned a case in which he claimed that four of his writing students (in an IELTS preparation course) scored band 5.5 on the Writing sub-test at one IELTS sitting. Six weeks later, at a subsequent sitting, they all scored 7.0 on the Writing sub-test. By way of explanation, several participants indicated that Cambodians could perform quite well on some types of Task 1 question, but poorly on others. Similarly, the candidates might be quite accomplished writers for Task 2 questions, but fail to respond to the whole question prompt, or fail to bring in sufficient world knowledge to contextualise their answer appropriately. These examples suggested that the IELTS Writing sub-test was not a consistently reliable measure of the writing skills of a Cambodian candidate. However, it could equally be the case that the Writing sub-test is an accurate measure, but the different prompts trigger different quality responses from the same candidate. This would be an interesting issue for future research to investigate.

When questioned about any specific impact that IELTS might have in terms of washback across the four macro-skills, one participant mentioned that IELTS, especially Task 2 writing, “forces them [Cambodians] to have a viewpoint…[in a developing country] where even having a viewpoint might be somewhat subversive. They prefer to sit on the fence.” Another participant noted that IELTS “exposes them [Cambodians] to a whole lot of, a huge amount of ideas and topics and things that they would not necessarily be exposed to, I think…And they have to sort of grapple with these ideas and then come up with some sort of coherent opinion and then the test sort of reinforces that.” A third participant noted how IELTS “forces them [Cambodians] to read, as Cambodians aren’t natural readers”. An example was cited of a candidate who, when asked what their favourite book was, said Headway (an English language course book, rather than, say, a particular novel); or when asked what their favourite movie was, said Tom and Jerry (a cartoon, rather than, say, an Oscar-winning film).

Some focus group participants noted that under-prepared candidates were easy to spot in terms of their lack of familiarity with the Speaking test format and with Task 1 writing requirements.

Some participants suggested that describing IELTS as a test of English proficiency was problematic because of “the fact that IELTS tests much more than English…it tests general knowledge, interpretation of statistics or graphs…critical thinking”. One participant noted candidates’ lack of knowledge about the world beyond Cambodia itself, and the difficulties this caused in providing exemplification to support an opinion. Related to this issue, several participants felt that IELTS was too “Eurocentric”.

One participant suggested that Cambodian test-takers should be given an extra 10 or 15 minutes for the Reading sub-test to make the IELTS Test fairer for Cambodians. Another participant lamented a lack of “the analytic outlook that westerners have” (eg, in relating the change of seasons and their weather patterns to actual months of the year).

To sum up, the IELTS examiners in this focus group offered various views about the impact of the IELTS Test in Cambodia. Although the impact on society at large was felt to be minimal, the impact on the lives of test-takers was felt to be substantial. In particular, the Test had the effect of ‘forcing’ Cambodian candidates to change their way of thinking in order to achieve appropriate IELTS scores to enable them to study overseas.



5.5 Focus group of parents/guardians 5.5.1 Background to results It was difficult to recruit parents and/or guardians to participate in the planned focus group. First, identifying parents and guardians who were interested in participating was not easy (although offering a participation fee of US$20 was helpful), and then arranging a mutually convenient time for the focus group to meet also proved to be a challenge. In the end, a group of just three parents/guardians attended the focus group: one was the mother of a test-taker (who also attended the interview as her mother’s personal interpreter); one was the older sister of a female test-taker; and one was the uncle of a male test-taker. None of the three participants appeared to speak English and virtually all of the interactions with them were conducted in Khmer. The focus group discussion, held at IDP’s office in Phnom Penh, was audio-recorded and lasted 36 minutes. (See Appendix 6 for question prompts.) Moore explained the nature of a focus group (something which none of the participants had ever previously participated in) with the key goal being to generate discussion amongst the participants.

5.5.2 Results The parents/guardians (hereafter ‘participants’) reported that the three test-takers under their care had each taken the IELTS Test one time only. When asked what they themselves knew about IELTS, one participant indicated that it was an internationally standardised test of English proficiency, while the other participants knew only that it was a test that enables someone to study overseas. When asked if they knew the composition of the Test, one participant claimed that it covered speaking, listening and writing skills, while the other participants were unaware of what was covered. None of the participants indicated any familiarity with any other English language proficiency test.

When asked if they thought the IELTS Test was expensive, none of the participants were aware of the actual cost (US$175). For two participants, the cost was ‘hidden’ in the GEP 12 (advanced English) course fee at ACE. The participant mother stated that even with three of her children studying English with the goal of eventually sitting the IELTS Test, the cost was not really a financial burden because it could be budgeted for. Another participant claimed that if the IELTS result was insufficient to meet the requirements of overseas study, then in a sense, the examination fee was forfeited. This participant also felt that by bringing the price down it might encourage more financially disadvantaged, but still capable Cambodians, to sit the Test. When further prompted whether he was in favour of a two-tier price system, he replied that, although it might be a good idea, it would be hard to implement in Cambodia because everyone would want the lower price. The third participant suggested that perhaps a lower fee could be charged if a candidate had a good recommendation from their school.

When asked if they perceived IELTS to be a burden in any other way, the participant mother felt that it was not a workload burden for her children because she trusted them to be able to study sufficiently hard to attain a good IELTS result. Another participant felt that there was a burden on the actual test day, and wondered why the Test could not be spread across two days to ease the stress and fatigue that a one-day sitting seemed to entail.

When asked about the impact they felt IELTS had in Cambodia, all three participants responded positively. The guardian sister claimed that IELTS had enabled her sister to obtain a scholarship for two years of study in the Netherlands, to be followed by four years of study in the USA. The participant mother noted that after taking IELTS, at least the student knows their level of English, and if it is sufficiently high, then it provides opportunities for overseas study. The guardian uncle stated that the impact of IELTS in Cambodia was “100% positive”.



To sum up, this small focus group of parents/guardians had quite a favourable, albeit limited, view of IELTS. They seemed to trust it as an accurate and unbiased measure of English language proficiency, and their only criticisms of it were fairly mild. Despite Moore attempting to facilitate a wider-ranging discussion, the focus group participants tended to answer the prompt questions in turn, with little reference to the immediately preceding turns of other participants. Thus, this particular focus group might more accurately be described as a “group interview” (see Edley and Litosseliti, 2010).

5.6 Interviews of embassy officials 5.6.1 Background to results The principal foreign countries that Cambodians typically target for overseas study are Australia, the USA and the UK. Other countries that also attract interest (though with somewhat fewer opportunities) include New Zealand, Canada and Singapore. The European Union (EU) also attracts some interest for its scholarships. The foreign embassies of these countries, including the EU, were targeted for participation in the IELTS impact study in Cambodia. The planned protocol for the interviews was adjusted from structured to semi-structured in order to facilitate each interviewee providing the most useful information for this study. (See Appendix 7 for question prompts.)

5.6.2 Results The participating foreign embassies in our study are set out in Table 12. In total, two First Secretaries and four Program Officers were interviewed.

Country Interviewee Audio-recording permitted 1 Australia Australian Yes

2 UK Cambodian No

3 USA Cambodian No

4 Japan Cambodian Yes 5 Sweden Swedish No

6 EU European Yes

Table 12: Summary of embassy officials interviewed

Neither Canada nor New Zealand had an embassy in Phnom Penh at the time of our data collection, so they were not approached to participate in this study. Other foreign embassies declined to participate stating that they were not familiar with, or did not use, IELTS results in their scholarship application evaluations. So, in the end, six rather than 10 embassies participated in the study. The interviews were semi-structured, allowing for question probes and follow-up questions. Perhaps not surprisingly, half of the interviewees did not permit audio-recording of the interview, either due to security concerns or citing confidentiality issues. In those cases, Moore and Heang both made hand-written notes during the interview, and then compared their notes within 24 hours to ensure they were in agreement. The audio-recordings that were made ranged from 21 to 36 minutes in duration. A summary of the results of the six interviews is set out in the following paragraphs.

All six interviewees claimed that IELTS results had been in use at their embassies for a varying number of years up to, and including, the time the interview took place. One interviewee stated that their embassy only accepted IELTS scores (and not TOEFL); another two stated that while both were accepted, most applicants used TOEFL because it was less expensive.



All interviewees stated that IELTS results were used as part of scholarship application formalities. The scores were accepted as ‘hard’ evidence (Rea-Dickins, Kiely and Yu, 2011) and used as accurate measures without any deep interpretation of the band scores presented. (One interviewee who did claim the need to interpret IELTS scores said that it was easy to do so.) All interviewees agreed that IELTS was a useful test, due to its “international” and “standardised” credentials. In terms of reliability, none of the interviewees could cite any instance in which it was subsequently found that a Cambodian studying overseas had an insufficient level of English proficiency to complete their studies, despite achieving an IELTS band score that permitted entry to their program of study.

All interviewees espoused the view that IELTS had the benefit of opening up overseas study opportunities for Cambodians. However, many also noted that US$175 was quite an expensive fee for Cambodians. (The annual per capita income in 2008 was US$768, United Nations, 2011). One interviewee questioned why, given its expense, the scores were not valid for longer than two years. Another interviewee commented that IELTS scores had the benefit of being usable to benchmark pre-departure English for Academic Purposes (EAP) programs linked to scholarships. A few interviewees perceived IELTS to be a ‘high stakes’ exam especially for Cambodian Government officials who might have only one opportunity to take the IELTS Test. Thus, their life chances (ie, overseas study and qualifications, and subsequent career advancement) hinged on whether or not they could achieve a suitable band score in their one and only attempt.

To sum up, the embassy officials were generally positive about IELTS as a test, and trusted/accepted that it was an accurate measurement instrument for candidates’ English proficiency.

5.7 Interviews of employers 5.7.1 Background to results Prospective employer participants were identified by the project’s research assistant. Given the prevalence and importance of international NGOs and UN bodies in Cambodia, we decided to extend the category of ‘employer stakeholder’ beyond private sector employers as originally planned. Heang contacted prospective participant organisations via telephone or email and an Expression of Interest flyer was forwarded to them by email. Some of the targeted employers declined to participate but in the end, seven agreed to be interviewed. The planned protocol for the interviews (see Appendix 8) was adjusted from structured to semi-structured in order to facilitate each interviewee providing the most useful information for this study. The business activities of the participant employers are summarised in Table 13, and a good representation was achieved of the different types of employer requiring high levels of English proficiency amongst their key employees.

Business type Interviewee Gender Cambodian 1 Bank Head of Human Resources F Yes 2 Accounting firm Operations Manager M No

3 Hotel Human Resources Manager F Yes

4 Private university Vice-President M Yes

5 Translation and interpreting Managing Director M Yes

6 International NGO Administration Officer M Yes

7 UN organisation Human Resources Manager M Yes

Table 13: Summary of employer stakeholders interviewed



5.7.2 Results The interviews were all audio-recorded, and ranged from 10 to 34 minutes in duration. A summary of the results of the seven interviews is set out in the following paragraph.

None of the employers claimed to formally use IELTS as part of their hiring practices. However, if IELTS results were reported by a job applicant, several employers said the results would be “noticed” and possibly used as corroborative evidence of language proficiency. While most of the employers acknowledged the usefulness of a standardised English proficiency test, such as IELTS represents, many used their own in-house language assessment instrument (usually involving an interview and a role play in English). They generally felt that IELTS was “positive” for Cambodia and many claimed that it had a good reputation as a standardised test used in Cambodia. Most employers linked IELTS to overseas study (particularly in Australia) rather than to general English proficiency (and the General Training module). Several employers suggested that the IELTS Test could help motivate students to improve their English and, therefore, have a positive impact on English proficiency for test-takers. All the employers felt that US$175 was expensive for Cambodians, and a few noted that if it were cheaper, it had the potential to become a commonly used requirement for employment.

5.8 Interviews of returned graduates from overseas 5.8.1 Background to results Using the Australian Alumni Association of Cambodia (AAA-C) network, we were able to identify returned graduates from overseas and approach them about participating in an interview. The planned protocol for the interviews (see Appendix 9) was adjusted from structured to semi-structured in order to facilitate each interviewee providing the most useful information for this study. There were seven participants, as shown in Table 14, holding positions ranging from accountant to deputy director.

Gender Current employer 1 M National Bank of Cambodia 2 F National Bank of Cambodia

3 M Ministry of Economy and Finance

4 F Ministry of Economy and Finance

5 M Ministry of Rural Development

6 F Ministry of Health

7 M Council of Ministers

Table 14: Summary of returned graduates from overseas

5.8.2 Results The interviews were all audio-recorded, and ranged from 14 to 44 minutes in duration. It is worth noting that all the participants in this category of stakeholder were government officials. (Indeed the Australian Government, through AusAID, is particularly supportive of Cambodian Government officials seeking scholarships to study overseas.) We were unable to interest any private sector employed returned graduates to participate in the study. A summary of the results of the seven interviews is set out in the following paragraphs.



All of the interviewees had a fairly positive view of IELTS and all agreed that IELTS helped develop skills needed for success at university. One interviewee stressed that Cambodian students often cheat on tests, sometimes without punishment even if caught, but in the case of the IELTS exam being invigilated by foreigners rather than Cambodians, test-takers “don’t dare to cheat”. Thus, IELTS was seen as an exam whose results were not distorted by cheating.

Some interviewees lamented the fact that IELTS is not discipline-specific and, therefore, felt that they were misspending time on general academic English at the expense of specific academic English. (This sentiment aligns with findings in Rea-Dickins et al, 2007, which studied ‘successful’ IELTS candidates after entering higher education institutions.) Most interviewees had taken the IELTS Test only once or twice, and most considered it to be quite expensive for Cambodians. In terms of criticism of IELTS, several interviewees complained about the impact of different topics (especially in the Reading sub-test), claiming that their score could fluctuate a lot depending on the particular reading topics, and, therefore, the Reading sub-test was not always an accurate reflection of their reading skills ability.

5.9 IELTS preparation course classroom observations Two teachers agreed to participate in classroom observations. One was a teacher of GEP Level 12; the other was a teacher of the 10-week IELTS Preparation course. The former was not (and had never been) an IELTS examiner; the latter was an IELTS examiner of nine years’ standing. The GEP lesson was observed in early June, while the IELTS lesson was observed in late July. Both lessons were audio and video-recorded.

5.9.1 GEP Level 12 lesson observation The GEP Level 12 lesson observation took place 6.00–7.30 pm in Week 8 of the 10-week term. Thus, the students were within a few weeks of sitting the end-of-course IELTS exam for the first time. Such students typically achieve an overall band score of 5.0, with no less than 4.0 in any skill area. The teacher was a middle-aged British national, with approximately six years ELT experience, including several years teaching at ACE. There were 20 students enrolled in this class, 11 males and nine females, ranging in age from 14 to 22 years.

The lesson proceeded quickly as the teacher used a variety of handouts and made frequent changes to seating arrangements for the various tasks undertaken. The entire lesson was conducted in English. Interestingly, the term “IELTS” was never mentioned throughout the lesson and, therefore, could probably be described as the ‘elephant in the room’. It was not clear why the exam was not mentioned by name, but the focus of the class work was directly relevant to both the listening and writing components of the IELTS Test.

Data from the lesson observed could be triangulated with data collected from other sources, namely the teacher’s own responses to the teacher questionnaires, and student responses to the TP questionnaire. What follows is a summary of how these data are generally corroborative and, thus, enhance our study’s reliability.

The main focus of the particular lesson was on listening skills, with a secondary focus on Task 2 writing and some vocabulary development. This lesson was therefore consistent with TP student responses from this cohort indicating that more time was spent on macro skills (especially writing) than on vocabulary or grammar.

The materials used by the teacher in the observed lesson were a combination of published and self-prepared. The published materials were from the book Improve your IELTS Skills, although the teacher claimed to use the textbook Action Plan for IELTS (which TP respondents from this class also reported using). (Because none of the materials used in the observed lesson were from Action Plan for IELTS,



no triangulation of the teacher’s textbook evaluation questionnaire responses could be matched to the observed lesson practices). Interestingly, the TP student respondents (N=8) from this class were rather vague about what supplementary materials were used by the teacher. Three did not answer this question, while of those who did, a few mentioned handouts and dictionaries. Only one student mentioned some of the supplementary materials used in the observed lesson: listening CD and handout; reading handout; and Task 2 writing topics handout.

Despite the teacher not being a certificated IELTS examiner, his responses to questionnaire items 41-48 (Appendix 3) concerning knowledge about the Test showed that he was well-informed and had an accurate view of the Test.

In the teacher questionnaire about preparing students for IELTS, item 33 asks in what way IELTS influences the way teachers teach. The observed teacher indicated that he teaches “according to the marking criteria, eg, by trying to eliminate grammar and spelling errors in the listening”. This was observed in practice in the observation lesson.

Also in the teacher questionnaire, item 40 asks respondents to rank the IELTS sub-tests in order of difficulty for most of the teacher’s students. This teacher ranked Listening and Writing as the most difficult. In questionnaire item 31, asking about how the Test influences the lesson’s content, the teacher stated that he “spent more time on the difficult areas”. Both of these responses were borne out in the observed lesson, which focused primarily on listening and writing. The students who were also TT respondents (N=7) also answered the same question in their questionnaire, and indicated that Writing was most difficult, while both Reading and Listening were also perceived as being difficult.

The teacher’s responses to teacher questionnaire item 58 concerning which of a range of activities take place in the teacher’s normal IELTS-preparation class all matched with what was observed in his teaching. For example, for Listening, 58L.1: Reading questions and predicting what listening texts will be about; 58L.5 Reading questions and guessing the types of answer required; and 58L.6 Practice in recognising previous information repeated in different words. For Writing, 58W.2 Describing graph/process/statistical data; 58W.3 Learning how to organise essays; 58W.4 Practising using words or phrases to organise a written text; 58W.7 Planning written answers to test questions; and 58W.9 Writing parts of test answers.

5.9.2 IELTS preparation course lesson observation The observed IELTS preparation lesson took place in Week 3 of the 10-week term. The teacher was a middle-aged Australian national with close to 30 years of teaching experience, including a decade at ACE (of which eight years involved teaching IELTS preparation courses). He was also a certificated IELTS examiner. Although 19 students were enrolled in the class, only 10 attended this particular lesson, seven males and three females. One of the 19 students was Korean, while all others were Cambodian nationals. The first 90 minutes (up to the tea break) was observed, but the lesson continued for a full 3 hours (5.30–8.30 pm).

The opening 90 minutes of the lesson focused first on Task 2 writing and then on the Speaking test interview. In contrast to the observed GEP Level 12 lesson, the IELTS Test was constantly referred to throughout the duration of this observation. The teacher provided explicit input for question interpretation and advice on how to structure a written response, and he also referred explicitly to the publicly available IELTS descriptors for various assessment criteria. Thus, the students could be in no doubt as to the centrality of the actual IELTS Test in relation to their preparation course endeavours.



As with the GEP Level 12 observation, data from the observed IELTS preparation lesson could be triangulated with data from other sources, namely the teacher’s own contributions to the IELTS examiners’ focus group discussion. Unfortunately, due to the 30 September cut-off for our data collection, no triangulation could be performed with student responses to the TP questionnaire. What follows is a summary of how the teacher’s contribution to the focus group data is generally corroborative of the observed lesson, and thus enhances our study’s reliability.

The materials the teacher used in the lesson were a combination of published sources and self-prepared handouts. The published materials were from the book, The IELTS Tutor, rather than the textbooks commonly used by IELTS preparation course teachers, Action Plan for IELTS and IELTS Foundation. As this teacher did not complete either of the teachers’ questionnaires, we cannot triangulate his observed materials usage with those instruments.

In the focus group, the teacher mentioned the issue of test familiarity as a factor in IELTS scores, agreeing that it could account for half a band score for first time test-takers. His awareness of the Test seemed to foreground a very practical approach to what Cambodian students preparing for the IELTS exam needed to understand about the Test. Much of his focus group contribution and classroom practices were concerned with the Writing sub-test. He commented, for example, that the Task 1 writing was “very artificial” and seemingly inauthentic: “When was the last time you had to take a visual in a test? It’s a rare thing”. He added that it was not particularly helpful for some fields of academic study, such as Humanities. Task 2 writing on the other hand was “quite good”. Indeed, his observed teaching of Task 2 writing indicated a positive attitude towards this component of the IELTS Test.

A particular view expressed by the teacher in the focus group concerning the issue of ‘having a point of view’ in Task 2 writing was also emphasised in the observed lesson. “It forces them to have a viewpoint, especially Task Two [Writing sub-test]…They prefer to sit on the fence, you know, nothing is black and nothing’s white, everything’s grey and everything’s in the middle and everything’s okay. It’s rare to have an extreme viewpoint either way. So it [Task 2] forces them into that because the [exam] descriptors say…‘have a viewpoint’. A viewpoint doesn’t mean ‘I don’t know’ or ‘I’m not sure’ or ‘Work it out for yourself’.”

The teacher also stated that Cambodian test-takers were “okay if they’re writing generalisations, but once it comes to specific exemplification, then you need some sort of raw [world] knowledge”. He further commented on the general lack of world knowledge amongst Cambodian test preparation students: “Their general knowledge is almost non-existent outside the borders of Cambodia. They know everything inside Cambodia but on a world level, it doesn’t exist…Where I think they are disadvantaged: a lot of the students don’t have the world knowledge at all to provide exemplification for what they’re saying”. In his observed teaching, the teacher helped broaden students’ knowledge through asides when discussing whatever matter was at hand.

In the focus group, the teacher also commented on the academic writing tradition being quite different between English and Khmer, and lamented the fact that this wasn’t well known by either IELTS preparation course teachers or their students. In his observed lesson, he explicitly made the same point when modelling the structure of a Task 2 Writing response on the whiteboard.



6 RESEARCH QUESTIONS ANSWERED 6.1 Research question 1 What impact does IELTS have on education and society in Cambodia?

This is the overriding research question and, as such, can only be properly answered when all other research questions have been addressed. Therefore, we defer providing an answer until the end of the Discussion in Section 7.

6.2 Research question 2 What are the profiles of the candidates taking the IELTS Test?

[Source: TT questionnaire]

In general, IELTS test-takers in Cambodia can be described as comparatively well educated, professionally or educationally goal-oriented, and people who make use of English on a regular basis in their professional and/or personal circumstances. Indeed, the profile of our sample reflects Cambodia’s emerging middle class, which is quickly becoming a relatively upwardly mobile social group as the country rapidly develops. Our sample of test-takers is almost exclusively comprised of Cambodian nationals (with one Korean and Filipino included), and is relatively gender balanced (male, 58%; female, 42%). They represent a mixed age group, with 27% between the ages of 16 to 19 years, 17% between 20 to 24 years, 34% between 25 to 29 years, and 22% between 30 to 46 years. The majority is employed (57%) while the remainder are students (university 26%; high school, 17%), with a small number pursuing Masters degrees (12%), and one pursuing a PhD. The majority use Khmer at home (87%), with only 4% using a mixture of Khmer and English, and a small number using combinations of other languages (Khmer, Chinese, Tagalog, English, Korean) in their home environment. Only 10% live(d) with a native speaker of English as a child.

The experience of the IELTS candidates in our sample demonstrates the growing emphasis on English in the Cambodian educational system. A large majority of test-takers received, or are receiving, English instruction in secondary school (77%) and university (76%). Additionally, 80% have enrolled in extra language classes outside the formal education system. Some started English language study at a young age, with 29% beginning at the primary level, and 6% at the kindergarten level.

During their daily life, our sample of test-takers claimed to use English frequently, responding that they often or sometimes used English for socialising, contacting friends overseas, or simply through media such as television and the internet. Test-takers who were older professionals reported using English for work more often than the younger test-takers, who reported using English when studying specialised texts such as university textbooks. Overall, there seemed to be a comparatively high frequency of English use amongst test-takers through their social, professional and educational activities. Moreover, 12% reported having already lived and studied in a foreign country, most often for a short time (1-6 months), and most often in Australia or New Zealand.

Although most of the test-takers in our sample had only taken the exam one time (76%), 20% were sitting the exam for a second time. One third (35%) were sitting the IELTS Test to prepare or qualify for study abroad opportunities, and another third (32%) to qualify for educational scholarships. Almost all (99%) reported taking the Academic module of the IELTS exam. Only 5% were sitting the exam for employment purposes, and even fewer for immigration purposes (1%). The majority (82%) had not taken any similar English language assessment test, but of those who had, the TOEFL exam was most common (16%).



6.3 Research question 3 What is the washback of the IELTS Test on courses preparing candidates to take it?

[Sources: Teacher questionnaires; TP questionnaire; Classroom observation]

Not surprisingly, the IELTS Test and its content, structure and associated testing strategies all inform the test preparation courses we investigated. All teachers of IELTS Test preparation courses in the current study indicated that the IELTS Test not only impacts their decisions regarding the activities and materials to be used in their classes, but also motivates their students. While there were mixed opinions regarding the benefit of the additional intrinsic and extrinsic pressure placed on students related to performance on the Test, teachers indicated that they believed that students did achieve the course goals and improved their language skills in their IELTS preparation courses.

Almost all teachers in our sample stated that the Test informs their choice of teaching content as they address and develop specific skills related to students’ performance on the exam. The type of question, the content (in particular related to writing tasks), and the exercises are all chosen to specifically prepare students for test tasks, expose them to typical test topics, and to make them aware of the Test framework. Types of practice activities in class (eg, having students read questions, anticipate topics of listening passages, and being familiar with the types of answers required for specific types of question) were all heavily influenced by the Test.

Even though most teachers indicated that they continue to follow a communicative approach in terms of their teaching methodology, the IELTS Test also influences the teachers’ actions in the class. For example, teachers did indicate that their classes had become more systematic and more similar to a workshop environment and, as a consequence, less communicative due to the emphasis on test-based task-specific skills. Some teachers were concerned with the possibility that such a systematic, repetitive and formulaic approach to test preparation might limit the creative development of their students. Teachers also noted that although students were generally well-motivated by the IELTS exam, many had unrealistic expectations of how much they could improve their scores on the basis of a test preparation course.

In terms of specific learning on IELTS Test preparation courses, vocabulary development and general knowledge related to global issues were two areas teachers identified as important for test preparation and which they always included in their courses. Listening skills development was often identified by teachers as necessary and informed by the Test, and therefore included in the curriculum of their courses. While listening strategies (eg, identifying repeated words, pre-reading questions, predicting content, etc) were seen as important, authentic listening passages and situations (eg, group discussion techniques and using information from lectures in writing assignments) were deemed as less important. Reading skills (eg, analysing text organisation, identifying main and supporting ideas, interpreting graphic information, reading efficiently and effectively) were also included because of their relevance to the Test. Concerning writing skills, emphasis was placed on the organisational structure (eg, essay organisation, paragraphing, using transitional signals, use of transition signals, etc), the goal (or type) of writing tasks, and the writing process. Strategies and skills related to improving students’ speaking skills, including using examples and organising ideas, were also identified and included in teachers’ courses. In some cases, utilising practice tests formed a basis for some class activities.



In addition to a focus on developing vocabulary, world knowledge and the four macro skills, teachers also ensured time was spent developing critical thinking skills. Some teachers also encouraged students to develop effective learning skills and strategies and become more autonomous learners. Our study also enquired of students how the IELTS Test impacted on their test preparation courses. They were specifically asked about the occurrences of different types of activities related to different skill areas. For listening skills development, activities related to questions associated with listening passages were most common, eg, reading the questions and predicting what listening passages would be about (91%); reading questions and guessing the type of answer required (92%). Listening to and taking part in seminar or workshop activities were less common (34%). For reading skills development, identifying main ideas and increasing reading speed (90% and 86%, respectively) were important reading skills that were included in class activities. With regard to writing skills, over 95% of students indicated that describing statistics, graphs and diagrams, learning how to organise essays, and making use of transition words were included in their classes. Other aspects of writing were less common, eg, learning how to write in different styles (79%); planning written answers to test questions (62%); and copying out good paragraphs and model answers (61%). For speaking skills development, practising making a point and providing supporting examples (85%) and using words or phrases to organise your speech (80%) were the most commonly reported activities, while other frequently reported activities included engaging in group discussions or debates (75%) and using filler words to cover silences in your speech (71%).

Regarding class activities specifically related to the IELTS exam, one third of students in our sample reported using practice tests in their preparation classes. Other related activities reported were techniques for taking the exam (18%), information about the exam (17%), and looking at past papers (17%). Numerous specific test-taking techniques were also mentioned as being included in test preparation classes (eg, time management, skimming/scanning, predicting answers, understanding exam instructions, etc).

From the summaries above of how teachers adapt their course content and teaching activities to suit the perceived needs of students preparing to sit the IELTS Test, and how students perceive the content and activities of their test preparation courses, it is clear that there is substantial washback from the Test to the course. At another level, however, we can also see very clearly the impact of the IELTS Test on ACE’s courses preparing candidates to take the exam in Cambodia. In the case of the GEP Level 12 completion, the IELTS Test (Academic module) is the benchmark used for measuring students’ English language proficiency. The target score is 5.0 overall, and no score below 4.0. This use of the IELTS Test suggests that the GEP beyond Level 10 is more of a transition to academic English than an extension of ‘general English’. After all, the IELTS Test (Academic module) is intended for measuring English proficiency for academic, not general, purposes. The other IELTS preparation courses offered at ACE (ie, IELTS Orientation and IELTS Preparation course) are of 15 and 45 hours duration respectively, and are aligned completely with the IELTS Test. Indeed, the teachers ‘teach to the test’, and its washback affects virtually every minute of how class time is spent.



6.4 Research question 4 What are the profiles of the participants who have already taken the IELTS Test?

[Source: Historical profile from 1992-2010 provided by IDP Education (Cambodia)]

Our questionnaire instrument for test-takers asked respondents, in an open-ended question item, to indicate when they took the IELTS Test and to include all dates if they had taken the Test more than once. Our findings showed that 14% of TP and 33% of TT respondents claimed to have taken the Test more than once. However, given the open-ended nature of this question, we are not confident that a profile of these respondents would be an accurate description of that sub-category of test-taker. To collect an accurate account of such information would entail a more prescribed question item (or items) that could be verified through a cross-check of IELTS Test records. In lieu of this information, however, we can report a summarised historical profile of test-takers in Cambodia. Table 15 provides the overall profile of more than 10,000 IELTS test-takers since the Cambodian test centre’s establishment in 1992 until the end of 2010.

Module Academic 89.7%

General Training 10.3%

Gender Male 58.5%

Female 41.5%

Age of test-takers

16-20 years 27.7%

21-25 years 28.9%

26-30 years 20.5%

31+ years 19.7%

Nationality Cambodian 96.1%

Other 3.9%

Overall band score

5.0 23.7%

5.5 23.4%

6.0 15.8%

6.5+ 13.3%

Table 15: Summarised historical profile of IELTS test-takers in Cambodia 1992–2010

6.5 Research question 5 What is the impact of IELTS on the participants who have taken the Test (including Cambodian graduates returned from overseas study)?

[Source: Returned graduate interviews]

The impact of IELTS on participants who have taken the Test is profound. The Test is the gateway to both scholarships and overseas study (particularly at tertiary level). In this sense, test-takers are very strongly motivated to prepare properly for the exam (eg, by enrolling in IELTS preparation courses) and to do their best to attain the highest scores they can achieve. For those test-takers who succeed, the reward is potentially very positive, with many attractive career and life-altering possibilities. On the other hand, for those test-takers who do not succeed in achieving the required IELTS scores, the reality they face is one of uncertainty and financial stress (eg, if they decide to continue studying and re-sit the Test) or possibly loss of face among their work colleagues (eg, if they have no opportunity to re-sit the Test and therefore no chance to study abroad). Although the latter case is rare, we did hear anecdotal evidence of this occurring from time to time in Cambodia, and this exemplifies the status of IELTS as a ‘high-stakes’ test for some test-takers.



The data collected through interviews with graduates returned from overseas all confirmed the positive impact that success at IELTS had ultimately meant for them. They all had benefited from obtaining scholarships to study in Australia, and then were able to return to jobs in their various government ministries upon successful completion of their studies. For many, they also received promotion and higher pay upon returning.

6.6 Research question 6 What are the profiles of IELTS examiners in Cambodia?

[Source: ACE-prepared list]

There were 10 IELTS examiners in Cambodia during the period of data collection in 2010 – four women and six men, ranging in age from 35 to 65 years old, with an average of 51 years. Six were Australian, two were British, one was Burmese, and one was Canadian. Their experience as English teachers ranged from seven to 30 years, with an average of 19.6 years, while their experience as IELTS preparation course teachers ranged from 0 to 9 years, with an average of 6.2 years. Their approximate years as certificated IELTS examiners ranged from at least two to 15, with an average of 9.3 years.

6.7 Research question 7 What is the impact of IELTS on the examiners in Cambodia?

[Source: IELTS examiners’ focus group]

The focus group of IELTS examiners did not explicitly express any impact that the Test had on them as IELTS examiners. As most of them were also IELTS preparation course teachers, they tended to speak of the impact the exam had in terms of their experience of preparing students for the Test and the washback it created in that context.

These examiners did speak quite openly about what they perceived as ‘flaws’ in IELTS as a measure of English proficiency (eg ,that it was ‘Eurocentric’, required ‘world knowledge’, and ‘critical thinking’). Others noted that test-taker performances could vary dramatically for the Writing component, for example, from 4.5 to 7.0 within a six-week period. These criticisms suggest an inherent tension for examiners in being aware that the IELTS Test might not be perfect on the one hand, and needing to administer it as though it were perfect on the other.

The focus group discussion indicated that IELTS examiners were professional in their approach to their examining tasks and, though certainly sympathetic to Cambodian test-takers, they were careful to follow the prescribed testing protocols in all their interactions with test-takers. Indeed, the fact that IELTS is an international standardised exam seemed to provide a strong sense of the need for the Cambodian testing to be as rigorous as that of any other test site worldwide. This, in turn, seemed to provide some psychological reassurance to examiners that, by following the protocols, they are properly acquitting their role as examiners, and the notion of the exam having a positive or negative impact on them becomes somewhat irrelevant. Again, none of the examiners stated that the IELTS Test had either a negative or positive impact on them as examiners.



6.8 Research question 8 What are the profiles of the teachers preparing candidates to take the IELTS Test?

[Sources: Teacher questionnaires; ACE-prepared list]

The 11 IELTS teacher respondents were drawn from a total pool of 18 such teachers at ACE involved in IELTS preparation course teaching (across the range of ACE’s GEP (Levels 11A, 11B, and 12) to short and long IELTS preparation courses). Of these 18 teachers, four were female and 14 were male; six were Australian, four were British, two were American, two were New Zealanders, and one each was from Canada, Ireland, Singapore and Slovenia. Their ages ranged from 31 to 62 years old, with an average of 45.5 years. Their experience as English teachers ranged from 3.5 to 29 years, with an average of 13.8 years; and their years as IELTS preparation course teachers ranged from three months to 13 years, with an average of 4.5 years. Five were IELTS examiners as well as IELTS preparation course teachers.

The IELTS preparation course teachers who completed the questionnaires for the current research project were comparatively experienced (15.4 years on average) and qualified (ie, a majority had completed BA degrees in addition to certification or diplomas in Teaching English as a Foreign Language (TEFL), or equivalent). Most had previous experience teaching IELTS preparation courses.

6.9 Research question 9 What is the washback of the IELTS Test on the teachers preparing candidates to take the Test?

[Sources: Teacher questionnaires; classroom observation; IELTS examiners’ focus group]

As in the case of Research Question 7 (which considered the impact of the IELTS Test on IELTS examiners), this question is difficult to answer conclusively. The main difficulty arises from trying to separate the teacher’s teaching procedures in this case from the textbooks and other materials they use (which are covered by Research Question 11). However, unlike the case of the Sri Lankan impact study (which did not involve the IELTS Test), in which Wall and Alderson (1993) reported that local teachers did not adapt their teaching methods to suit new content, the Cambodia study shows clear evidence of teachers adapting their teaching to suit the IELTS content. This can be explained by the fact that the teachers in the Cambodian context were all native or near native speakers, and many were also IELTS examiners, which would make adapting teaching to the IELTS Test comparatively easy.

As can be seen in the Research Question 7 discussion above related to the washback effect of the IELTS exam on the content of courses, the influence of the Test dominates the materials and curriculum development of these courses, and therefore clearly impacts the teachers of these types of courses. Moreover, as an international standardised benchmark exam, the IELTS Test also has a wider sphere of influence including washback on teachers who work with students in more general academically focused English classes that are not specifically connected to the IELTS exam through course title or content. In our sample, teacher respondents indicated that if the IELTS Test were not required, the majority of teachers (N=8, 73%) would not have prepared students in the same way for their future studies, but rather would have focused on more traditional English for Academic Purposes (EAP) study skills (ie, research skills, summary writing, critical thinking skills, note taking etc). These views are testament to the powerful influence the IELTS Test has on both course content and teaching activities.

The classroom observations of two IELTS preparation lessons (one GEP Level 12; one IELTS preparation course) provide useful evidence of washback from the IELTS Test on teachers preparing students to take the exam. In these lessons, the teachers focused on: exam structure and exam question types; test-taking strategies (eg, time management); and how to process exam-type questions. They also assigned homework to maximise the value of classroom time and to avoid long stretches of time that are not interactive. There was also evidence that they had looked for materials and activities that



were most appropriate for their students, and this involved designing new materials for specific needs. This action is particularly needed where the purpose for taking IELTS as part of GEP Level 12 (as a capstone) is quite different from taking the IELTS for overseas study. The former’s target score is IELTS 5.0 while the latter’s is 6.0 or higher. Also, the student cohort of GEP Level 12 is younger, including substantially more teenagers, than in IELTS preparation courses, and the teacher needs to take this factor into account in presenting materials and related activities.

From the lesson observations, it was also clear to see that the teachers often had a focus on the accuracy of their students’ work during class time. They also devoted time to developing students’ critical thinking skills (eg, by demanding reasons for particular responses). Through the selection of reading and listening materials, the teachers also provided opportunities for their learners to increase their ‘world knowledge’ beyond what Cambodians would typically learn in their domestic schooling.

One of the observed teachers provided a model structure for a Task 2 essay that could be seen as a kind of IELTS Test ‘artefact’ (ie, something that, without the IELTS Test, the teacher would probably not have taught). Furthermore, the requirements of Task 2 Writing raised an interesting issue concerning the ‘truthfulness of response’ for the same teacher. He emphasised that to answer Task 2 questions does not involve being honest or truthful, but just to take a position and argue it consistently.

The data collected from the IELTS examiners’ focus group also shed light on the IELTS Tests’ washback on IELTS preparation course teachers, since most of the focus group participants were also IELTS preparation course teachers. They noted, for example, that IELTS Writing required the development of world knowledge and an ability to form an opinion about a matter (even if it was not a ‘truthful’ opinion) in order to be able to provide a satisfactory response. These teachers felt, and many language teachers would agree, that these are not language proficiency skills as much as general education and critical thinking skills. In this sense, these teachers are finding that IELTS preparation teaching in the Cambodian context involves much more than it might in, say, a European EFL context where candidates would already possess a level of world knowledge appropriate to IELTS questions, and already have developed critical thinking skills appropriate for tertiary level education.

6.10 Research question 10 Which textbooks and other materials are used on IELTS preparation courses?

[Sources: Teacher questionnaires; TP questionnaire; classroom observations; ACE teacher resources room inspection]

The responses from the teacher questionnaire concerned with materials evaluation (Appendix 4) indicated the use of three textbooks in IELTS preparation courses:

1. Roberts, R, Preshous, A, and Gakonga, J, 2004, IELTS Foundation, Macmillan Education in association with The Open University, Oxford

2. Jakeman, V, and McDowell, C, 2006, Action Plan for IELTS, Cambridge University Press, Cambridge

3. O’Connell, S, 2002, Focus on IELTS, Longman, New York.

This relative conformity in using the same textbooks is to be expected given that all respondents were employed at the same institution having a unified curriculum. Also, the availability of additional IELTS preparation textbooks is more limited in the Cambodian context than in many countries.

The GEP Level 12 lesson observed made use of two published course books, “Listening and Speaking Skills” and “Reading Skills”, both from Macmillan’s Improve Your IELTS Skills series. The teacher also introduced some supplementary Task 2 writing materials that appeared to be self-produced.



The IELTS preparation course lesson observed used the course book The IELTS Tutor, published by the University of Queensland, and teacher-produced supplementary writing materials.

During one of the data collection visits to Phnom Penh, Moore explored the ‘Teachers’ Resource Room’ at ACE, specifically to see the extent and quality of the IELTS books and materials available to IELTS preparation course teachers. He counted more than 40 different books (excluding multiple copies or sets containing a teacher’s book, a student’s book, and a homework book). There appeared to be an impressive range of books, but some were dated as much as 10 years old. Moore was told that other IELTS-related teaching materials were available but were hidden from public view so as to reduce their usage and familiarity to students (an important consideration for mock testing).

6.11 Research question 11 What is the washback of IELTS on these preparation course materials?

[Sources: Teacher questionnaires; classroom observations]

The titles of the textbooks used by teachers in the IELTS preparation courses indicate that they were specifically designed to prepare students for the exam. These textbooks include language skills activities (ie, reading, writing, listening, speaking), practice tests and test-taking strategies that are directly relevant to students’ performance on the test.

Teachers reported that the textbooks included useful advice, strategies and test-taking techniques. Overall, IELTS Foundation and Action Plan for IELTS were characterised by teacher respondents as textbooks that specifically address the skills and content necessary for: students to become familiar with the question and answer types on the Test; the test-taking skills and strategies useful to help students improve their performance; and the content to which the students need to be exposed in order to perform successfully on the Test.

From the test preparation course teachers’ perspective, and specifically regarding activities and skills included in the textbooks, teachers found both the skills and question or task types included in the textbooks were relevant to, and supportive of, the skills necessary for the IELTS exam. The teachers positively reviewed the writing sections in the textbooks. While the listening and reading components of the textbooks were sometime criticised as potentially repetitive and uninteresting, as well as Eurocentric or too focused on British culture, in large part these parts of the textbooks were also seen as relevant and useful in preparing students for the Test.

Classroom observations also provided useful evidence of the washback of the IELTS Test on preparation course materials. For example, the course materials were found to sometimes have exam item-like qualities. One observed instance of this was when the teacher designed a paired speaking activity to be a simulation of the three stages of the speaking interview. Course materials might be skill-building in focus (ie, building skills that are directly applicable to exam questions). This occurred when one of the teachers designed materials for an activity involving paraphrasing Task 2 prompts and writing a “thesis statement”. Both observed teachers introduced writing materials (Task 2 topics) they had specifically developed for their IELTS preparation courses. The IELTS certificated teacher also introduced ‘official’ IELTS materials into his lesson, namely the “public descriptors” used for assessing speaking skills. All of these observations indicate how the IELTS Test provides washback on preparation course materials.



6.12 Research question 12 What do IELTS-preparation lesson observations and analyses indicate about the washback of the IELTS Test on the lessons?

[Source: Classroom observations]

Lesson observations of two types of test preparation course (GEP Level 12 and IELTS Preparation) provided strong evidence that the IELTS Test was influencing these lessons. In addition to relevant points made in response to Research Questions 9 and 11 above, other aspects of washback are noted here.

Some parts of the lessons involved the use of mock test questions and exam conditions. Thus, students worked on these tasks under strict time pressure and in silence. In one of the lessons, students were told how long to spend on paraphrasing from the question rubric a thesis statement for Task 2 writing (ie, five minutes). In the other lesson, the teacher played an audio-recording one time only, which simulates actual exam conditions.

In both lessons, the teachers at times focused on test-taking strategies. For example, one teacher paid attention to spelling in the listening test activities and, for writing practice, gave students a five-minute task to plan how they would answer an essay question, including how they would structure their essay. The other teacher gave the scenario of a test day and a student being about to start the writing test. He asked which task should be done first, and then answered: “Task 2, because it is worth more”. He also tried to lessen the anxiety of Task 2 writing by stating that a 250-word essay was “only about seven words/minute”, and this was not so hard once one had written one’s thesis statement.

Both lessons were structured around a mix of macro-skill activities (ie, the four components of the IELTS Test), although not all four were observed in either lesson. One lesson emphasised listening skills, but also included reading and writing; the other was balanced between writing and speaking.

In one of the lessons, the teacher provided a lot of feedback after activities, to ensure that everyone could properly evaluate how well they had done. For example, he was constantly calling upon students by name to give their answers to questions. Whenever the answer given was not what he was looking for, he would request elaboration or invite other students to give their responses (and sometimes justifications).

6.13 Research question 13 What are the profiles of the main local users of IELTS Test results in Cambodia?

[Sources: Embassy interviews; employer interviews]

The main local users of IELTS Test results in Cambodia are embassy staff at local embassies in Phnom Penh, who process scholarship applications for study in the embassy’s home country. Based on our interviews with embassy officials, it seems there are only one or two people at each embassy who deal directly with such applications. There is also a relatively small number of such embassies, but the principal ones are Australia, the UK and the USA. During 2010, Canada and New Zealand were handling Cambodian scholarship applications in regional embassies rather than in Phnom Penh. Representatives from the embassies of Japan and Sweden and the EU mission stated that IELTS scores were accepted for their scholarship programs, but were only collected and not processed in their offices in Phnom Penh. We were unable to confirm whether other donor countries used IELTS scores in processing Cambodian scholarship applications for study in the home country, but it is likely that Singapore and Korea would be such countries. The officials we spoke to from the embassies of Korea, Japan and the USA all stated their country’s traditional preference for TOEFL scores but, in the case of Japan and the USA, indicated that IELTS scores were often accepted in place of a TOEFL score.



Apart from local embassies, we anticipated that major private sector or NGO employers would also be “main local users” of IELTS Test results in Cambodia, but this proved not to be the case. Indeed, from interviews with employers representing a wide range of commercial enterprises and development missions, we found no organisation formally using IELTS scores as part of their recruitment process.

6.14 Research question 14 What is the impact of the IELTS Test on these users?

[Sources: Embassy interviews; employer interviews]

It is not easy to discern the impact of the IELTS Test on the embassies (and their officials) that use the IELTS Test. The officials we interviewed all shared the view that the IELTS Test could be trusted for their purposes because it was an “international standardised” exam. Indeed, several officials drew attention to their awareness of the Test’s apparently good international reputation and, therefore, there was never a thought about its reliability or validity. The embassy official’s job, in practice, was to process the IELTS scores and not to consider whether they were trustworthy in the circumstances of a particular candidate. They had some justification for taking this approach, as none was aware of any instance in which it transpired subsequently that a scholarship student could not cope with the English language demands of their overseas degree program despite achieving an IELTS score that had been deemed suitable for that program.

In the case of local employers in Phnom Penh, it is safe to say that IELTS has virtually no impact on them. A few employers stated that they would “notice” a good IELTS score submitted by a job applicant, but that it would not override essential non-English skills criteria to be met for a particular position (eg, computer literacy, leadership, etc). Thus, in contrast to some English proficiency tests such as the CET in China (Huang and Garner 2009), an IELTS score in Cambodia has very little impact on gaining direct entry to good jobs.

6.15 Research question 15 What is (or has been) the impact of the IELTS Test on other stakeholders (eg, parents of test-takers)?

[Source: Parent/guardian focus group]

Parents and guardians of IELTS test-takers would seem to be a significant stakeholder group since, without their financial support and encouragement, their children would likely not be able to sit the Test, or to achieve a band score that could enable access to overseas scholarship opportunities.

Our only data in relation to parents and guardians was obtained through an interpreted focus group comprised of just one parent and two guardians. As previously mentioned, the dynamics of this group were more like those of a group interview rather than a focus group, and discussions of topics were generally non-collaborative and lacking in depth. Their understanding of the IELTS Test was quite limited, and they trusted it to be an accurate measure of English proficiency. The only concern to emerge from this focus group was that the cost of IELTS (US$175) was expensive for Cambodians, particularly if they had to re-sit the exam. Nevertheless, it was felt to be manageable and not a barrier or burden that was insurmountable for families to meet.

On the positive side, there was great pride taken by families in having a child who was in a position to do well on the IELTS Test (ie, properly prepared) as it would open doors to overseas study and, ultimately, a good career upon returning to Cambodia. So, parents and guardians had a vested interest in the success of their children who take, or plan to take, the IELTS Test.



7 DISCUSSION The results of our impact study have been reported in Section 5, and the extent to which they have answered the research questions which have guided this study has been reported in Section 6. We now discuss the results of our study in two parts. Part 1 deals with understanding the questionnaire-derived results by way of a baseline comparison with the results reported in Hawkey (2006), while Part 2 deals with understanding all of our results in a more holistic way, using a critical language testing framework.

7.1 Part 1: Questionnaire-derived results In this first part of our discussion, we focus on comparisons between the results reported in Hawkey’s (2006) study of 572 IELTS candidates drawn from 193 IELTS test centres around the world, and some of our own findings in the Cambodian data using substantially the same quantitative instruments. The questionnaires used in these two studies focused on test-takers, test preparation students and their teachers, and we discuss below only the more important comparisons between the two studies.

While the respondents in Hawkey’s study were quite diverse in terms of nationality, it is not surprising that the respondent pool in the current project was much more homogeneous (Cambodian nationality, 99%). Both Hawkey’s sample and our own were roughly balanced in terms of gender, with females making up the majority in Hawkey’s study (55%) whereas males made up the majority in the current study (58%). While Hawkey’s test-taker sample revealed a relatively youthful group, with 72% of respondents under age 25 (see Table 16), our study focused on two different stakeholder groups: test preparation course student respondents who, like those in Hawkey’s study, are relatively young (64% under age 24), and test-takers who are noticeably older (77% over age 25). Based on these age differences, it would be expected that these two groups would have different educational experiences, as well as different goals and objectives related to IELTS Test preparation and the use of resulting IELTS scores. This in fact was borne out and has been detailed in Section 5.2.2.

Current Study Test Preparation (TP) Respondents

Test-taker (TT) Respondents

Totals Hawkey (2006)

Age group

N Age group

N Age group

N Age group

N

16-19 43 41% 16-19 13 12% 16-19 56 27% 15-20 181 35%

20-24 23 23% 20-24 12 11% 20-24 35 17% 21-25 192 37%

25-29 20 20% 25-29 51 49% 25-29 71 34% 26-30 84 16%

30-46 16 16% 30-46 30 28% 30-46 46 22% 31-35 33 6%

36-40 23 4.5%

41-45 6 1% 64 / 12%

46-50 2 0.5% Total 102 100% Total 106 100% Total 208 100% Total 521* 100% *51 test-takers did not specify their age

Table 16: Research sample age comparison with Hawkey (2006)



Unlike the respondents in Hawkey’s study (secondary to college 38%; secondary only 16%), the majority of respondents in the current project (77%) studied English at the secondary school level, which, in fact, was far above the totals for all levels combined in Hawkey’s study (56%). Additionally, while outside-of-school classes was not a frequent option of respondents in the Hawkey study (outside classes only, 1%), in the Cambodian context, 80% reported being involved in extra classes. These comparisons must be interpreted with caution, however, because of the slightly different phrasing for this item in each study. In Hawkey’s study, the question was phrased “outside classes ONLY”, whereas in our study it was phrased “extra language classes”. Also, in Hawkey’s study, it appears that the percentages are based on respondents answering only once across all the levels, whereas in our study they could answer multiple times, depending on the levels at which they had language education experience.

In Hawkey’s study, the majority of respondents (89%), completed the Academic module of IELTS, compared with 99% in the current study. However, a larger percentage of respondents in Hawkey’s study (96%) had participated in a preparation course, 40% of whom indicated that ‘IELTS’ was included in the name of the course. The comparative figures in the current study, which used slightly different methods to record this data, were 40% and 51% respectively (see Table 4). Whereas in Hawkey’s study, 36% of test-takers identified the United Kingdom, and 12% Canada as their destination country, none in the current study indicated these countries. In both Hawkey’s and the current study, respectively, Australia (28% and 37%) and New Zealand (12% and 18%) were common destination countries, whereas the United States was a more popular destination in Hawkey’s study, indicating a more North American / United Kingdom focus on the part of respondents in that study. The Cambodian nationals in the current study seem to favour Australia and New Zealand, although other destinations within the Pacific Rim were seen as possibilities (eg, Singapore and Philippines).

Teachers in Hawkey’s and our study (Hawkey, N=83; current study, N=11) represented a broad range of ages and experience levels, and the majority in both cases reported completing post-secondary degrees or other qualifications. Approximately half of the teacher respondents in both studies reported that they were IELTS examiners, and approximately one third indicated that they had received specific training in IELTS preparation course teaching. Almost all teachers (Hawkey, 90%; current study, 91%) stated that the IELTS Test not only influences the content and types of activities they utilise in their classes, but also the methodology they employ (Hawkey, 63%; current study, 64%). Although these findings align with each other, they are in contrast to those of previous studies that found no evidence of test washback resulting in a change of teaching methodology (see, for example, Alderson and Hamp-Lyons, 1996).

Teachers in both studies indicated that students were able to realise significant achievement and success in their classes due in large part to student motivation, as well as clear and focused learner and course goals. At the same time, the common complaints in Hawkey’s study were echoed by the teachers in the current study, ie, that the IELTS preparation course could easily become repetitive and monotonous, lacking a ‘dynamic,’ ‘reflective,’ or ‘personal’ quality (Hawkey, 2006, p 108). Also, in each study there were overlaps among the most commonly emphasised skills (either by the textbook or the teacher) covered in classes, including identifying main ideas, identifying overall meaning, and predicting information. In general, there seem to be clear consistencies among the teachers, course related activities, reasons for success, and skills covered across the contexts of both studies. Teachers in both studies also indicated that, in general, the scores that students received on the Test roughly approximated their (the teachers’) estimates of the students’ skill levels.



From the students’ own perspectives, a large number (Hawkey, 43%; current study, 51%) reported that they felt they had reached their goals and had been successful in their course. On the other hand, and potentially more troubling, 43% and 27% of students (Hawkey and the current study, respectively) felt that they had not been successful. Moreover, test-takers in both studies reported worrying about their performance on the exam. Students in both studies ranked the Reading sub-test as the most difficult, with slight differences in the comparative ranking of the other sub-tests. The majority of test-takers in both studies (Hawkey, 72%; current study, 80%), however, indicated that they believed the Test was fair and accurate.

7.2 Part 2: All results As noted above, to ensure a proper discussion of the full results of our impact study we need to consider them from a more holistic perspective. Indeed, to properly understand and appreciate the full impact of IELTS in Cambodia, we need to locate these findings within a theoretical framework capable of illuminating the broader subject of test impact in a small developing country. Critical language testing (CLT) would seem to be a very good fit for this task. As discussed in Section 2, Lynch (2001) provides a framework (see Table 17) linking Pennycook’s (1999, 2000) ideas about critical applied linguistics with Shohamy’s (2001) 15 CLT principles, to enable the operationalising of a CLT model.



Critical perspective characteristic 1: An interest in particular domains such as gender, class, ethnicity, and the ways that language and language-related issues (like all human relations and activities) are interconnected with them. Shohamy’s CLT Principles:

1) Critical language testing (CLT) is not neutral, but is shaped by cultural, social, political, educational and ideological agendas.

3) CLT views test-takers as political subjects within a political context. 4) CLT views tests as tools within a context of social and ideological struggle. *5) CLT asks questions about which and whose agendas tests serve. *6) CLT claims that testers need to understand the tests they create within a larger vision of society and its

use of those tests. *7) CLT examines tests in terms of their measurement and assessment of knowledge versus their definition

and dictation of knowledge. *8) CLT questions the nature of knowledge that tests are based upon: whose knowledge? Independent

‘truth’ or negotiated and challengeable? 9) CLT examines the influence and involvement of the range of stakeholders in a testing context. 10) CLT perceives the embeddedness of tests within social and educational systems.

Critical perspective characteristic 2: The notion that our research needs to consider paradigms beyond the dominant, postpositivist-influenced one. Shohamy’s CLT Principles:

*7) CLT examines tests in terms of their measurement and assessment of knowledge versus their definition and dictation of knowledge.

*8) CLT questions the nature of knowledge that tests are based upon: Whose knowledge? Independent ‘truth’ or negotiated and challengeable?

11) CLT admits to the limited knowledge of any tester and the need for multiple sources of knowledge. 12) CLT challenges the dominant psychometric traditions and considers ‘interpretive’ approaches to

assessment that allow for different meanings and interpretations rather than a single absolute truth. *13) CLT considers the meaning of test scores within this interpretive framework, allowing for the possibility of

discussion and negotiation across multiple interpretations. 15) CLT challenges the primacy of the ‘test’ as assessment instrument and considers multiple procedures for

interpreting the knowledge of individuals.

Critical perspective characteristic 3: A concern for changing the human and social world, not just describing it: the ‘transformative agenda’, with the related and motivational concern for social justice and equality. Shohamy’s CLT Principles:

2) CLT encourages an active, critical response from test-takers. *5) CLT asks questions about which and whose agendas tests serve. *6) CLT claims that testers need to understand the tests they create within a larger vision of society:

What vision do the tests create? What vision and purposes are they used for? 14) CLT challenges the knowledge that tests are based upon and advocates a democratic representation of

the multiple groups of society.

Critical perspective characteristic 4: The requirement that critical applied linguistics be self-reflexive. Shohamy’s CLT Principles:

*5) CLT asks questions about which and whose agendas tests serve. *8) CLT questions the nature of knowledge that tests are based upon: Whose knowledge? Independent

‘truth’ or negotiated and challengeable? *13) CLT considers the meaning of test scores within this interpretive framework, allowing for the possibility of discussion and negotiation across multiple interpretations.

Notes: Numbering refers to the order in which Shohamy (2001) presents the principles. Asterisks (*) indicate that the principle appears under more than one characteristic.

Table 17: Shohamy’s critical language testing principles and the critical perspective (reproduced from Lynch 2001, p 363)



Our discussion of the full findings of our impact study follows below, set out in four sub-sections, addressing all 15 principles of CLT. (We include each principle’s number in square brackets after it has been mentioned or exemplified).

7.2.1 Critical perspective characteristic 1 ! an interest in particular domains such as gender, class, ethnicity, and the ways that

language and language-related issues (like all human relations and activities) are interconnected with them.

This critical perspective brings a focus on the particular domain of Cambodian nationals as a distinct group of test-takers. (This political domain also overlaps significantly with an ethnicity domain since 90% of Cambodians are ethnic Khmers). The central question to address under this characteristic is to what extent Cambodians as a group are advantaged or disadvantaged by the IELTS Test.

The first point to note is to state the obvious: Cambodia is a developing country in which English is not a traditional language (although it is becoming increasingly widespread), while the IELTS Test is a product of educational authorities in developed countries (ie, the United Kingdom and Australia) [3]. The use of IELTS as a gate-keeping instrument, whether for admission to higher education or for migration purposes, is controlled by western institutions and Cambodians must abide by the rules and regulations that are set and imposed by these institutions. Generally speaking, for Cambodians to be socially mobile they must speak English and their pathway to success, to the extent that it involves overseas study or scholarships, is determined by non-Cambodian gate-keepers such as the IELTS Test.

The involvement of international aid to assist the economic and social development of Cambodia introduces a further complicating element in the relationship between Cambodians and their social advancement [3]. The foreign aid support offered by many English-speaking countries typically includes scholarship funding and related English support programs. Australia, for example, funds a substantial number of scholarships for Cambodians to study at Australian universities. This aid obviously acts as a strong inducement for Cambodians to study English and engage with the IELTS testing enterprise in order to realise their dreams of English proficiency and the best possible career prospects [4]. However, this aid also benefits Australian universities by contributing to student fee income and enriching university life through diversity of the student population. Although it seems to be the case that the IELTS Test serves the agendas of both developed and developing countries, the developing countries agendas in this case are in response to those of the developed countries [1].

The IELTS Test also serves the agendas of universities in developed countries, which have become increasingly reliant on international students as a source of income in recent years [9]. These universities need to be able to cope with processing the English proficiency ratings of non-native English speakers. The IELTS Test has come to be widely used and accepted as an appropriate and sufficient measure of English proficiency. And, with its enduring and extended use in university settings, the sense is created (whether justifiably or not) that IELTS Test scores are reliable and trustworthy, and this in itself helps further embed the Test’s usage in these settings [10]. Yet, our data provided some evidence that IELTS scores need to be interpreted with care and caution. For example, we found several Cambodian test-takers who, independent of one another, expressed criticism of the Reading sub-test, claiming that their reading skills assessment varied dramatically depending on the nature of the topics. Such a finding, if backed by suitable evidence, would seem to suggest that the IELTS Reading sub-test should not be the sole contributor to assessing the reading skills of an overseas university applicant.



In terms of the IELTS Test’s assessment and measurement of knowledge versus its definition and dictation of knowledge, several Cambodian stakeholders commented that IELTS assumes a Eurocentric view of knowledge in terms of topics selected for examination purposes [4]. One impact of this was that rather than simply teaching macro-skills for academic purposes, test preparation course teachers were often also teaching ‘about the world’ to extend the knowledge base of their students to what are seen to be important issues in western cultures (eg, education, environment, entertainment) [7]. There is no room for negotiating or challenging these topics in the IELTS Test; they have to be accepted and the test-taker’s English proficiency is determined through their ability to handle this knowledge (or lack of it) [8].

The emergence in recent years of a Cambodian variety of English (Keuk, 2009; Moore and Bounchan, 2010) is also a matter of relevance in the testing of English proficiency in Cambodia. While no respondents in our project mentioned Cambodian English, it is almost certainly the case that Cambodian test-takers modified their colloquial English to be more aligned with standard dialects for the purposes of the IELTS Test. (This supports Davies (2009) who noted the tendency for interlocutors to use their own local dialect in non-formal settings, while using a standardised norm in formal settings such as English language assessment.) Such behaviour is a clear indication of the impact of the IELTS Test, and of its ability to decide what counts as appropriate English language knowledge. Whilst the IELTS Test is concerned chiefly with the use of English in academic contexts in English-medium universities, the use of a non-standard variety of English in some local academic contexts could create unanticipated hurdles for a Cambodian IELTS test-taker.

An interesting footnote to IELTS testing of Cambodians is that in the 1990s, female candidates constituted only 22.8% of test-takers, whereas in the 2000s, they almost doubled to 43.5% (calculated from figures supplied by IDP Education (Cambodia)). This evolution shows that the IELTS Test seems to be increasingly accessible to Cambodian women, and this bodes well for their future educational opportunities [4].

7.2.2 Critical perspective characteristic 2 ! the notion that our research needs to consider paradigms beyond the dominant,

postpositivist-influenced one.

This critical perspective is addressed through the mixed-methods design of the current study. Although we used previously validated questionnaire instruments to collect data on test preparation course students and teachers, and test-takers, we complemented this data with classroom observations and interviews and focus groups with a wide range of stakeholders, each with valid perspectives.

Our study sits within the interpretive paradigm [12] and thus allows for “different meanings and interpretations rather than a single absolute truth” aimed for in the dominant psychometric traditions (per Lynch, 2001, p 363). By using a mixture of methods and drawing on a wide variety of stakeholders, our research is better able to investigate the complexities inherent in a test impact study [9].

Not surprisingly, the IELTS Test was designed for a ‘generic’ non-native English speaking test-taker, and not specifically for Cambodian test-takers. However, in our study we found various stakeholders complaining that the IELTS Test was too Eurocentric in terms of expected ‘world knowledge’. Indeed, in this respect, the Test clearly advantages European test-takers and disadvantages non-European test-takers. There is a real need for tapping into multiple sources of knowledge to create a test that is fair for all test-takers [11].



IELTS Test scores are decided on the basis of test-taker responses to multiple items in the Listening and Reading sub-tests; to two tasks in the Writing sub-test; and to interviewer prompts in the Speaking sub-test. Because of the nature of a ‘mass’ and anonymous testing system, the individual test-taker has, for the most part, no distinct identity in the scorer or rater’s mind. Thus, it is virtually impossible (and certainly impractical) to discuss or negotiate multiple interpretations of test scores as would be desirable in a more democratic and non-positivist approach. The scores are simply reported as macro-skill achievements and an overall composite score, with virtually no regard to any other humanising or otherwise relevant information about a test-taker’s ability in English [13]. The IELTS Test is a classic example of the “primacy of the ‘test’ as assessment instrument” (Lynch, 2001, p 363), with no recourse to any other procedure for interpreting the knowledge of individuals [15]. The onus is on the receiving institution to interpret the reported IELTS band score in conjunction with other appropriate measures to determine whether the applicant meets the English proficiency requirements of that particular institution. In practice, however, many users of language test scores rely almost exclusively on the reported scores in making their decisions about suitable language proficiency.

7.2.3 Critical perspective characteristic 3 ! a concern for changing the human and social world, not just describing it:

the ‘transformative agenda’, with the related and motivational concern for social justice and equality.

This critical perspective relates to our concern for hearing the voices of Cambodians not only as a nation but also as individuals with various stakeholder roles in relation to the IELTS enterprise. By giving voice to these individuals, their concerns and interests can be aired and publicly debated and, in the process, their democratic rights are respected.

Our study found general praise for IELTS among most stakeholders, but mixed praise among the test-takers, especially from Cambodian graduates returned from overseas. Various criticisms raised by test-takers included the view that the Test was too Eurocentric; that the reading topics varied greatly in terms of degree of difficulty; that the reading topics were not discipline-specific (and therefore sometimes irrelevant); and that the Test was too expensive. All of these issues are worthy of further investigation by IELTS [2].

Despite providing the opportunity for various stakeholders to exercise their democratic right to be heard, the only stakeholders who seemed able and willing to articulate and argue a clear opinion about IELTS were the returned graduates from overseas and the IELTS examiners. Perhaps this is understandable given the direct impact that IELTS has had in their lives. However, it is somewhat regrettable that other stakeholders (eg, parent/guardians; embassy staff; employers) seem to have missed the opportunity to express any view critical of the IELTS Test (assuming, of course, that they held such views). Not surprisingly, these tendencies to express (or not express) opinions also match the degree of stakeholder exposure to democratic values more generally: the examiners and returned graduates have certainly been exposed to democratic values, whereas it is far from certain whether the Cambodian parents, embassy staff and employers have [14].

CLT claims that testers need to understand their tests within the context of a larger vision of society. In particular, they need to consider the vision the tests create and the purposes to which the tests are used [6]. In the case of IELTS, the vision the Test seems to create is that assessing English language proficiency is basically unproblematic, and the IELTS Test can accurately measure a candidate’s English proficiency. Indeed, our study found a widespread belief, particularly among the non-test-taker stakeholders, that IELTS was a fair and accurate test. However, as previously noted in this report, test-takers themselves sometimes voiced misgivings about the Test in terms of its fairness and



accuracy. At the same time, the fact that it is an expensive test (albeit for justifiable reasons associated with test development and administration) was noted by virtually all of the Cambodian stakeholders, many of whom felt that, relative to other educational costs in Cambodia, it was excessively high. The ethical issue of whether language testing, as part of the educational sector, should be profit making, is a moot point and a novel concept in Cambodia [6].

By publicly expressing their views, IELTS test-takers and other Cambodian stakeholders are democratically empowered – they are not faceless/voiceless statistically generated demographics, but rather individual sentient human beings who have to deal with the IELTS Test, in many cases whether they want to or not. It is, therefore, important for IELTS (as with other testing authorities) to take account of the views of all stakeholders impacted by language testing regimes to better understand the impact their test has on the lives of real people [14].

7.2.4 Critical perspective characteristic 4

! the requirement that critical applied linguistics be self-reflexive.

This critical perspective relates to our concern as responsible members of the language testing community to step back from our work and examine our assumptions and professional practices in a critical light. This is particularly important as one of the researchers (Moore) was, until 2009, an IELTS examiner for 10 years (including two years as a Senior Examiner).

Among the agendas that the IELTS Test serves must be included the agendas of academics, such as two of the authors of this report [5]. Our academic positions require that we are seen to be ‘research active’ and secure grants from outside our universities in order to be respected as academics. We did not set this agenda but, as long-serving members of the academic community, we have in practice accepted it (and, it must be said, our own teaching and professional development have benefitted from our involvement in research activities). Universities set and retain such agendas at least partly because it influences how they are perceived by the public, the private sector and the government, which in turn impacts on their funding.

Academics cherish the right of ‘academic freedom’ to research what they like and to report their findings, whether favourable or unfavourable to their sponsors. We have been able to undertake the current study according to our own conceptualisation of what an impact study of Cambodia should comprise, and with a view to publicly reporting the findings without ‘fear or favour’ [5].

We must also acknowledge that an impact study of IELTS in Cambodia would ideally have Cambodian researchers playing leading roles in devising and undertaking the research. This was not possible, however, because at present there are no Cambodian academics fitting the profile of this type of researcher. Cambodia is still recovering from the devastation wreaked on its educational system by the Khmer Rouge regime in the late 1970s, including the elimination of most of its teachers and academics. Today, almost all Cambodian academics get paid for their teaching hours but receive no income for research. Thus, there are very few Cambodian academics conducting western-style research (see Moore, 2011) [8].

In addition to not being Cambodian nationals ourselves, none of the authors was in full-time residence in Cambodia throughout the duration of this study, although each spent several months in-country in 2010. Thus, a considerable amount of our project work was actually performed off-site, namely in Sydney and Tokyo. It is hard to know whether this has had a detrimental effect on conducting this research, however, we report it in the spirit of self-reflexivity.



7.3 Conclusion of discussion As with ‘critical discourse analysis’, with its agenda to uncover and draw attention to power relations and to champion the less powerful, critical language testing offers both a fresh perspective on language testing and, perhaps, an unsettling one for the language testing establishment. To conclude our discussion we would like to draw attention to the power of Lynch’s CLT framework to ‘unpack’ a language test and expose it to critical scrutiny. Indeed, it has given us a much better understanding of the full impact of IELTS in Cambodia, ie, well beyond the understanding gained from simply mechanically providing data analysis and results reporting.

At this point it is now appropriate to return to the first, and as yet, unanswered research question, What impact does IELTS have on education and society in Cambodia? Drawing from our results and discussion above, we can claim that IELTS currently has a minor impact on education and society in Cambodia in general terms, but has a significant impact on facilitating opportunities for Cambodians to pursue higher education (especially postgraduate studies) overseas at English-medium institutions. The full impact of IELTS’ role in contributing to the decisions as to which Cambodians can (and cannot) pursue overseas study is ultimately unknowable but will inevitably play out over the coming decades.

8 SUMMARY OF KEY FINDINGS 1. The IELTS Test has grown steadily and substantially since it was first offered in Cambodia in

1992 to just a handful of people. In 2010, there were more than 1000 IELTS test takings, of which 96% were the Academic module.

2. The IELTS Test is the major gate-keeping mechanism for Cambodians to access the majority of scholarships available to them for overseas study.

3. The IELTS Test is widely known among those Cambodians who hope to study overseas at an English-speaking institution.

4. The IELTS Test is quite respected among all the categories of major stakeholders who participated in this study.

5. The IELTS Test is generally trusted as an accurate measurement tool for academic English proficiency, although some concerns were raised about the validity and reliability of the Reading sub-test (ie, by returned graduates from overseas) and the Writing sub-test (ie, by IELTS preparation course teachers and IELTS examiners).

6. Several different stakeholder groups (eg, IELTS examiners; IELTS preparation course teachers; Cambodian test-takers) all indicated that IELTS was too Eurocentric and that this meant Cambodian candidates had to acquire more ‘world knowledge’, in addition to developing their English proficiency skills in order to achieve a satisfactory band score.

7. The IELTS Test has a very strong impact on the content of some IELTS preparation courses, and a tension was noted between teaching narrowly to the exam and developing real English proficiency. Some teachers stated that students would be better off in the long run to take general EAP courses rather than IELTS-specific preparation courses.

8. The IELTS Test is widely considered to be expensive for Cambodian test-takers.



9. Although the IELTS exam is perceived to be a “high stakes” test, there appear to be few occasions in Cambodia where failure to achieve a required band score has resulted in a candidate having no further opportunities to improve their English or to re-apply to undertake overseas study.

10. The study found no evidence of IELTS scores being used for purposes other than that for which they were intended (ie, measuring English proficiency for academic or migration purposes). We found no employers, even in the higher education sector, who demanded IELTS scores from potential employees.

11. The returned graduates from overseas and the IELTS examiners were the two categories of stakeholder that were able to provide the most clearly articulated view of the impact of IELTS in Cambodia. The views of most other stakeholders concerning the impact of IELTS were, by contrast, more indirect and vague.

12. From a research methodological perspective, utilising the Critical Language Testing paradigm to evaluate the impact of the IELTS Test in Cambodia provides a principled and effective way to give voice to all stakeholders impacted by this globally powerful language test.

9 CONCLUSIONS, LIMITATIONS AND FURTHER RESEARCH 9.1 Conclusions The research reported in this study has shown that an IELTS Test impact study covering a whole country is a viable proposition if the research is well planned and efficiently carried out. Moreover, this study has shown, for the first time, how an IELTS Test impact study can be realised within a critical language testing paradigm. The results of the IELTS impact study in Cambodia also provide useful further validation of the IELTS exam, particularly in terms of the context of usage in a small, developing country. Our study has shown that IELTS impacts Cambodia in a number of ways and at a number of levels. Although most stakeholders were asked the direct question “What impact does IELTS have in Cambodia?”, few were able to give clear, well-articulated answers. Thus, the answers to this particular question have largely been determined from combining direct responses and with indirect, inferred understandings gained from the various analyses of the data.

Hawkey (2006) notes the need for a balance to be struck in impact research between what he calls two extremes: “One [extreme] would tend to neglect the social consequences or consequential validity of a high-stakes test; the other would tend to enquire into its effects on too many stakeholders, or pursue too many intervening variables for clear washback or impact connections to be made” (p 18, emphasis in original). In conducting our study of the impact of IELTS in Cambodia, we believe that we have struck the right balance between these extremes, and hope that our report has provided sufficient evidence to reflect this view.

9.2 Limitations As with any study of similar scope and objectives, our study has been constrained by various factors, which in turn have impacted on what we have been able to achieve. For example, this study has only investigated IELTS in terms of its operation on-site in Cambodia. It has not considered the dimension of overseas higher education authorities using IELTS results from Cambodia or its citizens. In terms of IELTS preparation courses, we considered only those offered by ACE, since these are reputable, well-established, well-attended and convenient to sample. A more comprehensive study of IELTS preparation courses in Cambodia would need to sample non-ACE courses, their teachers and students.



In terms of the methodology used in this research project, we sampled relatively small numbers of participants, and selected them through convenience sampling. A larger focus group for parent/guardians in particular, would have been desirable and might have yielded valuable information. More triangulation of results would also have been desirable, for example, using different instruments from the same source; or one instrument from different sources.

The scope of this relatively small-scale IELTS impact study in Cambodia was constrained by the availability of time and money. With more resources, wider-scale sampling of data might have generated valuable information. The timeframe of one calendar year for undertaking an impact study devoted to an entire country also limited what was possible to achieve in terms of available resources. Lastly, as none of the principal researchers is Cambodian nor resided in Cambodia during the full course of the research project, it is possible that some aspect of the project has overlooked or under-researched what might have been considered important from a Cambodian perspective.

9.3 Further research We see many opportunities for further research stemming from this project. For example, IELTS impact studies could usefully be undertaken in other small developing countries to determine to what extent the current study’s findings are specific to the Cambodian context or indeed are more generalisable to the contexts of all small developing countries. IELTS impact studies could also usefully be undertaken in larger developing or developed countries, depending on the perceived need for such studies. All would contribute to further validating the IELTS examination.

We have noted that there are very few examples of critical language testing case studies in the published testing literature. Given the international significance of IELTS in the English language testing market, it is important that it continue to be scrutinised critically for its impact on all stakeholders. Lastly, (critical) impact studies of other market-leading English language tests should also be undertaken to properly understand their positive or negative impacts in the various countries (developed/developing) in which they operate.

ACKNOWLEDGEMENTS The authors of this report would like to thank all who contributed to the successful completion of this research project. In particular, we are grateful to IELTS Australia for funding the project and to IDP Education (Cambodia) for administrative and technical support throughout the life of the project. The Phnom Penh team, coordinated by Heang Chan Veasna, was comprised of Net Wanna, Ou Socheat, Chan Sreytoch, Veng Mengkeang, Korng Sothea, Uk Borasy, Phoeun Bunna, Sok Pisey and Kin Sina.

Our greatest thanks are owed to the several hundred participants, mostly Cambodian nationals, who generously gave their time to answer the many questions we posed.



REFERENCES Alderson, J and Hamp-Lyons, L, 1996, ‘TOEFL preparation courses: A study of washback’ in Language Testing 13(3), pp 280-297

Alderson, J and Wall, D, 1993, ‘Does washback exist?’ in Applied Linguistics 14, pp 115-129

Blackhurst, A, 2004, ‘IELTS Test performance data 2003’ in Research Notes 18, pp 18-21

Brown, JD, 2001, Using surveys in language programs, Cambridge University Press, Cambridge

Cheng, L, Watanabe, Y, and Curtis, A, 2004, Washback in language testing: Research contexts and methods, Lawrence Erlbaum, Mahwah, NJ

Coleman, D, Starfield, S, and Hagan, A, 2003, ‘The attitudes of IELTS stakeholders: Student and staff perceptions of IELTS in Australia, UK and Chinese tertiary institutions’, IELTS Research Reports Volume 5, IELTS Australia Pty Limited, Canberra, pp 159-235

Davies, A, 2009, ‘Assessing World Englishes’ in Annual Review of Applied Linguistics, 29, pp 80-89

Dornyei, Z, 2003, Questionnaires in second language research: Construction, administration, and processing, Lawrence Erlbaum, Mahwah, NJ

Duff, PA, 2008, Case study research in applied linguistics, Lawrence Erlbaum/Taylor Francis, New York

Edley, N, and Litosseliti, L, 2010, ‘Contemplating interviews and focus groups’ in Research methods in linguistics, ed L Litosseliti, Continuum, London, pp 155-179

Hamp-Lyons, L, 1997, ‘Washback, impact and validity: Ethical concerns’ in Language Testing 14 (3), pp 295-303

Hamp-Lyons, L, 2000, ‘Social, professional and individual responsibility in language testing’ in System 28, pp 579-591

Hawkey, R, 2006, Impact Theory and Practice: Studies of the IELTS Test and Progetto Lingue 2000, Cambridge University Press, Cambridge

Hogan, MJ, 2009, ‘Assessment and its place in international ELT’ in CamTESOL conference on English language teaching: Selected papers volume 3, ed N Tao, CamTESOL, Phnom Penh, pp 1-8

Huang, D, and Garner, M, 2009, ‘A case of test impact: Cheating on the College English Test in China’ in Language Testing Matters: Investigating the wider social and educational aspect of assessment, eds L Taylor and C Weir, Cambridge University Press, Cambridge, pp 59-76

Green, A, 2007, IELTS washback in context: Preparation for academic writing in higher education, Cambridge University Press, Cambridge

IDP Education (2010a) General English Program & IELTS Preparation Courses, accessed December 17, 2010 from <www.cambodia.idp.com/ace/study_program/general_english_program.aspx>

IDP Education (2010b) Guided Individual Learning (GIL) Centre, accessed December 17, 2010 from <www.cambodia.idp.com/ace/the_guided_individual_learning.aspx>



Ingram, I, and Bayliss, A, 2007, ‘IELTS as a predictor of academic language performance, Part 1’, IELTS Research Reports Volume 7, IELTS Australia, Canberra and British Council, London, pp 132-199

Jakeman, V, and McDowell, C, 2006, Action Plan for IELTS, Cambridge University Press, Cambridge

Keuk, CN, 2009, ‘English language variety in Cambodia’ in CamTESOL Conference on English Language Teaching: Selected Papers Volume 4, 2008, ed Im Koch, CamTESOL, Phnom Penh, pp 98-107

Lynch, B, 2001, ‘Rethinking assessment from a critical perspective’ in Language Testing 18(4), pp 351-372

McNamara, T, 1998, ‘Policy and social considerations in language assessment’ in Annual Review of Applied Linguistics 18, pp 304-319

McNamara, T, 2007, ‘The social-political and power dimensions of tests’ in Language Testing and Assessment, (Volume 7 of Encyclopaedia of Language and Education, 2nd ed), eds E Shohamy and NH Horberger, Springer, Dordrecht, The Netherlands

Merrifield, G, 2008, ‘An impact study into the use of IELTS as an entry criterion for professional associations – Australia, New Zealand and the USA’, IELTS Research Reports Volume 8, ed J Osborne, IELTS Australia, Canberra, pp 279-321

Messick, S, 1989, ‘Validity’ in Educational Measurement, 3rd edition, ed RL Linn, ACE/Macmillan, New York, pp 13-103

Messick, S, 1996, ‘Validity and washback in language testing’ in Language Testing 13, pp 241-256

Mickan, P, and Motteram, J, 2008, ‘An ethnographic study of classroom instruction in an IELTS preparation program’ in IELTS Research Reports Volume 8, ed J Osborne, IELTS Australia, Canberra, pp 14-39

Moore, SH, 2011, ‘The struggle to develop a ‘research culture’ in a developing country’ in TESOL Quarterly 45(2), pp 334-343

Moore, SH, and Bounchan, S, 2010, ‘English in Cambodia: changes and challenges’ in World Englishes 29(1), pp 114-126

O’Connell, S, 2002, Focus on IELTS, Longman, New York

O’Loughlin, K, and Arkoudis, S, 2009, ‘Investigating IELTS exit score gains in higher education’, IELTS Research Reports Volume 10, ed J Osborne, IELTS Australia, Canberra and British Council, London, pp 95-180

Pennycook, A, 1994, The Cultural Politics of English as an International Language, Longman, London and New York

Pennycook, A, 1999, ‘Introduction: Critical approaches to TESOL’ in TESOL Quarterly 33, pp 329-348

Pennycook, A, 2001, Critical applied linguistics: A critical introduction, Lawrence Erlbaum, Mahwah, NJ



Rea-Dickins, P, Kiely, R, and Yu, G, 2007, ‘Student identity, learning and progression: The affective and academic impact of IELTS on ‘successful’ candidates’ in IELTS Research Reports Volume 7, IELTS Australia, Canberra and British Council, London, pp 59-136

Rea-Dickins, P, Kiely, RN, and Yu, G, 2011, ‘Uses and impact of test scores in university admissions processes: The language test as the ‘hard’ criterion’ in Language Testing: Theories and Practices, ed B O’Sullivan, Palgrave Macmillan, Houndmills, Basingstoke, Hampshire, pp 262-281

Richards, K, 2003, Qualitative Enquiry in TESOL, Palgrave Macmillan, Basingstoke, UK

Roberts, R, Preshous, A, and Gakonga, J, 2004, IELTS Foundation, Macmillan Education in association with The Open University, Oxford

Ross, S, 2008, ‘Language testing in Asia: Evolution, innovation, and policy challenges’ in Language Testing 25(1), pp 5-13

Saville, N, and Hawkey, R, 2004, ‘The IELTS impact study: Investigating washback on teaching materials’ in Washback in language testing: Research contexts and methods, eds L Cheng, Y Watanabe, and A Curtis, Lawrence Erlbaum, Mahwah, NJ, pp 73-96

Shohamy, E, 1997, ‘Critical language testing and beyond’, plenary address at the annual American Association for Applied Linguistics Conference, March 1997, Orlando, Florida, USA

Shohamy, E, 2001, The power of language tests: A critical perspective on the use of language tests, Pearson Education, Harlow, UK

Stewart, D, Shamdasani, P, and Rook, W, 2007, Focus groups: Theory and practice, Sage, London

Teddlie, C, and Tashakkori, A, 2009, Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioural sciences, Sage, London

United Nations, 2011, accessed 19 February 2011 from <http://data.un.org/CountryProfile.aspx?crName=Cambodia>

Wall, D, and Alderson, C, 1993, ‘Examining washback: The Sri Lankan impact study’ in Language Testing 10, pp 41-69



LIST OF ABBREVIATIONS AND ACRONYMS

ACE Australian Centre of Education, Phnom Penh

ADS Australian Development Scholarships

CAE Cambridge Certificate in Advanced English

CLT Critical Language Testing

ELTO English Language Training for Officials

FCE Cambridge First Certificate in English

GEP General English Program

GESE Trinity College Graded Examinations in Spoken English

IDP IDP Education (Parent company of IELTS Australia)

IELTS International English Language Testing System

NGO Non-Government Organisation

NZDS New Zealand Development Scholarships

PET Cambridge Preliminary English Test

TOEFL Test of English as a Foreign Language

TOEFL iBT TOEFL Internet-Based Test

TOEIC Test of English for International Communication

TP Test Preparation Course Student

TT Test-taker



APPENDIX 1: TEST-TAKER (TT) QUESTIONNAIRE STUDY OF THE IMPACT OF THE INTERNATIONAL ENGLISH LANGUAGE TESTING SYSTEM (IELTS) STUDENT QUESTIONNAIRE

Dear Participant,

As part of the continuing programme to update and refine its International English Language Testing System (IELTS), a number of studies are undertaken in order to determine the impact of the test. This current project is a case study investigation of the impact of IELTS in Cambodia. The project contributes to the ongoing test validity research of IELTS worldwide but, unlike other IELTS impact studies to date, has a focus exclusively on a small, developing country. The aim of the project is to determine the nature and extent of the impact of IELTS in Cambodia.

Your responses to this questionnaire will be treated in confidence, and only used for the stated purposes of the study.

Thank you very much for your time and co-operation. We should also be grateful if you would complete and sign the consent note below.

Yours sincerely Dr. Stephen Moore Senior Lecturer Linguistics Department Faculty of Human Sciences Macquarie University North Ryde NSW 2109 Australia

Your consent to participate in the IELTS Impact Study I understand that: • the purpose of the study is to collect and analyse information from those familiar with the IELTS; • my name will not appear in any project publication; • the information I give, but not my name, may be quoted; • I am free to refuse to participate in the study and may withdraw at any time; • my completed questionnaire is for the study team only; it will not be shown to anyone not connected with

the study. Signature: …………………………………… Date: ……………………………

Please give the information about yourself requested below in the relevant spaces and by ticking (!) the appropriate .

Home country

Nationality Full name and current address

Male Female Age: ______

Answer one of the following questions:

A. Do you work? If yes, what is the name of your employer? ---------------------------

B. Are you a university student? If yes, what is your major subject (in university)? ----------------------

C. Are you a high school student? Yes No



There are 3 Parts / modules in this questionnaire:

• PART 1: for all students and candidates, on participants' language learning background. • PART 2: again for everyone to complete, about the ways people learn, study and take tests. • PART 4: only for students who have taken the IELTS, asking them about the test.

You will be advised on which Parts you should complete.

Please now complete PART 1 of the questionnaire, which is about your language learning background.

PART ONE: ENGLISH LANGUAGE BACKGROUND

1. What language(s) do you usually speak at home when in your country?

Yes No 2. As a child, did you live with a parent, guardian or other close

relation who was a native speaker of English?

3. Did you study English in: Yes No 3.1 kindergarten (age 3-6)? 3.2 primary school (age 7-11)? 3.3 secondary school (age 12-17)? 3.4 college / university? 3.5 extra language classes? 4. How many of your English

lessons were taught (or assisted) by native English speaking teachers?

More than half

About half

Less than half

A few

None

5. How much of the time in

class did your teachers of English speak to you in English?

All the time

More than half

About half

Less than half

Never

6. In your home country, how often: Often Some-

times Rarely Never

6.1 do you use English for socialising (talking or writing to friends)?

6.2 are you in English contact with friends overseas? 6.3 are you exposed to English in the media? 6.4 do you use English for work? 6.5 do you read texts in English in your specialist subject? 6.6 do you write in English in a day? 7. Have you studied in an English-speaking

country? Where, when and for how long?

Very often Quite

Often Once/twice

only Never 8. How often have you stayed in English-

speaking countries for a week or more?

Thank you for completing Part 1.



Please now complete PART 2 of the questionnaire, which is about the ways you learn and study languages

PART 2: LEARNING, STUDY AND TEST -TAKING APPROACHES

A. Please put a tick (!) in the column that best describes your language learning or study habits.

Strongly disagree

Moderately

disagree

Slightly disagree

Slightly agree

Moderately

agree

Strongly agree

1. When I begin learning a new language, I think about the level to which I want to learn it.

2. I set specific goals for myself in language learning.

3. I plan how I am going to learn so that I can use my time effectively.

4. I think about how I learn languages best.

5. I decide in advance to pay special attention to particular skills in English so I can learn them best.

6. I learn best when I am taught language rules.

7. I make notes of the mistakes I make in English so that I can learn from them.

8. I make efforts to improve my ability in English by spending time with native speakers of this language.

9. I try to improve my ability in English by asking other people to tell me if I have understood or said something correctly.

10. I repeat new words to make sure I have understood them correctly.

11. I make charts, diagrams or tables to organise what I have learned.

12. I try to make sure I remember new words by using them in new situations.

13. I learn new words in English by thinking of words that I know that sound like the new word.

14. I learn new words in English by dividing them into parts that I understand so I can figure out what they mean.

15. I learn new words in English by remembering where the new word was on the page, or where I first saw or heard it.

16. I learn grammar in English by comparing the rules in my language with grammar rules in English.

17. I learn grammar in English by memorising the rules and applying them to new situations.

18. When I speak or write in English I know when I make grammar mistakes.

19. When I listen to or read incorrect English I recognise grammar mistakes.

20. I make special efforts to improve my English by listening to programmes in English on the radio.

21. I try to improve my English by watching television programmes or films in English.

22. I try to improve my listening and reading in English by guessing the meanings of new words from the context.

23. I try to improve my reading in English by making notes to help me remember what I have read.



24. I try to improve my English by summarising new information I hear or read.

25. I try to improve my writing in English by analysing how writers organise their paragraphs.

26. I try to improve my writing in English by showing my writing to another person.

27. I try to improve my writing or speaking by including expressions I have read or heard.

28. I try to improve my writing in English by putting words and phrases that I meet into new practice sentences.

29. I try to improve my speaking in English by repeating sentences in English until I can say them easily.

30. I try to improve my speaking and writing in English by using my knowledge of grammar rules to help me form new sentences.

31. I try to improve my speaking in English by repeating what I hear native speakers say.

32. When I speak English I know when I have mispronounced something.

33. After I finish a conversation in English, I think about how I could have said things better.

34. After I have said something in English, I check whether the person I am talking to has really understood what I meant.

35. I want to improve my general ability to use English.

36. I encourage myself to use English even when I am afraid of making mistakes.

37. I read for pleasure in English.

38. I have several close friends who are native speakers of English.

39. I try to learn about the culture of English speakers.

40. I learn more by reading textbooks than by listening to lectures.

41. I enjoy working on an assignment with two or three classmates.

42. I learn better in class when the teacher gives a lecture.

43. I learn better working alone than in a group.

44. I enjoy learning in class by doing experiments.

45. I remember things I have heard in class better than things I have read.




If you have already taken the IELTS, please now complete PART 4 of the questionnaire,

B. Please tick (! ) the appropriate for each item on the ways you do tests. Never Rarely Often Always

1. I decide which parts of the test are the most important before I start.

2. I glance through all the questions in the test before I start doing them.

3. I plan so that I have enough time to answer each question.

4. I read and think about the instructions in detail before I try to answer the questions.

5. I highlight (or underline) key words in the instructions and keep them in mind while completing the task.

6. I read the whole text before I start to answer the questions.

7. I use the words in the questions to find the sentence in the text which contains the answer.

8. I first skim a text to look for the main ideas.

9. Before I listen to something, I try to guess what information is coming.

10. I try to predict the questions when listening.

11. I expect to hear the information for the answers in the same order as the questions.

12. I make a plan of my whole answer before I write.

13. I write a draft of my whole answer.

14. I do not write much more than the minimum word requirement.

15. I try to include words or phrases to organise my speaking /writing (eg firstly, furthermore, secondly, I have two points…)

16. I practise using set conversational phrases (eg let me see now...; what shall I say... etc) to fill in silences.

17. I am prepared to speak first in a conversation test.

18. I don't wait to be asked before speaking.

19. I check my answers before I leave the exam.

20. After a test, I usually feel that I have done as well as my knowledge and ability deserve.



PART 4: FOR THOSE WHO HAVE TAKEN IELTS

1.1 Are you a student at present? Yes No

1.1a. If so, at what level are you studying? High School

Undergraduate

Post-graduate

Other:

1.1b. What degree are you studying for (eg High school diploma, BA, MSc., PhD)?

1.1c. In which country are you studying now?

1.2 If you are not a student at present, please describe your current status (eg waiting to enter University, already graduated, teaching) :

2. When did you take the IELTS Test? (Please include all

dates if you have taken the test more than once.)

3. Why did you take IELTS?

4. Have you taken other similar tests (TOEFL; CAE/CPE/FCE; Michigan, others?) Why? / Why not?

5. For the IELTS reading and writing section did you take

the:

General Training (GT) module?

Academic (A) module?

6. If you know your IELTS band score, (range 1 to 9):

6.1 which band score(s) did you get when you took the IELTS? 6.2 which band score(s) did you need for your studies, your

work, your goal, or other requirement?

6.3 which band score(s) did you expect to get after taking the test?

Very much

1

2

3

Very little

4 7. Did you worry about taking the IELTS Test?

8. What worried you most about the test?



9. Please rank the sections of the IELTS Test according to how difficult you found them

The most difficult 1

Second most difficult

2

Third most difficult

3

The easiest

4 9.1 Listening 9.2 Reading 9.3 Writing 9.4 Speaking 10. Did you feel that you performed to the best of your ability in the test? Yes No

11. When you took the IELTS Test, what affected your performance?

A lot Quite a lot Not a lot Not at all

11.1 Difficulty of the language used on the test

11.2 Difficulty of the questions included in the test

11.3 Unfamiliarity of the topics on the test

11.4 Time pressure

11.5 My fear of tests

11.6 Others (please specify)

12. Do you think IELTS is a fair way to test your proficiency in English? Yes No

Why (not)? 13. Except for language ability, what other knowledge or skills (if any), are needed for a good IELTS

score?



14. In your opinion, is the IELTS exam appropriate for the

following groups? Yes No No opinion

14.1 all ages 15+

14.2 university students (under-graduates)

14.3 graduate students (post-graduates, Masters, PhD)

14.4 professional people

14.5 all nationalities/cultures

14.6 students in all subject areas

15. What did you like about the IELTS Test? 16. What did you dislike about the IELTS Test?

17. What advice would you give someone who is going to take the IELTS Test? 18. Please write here any comments other about IELTS, which were not covered in the above items:

That is the end of the questionnaire. Thank you very much for responding to the items and answering the questions.



APPENDIX 2: TEST PREPARATION (TP) COURSE STUDENT QUESTIONNAIRE STUDY OF THE IMPACT OF THE INTERNATIONAL ENGLISH LANGUAGE TESTING SYSTEM (IELTS) STUDENT QUESTIONNAIRE

Dear Participant,





Your consent to participate in the IELTS Impact Study I understand that: • the purpose of the study is to collect and analyse information from those familiar with the IELTS; • my name will not appear in any project publication; • the information I give, but not my name, may be quoted; • I am free to refuse to participate in the study and may withdraw at any time; • my completed questionnaire is for the study team only; it will not be shown to anyone not connected with

the study. Signature: …………………………………… Date: ……………………………

Please give the information about yourself requested below in the relevant spaces and by ticking (!) the appropriate .

Home country

Nationality Full name and current address

Male Female Age: ______

Answer one of the following questions:

A. Do you work? If yes, what is the name of your employer? ---------------------------

B. Are you a university student? If yes, what is your major subject (in university)? ----------------------

C. Are you a high school student? Yes No



There are 2 Parts modules in this questionnaire:

• PART 1: for all students and candidates, on participants' language learning background. • PART 3: only for students who are taking or have taken a course to prepare them for the

IELTS, on their experiences on the course. You will be advised on which Parts you should complete.

Please now complete PART 1 of the questionnaire, which is about your language learning background.

PART ONE: ENGLISH LANGUAGE BACKGROUND

1. What language(s) do you usually speak at home when in your country?

Yes No 2. As a child, did you live with a parent, guardian or other close

relation who was a native speaker of English?

3. Did you study English in: Yes No 3.1 kindergarten (age 3-6)? 3.2 primary school (age 7-11)? 3.3 secondary school (age 12-17)? 3.4 college / university? 3.5 extra language classes? 4. How many of your English

lessons in your country were taught or assisted by native speakers of English?

More than half

About half

Less than half

A few

None

5. How much of the time in

class did your teachers of English speak to you in English?

All the time

More than half

About half

Less than half

Never

6. In your home country, how often: Often Some-

times Rarely Never

6.1 do you use English for socialising (talking or writing to friends)?

6.2 are you in English contact with friends overseas? 6.3 are you exposed to English in the media? 6.4 do you use English for work? 6.5 do you read texts in English in your specialist subject? 6.6 do you write in English in a day? 7. Have you studied in an English-speaking

country? Where, when and for how long?

Very often Quite

often Once/twice

only Never 8. How often have you stayed in English-

speaking countries for a week or more?




If you attended / are attending IELTS preparation classes, please now complete PART 3 of the questionnaire,

PART 3: FOR THOSE ATTENDING IELTS PREPARATION CLASSES

1.1 Are you still attending IELTS preparation classes? Yes No

If yes, where and who is organising?

If taking a class at ACE, which class?

IELTS Overview IELTS Preparation General English Program (GEP) Level 11 or 12

1.2 Did you attend such classes in the past? Yes No

If yes, where and who organised?

2. What kind of IELTS preparation classes and for how long? Yes No Weeks Hours/week

2.1 A course with “IELTS” in its title?

2.2 Part of a general English course?

2.3 Part of an English language study skills course?

2.4 Part of an English for Academic Purposes (EAP) course?

2.5 Part of an English for Occupational Purposes (EOP) course?

Very useful Quite useful Not very useful

3. What proportion of 100% of the time on your IELTS-preparation course is / was spent working on the following, and how useful do you believe they are for the IELTS Test?

% of time

for IELTS

3.1 Reading 3.2 Writing 3.3 Listening 3.4 Speaking 3.5 Vocabulary 3.6 Grammar 3.7 Other (please specify)

Total 100%



4. Did / Do any of the following happen in your IELTS preparation classes? Yes No Not

sure Listening: 4L.1 Reading the questions and predicting what listening passages would be

about.

4L.2 Listening to live, taped or video talks / lectures and taking notes. 4L.3 Listening and taking part in seminar / workshop activities. 4L.4 Using information from a lecture or talk to write reports. 4L.5 Reading questions and guessing the type of answer required. 4L.6 Practice in recognising previous information repeated in different words.

Reading:

4R.1 Analysing text structure and organisation. 4R.2 Interpreting statistics / graphs / diagrams. 4R.3 Reading texts to predict test questions and tasks. 4R.4 Learning quick and efficient ways of reading texts in English. 4R.5 Reading articles, reports, books in your specialist subject area. 4R.6 Using English-only dictionaries to complete reading tasks. 4R.7 Reading quickly to get the main idea of a text.

Writing:

4W.1 Copying out good paragraphs and model answers. 4W.2 Describing a graph / a process / statistical data. 4W.3 Learning how to organise essays. 4W.4 Practising using words or phrases to organise a written text

(eg firstly, furthermore, secondly, therefore)

4W.5 Learning how to write in different styles. 4W.6 Short report writing. 4W.7 Planning written answers to test questions. 4W.8 Editing written work. 4W.9 Writing parts of test answers 4W.10 Writing long essays, reports (ie over 1000 words).

Speaking:

4S.1 Practising making a point and providing supporting examples. 4S.2 Planning and delivering oral presentations. 4S.3 Group discussions / debates. 4S.4 Practising using filler words to cover silences in your speech

(eg well...you see…)

4S.5 Practising using words or phrases to organise your speech (eg firstly, furthermore, secondly, I have two points)



5. How much of the following kinds of specific exam practice do / did you do on your preparation

course (each type as approximate percentages [%], added together should equal 100%):

Specific IELTS Exam Practice Approx % of course time

5.1 Information about the contents and format of the test

5.2 Looking at past papers

5.3 Taking practice tests

5.4 Receiving feedback in the form of IELTS band scores. 5.5 Techniques for taking the test (please specify)

5.6 Others (please specify)

Total 100% 6. Did you use (a) textbook(s) on your IELTS preparation course? If so, what was / were the title(s)

(approximately if you cannot remember exactly), and what did it / they contain?

7. If you did use (an) English textbook(s), please give your opinions of the good and less good points here.

Good Points

Not so good points

8. What other materials did you use on the preparation course (eg online materials, handouts, worksheets, supplementary materials)? How were they (eg useful / not useful; difficult / easy; enjoyable / not enjoyable)?

9. What does a successful student do on the IELTS preparation course that an unsuccessful one does not do?

10. Do you think you were / are successful on the preparation course(s)? Why? Why not?



11. If an IELTS score had not been a requirement (for your application or for a scholarship), would

you have prepared for your studies abroad in the same way?

Yes No

Why / Why not?

12. If you are already studying / working in an English-medium situation now, do you find that your IELTS preparation course provided you with the language knowledge and skills you need? Please comment.

13. Would the IELTS preparation course be a good way to learn English for someone who is not going to take IELTS? Why? / Why not?

14. Would the IELTS preparation course be useful for someone who is not going to university? Why / Why not?

15. Please note here anything else you wish to say about your IELTS preparation course.


That is the end of the questionnaire. Thank you very much for responding to the items and answering the questions.



APPENDIX 3: IELTS TEST PREPARATION COURSE TEACHER QUESTIONNAIRE

STUDY OF THE IMPACT OF THE INTERNATIONAL ENGLISH LANGUAGE TESTING SYSTEM (IELTS) TEACHER QUESTIONNAIRE (for Teachers preparing students for IELTS)

Dear Participant,





Your consent to participate in the Impact Study I understand that:

! the purpose of the study is to collect and analyse information from those familiar with international English language tests;

! my name will not appear in any project publication; ! the information I give, but not my name, may be quoted; ! I am free to refuse to participate in the study and may withdraw at any time; ! my completed questionnaire is for the study team only; it will not be shown to anyone not

connected with the study. Name: ……………………………………….. Signature: ……………………………………. Date: ……………………………. Thank you.



QUESTIONNAIRE FOR TEACHERS PREPARING STUDENTS FOR IELTS Please tick (! ) the boxes or write in your responses as appropriate.

SECTION A: About you 1. Your full name:

2. Form of address: Miss Mrs Mr Dr Other, please specify:

3. Your age, please: below 30 31-40 41-50 51-60 61+

4. Country you now work in:

5.

Name, address and email of institution where you work:

6. Your position there:

7. Number of years you have been teaching English:

8. Your qualifications:

9. Have you been trained as an examiner for IELTS or other international proficiency test(s)? Yes No

If, yes, please explain a little:

10. Have you received any training in how to prepare students for IELTS? Yes No

If yes, please describe briefly:

SECTION B: About your students

11. Ages of your IELTS students? 11a. Their country(ies)

of origin:

12. Your IELTS students’ level(s) of education?

secondary up to 16

years

secondary 17-19 years

degree or equivalent

post-graduate

13. Applying for which country(ies)?

All Most Half A few Academic module 14. Taking which IELTS

modules? General training module

All Most About half A few None 15. Proportion of students who have

already taken IELTS? If you have taught students who have already taken IELTS, please answer items 16-20. If not please go to Section C.



16-20 Comparing your students’ results in the IELTS Test with your own assessment of their language ability:

Compared with their IELTS results, I consider higher about the same

not consistently

related lower

16. my students’ actual general English proficiency level is

17. my students’ actual level in reading is

18. my students’ actual level in writing is

19. my students’ actual level in listening is

20. my students’ actual level in speaking is SECTION C: About the IELTS 21-26. Do you consider the IELTS Test appropriate to candidates' future English language needs:

Yes No Not sure 21. at undergraduate level? 22. at postgraduate level? 23. at pre-university level? 24. for vocational studies? 25. in their professional work? 26. for immigration purposes? 27. How does a current IELTS Band 6 compare with a Band 6 in previous years?

Higher Lower Unchanged Not sure

Yes No Don’t know

28. Does the IELTS Test provide positive motivation for your students?

29. Does the test cause unhelpful stress for your students?

30. Does the IELTS Test influence your choice of the content of your IELTS preparation lessons (ie what you teach)?

31. If yes, please note here how the test influences your decisions on lesson content:

Yes No Don’t know 32. Does the IELTS Test influence your choice of

methodology (ie the way you teach) for IELTS preparation lessons?



33. If yes, please note here how the IELTS test influences

the way you teach:

Please complete the following statements: 34. What I like about teaching for IELTS is ..…………………………………………………………………………………………………………………………………. 35. What I dislike about teaching for IELTS is …………………………………………………………………………………………………………………………………… 36. What knowledge or skills other than proficiency in the English language do you think help students achieve a good IELTS grade?

37. What advice would you give to a colleague who was about to prepare students for IELTS for the first time?

38. Compared with other English language classes you have taught, do you think your IELTS preparation classes are:

more successful?

as successful?

less successful?

39. Why?

Rank order 40.1 reading 40.2 writing 40.3 listening

40. Please rank the IELTS Test sections in order of difficulty for most of your students (1 = most difficult …4 = least difficult etc):

40.4 speaking

41- 48 Are the following statements about the IELTS Test correct?

Yes No Not sure

41. The IELTS Test includes a section testing grammar.

42. In the speaking module, candidates have to both ask and answer questions.

43. Reading and writing together carry more than half of the marks.

44. Candidates have two opportunities to hear the cassette in the listening module.

45. Candidates have to write at least 150 words for the first task in the writing module.

46. Candidates often need to refer to the reading texts when they do the writing module.

47. The reading module has three sections.

48. In the listening module, candidates may have to label a diagram.



SECTION D: About IELTS preparation classes

What kind of IELTS preparation classes are you teaching? (If you teach more than one type of IELTS class, indicate on which class you are basing your answers for this questionnaire)

IELTS Preparation IELTS Overview General English Program (GEP) (Levels 11-12)

Yes No Weeks long Hours/week

49. a course with “IELTS” in its title

50. part of a general English course 51. part of an English language study skills course 52. part of an English for academic purposes ( EAP) course

53. part of an English for occupational purposes (EOP) course

54. other (please specify) 55. How many students on average attend the IELTS class(es) you teach?

1-5

6-10

11-15

16-20

21-25

26+

56. Are the IELTS courses normally taught by one, or more than one teacher? Please explain:

57. What proportion of the time on your IELTS-preparation course is normally spent working on the following, and how useful do you believe they are for the ELTS test?

For IELTS Skill, component etc % of time

very useful quite useful not very useful

57.1 Reading 57.2 Writing 57.3 Listening 57.4 Speaking 57.5 Vocabulary 57.6 Grammar 57.7 Others (please specify)

58. Which of the following activities take place in your normal IELTS-preparation class?

Yes No Not Sure Listening: 58L.1 reading questions and predicting what listening texts will be about

58L.2 listening to live, cassette or video talks / lectures and taking notes

58L.3 listening and taking part in seminar / workshop activities

58L.4 using information from a lecture or talk in written reports

58L.5 reading questions and guessing the types of answer required

58L.6 practice in recognising previous information repeated in different words

Reading:

58R.1 analysing text structure and organisation

58R.2 interpreting statistics / graphs / diagrams

58R.3 reading texts to predict test questions and tasks

58R.4 learning quick and efficient ways of reading texts in English

58R.5 reading articles, reports, books in your specialist subject area



58R.6 using monolingual dictionaries to complete reading tasks

58R.7 reading quickly to get the main idea of a text

Writing:

58W.1 copying out good paragraphs and model answers

58W.2 describing graph / process / statistical data

58W.3 learning how to organise essays 58W.4 practising using words or phrases to organise a written text (eg firstly, furthermore, secondly, therefore)

58W.5 learning how to write in different styles

58W.6 short report writing

58W.7 planning written answers to test questions

58W.8 editing written work

58W.9 writing parts of test answers

58W.10 writing long essays, reports (i.e over 1000 words).

Speaking:

58S.1 practising making a point and providing supporting examples

58S.2 planning and delivering oral presentations

58S.3 group discussions / debates

58S.4 practising using filler words to cover gaps in speech (e.g well..., you see…)

58S.5 practising using words or phrases to organise a speech (eg firstly, furthermore, secondly, I have two points)

59. How much, if any, of the following kinds of specific exam practice do you give on your preparation course (as approximate percentages (%) of the course): approx % of course time

59.1 information about contents and format of the test 59.2 looking at past papers 59.3 taking practice tests 59.4 marking and giving feedback in the form of IELTS band scores

59.5 techniques for taking the test (please specify)

59.6 others (please specify)

60. Do you use (a) textbook(s) on your IELTS

preparation course. If so, what is / are the title(s) (approximately if you cannot remember exactly)

61. If you do/did use (a) textbook(s), please give your opinions of their good and less good points here: Good points: Not so good points:



62. What other teaching materials do you use on your IELTS preparation course(s) and why? IELTS prep-course teaching materials Reason

63. What does a good/ successful student do on the IELTS preparation course that an unsuccessful one does not?

64. If an IELTS score had not been a requirement would you have prepared your students for their future studies abroad in the same way? Yes No

65 Would your IELTS preparation course be a good way to learn English for someone going to university but who is not going to take IELTS? Why? / Why not? 66. Would the IELTS preparation course be useful for someone who is not going to university? Why? Why not? 67. Please note here anything else you wish to say about your IELTS preparation course. Thank you very much for your help with the Impact Study.



APPENDIX 4: MATERIALS EVALUATION QUESTIONNAIRE

STUDY OF THE IMPACT OF THE INTERNATIONAL ENGLISH LANGUAGE TESTING (IELTS) ON TEXTBOOKS Instrument for the Analysis of Textbook Materials (IATM)

Dear Participant,




Yours sincerely

Dr. Stephen Moore Senior Lecturer Linguistics Department Faculty of Human Sciences Macquarie University North Ryde NSW 2109 Australia

Your consent to participate in the Impact Study I understand that:

! the purpose of the study is to collect and analyse information from those familiar with international English language tests;

! my name will not appear in any project publication; ! the information I give, but not my name, may be quoted; ! I am free to refuse to participate in the study and may withdraw at any time; ! my completed questionnaire is for the study team only; it will not be shown

to anyone not connected with the study.

Name: ………………………………………. Signature: …………………………………… Date …………………………………... Thank you.

A little about you (please write, type, tick (!) boxes , leave blank as appropriate)



Your full name

Form of address Miss Mrs Mr Dr Other (please specify)

Country where you were born

Your first language

Name and address of institution where you work

Your position Your academic / professional qualifications

International English language test(s) with which you are familiar

Test(s) Experience Your experience with this / these test(s) (teaching test preparation courses, test administration, being trained as examiner etc), if any

Test(s) Comment(s)

Your brief opinion of this test / these tests (eg any comments on: test components, levels, topics, skills, format, scoring, administration, reliability in predicting a student’s English language competence and performance?)



Notes on the use of the Instrument for the Analysis of Textbook Materials (IATM)

1. This questionnaire seeks your analysis and evaluation of the content, level and approaches of a textbook and its support materials.

2. We are especially interested in the relationships between textbooks and international tests for the certification of language performance.

3. The questionnaire invites: ! objective analytic responses on features of the book (using a tick (! ) in the appropriate boxes ) ! additional comment on most items and on relevant matters not covered, to be written in the

appropriate spaces ! your evaluation of the book’s coverage of the four skills, and of the book as a whole, to be written in

the spaces provided. We are grateful for your help and look forward to reading your responses. Thank you.

INSTRUMENT FOR THE ANALYSIS OF TEXBOOK MATERIALS (IATM)

0. The textbook being analysed: Title:

Author(s)

Publishers

Place of Publication Year of publication

What materials in addition to

this book, if any, do you use when teaching students?

Which students are you teaching using this book?

Questions on the kind of book this is, in general aim and organisation

1. What kind of book would you say this is? (please tick (!) the box where appropriate).

1.1 a language teaching book with no specific reference to international tests

1.2 a book of practice tests only

1.3 a language teaching book and an international test preparation book combined

1.4 If it is a test-related book, for

which test(s)?

1.5 Any other comment on the type of book this is?

• If the book is a book of practice tests only, please go to Question 4.

• • If the book contains teaching material as well as practice tests, please go to Question 2, about the way the book is organised.



2. The book’s units / chapters etc seem to be organised mainly according to:

(please tick (! ) the box(es) where appropriate, more than one possible).

2.1 topics,

themes

2.2. language skills

2.3 grammatical structures

2.4 tests, tasks

2.5 notions, functions

2.6 Other (please specify)

2.7 Any further comment on the organisation of the book?

Now a question on whether the book tries to break the language down and teach the elements of the listening, reading, writing and speaking skills.

3. Your analysis of the book’s explicit practice of language features. (Please tick (! ) appropriate boxes .)

A lot

A little

None

A lot

A little

None

A lot

A little

None

3.1 recognition of sounds

3.2 pronunciation

of sounds

3.3 stress and intonation

3.4 grammar 3.5 sentence

patterns 3.6 notions,

functions

3.7 word formation 3.8 connotation 3.9 collocation

3.10 idioms 3.11 linking words,

expressions 3.12 punctuation

3.13 Other language components or features given explicit practice:

3.14 Related comments on how the book treats language features:

Questions 4, 5 and 6 ask whether the book teaches and/or tests particular enabling or micro-skills, using a variety of techniques and activities?

Try checking 4, 5 and 6 before you comment, as skills, question / tasking and activities clearly overlap.



4. Enabling skills you think are covered in the book:

(please tick (! ) appropriate boxes .) 4.1 understanding and conveying meaning through stress and intonation

4.2 retrieving and stating factual information

4.3 identifying main points

4.4 drawing conclusions

4.5 identifying overall meaning

4.6 predicting information

4.7 making inferences

4.8 evaluating evidence

4.9 distinguishing fact from opinion

4.10 recognising roles

4.11 identifying attitudes

4.12 planning and organising information

4.13 Other skills covered by the book

(please specify):

4.14 Further comment on skills covered or not covered by the book:

5. Your summary of the use of question / tasking techniques in the book: (please tick (! ) appropriate boxes .) Frequent

A little

None

Frequent

A little

None

5.1 multiple / dual choice 5.2 conversion 5.3 true / false 5.4 sequencing 5.5 matching 5.6 paraphrasing 5.7 substitution 5.8 open-ended questions 5.9 linking / joining 5.10 note taking / making 5.11 expansion 5.12 correcting / editing 5.13 gap filling / completion 5.14 summarising

5.15 Other techniques

(please specify)

5.16 Further comment on question and task techniques covered or not covered by the book



6. Your evaluation of the extent to which the materials provide / encourage the following kinds of

communicative opportunities. (Please tick (! ) appropriate boxes .) A

lot

Quite a lot

Very little

None

A lot

Quite a lot

Very little

None

6.1 pair communication

6.2 group discussions and debates

6.3 games, puzzles, quizzes

6.4 role play, simulations

6.5 surveys, other project work

6.6 report writing

6.7 review writing 6.8 essay writing

6.9 creative writing

6.10 IT eg telephone, fax, letters, email, web

6.11 listening, reading, viewing for personal interest

6.12 other

communicative opportunities

(please specify):

6.13 Further comment on the communicative opportunities offered by the book:

Questions 7 and 8 ask for information on test types and topics to check the coverage of the books.

7. How would you categorise the text types (heard, spoken, read, written) in the book? (Please tick (! ) appropriate boxes .)

7.1 public announcement

7.2 lecture/ talk

7.3 press report

7.4 textbook/ journal article

7.5 correspondence

7.6 fiction

7.7 discussion

7.8 face-to-face conversation

7.9 radio/ TV report

7.10 manual / brochure

7.11 advertising

7.12 maps, charts, tables and graphs

7.13 interview

7.14 telephone

7.15 email

7.16 internet

7.17 other text type(s)? (please specify)



8. And the book's text topics (heard, spoken, read, written)? (Please tick (! ) appropriate boxes .)

8.1 accommodation

8.2 health

8.3 physical environment

8.4 leisure and sports

8.5 daily routines

8.6 education training

8.7 science

8.8 travel

8.9 shopping

8.10 world of work

8.11 arts

8.12 current affairs

8.13 food and drink

8.14 social environment

8.15 customs

8.16 moral issues

8.17 Other topics: (please specify):

8.18 Any inappropriate topics: (please exemplify and explain):

• If the book has no recorded texts, please go to Question 11.

O Questions 9, 10, seek your views on the authenticity of the listening and reading texts and tasks.

9. Authenticity of listening texts and tasks: (Please tick (! ) appropriate boxes .)

9.1 Do the listening text(s) appear: scripted? authentic? some of each

9.2 Do the recorded texts include redundancies such as:

repetition? rephrasing? hesitation?

9.3 Please comment on the authenticity or realism of the listening tasks:



10. Authenticity of reading texts and tasks: (please tick (! ) the appropriate boxes )

10.1 Do the reading texts seem:

adapted or written for the book?

authentic?

some of each?

10.2 Please comment on the authenticity or realism of the listening tasks:

O Most of the information you have been asked to provide so far has been relatively objective.

Questions 11 and 12 here are very important as they request you to give your evaluation of how the book treats the main language skill areas, and of the book as a whole.

11. Please give your comments on the book’s treatment of the four language skills:

11.1 Listening

11.2 Reading

11.3 Writing

11.4 Speaking

12. Please now evaluate the whole textbook, preferably in terms of: • type • level • contents • pedagogical approach • interest



13. Finally please evaluate the book’s relationship with the international test for which you use it to prepare your students.

How does the book help your students to cope with the international test you are preparing them for?

That is the end of the questionnaire. Thank you very much for answering the questions.



APPENDIX 5: IELTS EXAMINER FOCUS GROUP DISCUSSION POINTS

1. How many years have you been an IELTS examiner?

2. Have you examined IELTS in countries other than Cambodia?

If so, where?

3. Have you noticed any differences in examining IELTS in Cambodia compared with in other countries?

If so, what?

4. What impact do you think IELTS has in Cambodia?

5. What evidence is there that supports your claim(s) about the impact of IELTS in Cambodia?

6. What impact do you think IELTS has on:

-students who take it?

-teachers who prepare them for it?

-subject lecturers who teach the students who have taken it?

-other stakeholders?

7. Is there anything further you would like to say about the impact of IELTS in Cambodia?



APPENDIX 6: PARENT/GUARDIANS FOCUS GROUP DISCUSSION POINTS

1. How many times has your child taken the IELTS exam?

2. What do you know about the IELTS exam?

3. What impact do you think IELTS has in Cambodia?

4. Do you think it is helpful or an unnecessary burden for students? Why?

5. What impact do you think IELTS has specifically on:






APPENDIX 7: EMBASSY OFFICIALS INTERVIEW QUESTIONS

1. How long have you used IELTS exam results?

2. In what way do you use IELTS exam results? (eg what sort of decision making? High stakes or low stakes?)

3. Do you think that IELTS results are useful? Why/why not?

4. Do you find IELTS results easy to interpret? Why/why not?

5. Have you ever based a decision on IELTS scores and then found them to have been misleading? Any exemplifications of this point?

6. What impact do you think IELTS has overall in Cambodia?







APPENDIX 8: EMPLOYERS INTERVIEW QUESTIONS

1. How long have you used IELTS exam results?

2. In what way do you use IELTS exam results? (eg what sort of decision making? High stakes or low stakes?)

3. Do you think that IELTS results are useful? Why/why not?

4. Do you find IELTS results easy to interpret? Why/why not?

5. Have you ever based a decision on IELTS scores and then found them to have been misleading? Any exemplifications of this point?








APPENDIX 9: RETURNED GRADUATES INTERVIEW QUESTIONS

1. Why did you take the IELTS exam?

2. When and where did you take the IELTS exam? (If you took it more than once, how many times?)

3. Do you recall your overall or individual band scores achieved to enable you to study/work abroad?

4. Do you think the IELTS exam prepared you well for the challenges you faced overseas? Why/why not?

5. Would you recommend the IELTS exam as useful preparation for others who plan to study in English-speaking countries? Why/why not?








APPENDIX 10: LESSON OBSERVATION PROTOCOL

IMPACT STUDY LESSON OBSERVATION ANALYSIS FORM Type of institution Course type Class number, m / f Nationalities, ages, level, etc Recorded / not recorded

Date, time

Teacher Materials, references etc

Episode Timing (mins)

Activity, participation, materials

Comments

1 2 3 4

5 6 7 8 9 Comments

Student Communicative Opportunity Analysis data:

Class listening to T

Individual/ pair/ gp / (other) listening to T

Individual listening in pair / group / (other)

Individual speaking in pair/ group / (other)

Individual speaking to class

Writing

Reading

Other

Note:



APPENDIX 11: GROUP STATISTICS: LEARNING AND STUDY HABITS

Group Statistics: Learning and Study Habits N Mean Std.

Deviation Std. Error

Mean TP 101 4.79 1.381 .137 1. When I begin learning a new language, I think about the level to

which I want to learn it. TT 105 4.90 1.447 .141

TP 102 5.25 .999 .099 2. I set specific goals for myself in language learning.

TT 106 4.99 1.159 .113

TP 99 5.06 1.086 .109 3. I plan how I am going to learn so that I can use my time effectively. TT 106 4.77 1.157 .112

TP 101 5.13 .956 .095 4. I think about how I learn languages best.

TT 106 5.11 .876 .085

TP 101 4.81 1.164 .116 5. I decide in advance to pay special attention to particular skills in English so I can learn them best. TT 106 4.67 1.058 .103

TP 102 4.42 1.057 .105 6. I learn best when I am taught language rules.

TT 106 4.38 1.215 .118

TP 102 4.88 1.102 .109 7. I make notes of the mistakes I make in English so that I can learn from them. TT 106 4.93 1.149 .112

TP 102 4.65 1.240 .123 8. I make efforts to improve my ability in English by spending time with native speakers of this language. TT 106 4.55 1.303 .127

TP 102 4.50 1.257 .124 9. I try to improve my ability in English by asking other people to tell me if I have understood or said something correctly. TT 106 4.46 1.360 .132

TP 102 4.74 1.143 .113 10. I repeat new words to make sure I have understood them correctly. TT 105 4.93 1.085 .106

TP 102 3.74 1.482 .147 11. I make charts, diagrams or tables to organise what I have learned. TT 106 3.57 1.250 .121

TP 101 4.91 1.040 .104 12. I try to make sure I remember new words by using them in new situations. TT 106 4.49 1.221 .119

TP 99 4.33 1.262 .127 13. I learn new words in English by thinking of words that I know that sound like the new word. TT 105 4.30 1.264 .123

TP 102 4.30 1.265 .125 14. I learn new words in English by dividing them into parts that I understand so I can figure out what they mean. TT 105 4.41 1.246 .122



TP 102 3.86 1.350 .134 15. I learn new words in English by remembering where the new word was on the page, or where I first saw or heard it. TT 106 3.71 1.460 .142

TP 102 3.96 1.566 .155 16. I learn grammar in English by comparing the rules in my language with grammar rules in English. TT 106 3.65 1.627 .158

TP 102 4.75 1.041 .103 17. I learn grammar in English by memorising the rules and applying them to new situations. TT 106 4.78 1.163 .113

TP 102 4.15* 1.197 .119 18. When I speak or write in English I know when I make grammar mistakes. TT 105 4.71* 1.183 .115

TP 102 4.28 1.189 .118 19. When I listen to or read incorrect English I recognise grammar mistakes. TT 106 4.57 1.163 .113

TP 102 5.02 1.235 .122 20. I make special efforts to improve my English by listening to programmes in English on the radio. TT 106 5.07 1.157 .112

TP 102 5.37 1.014 .100 21. I try to improve my English by watching television programmes or films in English. TT 106 5.29 1.078 .105

TP 102 4.95 1.057 .105 22. I try to improve my listening and reading in English by guessing the meanings of new words from the context. TT 106 4.80 1.055 .102

TP 102 4.56 1.223 .121 23. I try to improve my reading in English by making notes to help me remember what I have read. TT 106 4.51 1.244 .121

TP 102 4.38 1.211 .120 24. I try to improve my English by summarising new information I hear or read. TT 106 4.08 1.317 .128

TP 101 4.72 1.184 .118 25. I try to improve my writing in English by analysing how writers organise their paragraphs. TT 106 4.46 1.205 .117

TP 100 4.41 1.364 .136 26. I try to improve my writing in English by showing my writing to another person. TT 106 4.11 1.355 .132

TP 102 4.85 1.047 .104 27. I try to improve my writing or speaking by including expressions I have read or heard. TT 106 4.67 1.177 .114

TP 102 4.72 1.230 .122 28. I try to improve my writing in English by putting words and phrases that I meet into new practice sentences. TT 106 4.80 1.108 .108

TP 102 4.42 1.254 .124 29. I try to improve my speaking in English by repeating sentences in English until I can say them easily. TT 105 4.45 1.352 .132

TP 102 4.79 1.028 .102 30. I try to improve my speaking and writing in English by using my knowledge of grammar rules to help me form new sentences. TT 106 4.82 1.217 .118



TP 102 4.71 1.148 .114 31. I try to improve my speaking in English by repeating what I hear native speakers say. TT 106 4.75 1.271 .123

TP 100 4.35* 1.234 .123 32. When I speak English I know when I have mispronounced something. TT 105 4.80* 1.069 .104

TP 102 4.85** 1.197 .119 33. After I finish a conversation in English, I think about how I could have said things better. TT 106 5.19** .863 .084

TP 102 4.79 1.213 .120 34. After I have said something in English, I check whether the person I am talking to has really understood what I meant. TT 106 4.92 1.030 .100

TP 102 5.36 1.106 .110 35. I want to improve my general ability to use English.

TT 106 5.60 .813 .079

TP 102 5.16 1.184 .117 36. I encourage myself to use English even when I am afraid of making mistakes. TT 106 5.31 .999 .097

TP 101 4.88 1.219 .121 37. I read for pleasure in English.

TT 106 4.93 1.044 .101

TP 102 3.42 1.601 .159 38. I have several close friends who are native speakers of English.

TT 105 3.61 1.848 .180

TP 102 3.92 1.433 .142 39. I try to learn about the culture of English speakers.

TT 106 4.22 1.387 .135

TP 102 3.97 1.382 .137 40. I learn more by reading textbooks than by listening to lectures.

TT 106 4.15 1.271 .123

TP 102 4.35 1.332 .132 41. I enjoy working on an assignment with two or three classmates.

TT 106 4.21 1.336 .130

TP 102 4.64 .983 .097 42. I learn better in class when the teacher gives a lecture. TT 105 4.49 1.084 .106

TP 102 3.89 1.628 .161 43. I learn better working alone than in a group. TT 106 3.60 1.566 .152

TP 102 4.60 1.221 .121 44. I enjoy learning in class by doing experiments.

TT 106 4.72 1.119 .109

TP 102 4.37 1.234 .122 45. I remember things I have heard in class better than things I have read. TT 106 4.58 1.309 .127

* Significant at the .01 level.

** Significant at the .05 level.



APPENDIX 12: GROUP STATISTICS: TEST-TAKING APPROACHES

Group Statistics: Test-Taking Approaches N Mean

Std. Deviation

Std. Error Mean

TP 101 2.80 .825 .082 1. I decide which parts of the test are the most important before I start. TT 106 2.92 .902 .088

TP 102 3.11 .855 .085 2. I glance through all the questions in the test before I start doing them. TT 106 3.18 .814 .079

TP 102 3.05 .860 .085 3. I plan so that I have enough time to answer each question.

TT 106 3.08 .794 .077

TP 102 3.45 .669 .066 4. I read and think about the instructions in detail before I try to answer the questions. TT 106 3.41 .714 .069

TP 102 3.35 .753 .075 5. I highlight (or underline) key words in the instructions and keep them in mind while completing the task. TT 106 3.15 .848 .082

TP 102 2.39 .924 .092 6. I read the whole text before I start to answer the questions.

TT 106 2.36 .886 .086

TP 102 3.51 .609 .060 7. I use the words in the questions to find the sentence in the text which contains the answer. TT 106 3.45 .554 .054

TP 101 3.18 .780 .078 8. I first skim a text to look for the main ideas.

TT 105 3.19 .681 .066

TP 101 3.30 .701 .070 9. Before I listen to something, I try to guess what information is coming. TT 106 3.11 .747 .073

TP 102 3.12* .735 .073 10. I try to predict the questions when listening.

TT 106 2.70* .841 .082 TP 102 3.25 .740 .073 11. I expect to hear the information for the answers in the same order

as the questions. TT 105 3.18 .718 .070

TP 102 2.83 .797 .079 12. I make a plan of my whole answer before I write.

TT 106 2.76 .811 .079

TP 101 2.39 .969 .096 13. I write a draft of my whole answer.

TT 106 2.24 .846 .082



TP 101 2.91 .981 .098 14. I do not write much more than the minimum word requirement.

TT 106 2.89 .908 .088

TP 101 3.52 .687 .068 15. I try to include words or phrases to organise my speaking /writing (eg firstly, furthermore, secondly, I have two points…) TT 106 3.35 .618 .060

TP 101 2.79 .864 .086 16. I practise using set conversational phrases (eg let me see now...; what shall I say... etc) to fill in silences. TT 106 2.71 .839 .082

TP 102 2.74 .855 .085 17. I am prepared to speak first in a conversation test.

TT 106 2.70 .807 .078

TP 102 2.34 .850 .084 18. I don't wait to be asked before speaking.

TT 104 2.27 .906 .089

TP 102 3.24 .881 .087 19. I check my answers before I leave the exam.

TT 106 3.19 .829 .081

TP 102 3.05 .849 .084 20. After a test, I usually feel that I have done as well as my knowledge and ability deserve. TT 106 2.96 .755 .073 * Significant at the .01 level.

** Significant at the .05 level.

perceptions of ielts in cambodia: a case study of test

Documents