Textbook Evaluations: The case of Grade Five’s English

For Me Classbook B

By: Jamel Abdenacer ALIMI

e-mail: jamel_alimi@yahoo.com

Date: 11 July, 2006.

… no one is really certain what criteria and constraints are actually operative in ELT contexts worldwide, and textbook criteria are emphatically local. Not all the criteria described would be deployed simultaneously, nor is the list definitive.

Leslie E. Sheldon (1988: 241)

The complex issue of ELT materials selection has, over the past few decades, spawned several published checklist-format criteria for course book analysis and evaluation (see, for example, Tucker 1975; Rivers 1981; Williams 1983; Sheldon, Op.Cit; Ur 1996; Littlejohn 1998; McDonough and Shaw 2001). Between them, the proposals so far advanced have considerably helped interpret their target coursebooks’ intentions, content, and method. Taken separately, however, each of these schemes appears to betray serious limitations, which pressingly calls into question their much-claimed standards of informedness, systematicity, and appropiateness.

The present paper addresses itself exclusively to the Sheldon textbook evaluation checklist (or STEC) mentioned earlier. Within its purview, it will seek to assess the said framework in terms of the theoretical import of its constituent criteria and, then, following its practical application to one of the classbook series currently in use at Sultanate of Oman’s basic education (BE) schools. To this end, we propose to divide the remainder sections as follows: Section One will provide a general overview of the factors in connection with the textbook to be evaluated. Sections Two and Three will, respectively, supply a brief theoretical assessment of the STEC criteria and an illustrative application of its very rubrics to Grade Five’s English For Me Classbook B ( English Language Curriculum Department 2003-2004 a). Section Four will, finally, discuss the outcomes arising from the preceding Section and consider the late 1980s sample proposal in its presently wider implications.

1- PRE-ASSESSMENT OF THE SHELDON CHECKLIST: FOCUS ON BACKGROUND FACTORS

The present Section aims to shed light on the Omani ELT context as representatively expounded by Sheikh Hamdan bin Khamis (SHK) BE School, where I am currently employed. The two-part survey will, in turn, look at some of the micro and macro factors in

1

connection with fifth graders. The data provided draw much on the prompts figuring in McGrath (2002:18-21).

1.1 MICRO FACTORS: 1.1.1 Learner Factors :

The learners here concerned with the Sheldon checklist-based textbook evaluation are all male fifth graders in their early teens (10 to 11 years). They all come from large, most often barely educated families and are brought up according to strict Sunni, Shiite, or Abadhi religious precepts. Depending on their parents’ early origins, they speak Belushi or Swahili as their first mother tongue in addition to Arabic1. Many of them come from relatively low-income families.

In addition to the above, the students in question tend to show fairly positive attitudes towards the English language, its speakers, the teachers and the School. In contrast, their in-class group homogeneity, aptitude, and proficiency are alarmingly limited. The situation is even worse given their dependency on language coursebooks, passivity in the learning process, incapability in adequately specifying their own wants and needs, and preference to rote-learning.

1.1.2 Learners’ Needs Based on the Classbook and Skills Book prescribed for Grade Five as well as class tests specifications, the needs of these fifth graders may be summed up in the following profile categories:

a- Dialect: Britishb- Language-skill emphasis: reading; writingc- Contexts and situations of use: everyday conversational Englishd- Sub-skills: listening; reading for specific details; drawing; …e- Notions: time; locations; occupations; …f- Functions: greetings; thanking; denying; narrating; describing; ….g- Large-system emphasis: vocabulary; pedagogic grammarh- Language forms (e.g., grammatical structures, lexical items, and features of stress and

intonation): varied but basici- Future use of the language system: both receptively and productivelyj- Attention given to mechanics (handwriting, spelling, punctuation): reasonably

attended to.

1.1.3 Teacher Factors

The teachers of English here involved are three Tunisians, three Omanis, one Sudanese, one Egyptian, and one Indian (a PhD holder in English Literature). Apart from my two Tunisian fellow countrymen, none of the rest has ever been to Britain or any other English-speaking countries, which, at times, betrays a serious lack of familiarity with the target culture. Their competence as target language users and analysts is well below native-like standards (Dr. Vijay not included). As yet, their methodological competence and awareness, including, most importantly, the ability to adapt course books and to prepare supplementary materials, are well-developed. Their teaching beliefs and styles have been tuned to recently developed methodology to teaching English to young learners2, following a month-long intensive training course.

1.1.4 The Institutional and Specific Programme Factors

a- Level within the educational system: Basic Education (Cycle 2)

2

b- Sector: publicc- Role of the target language: overwhelmingly viewed as a curriculum subject (and not

as a skill)d- Time available for the study of the L2:

Per day: one 45-minute period (from Saturday to Wednesday) Per week: 5 periods (running from September to Mid-December and, then, from Mid-February to Mid-June)

e- Time-table: just one 45-minute lesson any time from Period 1 to Period 8f- Class size: 34 to 36 studentsg- Physical environment: spacious classrooms; flexible seating arrangements; clear

acousticsh- Additional resources available: sufficient cassette and video recorders; overhead

projectors; two computer laboratories; one Learning Resources Centrei- Aims of the programme: to “provide children with a firm foundation [ in grammar and

lexis] on which to build in later grades” (English Language Curriculum Department 2003-2004b: viii)

j- Syllabus: threshold; elementaryk- Form of evaluation: pen-and-pencil tests; portfolio-based continuous assessmentl- Decision-making mechanisms: top-down directives from the English Language

Curriculum Departmentm- Freedom given to teachers: very insignificant (teachers are required to cover all the

five 15-lesson units and observe as strictly as possible the way each of the lessons is detailed in the Teacher’s Book)

1.2 MACRO FACTORS

The English language occupies a very important position in the socio-political system in Oman— coming only second after Arabic, the mother tongue of the country3. It is widely used, though not necessarily needed, in many walks of life and is increasingly regarded as a means for future job employment and promotion. It is also the official language of the Sultanate for international affairs and diplomatic ties with non-Arab-speaking countries.

The above data, while strictly concerned with the factors regarding the Grade Five students described earlier, are, to a great extent, also indicative of the situations experienced by peer learners in other grades not only at SHK School but also other BE institutions across the country. This will allow the observations and other related inferences in Section Three to be viewed from a realistically more generalizable perspective. By the same token, this will help situate the illustrative application of the Sheldon framework in the subsequent Section in its broader, externally-bound dimensions.

2- THEORETICAL ASSESSMENT OF THE SHELDON CHECKLIST

The Sheldon checklist consists of 53 evaluative criteria referring to seventeen different sections (see Appendix One for full details). This comparatively wide array of rubrics and accompanying question-format items is drawn up with the overt intent of “mak(ing) evaluation and selection more systematic and informed” (Sheldon 1988: 237).

Bearing in mind the set of factors in Section One, though, it soon appears that not all the criteria posited there-in would bear the same importance throughout. Accordingly, some of them will be highly prioritized while others will necessitate supplementation or, simply, deletion. Falling in the first category are, most likely, the rubrics labelled “Rationale”, “User definition”, “Lay-out/graphics”, “Accessibility”, “Linkage”, “Selection/grading”, “Appropriacy”, “Authenticity”, “Cultural bias”, and “Stimulus/practice/revision” . Those of

3

relatively less importance and, thus, potentially prone to deletion plausibly refer to “Guidance”, “Overall value for money”, “Availability”, “Physical characteristics”, and “Educational validity”. Addition, on the other hand, would concern such criteria as the following:Lay-out/graphics: Are the pictures showing boys and girls equitably distributed? Are they gender-biased?Accessibility: Is the core vocabulary explained in a separate English-Arabic glossary section in the textbook?

It goes without saying that the above remarks are by no means exhaustive. Other modifications to the checklist at issue can surely be made. These may address, inter alia, its question-format presentation, its excessive number of items, its too much time-consuming demands, its lack of scoring scale, and its failure to provide well-defined criteria for appraising coursebooks specifically in terms of the way they handle lexis, grammar, pronunciation, and the four macro LSRW skills. Further shortcomings are to be highlighted at the end of the next Section which attempts to provide a practical application of the checklist under consideration to a BE textbook selected for that purpose.

3- ILLUSTRATIVE APPLICATION

3.1 FACTUAL DETAILS:

Title: English For Me Classbook 5BAuthor: English Language Curriculum DepartmentPublisher: Oman’s Ministry of Education Price: FreeISBN: None No. of Pages: 63Components: SB/TB/Cassettes/Stimulus visual posters/ Flash cardsLevel: Basic Education/ Elementary Physical size: A4Length: 1 Semester Units: 5 Lessons/sections: 15 Hours: 85Target skills: LSRWTarget learners: 10 to 11 year-oldsTarget teachers: Teachers trained in teaching English to young learners

3.2 ASSESSMENT:

ASSESSMENT (* Poor ** Fair *** Good **** Excellent)

Factor Rating and commentsRationale: ……………………………………………………… PoorAvailability: …………………………………………………… PoorUser definition: ………………………………………………… PoorLayout/graphics: ……………………………………………….. ExcellentAccessibility: …………………………………………………… PoorLinkage: ………………………………………………………… FairSelection/grading: ………………………………………………. PoorPhysical characteristics: ………………………………………… ExcellentAppropriacy: ……………………………………………………. GoodAuthenticity: …………………………………………………….. FairSufficiency: ……………………………………………………… PoorCultural bias: …………………………………………………….. Fair

4

Educational validity: …………………………………………….. GoodStimulus/practice/revision: ………………………………………. FairFlexibility: ……………………………………………………….. FairGuidance: ………………………………………………………… GoodOverall value for money: ………………………………………… Fair²

3.3 APPLICATION OUTCOMES:

As might obviously emerge from the previous sub-section, the Sheldon checklist, even in its unmodified state, proves powerful enough to determine the rather modest level of efficiency the illustrative textbook is at. For a learning/teaching aid which claims to “provide children with a firm foundation on which to build in later grades” (English Language Curriculum Department 2003-2004b: viii), the textbook actually seems to prove “Poor” to “Fair” thirteen times out of seventeen, “Good” against three evaluative rubrics, and “Excellent” with regard to two somewhat peripheral criteria. A thorough reworking (and not just mere adaptation) of it would be a matter of more obligation than necessity should the above ratings be awarded by a greater number of assessors.

What the checklist-based assessment fails to pinpoint, though, is an explicitly clear pronouncement about the illustrative classbook’s strengths and weaknesses— notably with regard to its approach and its effectiveness (or lack of it) to teaching the LSRW skills. The investigative questions in the given rubrics are rather of little assistance in this respect, which largely explains the “major caveat” pointed at in the introductory quotation above (Sheldon, Op.Cit: 241). Help might be best sought in Littlejohn’s 1998 alternative checklist (Tomlinson (ed) 1998), where an inside-the-Trojan-Horse-type evaluation is made possible4.

Notwithstanding this foible, the above-mentioned ratings do strongly suggest the state of imperfectness the chosen material is. For nowhere does it care to specify the rationale it was written for in the first place, the methodology it operates correspondingly, or the pedagogy it aims to promote. At no point, either, does it give the impression that it works according to a discernible system in the selection and grading of the reading and writing activities (the listening skill is comparatively well-catered for; the speaking one, curiously enough, is totally left out here and is relegated to ancillary materials). This quasi-total inefficiency, as revealed by the assessment analysis, could but result in yet lower levels of English proficiency in students (see subsections 1.1.2 and 1.1.3 above).

The next Section will consider some of the implications surrounding the checklist method for textbook evaluation as sampled by the Sheldon proposal.

4- IMPLICATIONS

A close look back at the preceding two Sections would undoubtedly reveal two basic aspects of the STEC and, indeed, other similarly “off-the-shelf” ones: their obvious merits and, simultaneously, potential limitations. The former, as McGrath (Op.Cit) states, can be summed up in the following points:

1. It is systematic, ensuring that all elements that are deemed to be important are considered.

5

2. It is cost effective, permitting a good deal of information to be recorded in a relatively short space of time.

3. The information is recorded in a convenient format, allowing for easy comparison between competing sets of material.

4. It is explicit, and provided the categories are well understood by all involved in the evaluation (…), offers a common framework for decision-making (26-7).

The limitations, on the other hand, are not less conspicuous. These, as could be discerned from the introductory excerpt, are particularly due to

a- the impossibility of ensuring a hundred per cent systematicity of the checklist criteria for each single ELT setting across the world. The terms “exhaustiveness” and “discreteness” which were set by Tucker (1978: 319; quoted in Skierso 1991: 440) as two indispensable parameters for materials evaluation, have remained unattainable to date.

b- the practically unavoidable subjectivity which may easily be sensed in the very “judgement [exercised by any checklist compiler] in selecting the parameters to ask about (and not ask about)…” (Roberts 1998: 123).

c- the high instability of the checklist categories which, as McGrath (Op.Cit: 27) observes, “are as much a reflection of the time at which they were conceived and of the beliefs of their designer as are published materials themselves”.

d- the numerous demands that checklists, more often than not, pose to evaluators, be the experienced or less trained, mainly because of the inappropriateness, opacity, and complexity of many of the criteria to the evaluative purpose (McGrath, ibid: 31; Sheldon, Op.Cit: 245; Roberts, Op.Cit: 123). The questions referring to the rubrics labelled “Appropriacy” and “Authenticity” in the Sheldon checklist are quite illustrative in this regard.

The immediate implication of such demerits is the plain realization that “the checklist method is not a watertight category [as compared to the in-depth one]” (McGrath, Op.Cit: 26). This, in its turn, prompts the need for other types of approach n view of meaningfully seeking out the real teaching/learning value of textbooks (Allwright 1981; O’Neill 1982). These include, à titre indicatif, team self-evaluation based on internally-designed checklists (Blue and Grundy 1996; Chambers 1997), classroom-based observation (Sheldon, Op.Cit: 245-6), and staff/student-driven feedback (Sharp 1990). Discussion of the techniques developed in each of these alternative proposals is, understandably enough, beyond the focus of the task requirements.

5- CONCLUSION

The present paper has considered the issue of the checklist approach to textbook analysis and evaluation via a sample scheme drawn up by Sheldon (1988). The theoretical assessment and subsequent practical application of the criteria in the said framework to an illustrative coursebook have demonstrated the twin merits and constraints that are associated with such a materials evaluation instrument. A set of related implications have also been highlighted, following the discussion about the numerous limitations inherent in the method under consideration. The whole issue will remain open to further debate.

6- END NOTES

6

1- In some other grades, a few students who come from the southern province of Dhofar speak Jabbali -- a variety of language totally different from those mentioned earlier. The issue of home languages is one of the several taboos here.2- See Cameron (2001) for an excellent discussion.3- A close look into Oman’s native population strikingly belies this cliché statement which is daily chanted in the National Anthem.4- See also Ellis’s contribution in the same edited book for a task-focused framework for textbook analysis and evaluation.

7- REFERENCES

Allwright, R. (1981), “What do we want teaching materials for?”, ELT Journal 36, 1: 5-18.

Blue, G. and P.Grundy (1996), “Team evaluation of language teaching and language courses”, ELT Journal 50, 3: 244-53.

Cameron, L. (2001), Teaching Languages to Young Learners, Cambridge: Cambridge University Press.

Celce-Murcia, M. (Ed), Teaching English as a Second or Foreign Language, 2nd Edn., Boston:Heinle and Heinle.

Chambers, F. (1997), “Seeking consensus in coursebook evaluation”, ELT Journal 51, 1: 29-35.

Ellis, R. (1998), “The evaluation of communicative tasks”. In B. Tomlinson (Ed), Materials Development in Language Teaching, Cambridge: Cambridge University Press, 217-38.

English Language Curriculum Department (2003-2004a), English For Me. Class Book 5B, Muscat: Ministry of Education.____________________________________ (2003-2004b), English For Me. Teacher’s Book 5B, Muscat: Ministry of Education.

Johnson, K. and H. Johnson (eds) (1998), Encyclopedic Dictionary of Applied Linguistics, Oxford: Blackwell.

Littlejohn, A. (1998), “The analysis of language teaching materials: inside the Trojan Horse”. In B. Tomlinson (Ed), Materials Development in Language Teaching, Cambridge: Cambridge University Press, 191-213..Madsen, H. and J. Bowen (1978), Adaptation in Language Teaching, Rowley, MA: Newbury House.

McDonough, J. and C. Shaw (2003), Materials and Methods in ELT. 2nd Edn., Malden, Ma., USA: Blackwell.

McGrath, I. (2002), Materials Evaluation and Design for Language Teaching, Edinburgh: Edinburgh University Press Ltd.

O’Neill, R. (1982), “Why use textbooks?”, ELT Journal 36, 2: 106-11.

7

Rivers, W. (1981), Teaching Foreign-Language Skills, Chicago: University of Chicago Press.

Roberts, J.T. (1998), “Evaluation of course books or materials evaluation”. In K. Johnson and H. Johnson (eds), Encyclopedic Dictionary of Applied Linguistics, Oxford: Blackwell, 119-25.

Sharp, A. (1990), “Staff/student participation in course evaluation: a procedure for improving course design”, ELT Journal 44, 2: 132-7.

Sheldon, L. (1988), “Evaluating ELT textbooks and materials”, ELT Journal 42, 4: 237-46.

Skierso, A. (1991), “Textbook selection and evaluation”. In M. Celce-Murcia (Ed), Teaching English as a Second or Foreign Language, 2nd Edn., Boston:Heinle and Heinle, 432-53.

Tomlinson, B. (Ed) (1998), Materials Development in Language Teaching, Cambridge: Cambridge University Press.

Tucker, C.A. (1975/1978), “Evaluating beginning textbooks”, English Teaching Forum 13, 3-4 (Special Issue PT 2), 356-6 [originally printed in 1968 in English Teaching Forum 6, 5: 8-15, and subsequently reprinted as Appendix 3 in H. Madsen and J. Bowen (1978), Adaptation in Language Teaching, Rowley, MA: Newbury House, 219-37].

Ur, P. (1996), A Course in Language Teaching: Practice and Theory, Cambridge: Cambridge University Press.

Williams, D. (1983), “Developing criteria for textbook evaluation”, ELT Journal 37, 3: 251-55.

8- APPENDIX

Textbook evaluation sheet by Sheldon (1988:243-5) Rationale - Why was the book written in the first place, and what gaps is it intended to fill?- Are you given information about the Needs Analysis or classroom piloting that were undertaken?- Are the objectives spelt out?

Availability - Is it easy to obtain sample copies and support material for inspection?- Can you contact the publisher’s representatives in case you want further information about the content, approach, or pedagogical detail of the book?

User definition - Is there a clear specification of the target age range, culture, assumed background, probable learning preferences, and educational expectations?- Are entry/exit language levels precisely defined, e.g. by reference to international ‘standards’ such as the ELTS, ACTFL or Council of Europe scales, or by reference to local or country-specific examination requirements?

8

- In the case of an ESP textbook, what degree of specialist knowledge is assumed (of both learners and teacher)?

Layout/graphics - Is there an optimum density and mix of text and graphical material on each page, or is the impression one of clutter?- Are the artwork and typefaces functional? colourful? appealing?

Accessibility - Is the material clearly organized?- Can the student find his or her location in the material at any point, i.e. is it possible to have a clear view of the ‘progress’ made, and how much still needs to be covered?- Are there indexes, vocabulary lists, section headings, and other methods of signposting the content that allow the student to use the material easily, especially for revision or self-study purposes?- Is the learner (as opposed to the teacher) given clear advice about how the book and its contents could be most effectively exploited?

Linkage - Do the units and exercises connect in terms of theme, situation, topic, pattern of skill development, or grammatical/lexical ‘progression’?- Is the nature of such connection made obvious, for example by placing input texts and supporting exercises in close proximity?- Does the textbook cohere both internally and externally (e.g. with other books in a series)?

Selection/grading - Does the introduction, practice, and recycling of new linguistic items seem to be shallow/steep enough for your students?- Is there a discernible system at work in the selection and grading of these items (e.g. on the basis of frequency counts, or on the basis of useful comparisons between the learner’s mother tongue and English)?- Is the linguistic inventory presented appropriate for your purposes, bearing in mind the L1 background(s) of your learners?

Physical characteristics- Is there space to write in the book?- Is the book robust? too large? too heavy?- Is the spine labelled?- Is it a book that could be used more than once, especially if it is marked by previous students?

Appropriacy

- Is the material substantial enough or interesting enough to hold the attention of learners?- Is it pitched at the right level of maturity and language, and (particularly in the case of ESP situations), at the right conceptual level?- Is it topical?

Authenticity

9

- Is the content obviously realistic, being taken from L1 material not initially intended for ELT purposes?- Do the tasks exploit language in a communicative or ‘real-world’ way?- If not, are the texts unacceptably simplified or artificial (for instance, in the use of whole-sentence dialogues)?

Sufficiency

- Is the book complete enough to stand on its own, or must the teacher produce a lot of ancillary bridging material to make it workable?- Can you teach the course using only the student’s book, or must all the attendant aids (e.g. cassettes) be deployed?

Cultural bias

- Are different and appropriate religious and social environments catered for, both in terms of the topics/situations presented and of those left out?- Are students’ expectations in regard to content, methodology, and format successfully accommodated?- If not, would the book be able to wean students away from their preconceived notions?- Is the author’s sense of humour or philosophy obvious or appropriate?- Does the coursebook enshrine stereotyped, inaccurate, condescending or offensive images of gender, race, social class, or nationality?- Are accurate or ‘sanitized’ views of the USA or Britain presented; are uncomfortable social realities (e.g. unemployment, poverty, family breakdowns, racism) left out?

Educational validity

- Does the textbook take account of, and seem to be in tune with, broader educational concerns (e.g. the nature and role of learning skills, concept development in younger learners, the function of ‘knowledge of the world’, the exploitation of sensitive issues, the value of metaphor as a powerful cognitive learning device)?

Stimulus/practice/ revision

- Is the course material interactive, and are there sufficient opportunities for the learner to use his or her English so that effective consolidation takes place?- Is the material likely to be retained/remembered by learners?- Is allowance made for revision, testing, and on-going evaluation/marking of exercises and activities, especially in large-group situations; are ready-made achievement tests provided for the coursebook, or is test development left for the hard-pressed teacher? Are ‘self-checks’ provided?

Flexibility

- Can the book accommodate the practical constraints with which you must deal, or are assumptions made about such things as the availability of audio-visual equipment, pictorial material, class size, and classroom geography; does the material make too many demands on teachers’ preparation time and students’ homework time?

10

- Can the material be exploited or modified as required by local circumstances, or is it too rigid in format, structure, and approach?- Is there a full range of supplementary aids available?

Guidance - Are the teacher’s notes useful and explicit?- Has there been an inordinate delay between the publication of the student’s and teacher’s books which has meant that teachers have had to fend for themselves in exploiting the material?- Is there advice about how to supplement the coursebook, or to present the lessons in different ways?- Is there enough/too much ‘hand-holding’?- Are typescripts, answer keys, ‘technical notes’ (in the case of ESP textbooks), vocabulary lists, structural/functional inventories, and lesson summaries provided in the Teacher’s Book?- Is allowance made for the perspectives, expectations, and preferences of non-native teachers of English?

Overall value for money - Quite simply, is the coursebook cost-effective, easy to use, and successful in your teaching situation, in terms of time, labour, and money?- To what extent has it realized its stated objectives?

11