CHAPTER I1

THEORETICAL BACKGROUND

Page No.

2.1 INTRODUCTION 37

2.2 THE COMMUNICATIVE APPROACH TO LANGUAGE TEACHING 37

2.3. EVALUATION OF ENGLISH LANGUAGE TEACHING

2.3.1. Introduction

2.3.2. Aims and Purposes of Evaluation

2.3.3. The Timing of Evaluation

2.3.4. Principles of Evaluation

2.3.5. Methodology of Evaluation

2.3.5.1. Introduction

2.3.5.2. Self-Report Methods : Interviews and Questionnaires

2.3.5.3. Data Analysis

2.3.6. General Considerations 55

CHAPTER ll

THEORETICAL BACKGROUND

2.1 INTRODUCTION

Planned innovations in language teaching began about a 100 years ago and have

been going on ever since. Research studies on teaching methods looked for new

emphases in curriculum design and human relations. The focus turned to be on

communication in the mid-seventies of the present century. Communication or

communicative competence is the key concept that has epitomized the practical, theoretical

and research preoccupations in educational linguistics and language pedagogy. The term

'communicative competence' first used by Hymes (1972) in deliberate contrast to

Chomsky's 'linguistic competence', reflects the social view of language. The various trends

of language teaching and the concept of communicative competence have merged in the

idea of communicative language teaching as central focus for new thought and fresh

approaches in language pedagogy.

2.2 THE COMMUNICATIVE APPROACH TO LANGUAGE TEACHING

Through concepts such as 'communicative' or 'functional' language teaching or

'communicative competence', theorists have attempted to bring into language teaching

insights which they have derived from speech act theory, discourse analysis and the

ethnography of communication. The sociolinguistic emphasis is expressed by contracting a

'communicative' or 'functional' approach with 'linguistic', 'grammatical', structural, or

'formal' approaches to language teaching.

Widdowson (1978) defined a set of contrasting concepts which distinguish between

language as a formal system and language use as communicative events. The point of

view that Widdowson advocated was that it is important for language teaching to make

these distinctions and that a shift of emphasis is needed from teaching a second language

as a formal system to teaching second language as communication. The distinctions

themselves can be regarded as contributions to linguistic theory. Examples of these

concepts are :-

Linguistic categories

Correctness

Usage

Significance

Sentence

Proposition

Cohesion

Linguistic skills

(eg.speaking, hearing)

Communicative categories

appropriacy

use

value

utterance

illocutionary act

coherence

communicative abilities

(eg.saying, listening, talking)

Widdowson was aware of the fact that the demands of language pedagogy in terms

of such distinctions may run ahead of linguistic theory. But he was of the opinion that

practical needs may stimulate development of new linguistic theory, in line with the

desirable reciprocal flow of ideas.

The main distinction between the formal or structural theories and the

communicative theory is seen in the fact that the former theories view language outside a

particular context of language use while communicative theory presents the second

language in a more clearly specified social context and situation. Advocates of structural

approach were not unmindful of situations of language use. But the situations were left

open and relatively undefined. Theorists talked about speaking and listening as skills in

general. Provided emphasis was laid on 'the primacy of speech' and opportunities for skill

practice existed, it was thought enough was done to make language teaching realistic and

relevant for potential language use.

By basing themselves on speech act theory and the analysis of discourse and

introducing perspectives of sociolinguistics generally, theorists since the last decade have

attempted to come closer to the reality of language use. Henceforth, uses of language

were to be specified in social settings much more precisely, in the expectation that

language pedagogy would thereby become more relevant to the declared or putative

needs of language learners. The theorists' energies have been directed to bringing these

sociolinguistic perspectives into language curriculum through new curriculum designs, and

through new materials, teaching techniques, and testing with a communicative orientation.

Several educational linguists and language teachers have been involved in efforts to give

concrete shape to this direction of language teaching.

The Council of Europe Modern Languages Project, was one of the main pioneering

endeavours in the respect of the design of a curriculum based on communicative principles.

The rationale of this project and others of a similar nature was that in order to determine

what language functions should include, one has to set out from the language needs of

language learners. The definition and identification of these language needs has

contributed a first and important stage in the procedures to make language teaching

communicative. The second stage has been the definition of language categories in

semantic and sociolinguistic terms accompanied by examples of language items. While

these procedures have stimulated a great deal of interest among practitioners, the gap

between the inventories of language items in a 'syllabus' and the teaching materials,

teaching techniques and testing procedures which carry these syllabi into effect has been

difficulttobridge. Teaching materials and techniques which are based on

sociolinguistic principles usually identify learners in a specific role of language use, for

example, as tourists, or university students, or migrant workers. Often the interactants are

specified : shop-assistant - customer: foreign traveller - policeman; physician - patient, and

so on. Situations of language use are indicated and sometimes described in a detailed

scenario : for example, visiting a city, arriving at a hotel; reading academic papers;

participating in seminar discussions, asking a neighbour for help; visiting a doctor's surgery.

Next, speech acts are analysed which regularly occur in the given situation : introducing

oneself, enquiring, gathering information, asking permission, asking for help, giving reasons

or explanations, and the like. Eventually, the linguistic manifestations of the speech act or

acts are presented in a text, a dialogue, a flow-chart, a table with explanations or an

excerpt from a newspaper, etc. Learners are usually invited to enter vicariously into the

situation so that they become participants. The learning tasks, therefore, frequently involve

problem solving, simulation and role-playing. There may be conventional drill-type

exercises, but the difference from structural practice lies in the fact that the linguistic forms

to be practised have an identifiable place in a sociolinguistic context which is presented to

learners as a concrete, practical situation in which they can feel at home and in which they

need the language items to be learnt. Ideally, the practice is never entirely repetitive or

imitative but offers natural options of language use which reproduce the kinds of choices

that occur in spontaneous communication.

Similar principles have been applied to testing. Although communicative testing as

an idea has appealed'to language teachers for several years, the construction of such tests

has proved troublesome. The aim is not to test only formal correctness, but also social

appropriateness in a given context. The test items usually define a situation and a problem

requiring a speech act. The item consists of the response the learner is expected to make.

It can be formulated as a multiple-choice or as an open-ended test item.

The sociolinguistic orientation has opened new perspectives in language teaching

which have only recently begun to influence language pedagogy. The consensus among

theorists and practitioners is that this sociolinguistic component complements and modifies

a 'structural' or 'grammatical' approach to language but does not supercede it. The problem

that has engaged the attention of several linguists is how to combine for teaching purposes

a structural and a sociolinguistic approach to language most effectively.

Some language teaching theorists have derived a different conclusion from the

sociolinguistic expansion of the view of language. They see in it further proof, in addition to

the evidence provided by structural linguistics and transformational generative grammar,

that language is too complicated to be taught by mainly analytical methods, structural or

sociolinguistic. Instead they recommend ways which systemize and supplement language

'acquisition' processes, that is, natural language learning without formal tuition. While these

conditions are best examined as psychological or methodological questions in pedagogy, it

can be pointed out that these views have been reinforced by a sociolinguistic interpretation

of language because sociolinguistics has placed language and language learning into a

social context of interaction, and non-analytical approaches to language learning are based

on the principle that the learner must become a participant in a real-life context of language

use as a condition of effective language learning

Attempts have been made to combine analytical and non-analytical approaches in a

multilingual curriculum as in the following scheme by Allen (1 980) :-

Fig.3.: An adaptation of Allen's three levels of communication competence in second

language education.

Level 1 Level 2 Level 3 Structural Functional Experiential

Focus on language

(formal features)

a) Structural control

Focus on language Focus on use of

(discourse features) language

a) Discourse control a) Situational or

topical control

b) Materials simplified b) Materials simplified b) Authentic language

structurally functionally

c) Mainly structural

practice

c) Mainly discourse c) Free practice

practice

This model expresses the view that a language curriculum must have a structural

level, as recognized in most conventional language programmes, but the structural

component by itself is insufficient. Discourse analysis and speech act theory provide the

basis for a second-level component of the curriculum. Both of these must, however,

become integrated at a third level where the language is used instrumentally in real-life

activities. According to this conception the language curriculum must have all three

components. Although the emphasis at different stages of the curriculum may shift from

level 1 to level 2 and then to level 3, in Allen's view the curriculum at all times should

include all three components. In other words, a curriculum should be based on a formal and

functional analysis and at the same time offer opportunities for experiential participation in

real-life communication which by its very nature is non-analytical.

Stern (1 983), recognising that a language curriculum must also have a sociocultural

aspect, has modified ~ l l e n j s scheme and suggested as a synthesis a fourfold curriculum

framework, as follows :-

Fig.4 : Sketch of a fourfold curriculum framework for a second language teaching

Mainly analytical mainly non-analytical

(involving language study and practice) (involving language

use in authentic

Language teaching thus approaches language learning objectively and analytically

through the study and practice of structural, functional, and sociocultural aspects.

2.3. EVALUATION OF ENGLISH LANGUAGE TEACHING

2.3.1. Introduction

Evaluation is the systematic collection and analysis of all relevant information

necessary to promote the improvement of the curriculum, and assess its effectiveness and

efficiency, as well as the participants' attitudes within a context of particular institutions

involved (Brown, 1989). The purpose of evaluation is to collect information systematically in

order to indicate the worth or merit of a programme or project (from certain aspects or as a

whole) and to inform decision making. The term 'programme' refers to any organized

educational activity offered on a continuing basis. The word 'programme' therefore

embraces both individual language programmes on training courses as well as a wider

range of English language teaching-related operations at one or more locations within a

single country.

The term 'project' refers to activities funded to achieve a particular task, usually

based on a formal contract in which staff duties are defined and measurable outcomes or

products are specified, all within a stated timescale.

Evaluation can be done for purposes of accountability and for purposes of

programme development. In general, accountability refers to the answerability of staff to

others for the quality of their work. 'Others' could be bureaucrats, employees, senior school

staff, parents, students and the community. Accountability can be contractual, where job

descriptions and planned outcomes are clearly specified in formal contracts. Professional

accountability abides where there may be an expectation that staff and administrators

should be answerable for their work as it affects others, for example in the use of

resources, in their professional practice, or in programme outcomes. Accountability oriented

evaluation is intended to assess the degree to which staff have met contractual or

professional accountability demands. It is usually summative in focus, in that it examines

the effects of a programme or project at significant end points of an educational cycle.

In comparison, development-oriented evaluation is intended to bring about

programme improvement and will normally be formative in nature; it 'regards the

programme as fluid and seeks ways to better it' (Cronbach 1982). 'Formative evaluators

... aim ... to ensure that the programme be implemented as effectively as possible. The

formative evaluation watches over the programme, alert both for problems and good ideas

that can be shared' (Morris and Fitz-Gibbon 1978).

Labelling evaluation data as formative and summative must relate closely to the

purpose for which it has been collected. Where data are used to evaluate effectiveness

against specified criteria, it is summative, and where they are used to influence change, it is

formative. Data can serve both formative and summative purposes in both developmental

and accountability oriented evaluations.

2.3.2. Aims and Purposes of Evaluation

A major reason for programme evaluation is that a wider audience can benefit from

the educational experience than those immediately involved Evaluations can provide

evidence which can inform theoretical disputes about directions to be followed in language

teaching or in teacher education and can also provide context-sensitive information on

implementation. For example, evaluations help to indicate whether particular approaches or

techniques are suitable under given conditions; whether certain textbooks or materials are

appropriate or inappropriate, effective or not for various contexts, purposes and groups of

learners.

Aims of evaluation, as represented in public documents - Aldensons 'Report on Visit

to Project' (November 1985) and Celani et al (1988) are eight fold :

- to attempt to carry out an evaluation which will be participatory and informative in nature

- to help teachers to become more aware and self-critical, to see themselves as agents -

not patients

- to help similar programmes do well (or better)

- to help understand the nature of innovation by monitoring developments and guiding

them by provision of feedback

- to show how the programme evaluates itself and to investigate the evaluation process

- to encourage authorities to take evaluation more seriously and to build it into the design

ab initio of projects

- to show supporting agencies what has been achieved

- to have defences in case of possible criticism. In general terms, it may be said that

evaluations are required for a variety of reasons, and one must never lose perspective

of why the evaluation is required.

- Purposes for evaluation may be listed as follows :-

- to discover whether the current approach should be continued and extended

- to identify the effects of the existing approach

- to inform decisions on the future of the programme

- to decide whether the programme has the intended effect

- to identify areas of improvement

- to show the positive achievements of teachers and pupils

- to motivate teachers.

2.3.3. The Timing of Evaluation

Systematic evaluation throughout a programme (formative evaluation) should be

integrated with summative evaluation. Systematic formative evaluation can operate as a

form of quality assurance, the monitoring of progress and the provision of immediately

useful information for decision making and change. It is used to help guide the programme

or project in the light of regular feedback on the way it is developing, through gathering

information which might enable corrections to be made. Evaluation conducted for this

purpose should be explicit and systematic and should be fed back to bureaucrats as well as

insider staff.

For evaluation to be comprehensive, data need to be generated which not only

provide a full account of what has taken place but which also contribute to an

understanding of the reasons behind the practices that affect success. Summative data

collected on the results obtained at the end of a programme (for example, student test

scores) are unlikely to help us understand the processes that gave rise to them. In this

case, summative data are of little use on their own in deciding how improvements might be

made.

Evaluation should usefully integrate formative and summative dimensions and be

concerned both with the results of a programme as well as an understanding of how these

results came about - that is with processes and activities during implementation as well as

with end products.

Formative evaluation during a programme (preferably throughout its life) can help

those involved to negotiate its direction with a deeper understanding of its internal and

external dynamics. In many ways this 'middle period' is the most important time to be

getting information, as it enables staff to take any steps necessary in terms of readjustment

and to justify their actions to the bureaucracy. Regular feedback sessions with students

and lesson proforma sheets filled out by teachers provide a valuable permanent record of a

course and allow for ongoing modification.

When evaluation is conducted only at the end of the programmes or projects, it

frequently means that crucial information for the evaluation is no longer available. If there is

no commitment to documentation during implementation, then any summative evaluation is

likely to be less informative or effective since so much data would be lost, for example,

what's been done in lessons, feedback on this from staff and students, comments on

materials and methodology, test data, continuous assessment data, time allotted,

attendance records, classroom observations, etc.

The durat~on of evaluator involvement is related to the timing of evaluations in the

life-cycle of a project or programme. If evaluation is to be valid and reliable, and to have

credibility with an outsider audience and with insider staff, it needs to cover a

representative period or periods. This means longer time-scales and meaningful episodes.

It also necessitates that those involved in the day-to-day running of the programme have a

prime responsibility for collecting the data that enable comprehensive evaluations.

2.3.4. Principles of Evaluation

Evaluation is always perceived as threatening the interests of those involved in the

object of the evaluation. For this reason, on the whole, evaluation tends to be neglected.

Only when external pressures or the need for external assistance dictate, do projects and

programmes tend to be evaluated, and such evaluation is then perhaps necessarily

perfunctory, superficial and inadequate. Yet, when it is finally carried out, evaluation has a

potentially powerful impact. An external evaluation centred on the basis of a visit by an

'expert' is not a very coherent way of evaluating language education. Instead, a

participatory model, centred on insides, perhaps benefiting from the advice of outsiders is

more welcome.

The principles of participatory evaluation may be outlined as :-

I) The study should be centred on needs.

2) It should aim at finding out student attitudes to objectives, course content, teacher's

competence. The classroom environment, and a global opinion.

3) The evaluation should provide opportunities for discussion and exchanges of opinion.

4) It should produce concrete recommendations.

5) It should be responsive to the variety of situations in a country.

6) The evaluation study should have a clear sense of its purpose.

A language evaluation should aim to avoid.

- exposing weakness in public (referring to individual teachers' or university teams'

weaknesses).

- Being externally oriented rather than internally useful.

- standardized questionnaires not capable of reflecting the real situation.

- excessively long procedures or instruments.

2.3.5. Methodology of Evaluation

2.3.5.1. Introduction

A data collection method should be chosen because it is the best means to tell us

what we want to know. Therefore, the first step is to determine exactly what it is what we

want to know : what the objectives of the evaluation are and what information will help

achieve these objectives. For example, in an evaluation with a narrow summative focus for

accountability purposes, methods generating qualitative evaluation data would be required

in order to match formal plans against measurable outcomes; or in a more comprehensive

summative evaluation, additional methods would be used to obtain process and

implementation data where the purpose is largely developmental, data collection can fulfill

more diverse functions : to document its actual development; to provide course records to

record unintended outcomes; to map the evolution of objectives; to identify the contextual

factors affecting implementation; to record the perceptions and reactions of teachers and

students to the programme; to monitor progress in language learning; to support and inform

materials development; to monitor classroom processes; and to show how far criteria of

quality and worthwhileness relevant to the programme are not being met.

In the past, there has been a tendency among some evaluators to adopt one

paradigm of enquiry to the exclusion of any other : some adopting naturalistic and

qualitative design and others relying wholly on experimental and qualitative models.

However, it is now a more widespread view that the purpose of the evaluation should

override such quasi-ideological preferences in favour of principles of utility and relevance

(Patton 1986). The position is greatly reinforced by a general preference for a broad,

inclusive approach to evaluation.

Methods should therefore be chosen according to the information required by

different evaluation purposes and also according to the realities of logistics (access and

resources) and the characteristics of informants.

In terms of logistics, methods are determined by such issues as the following:-

1) Access - who can be actually reached for data collection ? How long can informants

be expected to be involved in giving information (eg. being interviewed, filling in

questionnaires) ? Potential informants will all have their own commitments and demands on

their time, which evaluators will often have to work with rather than against. For example, to

see a busy teacher might require quite a careful advance arrangement.

2) Communications - how reliable and quick are communications ? For example, is

there a functioning telephone system by which to reach institutions ?

3) Resources - what time is there to get, analyze and present information ? What are

the cost limits on data collection ?

4) Access consequences - who is affected by the data collection methods used ?

Who is affected by the sampling decisions ? The choice of informants and the route by

which data is obtained will be affected by possible interpersonal consequences of data

collection. When considering access consequences, the advice of Sanders (1992) should

be well taken : 'One must always consider three aspects of good programme evaluation - communication, communication and communication. As long as you listen and respond,

share information, discuss your intentions and obtain feedback, clarify expectations,

provide clear and useful reports in a timely manner, and maintain an open evaluation

process, the evaluation seas will be smooth.

2.3.5.2. Self-Report Methods : Interviews and Questionnaires

Interviews and questionnaires are commonly referred to as self-report methods

because information is obtained at second hand through informants' accounts, rather than

by direct 'first-hand' description such as test-scores, documentary evidence or classroom

observation. Self-report data are "the personal responses of programme faculty, staff,

administration and participants. Self-reports typically take one of two forms : questionnaires

and interviews. Questionnaires asking about different individuals' experiences with a

programme enable the evaluator to collect information efficiently from a large number of

people. Individual or group interviews, are more time consuming, but provide face to face

descriptions and discussion of programme experiences." (King et al 1987).

Evaluators cannot provide a comprehensive account of a programme without the

accounts of insiders like learners, teachers, administrators, parents and education authority

officers, because they need to elicit insiders' experience of events to verify their

descriptions.

Self-report data, by definition, are 'indirect' in nature, as they either consist of a

description of events through the eyes of an intermediary, or represent the views and

perceptions of an individual, which cannot be directly accessed. In general, the less direct

data are, the harder it is to defend their credibility. On the other hand, in educational

settings direct evaluation data (documents, observations, test scores) though desirable, are

often unobtainable; furthermore, the perceptions of participants, however subjective, are

crucial means to understand programme implementation and effects and are only

obtainable by self-report methods.

Self-report data are unreliable if unsupported by other data because of 'post-event

reconstruction' by the informant, and the tendency of interviews or question wording to

affect responses. Post-event reconstruction is the tendency to create an account of reality

which is favourable to the evaluator, who has a vested interest in 'sounding good' when

reporting. The interviewer effect refers to the tendency of an interaction between

informants and interviewers to bias responses : there is a known tendency among

respondents to wish to provide what they think is wanted of them. Interviewers can

sometimes reinforce this by 'cueing' respondents when this is taking place. Low (1991)

drawing upon the conversational analysis literature, suggests that there might be a

reluctance to disagree in spoken interactions. Finally, there is a tendency for the nature of

question wording to influence answers, a fact fully recognized in market research.

In many cases, data collected through self-report can be objective and/or

quantifiable. However, in those cases where self-report data are supplied in response to

open-end questions, where they are not elicited in a structured fashion, or responses are

subjective and not open to quantification, there may be inherent limitations, particularly

regarding reliability and generalizability. Some evaluators might restrict the role of such

data to the verification of more quantifiable data, and only prefer it if logistics demand.

Some evaluators (Cronbach 1982, Parlett and Hamilton 1972, Patton 1987 and Stake

1986) might feel that qualitative and quantitative dimensions are relevant to the description

and evaluation of programmes. This is because of the need to 'triangulate data' - that is, to

confirm data from one source by cross-referring it to data from another (Cohen and Marion

1980). It is also because the forms of data can be complementary - quantitative findings

like classroom interaction patterns, test-scores, attendance figures, etc. can identify

objective trends and patterns; qualitative self-report data can help us to understand them

(Delamont 1976, Lynch 1992).

2.3.5.3. Data Analysis

There are no rigid rules to make data collection decisions beyond ensuring that the

methods chosen are appropriate to the situation and for the purposes for which information

is sought. The quantitative and qualitative dimensions of data need not be isolated from

each other, and can be complementary. Cronbach (1982) has observed that it is not

adequate or desirable to try to compress educational outcomes into a single dimension of

measurement. Conversely, it is a good practice in questionnaire or interview design to start

from open-ended elicitation to more focussed, quantifiable response categories.

Qualitative methods are often closely associated with naturalistic inductive designs

rather than by hypotheses (Patton 1987). They are normally explanatory, descriptive, and

discovery oriented in purpose. They try to describe complex events, attitudes and sets of

behaviour in depth and detail. They can take account of unforeseen or diverse reactions

4 an perspectives of stakeholders. They can provide information on how teaching and h

learning processes actually take place and what they mean to participants. The data may

take the form of verbatim descriptions, interviews, written responses, or unstructured

observations. Qualitative methods allow depth and flexibility, as in an unstructured

interview. However, qualitative data are more difficult to code and analyze than the

quantifiable data which result from structural questions. The categories of description are

inducted from raw data rather than imposed from 'outside' these data on the basis of a

predetermined and circumscribed set of categories. Thus no 'a priori' decision is taken as

to what is important. A recognized method for analyzing data obtained by open-ended

questions is to develop, with references to the data, a number of categories by which they

might be coded.

Quantitative methods normally rely on constraining people to respond in terms of

fixed response categories, as in the example of the questionnaire. Quantified data tells us

the frequency with which certain responses are ascribed to the sample under review and

allow us to determine whether these frequencies are reflected in subsamples within the

data set - that is, the extent to which people differ in respect of specific pre-determined

critical variables. Patton (1987) observed that :

'The advantage of quantitative approach is that it measures the reactions of a great

many people to a limited set of questions, thus facilitating comparison and statistical

aggregation of data. This gives a broad generalizable set of findings. By contrast,

qualitative methods typically produce a wealth of detailed data about a much smaller

number of people and cares.

Qualitative data provide depth and detail through direct quotation and careful

description of programme situations, events, people, interactions and observe behaviour.

The detailed descriptions, direct quotations and case documentation of qualitative methods

are collected as open-ended narrative without attempting to fit programme activities or

people's experiences into predetermined, standardized categories such as the response

choices that constitute typical questionnaires or tests.

2.3.6. General Considerations

There are a number of considerations that all evaluators used to address no matter

what methods are employed. These are :-

I) Plann in~ : Evaluators need to be aware of the advantages and disadvantages of

the various methods employed and be familiar with what constitutes good practice in the

use of these method. In many ways, planning is the most crucial stage, as once the

evaluation is implemented, it is difficult to make radical changes to the instruments being

used or the procedures being followed. Any changes made will cause problems of

comparison with data collected earlier.

2) Validity : In all methods, the cardinal principle is to establish clearly what one wants

to find out. Validity is concerned with measuring what one wants to measure. It is crucial to

be explicit about what is to be measured and to ensure that data collection procedures

provide the data for this purpose. Methods which best allow this to be done should be

selected.

3) Trianaulation : A combination of data sources is likely to be necessary in most

evaluations because often no one source can describe adequately such a diversity of

features as is found in educational settings, and because of the need for corroboration of

findings by using data from these different sources, collected by different methods.

4) Reliability : It is imperative for all evaluators, no matter what methods they employ,

to try and ensure that the data that are collected are reliable.

5) Practicality : It is imperative to choose a method that is efficient and practical. Often

time and resources are serious constraints. Data should only be collected if they are going

to be used, and only in a form which will enable best use to be made of them. It is better to

collect a limited quantity of high quality (informative and reliable) data rather than a larger

amount of low quality information that is possibly unusable. This is best ensured by

continual reference back to the decisions which these evaluation data are designed to

6) Sampling : For results to be credible we need to take a reasonable sample of data

relating to the evaluation focus under review. The more restricted the sampling, the more

the data is open to question in terms of both its validity and reliability. It is important that

stakeholders consider carefully what would be an adequate sample size for them to have

faith in the results that emerge. Increasing the sample size has obvious knock-on effects in

terms of practicality. However, if compromises are to be made, practical expediency should

not thereafter threaten the validity and reliability of the study.

7) Piloting : In all methods, the value of piloting instruments before actually employing

them in final data collection is paramount. The biggest single threat to the reliability and

validity of a study occurs when insufficient attention is paid to the design and piloting of

evaluation instruments. All too often, attention IS concentrated on the actual collection of

data and their analysis (Davies 1992). Sufficient time and attention must be allocated to

the refining of evaluation instruments. They at least need to be tried out first on colleagues

and then preferably on a small sample from the intended target group of informants. This

will help identify ambiguities, other problems in wording and inappropriate items, and

provide sample data to clarify any problems in the proposed methods of analysis prior to

the collection of data in the study proper.

8) Reporting : Davies (1992) makes the important point that 'An evaluation is not a

history but an abstraction .... An evaluation must be an interpretation.' Care must be taken

to ensure that the audience for reports have sufficient information to judge the reliability

and validity of the procedures followed, but they need to be able to separate the wood from

the trees. A surfeit of information should be avoided. Care should be taken in structuring

and presentation of evaluation reports. The 'transmissibility' of data is an important

consideration. The value of an evaluation is a function of its usefulness and accessibility to

immediate stakeholders and perhaps eventually to a wider audience.Jargon and

overloading with detail should be avoided. The evaluation should ideally be a coherent

account written in simple English.