session 2 principles of language assessment shared

8/15/2019 Session 2 Principles of Language Assessment Shared

1/31

PRINCIPLES OF LANGUAGE ASSESSMENT

1/21/2016

Nguyen T. T. NganSpring semester, 2015-2016


2/31

By the end of the lesson, SS should be able to define theprinciples in assessing of a foreign/second language.

1. practicality2. reliability

Student-related reliabilityRater reliability (Inter-rater & Intra-rater reliability)Test administration reliabilityTest reliability

3. validityContent-related evidenceCriterion-related evidenceConstruct-related evidenceConsequential-validityFace validity

4. authenticity5. washback

1/21/2016 Nguyen T.T. Ngan


3/31

EXERCISE 6, p.38 Group workRead the materials and decide whether the scenariosfulfill the following assessment principle.

Group 1: PracticalityGroup 2: Rater ReliabilityGroup 3: Test Reliability

Group 4: Content ValidityGroup 5: Face ValidityGroup 6: AuthenticityGroup 7: Washback


1. Be prepared toexplain yourposition.

2. Briefly explain what is the principleand how to achieveit in assessment.


4/31


5/31

1. PRACTICALITYPractical:

Not excessively expensive

Within appropriate time constraintsEasy to administerHas specific & time-efficient scoring/ evaluationprocedure



6/31

How to obtain practicality? Are administrative details clearly established before thetest?Can students complete the test reasonably within the settime frame?Can the test be administered smoothly, without procedural‘glitches’? Are all materials and equipment ready?

Is the cost of the test within budgeted limits?Is the scoring/evaluation system feasible in the teacher’stime frame? Are methods for reporting results determined in advance?



7/31

Consider the following tests1) Test-takers are temporarily ill or too anxious.2) Two scorers yield inconsistent scores of the same

test (e.g. for different scoring standards)3) Same scorer is easier on some tests but harder onother tests (e.g. unclear scoring criteria,inexperience, inattention, fatigue, biases).

4) Bad quality of audio tape in a listening test; badquality of the test paper; too stuffy test room

5) Poorly written test items (e.g. ambiguous items thathave more than one correct answer)



8/31

2. RELIABILITY

Reliable

DependableConsistent



9/31

2. RELIABILITY

2.1 Student-related reliability Test-takers are temporarily ill or too anxious.

Advice to test-takers to be prepared in good physical orpsychological factors



10/31

2. RELIABILITY

2.2 Rater reliability

Avoid human error, subjectivity, or bias during the

scoring process

Inter-rater reliability Intra-rater reliability



11/31

Inter-rater reliabilityTwo scorers yield inconsistent scores of the same test(e.g. for different scoring standards)



12/31

Intra-rater reliability?Same scorer is easier on some tests but harder on other tests(e.g. unclear scoring criteria, inexperience, inattention,fatigue, biases).

Use consistent sets of criteria for a correct responseGive uniform attention to those sets throughout theevaluation timeRead through test at least twice to check for your

consistencyIf ‘mid-stream’ modifications are made, go back and applythe same standards to allRead the tests in several sittings to avoid fatigue



13/31

2. RELIABILITY

2.3 Test administration reliabilityBad quality of audio tape in a listening test; bad qualityof the test paper; too stuffy test room

Conditions in which the test is administeredNoiseLight

TemperatureDesks & chairs…



14/31

2. Administration reliability?Every student has a cleanly photocopied test sheetSound amplification is clearly audible to everyone in

the room Video input is equally visible to allLighting, temperature, extraneous noise, and otherclassroom conditions are equal for all ss

Objective scoring procedures leave little debate aboutcorrectness of an answer



15/31

2. RELIABILITY

2.4 Test reliabilityPoorly written test items (e.g. ambiguous items that havemore than one correct answer)

The quality of the test itself

Examples of low test reliability:

Too long tests (which may cause test-takers to becomefatigued)Poorly written test items



16/31

2. RELIABILITYStudent-relatedreliability

Rater reliability Testadministrationreliability

Test reliability

Physical orpsychologicalfactors

Avoidance ofraters’ errors,subjectivity, biasduring the scoringprocess

Inter-rater reliability Intra-rater reliability

Conditions in which the test isadministered

How the test is written



17/31

Consider the following tests1) Pen-and-paper test, no oral production tested when

trying to assess learners’ ability to speak 2)

The test covers 2 of the 10 objectives of the course.3) An oral interview that evaluates pronunciation &grammar; while the rater believes other factors (e.g. vocab use, fluency) should be included in the

evaluation process.4) After the test, test- takers claim that they didn’tperform well on the test because they were notaccustomed to these formats.



18/31

3. VALIDITY Validity is the extent to which inferences

made from assessment results areappropriate, meaningful and useful interms of the purpose of the assessment



19/31

3. VALIDITY

3.1 Content validity1) Pen-and-paper test, no oral production tested when

trying to assess learners’ ability to speak

Content validity of a test is achieved when it requiresthe test taker to perform the behavior that is beingmeasured (i.e. direct testing).



20/31

3. VALIDITY

3.2 Criterion-related Evidence2) The test covers 2 of the 10 objectives of the course.

Test which criteria/ objectives test-takers are expected toachieve


CONCURRENT VALIDITY PREDICTIVE VALIDITY

Results are supported by otherconcurrent performance beyond theassessment itself.

e.g. high score proved by actualproficiency

Results predict a test- taker’slikelihood of future success

e.g. placement test, languageaptitude test


21/31

3. VALIDITY

3.3 Construct validity3) An oral interview that evaluates pronunciation &

grammar; while the rater believes other factors (e.g. vocab use, pron, fluency) should be included in theevaluation process.

A construct is any theory, hypothesis, or model that

attempts to explain observed phenomena.



22/31

3. VALIDITY

3.4 Consequential validityIts effectOn learners (motivation, subsequent performance in acourse, independent learning, study habits, and attitude

toward school work)On the social consequences of a test’ interpretationand use



23/31

3. VALIDITY

3.5 Face validity4) After the test, test- takers claim that they didn’tperform well on the test because they were notaccustomed to these formats.

SS view the assessment as fair, relevant, and useful forimproving learning



24/31


25/31

3. VALIDITY1. Content validity2. Criterion-related reliability

3. Construct validity4. Consequential validity5. Face validity



26/31

4. AUTHENTICITYTest tasks are likely to be enacted in the ‘real world’.



27/31

How to achieve authenticityDo tasks represent real-world tasks?Is the language in the test as natural as possible?

Are items as contextualized as possible rather thanisolated? Are topics and situations interesting, enjoyable, and /or humorous?

Is some thematic organization provided, such asthrough a story line or episode?



28/31

5. WASHBACK‘Effects of testing on teaching and learning’ - Praises for strengths and constructive criticism of

weaknesses- Specification of the numerical scores on the various

subsections of the test

“SS need to have a chance to feed back on your feedback,- to seek clarification of any issues that are fuzzy,- and to set new & appropriate future goals for

themselves.”



29/31

How to achieve washbackReviewing the content after the test for ss to discovertheir areas of strengths and weaknessesSelf-assessment



30/31

PRINCIPLES APPLIED TO THE EVALUATION

OF CLASSROOM TESTS1. Are the test procedures practical ?2. Is the test reliable ?

3. Does the procedure demonstrate content validity ?4. Are the test-taker, test, test administration, &evaluation reliable with no bias?

5. Are the tasks as authentic as possible?6. Does the test offer beneficial washback to the

learner?



31/31

EXERCISE 6, p.38

1/21/2016 N T T N

session 2 principles of language assessment shared

Documents