assessment in language learning classrooms:more questions than answers

Assessment in language learning classrooms:More questions than

answers

Robert J. DickeyKeimyung University, S. Korea

Defining Assesment…

?

Defining Assesment…• the wide variety of methods that educators use to evaluate, measure, and

document the academic readiness, learning progress, and skill acquisition of students

• the use of empirical data on student learning to refine programs and improve student learning

• the process of gathering and discussing information from multiple and diverse sources in order to develop a deep understanding of what students know, understand, and can do with their knowledge as a result of their educational experiences; the process culminates when assessment results are used to improve subsequent learning

• the systematic basis for making inferences about the learning and development of students --. the process of defining, selecting, designing, collecting, analyzing, interpreting, and using information to increase students’ learning and development

• the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development

Opening Quiz!Rank the following in determining student grades.__ performance (as indicated in formal tests, quizzes, etc)__ (oral) participation in class__ improvement over the course__ behavior in class__ effort__ motivation__ assignments__ punctuality and attendance__ intuitive, informal assessments of class performance (Most important = 1, Least/Not important = 9)

Who are we testing?

?Elicitation is both testing and lesson confirmation

Who are we testing?

• The Students

• The Teacher

What are we testing?

?

What are we testing?

• Overall English– competence? proficiency? skill?

• Mastery of course content• Needs of students• Improvements during the course• Teaching performance• Coursebook / Curriculum

Learning Objectives

Bloom’s Taxonomy

Learning Objectives

When do we test?

?

When do we test?

• Pre-test• Midterms• Finals• Quizzes• More Frequent > > >

– Elicitation– Homework/Assignments– “heat of the action”

Why do we test?

?

Why do we test?

• Required (from above)• Duty (teacher’s sense)• Tradition (always been testing)• Expectation (parents, others)• Feedback (students, others) (request)• Ease (following coursebook pattern)

How do we test?

?

How do we test?

• 4 skills• “forms” : vocabulary,

grammar, spelling…• fluency/accuracy• paper & pencil• multiple-choice• fill-the-blank• open-ended• interview

• group-tests• rubrics• standardized• textbook-source• commercial (generic)• curriculum-based• standards-based• teacher-created• computer-based

Types of tests

• Proficiency (regardless of training)• Achievement (final and progress)• Diagnostic• Formative / Pedagogic• Placement• Selection/Gate-keeping• Aptitude• Encouragement !!!

Oral & Written test forms

Types of tests

Traditional & Alternative• One-shot, standardized exams• Timed, multiple-choice format• Decontextualized test items

• Scores suffice for feedback

• Norm-referenced scores• Focus on the "right" answer• Summative• Oriented to product• Non-interactive performance• Fosters extrinsic motivation

• Continuous long-term assessment• Untimed, free-response format• Contextualized communicative

tasks• Individualized feedback and

washback• Criterion-referenced scores• Open-ended, creative answers• Formative• Oriented to process• Interactive performance• Fosters intrinsic motivation

Other Assessment Instruments

• Portfolio / Journal / Notebook• Projects• Systematic set of observations• Communicative tests through real-

world tasks• “personal chats” with instructor

How do we score?

?

How do we score?

• Normative• Relative• “pure” scores

• “Good Job!” / “good scolding”

“The Curve”

“The Curve”

How do we feedback scores?

?

How do we feedback scores?

• Public / Private• Personalized / Relative to the class• Commentary / Numbers• Constructive

Principles in Assessment

• Practicality• Reliability• Validity

Practicality in Assessment

• Expense• Within time constraints

– preparation– implementation– grading and review

• Scoring system is specific and time-efficient

Reliability in Assessment

• Consistent & Dependable• Students (a bad day?)• Rater (inter-rater, intra-rater)• Conditions (noise, heat, etc)• Is a timed-test more fair? To whom?• More than one possible answer?• Too easy/difficult (grade distinctions)

Validity in Assessment• Rampantly misunderstood !1.Face Validity2.Content Validity3.Criterion Validity (compare to other test)

(concurrent, predictive)

concurrent, predictive4.Construct Validity (test assumptions match

matter under investigation)(speaking test must include vocabulary?)

Face ValidityMostly Under-appreciated•“looks right” to ALL users•Familiar tasks•Familiar format•Clearly do-able within allowed time•Clear and uncomplicated items•Clear directions•Difficulty is “a reasonable challenge”

Authenticity

No one agrees exactly what this means•Language as natural as possible•Contextualized testing•Meaningful topics•Thematic organization of materials•Tasks approximate “real-world” tasks

Washback

• Pedagogic (formative) testing INTENDS a positive washback

• Information flows to students and/or teacher

• Affects future studies:– Lesson plan designs– Student motivation and focus

Choices we make…

Aims of Assessment

• Assessment for Learning

• Assessment as Learning

• Assessment of Learning

Assessment for Learning• to give teachers information• to modify the learning activities• to differentiate & focus how individual students

approach their learning• emphasis is on teachers using the information

from carefully-designed assessments• to determine what students know• to gain insights on how, when, & whether

students use what they know• so instruction can be targeted

Assessment as Learning• a process of developing & supporting

metacognition for students• focuses on the role of the student as the critical

connector between assessment and learning– Students, as active, engaged and critical assessors

make sense of information, relate it to prior knowledge, and use it for new learning

• the vehicle for helping students develop, practice and become comfortable with reflection and with critical analysis of their own learning

Assessment of Learning• to confirm what students know• to demonstrate whether or not the students have

met the standards and/or show how they are placed in relation to others

• Focus on creating accurate and sound statements of proficiency or competence for students, so that the recipients of the information can use the information to make reasonable and defensible decisions

Integrating Assessment

A quick survey

• Instructors for English Conversation courses

• Universities in S. Korea• June 2015• 58 respondents• Online collection (solicited through

Facebook groups)

Included in final course grade• attendance 94.83%• participation 91.38%• final examination 91.38%• midterm examination 86.21%• assignments 53.45%• presentation (oral) 41.38%• quiz 36.21%• report (written) 12.07%• notebook 5.17%• Other includes

-- online assignments -- TOEIC test-- transcription of a conversation -- video assignment-- writing lab

Included in final examination• Oral-based open response 61.11%• paper-based multiple-choice 40.74%• paper-based vocabulary assessment 40.74%• paper-based fill-in the blank/complete the sentence 37.04%• paper-based grammar assessment 35.19%• oral-based vocabulary assessment 33.33%• paper-based open response 31.48%• oral-based grammar assessment 29.63%• listening (accuracy) 29.63%• pronunciation 24.07%• oral-based taught phrases 24.07%• oral-based "content" (non-linguistic materials) 24.07%• paper-based taught phrases 14.81%• reading (accuracy) 11.11%• paper-based "content" (non-linguistic materials) 9.26%• oral-based fill-in the blank/complete the sentence 3.70%• other

Questions?

Rob Dickeyrjdickey@content-english.org

File available at http://tinyurl.com/q7pm5vt