assessment literacy in practice - core
TRANSCRIPT
Assessment literacy in practice
Anthony Green
University of Bedfordshire
Centre for English Language Learning and Assessment
• Need for assessment literacy – users, assessors,
constructors
• Do language teachers lack assessment literacy?
• What is assessment literacy for language
teachers? Is it the same as assessment literacy
for testers?
• Towards a shared framework for assessment
literacy
November 2013University of Bedfordshire 2
Assessment literacy in practice
Expanding definitions of literacy
1. Multiple literacies: different forms of
communication require different ways of
„reading‟ - media literacy, digital literacy,
statistical literacy. Awareness of divergent
attitudes and social identities: cultural literacy.
2. Critical literacy: not only recovering
informational content, but reflecting on,
challenging and questioning the purpose of the
information received.
University of Bedfordshire 3
The changing landscape of language assessment
• Growth of language testing and assessment worldwide.
• Globalised access to higher education, migration, employment.
• Learning outcomes and tests for accountability in education and training.
• Lifelong learning and „portability‟ of qualifications.
• National and international standards and frameworks.
University of Bedfordshire 4
Stakeholders
Language assessment use by…
• policymakers
• government agencies and bureaucrats
• educationalists, employers
• media and general public
• test takers
• teachers and teacher educators
How much/ what kind of assessment literacy do teachers
need?
Do language teachers really lack assessment literacy?
November 2013University of Bedfordshire 6
What a language proficiency test looks like and what it aims to
measure
• constructs that tests assess
• differences between testing instruments
Bodies responsible for overseeing testing procedures
How to find publicly available documentation on language tests
and what to look for in it
• how tests are administered and scored
• how cut scores are determined and who determines them
A lack of LAC from the invisible profession?
What do users need to know?(Pill and Harding 2013)
November 2013University of Bedfordshire 7
A „typical‟ teacher reaction to a Masters
course in Language Assessment.:
“Why wasn‟t there a full course on
assessment during my initial [teaching]
degree? I‟ve needed to know these
things for years now”
Cumming (2009)
Evidence for an assessment literacy deficit among language teachers
• Anecdotal evidence
Evidence for an assessment literacy deficit among language teachers
• Tests of assessment literacy
• Plake & Impara 1993, Newfields 2006, Kaftandijeva
2008 …
• Observation and surveys of teacher practices
• Rea Dickins 2001, Coniam 2009, Goto Butler 2009…
• Surveys of provision and needs
• Hasselgreen, Carlsen and Helness 2004, Brown and
Bailey 2007, Tsagari 2008, Fulcher 2011…
November 2013University of Bedfordshire 8
University of Bedfordshire 9
10. A test score at the 61st percentile indicates that 61% of the items
have been answered correctly.
11. It is difficult to obtain reliable scores from a group in which the
range of abilities is very wide.
12. A test score never equals a student‟s true level of ability or
achievement.
13. A valid assessment instrument can be used with any group of
students.
14. If test-retest reliability is .95, then if an examinee retakes the test,
there is a 95% chance to get the same test score.
15. Systematic error usually leads to bias in assessment.
True/False assessment literacy test(Kaftandijeva 2008)
Training for language teachers(Brown and Bailey 2008)
Topics covered in most graduate language assessment courses
(by LT specialist instructors):
• Test Critiquing and Test Analysis
• Item Writing for Different Skills, Item Content Analysis, Item
Quality Analysis, Item Facility, and Item Discrimination
• Construct Validity, and Criterion-related Validity
• Sources of Testing Error, Reliability and Test Length
Strategies for Estimating Test Reliability: Test-retest
Reliability, Parallel Forms Reliability, Internal Consistency
Reliability, Standard Error of Measurement, and Interrater and
Intrarater Reliability
November 2013University of Bedfordshire 10
Training for language teachers(Jeong 2013)
November 2013University of Bedfordshire 11
“Non-LT” instructor of a language testing course
suggested that „language assessment course textbooks
were targeted for large-scale assessment usage, so the
content covered in these books was not directly related to
language teachers‟ needs‟.
Significant differences in content of courses depending
on instructors‟ background:
+ LT: test specifications, test theory, basic statistics.
- LT: classroom assessment, rubric development, and
test accommodations.
„It is important for LTs to preserve their specialty, but also it
is essential to share the knowledge and make it accessible
to those who are part of the language assessment culture.
It is the role of the LT community to make the field
approachable to others‟
November 2013University of Bedfordshire 12
Training for language teachers(Jeong 2013)
University of Bedfordshire 13
„Since most teachers are not engaged in the
construction of formal tests, there are strong
arguments for placing the emphasis in
professional development. – at least initially – on
the role of assessment in the learning process
rather than on theoretical and statistical issues in
testing‟
Assessment literacy for teachers ≠ statistical literacy(Brindley 2001)
University of Bedfordshire 14
Too little attention in teacher training
Language education programs at graduate level
typically devote little time or attention to assessment
theory and practice, perhaps just a short (often
optional) module
Specialist training too narrow
… not simply to train well-qualified engineers and
technicians for quality test development, but to educate
and build a much broader and inclusive…
community… concerned with the impact of testing on
society and with fairness issues in the broadest sense.
Criticism of current training in language assessment (Taylor 2009)
National (USA) Survey of Teacher Assessment Literacy (Plake & Impara.1993)
The American Federation of Teachers, the National Council on
Measurement in Education, and the National Education
Association (1990) seven standards for teacher development in
the area of assessment:
1. Teachers should be skilled in choosing assessment methods
appropriate for instructional decisions.
2. Teachers should be skilled in developing assessment
methods appropriate for instructional decisions.
3. Teachers should be skilled in administering, scoring, and
interpreting the results of both externally produced and
teacher produced assessment methods.
November 2013University of Bedfordshire 15
4. Teachers should be skilled in using assessment results
when making decisions about individual students, planning
teaching, developing curriculum, and improving schools.
5. Teachers should be skilled in developing valid pupil grading
procedures which use pupil assessment.
6. Teachers should be skilled in communicating assessment
results to students, parents, other lay audiences, and other
educators.
7. Teachers should be skilled in recognizing unethical, illegal,
and otherwise inappropriate assessment methods and uses
of assessment information.
November 2013University of Bedfordshire 16
National (USA) Survey of Teacher Assessment Literacy (Plake & Impara.1993)
National (USA) Survey of Teacher Assessment Literacy (Plake & Impara.1993)
Item 3. Choosing Assessment Methods
Mrs. Bruce wished to assess her students' understanding of the
method of problem solving she had been teaching. Which
assessment strategy below would be most valid?
a) Select a textbook that has a "teacher's guide" with a test
developed by the authors.
b) Develop an assessment consistent with an outline of what
she has actually taught in the class.
c) Select a standard test that provides a score on problem
solving skills.
d) Select an instrument that measures students' attitudes about
problem solving strategies
November 2013University of Bedfordshire 17
National (USA) Survey of Teacher Assessment Literacy (Plake & Impara.1993)
Item 23. Using Assessments in Grading
When a parent asks a teacher to explain the basis for his or her
child's grade, the teacher should:
a) explain that the grades are assigned fairly, based on the
student's performance and other related factors.
b) ask the parents what they think should be the basis for the
child's grade.
c) explain exactly how the grade was determined and show the
parent samples of the student's work.
d) indicate that the grading scale is imposed by the school
board and the teachers have no control over grades.
November 2013University of Bedfordshire 18
November 2013University of Bedfordshire 19
National (USA) Survey of Teacher Assessment Literacy (Plake & Impara.1993)
Accurate assessment: Effectively used (Stiggins et al. 2004)
University of Bedfordshire 20
Key 1: Clear Purpose
• Who will use the results? What will they use the results for?
Key 2: Clear Targets
• What are the learning targets? Are they clear? Are they appropriate?
Key 3: Sound Design
• Method. Quality of items. Sampling of content. Avoidance of bias.
Key 4: Effective Communication
• Managing and reporting information. Who needs to know what?
Key 5: Student Involvement
• Students need to understand targets. Students can become effective
self and peer assessors
Evidence from observation and surveys of practice• Teachers often skilled in assessing students, but may be
marginalised - perceive need for more training in LTA
• A lot of class time (25% or more) may be devoted to
assessment
• Teacher-made assessments often of poor quality and not
subject to any appraisal or review
• Heavily reliant on printed sources, not always linked to
what is taught
• Not generally shared with other teachers
• Most often focussed on grammar and vocabulary
• Teacher criteria often idiosyncratic, may not reflect mandated
targets
• Feel pressured into mechanistic test preparation practices
November 2013University of Bedfordshire 21
November 2013University of Bedfordshire 22
Evidence from surveys of provision and needs(Hasselgreen, Carlsen and Helness 2004)
• Knowledge of theory
• Technical skills
• Principles and
concepts
• Language pedagogy
• Sociocultural values
• Local practices
• Personal beliefs/
attitudes
• Scores and decision
making
November 2013University of Bedfordshire 23
November 2013University of Bedfordshire 24
Teachers Testers
November 2013University of Bedfordshire 25
core
November 2013University of Bedfordshire 26
Assessment as learning
Assessment for learning
Assessment of learning
External proficiency testing
Literacy in classroom assessment(Hasselgreen.2008)
• Assessment as learning
Teachers need to provide means, training and
opportunities for self and peer assessment…
• Assessment for learning
Teachers need awareness of learning processes and
trajectories, sources of difficulty, strategies for
overcoming roadblocks, effective delivery of feedback…
• Assessment of learning
Teachers need skills in summarising learning, selecting
methods, task design…
November 2013University of Bedfordshire 27
November 2013University of Bedfordshire 28
The art of classroom-
based assessment
Lacking in reliability and validity
(Clapham 2000)
The science of
standardised testing
„dehumanizing and oppressive
pseudo-assessments‟
(van Lier 2004)
Mars
Venus
What do informal ‘in-flight’ in-class assessment, external proficiency tests and points between all have in common?
Asking informative questions
„Being literate in assessment… means having the
capacity to ask and answer critical questions about the
purpose for assessment, about the fitness of the tool
being used, about testing conditions, and about what is
going to happen on the basis of the results‟
Inbar-Lourie (2013)
November 2013University of Bedfordshire 30
Assessment literacy for language teachers
Green 2014Exploring Language Assessment and Testing
ProSET EU TEMPUS funded project for teacher training in Russian FederationProset-tempus.net
November 2013University of Bedfordshire 31
Assessment literacy for teachers and test developers:A common agenda for teachersand testing professionals?
Assessment qualities
The „point‟ of
language
assessment
The
necessary
groundwork
Effective assessment comes at a PRICE(Green 2013)
University of Bedfordshire 33
lanning and
eflection lead to
mprovement, when supported by
o-operation and informed by
vidence
Planning
University of Bedfordshire 34
Planning should always be built into the
development of any new project - and
assessments are no exception. The more
thorough the plans, the better the assessment is
likely to be.
Planning needed for P-R-V-BC
Key resources: time and training
Reflection
Reflection and review are fundamental to good
quality systems. The effectiveness of planned and
current materials and practices in fulfilling their
purpose should constantly be questioned and
evaluated.
How can P-R-V-BC be improved?
Key resources: assessment cycle
University of Bedfordshire 35
Improvement
Improvement on current materials and practices is
always possible and should be the constant aim of
any assessment system. Reflection is directed
towards making improvements.
Key resources: iterative, evolutionary approach to
development, systems for validation/ evaluation
November 2013University of Bedfordshire 36
Cooperation and teamwork are essential to the
development of good assessment materials and
practices. Shared ideas and shared criticisms aid
reflection and inform improvement.
Key resources: programmed opportunities for
cooperation between teachers, supportive
environment.
November 2013University of Bedfordshire 37
Cooperation
Evidence is the basis for good decision-making.
Reflection based on good evidence is much more
constructive than prejudiced guesswork as a
means of improving quality.
Key resource: record keeping
Effective record-keeping helps everyone involved to
understand where procedures and materials could be
made better.
Systematic analysis informs effective improvement.
November 2013University of Bedfordshire 38
Evidence