wiseman facets of l2 writing ability
DESCRIPTION
Wiseman Facets of L2 Writing AiliTRANSCRIPT
Facets of Testing L2 Writing Ability
Pre-Convention Institute
TESOL Conference Philadelphia 2012
Cynthia S. Wiseman, Ed.D.Borough of Manhattan Community College
City University of New York
Agenda
Introductions General features of language assessment &
L2 writing assessment Activity Language Use Argument & L2 Writing Defining the construct: L2 Writing Framework of Task Characteristics Examination of items Review: crossword
EXAMS!!!
Homework
Classroom observation
Participation
Quiz
Self-Assessment
Portfolios
Explicit•T & Ss are aware of assessment•Clearly distinct from Teaching - Exam
Implicit•T & Ss may be unaware -- organic•Continuous – graded, ungraded
free, pre-writing•Instantaneous – in-class written free wr•Cyclical – reflective practice
Assessment is…
Purpose of L2 writing assessment: to collect information about teaching & learning of L2 writing to make decisions
Teacher & Student:–Formative decisions
• To correct S errors or not – based on student draft
• To change question of inquiry – Revise prompt? description of assignment?
• To model a structure – based on student writing, do we need to model a paragraph in class?
• To go to next lesson or review?
• To go more in-depth in content area – based on thin development, do we need to do more research, discussion, reading?
• Use a different strategy to respond to write an essay or explain a genre
–Summative decisions• To place, pass, fail or promote a student
Decisions about…. Individuals
– Selection for admission/employment – screening exams, like CATW
– Placement into course of study – Department exams, multiple measures
– Certification for profession – LAST (NY Teacher Certification Exam)
– Prediction of future performance – TWE, DIALANG
Program– Formative, to make changes to improve program
– Summative, to continue existing program
Research– To decide on new research questions or methodology
– To change/modify view/understanding of language phenomenon
Uses of Language Assessments
Intended use: – To collect information for making decisions– Beneficial consequences for stakeholders
• E.g., ESL writing teacher is teaching lesson on cohesive devices in essay writing
– To make decision about instruction
– To change/improve instruction so Ss will effectively use cohesive devices to improve writing
– Short fill-in-the-blank paragraph to get feedback on Ss’ learning & effectiveness of teaching
Introductory ActivityThink of an L2 writing assessment
development situation that you are familiar with. Describe the context, participants, and the test development process. What decisions were to be made based on the results of this assessment?
What were some of the intended beneficial consequences of assessment use in this situation?
Did any problems came up as a result of this assessment development?
Steps in creation of an assessment
Assessment Use Argument (AUA) Set of claims:
– Conceptual links between TTs performance on assessment and interpretation about the ability
– Decisions to be made– Consequences
– What would the AUA of the L2 writing assignment that you just described look like?
Assessment Use Argument --Bachman & Palmer (2010)
Assessment Development Use Bachman & Palmer (2010)
Initial Planning in test development What beneficial consequences do we want to happen? Who
are the stakeholders (i.e., intended TTs, etc.)? Who will be directly affected by the use of the assessment? How?
What are the specific decisions that need to be made to reach the intended consequences?
What do we need to know about the ELL’s language ability as demonstrated in writing to make the intended decision?
What sources could we use to make that decision? Is an existing assessment available? Is it appropriate?
Do assessment tasks correspond to TLU tasks? Does the developer provide evidence justifying intended uses? Do we really need to develop our own assessment?
Case scenario Educational Context: CC, US, diverse urban population,
85% ELL/bilingual background, open admissions Decision: To place S in ENG101, 3-credit composition
class? Beneficial consequences? What do we need to know about the ELL’s language ability
as demonstrated in writing to make the intended decision? What sources could we use to make that decision? Is an
existing assessment available? Is it appropriate?– ACT Compass (Writing Sample Test)
Do assessment tasks correspond to TLU tasks? Does the developer provide evidence justifying intended
uses? Do we really need to develop our own assessment?
Steps in Creating an Assessment Identify the target population/test context Identify the type of assessment Specify the specific purpose of the test Define the construct: Describe the Target Language Use
Domain & Target Language Tasks Write specifications for the test Write items/tasks that operationalize the construct &
incorporate task characteristics that correspond to TLU tasks
Create the test of items/tasks with clear instructions Create an answer key/rubric
Test Development ModelBachman & Palmer, 1996
What is L2 writing ability?
Communicative Language Ability --Bachman (1990)
Language characteristics• Organizational Characteristics
• Grammatical (vocabulary, syntax, phonology, graphology)
• Textual (cohesion, rhetorical, conversational organization)
• Pragmatic characteristics• Functional (ideational, manipulative, heuristic, imaginative)
• Sociolinguistic (genres, dialect/variety, register, naturalness, cultural references and figurative language)
• Topical characteristics
• Metacognitive Strategies/Competence
Target Language Use (TLU) Domain “…a set of specific language use tasks that
the TT is likely to encounter outside of the test itself, and to which we want our inferences about language ability to generalize.” (p. 44)– Distinguishing characteristics of language use
tasks to describe language use domain– Inferences that generalize to specific domains in
which TT is likely to need to use the language– Inferences about TT’s ability to use language in
a target language use domain
Language Use Task
Language use task: an activity that involves individuals in using language for the purpose of achieving a particular goal or objective in a particular situation
– Specific situations– Goal-oriented– Active participation of language users
TLU Domain Language Use Settings Language Use Tasks
English for Business Communication
Managing & operating an office
•Writing memos•Preparing reports•Taking phone msgs•Writing letters•Writing emails•Texting
Negotiating with clients & customers
•Writing proposals•Responding to written offers •Writing emails•Texting
Promoting products or services
• Writing advertising copy
• Writing solicitation pitch
Characteristics of Tasks Link between tasks in the domain of test tasks
and the domain of non-test tasks – Selection or design of tests that correspond in
specific ways to language use tasks Extent and ways TTs’ language ability is
engaged Degree of correspondence between
characteristics of given test task and a particular language use task: authenticity, validity of inferences, domain to which inferences generalize
Control of characteristics of the test task through test design and development
Language Task Characteristics
Characteristics of the setting •Physical Characteristics•Participants•Time of task
Characteristics of the test rubrics •Instructions•Structure•Time allotment•Scoring method
Characteristics of the input •Format•Language of input
Characteristics of the expected response
•Format •Language of expected response
Relationship between input and response
•Reactivity•Scope of relationship•Directness of relationship
Framework for Language Task characteristics
Characteristics of the setting
•Physical characteristics•Participants•Time of task
Characteristics of the test rubrics
•Instructions •Language (L1, L2)•Channel (aural,visual)•Specification of procedures and tasks
Structure •# of parts/tasks•Salience of parts/tasks•Sequence of parts/tasks•Relative importance of parts/tasks•# of tasks/items per part
Time allotment
Scoring method •Criteria for correctness•Procedures of scoring the response•Explicitness of criteria and procedures
Framework for Language Task characteristics
Characteristics of the Input
Format •Channel (aural, visual)•Form (language, non-language, both)•Language (native, target, both)•Length•Type (item, prompt)•Degree of speededness•Vehicle (live, reproduced, both)
Language of input
Language characteristics•Organizational Characteristics
•Grammatical (vocabulary, syntax, phonology, graphology)•Textual (cohesion, rhetorical, conversational organization)
•Pragmatic characteristics•Functional (ideational, manipulative, heuristic, imaginative)•Sociolinguistic (dialect/variety, register, naturalness, cultural references and figurative language)
•Topical characteristics
Framework for Language Task characteristics
Characteristics of the Expected Response
Format •Channel (aural, visual)•Form (language, non-language, both)•Language (native, target, both)•Length•Type (item, prompt)•Degree of speededness•Vehicle (live, reproduced, both)
Language of input
Language characteristics•Organizational Characteristics
•Grammatical (vocabulary, syntax, phonology, graphology)•Textual (cohesion, rhetorical, conversational organization)
•Pragmatic characteristics•Functional (ideational, manipulative, heuristic, imaginative)•Sociolinguistic (dialect/variety, register, naturalness, cultural references and figurative language)
•Topical characteristics
Framework for Language Task characteristics
Relationship between Input & Response
Reactivity •Reciprocal•Non-reciprocal•adaptive
Scope of relationship
•Broad•Narrow
Directness of relationship
•Direct •Indirect
Language Use Tests comprised of Tasks Language test: a procedure for eliciting
instances of language use from which inferences can be made about an individual’s language ability– Language test should consist of language use
tasks– The elemental activities and situations of
language use– Performance of a set of interrelated language
use tasks Framework of task characteristics
Test Items
Is this writing task similar to tasks that the 2nd language learner would have to do in real life?
What aspect of second language writing ability is the item testing?
Does this task require the TT to demonstrate that aspect of L2 writing ability?
Task Characteristics TLU Task Test Task
Characteristics of the Setting Physical characteristics Participants Time of task
Characteristics of the test rubrics Instructions Structure Time allotment Scoring method
Characteristics of the input Format Language of input
Characteristics of the expected response Format Language of expected response
Relationship between input and expected response Reactivity Scope of relationship Directness of relationship
EAP ESL Low Intermediate L2 writing class, 20-25 students/class, 40-minute class, 4x/week, college preparation program w/ 7 levels, level & program exit exam: basic skills writing
proficiency test: argumentative essay.
Common Core State Standards For ELA & Literacy Council of Chief State School Officers
(CCSSO) & National governors Association (NGA)– Aligned with college and work expectations
– Standard was included based on the best available evidence that its mastery was essential for college and career readiness in 21st c, globally competitive society
Language use domain: literate in a global 21st century world Close attentive reading to understand and enjoy complex
works of literature Critical reading for important points Able to handle large amounts of information Actively seek wide, deep, thoughtful engagement with
high-quality literary and information texts that builds knowledge, enlarges experience & broadens worldviews
Reflexively demos cogent reasoning and use of evidence essential to private deliberation and responsible citizenship in democracy
Writing: K-5: College & Career Readiness
anchor Standards text types and Purposes* Write arguments to support claims in an analysis of
substantive topics or texts, using valid reasoning and relevant and sufficient evidence.
Write informative/explanatory texts to examine and convey complex ideas and information clearly and accurately through the effective selection, organization, and analysis of content.
Write narratives to develop real or imagined experiences or events using effective technique, well-chosen details, and well-structured event sequences.
Anchor Standards for Writing 6-12
Text types and Purposes Write arguments to support claims in an analysis of substantive topics or texts, using valid reasoning and relevant and sufficient evidence. Write informative/explanatory texts to examine and convey complex ideas and information clearly and accurately through the effective selection, organization, and analysis of content. Write narratives to develop real or imagined experiences or events using effective technique, well-chosen details, and well-structured event sequences.
College & Career Readiness Anchor Standards for LanguageConventions of Standard English Demonstrate command of the conventions of standard English grammar and usage when writing or speaking. Demonstrate command of the conventions of standard English capitalization, punctuation, and spelling when writing.
Production & Distribtion of Writing Produce clear and coherent writing in which
the development, organization, and style are appropriate to task, purpose, and audience.
Develop and strengthen writing as needed by planning, revising, editing, rewriting, or trying a new approach.
Use technology, including the Internet, to produce and publish writing and to interact and collaborate with others.
Agenda
Rubrics: facets of L2 writing ability Comparison of rubrics measure L2 writing
ability Design of writing task Wrap-up & evaluation
Scoring
Answer key – objective scoring Rubric - subjective scoring
– Construct– Rubric– Partial credit
L2 Writing ability
Target Language Use Domain: Academic writing in a community college: Language Use Task: write narrative/persuasive essays
Control of content developmentRhetorical controlGrammatical controlControl of register & vocabularyTask fulfillment (McNamara, 1996)
This exceptionally executed essay takes a clear position and
exceptionally succeeds in expressing a point of view or telling a story. The thorough development of ideas includes at
least two outstanding points directly related to the topic, and the examples used, particularly those from personal experience, are rich, e.g., occasional citation of statistics or reference to personal readings.
The essay is clearly and logically organized with no digressions; the writer demonstrates skillful command of cohesive devices. Writer demonstrates ability to write in the
appropriate academic register and demonstrates extensive range of vocabulary for academic purposes, with few
problems in word choice or usage. A few grammatical errors are noticeable but rarely do the grammar errors interfere with meaning.
Sentence variety and complexity reflect a sufficient command of standard written English to ensure reasonable clarity of expression.
Holistic Scoring Scale: Criteria for Grading ESL Papers 6This exceptionally executed essay takes a clear position and exceptionally succeeds in expressing a point of view or telling a story. The thorough development of ideas includes at least two outstanding points directly related to the topic, and the examples used, particularly those from personal experience, are rich, e.g., occasional citation of statistics or reference to personal readings. The essay is clearly and logically organized with no digressions; the writer demonstrates skillful command of cohesive devices. Writer demonstrates ability to write in the appropriate academic register and demonstrates extensive range of vocabulary for academic purposes, with few problems in word choice or usage. A few grammatical errors are noticeable but rarely do the grammar errors interfere with meaning. Sentence variety and complexity reflect a sufficient command of standard written English to ensure reasonable clarity of expression. 5The focus of this competently executed essay is clear but there may be a few digressions. The writer provides substantial support in the development of the essay although all examples may not be entirely relevant or appropriate for the topic. The essay is effectively organized, demonstrating systematically competent use of cohesive devices. The writer demonstrates ability to use a variety of patterns of sentence construction but with some errors. Range of vocabulary for academic purposes is generally competent, and the writer demonstrates accurate and generally appropriate control of word choice, word forms and idiomatic expressions for academic writing. Some errors in language use, but errors do not generally interfere with meaning. 4In this adequately executed essay the writer’s position is clear despite some possible digressions and contradictions. The writer provides adequately detailed support of two or more points that directly relate to the topic. The essay is generally organized, demonstrating generally accurate and appropriate use of cohesive devices. The writer demonstrates some sentence variety with simple, compound, and some complex sentences though not always correctly. The essay may contain frequent errors that may occasionally interfere with meaning. Vocabulary is adequate in range, but there are some inappropriate or inaccurate word choices and word forms. 3The essay minimally succeeds in taking a position or relating a narrative with a discernable organizational pattern (introduction, body, conclusion) but may lack clear focus in development of the central idea. The writer makes an attempt at development although examples are sometimes irrelevant. The writer makes minimal use of cohesive devices and he/she demonstrates a minimal range of sentence variety and vocabulary, with some inaccurate and/or inappropriate word choices or inappropriate register. The essay demonstrates minimal control of language, with frequent errors, some of which interfere with meaning. . 2The paper represents limited success in writing a persuasive or narrative essay. The writer provides limited development of the topic with one or more points that directly or indirectly relate to the supporting argument or story. The writing shows limited evidence of organization of ideas (paragraphs are often one sentence) or accurate or appropriate use of cohesive devices. The range of vocabulary and word choice and the use of academic register is limited. The control of language is uneven, with frequent errors, many of which obscure meaning. The writing lacks sentence variety. 1The paper is a failed attempt to write an essay. The writer does not fully develop the topic, lacking related support. There is often no clear organizational pattern, lacking a clear beginning, middle and end. The writer does not use cohesive devices. The writer demonstrates a narrow range of vocabulary. There is little evidence of appropriate word choice or usage or academic register. The writer demonstrates little control, with frequent errors of all types. The errors generally obscure meaning. The writing lacks basic sentence structure and variety. In some cases, the paper may even be written in the writer’s first language.
Template for Holistic Rubrics
Score Description
5 Demonstrates complete understanding of the problem. All requirements
of task are included in response.
4 Demonstrates considerable understanding of the problem. All
requirements of task are included.
3 Demonstrates partial understanding of the problem. Most requirements
of task are included.
2 Demonstrates little understanding of the problem. Many requirements
of task are missing.
1 Demonstrates no understanding of the problem.
0 No response/task not attempted
Criteria Beginning 1 Developing 2 Accomplished 3 Exemplary 4 Score
1 Description reflecting beginning level of performance
Description reflecting movement toward mastery level of performance
Description reflecting achievement of mastery level of performance
Description reflecting highest level of performance
2 Description reflecting beginning level of performance
Description reflecting movement toward mastery level of performance
Description reflecting achievement of mastery level of performance
Description reflecting highest level of performance
3 Description reflecting beginning level of performance
Description reflecting movement toward mastery level of performance
Description reflecting achievement of mastery level of performance
Description reflecting highest level of performance
4 Description reflecting beginning level of performance
Description reflecting movement toward mastery level of performance
Description reflecting achievement of mastery level of performance
Description reflecting beginning level of performance
Creating a task to test L2 writing Describe the population you teach Define the purpose for an L2 writing
assessment task Define L2 writing ability: Think of a TLU
domain task in which that language ability would be demonstrated
Using the framework of task characteristics, design a task that would require L2 writing ability to accomplish
Share your task with a partner
Self-Assessment I could create an Assessment Use Argument (conceptual link between
assessment & intended decisions & consequences). I can define the construct of L2 writing ability. I can articulate a conceptual framework for designing and evaluating
L2 writing assessment tasks. I can identify the steps in constructing a language test. I can evaluate strengths and weakness of some specific L2 writing
items. I would be able to create a rubric to serve the purposes of an L2
writing assessment in my program. I practiced item/task evaluation. I created an L2 writing task that would be suitable for my program.
THANK YOU
for your participation!