understanding difficulty & complexity in writing tasks: toward a useful analytic framework anne...
TRANSCRIPT
UNDERSTANDING DIFFICULTY & COMPLEXITY IN WRITING TASKS: TOWARD A USEFUL ANALYTIC FRAMEWORKAnne Nebel
Lancaster University [email protected]
Georgetown University [email protected]
School of Foreign Service in Qatar
TBLT 2009 Lancaster
TBLT research
Classroom context (sequencing tasks for learning)
Speaking tasks Assessment context
(Chalhoub-Deville, 2001; Elder, Iwashita & McNamara, 2002; Wigglesworth, 2001)
Writing tasks(Ishikawa, 2008; Kuiken & Vedder, 2007)
Predominant models developed for speaking (Robinson, 2001, 2005; Skehan & Foster, 1999, 2001)
Writing assessment tasks
My research investigates the difficulty and complexity of a set of 8 writing tasks
Not experimentally designed tasks Fixed set of retired test tasks (ECCE) Writing task option pair: letter or essay
Task choice?
Challenge of task equivalence Within each test and across versions Potential for eliciting sufficient output
equally rich in features under assessment Raises validity & reliability issues
Fairness is a primary concern What decisions and conclusions are
reached from the test results? (Bachman & Palmer, forthcoming)
Research context (1)
High-stakes nature of language proficiency assessment (e.g. Greece) Employment opportunities Career advancement Pay scale May impact performance (affective
variables) Over-trained and test-savvy
Test context as a facet in the complexity equation?
Research context (2)
Low-stakes environments (e.g. Qatar) Opportunity to look at test tasks outside
the test context Test-like conditions (timed, no resources,
etc.) None of the affective and pragmatic
interference of a real test situation Motivation?
My research: writing assessment tasks
Perceptions of task difficulty Student (N=20 pilot questionnaire and raking activity) Teacher (N=5, 30-minute interviews & ranking activity) Student (N=? interviews and raking activity)
Analysis of task complexity 8 task set of 4 pairs (letter & essay) Various model adaptations – toward the development of a
framework
Analysis of performances on tasks Measures of CAF (which, why) Interactions between perceptions, task complexity and
performance (what, how)
How to analyze difficulty & complexity?
Expert judgments, VPAs, interviews, surveys and questionnaires.(e.g. Elder, Iwashita and McNamara, 2002; Hamp-Lyons, 1994)
Existing models (Robinson, 2001, 2005; Skehan & Foster, 1999, 2001; and Ishikawa, 2008; Kuiken & Vedder, 2007)
Taxonomies of characteristics (Bachman & Palmer, 1990; Carr, 2006)
Coding & classification systems (Silver et al., 2009)
DIFFICULTY
learner factors / perceptions
COMPLEXITY
task structure / analysis
Student perceptions of difficulty (phase I)
Task choice? letter (18), essay (2)
S1: “I chose the letter because it gives me the way how to write and explain my opinion.”
S4: “Letter, because it’s easiest.”
S14: “Because I can explain a situation better in a letter than in the essay.”
S17: “It’s easier to give an opinion on a letter (less formal) than writing an essay.”
S18: “In my view a letter is more personal and it is easy to write their own opinion.”
S6*: “Because I don’t know how to write an essay.”
“View from the learner” (Ortega, 2005)
“Whether through discourse analysis, psychological questionnaires, or post-task interviews, a process-product approach that considers the full landscape of variables contributed by task, leaner and linguistics outcomes, has the potential to illuminate our theories of the role of planning in task-based language learning” (Ortega, 2005: 116).
…of task complexity?
Approach to perceptions data (phase II)
PhenomenographyQualitative approach that makes use of in-depth interview data to explore how participants conceive of a phenomenon (Marton, 1981);
Reveals “categories of understanding” that can be ordered in an outcome space (Micari et al., 2007);
Characterizes “ways of seeing something in terms of the critical aspects of the phenomenon” (Pang, 2003: 145)
Theory and method
PCP (Kelly, 1955) / Repertory Grid Technique Personal construct theory is a theoretical framework that “provides a vocabulary with which to interpret the constituents and processes by which a person construes his or her experience” (Roberts, 1999:119).
Dyadic difference elicitation techniqueElicit constructs based on a comparison of elements.
“There is nothing sacrosanct about the triad” (Fransella, Bell and Bannister, 2004: 28).
Elements: 8 tasksConstruct elicited (sample):
Specific / non-specific Personal / general (“private/public orientation”
Hamp-Lyons & Mathias,1994: 54-55)
Real life / artificial or hypothetical Meaningful / meaningless Concrete / general or vague Specified content / need to invent content Voc-ed domain / academic domain 1st person / 3rd person Suggested structure / no suggestion of structure
Task ranking
Teachers (N=5) ranked 8 tasks in order of difficulty.
In every case, task 6 was ranked as the most difficult or the second most difficult.
In every case (but one), task 1 was ranked as the easiest or the second easiest task of the set.
Task analysis
(essay option, task pair 3)Should public and private
resources be used to support development in locations where natural disasters, such as floods, forest fires, and volcano eruptions, could occur? Explain your views. Give specific reasons and examples.
(http://www.lsa.umich.edu/UMICH/eli/Home/Test%20Programs/ECCE/General%20Information/2000ecceformb.pdf)
(letter option, task pair 1)
Write a letter to the Editor of the City Times. Name one person you think should go on the Millennium Exchange Program. Describe the person, and tell why you think this person should go. You may nominate yourself or someone else you know.
(http://www.lsa.umich.edu/UMICH/eli/Home/Test%20Programs/ECCE/General%20Information/ECCE%20Practice%20Tests/ECCE_06NDWriting3.pdf)
Task 6 (most difficult) Task 1 (easiest)
Task complexity analysis: constructs
-specific-personal -concrete (abstract)-meaningful
6 = more complex task
+ specific+ personal+ concrete+ meaningful
Task 6 (rebuild city with funds essay)
Task 1 (Millennium nominee letter)
Task complexity analysis: (Silver et al., 2009)
High cognitive demand analyze evaluate interpret create justify make decisions
6=more complex task
Low cognitive demand recall remember describe guided routine constrained
Task 6 (rebuild city with funds essay)
Task 1 (Millennium nominee letter)
Task complexity analysis: LACM and TCF(compared in Kuiken & Vedder, 2007)
Code complexity Vocab
Cognitive complexity Familiarity of topic Predictability Information
organization
6=more complex task
Resource directing Few elements Here and now No reasoning
demands Resource dispersing Single task Prior knowledge Task difficulty
LACM (Skehan) TCF (Robinson)
Task complexity analysis: 1 adaptationfor speaking (Elder, Iwashita and McNamara, 2002)
2 performance conditions (+/-)4 task dimensions:
Perspective: your own/another’s Immediacy: there-then/here-now Adequacy: complete/incomplete input Planning time: [constant]
Which is more complex: 6? Equal? Inconclusive
Complex tasks, complex writing? Although they don’t always take into account the
same task dimensions, the models are capable of accounting for complexity in writing tasks; which dimensions are essential and how do we operationalize them?
Can the models predict the corresponding features of writing that will demonstrate complexity of thinking in writing? Which features and why?
In language testing, an equally important consideration is how well we can measure these features with a scoring rubric or scale.
References
Bachman, L. and Palmer, A. (1990) Language Testing in Practice. Oxford: Oxford University Press.
Carr, N. (2006) The factor structure of test task characteristics and examinee performance. Language Testing 23, 269-289.
Chalhoub-Deville, M. (2001) Task –based assessments: characteristics and validity evidence. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 10, pp. 210-228.
Elder, K, Iwashita, N. and McNamara, T. (2002) Estimating difficulty or oral proficiency tasks: what does the test-taker have to offer? Language Testing 19 (4); 347-368.
Fransella, F., Bell, R. and Bannister, D. (2004) A Manual for Repertory Grid Technique. West Sussex: John Wiley and Sons.
Hamp-Lyons, L. and Mathias, S. (1994) Examining expert judgments of task difficulty on essay tests. Journal of Second Language Writing 3 (1); 49-68.
Ishikawa, T. (2008) The effect of manipulating task complexity along the [+/- here and now] dimensions on L2 written narrative discourse. In M. Garcia May (ed) Investigating tasks in formal language learning. Clevedon, UK: Multilingual Matters.
Kelly, G. (1955) The psychology of personal constructs. New York: Norton.Kuiken, F. and Vedder, I. (2007) Task complexity and measures of linguistics
performance in L2 writing. IRAL 45, 261-284.Micari, M., Light, G., Calkins, S. and Streetweiser (2007) Assessment beyond
performance: phenomenography in educational evaluation. American Journal of Evaluation 28 (40); 458-476.
Marton, F. (1981) Phenomenography: describing conceptions of the world around us [online version]. Instructional Science 10, 177-200. Retrieved 2/12/08 from http://www.ped.gu.se/bjorn/phgraph/misc/constr/html)
References
Ortega, L. (2005) What do learners plan? Leaner-driven attention to form during pre-task planning. In Ellis, R. (ed) Planning and task performance in a second language. Amsterdam/Philadelphia: John Benjamins. Pp. 77-110.
Pang, M. F. (2003) Two faces of variation—on continuity in the phenomenographic movement. Scandinavian Journal of Educational Research 47 (2); 14-156.
Roberts, J. (1999) PCP as a framework for research into teacher learning and thinking. Language Teaching Research 3 (2); 117-144.
Robinson, P. (2001) Task complexity, task difficulty and task production: exploring interactions in a componential framework. Applied Linguistics 22, 27-57.
Robinson, P. (2005) Cognitive complexity and task sequencing: studies in a componential framework for second language task design. IRAL 43, 1-32.
Silver, E., Mesa, W., Morris, K., Star, J. and Benken, B. (2009) Teaching mathematics for understanding: an analysis of lessons submitted by teachers seeking NBPTS certification. American Educational Research Journal, 46 (2); 501-535.
Skehan and Foster (1999) The influence of task structure and processing conditions on narrative retellings. Language Learning 49, 93-100.
Skehan and Foster (2001) Cognition and tasks. In P. Robinson (ed) Cognition in second language instruction (pp. 185-205). Cambridge: Cambridge University Press.
Wigglesworth, g. (2001) Influences on performance in task-based oral assessments. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 9, pp. 186-209.
