autonomy and english proficiency
TRANSCRIPT
Autonomy as a predictor
of English proficiency
Matthew T. AppleNara National College of Technology
Introduction keep it short!
Personal interest in autonomy since 2003, autonomy conference in HK and Zhejiang U, paper in MAYA
Japanese educator's interest in autonomous learning
NNCT interest in autonomy, jishu
But does autonomy actually help students get better at English? Let's find out.
Background
Participants
Methods
Results
Four main sections, hope to spend roughly 4 minutes per section.
Any questions, please keep until the end of the presentation
Run out of time? Send me an email.
Participants
Methods
Results
Background
First start with the background of the study
In particular, the theories about autonomy and its relationship to proficiency
Autonomy = ?
First question: what is autonomy?
Holec (1981) primarily from a political view
i.e., who has control in the classroom
David Little (1991), Leni Dam and others produced a series of small books (most of which are out of print) theorizing that autonomy as learner control of their own goals, materials, and reflections.
Benson (2001) neatly summarized all the autonomy ideas to date (sheep?)
Materials
Learningprocesses
Metacognitiveprocesses
Benson's 2001 synthesis of learner autonomy.
3 aspects.
What7s the difference between learning processes and metacognitive processes?
Problem: autonomy is a multidimensional construct.
No, not a construct, but a concept.
Constructs must be unidimensional. Where they intersect is the latent variable (statistcially speaking). And we need at least three (any two constructs can correlate w/ little meaning).
People and chairs = construct?
Qualitative
Quantitative
Benson himself noted a lack of evidence about studies connecting autonomy w/ proficiency
Two main type sof studies (gross simplication!);
The great qual-quan debate (perhaps a false dichotomy, but certainly true in autonomy studies.)
Qual = mostly case studies, perhaps as many as 3 or 4, usually highly advanced learners in ESL universities or gradu programs.
Quan = large numbers of students, again typically at universities, but relatively few studies in autonomy (Cotterall, 1997, was a learner belief study, not strictly autonomy. Yang (2006) was also beliefs.
The good learner
The good learner of Naiman (1985), Wen & Johnson (1995), etc.
The garden path:
They were successful learners who used autonomous strategies, therefore autonomous students will become proficient.
Ignores other individual differences such as language aptitude, motivation, anxiety, educational backgrounds, gender, age, socioeconomic factors, etc.
Not generalizable across learner populations.
Generalizability?
Generaliability needs greater numbers.
Case studies can't demonstrate that most students will follow the same path.
We need more quan studies to improve the generalizability that autonomy can predict proficiency for ALL ESL or EFL learners.
Hypothesis:Autonomy leads to English proficiency
So, let7s start with the hypothesis above.
Let's assume that it7s true, autronomy will lead directly to greater English proficiency.
All we need now is a group of students, measuring instruments, and analytic techniques. (oy)
Background
Methods
Results
Participants
Next, participants in the study.
The study mostly consisted of high school-age studnets; however, the learning context is somewhat complicated, as well as unique to Japan.
185 technical college Ss
The participants were students at a national technical college.
The college consisted of 7 grades.
1 through 3 grade are equivalent of the three years of hs in Japan, with a marked increase in technical classes (i.e., math, physics, engineering prep) and a corresponding decrease in general ed classes (i.e., English and japanese).
164 students in the study were in the 3rd grade, or high sch senior equivalent. (I hesitate to say equivalent, b/c they tend to take classes at the uni sopho or junior level, at least!)
21 were in the 6th grade, or senkoka 1st year. This is the equivalent age of uni 3rd yr. However, they tend to take classes close to grad school level.
M
E
S
C
I
5 departments represented by the participants in roughly equal n umbers
400
Magic number for engineering students
Why?
J
A
B
E
E
TOEIC 400 is required to receive JABEE nintei, or certification as an engineer
Japan
Accreditation
Board for
Engineering
Education
JABEE means...
The technical college joined the JABEE program in 2005 and has been struggling to meet the English requirement, mostly due to a combination of lack of student motivation and the few number of classes...
Studnets only take 5 English classes of 45 minutes in grades 1 and 2, and only 3 classes per week in 3rd year.
Sen 1 only take 2 classes per week.
400
Magic number is --
315
Participant TOEIC average was this...a bridge too far?
How can they get to 400?
67.5
In only 67.5 hours per year?
100 points = 100 hours?
100 points = 400+ hours (Boldt & Ross, 1997)
No reliable figures, but regardless, many studnets have difficulty increasing 85 points on average.
Background
Participants
Results
Methods
Methods = once more, into the breach
Reasons for ethods, as opposed to actual methods
Those interested in the hw of stats stick around later or send an email!
Autonomy = ?
Let's go back...
Autonomy is what?
Materials
Learningprocesses
Metacognitiveprocesses
A
Autonomy has three aspects.
IOW, there are three measured constructs that form the concept or latent variable of autonomy
Autonomy is a multidimensional construct.
Benson (2001)
Again, the terminology is not entirely accurate as far as stats are concerned.
Rubik's has multiple faces, but we don't consider it to be multidimensional. We do however, consider each side separately when solving the cube...and yet, all sides are obviously related.
Trying to solve all faces at the same time leads to an incomoplete cube...
Construct = ?
Another metaphor...weight lifting...
Each part of the body can be worked on separetly, despite the fact that all part sof the body are obviously related.
For example, here is somebody usig the benchpress.
What's the construct?
Yes, chest or pectoral muscles. And yet obviously, we need our arms and shoulders as well. But the pecs are the focus.
Construct = ?
How about now?
Yes. The biceps, or arm muscles. Yet we also need forearm muscles, triceps, and shoulders. Biceps are the focus.
Construct = ?
Now? Yes, the shoulders. I think you see the point.
Construct = ?
Now what are we working on?
We don't know for sure, b/c we are using legs, back, arms, and shoulders...leading to a severe back strain eventually! And we don't develop what we want...ie., the biceps.
So, it order to properly evaluate, and improve, the construct we want to measure, we need to isolate as much as possible a single costruct, BEFORE we put all the costructs together to form a related whole.
Materials
Learningprocesses
Metacognitiveprocesses
A
Thus, autonomy may consist of RELATED constructs, howver EACH construct must be measured separately/
So, how do we do this?
Rasch item fit analysis
First, check the items for item fit to the intended construct. Does each iterm measure a a different level of difficulty on the contsurct? i.e., can it distinguish between participants who have more or less autonomy?)
Factor analysis
(factor scores)
Then, double check to confirm that there are separet, but related constructs in the data. FA works well here, and factor scores will make the raw data into stanardized scores for later analysis.
Multiple regression analysis
Here's the meat of the analysis: do the independent variables of autonomy predict the dependent variable of English proficneyc?
Materials
Learningprocesses
Metacognitiveprocesses
A
A quick model: here are the three proposed constructs that measure autonomy
English Proficiency
Learning
Metacognitive
Materials
And here's the model of how they predict English proficneyc
TOEIC
Shimo (2008)
And here are the mesuring intsurments.
Shimo (2008) has 18 items...she itended originally to find 4 factors, but she found 2 factors in one study and 3 in another. How many are there, rteally
TOEIC was used for proficiecy measurement...more on that later...
Background
Participants
Methods
Results
You can see the statistical results on your handout. Table 1 shows the item fit stats for the Rasch analysis
Table 2 shows the monte carlo parallel analysis to help confirm how many factors there were in the data
Table 3 shows the traditional FA as another confirmation (you can see some items were mixed)
Table 4 shows the final regression analysis
Here7s the summary!
Lack of item fit to intended constructs
5 items did not fit the intended constructs. 4 of the five items contained words such as English ability, mistakes, or tests.
Students distracted by these words? Maybe another construct COULD have existed with a few more items to tease it out...
But in the end the three hypothesized constructs...
Materials
Learningprocesses
Metacognitiveprocesses
A
Of Benson (2001)...turned into...
Materials
Metacognitiveprocesses
Asomething resembling the Minshuto symbol..
learning processes and metacognitive processes fell into the same construct.
In Shimo this was termed Orientation toward reflective learning, and included goal setting, plannig, and reflectig on mistakes.
So, in a sense, it make sense...are control over learning processes different from metacognitive processes?
In the end, the FA also bore this out: only 2 factors.
Orientation towards improving learning environment
Orientation towards improving learning processes
Asomething resembling the Minshuto symbol..
learning processes and metacognitive processes fell into the same construct.
In Shimo this was termed Orientation toward reflective learning, and included goal setting, plannig, and reflectig on mistakes.
So, in a sense, it make sense...are control over learning processes different from metacognitive processes?
In the end, the FA also bore this out: only 2 factors.
Orientation towards improving learning environment
11% of TOEIC
Autonomy predicted 14% of TOEIC differences, and 11% alone was finding materials.
(3% was control over learning processes.)
Increasing 1 SD of autonomy would lead to 24 points on TOEIC.
Not bad, right?
Uh huh...one problem...
400
Second problem:
Magic number of TOEIC 400...however...
350 450
Error!
TOEIC error margin is AT LEAST 25 points per section (some say 35).
So, TOEIC 400 really is a range of scores between 350 ane 450...
And this is only a 68% chance as well..
hmmm...not exactly a good measurement of English proficney perhaps...
So is the study fatally flawed?:
1. TOEIC-IP is only passive.
2. All students were engineering majors.
3. 92% were male.
4. Other possible causes of TOEIC differences:
Language aptitude (particularly linguistic coding?)Multiple intelligencesMotivationField IndependenceTest anxietyetc.
Generalizability
Planning for improvement
Good first step.
More students. Close to 300 or 400 would be ideal.
Refine the constructs.
Pre and post...AFTER encourahing autonomy?
i.e., I am now using the language portfolio provided by the FLP SIG, will this encourage autonomy? Will this mprove English proficiency?
Stayed tuned!
Autonomy as a predictor of
English proficiency
Matthew T. [email protected]
3rd yrSenkoka 1
Row 11641.#NAN
Row 2211.#NAN
Column 1
170
15