designing and validating a visual analogue scale version of the vocabulary learning questionnaire...
TRANSCRIPT
Designing and validating a visual analogue scale version of the Vocabulary Learning Questionnaire Peter Yongqi Gu Victoria University of Wellington, New Zealand
SSU2015Klagenfurt, Austria16-17 October, 2015
In this session…
1. updating and validating a questionnaire on vocabulary learning strategies
2. exploring a VAS tool for survey research
Part I
Updating and validating the Vocabulary Learning Questionnaire (VLQ)
Updating VLQ• VLQ:
– All validated versions done in Chinese • Gu and Johnson (1996) Version 3• Gu and Hu (2003) Version 5
– No full validation info publically available• VLQ6:
– An ESL version– Provide full validation documentation
Creating VLQ6, an ESL versionVersion Purpose ProceduresVLQ6.0 Create English
version of VLQ5
Find all English statements in previous versions and match them with the 90 statements in VLQ5.
Make an ESL version
Go over each statement and simplify them. Aim: Most words should fall within the first 2000
most frequent words Include as few as possible academic words
and off-list words. Include as few as possible meta-language
jargons. Where jargons can’t be avoided, give simple
examples. Use simple sentence structures
Changes in the ESL versionAn example
• VLQ5 English version:– I consult the dictionary about the subtle differences in the
meanings of English words. – I create a mental image of the new word to help me
remember it.• VLQ6.2:
– I check the dictionary when I want to find out the similarities and differences between the meanings of related words.
– I create a picture in my mind to help me remember a new word.
Vocabulary profile (VLQ6.2)
• K1 (First 1000 most frequent) words: 89.28% • K2 (First 2000 most frequent) words: 5.18% • K1+K2 words: 94.46%• AWL words: 3.01%
Analyse confirm context corresponding create data focus indicates items link logical networks neutral paragraph response section similar similarities similarly strategies structure text topic
• Off-list words: 2.53%collocations EPP etc grammatical phrase prefix prefixes pronunciation repetition suffix suffixes textbooks untrue usage usages vocabulary
VLQ6: Updating proceduresVersion Purpose ProceduresVLQ6.1 Trial and fine-
tune for clarity Get ESL teacher feedback re clarity of
statements Trial 1, one advanced level ESL user, to ensure
clarity of statementsVLQ6.2 Create
online VAS version;
Trial among 5 students
Trial 2, four ESL students, think-aloud for both paper version and online versions. To catch understanding issues and simplify
statements further, and catch potential similarities and
differences for Likert scale and VAS versions.
Re-write problematic statements after trial 2. VLQ6.3 Formal
piloting Trial 3: Among a larger group of ESL learners
o Statistically compare Likert scale version with VAS version
o Catch potentially problematic itemso Catch problems with online
administration
VLQ6: Construct validation of VAS versionVersion Purpose ProceduresVLQ6.4 Construct
validationAdminister questionnaire among 100+ students Exploratory factor analysis
o Do the data support the existing structure of the questionnaire?
o Can the questionnaire be shortened? Should certain categories be deleted? Can some categories be combined? Can some items be deleted?
o Make sure resulting categories are meaningful.o Make sure resulting categories are internally
consistent (Cronbach’s alpha). Confirmatory factor analysis
o Model fit of questionnaire structureo Reduction of items
Predictive validity: Relating VLS to learning outcomes
Part II
Exploring a VAS tool for survey research
Likert Scale vs Visual Analogue Scale
Visual Analogue Scale (Slide bar):
Likert Scale:
Questions
1. Is the VAS (slide bar) version comparable to the Likert scale version?
2. Does the slide bar version elicit more interval-like data?
Problems of the Likert-type data
• Two major problems– Ordinal nature of data– Often not normally distributed (both item and
scale levels)• Over the years, statisticians have worked out
various ways in treating ordinal data;• Applied linguists have typically been treating
Likert-type data as interval data
Is the slide bar a better scale?
• Think-aloud and interviews• Matching the slide bar data against the
Likert-type data
How does the slide bar scale compare to the 7-point Likert scale? (Interview)
• I find the two interviews interesting, except that, the online one is quite similar to the paper interview. –Johnson
• In general, I find that the paper and online survey closely resembled each other. –Tran
• I think paper survey has no much differences from online survey, except for a little format. –Wendy
• I think the paper survey and the online survey are quite similar. –Roxy
How does the slide bar scale compare to the 7-point Likert scale? (Interview)
• “In some questions, the online scale may have helped to rank my agreement level clearlier than the paper scale (e.g. Between absolutely disagree and 10% agree). Nevertheless, in general, I don't think they made a big difference in my responses.”
--Tran
How do participants use the slide bar scale? (Johnson: range 5-80%)• No. extremely untrue. Give it 5. • I don’t do that. I will say 5%.• No. give it 10%. • No. this is absolutely untrue. I would give it 10. • No. I don’t do that. Let me say 20% of that. • No. that’s not true of me again. Maybe 30% of that. • No. for me no. maybe in my childhood tell me what to do . but
at this level, I don’t do it. I give it 30%.• Huh, it’s a good way. I would say 40 percent for me. I don’t do
this often.• Yes. About 50%. The 50% comes from general knowledge.• 50%. sometimes I read, sometimes I don’t.
How do participants use the slide bar scale? (Johnson: range 5-80%)• Yes. That is common. I would give it 55%.• I agree. I agree huh partly moderately between 55 and 60. In the
middle, 54, 55, 55. • This, the words I look up for, I will give it 60 percent. The reason
is …• Yeah, collocations. That’s right. I will give it huh … … 50, 65, 65.
The reason is that …• I agree with that. I give it 65. • I agree with that. Yes. I give it 75, 70. I give it 70. Yes. • Yes. I agree. I give it 75. The reason is …• That’s true of me. I give it 80%. It’s more applied.• Yes. I totally agree. 80% for me it’s 80%. And the reason …
Slide bar allows fine-grained choices(Johnson: Item 15)
Item 15: I know which word I can guess and which word I can’t.
• Yes. That is true. Sometimes when I, like, I probably when have come across it already, my mind is drifting out. It depends on the mind whether you can have a long term memory there. And then in the speech it takes times to process the information to answer that question. So for me it take time to activate before I can understand. I know which words to guess. I give it 58% confined of what words to guess.
• RA: 58?• Johnson: 56, maybe 55.
How do participants use the slide bar scale? (Tran: range 0-100%)• 0. Untrue of me. I learn that things that are necessary for me not just
because the teacher does not mention it would be useless.• 0 % because I don’t think I have ever done that.• En.. 10% because I don’t think the general topic would be very helpful
to the unknown word.• 20% because I did this once in a while only when it is very necessary to
do so.• 30% because I only do this when it is very important to remember.• 30% because for a certain time I tried to do that but after that I lost the
habit.• 40% because I remember some popular ones but not many of them.• I think 40. It would be quite difficult to use the knowledge of world to
guess about a word.• 50% because it is not all the time that part of speech is really helpful.
How do participants use the slide bar scale? (Tran: range 0-100)• 60% because I will look it up when I finish reading but not during the
reading.• 70% because that’s how I practice my speaking.• 70% because that help me to guess whether the word has negative or
positive meaning or how it fit into the eh essay. (72 on slide bar)• Em…80% because I did some reading comprehension where the reading
itself also provide definition of some technical and complicated term.• 90%, oh 80% because when the word is too complicated I would also
write it down several times.• 90% of me because that’s how I read newspaper and articles during my
free time. (92 on slide bar)• 100%. Em…yeah extremely true. that’s the only way to memorize the
spelling.• 100% because that’s my style to memorize new words.
Are the two versions significantly different?Paired-samples t-test (e.g., Extended dictionary strategies)
Mean N SD t sig Scale Likert data (Sum of 5 items) 25.9444 18 3.96265 -1.110 .282
Slide bar data (Sum of recoded 5 items) 26.7778 18 6.02500 Item 37 7-point Likert scale 4.667 18 1.4951 -2.129 .048
Slide bar recoded into 7 points 5.3333 18 1.37199 Item 38 7-point Likert scale 4.833 18 1.4653 -2.129 .048
Slide bar recoded into 7 points 5.5000 18 1.29479 Item 39 7-point Likert scale 5.222 18 1.1144 -.175 .863
Slide bar recoded into 7 points 5.2778 18 1.67352 Item 40 7-point Likert scale 5.667 18 .9701 1.699 .108
Slide bar recoded into 7 points 5.1667 18 1.42457 Item 41 7-point Likert scale 5.556 18 1.0966 .236 .816
Slide bar recoded into 7 points 5.5000 18 1.24853 Answer: Not quite.
Are the two versions significantly similar to each other?
Paired Samples Correlations N r Sig.Pair 1 DicExtendP &
DicExtendO_Recoded 18 .877 .000
Pair 2 P37 & O37Recoded 18 .574 .013Pair 3 P38 & O38Recoded 18 .543 .020Pair 4 P39 & O39Recoded 18 .596 .009Pair 5 P40 & O40Recoded 18 .511 .030Pair 6 P41 & O41Recoded 18 .644 .004
Answer: Yes
77% overlap
Summary
• The respondents’ attention was mainly focused on the content, not the format.
• They all perceived the two versions as very similar.
• The slide bar version offered a wider range and finer-grained choices.
Preliminary suggestions
• Online slide bar surveys are at least comparable, if not better, than paper based Likert scale versions in eliciting the same constructs.
• Online slide bar surveys produce data that resemble interval scale more than the Likert-scale does.
• Online surveys with a slide bar is now feasible.• More confidence than Likert data in using
parametric statistics
Caveat
• Human judgment is fundamentally imprecise, as such,• Slide bar data are not interval data. E.g., The distance
between 20 and 21 does not equal that between 60 and 61.
• Slide bar data can be over-sensitive. E.g., 63 for one person is not necessarily more than 60 for another person.
• Despite more confidence in using inferential statistics with slide bar data compared to Likert-type data, researchers should use caution in interpretation.
Research tools• A tool we choose to use defines the
epistomological perspective we take in observing our target construct.
• Researchers need to be aware of the strength and limitations of each tool.
• Make sure the tool we use is as good as it gets.
• I vote for methodological triangulation.
