cue forum2008
TRANSCRIPT
Research design: The backbone of academic inquiry
Peter Neff – Doshisha UniversityMatthew Apple – Nara National College of TechnologyDavid Beglar – Temple University Japan
CUE Forum 2008
Olympic Memorial Youth CenterNovember 1, 2008, 1:15 - 2:50 p.m.
Introduction: The importance of good research design
Approaching the studyDeveloping the study design
Break for Q&A
Designing the right instrumentImplementing your designConclusion
Further Q&A
Overview
Introduction: The importance of good research design
Poor design vs. Good design
Poorly-designed studyHit on an idea, dive right inNo background researchThrow together a survey, give to a
group of unwary participantsCollect data, then ponder how to
analyzeRun to a colleague for helpFish around for most “interesting”
findingsPray to get published
Poor design vs. Good design
Well-designed studyHit on an idea, do background researchFormulate relevant, specific, practical RQsConsider participants, context, data analysis
in advanceDecide/develop instrument; pilot and revise
itDecide on appropriate pre/post-test
instrumentsPlan stages and structure of data collectionPrepare participants adequately…Then carry out the study
The importance of good designA well-designed study provides
many benefits:– Demonstrates researcher
knowledge– Ties the study to an underlying
philosophy– Provides a clear path for the
researcher(s)– Helps avoid mishaps of previous
studies
The importance of good designOther benefits
– Leads to more concrete results, more definitive conclusions
– Improves chances of publication– Raises the status of SLA as a
field of inquiry
A word about mixed methods designsThe great quan-qual debateMixed methods – the “best” of
both worldsAdd a qualitative component to a
quantitatively-oriented study: – Participant interviews– Observational, audiovisual data– Open-ended survey questions
Plan for qualitative analyses (text analysis, response coding)
Part 1Approaching the Study
Approaching the studyHitting upon research ideasReview of the literatureFormulating research questions
Hitting upon research ideasIdentify the topic in a few wordsReflect on “doability” of research
– Can I research this?– Should I research this?– Am I interested in researching this?
Review of the literature can help redefine and revise ideas
Identifying the topic: Hints for starting to narrowPose a short question using “what”
or “how” Write a short title that consists of
one sentence under 12 wordsAsk a friend or colleague to read
your topic and gauge their reactionsDraft research questions to see if
the topic can be adequately explored
A “researchable” topic“Can I do this in my current
situation?”“Does this concern people at other
institutions?”“Does this add to the current body of
research related to this topic?”“Does this study contribute something
from a unique perspective?”
Filtering “probably not so good” ideas: To boldly go where no research has
gone before… (The “Star Trek” idea)My theory is clearly better than X
(The “Steven Krashen is so wrong” idea)My classroom is totally unique
(The “I don’t need theory” idea)This is a really cool technology /
methodology / text book (The “I am primarily a teacher” idea)
Filtering ideas: A few hintsReview research designs and
statistical techniquesReview teaching methods and
overall SLA research resultsEvaluate access to potential study
participantsPlan time for material creation,
study design, and implementation
Review of the literatureRelate the study to continuing
“dialogue” in current researchFinding a “gap” in the literature Provide a framework for the
importance of the study
Review of the Literature:Finding a “gap” in knowledge“We do not enough about X…”“This way of looking at X has never
been done…”“This way of learning about X has
not been duplicated in my context”“Previous research has
inadequately explored X…”
Finding literature: Some hints Google Scholar using key words or
researcher names Scour recent literature review articles Check for “cited” numbers online Get access to university databases Refer to recently published articles
– After the year 2000– During the previous 2 to 3 years
Examine “outside the field” articles
Finding literature: Separating the wheat…“Top tier” journal articlesMost-often-cited articlesRecent articlesResearch articles (not reviews)Books / Edited book-articlesMajor international conference
papersDissertations / dissertation abstracts
…from the chaff“In-house” journal articlesArticles from “proceedings” booksOnline journal articles with only .html
versionsUnedited books from small publishersNewspaper and magazine articlesWeb pagesAnecdotal evidenceYour own previous papers for an MA
course
Research questions: A few useful guidelinesNaturally flow from the literature reviewStrongly connected to the topicAt least two or three (not one)……but not five or six or moreAs specific as possibleDirectly concern variables in the studyDo not contain yes/no question words
RQs: What not to ask“Is X true/false?”“Will X happen if…?”“Does X cause Y?”
“What do participants think of X?”“Why does X happen?”
RQs: What to ask“What differences exist between…”“Compared to X, how does Y…?”“To what degree do X and Y differ…?”“When X is controlled for Y…, how
does Z…?”
“What are underlying patterns among…?”
“To what degree does X predict Y?”
Part 2Developing the Study Design
Research design
Cross-sectional design: A design in which data are collected from a sample at only one point in time.
Longitudinal design: A design in which data are collected at more than one point in time.
Randomized Control-Group Pretest-Posttest Design
Experimental Group 1 T1 Xa (Method a) T2
Experimental Group 2 T1 Xb (Method b) T2
Control Group T1 T2
Randomized Control-Group Pretest-Posttest Design
Reasonably strong conclusions can be reached about the effects of the treatments.
Problem 1: Within session variation (e.g., different teachers or room conditions) may intervene.
The solution? Randomly assigning participants, times, and places to the experimental and control conditions.
Problem 2: The pretest may interact with the treatment. This potential problem is dealt with in the next design.
Randomized Solomon Six-Group DesignPretested (Random assignment) T1 Xa (Method a)
T2
Pretested (Random assignment) T1 Xb (Method b)T2
Pretested (Random assignment) T1T2
Unpretested (Random assignment) Xa (Method a)T2
Unpretested (Random assignment) Xb (Method b)T2
Unpretested (Random assignment)T2
Randomized Solomon Six-Group Design
This design amounts to doing the experiment twice –once with and once without pretesting.
It is possible to know what effects, if any, are associated with pretesting.
If the results of the “two experiments” are consistent, greater confidence can be placed in the findings.
Counterbalanced Design
This design is useful when randomization is not possible and intact groups must be used.
Replication
Xa Xb Xc Xd
1 A B C D2 B D A A3 C A D B4 D C B C
Counterbalanced design The counterbalanced design rotates out
the participants’ differences (e.g., one group has more aptitude or motivation than the other groups) by exposing each group to all variations of the treatment.
Order-of-presentation effects are controlled.
Primary weakness: The possibility of carryover effects from one treatment to the next exists. Allowing time between treatments can alleviate this problem.
Control-Group Time-Series Design
Experimental Group 1 T1 T2 T3 T4 Xa (Method a) T5 T6 T7
T8
Experimental Group 2 T1 T2 T3 T4 Xb (Method b) T5 T6 T7
T8
Control Group T1 T2 T3 T4 T5 T6 T7 T8
Control-Group Time-Series Design
This design allows the researcher to determine growth over time, and the effect of an intervention.
The presence of a control group increases the trustworthiness of the results because the possibility of a contemporary event causing any gains can be determined.
Control-Group Time-Series Design
This design can be extended by exposing the participants to the intervention on multiple occasions.
This approach is more sensitive to partial gains in knowledge and tests the strength of the intervention more than once, thus giving the researcher a more accurate understanding of the effectiveness of the intervention.
Experimental Group 1 T1 T2 Xa T3 T4 Xa T5 T6
Experimental Group 2 T1 T2 Xb T3 T4 Xb T5 T6
Control Group T1 T2 T3 T4 T5 T6
Q&A Break
Part 3Designing the Right Instrument
Instrument Design Commonly used instruments in SLA
research– Scored tests– Rater scores– Surveys– Interviews
Consider your eventual data analysis when developing instruments
Instruments - Scored testsPluses
Quantitative items (M/C, Cloze/C-tests)– Simple to score large
# of participants– Easier to analyze
Qualitative items (short answer, timed essays)– Good complement to
quantitative scores– Can provide more in-
depth assessment of participants’ abilities
Minuses Quantitative items
– Limited to one type of data
Qualitative items– Take more
time/effort to score– Rater bias
Instruments – Performance ratingsAn assessment of participants’
performance in an assigned taskTasks may include presentations,
interviews, written essaysPerformances can be scored using a
Likert-scale, a rubric, or holisticallyUsually scored by at least two
“expert” raters; sometimes also by peers
Performance ratings
Rating criteria should be concretely established with little ambiguity
Avoid including too many (or too few) criteria for one performance task
All raters should undergo a “normative” training session prior to assessment
Use models to train ratersAvoid single-score holistic ratings
Instruments - SurveysOften used for:
– Collecting learner history data (L2 study experience, other background info)
– Assessing participants’ attitudes towards a predetermined construct (language learning motivation, anxiety using the L2)
– Determining reactions to an experimental treatment (teaching methods, innovative learning tasks)
Survey makingFor non-advanced learners – surveys
should be in their L1Build in redundancy - Include multiple
questions for each concept areaQuestions should be simply worded –
avoid negative or confusing wordingDepending on the purpose of the
survey, 20-40 items/session is a good range to shoot for
Survey makingAny survey used in a serious study
should be piloted in advanceIt is acceptable to make
adjustments to an existing instrument
Likert-scale items should usually have between 4 and 6 choices
A few qualitative questions can provide a nice complement to quantitative instruments
Instruments - InterviewsInterviews can provide an
excellent qualitative component to a larger study
It is not necessary to interview all participants – a subsample as small as 10-20% can
be acceptableUse your best judgment on
participants’ language ability– For intermediate-and-above learners,
L2 interviews are often fine
Conducting interviewsInform students they are being
interviewed, obtain consentRecord unobtrusively “Warm up” the participants before
getting into the heart of the interview
Collect more data than you need
Validating Instruments
Instrument ValidityThe construct = The heart of the
matterWhat construct do you wish to
measure? How do you define the construct?What are its component parts? Do
they form a unified whole?
Operationalizing the construct: The items Conceptualize the construct as a continuum: easy—difficult items and less able—more able persons. How have other researchers measured the construct? Write original or adapted items. Cover the estimated range of your participants. Write 50% more items than you intend to use. This will allow you to “cherry pick” the best items as well as items at various levels of difficulty.
Operationalizing the Construct: The ItemsMore able | More difficultpersons | items | x | item 1 xx | item 2 item 3 xxx | item 4 item 5 xxxx | item 6 item 7 item 8 xxxx | item 9 item 10 item 11 xxx | item 12 item 13 xx | item 14 x | item 15 |Less able | Less difficultPersons | items
Operationalizing the Construct: The ItemsAfter piloting the items,
statistically analyze the results.Examine dimensionality, item
difficulty, and item content.Select the best items to make an
efficient, highly reliable instrument.
Part 4Implementing Your Design
Implementing the designIncluding other researchersPractical issuesHandling ethical concerns
Including other researchers in the studyThe nature of the researchers
involved– Main researcher plus “helpers”– One researcher plus “other participants”
The nature of the instructors involved– Teaching methods– Students taught– Course goals– University program goals
Working with other researchersWork with people you know and trustEstablish a schedule early Define clear roles for each researcherDecide definite research goals prior
to data collection Keep in regular contactThe “band practice time” principle
Example research rolesHead researcher / contact personData entry specialistStatisticianInterviewerLiterature analystEditor / proofreader
Heading off potential problems Explain study commitments prior to
starting the study Agree on “ownership” prior to data
collection and data entry– Who will keep the data?– Whose name comes first, second, etc.?
Keep everyone aware of deadlines Include everyone in decision-making
processed and data analysis
Things to avoid in group researchPeople you don’t knowResearch groups larger than four
or five membersUsing someone for language skillsInvolving others just to get more
participantsForgetting to thank others for
their assistance
Improving relations with research helpersWrite clear instructionsThank profusely for their time and
effortOffer to send copies of final
research papers and/or resultsOffer to assist in future researchKeep in touch after research ends
Practical IssuesTiming of implementationLearning and research contextParticipant consentFinancial considerations
Timing of the implementationBeginning, middle, or end of semesterDay of the weekTime of dayExams and exam preparation periods“Culture Festivals” or other club-
related events“Open classes” or “parents’ day”
Learning and research contextDiffering course goals (I.e.,
listening class vs. reading class)Different major field of studyGender, age, year in schoolNumber of class meetingsPerception of the value of
research by institution heads
Participant consentAlways allow for “non-participation”
choice from potential participantsWrite clear instructions for
participants asking for their cooperation
Ask co-researchers or helpers to briefly inform participants about their choice
Financial considerationsCopies for questionnaires, exams,
etc.Computer analysis softwareMailing costsInterview travel costsRecording equipment
Reference booksJournal article costs
Heading off lack of cooperation problemsReview requirements of the study
– How many items in the questionnaire? – How many treatments?
Recognize that students are busy, tired, etc.
Plan to get between 30-50% more data than you need
Try to get more data at a later date
Ethical Considerations
Ethical considerations
Students should not be exploited just because they are there
In theory, they should derive benefit (directly or indirectly) from the research
Explain the basic purpose of your research before collecting data– 1-2 sentences should be fine
It is also good to briefly explain this at the beginning of any survey instrument
Ethics - ConsentProvide students with the chance to
opt out of participationVerbal consent is usually acceptable
for surveys, ratings, and test scoresA written consent form may be
necessary for more involved forms of participation (interviews, essay passages)
When in doubt, check recent articles in well-known journals for guidance
Ethics – Other considerationsThe role of this study within your
institutionPotential gender, proficiency or
other issues that may affect your data or conclusions
Have a plan for anonymizing the data (and consider making this conspicuous on your instruments)
Conclusion
In conclusion…There are many factors to consider when
embarking on a serious studySome points to take away…Most studies should fill a place (a “gap”)
within the current academic dialogue Research questions should reflect
continuity with the literature and should be specific
Carefully consider the design setup which will work best with the participant groups and the research and analytical goals
In conclusionFurther points…Different instruments work better in
different circumstances. Choose those which best reflect your aims.
Plan your analyses at the same time you are developing your instruments
Develop and pilot instruments which can cover responses from a range of participants
In conclusionEven more points…Work with others who will be serious
and committed, and then be an organized and conscientious leader
Consider how practical, pedagogical, and institutional concerns may affect your study
Do not forget current ethical guidelines for carrying out participant research
Good luck with your research!
Thank you for listening
Q&A