opening up the science: methodological transparency and...
TRANSCRIPT
Opening up the science:
Methodological transparency
and reproducibility in second language acquisi7on
research EmmaMarsdenUniversityofYork,UK
Towards Methodological Transparency & Reproducibility in SLA research
1. Openscience&theIRISdigitalrepositoryofmaterials
2. RelaConshipbetweentransparencyandquality
3. ReplicaCon&theimportanceoftransparency
4. ChallengesandRecommendaCons
Publiclyfundedresearchshouldbemadefreelyavailable
OPENPUBLICATION->citedmore
Wagner,A.B.(2010).
OPENDATA• CorporaoforalproducCondata,L1andL2:FLLOC,SPLLOC,CHILDES• Trønso• CLARIN
->increasedcitaCons Piwowar&Vision(2013)->strongerevidenceandbe]erstaCsCcalreporCng
Wicherts,J.,Bakker,M.,Molenaar,D.(2011).
The s7cks and carrots of open science
Part 1: Open science movement
Part 1: Open science movement
OPENMETHODSMakingmaterialsavailableMaterials=DatacollecContools,instruments,sCmuli,scoringandcodingprocedures,analysisprotocols
Materials design for L2 data collec7on: a lonely, mysterious business
• Someproblems:
Researcherscreateandkeepowninstruments->re-invenConofwheel,poorsystemaCcityofresearch
Maintenanceandaccesstoinstrumentsisadhoc
PublicaConsjusthavebriefdescripConswithshortsamples
Whatacontrastwithfullinstrument!
Part 1: Open science: METHODS
How much of the methods do we see in journals?
Mai,Z.,&Yuan,B.(2016).Unevenreassemblyoftense,telicityanddiscoursefeaturesinL2acquisiConoftheChineseshì…declefconstrucConbyadultEnglishspeakers.SecondLanguageResearch,32(2),247-276.
Part 1: Open METHODS in SLA
WhatisIRIS?InstrumentsforResearchIntoSecondLanguages q Asustainabledigitalrepositoryq 2200+materialsusedtocollectdata:• e.g.quesConnaires,grammaCcalityjudgmenttests,observaCon&interviewschedules,wordlists,sound&videofiles,languagetests,pictures,teachingmaterials,sofwarescripts,
• AndANALYSISPROTOCOLSandDATAq Downloadableq Uploadableq Searchable
Part 1: IRIS: Transparency of methods
Marsden,E.,MackeyA.,&Plonsky,L.(2016).TheIRISRepository:AdvancingresearchpracCceandmethodology.InA.Mackey&E.Marsden(Eds.),AdvancingmethodologyandpracFce:TheIRISRepository(pp.1-21).Routledge.‘AcademyResearchProject’AN110002
ScopeofcontentaswideasthefieldofL2research…
L2percepCon
&processing
MoCvaCon,idenCty,&strategies
…indiversecontexts
Languagelearning
PhonologyGrammarVocabularyPragmaCcs
Languageteaching,&
policyBilingualism
…withdiverseresearchaims
…withdiversetypesofdata
Part 1: IRIS: Transparency of METHODS Marsden, Mackey & Plonsky
Materialscanbeadaptedtosuitdifferentcontexts,learners,languages
Promotestransparency
Easiertoevaluatequalityandreliability
Qualityassurance:onlypeer-reviewedpublicaCons
SCmulates&facilitatesreplicaCon
RaKonalebehindIRIS
Part 1: IRIS: Transparency of METHODS Marsden, Mackey & Plonsky
• 2400files->over1100datacollecKoninstruments• newmaterialscontributedalmostdaily
• From1300researchers• Searchableacross330+parameters,
ü researchareaü typeofinstrumentü languagefeatureü L1,L2ü parCcipantcharacterisCcs:age,proficiencyü author
• MaterialsqualifyforIRISifusedfor:peer-reviewedpublicaKonsorPhDs
Part 1: IRIS: Transparency of METHODS Marsden, Plonsky, Mackey
• SubmissiontoIRISsupportedby37journals• Inacceptancele]ersfromeditorstoauthors,andinauthorguidelines
• UploadtoIRISinpublicaConguidelinesoftheAmericanAssociaFonofAL• FormallyendorsedbyBriFshAssociaFonofAL
• OnlyvenueinALthatqualifiesarCclesforCenterforOpenSciencebadges
• 16,500+downloadsofresearchmaterials
• Citedinhandbooks,methodsbooks,posiConpieces&syntheses
• Usedinmanyresearchmethodstrainingcourses
Part 1: IRIS: Transparency of METHODS Marsden, Plonsky & Mackey
Materials used for ‘year abroad’ research?
Part 2: Open methods to improve rigour and replicability…
The“methodologicalturn”inappliedlinguisCcs
“methodologicalissues…demandakindofprofessionalscruKnythatgoesdirectlytothecoreofwhatwedoandwhatweknow…”
Byrnes,2013,p.825“MethodologicalpracCcesandstudyqualityneedtobemeasured,notassumed”
Plonsky,2013
Benefits of transparent instrument design
Part2a:GrammaKcalityjudgmenttests
Indicatehowacceptablethefollowingsentencesarecompletely completelyunacceptable acceptable
1 2 3 41) Idon’tknowwhatisheeaCngfordinnertonight.2) Idon’tknowwhatheiseaCngfordinnertonight.
Part 2: Open methods to improve rigour and replicability
Frequent use of judgment tests because assumed to be…
Easyto…• develop• administer• score
CommonconvenConsàgreatercomparabilityacrossstudies
15
Part2a:Transparencyofinstrumentdesign:GrammaCcalityjudgmenttests
JTs are controversial Testsofexplicitorimplicitknowledge?(e.g.,Ellis,2005;GuCerrez,2013;Vafaeeetal.,2016)
TheoreKcalperspecKvesonlinguisKcknowledge->methodologicaldecisions
• DoinstrucConssay‘howacceptable…’or‘howcorrect…’?• Scaledvs.dichotomousresponse• Oralvs.wri]en• …
16
Part2a:Transparencyofinstrumentdesign:GrammaCcalityjudgmenttests
Methodological synthesis of JTs Howdowedesign,administerandreportonJTs?
K=299studies382JTsTotalsamplesizeinoursynthesis=24,679!
17
Part 2a: Instrument transparency: JTs. Plonsky, Marsden, Gass, Spinner, Crowther (in progress)
%ofJTsavailable
66
22
8
3
1
0 10 20 30 40 50 60 70 80 90 100
No
Yes, in article
Yes, on IRIS
Yes, other
Yes, article + IRIS
Transparency?
18
Part 2a: Instrument transparency: JTs. Plonsky, Marsden, Gass, Spinner, Crowther (in progress)
88%=306instrumentsunavailabletopublicfor
openscruCny.*****
252notavailableforreplicaCon,evenifyouhaveajournalsubscripCon
A‘SpecialCollec:onofJTs’
onIRISisimprovingthis!
• MedianNwholestudysampleL2learners=47NScontrols=20Overallstudy=60
• Fieldmedian=60(Plonsky,2013)
• L1s45%=English15%=Chinese9%=Japanese8%=French
• TLsEnglish=59%Spanish=17%French=10%
• 77%one-shotdesign->non-developmental,notwithin-subjectoverCme
Consequences of lack of availability Part 2a: Instrument transparency: JTs. Plonsky et al. (in progress)
Imaginethe‘n’ifJTsavailableacross
theglobe?
ThinkoftheL1–L2combinaCons!
ExisCngJTsusedtoinvesCgatechangeover
Cme,e.g.aferteachingor
yearabroad
Transparent design -> clearer opera7onalisa7on of knowledge type
20
41
26
24
20
12
8
7
5
4
1
0 20 40 60 80 100
items randomized
balance of (in)correct
identify errors
time limit
timed
context provided
breaks
rating of confidence
basis of judgment
comprehension Q
Part 2a: Instrument transparency: JTs. Plonsky et al. (in progress)
75%didnotdiscusskindofknowledgeelicited
->thereaderhastoinferwhatkindofknowledgetheresearcherselicited…
withoutfullreporCngofdesignfeatures,withoutseeingtheinstrument!
Consequences of lack of full transparency: Construct validity TimepressuremaKers.
52%wedon’tknowwhetherCmedornotModalitymaKers.
18%wedon’tknowwhetherwri]enorauralBreadthofconstruct-typesandtokensoflinguisFcfeaturetested?
51%wedon’tknowiftherewasonlyone‘version’Ifoneversion,narrowsbreadthofconstructbeingelicited
21
Part 2a: Instrument transparency: JTs. Plonsky et al. (in progress)
• Only15%reportedareliabilitycoefficient(e.g.,Cronbach’salphaforinternalconsistency)
vs45%reporCngreliabilityoverallinSLA(Plonsky,2013)
• Whenreported,meanreliability=0.80(=fieldaverage,Plonsky&Derrick,2016)
• Butreliabilitycalculatedonwholeinstrument• Weneedreliability:
ExcludingdistractorsSeparatelyforgrammaCcalandungrammaCcal
Consequences of lack of transparency: Reliability
22
Part 2a: Instrument transparency: JTs. Plonsky, Marsden, Gass, Spinner, Crowther (in progress)
Reliabilitycoefficientscanbegivenalongside
instrumentsonIRIS.
NewcoefficientspostedeveryCmethatJTused?
Part 2b: Instrument transparency: Self-paced reading
SimilaraimtoJTsEliciCngsensiCvitytogrammaCcalstructureandnormsBut,SPRsidenCfyprecisepointofdifficultyduringparsing,withoutanexplicitjudgment
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
Self-paced reading: an example
I
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs
Trytounderstandthissentence.
Clickforeachwordtoappear.
YouwillbeaskedaquesKonattheend!
Self-paced reading: an example
don’t
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs.
Self-paced reading: an example
know
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs.
Self-paced reading: an example
what
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs.
Self-paced reading: an example
is
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs.
CRITICALREGION
Self-paced reading: an example
she
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs.
CRITICALREGION
Self-paced reading: an example
having
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs.
Self-paced reading: an example
for
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs.
Self-paced reading: an example
dinner
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs
Self-paced reading: an example
tonight
SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?
Part 2b: Instrument transparency: SPRs.
Which picture best matches the sentence?
A B
Theword‘dinner’-aZerthecriCcalregion
Transparency of instrument design: SPR
Methodologicalsynthesis
63studiesreporKngatotalof71SPRtests
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
Transparency and reproducibility of SPR tests in SLA research
Availability No.ofstudies
FullsKmulionIRIS 3Anotherplace(author’swebsite) 3ExampleinarKcleandonIRIS 4Full-allitems-inarKcle 16JustexamplesinarKcle 28Notavailableatall 16
A‘SpecialCollec:onofSPRs’
onIRISisimprovingthis!
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
96%offullmaterialsnotyet
availableonpermanent,open
repository
Consequences of poor availability: Agenda limi7ng - L2s?
Language No.studies:L1 No.studies:targetlanguage
EnglishChineseSpanishGreekGermanKoreanDutchJapaneseFrenchRussianArabic
MulKplelanguages
19954333311118
4321116120200
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
Consequence of poor availability: Agenda limi7ng: Cross-linguis7c influence?
WeknowtheL1ma]ersforonlineprocessing52/71withlearnerswithoneL114comparedacrossdifferentL1s
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
Consequence of poor availability:
Agenda limi7ng. Who are the par7cipants?
56/71=universitystudents• Highermeta-linguisCcknowledgecouldaffectreadingCmes
(KeaCng&Jergerski,2015)
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
Beginner Interm Advanced NearNaCve Bilingual
7 18 50 3 8
• Todate,researchintoprocessingwhencomprehensionishigh• advancedlearnersorbilinguals• onlyanalyse‘correctlycomprehended’sentences
• Butlearnersprocesslanguageevenwhencomprehensionispoor!
(thoughseeKeaCng,Jegerski&VanPa]en,2016;Xu,2014)
Consequence of poor availability: Agenda limi7ng. Why ‘successful’ comprehension only?
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
ExisFngSPRscouldbeusedwithotherlearnerpopulaCons
Consequence of poor availability: Agenda limi7ng. Inves7ga7ng learning over 7me?
GrowinginterestintheroleofprocessinginacquisiCon(Altmann&Kamide,Dussias,Gruter,Lew-Williams&Fernald,Kaan,O'Grady,Pickering&Garod,Chang
etal.,2006;Elman,1990;Seidenberg&MacDonald,1999;SpecialissueofLinguisFcApproachestoBilingualism,5,4).
YET,SPRnotyetusedtoinvesCgatedevelopmentofprocessing:76%oneshotstudies24%cross-secConalstudies,withproficiencyasavariableOneawithin-subject,longitudinaldesign
Thoughsee:McManus,K.,&Marsden,E.(2016).L1explicitinstrucConcanimprove
L2onlineandofflineperformance.StudiesinSecondLanguageAcquisiFon,1-34
ExisCngSPRscouldbeusedinlongitudinaldesigns
e.g.before,during,aferyear
abroad
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
Consequences of lack of transparency: Construct validity - Comprehension of what?
CentraltenantofSPR:processingduringreadingcomprehensionComprehensionquesConusedatendoftrials
QuesConshouldnotfocusa]enConon“criCcalregion”56/71usedCQsBUTwhatdotheseCQsfocusa]enConon?17–noexample34-oneexampleOnly5studiesprovidedmulCpleexamplesofcomprehensionquesCons
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
OneexampleofaCQdoesnottelluswherea]enConisrepeatedly
focussedduringreading
Consequences of lack of full transparency: Construct validity S7muli design (k items, word frequency, length) Lengthma]ers:affectscomputaConalease(Pienemann&Kessler,2011).Yet…
• 25/71lengthofsentencenotspecified
• 5/71reportednumberofsyllablesperword• 19/29usingsegments:numberofsegmentsnotspecified
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
AbankofsCmulihelpsinstrument
designandresearch
training
Consequence of lack of transparency:
Data cleaning and analysis protocols Outlierremoval?• 12didnotreportanyoutlierremoval.• 44usedparCcipantRTs• 15usedbothparCcipantRTsanditemRTs.
Outliercutoff?• 14used2.5SD• 5used200ms-2000ms(asrecommendedbyJegerski&KeaCng,butforanadvancedL2reader).• Widevarietyofranges100ms–25000ms.
Analysealldataorjustcorrectresponses?• 7/56studiesanalysedall• 28/56analysedonlycorrectresponses• 21/56didn’treport
Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)
HavinganalysisprotocolsavailablewouldmakeanalysesmoresystemaCcandreproducible
Summary of Part 2: Greater transparency and availability should help:
• Agendase�ngandresearchaims(cross-secConal,L1-L2influence)• Scopeandpowerofstudies(differentproficiencies,n)• OperaConaliseourconstructs(design:items,Cming,sCmuli)• Comparabilityacrossstudies(datacleaning&analysis)
MoretransparencyandsystemaKcityofmethods->Benefitsforalltypesofvalidity&reliability
‘Be]erreporCng’unlikelytofullyaddressallthese
problems(acrossalljournals&techniques…)
Theactualmaterials
necessary
“ConducCngaresearchstudyagain,inawaythatiseitheridenCcaltotheoriginalprocedureorwithsmallchanges(e.g.,differentparCcipants),totesttheoriginalfindings”(Mackey&Gass,2005:364).
“essenCal…supportfortheory”(Porte,2012)StrongreplicaConmovementinPsychology
• “ManyLabs”&Reproducibilityprojects• Pre-registra:onofmaterials&analyses
Part 3: Replica7on research in SLA
Commentaries and calls for replica7on
46publishedcommentaries&callsforreplicaConinL2researchSantos1989…Polio&Gass1997…Porte2012…Vandergirf&Cross2017
23+fromotherdisciplines:Psychology,EducaCon,Sociology,Business,MarkeCng,OrganisaConScience
ReplicaCons…lackingpresCge,originality,orexcitement
(Makeletal2012,ciCngLyndsay&Ehrenberg,1993;Neuliep&Crandall,1993)OthersynthesesofreplicaCons:forPsychology:Makel,Plucker&Hegarty(2012)forEducaConresearch:Makel&Plucker(2014)
01234567
1973
1975
1977
1979
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
2007
2009
2011
2013
2015
Meanrateof1.5
replicaConsperyear
Norris&Ortega2000?
67replicaConarCcles,in26journals,of70iniCalstudies
Polio&Gass1997?
Howlongdoesittakeforastudytobereplicated?• mean6.64years(sd6.16)• range0-37years
Propor7on of replica7on in journals that published most replica7ons
SSLA TheMLJ FLA LL AP
Meanrate Totalrate
TOTALarKcles 562 1009 1528 855 1030
kreplicaKons 13 8 5 5 4
%oftotal 2.31% 0.79% 0.33% 0.58% 0.39% 0.88% 0.70%1973 – 2015, last complete year before synthesis
Comparedtootherdisciplines?1.07%Psychology,butNB:pre-replicaCon-boom&top100journals1%-3%Business,MarkeCng,CommunicaCon0.13%EducaCon(top100,1938-2014)
acrossthe26journalsthathavepublishedreplicaCons=0.26%
What do we replicate? Study design and findings ParKcipants?TheWEIRDestWesternEducatedIndustrializedRichDemocraCc(Mishraetal.2012)…
speaking(1/4)orlearning(1/2)EnglishBut‘ages’and‘proficiency’notreported(67%,37%studies)(Thomas1994)->replicability??
%ofreplicaKonswhoseiniKalfindingswere…
Weneedtoreplicatenullfindingstoowhenpower(n)islow,“nullfindings”≠“noeffect”
Schmidt,F..&Oh,I.-.S.(2016)TheCrisisofConfidenceinResearchFindingsinPsychology:IsLackofReplicaContheRealProblem?OrIsItSomethingElse?ArchivesofScienFficPsychology4,32–37
Null Nullbuttrend Statsigdifferences
other
3 3 83 7
How much do we change when we replicate?
Numberofchanges
changes=reasonsforreplicaKon%studies
0 331 282 213 94 75 1
Meanperstudy(stdev.) 1.34(1.31)
%studieswithchangesbetweenIandR…
How much do we change when we replicate?
Numberofchanges
changes=reasonsforreplicaKon
changesacknowledged,butnotreasonfor
replicaKon%studies
0 33 451 28 312 21 133 9 94 7 15 1 0
Meanperstudy(stdev.)
1.34(1.31) 0.91(1.04)
%studieswithchangesbetweenIandR…
How much do we change when we replicate?
Numberofchanges
s=reasonsforreplicaKon
sacknowledged,butnotreasonforreplicaKon changesnotacknowledged
byauthors%studies
0 33 45 461 28 31 252 21 13 153 9 9 94 7 1 45 1 0 0Meanperstudy(sd)
1.34(1.31) 0.91(1.04) 1(1.18)
%studieswithchangesbetweenIandR…
norelaKonshiptowhatthereplicaKoncallsitself(e.g.,‘parKalreplicaKon,’just‘replicaKon’,extension,conceptual)
Do “authorship overlaps” relate to whether findings are suppor7ve of ini7al study?
FindingsinrelaKontoiniKalstudy(%ofreplicaKonstudies)
Authoroverlap?(%totalRstudies)
%notorparKallynotsupporKve
%parKallyorverysupporKve
None(69%) 37 58Some/all(31%) 10 91
Overall 28 68
%“supporCveofIniCal”,asfuncConofauthorshipindependence
How do rela7ons between authorship teams relate to the nature of the conclusions?
FindingsinrelaKontoiniKalstudy(%ofstudies)
Authoroverlap?
(Kstudiestotal)
%notsupporKve
%parKallynot
supporKve%parKallysupporKve
%verysupporKve
%notreported
Nooverlap… (46) 20 17 30 28 4Someoverlap… (21) 5 5 43 48 0Overallk (67) 15 13 34 34 3
%headlinefindingsaccordingtoauthorshipindependencePsychology:92%supporCveifoverlapinauthorship65%supporCveifnooverlapEducaKon:89%supporCvewithoverlap,insamepublicaCon71%withoverlap,innewpublicaCon54%whennoauthoroverlap
vs.SLA:91%58%
QuesConableResearchPracCces(bias,p-hacking)
OrMaterialsavailabilityandfidelitytoiniCalstudy
??
How do studies use the findings and the data of the Ini7al study? Only6%ofReplicaConsprovidedIniCalstudy’seffectsize.
DataSharing!Opendataisassociatedwithstrengthofevidence
andqualityofreporCng
WichertsJM,BakkerM,MolenaarD(2011)WillingnesstoShareResearchDataIsRelatedtotheStrengthoftheEvidenceandtheQualityofReporCngof
StaCsCcalResults.PLoSONE6(11):e26828
Only6%ofReplicaConsusedIniCal’srawdatainanewanalysis
How do Replicators compare their findings to Ini7al Study?
%narraKveComparisons
%menKonedoriginalfindings
%dichotomousdecisionfrom
NullHypothesisSignificance
Test
%descripKvestaKsKcs %unclear %effectsizes
93 90 84 34 6 1
%ofReplicaKonStudiesthatcomparedtheirfindingstoIniKalStudyusing…
How do Replicators compare their findings to Ini7al Study?
LearningfromtheReproducibilityProjectinPsychology:masscoverage:
“inonly36%ofthestudiesweretheoriginalresultsreplicated”[becausep>0.05]
BUT…77%ofreplicaConeffectsizeswerewithina95%predicConintervaloforiginaleffectsize
(PaCletal.2016)
Transparency: How do replicators get hold of materials?
AvailabilityinIniKalStudy
Howpassedontoreplicator?%
%inar:cle
%passedoninprivate
%sharedauthorship %unclear
None(k=12) n/a 25 33 42parCalexamples(k=29) 54 12 31 4onefullinstrument,but
notall(k=26) 77 4 19 0
Only4%ofIniKalstudieshadmaterialsinIRISorsomeotheropenaccess
1) Replica7on: cultural changes in academia
• ReplicaConnotaneasyroute;it’sanessenFalroute
• FacilitatestudentapprenCceshipmodel
“alotishiddenbehindthefinal[publishedarCcle].”
Roxana,areplicator,SLAgradstudent/traineeteacherfromVasquez&Harvey2010p.436
Part 3: Replica7on
1)only4/67triedtoreplicateastudythathadnullfindings
2)FearthatreplicaConwon’t‘replicatefindingsoforiginal’Howtomake‘nullfindings’morepublishable?Transparentreplica:one.g.viapre-registra:onshouldhelpMethodsfullyreviewedandapproved->InPrincipleAcceptanceIPATHENdatacollectedReviewerscannot‘argueout’onbasisofmethodologicalflawPerspecFvesinPsychologicalScience,Cortex,JournalofChildLanguage
2) Effects of publica7on bias on replica7on effort Part 3: Replica7on
Part 3: Example of a mul7-site replica7on Morgan-Short, Marsden, Heil, et al. (manuscript)
h]ps://osf.io/
Part 3: Replica7on research in SLA
The study: Is comprehension affected by acending to gramma7cal or lexical forms?
Elpuebloazteca,comopuebloprimiCvo,podíaencontrarunasoluciónalosproblemaspresentadosporlasfuerzasdela(La1)naturaleza.Dabamuchaimportanciaasureligión.EnellasuDiosprincipalytodopoderosoeraTonaCuh(elsol)(Sol1).Seloadmirómucho.TonaCuhteníalasbondadesylosdefectosdeloshumanos,peroconungranpodersobrenatural.Segúnla(La2)religiónazteca,elsol(Sol2)TonaCuhnecesitabaqueloalimentaran(-n1)conunasustancia...
Part 3: Example of mul7-site replica7on in SLA. Morgan-Short, Marsden, Heil et al.
THENcomprehensionmeasuredby10mulCplechoicequesCons
ORhearthesameintheoralmodality(andspotforms)
EITHERreadawri]entext(andspotforms):
fromLeowetal.(2008)
AvailableonIRISwww.iris-database.org
Par7cipants: Sites by Modality
704parCcipantsacross7researchsites
• 4sitesranlisteningversion• UniversityofIllinoisatChicago*• UniversityofOregon• SouthamptonUniversity• KazimierzWielkiUniversity&AdamMickiwiczUniversity
• 3sitesranreadingversion• UniversityofYork*• GeorgetownUniversity• UniversityofSouthCarolina
Part 3: Example of mul7-site replica7on in SLA. Morgan-Short, Marsden, Heil et al.
Demonstra7on of the need for replica7on:
• Clearlyreplicatedfindingsin4sites• Notclearin2sites• Notreplicatedin1site
65
Part 3: Example of mul7-site replica7on in SLA. Morgan-Short, Marsden, Heil et al.
Challenges for mul7-site replica7on
1. Seekingcollaborators2. Parityinproficiencyatdifferentsites(Thomas,1994)
3. CompaCbilityofsofware(EPrime/superlabscripts)4. ResponsibiliCesfordataentry&analysis
weprovideddetailedprotocols
Part 3: Example of mul7-site replica7on in SLA. Morgan-Short, Marsden, Heil et al.
Benefits of mul7-site replica7on!
1. N=7042. Protocolsreadytobeusedagain
FourfurtherreplicaConsongoing:1inChina,1NaCvespeakers,1Heritagelearners,1aferyear
abroad
3. Havingdifferentsitestemperedourclaims:Oneindividualstudycouldhaveconcludedonbasisofsitewithdifference!Similarfindingsacross6sitessuggestsomethingspecialinone‘odd’site=MorereliablereasonforgeneraCngnewhypothesis
67
Part 3: Example of mul7-site replica7on in SLA. Morgan-Short, Marsden, Heil et al.
CONCLUSIONS: Challenges for open methods
and Recommenda7ons
Some challenges for open materials
Concern1:Openmaterialswillencouragebaduseofmaterials
wesCllneedcriCcalthinkingabout…
• purposeofmaterials
• useoradaptaFonofmaterials
• analysisandinterpretaFon
Osborne,R.(2013).Whyopenaccessmakesnosense.InN.Vincent&C.Wickham(Eds.)DebaCngOpenAccess.(pp96-105).London:TheBriCshAcademy.
Concern 2: Reluctance to share “Others might misuse my materials” • PutnotesaboutuseonIRIS• Wegatekeeptoreducebadscience:peerreview
“My next study might be ‘scooped’” • TheexisFngstudyshouldbeopentofullscruCny• Yourplanwon’tbesameasothers’ “I might be proved wrong” • Good!• but…unlikelytobesoclear-cut• ->citaCons!
“I don’t have time” • 15minutesofyourCmevs3yearsofPhDstudent’s!• SharingmagnifiesimpactofCme&publicmoney
“I can’t find my materials or data” • AllthemorereasonforIRIStoexist!
Concluding remark (1) toresearchers:
• Ethics:sharingwithall
• ECc:onestudyindividualismvssyntheFc,acrosscontextsandoverFme(Plonsky2012,Norris&Ortega2005)
• acollecFvemethodologicalmemory
Ratherthan:I’llsharewithyou(theoreFcalally),
butIwon’tsharewiththem
Concluding remark (2)
toyou,futurereviewersandeditorsofjournalsandbooks:• WithoutseeingFULLmaterials,canreviewersproperlyevaluate?
• Withoutopenaccesstomaterialsanddata,canresearchershaveproperly:
Ø builtonpreviousmethodssystemaFcally?Ø comparedtheirdatatopreviousdata?Ø reducedpotenCalbias?(giventhatindependencehelps)
Ask your journal editors…
toincen:viseandrecogniseopenscience
OpenScienceBadges(fromtheCenterforOpenScience)LanguageLearningadoptedtheschemein2015.StudiesinSecondLanguageAcquisiFon,availablefromnowAppliedLinguisFcs,boardapprovedTheModernLanguageJournal,boardapprovedLinguisFcApproachestoBilingualism,pendingLanguage,InteracFonandAcquisiFon,pending
Ø Trofimovich&Ellis(2015)EditorialLanguageLearning.[OpenScienceBadges]
Ø Blohowiak,B.,Cohoon,J.,de-Wit,L..,Farach,F.,Hasselman,F.,&DeHaven,A.(2016).BadgestoacknowledgeopenpracCces.osf.io/tvyxz
Effec7veness of journals recognising open materials with badges
KidwellMC,LazarevićLB,BaranskiE,HardwickeTE,PiechowskiS,FalkenbergL-S,etal.(2016)BadgestoAcknowledgeOpenPracCces:ASimple,Low-Cost,EffecCveMethodforIncreasingTransparency.PLoSBiol14(5):e1002456.doi:10.1371/journal.pbio.1002456
Heh guys, have you seen these on www.iris-database.org ?
Thankyouforlistening
With thanks to: Keycollaborators• SusanGass,Pa�Spinner,DusCnCrowther(JT)
• AlisonMackey(IRIS)• RosMitchell&FlorenceMyles(Learnerlanguagecorporaprojects)
• KaraMorgan-Short,JeanneHeil,DavidAbugaber(ReplicaCon)
• LukePlonsky(IRIS,JT,SPR,ReplicaCon)• SophieThompson(SPR,ReplicaCon)Funders
Phew! I’m glad I went to www.iris-
database.org
Selected references KidwellMC,LazarevićLB,BaranskiE,HardwickeTE,PiechowskiS,FalkenbergL-S,etal.(2016)BadgestoAcknowledgeOpenPracCces:ASimple,Low-Cost,EffecCveMethodforIncreasingTransparency.PLoSBiol14(5):e1002456.doi:10.1371/journal.pbio.1002456
Morgan-Short,K.,Heil,J.,Botero-Moriarty,A.,&Ebert,S.(2012).AllocaConofA]enContoSecondLanguageFormandMeaning.StudiesinSecondLanguageAcquisiFon,34(04),659-685.
OpenScienceCollaboraCon.(2015).EsCmaCngthereproducibilityofpsychologicalscience.Science,349(6251),aac4716.Doi:10.1126/science.aac4716
Plonsky,L.(2012).ReplicaCon,meta-analysis,andgeneralizability.InG.Porte(Ed.),ReplicaFonresearchinappliedlinguisFcs(pp.116-132).NewYork:CambridgeUniversityPress.
Polio,C.,&Gass,S.(1997).ReplicaConandreporCng:Acommentary.StudiesinSecondLanguageAcquisiFon,19(4),499-508.
VanPa]en,B.(1990).A]endingtoformandcontentintheinput:Anexperimentinconsciousness.StudiesinSecondLanguageAcquisiFon,12,287-301.
Wagner,A.B.(2010).OpenaccesscitaConadvantage:Anannotatedbibliography.IssuesinScience,Technology&Librarianship.60:Winter
LouiseCorC-VeerleVandenEynden-LibbyBishop-Ma]hewWoollard-2014ManagingandSharingResearchDataAGuidetoGoodPracCce,Sage
UKDataArchiveMANAGINGANDSHARINGDATAabestpracCceguideforresearchers2ndedh]p://www.admin.ox.ac.uk/media/global/wwwadminoxacuk/localsites/researchdatamanagement/documents/managingsharing.pdf
Klein,R.etal.(2014).InvesCgaCngvariaConinreplicability:A‘‘ManyLabs’’replicaConproject.SocialPsychology45(3),142–152.
Leow,R.P.,Hsieh,H.,&Moreno,N.(2008).A]enContoformandmeaningrevisited.LanguageLearning,58(3),665-695.
Marsden,E.,Mackey,A.,Plonsky,L.(2016)TheIRISrepository:AdvancingresearchpracCceandmethodology.InMackey,A.&MarsdenE.(Eds.)AdvancingMethodologyandPracFce:TheIRISrepositoryofInstrumentsforResearchintoSecondLanguages.Taylor&Francis.
Makel,M.&Plucker,J(2014).Factsaremoreimportantthannovelty:ReplicaConintheEducaConsciences.EducaFonalResearcher,43(6),304–316.
Makel,M.,Plucker,J.&Hegarty,B.(2012).ReplicaConsinpsychologyresearch:Howofendotheyreallyoccur?PerspecFvesinPsychologicalScience,7(6)537–542.
Wicherts,J.,Bakker,M.,Molenaar,D.(2011).WillingnesstoshareresearchdataisrelatedtothestrengthoftheevidenceandthequalityofreporCngofstaCsCcalresults.PLoSOne,6,11,e26828