construction of a multi- modal learner corpus of stem ... · modal learner corpus of stem student...

Construction of a multi-Construction of a multi-modal learner corpus of modal learner corpus of STEM student language STEM student language production: A pilot studyproduction: A pilot study

Ralph ROSE and Hinako MASUDARalph ROSE and Hinako MASUDACenter for English Language Education (CELESE)Center for English Language Education (CELESE)

Faculty of Science and EngineeringFaculty of Science and Engineering

Waseda University – Tokyo, JapanWaseda University – Tokyo, Japan

8th International Conference8th International Conferenceon Corpus Linguisticson Corpus Linguistics

2-4 March 20162-4 March 2016Malaga, SpainMalaga, Spain

Acknowledgments: Maiko Serizawa,Acknowledgments: Maiko Serizawa,Akari Nakai, Hideyuki Nagao, & Kenta NozakiAkari Nakai, Hideyuki Nagao, & Kenta Nozaki

2 March 20162 March 2016 CILC @ Malaga, SpainCILC @ Malaga, Spain

22

Learner CorporaLearner Corpora● Corpus definitionCorpus definition

– A "collection of machine-readable authentic texts A "collection of machine-readable authentic texts (including transcripts of spoken data) which is sampled (including transcripts of spoken data) which is sampled to be representative of a particular language or to be representative of a particular language or language variety" (McEnery language variety" (McEnery et al 2005, p. 5)et al 2005, p. 5)

● Learner corpus definitionLearner corpus definition– "[E]lectronic collections of natural or near-natural data "[E]lectronic collections of natural or near-natural data

produced by foreign or second language (L2) learners produced by foreign or second language (L2) learners and assembled according to explicit design criteria." and assembled according to explicit design criteria." (Granger et al 2015, Loc 328)(Granger et al 2015, Loc 328)

● Learner corpora construction boomLearner corpora construction boom


33

Learner CorporaLearner Corpora● LC enable focus on a narrow(er) populationLC enable focus on a narrow(er) population

– STEM students (HKUST: Wen et al 2005)STEM students (HKUST: Wen et al 2005)● LC allow contrastive interlanguage analysis (Granger, LC allow contrastive interlanguage analysis (Granger,

1996)1996)– Learning Prosody in a Foreign Language (LeaP) Learning Prosody in a Foreign Language (LeaP)

Corpus (Gut 2012): Contrast spoken German across Corpus (Gut 2012): Contrast spoken German across age, language background, etc.age, language background, etc.

– Japanese English as a Foreign Language LearnerJapanese English as a Foreign Language Learner((JEFLLJEFLL) Corpus (Tono 2007): Contrast written English ) Corpus (Tono 2007): Contrast written English across school level, topic, etc.across school level, topic, etc.

http://jefll.corpuscobo.net/


44

L1 and L2 contrastL1 and L2 contrast● Growing need for L1 dataGrowing need for L1 data

– "Very few learner corpora incorporate L1 data as an "Very few learner corpora incorporate L1 data as an integral part of the design. This will become more integral part of the design. This will become more important in future learner corpora projects as we are important in future learner corpora projects as we are beginning to realise the need to identify specific beginning to realise the need to identify specific features of L1-related errors or over/underuse patterns." features of L1-related errors or over/underuse patterns." (Tono 2003: 803)(Tono 2003: 803)

● Corpus Escrito del Español como L2 (CEDEL2; Corpus Escrito del Español como L2 (CEDEL2; Lozano and Mendikoetxea 2013): corpus of L1 English Lozano and Mendikoetxea 2013): corpus of L1 English and L2 Spanish writingand L2 Spanish writing


55

Speech and writing contrastSpeech and writing contrast● Spoken and Written English Corpus of Chinese Spoken and Written English Corpus of Chinese

Learners (SWECCL; Wen et al 2005)Learners (SWECCL; Wen et al 2005)● The Santiago University Learner of English Corpus The Santiago University Learner of English Corpus

(SULEC; (SULEC; http://www.sulec.es/http://www.sulec.es/))● Louvain International Database of Spoken English Louvain International Database of Spoken English

Interlanguage (LINDSEI; Gilquin et al 2010) and Interlanguage (LINDSEI; Gilquin et al 2010) and International Corpus of Learner English (ICLE; International Corpus of Learner English (ICLE; Granger et al 2009) complement each other for Granger et al 2009) complement each other for spoken-written contrastspoken-written contrast

http://www.sulec.es/


66

Writing process dataWriting process data● Written corpus data typically includes end product, or Written corpus data typically includes end product, or

occasionally the drafts (cf., Mäkinen and Hiltunen occasionally the drafts (cf., Mäkinen and Hiltunen 2014, Kreyer 2014).2014, Kreyer 2014).

● There is a growing research interest in fine-grained There is a growing research interest in fine-grained observations of the writing process.observations of the writing process.– Keystroke logging and language productionKeystroke logging and language production (Sullivan (Sullivan

and Lindgren 2006)and Lindgren 2006)– Inputlog tool for keystroke logging in Word (Leijten and Inputlog tool for keystroke logging in Word (Leijten and

Van Waes 2013; Van Waes 2013; http://www.inputlog.nethttp://www.inputlog.net))– Analysis of pauses during writing (Braaksma et al 2010, Analysis of pauses during writing (Braaksma et al 2010,

Hoàng 2015)Hoàng 2015)

http://www.inputlog.net/


77

A corpus for our needsA corpus for our needs● Desired:Desired:

– Constructed from production by English learners who Constructed from production by English learners who are science, technology, engineering, and mathematics are science, technology, engineering, and mathematics (STEM) students(STEM) students

– Contains samples of the functional uses of language Contains samples of the functional uses of language our students expect to use (research-oriented)our students expect to use (research-oriented)

– Includes both first and second language dataIncludes both first and second language data– Includes both speech and writingIncludes both speech and writing– Writing data shows writing processWriting data shows writing process– Is availableIs available

● None found...None found...


88

SELCor ObjectivesSELCor Objectives● Create a resource to answer several research Create a resource to answer several research

questionsquestions– What kind of linguistic patterns do STEM learners What kind of linguistic patterns do STEM learners

(mis)use for certain communicative functions?(mis)use for certain communicative functions?– What relationships can be observed between STEM What relationships can be observed between STEM

learners' writing and speech productive behaviors?learners' writing and speech productive behaviors?– Which aspects of the learners' L1 behavior are Which aspects of the learners' L1 behavior are

predictive/not predictive of their L2 behavior?predictive/not predictive of their L2 behavior?– What kind of relationship can be observed between use What kind of relationship can be observed between use

of linguistic patterns and STEM learners' proficiency?of linguistic patterns and STEM learners' proficiency?– How does the nature of the speaking tasks influence How does the nature of the speaking tasks influence

STEM learners' fluency?STEM learners' fluency?


99

SELCor ObjectivesSELCor Objectives● Create a resource that is useful to our teaching staffCreate a resource that is useful to our teaching staff

– Answering their own specific pedagogical questions Answering their own specific pedagogical questions (e.g., do STEM students have difficulty with a certain (e.g., do STEM students have difficulty with a certain English phoneme, stress pattern, lexical item, English phoneme, stress pattern, lexical item, grammatical structure, or communicative function?)grammatical structure, or communicative function?)

– Examining how STEM students approach the tasksExamining how STEM students approach the tasks– Providing sample materials to use in instructionProviding sample materials to use in instruction

● Create a resource that might be useful outside our Create a resource that might be useful outside our institutioninstitution– Other corpus researchersOther corpus researchers– ESP practitionersESP practitioners


1010

SELCor designSELCor design● ParticipantsParticipants

– Undergraduate and graduate students in the Faculty of Undergraduate and graduate students in the Faculty of Science and Engineering at Waseda UniversityScience and Engineering at Waseda University

– Recruited through the Waseda part-time job list Recruited through the Waseda part-time job list (commonly used by researchers within Waseda for (commonly used by researchers within Waseda for recruiting experimental participants)recruiting experimental participants)

● Demographic infoDemographic info– Personal: age, gender, academic orientation, dominant Personal: age, gender, academic orientation, dominant

hand, hearing difficultyhand, hearing difficulty– Linguistic: native language, other languages, English Linguistic: native language, other languages, English

test results (e.g., TOEIC, TOEFL), living abroad test results (e.g., TOEIC, TOEFL), living abroad experienceexperience


1111

SELCor designSELCor design● Japanese (L1)Japanese (L1)

– Reading aloud (Reading aloud (「狼と「狼と

男」男」))

● Participants spoke for Participants spoke for about two mins per task.about two mins per task.

● Speech was recorded in Speech was recorded in sound-attenuated room.sound-attenuated room.

● 15 min. writing task15 min. writing task– Expository or Expository or

argumentativeargumentative– Keylog info captured via Keylog info captured via

Java applicationJava application

● English (L2)English (L2)– Reading aloud ("The Reading aloud ("The

boy who cried wolf")boy who cried wolf")– Picture descriptionPicture description– Diapix task (spot the Diapix task (spot the

difference)difference)– Map task (giving Map task (giving

directions)directions)– Topic narrativeTopic narrative– Problem-solvingProblem-solving


1212

SELCor designSELCor design● Speech: Transcripts were created and annotated by Speech: Transcripts were created and annotated by

four native speakers of Japanese.four native speakers of Japanese.– All fully spoken wordsAll fully spoken words– All clipped wordsAll clipped words– Utterance unitsUtterance units– Filled pauses (Filled pauses (umum//uhuh))– Use of Japanese (L1)Use of Japanese (L1)– Non-linguistic data (laughter, ingresses, etc.)Non-linguistic data (laughter, ingresses, etc.)– Each recording was transcribed by one individual. At Each recording was transcribed by one individual. At

present, transcripts have not been cross-validated.present, transcripts have not been cross-validated.● Writing: Keylog info on inserts, removes, and cursor Writing: Keylog info on inserts, removes, and cursor

movements saved in XML formatmovements saved in XML format


1313

Results: L1-L2 contrastResults: L1-L2 contrast

t(11) = 3.9, p<0.01t(11) = 3.9, p<0.01RR22 = 0.35 = 0.35

t(11) = 2.2, p=0.05t(11) = 2.2, p=0.05RR22 = 0.07 = 0.07

t(11) = 2.4, p<0.05t(11) = 2.4, p<0.05RR22 = 0.09 = 0.09

Statistical tests with Statistical tests with lmelme in in RR using using languagelanguage as fixed and as fixed and participantparticipant as random factors as random factors

Participants’ L2 speech behavior patterns after their L1 speech Participants’ L2 speech behavior patterns after their L1 speech behavior. This is consistent with previous findings for unplanned behavior. This is consistent with previous findings for unplanned speech (Derwing et al 2009, Cox and Baker-Smemoe 2012, De Jong speech (Derwing et al 2009, Cox and Baker-Smemoe 2012, De Jong et al 2015, Rose 2015)et al 2015, Rose 2015)

Fluency measures observed in reading aloudFluency measures observed in reading aloud


1414

Results: speech-writing contrastResults: speech-writing contrast

t(11) = 5.6, p<0.001t(11) = 5.6, p<0.001RR22 = 0.58 = 0.58

t(11) = 2.4, p<0.05t(11) = 2.4, p<0.05RR22 = 0.30 = 0.30

Relationship between L2 speech rate and writing activityRelationship between L2 speech rate and writing activity

Participants who speak faster show a higher keyboard activity Participants who speak faster show a higher keyboard activity (inserts + deletes) rate but also a larger proportion of inserts to (inserts + deletes) rate but also a larger proportion of inserts to deletes: They type faster but backtrack more.deletes: They type faster but backtrack more.


1515

Results: speech-writing contrastResults: speech-writing contrast

t(11) = 1.8, p=0.1t(11) = 1.8, p=0.1RR22 = 0.18 = 0.18

t(11) = 2.6, p<0.05t(11) = 2.6, p<0.05RR22 = 0.30 = 0.30

Relationship between L2 speech and writing pausesRelationship between L2 speech and writing pauses

Participants who pause (silently) more in speech also pause Participants who pause (silently) more in speech also pause more when writing. Participants who use more when writing. Participants who use umum//uhuh more in more in speech, pause longer when writing.speech, pause longer when writing.


1616

Discussion: SELCor design issuesDiscussion: SELCor design issues● SELCor shows patterns and trends consistent with SELCor shows patterns and trends consistent with

previous observations.previous observations.● SELCor allows observation of intra-learner variation.SELCor allows observation of intra-learner variation.● SELCor is informative on some novel research SELCor is informative on some novel research

questions.questions.– Relationship between on-line speech and writing Relationship between on-line speech and writing

processesprocesses

● Some problems and limitations remainSome problems and limitations remain– Participant interest level in tasks was not high, hence Participant interest level in tasks was not high, hence

concerns about naturalness (cf., Gilquin 2015)concerns about naturalness (cf., Gilquin 2015)– Lack of L1 spontaneous speech sample prevents some Lack of L1 spontaneous speech sample prevents some

desired contrasts.desired contrasts.


1717

SummarySummary● SELCor design is characterized by several key SELCor design is characterized by several key

features.features.– Focuses on STEM learnersFocuses on STEM learners

● As participantsAs participants● With respect to developmental needsWith respect to developmental needs

– Contains L1 data for baseline comparisonsContains L1 data for baseline comparisons– Contains both speech and writing samplesContains both speech and writing samples– Allows intra-speaker comparisonsAllows intra-speaker comparisons


1818

Future workFuture work● Expand corpus with more participantsExpand corpus with more participants● Incorporate non-STEM student data for baseline Incorporate non-STEM student data for baseline

comparisoncomparison● Collaborate with other universities in JapanCollaborate with other universities in Japan● Incorporate other L1-L2 language pairingsIncorporate other L1-L2 language pairings


1919

ReferencesReferencesBraaksma, M., Rijlaarsdam, G., and van den Bergh, H. 2010. Hypertext writing Braaksma, M., Rijlaarsdam, G., and van den Bergh, H. 2010. Hypertext writing versus linear writing: Effects on pause locations and production activities and its versus linear writing: Effects on pause locations and production activities and its relation with text quality. Paper presented at the SIG Writing Conference, Heidelrelation with text quality. Paper presented at the SIG Writing Conference, Heidel--berg, Germany.berg, Germany.

Cox, T. and Baker-Smemoe, W. (2012). The relationship between L1 fluency and Cox, T. and Baker-Smemoe, W. (2012). The relationship between L1 fluency and L2 fluency across different proficiency levels and L1s. Presentation at Workshop L2 fluency across different proficiency levels and L1s. Presentation at Workshop Fluent Speech (Utrecht University, The Netherlands).Fluent Speech (Utrecht University, The Netherlands).

Derwing, T. M., Munro, M. J., Thomson, R. I., & Rossiter, M. J. (2009). The relaDerwing, T. M., Munro, M. J., Thomson, R. I., & Rossiter, M. J. (2009). The rela--tionship between L1 fluency and L2 fluency development. tionship between L1 fluency and L2 fluency development. Studies in Second LanStudies in Second Lan--guage Acquisitionguage Acquisition, 31(4), 533-557., 31(4), 533-557.

De Jong, N., Groenhout, R., Schoonen, R., Hulstijn, J.H. (2015). Second language De Jong, N., Groenhout, R., Schoonen, R., Hulstijn, J.H. (2015). Second language fluency: Speaking style or proficiency? Correcting measures of second language fluency: Speaking style or proficiency? Correcting measures of second language fluency for first language behaviour. fluency for first language behaviour. Applied PsycholinguisticsApplied Psycholinguistics, 36(2): 223-243., 36(2): 223-243.

Gilquin, G. 2015. From design to collection of learner corpora In Granger, Gilquin, Gilquin, G. 2015. From design to collection of learner corpora In Granger, Gilquin, and Meunier (Eds.), pp. 9-34.and Meunier (Eds.), pp. 9-34.

Gilquin, G., De CGilquin, G., De Cock, S., and Granger, S. 2010. ock, S., and Granger, S. 2010. Louvain International Database of Louvain International Database of Spoken English Interlanguage. Handbook and CD-ROMSpoken English Interlanguage. Handbook and CD-ROM. Louvain-la-Neuve: . Louvain-la-Neuve: Presses universitaires de Louvain.Presses universitaires de Louvain.

Granger, Sylviane (ed). 1998. Granger, Sylviane (ed). 1998. Learner English on ComputerLearner English on Computer. London: Addison . London: Addison Wesley Longman Limited.Wesley Longman Limited.

Granger, S., Dagneaux, E. Meunier, F. and Paquot, M. 2009. Granger, S., Dagneaux, E. Meunier, F. and Paquot, M. 2009. International Corpus International Corpus of Learner English. Handbook and CD-ROM. Version 2of Learner English. Handbook and CD-ROM. Version 2. Louvain-la-Neuve: . Louvain-la-Neuve: Presses universitaires de Louvain.Presses universitaires de Louvain.

Granger, Sylviane, Gilquin, Gaëtanelle, and Meunier, Fanny. (2015) Granger, Sylviane, Gilquin, Gaëtanelle, and Meunier, Fanny. (2015) The CamThe Cam--bridge Handbook of Learner Corpus Researchbridge Handbook of Learner Corpus Research. London: Cambridge University . London: Cambridge University PressPress

Gut, Ulrike. 2012. The LeaP corpus: A multilingual corpus of spoken learner GerGut, Ulrike. 2012. The LeaP corpus: A multilingual corpus of spoken learner Ger--man and learner English. In Schmidt, Thomas and Wörner, Kai (eds.), p. 3-23. man and learner English. In Schmidt, Thomas and Wörner, Kai (eds.), p. 3-23. Multilingual Corpora and Multilingual Corpus AnalysisMultilingual Corpora and Multilingual Corpus Analysis. Amsterdam: John Ben. Amsterdam: John Ben--jamins.jamins.

Hoàng Thi Đoan Ha. 2015. Hoàng Thi Đoan Ha. 2015. Metaphorical Language in Second Language Learners’ Metaphorical Language in Second Language Learners’ Essays: Products and ProcessesEssays: Products and Processes. Ph.D. Dissertation. Victoria University of . Ph.D. Dissertation. Victoria University of Wellington, Australia.Wellington, Australia.

Kreyer, Rolf. 2014. “The people on the island Kreyer, Rolf. 2014. “The people on the island stasta stosto steal all the fish“: What we steal all the fish“: What we can learn from deletions in authentic learner texts? In can learn from deletions in authentic learner texts? In Proceedings of the 35th Proceedings of the 35th ICAME ConferenceICAME Conference. Nottingham, p. 56.. Nottingham, p. 56.

Leijten, Mariëlle and Van Waes, Luuk. 2013. Keystroke Logging in Writing ReLeijten, Mariëlle and Van Waes, Luuk. 2013. Keystroke Logging in Writing Re--search: Using Inputlog to Analyze and Visualize Writing Processes. search: Using Inputlog to Analyze and Visualize Writing Processes. Written ComWritten Com--municationmunication, 30: 358-392., 30: 358-392.

Lozano, Cristóbal and Mendikoetxea, Amaya. 2013. Learner corpora and second Lozano, Cristóbal and Mendikoetxea, Amaya. 2013. Learner corpora and second language acquisition: The design and collection of CEDEL2. In A. Díaz-Negrillo, N. language acquisition: The design and collection of CEDEL2. In A. Díaz-Negrillo, N. Ballier, & P. Thompson (Eds.), Ballier, & P. Thompson (Eds.), Automatic Treatment and Analysis of Learner CorAutomatic Treatment and Analysis of Learner Cor--pus Datapus Data (pp. 249-264). Amsterdam/Philadelphia: John Benjamins. (pp. 249-264). Amsterdam/Philadelphia: John Benjamins.

Mäkinen, M. and Hiltunen, T. 2014. Approximating the norm: Exploring EFL stuMäkinen, M. and Hiltunen, T. 2014. Approximating the norm: Exploring EFL stu--dents' use of formulaic expressions in economics papers. In dents' use of formulaic expressions in economics papers. In Proceedings of the Proceedings of the 35th ICAME Conference35th ICAME Conference. Nottingham, p. 133.. Nottingham, p. 133.

McEnery, A., Xiao, R., and Tono, Y. (2005) McEnery, A., Xiao, R., and Tono, Y. (2005) Corpus-Based Language Studies: An Corpus-Based Language Studies: An Advanced Resource BookAdvanced Resource Book. London, U.K.: Routledge.. London, U.K.: Routledge.

Rose, R. (2015). Temporal Variables in First and Second Language Speech and Rose, R. (2015). Temporal Variables in First and Second Language Speech and Perception of Fluency. Perception of Fluency. Proceedings of the 18th International Congress of Phonetic Proceedings of the 18th International Congress of Phonetic SciencesSciences (ICPhS 2015), p. 0405.1-5. the University of Glasgow: Glasgow, UK. (ICPhS 2015), p. 0405.1-5. the University of Glasgow: Glasgow, UK.

Sullivan, K.P.H. and Lindgren, E. (Eds.). 2006. Studies in Writing: Vol. 18. Sullivan, K.P.H. and Lindgren, E. (Eds.). 2006. Studies in Writing: Vol. 18. ComCom--puter Key-Stroke Logging and Writing: Methods and Applicationsputer Key-Stroke Logging and Writing: Methods and Applications. Oxford: Elsevier. . Oxford: Elsevier. Glasgow, UK.Glasgow, UK.

Tono, Yukio. 2003. Learner Corpora: design, development and applications. In D. Tono, Yukio. 2003. Learner Corpora: design, development and applications. In D. Archer, P. Rayson, A. Wilson, & T. McEnery (Eds.), Archer, P. Rayson, A. Wilson, & T. McEnery (Eds.), Proceedings of the 2003 CorProceedings of the 2003 Cor--pus Linguistics Conference UCRELpus Linguistics Conference UCREL. Lancaster University: United Kingdom, . Lancaster University: United Kingdom, UCREL Technical Paper #16.UCREL Technical Paper #16.

Tono, Yukio. (ed.) 2007. Tono, Yukio. (ed.) 2007. NihoNihonjin Chukousei 10,000-nin no Eigo Corpusnjin Chukousei 10,000-nin no Eigo Corpus [JEFLL [JEFLL Corpus: A Corpus of 10,000 Japanese EFL Learners]. Tokyo: Shogakukan.Corpus: A Corpus of 10,000 Japanese EFL Learners]. Tokyo: Shogakukan.

Wen, Q., Wang, L., & Liang, M. 2005. Wen, Q., Wang, L., & Liang, M. 2005. Spoken and written English Corpus of ChiSpoken and written English Corpus of Chi--nese learnersnese learners. Beijing: Foreign Language Teaching and Research Press.. Beijing: Foreign Language Teaching and Research Press.

construction of a multi- modal learner corpus of stem ... · modal learner corpus of stem student...

Documents