using phones for & of oral skills development

Using mobile phones for Using mobile phones for the assessment of and the assessment of and

for oral skills for oral skills development in development in

secondary education.secondary education.

Ton Koenraad

Hogeschool Utrecht, University of Applied Sciences, Faculty of Education

TELLConsult

GLoCALL 2010

Anglia Network Europe www.anglianetwork.eu

Step by step

- 28 January- 15 April- 20 May- 24 June- Speaking Tests

Examinations

English for Kids FoundationEnglish for Kids Foundation

““Voice for children”Voice for children”

A, not -for-profit organisationA, not -for-profit organisation

EFKF projects:EFKF projects:

India

NicaraguaVery Young

Learners

South Africa Elandsdoorn

Kenia Teacher support

SurinamEDS

GobabisNamibia

Cambodia

Why this Project?Why this Project?

Anglia: flexibility / assessment of

- sharing innovative speaking practice materials - delivery of formal language assessments

English for Kids: mobile as infrastructure, & schools assessment for:

washback effect of testing

Issues to be researched Issues to be researched

general organisationtask and test design, teacher competences face & content validity aspectssystem usability

Research design / instrument development

How it worksHow it works

Teachers:

Personal online workspace to set spoken

questions/tasks.

Questions are easy to set online by using a

microphone or uploading audio files.

Audio player allows teachers to review spoken

work and leave feedback.

Over time teachers establish a digital portfolio

of student work.12

How it worksHow it works

Students:Connect using:

Mobile phonesiPod TouchSkypeLandlineComputer

Access spoken exercises & leave voice responses.

Connect with other students for role play.

Personal online workspace to store work, listen & receive

feedback.

Listen to exemplar and sample questions posted by teachers.13

PilotPilot

2 secondary ed. EFL teachers2 Anglia member schoolsVolunteer students (n= 20) Assessment: asynchronous, interview

formatOral presentation skills

Pre-Questionnaire: Pre-Questionnaire: Learner Profiles Learner Profiles (2)(2)

Aspect Group A Group B

Speaking Skills:(Self reported)

Fairly good Good

Like speaking in class

So, so Definitely

Actual speaking hardly Very frequently

Telecollaboration at school

n/a Slightly more than once

Tel. Experiencein projects n/a

Very occasionally

Tel. ExperienceIRL

seldom seldom

Pre-Questionnaire: Pre-Questionnaire: Learner perceptions: Learner perceptions:

L2 in class & IRLL2 in class & IRLAspect Group A Group B

L2 in lessons Once in 3 lessons

Practically every lesson

Answers /Discussion Only now and then

(very) frequently

Pairwork Hardly ever sometimes

Use of English IRL 1. Chat in games

2. Holidays3. Skype

1. Holidays,2. Chat in

games3. Txt chat

Assessment of Oral skillsAssessment of Oral skills

Aspect Group A Group B

Assessed Tasks in 2010

All: 1 All: 3.5

School reports: Oral skills included?

35% ? 70% ?

Expectations:Is tele-testing valid?

Yes: 35%?: 55%

Yes: 60%?: 40 %

Post: Valid Yes: 60% 40%

TopicsTopics

Introduction/warming up

Your holiday this yearA good schoolSocial networksA million eurosThe climate

Post-Questionnaire Post-Questionnaire (1)(1)

Aspect Group A Group B

Technically OK? Yes So, so

Read Instructions

Yes Sure

Different from expectation

Yes Yes

Questions: complexity,

speed, loudness,

Hard to remember,

Speed bit fast,Not loud enough

Idem, butSpeed OK

Pupils’ CommentsPupils’ Comments

Time constraint is unnaturalWas interrupted: new session neededRetries: worries about costsQuestions could be louderQuestions: peer voice is more inviting

Teacher PerceptionsTeacher Perceptions

System usability-System: fairly user-friendly 

Topics- More alignment with pupils’ interests might be needed  

Validity-Content measured in time is less suitable as criterion when no interaction is possible- computer-based testing, as such, not perceived as unusual or unfriendly.- Retry option?

Reviewing work onlineReviewing work online

22

Teacher PerceptionsTeacher Perceptions

Teacher competencies:- Knowledge of CEFR -> difficult, training / practice needed- Evaluation categories (content, accuracy, complexity, fluency.) useful; scoring doable in one session.But …would prefer a grading scale that results in a CEF-level: better match to Dutch current grade system

Teacher perceptionsTeacher perceptionsImplementation

- Use as practice material and preparation for speaking test. Actual testing: rather face-to-face

- Gives students the opportunity to practice outside the classroom, extra practice

- Chances for providing individualised feedback

- May help reduce anxiety of insecure & shy students:

Conclusions / next stepsConclusions / next steps

Improve briefing (demo, online tutorial) + raise awareness implications of re-tries

Redesign questions (granularity)

Try-out alternatives:- system access (landlines, computers)- content aligned to syllabus / textbook

ConclusionsConclusions

Pupils, teachers & management have concerns about costs

Also found in other projects:

[…] cost to the end user is a major consideration and can be a barrier to successful uptake when using mobile devices (Kukulska-Hulme & Shield, 2007)

Thank you for Thank you for your attention.your attention.

Comments, Questions?Comments, Questions?

www.koenraad.info

Ton.Koenraad@gmail.com

AnnexAnnex

Literature SelectionResearch data

Learnosity

Literature selectionLiterature selection Collins, T. (2005). ‘English Class on the air: Mobile

Language Learning with CellPhones’, Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologies (ICALT’05).

http://csdl2.computer.org/comp/proceedings/icalt/2005/2338/00/23380402.pdf

Fallahkair, S., Pemberton, L. & Griffiths, R. 2007. ‘Development of a cross-platform ubiquitous language learning service via mobile phone and interactive television’. Journal of Computer Assisted Learning, 23 (4), 312-325.

Kukulska-Hulme, Agnes; Sharples, Mike; Milrad, Marcelo; Arnedillo-Sanchez, Inmaculada and Vavoula, Giasemi (2009). Innovation in Mobile Learning: A European Perspective. International Journal of Mobile and Blended Learning, 1(1), pp. 13–35.

Literature selection Literature selection (2)(2)

Naismith, L., Lonsdale, P., Vavoula, G. & Sharples, M. (2004). ‘Literature Review in Mobile Technologies and Learning’. FutureLab Report 11. http://www.futurelab.org.uk/resources/documents/lit_reviews/Mobile_Review.pdf.

Sharples, M. (Ed.). (2006). Big issues in mobile learning. Report of a workshop by the Kaleidoscope Network of Excellence Mobile Learning Initiative, University of Nottingham, UK.

Shield, Lesley and Kukulska-Hulme, Agnes (2008). Special issue of ReCALL on Mobile Assisted Language Learning. Cambridge University Press.

Thornton, P. & Houser, C. (2005). ‘Using mobile phones in English education in Japan’. Journal of Computer Assisted Learning, 21, (3): 217-228.

Pre-Questionnaire: Pre-Questionnaire: Learner Profiles Learner Profiles (1)(1)

Aspect Group A Group B

Group size 12 8

Girls 5 4

Boys 7 4

Years of English 3 3

Positive Attitude toLearning English

2.42STD: 0.51

3.11STD: 0.33

Average scoreat Secondary

6.37STD: 0.9

7.6STD: 1.7

Speaking Skills:(Self reported)

6.58STD: 0.9

7.55STD 0.68

Pre-Questionnaire: Pre-Questionnaire: Learner Profiles Learner Profiles (2)(2)

Aspect Group A Group B

Like speaking in class

2.67STD: 0.78

3.780.44

Actual speaking 1.5STD: 0.52

3.56STD: 0.53

Telecollaboration at school

n/a 2.00.0

Tel. Experiencein projects n/a

1.38 STD: 1.0

Tel. ExperienceIRL

1.83STD: 1.19

1.44 STD: 0.73

Pre-Questionnaire: Pre-Questionnaire: Learner perceptions: Oral L2 in Learner perceptions: Oral L2 in classclass

Aspect Group A Group B

L2 in lessons 2.1STD: 0.50

5.00.0

Answers /Discussion 2.2 / 1.6 3.9 / 3.2

Pairwork 1.6 1.9

English IRL 1. Chat in games

2. Holidays3. Skype

1. Holidays,2. Chat in

games3. Txt chat

Assessment of Oral skillsAssessment of Oral skills

Aspect Group A Group B

Assessed Tasks in 2010

All: 1 All: 3.5

Formal reports: Oral skills included?

35% ? 70% ?

Is tele-testing valid? Yes: 35%? : 55%

Yes: 60%?: 40 %

Post: Valid Yes: 60% 40%

Post-Questionnaire Post-Questionnaire (1)(1)

Aspect Group A Group B

Likert scale Disagree 1 Agree 4

Technically OK? 2.43STD: 1.13

1.80.84

Read Instructions

2.8STD: 0.7

3.00.7

Different from expectation

2.7STD: 0.76

2.81.3

Questions: complex, speed,

loudness,

Not loud enoughHard to

remember

idemSpeed OK

Post-Questionnaire Post-Questionnaire (2)(2)

Aspect Group A Group B

Likert scale Disagree 1 Agree 4

Answer time left 3.4STD: 0.5

3.21.1

Expected Mark Just sufficientSTD: 1.4

O.K1.1

Problem Topics Networks; 1M Euros Good School; Climate

Fun to do 2.8STD: 0.4

2.40.9

More pleasant without teacher

2.07STD: 0.6

2.01.0

www.learnosity.comTwitter @learnosity

Speak. Listen. Learn.