evaluating elearning

M-ICTE2009 Conference, Lisbon, 22-24 April

1

José Reis Lagarto

evaluating elearning

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto 2

Who am I… ?Professor at Portuguese Catholic UniversityPedagogical coordinator of the elearning master degree

“ICT and Education”related to Teachers Training

jlagarto@ucp.pt

Initial question… what really eLearning is?

3

Distance Education?

knowledge mangement?

Mobile learning ?

Face to face learning?

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

4

eLearning is the use of Internet technologies to distribute a set of solutions that allow the increasing of knowledge and abilities, either at individual level, either at institutional level.

Rosenberg (2007)

Formal training Knowledge management

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

evaluating?Why?

5

Are the learning goals achieved ?

Are there areas where improvementis possible?Can we be

obtain a better position in the market?

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

6

Evaluatingwhat?

Micro vision

Macrovision

For a component of a course

For a specific course

Curriculum/program

For a organization /SME

For a particular activity sector

On a social levelm- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

7

O que avaliar ?

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

Evaluatingwhat?

Micro vision

Macrovision

For a component of a course

For a specific course

Curriculum/program

For a organization /SME

For a particular activity sector

On a social level

8

evaluating

When ?

Before training

During trainingdelivery

After Training

Are there conditions for sucess ?

Are the training strategy And materials adequate ?

The impact of training in the organization

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

Paulsen & Vieira (2006) had indicated some criteria to judge the global quality of an elearning program.

Credibility of the training institutionGuarantee of quality criteria (standards)Previous information and counselingCosts of the courseSupport to e-student (technical and counseling)Individual preferences (learning styles)

9m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

10

1 - satisfaction -the students had liked the training?- the drop out rate was low?

2 - learning -what had they learned?- had the aims of learning been reached?

3 - abilities - Are the students applying the new abilities in its workplace?

4 - results - Is the organization improving its outcomes by the training of its workforce?

- how much earns the institution with the training of its collaborators?

The Kirkpatrick frameworkThe Kirkpatrick framework

levels

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

11

student’s satisfaction

level 1

Coordination of the course

Contents related to personal objectives

Technology (LMS …)

Tutorial model

Pedagogical model / training design

evaluation criteria

?

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

---

12

Evaluation tools

Questionnaires

Interviews (singles or in focus group)

Forums

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

student’s satisfaction

level 1

13

Satisfaction questionnaire (example)

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

14

Results of satisfaction quiz– Research Methodologies in Education - 2007/2008

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

15

• Diagnosis• Lists of confidence

• minimum technical characteristics • Scales of abilities

•formal abilities (academic or professional qualification) •specific abilities in some areas (previous linguistics abilities and digital literacy)

Learning outcomes

Level 2

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

Mestrado em Ciências da Educação, especialização em Informática Educacional CONDIÇÕES DE ACESSO Edição 2007/2009 a) Habilitaçõesa) Habilitações: Pode candidatar-se ao curso de mestrado em Ciências da Educação (Especialização em Informática educacional (e-learning))

os candidatos com o grau de licenciado. b) Competências de utilização ao nível da informática: b) Competências de utilização ao nível da informática: O curso de informática educacional tem como pressuposto que os alunos detenham competências médias no domínio da

utilização dos softwares de utilização corrente (processador de texto, folha de cálculo, apresentações) e da utilização da Internet e da Web.

Deste modo, a frequência do curso está dependente da existência dessas mesmas competências. Como a UCP não fará verificação dos pré-requisitos, o candidato será responsável pelas suas capacidades no domínio das ferramentas informáticas. No anexo I define-se o perfil desejável de competências informáticas de entrada no Mestrado.

c) Competências linguísticas c) Competências linguísticas Embora a língua oficial do Mestrado seja o Português, os candidatos deverão possuir competências de leitura e interpretação

de textos escritos em Inglês, dado que alguns materiais pedagógicos distribuídos ou a pesquisar pelos alunos poderão estar nesse idioma.

d) Requisitos tecnológicosRequisitos tecnológicos Para a frequência do curso, os alunos devem dispor, em permanência, de um computador multimédia e acesso à Internet,

preferencialmente em banda larga. Anexo I

Pré-requisitos de competências informáticas Pré-requisitos de competências informáticas para acpara acesso ao Mestrado em Informática Educacional Consideram-se pré-requisitos informáticos gerais:

16

Academic qualification

Digital literacy

Adequate knowledge foreign language

Hardware requirements

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

Scale of abilities

17

• formative• self evaluation• hetero-evaluation

• peers• tutor/professor/system

• sumative• self evaluation• hetero-evaluation

• peers• tutor/professor/system

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

Learning outcomes

Level 2

18

online self evaluation tests

assignments

Project work

Fóruns participation

participation in sincronous

sessions

• Setting of a minimum number of participation

• Setting of evaluation criteria of each message

• Interaction• Confrontation• Lidership• Different

approaches• …

• Number of messages posted per day

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

Learning outcomes

Level 2

19

Face to face evaluation(quality recognition)

• Testes online tests

-Múltiple choice- true - false- fill in gaps- de correspondência- …

• written testes escritos

• projects discussion

• Labs work

• Chats’ participation

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

Learning outcomes

Level 2

20

evaluating learning outcomesevaluating learning outcomes

on-line individual works

Particpation in chats

forum participation

on-line working groups

face to face evaluation- tests / projects/ …

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

21

evaluation final evaluation (%)

Face to face(teste ou work debate) 50 a 60Assignments / required along the training period 15 a 20

Colaborative work - written documents sumbited to tutor.

20 a 30

Participation in online sincronous activities (according teacher strategies)

5

% of different evaluation components in a single Curricular Unit

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

22

evaluation of abilities

level3

external evaluation

department director opinion

self evaluationLess complaints

Less acidents

Raise off sales

……

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

23

It is possible to evaluate at this level , if

The training is a consequence of clear needsThe learner has strong motivation

There are logistics conditions allowing th evidence of new skills

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

evaluation of abilities level3

24

Results inside

organisation

Nível 4

Level 4 measures the results of learning at the institutional level

• The training contributes to the objectives of the business?

• The training responds efficiently to the needs identified?

• What level of investment in training is recovered?

ROIm- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

25

Evaluating e-learning

Evaluating learning

Evaluating communication tools

Conclusion:Knowledge

Skills

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

26

Thanks for your attention!

jlagarto@ucp.pt

m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto