Evaluating the Impact of Educational

Technology

Erno Lehtinen

University of Turku

Finland

European Association for Research on Learning and Instruction (EARLI)

New challenges of learning - are the schools prepared?

New challenges Managing complex, ill-

defined problems Managing rapid change Continuous surpassing of

existing expertise Learning socially shared

and distributed expertise Effective use of

cultural artifacts

Traditional schooling Dealing with well-defined

pieces of knowledge Conveying established

traditions of knowledge Fulfilling external

standards Cultivating individual

minds Emphasis of pure mental

operations

Self-evident benefits of information and communication technology

the power of multimedia interactivity possibilities for multiple representations possibilities to simulate real phenomena world wide access to information tools for synchronous and asynchronous

communication How real are these (technology-driven) benefits?

Desires for using ICT to improve learning: Theory-driven arguments

Technology as inspiration for advancement of the theories and models of learning applications of constructivist epistemology new notions of collaborative learning communities for progressive inquiry and knowledge

building authentic and anchored learning from knowledge acquisition to participation metaphor

of learning

Promises of ICT: results of the experimental research

Reviews and meta-analyses of more than 1500 experiments show that: ICT students learn more ICT supports social interaction ICT improves student motivation but also

causes motivational problems the quality of learning depends on the type of

ICT application

Promises of ICT: Experiences in the innovative projects

Higher order learning in traditional school tasks complex problem solving meta-cognitive skills

Results going beyond the aims of traditional education: communities of distributed expertise new media skills active participation

ICT and the quality of learning in ordinary classrooms

A few very good spontaneous examples Very much superficial quasi-activities In many technologically well equipped

schools ICT has only occasionally been applied in teaching and learning

Due to negative side effects many teachers do not use Internet

How to explain this contradiction between different results?

Critical reflection of the experimental evidence

Publication bias: only articles with positive effects are submitted and accepted

“Unfair” experimental designs Anatomy of the experimental evidence

the size of experimental groups the duration of experiments

Analysing the impact of ICT: methodological challenges

Why it is so difficult to measure the impact of ICT in education

Moving from ”traditional” to ICT environments: difficulties to define the ”independent variable” (systemic change)

In innovative use of ICT also the learning aims change

New methods are needed: Example CSCL

Collaboration around and through technology

Analysis of document structures Content analysis of CSCL documents Analysis of participation activity Analysis of participation structures

Critical reflection of innovative projects

Marketing of scientific ideas and applications Theoretical concepts have a tendency to turn

into ideological slogans Individual differences have been neglected

New ICT based environments change the motivational and social interpretations

virtual open learning environments are obviously increasing achievement differences between learners

Unrealistic investments in terms of preparation and support of the teaching

A scaling up problem: why ”best practices” do not transfer

Experimental evidence is partly misleading selected teachers, major investments etc.

Shortage of realistic models Shortage of ”killer applications”; ICT based

practices able to solve the hard nuts of teaching and learning

Shortage of models for long term everyday use Direct transfer of practices is not possible – a

transformation an localisation is needed

Preparing students for information society and the responsibility of the educational systems

New literacy for all students – avoiding new inequality

Innovative use of new technologies to improve access to education and quality of learning – not fashionable application of virtual environments at the expense of quality

Stronger teacher involvement in the development of the ICT tools an practices: Teachers must have ownership (eg. learning

objects approach) Innovative use of ICT is not a private

business of isolated teachers but it emphasises the community nature of the schools

The use of ICT in education should not be fully commercialised but a coordinated public-private partnership is needed

The duration of the experiment

Effect sizes in experiments of different duration

1-3

we

ek

s

4-7

w

8-1

1 w

12

-15

w

> 1

5 w

A ssig . 1 A ss ig . 2 A ss ig . 1 5A ssig . 3 A ss ig . 5 A ss ig . 6 A ss ig . 7 A ss ig . 8 A ss ig . 9 A ss ig . 1 0 A ssig . 1 2 A ssig . 1 6 A ssig . 1 7C ourse info A ssig . 1 1

4 72 5T utor 2

5 7T utor 1

6 3 2 1 3R iM i

1 1 3T oN i

5 5 1 0 1 1 0 2M aS a

9 0 1 0 7 1 1 8T e M i

7 1M iR i

5 0 1 4 6 1 6 61 6 5A nL a

6 4 8 3 1 8 38 4 9 3 9 4 9 6 9 7 1 8 0 1 8 4 1 8 51 7 9M iM e

6 8 1 6 91 2 3 1 7 11 3 1M iE l

H aS a

0

20

40

60

80

100

120

140

160

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61

login course create commenting

Next

BackBack