advancements in large scale assessments heiko rolke

7/24/2019 Advancements in Large Scale Assessments Heiko Rolke

1/37

Advancements in Large-scale

Assessment

Heiko Rlke & Krisztina TothDIPF - Deutsches Institut fr Internationale

Pdagogische Forschung

Frankfurt am Main

TAO Days | 10.09.2012 Seite 1


2/37

Overview

Trends in LSA

Closer Look at Trends

Example: Making Use of Data

Challenges in LSA

TAO Days| 10.09.2012 Seite 2


3/37

Trends

Main trend: CBA

Sub-trends/effects of CBA:

CAT

towards formative assessment

Complex items

Simulation, interconnection

Interweaving different tests, questionnaires, etc.

Big data



4/37

Trend: CBA

Computer-based Assessment has to serve a purpose

Costs?

Time!

Validity!



5/37

CAT

Also not practiced for its own sake

Not as important in pure summative assessment

But:

Strict time limits (e.g. PIAAC)

Growing demand for formative aspects in summative

assessment



6/37

Complex Items

Closer at reality

Simulation

Most important:

Ability to assess new domains



7/37

Examples of Complex Items

Simulation of an Email / Web Scenario

The test person receives an email and should book

cinema tickets online.

Dynamic Model (MicroDYN)

The test person should explore and master a dynamic

system with input (exogenous) variables influencing

output (endogenous) variables.

Automaton with Finite State Machine (MicroFIN)

The test person should interact with a mobile phone and set time to

summertime.


8/37

Interweaving of Tests and

Questionnaires

Framing

test only if preconditions are fulfilled

Double-check

findings of questionnaire



9/37

Big Data

Log data, not only results

Find out what is going on

Make use of data

E.g. partial scoring

-> Examples from

PIAAC PS-TRE



10/37

Simulation-based assessment

Modern assessments:

real life situations: e.g. web environments

complexity of instrumentvarious ways

Monitoring studentsactions - log files Analysis of test-taking paths is a relatively new field in education

(reason: printed vs. computer-based data collection)

Process data requires to consider and evaluate methods

Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in Large-Scale Assessments10


11/37

Aim

to investigate how log data can be integrated intothe process of

educational assessment and evaluation to support researchers

and practitioners in making use of the data assembled in

computer-based test delivery



12/37

Aim

to investigate how log data can be integrated intothe process of

educational assessment and evaluation to support researchers

and practitioners in making use of the data assembled in

computer-based test delivery



13/37

Hypertext Item

13


14/3714


15/3715

829709 15 {"sender":"15","type":"loading","data":""}

833463 15 {"sender":"15","type":"loaded","data":""}

848153 15 {"sender":"15","type":"user_interaction","":"

\u000d\u000a

\u000d\u000a}":"15","type":"unloaded","data":""}

Example

log


16/37

Promising methods

Statistics and visualisation

Clustering

Classification



17/37

Process Measures


1. Number of page visits

2. Number of different page visits

3. Visit of relevant page

4. Time spent on the relevant page

5. Ratio of time spent on the relevant page

6. Ratio of time spent on the opening screen

7. Completion time


18/37

Statistics

Process measures MeanStandard

Deviation

Number of page visits 5.18 3.13

Number of different page visits 2.38 1.38

Time spent on the relevant page 10.87 9.77

Ratio of time spent on the relevant page .14 .11

Ratio of time spent on the opening screen .64 .18

Completion time

67.74

27.49



19/37

Distribution of different page

visits



20/37

Distribution of different page

visits



21/37

Visualisation

21Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements inLar e-Scale Assessments


22/37

Visualisation

22Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements inLar e-Scale Assessments


23/37

Cluster Analysis

Features Cluster 1 Cluster 2 Cluster 3 Cluster 4

Nr. of page visits

Nr. of different page visitsCompletion time

Relevant page visited (Y/N)

Ratio of time on relevant page

Ratio of time on starting page

Distribution of sequences (%) 23% 16% 34% 27%

Ratio of correct responses

23Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in Large-Scale Assessments


24/37

Cluster Analysis


Nr. of page visits

Nr. of different page visitsCompletion time

Relevant page visited (Y/N)

Ratio of time on relevant page

Ratio of time on starting page


Ratio of correct responses 25.0% 1.7% 92.6% 84.4%



25/37

Cluster Analysis


Nr. of page visits .72 5.41

Nr. of different page visits .36 2.54Completion time 28.38 62.08

Relevant page visited (Y/N) No No

Ratio of time on relevant page 0 0

Ratio of time on starting page .93 .60

Distribution of sequences (%) 23% 16%

Ratio of correct responses 25.0% 1.7%



26/37

Cluster Analysis


Nr. of page visits 4.73 9.57

Nr. of different page visits 2.29 4.53Completion time 59.60 91.60

Relevant page visited (Y/N) Yes Yes

Ratio of time on relevant page .20 .17






27/37

Cluster Analysis


Nr. of page visits 5.41 4.73

Nr. of different page visits 2.54 2.29Completion time 62.08 59.60

Relevant page visited (Y/N) No Yes

Ratio of time on relevant page 0 .20






28/37

Cluster Analysis


Nr. of page visits .72 5.41 4.73 9.57

Nr. of different page visits .36 2.54 2.29 4.53Completion time 28.38 62.08 59.60 91.60

Relevant page visited (Y/N) No No Yes Yes

Ratio of time on relevant page 0 0 .20 .17

Ratio of time on starting page .93 .60 .62 .45





29/37

Summary and Conclusions



30/37

Summary and Conclusions


Nr. of page visits .72 5.41 4.73 9.57

Nr. of different page visits .36 2.54 2.29 4.53Completion time 28.38 62.08 59.60 91.60

Relevant page visited (Y/N) No No Yes Yes

Ratio of time on relevant page 0 0 .20 .17

Ratio of time on starting page .93 .60 .62 .45





31/37

Future work

Pilot studysample size

Validation

Other types of items require new process measures



32/37

32


33/37

Future work

Pilot studysample size

Validation

Other types of items require new process measures

We have a lot of work to do:

Software developer

Test developer

Psychometricans



34/37

(Selected) Challenges

Authoring and Management

Delivery

Re-use and exchange



35/37



Delivery

Re-use and exchange



36/37



Delivery

Re-use and exchange



37/37



Delivery

Re-use and exchange

advancements in large scale assessments heiko rolke

Documents