Bringing e-assessment to a higher maturity on a national scale
dr. A. Hartoog, SURF, Utrecht
drs. N.R. Bos, Fontys, Eindhoven
Educause october 2011
Program
1 introductionproblems, questions, all somehow related to e-assessment
10 min
2 discussion how is this situation in the US and/or Canada?
15 min
3 elaboration how are we tackling these issues in the Netherlands, on a national scale
15 min
4 questions/answers 10 min
Educause october 20112
context
3Educause october 2011
The Netherlands: - surface area 40.000 km2 (US is 270 times bigger)- population density: 491 people/km2 / (US: 34 people/km2)
- about 60 universities,- about 600.000 students- two handfuls of government funded and commercial
R&D institutes
Problems, questions…(1)
- many students drop out in their first year of study
- and if they don’t, their success rate is less than optimal
- can we improve this situation via (digital) testing?
Educause october 20114
Problems, questions…(2)
- Assessment is a key proces in education
- Testing students properly is tedious and time-consuming
- Teachers in general are no trained specialists in the assessment proces, and are usually already overloaded
- How to improve this situation?
Educause october 20115
Problems, questions…(3)
- Clinical reasoning is a key competence for doctors
- proper assessment in their education is crucial
- given the high number of students in medical schools,
- how on earth do you assess this competency reliably
Educause october 20116
Problems, questions…(4)
- teaching statistics is a key component in many educations
- statistics is often a stumbling block for students
- statistics typically has slow and fast students
- how do you make statistics education, training and assessment more effective and fun for the slow and fast alike
Educause october 20117
Problems, questions…(5)
- apprentices in nursing experience a gap between education and practise, leading to drop-outs
- there are many educations in nursing, with the associated variety in end-terms
- how to tackle this?
Educause october 20118
Discussion
- can e-assessment reduce teacher workload?
- does information technology raise the quality of testing?
- can e-assessment reduce students dropping out?
- is it possible to test complex skills with closed questions?
- how could e-assessment improve statistics education?
- could e-assessment reduce the reality gap in nursing?
- would a national infrastructure for e-assessment be useful?
- how can you stimulate collaboration between educations?
- how do you introduce e-assessment in law education?
all the above on a national scale!
Educause october 20119
So, what we did?
Context:Role of SURF in Higher Education in the Netherlands
Program:Testing and test driven learning
Impact:Things that work, things that don’t
Educause october 201110
Educause october 2011
SURF: ICT innovation by and for higher education and research
Research universities, universities of applied sciences, and research institutions work together in SURF in order to: • improve the quality of research and
education by:• creating pioneering ICT innovations.
For more than twenty years, collaboration within SURF has led to products and services that the participating institutions could not have achieved on their own.
11
between all Dutch institutions for higher education & research
Educause october 2011
42Universities of
applied sciences
Dutch Organisation for Applied Scientific
Research (TNO),Telematics Institute
14Universities
National Library
Smallerinstitutions of
higher education
Netherlands Organisation for
Scientific Research (NWO)
Royal Netherlands Academy of Arts
and Sciences (KNAW)
CommercialR&D institutes
Collaboration=
Success factor!
SURF is a partnership
12
Collaboration within SURF makes it possible to:
• carry out a joint (public/private) survey of what higher education & research requires/will receive
• cluster ICT requirements on the demand side (demand pull)
• arrive at sustainable ICT services suitable for joint use
• be innovative in contracting out services
• encourage the use of innovative services (supply push).
SURF as an innovator
Educause october 2011 13
SURF subjects
- advanced networks
- security/identity management/federation
- collaborative environments
- software procurement/licence management
- digital rights & open access
- open educational resources
- SURF academy
- information architecture
- national information brokerage
Educause october 201114
The SURF method
Implementation
SURF directs and coordinates the implementation of funded projects: • Projects are carried out close to the source
of expertise: at institutions of higher education and in enterprise.
• SURF sets up the programme, invites tenders, ensures independent quality assessment and monitoring, makes project results available.
Educause october 2011 15
Program “Testing and test driven learning”
Problems in higher education in the Netherlands:
- high drop-out rate- high workload on teachers
SURF launched a national program to tackle these problems with e-assessment
Educause october 201116
Purpose of this program
- improving transition from secondary higher education with diagnostic tests and revision material, to be jointly developed by institutions,
- improving students’ progress by means of progress tests,
- reducing the time needed to develop tests by collaboration, i.e. developing collections of digital tests,
- reducing the time taken to mark tests by deploying computer-based testing,
- providing expertise and professional development for instructors in the field of testing.
Educause october 201117
programme composition
the program focusses on three main area’s:
1. enabling institutions to jointly develop computer-based tests and assessments
2. realising an expertise network, aimed at improving quality of testing
3. realising a national technical infrastructure to combine successful national initiatives for computer based testing
Educause october 201118
What kind of projects?bachelor nursing:
- “body of knowledge”
medicine:- interuniversity progress test medicine (2)- testing of clinical reasoning- testing with 3D images (research)
math and statistics- foundation of national test database- research into effects of progresstesting- test-based learning (adaptive testing) (2)
teacher education- research into testing abilities of teacher educators- research into methods for portfolio assessments
-> and: new tender by 1 november
Educause october 201119
Math & Statistics projects
- Stumbling block- Faster and slower students- Based on earlier project “Mathematics Garden”
-> computer adaptive monitoring
- Students: can practicecan adjust their learning process- Reduction in drop out
- Teachers: can adjust their teaching process- Reduction in workload
University of Amsterdam, University of Utrecht, Twente University
Medical projects
- Low dropout rates students but high workload teachers (doctors)
- Development of questions on a national scale- Sharing these in national databases
- Different goals:- Measuring progress- Measuring Clinical Reasoning (complex skills)- Improvement gap practice/ theory
- Involvement: all nursing schools, all medical schools
Research projects
3D Testing (CT/MRI)
The project aims at improving the psychometric quality and validity of assessment of 3D images. Also gain insight into the cognitive processes required in the interpretation of 3D images
Progress testing and study success- First year economic students will get weekly
progress tests- Will this increase [retention rates]?- Does this also effect workload for teachers, costs of
assessment, quality of testing?
Network of expertise
Fysically:- a Special Interest Group of inspired specialist- seminars, schools, master classes (SURF Academy)- small scale practical research projects
Virtually:web portal that acts as a centre of expertise:- good practises- white papers- links to literature- ask and find specialists- blogs, discussions forum- agenda
free accessible and maintained by the higher education community
Educause october 201123
National infrastructure
ambition:
investigate, realize and exploit the advantages of a common infrastructure for computer based assessment on a national scale
healthy in all respects, to stand on its own when the program finishes, and external funding is discontinued
Educause october 201124
what’s to gain? (1)
B
user i/f
applications
middleware
operating systems & networks
hardware
A
Ccombi
?
user i/f
applications
middleware
operating systems & networks
hardware
user i/f
applications
middlewareoperating systems & networks
hardware
user i/f
applications
middlewareoperating systems & networks
hardware
sharing systems:
Educause october 201125
what’s to gain? (2)
A
B
Ccombi
?
helpdesk
funct.mgnt
appl.mgnt
technical mgnt
infra.mgnt
helpdesk
funct.mgnt
appl.mgnt
technical mgnt
infra.mgnt
helpdesk
funct.mgnt
appl.mgnt
technical mgnt
infra.mgnt
helpdesk
funct.mgnt
appl.mgnt
technical mgnt
infra.mgnt
sharing system management:
Educause october 201126
lessons learned up to now
- subject domains in education (math, health, law, humaniora, etc) are very different
- educational institutions are very different
- the advantages of sharing an infrastructure are not always obvious
- computer based testing is far from mature
Educause october 201127
so, what do we do
- confine within domains- investigate and explore along the functional lines- make advantages explicit- join already successful initiatives- take small steps, but think big- develop business models
and this might actually work:- example
Educause october 201128
collaboration in medical and nursing education
education in nursing and
medicine
BoKSBody of Knowledge
& Skills
iPTMinter University
Progress Test Medicine
ClinicalReasoning
Testingin 3D
AdaPT
?
?
?
Educause october 201129
Questions, answers
shoot!
30Educause october 2011
Impact, how to measure
- this four-year program is one year underway
- high level goals:- reduce number of students that drop-out- reduce workload on teaching staff- raise quality level of e-assessment
How do you measure this?basic problem: there are many factors (not related to the program) that may have a dominant influence on above subjects.
Educause october 201132
evaluation and impact
- decompose high-level program goals to manageable project proportions
- find ways to measure these parameters, and assess them during the project
- develop a model to arrange the findings in levels of evidence (based on the knowledge-pyramid from Kennisnet)
- inspiration (this is great: it should work)- perception (this actually works for me)- existence (others confirm that it works)- evidence (scientific proof that it works)
Educause october 201133
Evidence:measured benefits
Perception:experienced benefits
Existence:deployment of the project
Inspiration:Idea
34Educause october 2011