developing data systems- slovenia
TRANSCRIPT
Slovenian Educational Data System ALA Tool
as a School Performance Feedback System
Fifth thematic conference – Education
Governance: The Role of Data, CERI, OECD
dr. Darko Zupanc, National Examination
Centre (NEC), Ljubljana, Slovenia
Tallinn, Estonia, 12-13 February 2015
• Slovenia introduced the external General Matura Examination at the end of
general upper secondary schools - USSs (Gimnazija) in 1995; and the
Vocational Matura for professional and technical schools in 2002
• Matura is a typical high stakes examination
– a final exam at the and of USE
– a candidate in General Matura has to pass three compulsory subjects
(mother tongue, mathematics and first foreign language) plus two subjects
selected by the student
– in Vocational Matura student has to pass two compulsory subjects plus
mathematics or foreign language and one coursework
– candidates’ achievements at Matura are also important for the enrolment
into tertiary education
• The Matura Examination is a ‘Curriculum-Based External Examination
System’ – CBEES (Bishop 1998)
• Collected data have the potential to be used as feedback for improvement in
Slovenian schools
• In 2007 National Examinations Centre (NEC) developed the Assessment
of/for Learning Analytic (ALA) Tool (Zupanc, Urank, Bren 2009).
• In 2012 ALA Tool was upgraded with Value Added measures and options for
criterion-referenced interpretations of external assessment outcomes
• It gives new applicability to Matura results for ongoing analyses of student
assessment data, monitoring and evaluation of the school system at the
national, school and classroom level for several consecutive years (2002–
2014)
• It presents continuous follow-up of schools and classes (teachers) in schools
regarding effectiveness and school improvement on a more empirical, data-
driven basis, i.e. useful Management Tool in Education
• ALA Tool is used for decisions about improvement in teaching and students’
learning, and for a higher level of equity within and between schools
• There are no rewards, sanctions or accountability interventions for schools
Slovenian Assessment of/for Learning Analytic (ALA) Tool
as a School Performance Feedback System (SPFS)
ALA Tool is designed for users at different levels
• Subject teachers’ groups and individual teachers [classroom level]
– Improvement for future instructions
– Analyses as secondary level of formative assessment
• Principals and school management in 80 Gimnazija with General Matura and
150 professional and technical USSs with Vocational Matura [school level]
– self evaluation teams for quality assurance
– some schools present the analyses to the parents and
– some schools to the school boards
• Subject testing committees in NEC, curriculum developers, other subject
experts and decision makers [national level]
• Stakeholders have access to data a few weeks after the exam session
results
The main goals of the ALA Tool
• support management at classroom, school and national level for self-
evaluation and decision making
• compare a class or a school to other schools sharing common
characteristics
• compare students' achievements between classes (teachers) within a school
• analyse possible drop-out rates
• recognise longitudinal trends
• analyse teachers‘ internal grading and external marking
• analyse students' selection of different optional subjects in classrooms and
schools and selection of different tiers of compulsory subject exams
• compare outcomes in one subject to another and to overall achievement
• analyse students' outcomes in written and oral assessment, coursework
• analyse educational progress of student with value-added measures
• analyse achieved objectives and performance standards with items analysis
Data in the ALA Tool
• Data are centrally and automatically gathered within secured infrastructure;
servers are behind a firewall and there is a strict control of login accounts
• Our system provide integrated tools to enable easy data comparisons
• It includes the achievement of the thirteen year cohort, from 2002 to 2014
• General Matura: 13 x 2 = 26 exam sessions, 137,500 candidates, or 112,000
of those who sat for all five subjects of General Matura for the first time; data
are gathered for 112,000 x 5 = 560,000 exams
• Vocational Matura: 13 x 3 = 39 exam sessions, 187,000 candidates, or
140,500 of those who sat for all four subjects of Vocational Matura for the first
time; data are gathered for 140,500 x 4 = 562,000 exams
• Data on students’ achievements at the end of schooling include the
achievements of the entire yearly cohort, i.e. 255,500 secondary school
students, this is almost 12.5% of the entire Slovene population
• Altogether more than a million exams (1,120,000) in different Matura
subjects in thirteen years
Assessment of/for learning
Analytic Tool
Distributions of overall achievement - an USS & National
Differences in achievement between a school and
the national average – trends
Differences in average grades - an USS - National
Value
Added
Measure
-
Math for
two
Classes /
Teachers
in one
school
Differences in facility and discrimination indexes
for criterion referenced interpretation
Look at the test item – (Subject content: Equation of the Circle)
Test item P1.07 Solution and mark scheme
Maintenance of the system, support for the users and …
the use is not widespread
• ALA Tool is free of charge for schools
• ALA Tool is maintained by the NEC
• We perform customer support via E-mail, phone and regular contacts with
school staff
• There are user manuals for both ALA Tools with examples on the website
• From time to time we organize workshops for ALA Tool users and
interpretation of data analyses; attendance is voluntary
• Workshops were carried out last year; 93 participants from 63 schools out of
230 schools (80 with General Matura +150 with Vocational Matura)
• The use of the ALA Tool is lower than expected
• Half of the USSs with General Matura use our ALA Tool (40 out of 80)
• Only a quarter of the USSs with Vocational Matura use our Tool (40 out of
150); in the system, there‘s much less information on these schools
Plans for the future
• The biggest challenge for the future is to promote the use of the Tool among
teachers in schools and among other professionals
• We plan to prepare an upgraded version of the ALA Tool – with inclusion of
students‘ SES data in the databases as well as instruments for variability
analyses taking SES into account
• We are going to incorporate data from Statistical Office of the Republic of
Slovenia (SURS) for Contextual Value Added measures
• We plan to develop a similar Tool for primary school and lower secondary
school levels
• We wish to have it adopted as a joint Tool by researchers and practitioners in
the educational effectiveness and improvement processes in education
• We plan to present data with additional new formats, e.g. OD Graphs
