workshop themes & results from the … vincent-lancrin... · carlos gonzález-sancho oecd...
TRANSCRIPT
WORKSHOP THEMES &
RESULTS FROM THE OECD/CERI
SURVEY
Stéphan Vincent-Lancrin
Carlos González-Sancho
OECD Directorate for Education and Skills
New York City, 30 June – 1 July 2014
International workshop
“Fostering Innovation and Improvement in Education:
the Contribution of Longitudinal Information Systems”
Outline
• Welcome and thanks
• CERI Innovation Strategy for Education and Training
• Why this work and this workshop
• The CERI survey on longitudinal information systems
• Results
• Conclusions and steps forward
innovation in education
Innovation in education
Innovation in
education
Technology
School organisation
System organisation
Research and
Development
General Purpose Technology
Longitudinal information
systems in education
• Next generation = integration of
statistical data systems and learning
management systems with quick
feedback and visualisation tools
(expert systems)
• Building on OECD/SSRC/Stupski
workshop, October 2010
• Engage in the discussion about
“Big Data” in education
Innovation in
education
Technology
School organisation
System organisation
Research and
Development
why this work and this workshop
The opportunities
• Quick feedback to stakeholders:
– A new tool for formative assessment and the design of quick remedial strategies (an expert system for teachers?)
– A tool for cultural change (personalisation?)
• Platforms to network and mobilise practical knowledge: connect teachers and schools with same concerns (learning communities)
• Platforms to post relevant instructional material to support teachers and improve knowledge management – and develop a stronger educational industry
• Platform to improve efficiency and reduce administrative costs (admission process, student transfers, statistical collections, etc.)
• Creation of a better data infrastructure for educational research and the evaluation of educational innovations
• Big Brother and Gattaca: Privacy and safety: who should have access to what? What level of details? For how long? For what purposes?
• Quality and speed of feedback: how to ensure the data are of good quality (accurate, comparable)? That they are relevant?
• Accountability: is it the way to make the systems relevant to people? is there a risk of rejection because of a use for accountability purposes (sanctions/rewards)? Are evaluators (inspectors, etc.) still needed?
• Data driven education: an inappropriate narrowing down of educational objectives?
• How to ensure people use the data that are collected for improvement and innovation?
Several tricky questions
• Get an idea of the state of play of current systems and identify the new horizons for next-generation systems
• Identify the most important policy questions and define key recommendations that would allow one to reap the benefits of these systems
• Identify what role international collaboration and exchange could play in this regard
• (Learn and continue the international conversation)
Objectives of the workshop
The CERI survey on longitudinal information systems
26 systems surveyed in 2010
20 systems surveyed in 2013 outside the US
18 US state-wide systems from 2013 DQC survey
Countries/economies:
• Australia [3], Austria [2], Belgium [2], Brazil [2], Canada [2], Chile [2], Czech Republic, Estonia, France, Germany, Hungary, India, Israel, Italy, Japan, Korea [2], Lithuania, Mexico, Netherlands [3], New Zealand, Norway [2], Portugal, Slovak Republic [2], Slovenia, South Africa, Spain [2], Sweden [2], Turkey, UK [2], US [20]
Survey administered to systems managers in operating agencies
US state-wide systems: equivalent survey items identified in consultation with DQC
64 systems from 32 countries
1. Goals of the system
2. Data model
3. Coverage and frequency of collection
4. Data linkages
5. Quality processes
6. Access and privacy
7. Comparison possibilities
8. Accountability usage
9. Instructional support, networking facilities and PD
10. Other features
Good but not perfect overlap between OECD and DQC surveys: results presented separately
Results are preliminary –updates and more quality checks pending
Survey sections
About 80% in use <10 years, of which 10% <5 years
About 20% in place 10+ years, of which 5% set up 20+ years ago
Systems cover compulsory stages of schooling, but not only…
• About 60% cover also early (i.e. pre-primary) education
• Almost 50% cover either post-secondary or tertiary education
• Less than 10% are HE-systems only
… and up to a third provide comprehensive P-20 coverage
• Mainly found in the US but also in Estonia, Belgium, Slovakia, Lithuania and the Netherlands
Most are public and national or regional/state level, with data on all or a representative sample of the students in the jurisdiction
Most systems are recent, public and cover
K-12 schooling at national/state level
Longitudinal identifiers and linkages
• Schools and students are uniquely identified by most systems
• Fewer systems, especially outside the US, provide teacher and course identifiers
• Systems tend to enable linkages between most of the data elements available for identified entities
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
OECD 2010-13 DQC 2013
School ID Student ID Teacher IDCourse code All 4 Student-Teacher link
Student assessment data
• Still largely focused on conventional attainment and summative performance indicators
• Data on students ‘soft’ and generic skills rarely available
• Formative assessment and item/question level data missing from most systems
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
OECD 2010-13 DQC 2013
Course Grades Graduation
Skills in thinking and creativity Item/exercise/question level data
Formative assessment data
Teacher data
• Information systems with teacher-level data are much more common in the US
• Teacher data mainly focus on credentials, seniority and teaching duties
• Data on teacher evaluation and professional development are absent from most systems
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
OECD 2010-13 DQC 2013
Years of Service Level of education
Subjects taught Professional development
Evaluation
School data
• Virtually all systems provide school admin data and are able to group schools by type/status
• School evaluation data is rarely found outside the US
• Systems could make progress in recording school participation in networks and innovation programmes
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
OECD 2010-13 DQC 2013
Administrative Type of school
Networks and programmes Evaluation
Data access restrictions
• Full access to data remains the privilege of education authorities and school leaders, even after anonymisation
• Re-identification risks may persist, but limitations of access are at odds with effective data-use
• Access to elements such as assessment results is highly restricted
Full access to anonymised student-level data
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
OECD 2010-13 DQC 2013
Administrators Principals Teachers Parents Researchers
Speed of feedback
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Administrators Principals Teachers Parents Students Researchers
Real Time <1 month >1 month• Timeliness of
feedback is a critical condition to maintain data value
• Most systems outside the US take more than 1 month to make data available, regardless of access rights
– many impose >6 months delays
• Cited reasons for delay include data cleaning and anonymisation
OECD 2010-13 only
Analysis and comparison tools
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
OECD 2010-13 DQC 2013
Integrated tools Between schools
Between students Between teachers• Only about half of the
systems integrate tools for comparison
– Dashboards, automated reports or other tools for data mining and analysis
• When such tools are available, they more often enable comparisons between schools than between individuals
• Tools tend to allow peer-comparisons
Peer comparison
Quality assurance mechanisms
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
OECD 2010-13 DQC 2013
Support for data reporting Training for data reporting
Automated rules Check on subsamples• Support and training
for data reporting available on many systems
– Both definitions and technical standards
• Automatisation of quality checks and inconsistency alerts missing in about a third of the systems
– Key when data are directly imported from multiple sub-systems
n/a
Use of systems for high-stakes
accountability and ranking
0%
20%
40%
60%
80%
100%
OECD 2010-13 DQC 2013
Schools Students Teachers
0%
20%
40%
60%
80%
100%
OECD 2010-13 DQC 2013
Schools Teachers
• Systems often used to support high-stakes accountability, especially in the US
– e.g. admission decisions, career promotion, financial sanctions or school closure
• Systems less often used to inform publicised performance evaluations, especially outside the US
• May contribute to negative perception as punishing tools
System data used for high-stake assessment of
System data used to grade/rank performance of
Training and materials to improve
instruction
• Some systems, especially in the US, provide training to use data to inform instruction
• However, very few are linked to repositories of digital materials that may be used during or to complement classroom instruction
• Data systems still far from becoming expert systems with recommendation engines and pedagogical tagging of materials
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
OECD 2010-13 DQC 2013
Training to use the data to inform instructionCommunities of practiceTeacher access to materialsStudents/parents access to materials
n/a
Conclusions
Summary of survey results
• Student and/or school data systems mainly. Teacher IDs and course codes needed in order to effective support instruction
• Richer student assessment data needed. This may involve more efficient methods of data collection and new instruments
• Data access restrictions remain pervasive, marked differences by role
• Current speed of feedback at odds with support to ongoing interventions
• Analysis and comparison tools remain insufficiently integrated
• Automated inconsistency rules but very limited ability of stakeholders to check the data (expertise)
• High-stake uses are common and stated as goals. Other uses need to be promoted to change perceptions about aims of data systems
• Still far from becoming expert systems with the ability to provide personalised advice and support for teaching and learning
