THE USE OF STUDENT ASSESSMENT FOR POLICY …...Ramya Vivekanandan Rodrigues Education Policy & Reform Unit, UNESCO Bangkok 27 March 2013, NEQMAP Meeting Bangkok, Thailand Outline of
21
THE USE OF STUDENT ASSESSMENT FOR POLICY AND LEARNING IMPROVEMENT Ramya Vivekanandan Rodrigues Education Policy & Reform Unit, UNESCO Bangkok 27 March 2013, NEQMAP Meeting Bangkok, Thailand
THE USE OF STUDENT ASSESSMENT FOR POLICY AND LEARNING IMPROVEMENT
Ramya Vivekanandan Rodrigues Education Policy & Reform Unit, UNESCO Bangkok 27 March 2013, NEQMAP Meeting Bangkok, Thailand
Outline of presentation
1. Introduction to the Report
2. Findings from the Survey
3. Conclusions
• Objectives • Methodology • Scope
1. Introduction to the Report
1. Introduction
Objectives: To better understand
1. The dissemination and analysis of assessment data 2. Interventions using assessment results 3. Side-effects of assessments
Methodology:
Survey administered by UNESCO Bangkok in late 2011
Respondents: Ministry of Education Officials and/or National experts
Presenter
Presentation Notes
In the Asia-Pacific region, there has been strong emphasis on better monitoring and evaluation of the quality of education. Learning assessment is seen as a key part of this effort. As such, UNESCO Bangkok developed this survey in order 1) to better understand how assessments operate in the Asia-Pacific region and their impact on policy and learning outcomes, as well as unexpected side-effects and 2) to add to the existing knowledge base in regard to the dissemination and analysis of assessment data and interventions initiated and implemented using assessment results The survey also contributes to the progress of the SABER initiative and UNESCO Institute for Statistics (UIS) efforts to establish a knowledge base on learning outcomes. SABER – Systems Assessment and Benchmarking for Education Results. Initiated by the World Bank and conducted in collaboration with UNESCO Bangkok in 2011, involving 12 countries in the East Asia region. Will be refined and rolled out this year and in 2013, as per the information given in Satoko’s presentation. UIS: UIS is developing a new programme entitled the Observatory of Learning Outcomes. With an objective to monitor global trends in student achievement, this programme aims to set up a global database of internationally comparative indicators of learning outcomes by the end of primary school and a catalogue of measures that countries are using to understand the learning levels of their school children.
1. Introduction
Scope: 17 out of 48 Member States of UNESCO’s AP region responded:
Types of assessment covered in the survey: 1. International large-scale assessments 2. Examinations 3. National/sub-national assessments
SUB-REGION RESPONDENTS
Central Asia Kazakhstan, Kyrgyzstan, Mongolia, Uzbekistan
East Asia and the Pacific
Australia (Victoria), Cook Islands, Lao PDR, Myanmar, New Zealand, Palau, Philippines, Tokelau, Thailand
South & West Asia Bhutan, Iran, Nepal, Sri Lanka
Presenter
Presentation Notes
Survey questionnaire prepared in English & Russian and sent to Ministry of Education officials or national experts in the 48 Member States of UNESCO’s AP region. 17 Member States out of 48 responded: Survey covered international large-scale assessments, examinations and national/sub-national assessments. Classroom assessments were not within the scope of this survey. For each type of assessment questions were asked focusing on: 1) education assessment practice; 2) analysis of assessment data; 3) the use of assessment data for policy and learning improvement and 4) the side effects of assessments
• International large-scale assessments • Examinations • National/sub-national assessments • Perceived side-effects of assessments
2. Findings from the Survey
2. Findings: International Assessments
More countries/jurisdictions are participating in international assessments such as PISA, TIMSS and PIRLS
PISA: From 5 (2000) to 16 (2012)
TIMSS: From 11 (1995) to 14 (2011)
PIRLS: From 6 (2001) to 8 (2011)
8 7
6 6 6
4 4
0
1
2
3
4
5
6
7
8
9
National report to key stakeholders
Summary/brochures or presentations made to
key stakeholders
Natiional Report available online
International report to key stakeholders
Conferences/seminars organized for stakeholders
Press release Feedback to schools/educators
2. Findings: Dissemination of results from international assessments
The survey indicates that feedback to participating schools is not very common.
Presenter
Presentation Notes
Most countries which responded to the survey indicated that they release their national results on these assessments online as well as via national reports, summaries/brochures or presentations to key stakeholders. However, dissemination via press releases and giving feedback to participating schools is more uncommon, which leads us to wonder why.
8 7
5
7
5 4
7
4
1 0
1
2
3
4
5
6
7
8
9
Snapshots ofspecific
groups ofstudents
Snapshots ofnational
performance
Snapshots ofspecific
groups ofschools
Progress ofnational
performance
Progress ofspecific
groups ofstudents
Progress ofspecific group
of schools
Analysis atnational level
Analysis atother levels
Analysis atregional level
2. Findings: Analysis of data from international assessments
The survey indicates that very little of the analysis happens at the regional level.
Presenter
Presentation Notes
Table 2. Analysis of international assessment data that has been made public
7 6
5 4
3 3 3 2
1 0
1
2
3
4
5
6
7
8
Conferences for policy-makers
Professional development of
teachers
Review of curriculum
Intervention on learning theme
Intervention for specific group of
students
Professional development for
proncipal
Conferences for unions
Intervention for specific groups of
schools
Feedback to students
2. Findings: Use of International Assessments
In addition, “ranking” in such assessments has significant influence on national policy formulation, sometimes igniting hot policy debates that lead to whole-sector reforms (i.e. Kyrgyzstan, Myanmar)
Presenter
Presentation Notes
Table 3. The activities following the results of the most influential international large-scale assessment
2. Findings: Examinations
Two main observations for when examinations are held: 1. Just before/after transitional points:
Most influential examination:
From primary to lower secondary
From lower to upper secondary
At the end of upper secondary
2. At the upper secondary level
Graduation from upper secondary education
Entrance to post-secondary, non-tertiary education
14
10 10 9 8 8 6 6
3
0
2
4
6
8
10
12
14
16
Intervention for specific group of
students
Professional development for
proncipal
Seminar conferences for
unions
Conferences for policy-makers
Feedback to students
Professional development of
teachers
Review of curriculum Intervention on learning theme
Intervention for specific groups of
schools
2. Findings: Use of Examinations
Compared to other types of student assessments, results of examinations are more likely to be used for learning improvements.
Presenter
Presentation Notes
TAKA-This chart is useful if we are to make a point that the results from the examination is more likely to be used for feedback to the students. I circled the parts that particularly intriguing about this chart. Table 4. Activities following the most influential examination
2. Findings: National/Sub-national Assessment
Refers to large-scale assessment surveys designed to measure the achievement of students in a curriculum area for the purpose of monitoring educational quality
Most of the countries which participated in the
survey have national / sub-national assessment in place
12 12 12
9 8
7 6 6
5
0
2
4
6
8
10
12
14
Conferences for policy-makers
Professional development for
principal
Review of curriculum
Professional development of
teachers
Intervention on learning theme
Intervention for specific groups of
schools
Conferences for unions
Intervention for specific group of
students
Feedback to students
2. Findings: Use of National Assessments
Results of national / sub-national assessments seem to be used mainly for making policy improvements.
Presenter
Presentation Notes
TAKA-I thought it is more useful to include a chart for the use of national assessment than the one for examination, as they both illustrate that feedback to students is more uncommon use of the results of assessments.
2. Findings: Side Effects of Assessments
Types of Side Effects: Pressure from increased workload / preparation
Extra paid private tutoring outside classroom
Teaching to the test
Focusing on performance of certain groups
Presenter
Presentation Notes
Although assessments are useful tool for policy and learning improvement, they can also generate side effects such as increased stress/workload, extra cost burden for families in the form of private tutoring, pressure on the teacher to “teach to the test” and focusing on the performance of certain groups of students. The survey aimed to investigate aspects of these four types of unexpected side-effects and the source of such perceptions and how widespread they are among in the region. In so doing, the survey identified 6 groups of possible stakeholders: (1) students/students unions (2) teachers/teacher associations or unions, (3) parents and wider communities, (4) ministry officials/policy-makers, (5) cluster of schools, (6) the media. The survey was sent to Ministry officials and experts in Member States in the AP region. The respondents were asked to estimate the degree to which each stakeholder group perceives the side effects, on a scale of 0-5, where 0 indicates no side effects. While the information gleaned is useful, it is recommended that the actual stakeholders be asked about their perceptions. Further research could look into this e.g. via focus group research.
2. Findings: Side Effects of Assessments
Stakeholders affected by the side effects
Students/student unions
Teachers/teacher associations or unions
Parents and wider communities
Ministry officials/policy-makers
Cluster of schools
The media
Presenter
Presentation Notes
In so doing, the survey identified 6 groups of possible stakeholders: (1) students/students unions (2) teachers/teacher associations or unions, (3) parents and wider communities, (4) ministry officials/policy-makers, (5) cluster of schools, (6) the media. The survey was sent to Ministry officials and experts in Member States in the AP region. The respondents were asked to estimate the degree to which each stakeholder group perceives the side effects, on a scale of 0-5, where 0 indicates no side effects. While the information gleaned is useful, it is recommended that the actual stakeholders be asked about their perceptions. Further research could look into this e.g. via focus group research.
• Analysis of assessment data • Use of assessment data • Recommendations
3. Conclusion
3. Conclusions: Analysis of Assessment Data
Type of Analysis
• The survey suggests that snapshot analysis of assessment results is more common than longitudinal or progress analysis.
Level of Analysis
• Analysis is more commonly conducted at the national level rather than at regional, school or student levels
Presentation of Analysis
• Countries seem more likely to limit analysis of assessment data to the presentation of simple descriptive analysis of national performance rather than measuring the progress made by students or schools.
Presenter
Presentation Notes
The results from the survey indicate that countries in the AP region tend to conduct and release snapshot analysis more than longitudinal analysis. Snapshot analysis Simple frequency, pivot tables present the performance of specific groups of students/schools or overall regional/national performance and often referred to as cross-sectional analysis. Such types of analysis present a snapshot picture of overall student achievements at a specific point in time and they are commonly used among policymakers. One of the greatest advantages of such analysis is that they can be produced in a short period of time; however, it is known to mask the variances in student achievement. Progression analysis Longitudinal analysis is the study of a population over time, as opposed to cross-sectional analysis. Such analysis uses the assessment data for the same students at different points in time. One of the great benefits of having longitudinal data is the ability to do value-added analysis which allows estimating value-added impact(s) of student, school or teacher factors, while controlling for previous student outcomes or by making comparison among students with the same prior outcomes.
3. Conclusion: Use of Assessment Data
Use of Assessment
Data
Respondents reported that participating in international assessments led to : review or changes in their curricula,
professional development of teachers and organization of seminars or conferences for policy makers.
For national/sub-national assessments, the survey indicated that
policy makers, researchers and school leaders are the primary audience for results.
But students themselves are not – only few countries appear to give feedback directly to them.
There were at least three countries which said that international assessments did not have a direct impact
on policy and learning improvements.
Presenter
Presentation Notes
One of the main objectives of this survey was to find out how Member States use various assessments for policy formulation and other interventions and initiatives in practice. For each type of assessment, the survey aimed to understand the following question: Have the results of this assessment lead to any of the following? (several choices given to tick) reviews of or changes to curriculum specific intervention programmes professional development activities organization of seminars or conferences and feedback to students. Respondents reported that participating in international assessments led to review or changes in their curricula, professional development of teachers and organization of seminars or conferences for policy makers. There were at least three countries which said that international assessments did not have a direct impact on policy and learning improvements. In regard to national/sub-national assessments, the survey indicated that policy makers, researchers and school leaders are the primary audience for results. But students themselves are not – only few countries appear to give feedback directly to them.
3. Conclusions: Recommendations
Follow-up study focusing on the analysis and use of assessment results
In order to identify the needs of Member States and the potential for capacity building of Ministries of Education, further investigation is needed on: The type of primary and secondary data being collected at the national level
on achievement and pathways of students
The analysis and statistical techniques used in synthesizing data
Establishment of a network for greater regional collaboration for evidence-based decision making is highly encouraged.
