the role of timss, pirls & pisa in providing a true ... · skills and knowledge of 15-year-old...
Post on 22-Aug-2020
0 Views
Preview:
TRANSCRIPT
The role of TIMSS, PIRLS & PISA in Providing a True Picture of Maltese
Students’ Attainment
Dr. Vincent Marmarà Ph.D.(Stir.), M.Sc.(Sheff.) Statistics, B.Sc.(Hons.)(Melit.), FRSS
for the Malta Union of Teachers
November 2018
Contents1) Main objectives
2) Methodology
3) PISA, TIMSS & PIRLS
4) One-to-one interviews
5) The quantitative study:
Methodology & Demographics
Results – Familiarity with PISA,TIMSS and PIRLS
Results – Performance of the three assessments
Results – Involvement in the three assessments
Results – a. Your workplace and the three assessments – level of importance
b. Your workplace and the three assessments – level of preparation (academic staff)
c. Your workplace and the three assessments – level of preparation (students)
Results – Opinions about the three assessments
Results – The level of students’ knowledge in relation to these assessments
Results – The three assessments and the Maltese syllabus
Results – The three assessments and the language impact
Results – Design and implementation of the assessments
Conclusion of the quantitative study
2
Main Objectives3
Main ObjectivesBased on the information provided by the Malta Union of Teachers (MUT), thescope of work covered the following:
To review and understand PISA, TIMSS and PIRLS by analysing the documentation related to these assessments and the perceptions of
these tests amongst educators and key personnel within the education sector
4
Methodology5
MethodologyThe Methodology for this project included:
1. Setting the necessary meetings with the Malta Union of Teachers (MUT) to understand theircontribution in the whole process;
2. Setting the necessary meetings with the Maltese educational authorities to understand theircontribution in the whole process. These meetings helped to understand the assessmentsbeing carried out in further detail as well as the curriculum and the process of data collection;
3. Setting the necessary meetings to understand the educational system and the perceptions ofkey stakeholders in relation to these tests. The latter meetings were held with the followingmembers:
a. Representatives from the Education department and officials from the Department forCurriculum, Lifelong Learning and Employability;
b. Officials from the Faculty of Education, University of Malta;
c. Independent Schools representatives;
d. Church Schools representatives;
4. Obtained the required data and documentation in relation to the scope of this project to carryout the analysis;
6
Methodology5. Furthermore, a quantitative survey study was carried out amongst Learning Support Educators
(LSEs), Kindergarten Educators (KGEs) Teachers, Heads of Departments (HoDs), Assistant Heads(AHs), Heads of Schools (HoSs) and Education Officers (EOs) to:
“Study and analyse the role of international studies (TIMSS, PIRLS and PISA) to provide a true picture of Maltese Students’ Attainment.”
Throughout this study we analysed questions such as:
a. “What do you consider your level of knowledge to be about these assessments?”
b. “Do you think that your workplace is giving enough importance to these assessments?”
c. “To what extent is your workplace preparing the students for these assessments?”
Disclaimer: This data collection was carried out during the month of October when someeducators were being given training related to the TIMSS assessment. Hence, some educators mayhave been more knowledgeable about this particular assessment.
7
PISA, TIMSS and PIRLS8
PISAWhat is PISA?
The Programme for International Student Assessment (PISA) organised by the
Organisation for Economic Co-operation and Development (OECD) is a triennial
international survey which aims to evaluate education systems worldwide by testing the
skills and knowledge of 15-year-old students. Moreover, the PISA programme investigates
and compares the performance of schools and education systems worldwide.
PISA focuses primarily on the assessment of student performance in reading,
mathematics and science as they are foundational to a student’s continuing education.
However, it also takes into account student attitudes, motivations, and collaborative
problem-solving.
Reference: oecd.org
9
Key findings for PISA 2015• Based on the current literature, no specific issues or flaws with respect to the actual
research methodology exists (data collection and analysis);
• PISA 2015: 38 countries had a mean Science score higher than Malta (465), while 32countries had a lower mean score. Malta’s mean science score was found to be lowerthan the international average (493);
• When comparing European countries, Malta scored higher in Science than in Greece
(455), Bulgaria (446), Romania (435), Cyprus (433), Moldova (428), Albania (427),
Turkey (425), Montenegro (411), Macedonia (384) and Kosovo (378);
• Malta scored significantly lower than Estonia (534), Finland (531), Slovenia (513),
Netherlands (509), Germany (509), United Kingdom (509), Switzerland (506), Ireland
(503), Denmark (502), Belgium (502), Poland (501), Portugal (501), Norway (498),
Austria (495), France (495), Sweden (493), Czech Republic (493), Spain (493), Latvia
(490), Luxembourg (483), Hungary (477), Croatia (475), Italy (481), Lithuania (475)
and Iceland (473);
10
Key findings for PISA 2015• The mean Science score of Maltese students in the PISA 2015 cycle (465) was 4 points
higher than the PISA 2009 cycle (461). Compared to PISA 2009, the mean Sciencescores in PISA 2015 increased by 2 points in State and Church schools and by 20points in Independent schools;
• Mathematics was a minor domain in the PISA 2015 study. 33 countries had a meanMathematics score higher than Malta (479), while 34 countries had a lower meanscore. Malta’s mean mathematics score was lower than the international average(490);
• Reading was a minor domain in the PISA 2015 study. 41 countries had a mean readingscore higher than Malta, while 28 countries had a lower mean score. The readingaverage score for Maltese students (447) was lower than the international average(493);
• Independent and Church schools are scoring higher than the State schools. Studentattainment in Science differs significantly between school types. Male and femalestudents attending church (509) and independent schools (540) are scoringsignificantly higher in Science, compared to the international average (493).Conversely, male and female students attending state schools (427) are scoringsignificantly lower. 11
TIMSSWhat is TIMSS?
The Trends in International Mathematics and Science Study (TIMSS) is a comparative
international study of mathematics and science achievement organised by the
International Association for the Evaluation of Educational Achievement (IEA). IEA is an
independent international cooperative of national research institutions and government
agencies that has been conducting studies of cross-national achievement since 1959.
TIMSS has the goal of helping countries make informed decisions about how to improve
teaching and learning in mathematics and science.
Reference: timssandpirls.bc.edu
12
Key findings for TIMSS 2015
• TIMSS 2015: Malta’s mean Science score (481) was lower than the internationalaverage (500) and was ranked 22nd place out of the 39 participating countries;
• Science attainment of Maltese students was comparable to students from United
Arab Emirates (477). It was found to be higher than 16 other countries including
Malaysia (471), Bahrain (466), Qatar (457), Iran (456), Thailand (456), Oman (455),
Chile (454), Georgia (443), Jordan (426), Kuwait (411), Lebanon (398), Saudi Arabia
(396), Morocco (393), Botswana (392), Egypt (371) and South Africa (358);
• The 21 countries that scored higher than Malta in Science included Singapore (597),
Japan (571), Chinese Taipei (569), Republic of Korea (556), Slovenia (551), Hong Kong
(546) Russian Federation (544), England (537), Kazakhstan (533), Ireland (530), United
States (530), Hungary (527), Canada (526), Sweden (522), Lithuania (519), New
Zealand (513), Australia (512), Norway (509), Israel (507), Italy (499) and Turkey (493);
13
Key findings for TIMSS 2015• On average, Independent school students (550.3 females, 556.4 males) scored higher
in Science than Church school students (519.3 females, 517.3 males) who in turn
scored higher than State school students (452.9 females, 434.3 males).;
• Malta’s mean Mathematics score (494) was 6 scale points lower than the
international average (500) and was ranked 20th out of the 39 participating countries;
• On average, Independent school students (541.3 females, 553.0 males) scored higher
in Mathematics than Church school students (518.3 females, 524.3 males) who in
turn scored higher than State school students (473.0 females, 458.1 males);
• In the TIMSS 2015 cycle, Maltese students scored higher than the 2007 cycle in all
content and cognitive domains. This improvement was achieved by both female and
male students.
14
PIRLSWhat is PIRLS?
The Progress in International Reading Literacy Study (PIRLS) is a comparative
international study of the reading attainment of ten-year-olds (Year 5 students). PIRLS is
organised by the International Association for the Evaluation of Educational Achievement
(IEA). Furthermore, it investigates reading literacy and the factors involved in acquiring
this skill.
Reference: timssandpirls.bc.edu
15
Key findings for PIRLS 2016
• PIRLS 2016: Malta’s mean reading score (452) was significantly lower thanthe international average (500) and was ranked 40th of 50 participatingcountries;
• Reading attainment of Maltese students was comparable to students fromUnited Arab Emirates (450) but was higher than nine countries includingBahrain (446), Qatar (442), Saudi Arabia (430), Iran (428), Oman (418),Kuwait (393), Morocco (358), Egypt (330) and South Africa (320);
• In all countries, female students scored higher in reading literacy than males.In Malta, female (463) students scored 21 scale points higher than males(442);
16
Key findings for PIRLS 2016• On average, Church school students (470.6) scored higher in reading than State
school students (447.5) who in turn scored higher than Independent school students(411.8) (*). Mean reading scores of female students exceed those of males by 26.2scale points in State schools, 15.4 scale points in Church schools and 5.2 scale pointsin Independent schools;
(*) There is societal bilingualism in Malta and this is reflected in the different schoolsectors in Malta. The first language of most students in State schools is Maltese. Thelanguage situation in Church schools is more varied. The first language of most studentsin Independent schools is English. This is made clear by the present PIRLS results whereMaltese was the test language. There was a significant decrease in the performance forstudents from Church schools, but especially for those from Independent schools. It isclear that the language of the test was a huge factor which determined the generalunderperformance of Maltese students on the PIRLS assessment. Source:curriculum.gov.mt (PIRLS 2016, Malta Report)
17
One-to-one interviews
18
One-to-one interviews
19
Several one-to-one interviews were carried out with key players within our educationalsystem. The following are the salient points:
1. There are socio-economic differences between state schools, church schools andindependent schools;
2. Our system is very much exam oriented. If students fail the exams, they will most likelybe lost in the system. Hence, our students are not trained to focus on other tests whichare not their exams;
3. The items found in these tests (PISA, PIRLS, TIMSS) are not in sync with the knowledgeprovided to students;
4. Family background (incl. economic status) is still a dependent factor on the type ofschool (state, church and independent) and this affects test scores;
5. In most foreign countries, the above tests are given more importance, hence greaterawareness;
6. These tests are mainly based on reasoning and thinking skills. This is an issue for Maltaas our system is still exam oriented;
One-to-one interviews
20
7. Our educational system is not based on evidence and research;
8. In order for our ratings to improve we need to put less emphasis on coaching and moreemphasis on reasoning and thinking skills. Decisions need to be taken based on evidence;
9. The vision of the different categories of schools towards these tests is different.
10. Perceptions about PISA, PRILS and TIMSS: Teaching in the Independent schools incorporatesthe same mind set of these assessments.
11. Science teaching: For the independent schools science is a core subject involving around 4lessons per week. This is completely different when compared to other schools (a lowernumber of science lessons);
12. More focus needs to be put on primary schooling if we want to improve Malta’s ranking.
Teachers are not confident enough when teaching science at a primary level;
13. More focus needs to be placed on the science subjects, especially at primary level.
The quantitative study
21
Methodology22
• The survey was carried out online. Educators (Learning Support Educators
(LSEs), Kindergarten Educators (KGEs) Teachers, Head of Departments (HoDs),
Assistant Heads (AHs), Heads of Schools (HoSs) and Education Officers (EOs))
were approached several times through e-mail by inviting them to participate
in this research study;
• A sample size of 475 individuals was collected amongst the educators;
• Level of confidence: 95%;
• Confidence interval: +/- 4.3%;
• The data was collected between the 12th October and the 26th October 2018.
Methodology and Demographics
23
Results – Familiarity with PISA,PIRLS
and TIMSS24
Results – Familiarity with PISA,PIRLS and TIMSS
25
• Most of the respondents heard about these assessments. PISA is the most familiar
assessment amongst the educators (86.6%) while PIRLS is the least familiar (73.6%)
86.6%
13.4%
Have you heard about PISA?
73.6%
26.4%
Have you heard about PIRLS?
84.1%
15.9%
Have you heard about TIMSS?
Yes
No
Results – Familiarity with PISA,PIRLS and TIMSS
26
• Result 3.1 by school type
85.5% 87.9% 85.7%
14.5% 12.1% 14.3%
0%
20%
40%
60%
80%
100%
State Church Independent
Have you heard about PISA?
78.0%
64.8%53.6%
22.0%
35.2%46.4%
0%
20%
40%
60%
80%
100%
State Church Independent
Have you heard about PIRLS?
85.3% 83.7%
69.0%
14.7% 16.3%
31.0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
State Church Independent
Have you heard about TIMSS?
Yes
No
Results – Familiarity with PISA,PIRLS and TIMSS
27
• Result 3.1 by occupation
50.0%
55.0%
10.1%
0.0%
0.0%
0.0%
3.1%
50.0%
45.0%
89.9%
100.0%
100.0%
100.0%
96.9%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Learning Support Educator (LSE)
Kindergarten Educator (KGE)
Teacher
Education Officer (EO)
Assistant Head (AH)
Head of School (HoS)
Head of Department (HoD)
Have you heard about PISA?
Yes
Results – Familiarity with PISA,PIRLS and TIMSS
28
• Result 3.1 by occupation
61.4%
57.9%
28.8%
0.0%
2.3%
0.0%
18.2%
38.6%
42.1%
71.2%
100.0%
97.7%
100.0%
81.8%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Learning Support Educator (LSE)
Kindergarten Educator (KGE)
Teacher
Education Officer (EO)
Assistant Head (AH)
Head of School (HoS)
Head of Department (HoD)
Have you heard about PIRLS?
Yes
No
Results – Familiarity with PISA,PIRLS and TIMSS
29
• Result 3.1 by occupation
56.5%
55.6%
12.6%
0.0%
0.0%
0.0%
9.1%
43.5%
44.4%
87.4%
100.0%
100.0%
100.0%
90.9%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Learning Support Educator (LSE)
Kindergarten Educator (KGE)
Teacher
Education Officer (EO)
Assistant Head (AH)
Head of School (HoS)
Head of Department (HoD)
Have you heard about TIMSS?
Yes
No
Results – Familiarity with PISA,PIRLS and TIMSS
30
• When asked about the level of knowledge regarding these assessments, 26.9% have
some knowledge about PISA, while 33.3% are not knowledgeable at all about PIRLS.
Only 27.6% are knowledgeable or very knowledgeable about PISA and only 22.1% are
knowledge or very knowledgeable about PIRLS.
7.2%
20.4%
26.9%
25.2%
20.4%
0% 5% 10% 15% 20% 25% 30%
Very knowledgeable
Knowledgeable
Some knowledge
Slightly knowledgeable
Not knowledgeable at all
Level of knowledge (PISA)
5.0%
17.1%
25.4%
19.3%
33.3%
0% 5% 10% 15% 20% 25% 30% 35%
Very knowledgeable
Knowledgeable
Some knowledge
Slightly knowledgeable
Not knowledgeable at all
Level of knowledge (PIRLS)
Results – Familiarity with PISA,PIRLS and TIMSS
31
• 25.2% of the respondents are knowledgeable about TIMSS; however 67% are either
‘Not knowledgeable at all’ or ‘Slightly knowledgeable’ or have ‘Some knowledge’.
• The mean and median of the level of knowledge of these assessments is also
illustrated in the table below.
7.8%
25.2%
24.8%
20.7%
21.5%
0% 5% 10% 15% 20% 25% 30%
Very knowledgeable
Knowledgeable
Some knowledge
Slightly knowledgeable
Not knowledgeable at all
Level of knowledge (TIMSS)
Mean/Median
PISA Some knowledge
PIRLS Slightly knowledgeable
TIMSS Some knowledge
Results – Familiarity with PISA,PIRLS and TIMSS
32
• The mean and median of the level of knowledge of these assessments by school
type and occupation are illustrated in the tables below.
State Church Independent
Mean Median Mean Median Mean Median
PISA 3 3 3 3 3 3
PIRLS 2 3 2 2 2 1
TIMSS 3 3 3 3 3 3
PISA HoD HoS AH EO Teacher KGE LSE
Mean 3 3 3 4 3 2 2
Median 3 3 4 4 3 1 2
PIRLS HoD HoS AH EO Teacher KGE LSE
Mean 3 4 3 4 2 2 2
Median 3 4 3 4 2 1 1
TIMSS HoD HoS AH EO Teacher KGE LSE
Mean 3 4 4 4 3 2 2
Median 3 4 4 4 3 1 1
1:Not knowledgeable at all
2:Slightly knowledgeable
3:Some knowledge
4:Knowledgeable
5:Very knowledgeable
Results – Performance of the three
assessments33
Results – Performance of the three assessments
34
• More than half of the interviewed educators think that the Maltese students
performed below average in all three assessments when compared to other
countries.
56.0%
41.4%
2.6%
Performance of Maltese students (PISA)
57.1%
40.6%
2.4%
Performance of Maltese students (PIRLS)
59.3%
37.7%
3.0%
Performance of Maltese students (TIMSS)
Below average
Average
Above average
Results – Performance of the three assessments
35
• Result 3.2 by school type.
• Respondents working in independent schools were more optimistic and think that the
Maltese students’ performance was of an average performance.
58.1%54.4%
48.1%
39.7% 41.2%
51.9%
2.2% 4.4%0.0%
0%
10%
20%
30%
40%
50%
60%
70%
State Church Independent
Performance of Maltese students (PISA)
59.9%53.8%
44.4%37.6%
43.4%
55.6%
2.5% 2.8% 0.0%0%
10%
20%
30%
40%
50%
60%
70%
State Church Independent
Performance of Maltese students (PIRLS)
61.8%56.6%
48.1%
35.3%38.9%
51.9%
2.8% 4.4%0.0%
0%
10%
20%
30%
40%
50%
60%
70%
State Church Independent
Performance of Maltese students (TIMSS)
Below average
Average
Above average
Results – Performance of the three assessments
36
• Result 3.2 by occupation
24.3%
20.0%
56.9%
92.9%
64.1%
69.2%
66.7%
70.3%
80.0%
39.9%
7.1%
33.3%
30.8%
33.3%
5.4%
0.0%
3.2%
0.0%
2.6%
0.0%
0.0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Learning Support Educator (LSE)
Kindergarten Educator (KGE)
Teacher
Education Officer (EO)
Assistant Head (AH)
Head of School (HoS)
Head of Department (HoD)
Performance of Maltese students (PISA)
Above average
Average
Below average
Results – Performance of the three assessments
37
• Result 3.2 by occupation
32.4%
15.4%
56.8%
85.7%
67.5%
76.9%
64.5%
64.9%
76.9%
40.0%
14.3%
32.5%
23.1%
35.5%
2.7%
7.7%
3.2%
0.0%
0.0%
0.0%
0.0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Learning Support Educator (LSE)
Kindergarten Educator (KGE)
Teacher
Education Officer (EO)
Assistant Head (AH)
Head of School (HoS)
Head of Department (HoD)
Performance of Maltese students (PIRLS)
Above average
Average
Below average
Results – Performance of the three assessments
38
• Result 3.2 by occupation
32.5%
15.4%
60.9%
92.9%
65.0%
80.8%
60.6%
57.5%
84.6%
35.5%
7.1%
35.0%
19.2%
39.4%
10.0%
0.0%
3.5%
0.0%
0.0%
0.0%
0.0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Learning Support Educator (LSE)
Kindergarten Educator (KGE)
Teacher
Education Officer (EO)
Assistant Head (AH)
Head of School (HoS)
Head of Department (HoD)
Performance of Maltese students (TIMSS)
Above average
Average
Below average
Results – Involvement in the three
assessments39
Results – Involvement in the three assessments
40
• Very few of the respondents were involved in these three assessments. 16.8% of the
educators were involved in PISA while only 8.8% were involved in PIRLS.
16.8%
8.8%
15.8%
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
PISA PIRLS TIMSS
Involvment in these assessments
16.6%
18.5%
20.7%
9.1% 8.9%
6.9%
15.3%
17.7% 17.2%
0%
5%
10%
15%
20%
25%
State Church Independent
Involvment in these assessments (by school type)
PISA
PIRLS
TIMSS
Results – Your workplace and the
three assessments – level of
importance41
Results – Your workplace and the three assessments – level of
importance
42
• Only 4.8% (PISA), 2.7% (PIRLS) and 3.8% (TIMSS) of the educators said that a high level of
importance is given by their workplace to these three assessments respectively. The
majority said that their workplace gives a ‘Neutral’ level of importance (38.3%, 43.5% and
37.6% respectively.)
4.8%
20.7%
38.3%
19.8%
16.3%
0% 10% 20% 30% 40% 50%
Very high level of importance
Good level of importance
Neutral
Low importance
No importance at all
Level of importance (PISA)
2.7%
17.5%
43.5%
16.6%
19.7%
0% 10% 20% 30% 40% 50%
Very high level of importance
Good level of importance
Neutral
Low importance
No importance at all
Level of importance (PIRLS)
Results – Your workplace and the three assessments – level of
importance
43
• The mean and median of the Likert scale is at ‘Neutral’ level of importance for all the
three assessments. When analysing the Likert scale by school type, all the mean and
medians for all the assessments are ‘Neutral’; however educators from the independent
schools rated the level of importance by their workplace at a higher level (E.g. PISA:
Independent (Mean=3.1), Church (Mean=2.9) and State (Mean=2.7)).
3.8%
25.2%
37.6%
17.1%
16.2%
0% 10% 20% 30% 40%
Very high level of importance
Good level of importance
Neutral
Low importance
No importance at all
Level of importance (TIMSS)
Mean/Median
PISA Neutral
PIRLS Neutral
TIMSS Neutral
Results – Your workplace and the
three assessments – level of
preparation (academic staff)44
Results – Your workplace and the three assessments – level of
preparation (academic staff)
45
• Around a third (33.6%, 36.8% and 32.9%) of the respondents said that their workplace level
of preparation for the academic staff for such assessments is ‘Neutral’. On the other hand,
a considerable amount (31.0%, 30.0% and 27.3%) of the respondents said that there is ‘no
preparation at all’ for these assessments.
0.9%
14.2%
33.6%
20.4%
31.0%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Very high level of preparation
Good level of preparation
Neutral
Low preparation
No preparation at all
Level of preperation for academic staff (PISA)
1.3%
12.1%
36.8%
19.7%
30.0%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Very high level of preparation
Good level of preparation
Neutral
Low preparation
No preparation at all
Level of preperation for academic staff (PIRLS)
Results – Your workplace and the three assessments – level of
preparation (academic staff)
46
• When calculating the mean for the latter question, this resulted in a ‘low preparation’
rating for all assessments while the median for PISA resulted in a ‘Low preparation’ rating
and for PIRLS and TIMSS the result was ‘Neutral’.
Level of preparation for academic staff
Mean Median
PISA Low preparation Low preparation
PIRLS Low preparation Neutral
TIMSS Low preparation Neutral
3.0%
16.5%
32.9%
20.3%
27.3%
0% 5% 10% 15% 20% 25% 30% 35%
Very high level of preparation
Good level of preparation
Neutral
Low preparation
No preparation at all
Level of preperation for academic staff (TIMSS)
Results – Your workplace and the three assessments – level of
preparation (academic staff)
Level of preparation for academic staff
State Church Independent
Mean Median Mean Median Mean Median
PISALow
preparation
Low
preparation
Low
preparation
Low
preparationNeutral Neutral
PIRLSLow
preparationNeutral
Low
preparation
Low
preparationNeutral Neutral
TIMSS Neutral NeutralLow
preparation
Low
preparationNeutral Neutral
47
• The mean and median of the Likert scale by school type is illustrated below. It is evident
that the educators at the independent schools claimed that their workplace prepare
more their academic staff for these assessments.
Results – Your workplace and the
three assessments – level of
preparation (students)48
Results – Your workplace and the three assessments – level of
preparation (students)
49
• 38.0%, 36.5% and 34.3% of the respondents rated their workplace level of preparation for
the students for such assessments as ‘Neutral’. On the other hand, a considerable amount
of the respondents (30.8%, 31.5% and 30.0%) said that there is ‘no preparation at all’ for
these assessments.
0.9%
10.0%
38.0%
20.4%
30.8%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Very high level of preparation
Good level of preparation
Neutral
Low preparation
No preparation at all
Level of preperation for students (PISA)
1.4%
10.0%
36.5%
20.5%
31.5%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Very high level of preparation
Good level of preparation
Neutral
Low preparation
No preparation at all
Level of preperation for academic staff (PIRLS)
Results – Your workplace and the three assessments – level of
preparation (students)
50
• When calculating the mean for the latter question, this resulted in a ‘Low preparation’ for
all assessments while the median for PISA and PIRLS resulted in a ‘Low preparation’ rating
and a ‘Neutral’ rating for TIMSS.
Level of preparation for students
Mean Median
PISA Low preparation Low preparation
PIRLS Low preparation Low preparation
TIMSS Low preparation Neutral
2.2%
13.5%
34.3%
20.0%
30.0%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Very high level of preparation
Good level of preparation
Neutral
Low preparation
No preparation at all
Level of preperation for academic staff (TIMSS)
Results – Your workplace and the three assessments – level of
preparation (students)
Level of preparation for students
State Church Independent
Mean Median Mean Median Mean Median
PISALow
preparation
Low
preparation
Low
preparation
Low
preparationNeutral Neutral
PIRLSLow
preparation
Low
preparation
Low
preparation
Low
preparationNeutral Neutral
TIMSSLow
preparationNeutral
Low
preparation
Low
preparationNeutral Neutral
51
• The mean and median of the Likert scale by school type is illustrated below. Similar to
the prior findings, it is evident that the educators at the independent schools claimed
that their workplace prepares their students more for these assessments.
Results – Opinions about the three
assessments52
Results – Opinions about the three assessments
53
• The majority of the respondents have a “neutral” opinion or said “it is just a test” for all
the three assessments when asked ‘What is your opinion about these assessments?’
15.3%
35.3% 35.3%
14.0%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Positive Neutral It is just a test Negative
Opinion about PISA
12.9%
37.5%36.2%
13.4%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Positive Neutral It is just a test Negative
Opinion about PIRLS
Results – Opinions about the three assessments
54
16.0%
34.9% 34.9%
14.3%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Positive Neutral It is just a test Negative
Opinion about TIMSS
Results – Opinions about the three assessments
55
• Result 3.5 by school type.
13.7%
17.0%
28.6%
37.3%
34.0%
28.6%
37.3%
32.1%
28.6%
11.8%
17.0%
14.3%
0%
5%
10%
15%
20%
25%
30%
35%
40%
State Church Independent
Opinion about PISA (by school type)
12.5%
15.7%
7.1%
39.4%
31.4%
50.0%
37.5% 37.3%
21.4%
10.6%
15.7%
21.4%
0%
10%
20%
30%
40%
50%
60%
State Church Independent
Opinion about PIRLS (by school type)
Positive
Neutral
It is just a test
Negative
Results – Opinions about the three assessments
56
• Result 3.5 by school type
14.5%
20.8%
14.3%
37.0%
28.3%
42.9%
35.8% 35.8%
21.4%
12.7%
15.1%
21.4%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
State Church Independent
Opinion about TIMSS (by school type)
Positive
Neutral
It is just a test
Negative
Results – The level of students’ knowledge in
relation to these assessments57
Results – Do you think these assessments capture what
students know about the topics and provide a true picture?
58
• Only a fraction of the respondents (6.0%, 5.7% and 5.9%) said that these assessments provide
a true picture of the students’ knowledge. Many of the respondents (33.6%, 30.4% and
35.9%) said that only a small part of the content is relevant. There was a significant amount
of respondents (25.0%, 33.9% and 25.7%) who were unsure about the latter question.
6.0%
12.9%
33.6%
22.4%25.0%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Yes Most of thecontent
Small partsof the
content
No I don't know
Does PISA capture what students know about the subject and provide a true
picture?
5.7%
10.0%
30.4%
20.0%
33.9%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Yes Most of thecontent
Small partsof the
content
No I don't know
Does PIRLS capture what students know about the subject and provide a true
picture?
Results – Do you think these assessments capture what
students know about the topics and provide a true picture?
59
5.9%
11.4%
35.9%
21.1%
25.7%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Yes Most of thecontent
Small partsof the
content
No I don't know
Does TIMSS capture what students know about the subject and provide a true
picture?
Results – Do you think these assessments capture what
students know about the topics and provide a true picture?
60
• Result 3.6 by school type.
• Educators at the Independent schools selected the option ‘Most of the content’ more often
for the ‘PISA’ assessment.
5.6%7.8% 7.1%
11.9%13.7%
21.4%
31.9%
41.2%
28.6%
23.1%
15.7%
28.6%27.5%
21.6%
14.3%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
State Church Independent
Does PISA capture what students know about the subject and provide a true picture? (by
school type)
6.3% 6.1%
0.0%
12.5%
4.1%
0.0%
31.3%32.7%
21.4%
18.8%
16.3%
35.7%
31.3%
40.8%42.9%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
State Church Independent
Does PIRLS capture what students know about the subject and provide a true picture? (by school type)
Yes
Most of the content
Small parts of thecontent
No
I don't know
Results – Do you think these assessments capture what
students know about the topics and provide a true picture?
61
• Result 3.6 by school type.
6.7% 5.8%
0.0%
12.1%
9.6%
7.1%
36.4%
40.4%
21.4%20.6%19.2%
28.6%
24.2% 25.0%
42.9%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
State Church Independent
Does TIMSS capture what students know about the subject and provide a true picture? (by school type)
Yes
Most of the content
Small parts of thecontent
No
I don't know
Results – The three assessments and the
Maltese syllabus62
Results – The three assessments and the Maltese syllabus
63
• 44.9% (PISA), 48.2% (PIRLS) and 38.9% (TIMSS) of the respondents do not know if the
content of these assessments is in line with the Maltese syllabi. Very few of the
respondents said that it is in line with the Maltese syllabi (2.2%, 1.3% and 0.9%).
2.2%
14.1%
20.3%18.5%
44.9%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Yes Most of thecontent
Small partsof the
content
No I don't know
The content is in line with the syllabi? (PISA)
1.3%
13.4%
21.0%
16.1%
48.2%
0%
10%
20%
30%
40%
50%
60%
Yes Most of thecontent
Small partsof the
content
No I don't know
The content is in line with the syllabi? (PIRLS)
Results – The three assessments and the Maltese syllabus
64
0.9%
17.9%
24.8%
17.5%
38.9%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Yes Most of thecontent
Small partsof the
content
No I don't know
The content is in line with the syllabi? (TIMSS)
• Throughout this survey, it is evident that the number of responses ‘I don’t know’ is high.
This shows the lack of knowledge that exists amongst the Maltese educators regarding
these three assessments.
Results – The three assessments and the Maltese syllabus
65
• Result 3.7 by school type.
1.9% 2.0%
7.1%13.2%
19.6%
7.1%
17.0%
27.5%
35.7%
21.4%
9.8%
14.3%
46.5%
41.2%
35.7%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
State Church Independent
The content is in line with the syllabi? (PISA) by school type
1.9%0.0% 0.0%
13.9% 14.3%
7.1%
21.5% 22.4%
14.3%16.5%
12.2%
21.4%
46.2%
51.0%
57.1%
0%
10%
20%
30%
40%
50%
60%
State Church Independent
The content is in line with the syllabi? (PIRLS) by school type
Yes
Most of thecontentSmall parts ofthe contentNo
I don't know
Results – The three assessments and the Maltese syllabus
66
• Result 3.7 by school type.
1.2%0.0% 0.0%
17.0%
25.0%
7.1%
24.2%
28.8%
21.4%18.8%
13.5% 14.3%
38.8%
32.7%
57.1%
0%
10%
20%
30%
40%
50%
60%
State Church Independent
The content is in line with the syllabi? (TIMSS) by school type
Yes
Most of the content
Small parts of the content
No
I don't know
Results – The three assessments and the
language impact67
Results – The three assessments and the language impact
68
• A considerable amount of the respondents (61.4%) said that the language of the
assessments has an impact on the Maltese results. Only 5.7% think that the language is not
affecting the result.
• Those working in an independent school (78.6%) are the most in favour that the language is
affecting the results.
9.3%
23.6%
5.7%
61.4%
0% 10% 20% 30% 40% 50% 60% 70%
I don't know
Maybe
No
Yes
Does the language of the assessment impact on the results?
60.2% 59.3%
78.6%
5.3%7.4% 7.1%
25.1% 24.1%
7.1%9.4% 9.3% 7.1%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
State Church Independent
Does the language of the assessment impact on the results? (by school type)
Yes
No
Maybe
I don'tknow
Results – Design and implementation of the
assessments and improvement of the Maltese results69
Results – Design and implementation of the assessments and
improvement of the Maltese results
70
• Below are the opinions of the respondents when they were asked ‘In what ways can these
assessments be better designed and implemented?’
• 23.5% of the respondents think that such assessments need to be more in line with the
Maltese syllabi.
4.5%
5.3%
6.8%
6.8%
6.8%
14.4%
15.2%
16.7%
23.5%
0% 5% 10% 15% 20% 25%
Better timing to conduct these assessmnents
Incentives and motivation for the students
More student friendly
More training to the educators
The assessments are very long
Others
Do not know
Choice of the assessment's language
To be more related to our syllabi
Better ways for design and implementation of such assessments
Results – Design and implementation of the assessments and
improvement of the Maltese results
71
• There were different suggestions on how the Maltese results can be improved. 15.9% of the
respondents think that the Maltese students have to be better prepared for such
assessments. Less routine learning for students was another valid suggestion on how to
improve the results.
2.2%
2.9%
7.2%
8.0%
8.0%
10.9%
11.6%
15.9%
33.3%
0% 5% 10% 15% 20% 25% 30% 35%
Try to be more inline with these assessment
More importance to science subjects
Choice of the assessment's language
More training to the educators
To be more related to our syllabi
Less rote learning
Do not know
Better preparation for students for these assessments
Others
How to improve Maltese results
Conclusion of the quantitative
study72
Conclusion
❖ The majority of the respondents heard about PISA, PIRLS and TIMSS.
❖ Very few of the respondents are ‘very knowledgeable’ about these assessments
(7.2% (PISA), 5.0% (PIRLS) and 7.8% (TIMSS)).
❖ Most of the interviewees think that the performance of the Maltese students is
below average in these assessments.
❖ 16.8% of the educators were involved in PISA, 15.8% in TIMSS while only 8.8% were
involved in PIRLS.
❖ A ‘Neutral’ level of importance was given for the three assessments by the
educators.
❖ A considerable amount of the respondents (30.8% (PISA), 31.5% (PIRLS) and 30.0%
(TIMSS)) said that there is ‘no preparation at all’ for these assessments.
73
Conclusion
❖ Many of the respondents have a ‘neutral’ opinion or said ‘it is just a test’ for all
the three assessments.
❖ Only a fraction of the respondents (6.0% (PISA), 5.7% (PIRLS) and 5.9% (TIMSS)) said
that these assessments provide a true picture of the students’ knowledge.
❖ Very few of the respondents said that that these assessments are in line with the
Maltese syllabi (2.2% (PISA), 1.3% (PIRLS) and 0.9% (TIMSS)).
❖ Only 5.7% of the respondents think that the language is not affecting the result.
❖ The independent schools are more positive about the fact that their workplace is
giving ‘enough importance to these assessments’, that their workplace is
‘preparing the academic staff for these assessments’ and that their workplace is
‘preparing the students for these assessments’.
74
The Endvincentmarmara@gmail.com
top related