Teaching practices and attitudes among TALIS teachers, their characteristics

and association with student learning outcomes in National Tests and PISA

Ragnar F. Ólafsson Educational Testing Institute, Iceland.

International TALIS Seminar and Conference, Haarlem, March 11, 2015.

• „teaching is cultural; most teachers within a culture use similar methods“ (Stigler and Hiebert, 2004, p. 16)

• „teachers are proxies for an educational system‘s values (...) mathematics teachers in one country behave in ways that identify them more closely with teachers in their own country than teachers elsewhere (Andrews, 2010, p. 21)

Aims:

• Explore cultural differences in teacher practices and attitudes across Europe

• Examine whether such differences are associated with progress in PISA

Four datasets: Teacher questionnaires

TALIS 2008: Questions/items related to Collective Teacher Efficacy in mostly European 23 countries, co-authors Auður Pálsdóttir and Allyson Macdonald, University of Iceland.

TIMSS and PIRLS 2011: Questions/items about the background, teaching practices of teachers in reading, mathematics and science in 24 countries. In Northerns Lights on TIMSS and PIRLS, co-author Júlíus K. Björnsson.

Four datasets: Teacher questionnaires

PIRLS 2006: Questions on teaching practices and teacher background of reading teachers in 27 countries, mostly European.

TALIS 2013: Complete TALIS teacher questionnaire: Teaching practices, professional development, feedback, teacher background etc. of teachers in European countries.

Analysis

• Multidimensional Scaling

• Country means for each question/item were computed with IDB Analyzer.

• Country means were rescaled 0-1: Country mean divided by top score, multiplied by 100

Analysis

•MDS and HCA: Dimensions and country groups (clusters) identified and interpreted.

•Correlations between item country averages and country dimension scores.

•Dimension scores correlated with country progress in PISA.

• Self-reports

• Items paraphrased

• Correlations, individual countries may depart from the main trend

• Descriptive, exploratory

Dimension 1

• TEACHER PREPARATION

• West: Content and pedagogy of subjects taught are covered less well, compared with East, and the teachers felt less well prepared with respect to content, pedagogy and classroom practice in the subjects they taught.

• PROFESSIONAL DEVELOPMENT• East: More informal induction activities, more qualification

programmes in East.• PD in the East covered more topics (e.g. pedagogical

competencies, ICT, Classroom management, teaching cross-cultural skills, student career guidance and councelling).

• PD activities had more more positive impact on their teaching.

• They reported more need for PD in student behaviour and classroom management, school management and adminstration, new technologies in the workplace.

• FEEDBACK• Feedback sources: Teachers in the East reported having received

feedback from more individuals or bodies, such as Externals, the Principal, the school management team and assigned mentors. They also give such feedback themselves. Fewer in the West report receiving feedback from those individuals.

• What is being looked at in the feedback? Teachers in the East said all items on the list were considered with higher emphasis, compared with West, e.g. student performance, assessment practices, knowledge of subject field, feedback from parents etc. when they were receiving feedback.

• FEEDBACK (cont.)

• Teachers in the East report more effect of feedback and show more positive attitude to appraisal and feedback by agreeing more with items stating that proper recognition is given, that the appraisal is thorough, that weaknesses get discussed with the teacher and training plan developed etc.

• Did feedback lead to positive change? Teachers in the East invariably report more positive change following feedback – on all items considered, e.g. teaching practices, job responsibilites at this school, job satisfaction, likelyhood of career advancement etc.

• TEACHING PRACTICES

• Less Constructivism in East? They agree less to the statement that thinking and reasoning processes are more important than specific curriculum contents. Also do they not claim to use a variety of assessment stategies as much as the West, but use standardised tests more often and check student excercise books more often.

• The East however – in their teachings - let students practice similar tasks until everybody has understood and refer more often to problems from everyday life.

• DISCIPLINE The West spends more time on keeping order in the classroom and less time on actual teaching, compared with East. The West has to wait longer for class to get quiet, and looses a lot of time because students interrupt.

• West: Teacher-student relations are however better, with greater job satisfaction.

Dimension 2• Up: For PD activites, more scheduled time and/or salary

supplement is provided.• Up: Felt less need for PD of various sorts, e.g. to teach

students with special needs, improve pedagogical competencies etc.

• Up: Underperformance leads more to the appointment of a mentor or dismissal, while best performers are perceived to get recognition. They believe feedback has more impact.

• Down: Complained there were no real incentives for participating and that there was no relevant PD offered.

Hofstede‘s dimensions

• POWER DISTANCE People accept hierarchy, unequal distribution of power, no need to justify vs. Equality, distribution of power.

• INDIVIDUALISM Loosely-knit social framework, take care of self vs. Collectivism, loyalty.

• MASCULINITY Achievement, heroism, assertiveness, competition vs. Femininity, cooperation, quality of life.

Hofstede‘s dimensions

• UNCERTAINTY AVOIDANCE Ambiguity as uncomfortable, rigid codes vs. Relaxed attitude.

• LONG vs. SHORT TERM NORMATIVE ORIENTATION Thrift and effort to prepare for the future vs. Preference for time-honored traditions and norms.

• INDUGENGE vs. RESTRAINT Relatively free gratification, enjoyment, vs. Restraint, suppression, strict norms.

Correlations between MDS-dimensions and Hofstede’s National cultural dimensions

Dim1of2 Dim2of2 Dim1of3 Dim2of3 Dim3of3PowDist -.669** -0.204 -.696** -0.23 0.029Individual.vs.Collect0.384 .434* 0.363 .439* -0.243Masc.vs.Fem -0.269 0.206 -0.286 0.213 0.143UncertAvoid-.523** -0.348 -.540** -0.341 0.079LongTermOr -0.4 -0.184 -.445* -0.271 -.469*Indulg.vs.Restraint.818** 0.284 .824** 0.332 0.094** sign. at the 0.01 level (2-tailed).

*. Sign. at the 0.05 level (2-tailed).

Link with PISA progress

PISA_READprogress2000_2009 PISA_MATHprogress2003_2012 Dim1of2 Dim2of2 Dim1of3 Dim2of3 Dim3of3Dim1of2 -0.461 -.618** 1 -0.029 .994** -0.018 -0.107Dim2of2 -0.007 -0.205 -0.029 1 -0.054 .977** -0.191Dim1of3 -.474* -.594** .994** -0.054 1 -0.029 -0.016Dim2of3 -0.008 -0.141 -0.018 .977** -0.029 1 -0.024Dim3of3 -0.158 0.192 -0.107 -0.191 -0.016 -0.024 1** sign. at the 0.01 level (2-tailed).

*. Sign. at the 0.05 level (2-tailed).

TALIS 2008: Questions/items related to Collective Teacher Efficacy in 23 mostly European countries, co-authors Auður Pálsdóttir and Allyson Macdonald, University of Iceland.

TALIS 2008

TALIS 2008

Country clusters

• Eastern European

• Nordic

• Mediterranean

• Mid-European

• and outliers

Dimension 1

Culture of observation, feedback and improvement. It correlates most strongly with items assessing whether the principal or other management team members observe teaching in classes and give suggestions on how to improve teaching. It also correlates with items assessing the teachers’ confidence in the principal’s methods to determine whether the teacher is doing well or badly… (cont.)

Dimension 1 (cont.)

Culture of observation, feedback and improvement

…It correlates with measures assessing frequency and effect of feedback received, e.g. on teachers’ understanding of their main subject field, instructional practices, teaching in multicultural setting, and whether feedback led to the development of a training plan to improve teaching, and whether innovative teaching is rewarded.

• Eastern: Receive more professional development, yet express more need for it. They believe to a greater extent that reading professional journals has effect on their teaching.

• Nordics and Mid-Europe reject idea that “teaching facts is necessary” or that “instruction should be about problems with clear, correct answers that students can grasp quickly”

• Teachers in the West believe more often that professional development plan is put in place for bureaucratic purposes only while teachers in the East agree more with the notion that the feedback has an effect on their teaching practices.

• Culture of observation, feedback and improvement (COFI)

• Correlates with progress in PISA reading literacy; rho=-0.62

• Interestingly, COFI also correlates with country participation rates in TALIS (r=-0.67)

• TIMSS and PIRLS 2011: Teacher questionnaire regarding background, teaching practices and attitudes of reading, mathematics and science teachers of representative samples of students in participating countries.

• Aims:

• Are country groups from TALIS replicated in TIMSS and PIRLS 2011?

• Are similar dimensions differentiating between countries?

• Do the dimensions predict progress in PISA?

Country clusters

• Eastern European

• Mediterranean

• Anglo

• Nordic

• Germanic

• West: Better access to computers• West: Less communication with other teachers

about teaching practices, their own teaching experience, less cooperation between teachers on new teaching projects, they more rarely observe others teaching

• West: Teachers less often summarise at the end of lessons what has been covered in the lesson, and more seldom encouraged pupils to improve their own performance.

• West: Meet parents less often, send them progress reports less often.

• East: Use textbooks more, children‘s books, magazines for children.

• East: Pupils read aloud more, are systematically taught new vocabulary, learn scanning and skimming text.

• West: Pupils read books of „own choice“. • East: Pupils asked to link reading material to

own experience, link it with other material read earlier, explain and argue their interpretation of the text, identify main ideas, draw conclusions from text, describe author‘s style, author‘s viewpoint, author‘s intentions etc.

• East: More assigment of home reading and follow up whether homework is done.

• East: If pupil gets behind in reading, they get more time with teacher. In West, they are referred more often to specialists.

• East: Overall, more examinations and assessments.

• East: Mathematics teachers ask pupils more often to provide justifications for their answers, to memorize rules, work together (the whole class) more often with direct guidance from teacher.

• East: More time spent on professional development.

Correlation between PISA progress and dimensions 1 and 2

Dimension 1 Dimension 2

Progress_Reading 0.606*p=0.013n=16

-0.075p=0.782n=16

Progress_Mathematics 0.573*p=0.026n=15

-0.407p=0.132n=15

Progress_Science 0.410p=0.073n=20

-0.530*p=0.016n=20

• PIRLS 2006: Teacher questionnaire pertaining to teacher practices, attitudes and their background.

• Aims: Replicate country groups and dimensions – and link with PISA progress

PIRLS 2006: Teacher practices and attitudes in European countries

Country clusters

• Eastern European

• Nordic/Anglo

• Nordic

• Mid-European, incl. BeNeLux

• West: More books available in classroom reading corner and more time spent in reading corner.

• East: Assign homework in reading more often and expect students to spend more time on reading homework.

• East: Students with reading difficulties are more often assigned extra homework to improve reading, and sometimes other students called in to assist them.

• West: Greater access to specialised staff to attend to students with reading difficulties. In the East, the teachers themselves say they respond by devoting more time to the students having difficulties.

• West: Teachers report more often having the student with difficulties work in the regular classroom with a teacher aid or reading specialist or in a remedial reading classroom, compared with the East.

• East: More assessments and testing, classroom tests, teacher-made or textbook tests, and greater use of diagnostic reading tests.

• East: Grades given more often, and use various other material to assess progress such as portfolios

• East: Also, greater variety of methods used for reading assessment, e.g. multiple-choice questions, short open questions, asking students to write short paragraph about what they just read.

• East: Teacher listened more often to pupils reading aloud, asked them verbally about what they had read, listened to the pupils give overview of what they read, met them to discuss what they had read.

• East: Meet the parents more often to discuss progress, send more often progress reports to parents

• East: Teachers report having studied a number of areas more often than their counterparts in the West, e.g. the language of the test, literature, the pedagogy/teaching reading, psychology, remedial reading and reading theory

• East: More professional development, e.g. more in-service professional development workshops or seminars that dealt directly with reading (reading theory, instuctional methods) compared with teachers in the west

• East: More reading of books, professional journals related to teaching

• West: Teachers prepare more in school while teacher in East prepare more at home

• East: Teachers say they had more enthusiasm for teaching when they began, compared with their enthusiasm now

Link with progress

• The Pearson correlation between Progress in PISA literacy (from 2000 to 2012) and dimension 1 was r=-0.62; rho= -0.76.

Summary• Country clusters and dimensions persist

broadly accross datasets, countries and disciplines

• East – West differences similar accross studies:– Culture of observation, feedback and improvement

- TALIS– Engagement – PIRLS and TIMSS

• Correlations with PISA progress

• Correlation between dimensions and progress does not say anything about individual countries or questions/items.

• Correlation at country level does not imply correlation at the level of the individual.

Caveat

• Social desirability? Countries high on the Human Development Index (e.g. Nordic countries and Canada) generally display less social desirability or extreme responses.

• Experience of foreign students that come to Iceland indicates that there are more examinations and homework in Eastern European countries than in Iceland.

• Confirmation from TALIS NPMs from Eastern Europe countries

Future directions

• Correlate with progress on other measures of achievement

• Examine in more depth and breadth the nature of the cultural differences between the country clusters (e.g. Nordic, Anglo, Germanic, East European, Mediterranean) and how they are related to school activites

• How are the teaching practices related to other fundamental practices in these societies

Future directions

• Examine cultural differences within countries. – Can we replicate the local geographies of these

countries with MDS of teaching practices at school or community level?

– Does the same main dimension emerge if we work with school or district means? – and does it correlate with progress? The terms used to compare communities or countries may not be the same that we use to compare individuals within those communities.

• One can change teaching practices

• “by using methods known to change culture. Primary among these methods is the analysis of practice, which brings cultural routines to awareness so that teachers can consciously evaluate them and improve them.” (Stigler og Hiebert, 2004).

Association between TALIS_ISL and grades on National tests

• School grade averages in Icelandic, Mathematics and English, 7th and 10th grade, were correlated with teacher averages at school level on TALIS 2013 questions.

• The more systematic correlations are reported here.

Higher grades associated with...

• A higher percentage of women teachers at school.

• Higher age and longer service.

• Higher percentage of teachers with full-time employment.

• Lower grades where teachers had had long career in other jobs, unrelated to education.

Higher grades if...

• Content, Pedagogy and Classroom practice were parts of formal teacher education for greater proportion of subjects taught.

• If the teachers felt prepared in content, pedagogy and classroom practice in subjects they taught.

Counter-intuitive links

• Taking part in induction program or having an assigned mentor for support is associated with lower grades.

• Also, lower grades were observed where teachers had sought PD on student behaviour and classroom management.

Lower grades if...

• Where Principal or School Managment give feedback after classroom observation.

• Also lower grades where feedback was believed to be there only to fulfill administrative requirements.

• Grades were lower where many had special needs, were low academic achievers, with behavioural problems or came from socioeconomically disadvantaged homes.

• A high percentage of students with different first language was however not associated with lower grades.

• Lesser grades if more time is spent keeping order in the classroom (maintaining discipline) but higher grades were more time is spent on actual teaching and learning

• Thank you!