sydney symphony orchestra tuned-up! evaluation report · evaluation report . authors: rachel...

B Y P G r o u p h t t p : / / b y p g r o u p . c o m

Sydney Symphony Orchestra TunED-Up! Evaluation Report

Prepared for Sydney Symphony Orchestra by Rachel Smithies, BYP Group 1 December 2016

2

2

SSO TunED-Up! Evaluation Report

Authors: Rachel Smithies, BYP Group © 2016 Sydney Symphony Orchestra Disclaimer: The information contained in this report has been derived from survey and interview responses and data provided by ACARA and CESE. The authors advise that while every effort has been taken to ensure the accuracy of information on the following pages, the author does not accept responsibility for the original sources of information or the views expressed in this report.

http://bypgroup.com/

Table of Contents Table of Contents ........................................................................................................... 3 1. Executive Summary ................................................................................................ 4 2. Introduction ........................................................................................................... 8 3. Program Reach ..................................................................................................... 10 4. Alumni Survey ...................................................................................................... 14 5. Impact on Students .............................................................................................. 22 6. Recommendations ............................................................................................... 30 References ................................................................................................................... 32 Appendix 1: Full Educational Outcomes Analysis ........................................................ 34 Appendix 2: Distribution of Participants ...................................................................... 39 About the Authors ....................................................................................................... 40

4

1. Executive Summary This is an evaluation report on the Sydney Symphony Orchestra TunED-Up! Program. Findings are based on analysis of school statistics and an alumni survey.

1.1. Background TunED-Up! was launched in 2014 to address a crisis in music education, with music not taught in 60% of primary schools in NSW despite being a mandatory subject. The aim of the program is to address the unmet need for primary school teachers to teach music with confidence and consistency, aligned with the national curriculum. 121 participants have been through the program over 2014-2016 representing between them 102 schools. This match is not exact, as some participants worked at multiple schools, and others were in pre-service training.

1.2. Program reach The program is very successful at reaching a wide range of schools. The participating schools are representative of all NSW schools in terms of:

• location (whether metropolitan, regional or remote) • the socio-economic background of families at the school • the size of the school • the proportion of students who have a language background other than

English There is a higher rate of Indigenous student enrolments at the participating schools than at NSW schools in general (11% vs 9%). We estimate the program has reached some 43,350 students over 2014-2016.1

1.3. Alumni survey

Program impact Participants are overwhelmingly positive about the program. They rate its impact on their professional skills as 4.4 out of 5, and its impact on their professional development as 4.5 out of 5. TunED-Up! is more positively rated than the typical music professional development program, which past research has shown teachers rate as 3.7 (impact on skills) and 3.6 (impact on confidence).2

1 Calculated as the number of participating schools (102) times their average enrolment (425 students). 2 Petrova, 2012, p. 295. (Results converted from 7 to 5-point scale for comparability.)


5

Current experiences of teaching music in primary schools Participants report that music teaching at primary schools is still, in many cases, in a precarious position. Only half say music is taught across their whole school (51%). Just under half say that music is not taught across their whole school but only in some years or stages (43%), with the rest reporting it is extra-curricular only. Around a third of past participants are in casual or temporary roles (37%). Among the music activities teachers do, a third have delivered music teaching in RFF classes (Relief from face-to-face – i.e. when the regular teacher is rostered off). A lot of activity is done outside of classrooms, with four-fifths of teachers delivering music teaching for extracurricular activities (79%). When asked about the support for music teaching in their schools, participants rate staffing numbers as poor-to-neutral (2.8 out of 5). School facilities and the amount of time available are rated as neutral (3.1 and 3.0 out of 5). They are somewhat more positive about the support from school leadership, rated as neutral-to-positive (3.6 out of 5).

1.4. Impact on students Previous research has found that the arts ‘teach vital modes of seeing, imagining, inventing and thinking’.3 These in turn can have positive effects on students’ educational results in non-arts spheres. For example, there is evidence that music teaching is associated with better scores in mathematics, and in English (particularly reading and verbal skills).4 There is also evidence that active participation in arts learning in school leads to students attending school more often.5 In this evaluation we draw on data collected by ACARA (the Australian Curriculum Assessment and Reporting Authority) and by the CESE (the NSW Centre for Education Statistics & Evaluation) to look at whether such patterns occur for participants in TunED-Up!. Educational outcomes are measured using scores from Australia’s national standardised testing system, NAPLAN.

Methodology The analysis uses a quasi-experimental design, looking at how school outcomes change from before to after participation. The focus is relative changes over time, comparing schools who have participated in TunED-Up! with other NSW primary schools.

3 Winner & Hetland, 2008, p. 31. 4 Hetland & Winner, 2001, Smithrim & Upitis, 2005; Standley, 2008; Johnson & Memmott, 2007; Broh, 2002; Eason et al, 2013; Kelly, 2012. 5 Harris Interactive Inc., 2006; Eason & Johnson, 2013; Martin et al, 2013.


6

Our analysis6 lets us isolate three types of effects: • A time trend: do outcomes at individual schools change over time? • A group comparison: are participant schools consistently different in their

outcome levels than non-participant schools, both before and after participation?

• A relative change trend: do outcomes at participating schools change over time in a way that is different from non-participating schools?

The relative change trend is the focus of this report. If TunED-Up! affected outcomes, we would see that in the relative change trend.

Numeracy outcomes We find positive associations for the 2015 cohort. Their students’ Year 3 Numeracy scores improve more over 2014-2016 than those of non-participant schools. Their students also show greater improvements in numeracy scores from Year 3 to Year 5 than do other NSW schools. While the finding is not repeated in the 2014 or 2016 cohorts, there are data limitations for both these groups: the 2014 cohort is very small, while we lack a third time point for the 2016 cohort. On balance it seems likely participation in TunED-Up! has a positive effect on numeracy scores.

Reading and Grammar & Punctuation scores We find positive associations for the 2015 cohort. Their students’ Year 3 Reading, and Grammar & Punctuation scores, improve more over 2014-2016 than those of non-participant schools. Again, the finding is not repeated in the 2014 or 2016 cohorts, but as noted above there are data limitations for those these groups.

Spelling and Writing scores Our findings on Spelling and Writing scores are mixed. While the 2015 cohort shows positive effects for the domain of spelling, the 2016 cohort shows negative associations for both spelling and writing: scores drop from 2015 to 2016. It should be noted the 2016 cohort started with relatively high scores on spelling and writing in 2015 – higher than the NSW average. It may be that the 2015 scores were outlier scores, and the downward trend we observe in 2016 is due to scores moving back to their usual level. The 2016 cohort also differs from previous cohorts in having an even higher level of Indigenous student enrolment (16% vs 11%).

6 The analytic technique is called a General Linear Model with repeated measures and a mixed design of one within- and one between-subjects factor.


7

Future analysis of the 2017 scores may clarify the overall trend. At this point, we cannot conclude that participation in TunED-Up! affects Spelling and Writing scores.

Attendance rates We do not find any association between participation in TunED-Up! and schools’ average attendance rates. This is contrary to the existing research evidence. Anecdotal feedback from past participants suggests some do notice an effect on attitudes to school attendance, among students with poorer initial attitudes. This specific effect might be lost when looking at whole-of-school data.

The link between program implementation and outcomes The alumni survey findings suggest many participants encounter challenges in fully implementing the TunED-Up! program when they return to work, due to the precarious nature of music teaching in primary schools. It is striking that the cohort with the clearest evidence for positive impacts, the 2015 cohort, is also the one with the highest percentage of teachers reporting their schools have classroom music across the whole school (62%, compared to 47% of the 2014 cohort and 41% of the 2016 cohort). It could be that we would find stronger evidence on the impacts of TunED-Up! if all participants were able to fully implement the TunED-Up! program.


8

2. Introduction This is an evaluation report on the reach and impact of the Sydney Symphony Orchestra TunED-Up! program.

2.1. Background TunED-Up! is a relatively new SSO education program, targeting primary school teachers. It was launched in 2014 to address a ‘crisis in music education’ (quoting Rod Kemp, Federal Arts Minister 2001-2007)7, with music not taught in 60% of primary schools in NSW despite being a mandatory subject. The aim of the program is to address the unmet need for primary school teachers to teach music with confidence and consistency, aligned with the current state syllabus and the national curriculum. It is designed for generalist teachers to build skills and confidence, and provides resources, pedagogy techniques and programming strategies. TunED-Up! began as a five-day intensive residential program in 2014 and 2015. In 2016 a residential option was provided for regional teachers and day immersion for local metropolitan teachers. All 2014 to 2016 participants received a full scholarship (tuition, meals and accommodation). Participants become part of a support network and can subsequently participate in ‘ToppED Up!’, a two-day course to consolidate and build their skills and network. To date there have been 3 iterations of TunED-Up!:

• January 2014: pilot, with 19 participants • January 2015: 50 participants • January 2016: 52 participants

2.2. Methodology This evaluation looks at:

• The profile of participating schools • Past participants’ view of how the program affected their teaching, and

current experiences of teaching music • The impact on students at participating schools in terms of attendance and

educational outcomes.

7 Kemp’s comments were based on the 2005 National Review of School Music Education (Department of Education, 2005). The quote comes from Petrova (2012). Petrova (2012) provides further evidence on the poor state of primary school music education.


9

We draw on two main information sources. The first is government data on primary schools. Datasets were obtained from ACARA (the Australian Curriculum Assessment and Reporting Authority) and the CESE (the NSW Centre for Education Statistics & Evaluation). The second information source is an online survey of program alumni. Results are tested for statistical significance at the 10% level.


10

3. Program Reach To understand the reach of TunED-Up!, we benchmark characteristics of the participating schools against those of New South Wales primary schools in general.8 We use 2015 school profile data, the most recent available.9 Results are tested for statistical significance.

3.1. Identifying participant schools Over 2014-2016, 122 teachers have participated in TunED-Up!. They represent 102 schools. The match between teacher and school is not one-to-one because:

• Some teachers worked at multiple schools • Some schools have had multiple teachers participate over time • Some teachers were not attached to a school; for example the 2016 cohort

included several pre-service teachers. One school was excluded from the analysis as it is based in ACT. It would not be appropriate to compare ACT to NSW schools as they vary systematically. ACT schools are much more likely to be metropolitan and are significantly larger (in student numbers). ACT schools have significantly higher ICSEA scores (which measure the level of socio-educational advantage of students, based on family background). ACT schools have a significantly lower proportion of Indigenous students.

3.2. School characteristics The schools participating in TunED-Up! have a similar size profile as non-participants (average enrolments of 425, compared to 315). Participating schools are relatively more likely to be government schools, and less likely to be Catholic schools, than non-participant schools. There are similar proportions of Independent schools among participating and non-participant schools.

8 We use ‘primary schools’ to include all schools who offer primary level education. This includes so-called ‘combined’ schools who cover both primary and secondary levels. 9 This analysis uses data from ACARA (the Australian Curriculum Assessment and Reporting Authority).


11

Figure 1: Breakdown of school sectors of Tuned-Up! schools compared to the general NSW profile

3.3. School demographics The schools participating in TunED-Up! are representative of all NSW schools in terms of the socio-economic background of families at the school. This is measured using their ICSEA score.10 The average score of participating schools is 1014, compared to 1006 for non-participants. The participating schools are representative of all NSW schools in terms of the proportion of students who have a language background other than English, with 24% compared to 23% for non-participants. The 2016 cohort has a higher proportion of students who have a language background other than English, with 33%. The participating schools have a higher rate of Indigenous student enrolments than NSW schools in general, with 11% compared to 9% for non-participants. The 2016 cohort in particular has a much higher rate of indigenous enrolment, at 16%.

3.4. School location The geographical classification used by the Australian Department of Education and Training is the ‘MCEECDYA geo-location’. Another common measure of location is the ABS Remoteness Area Structure. By either of these measures the schools participating in TunED-Up! represent the same range of locations as non-participant schools.

10 The ICSEA (Index of Community Socio-Educational Advantage) is designed by ACARA (the Australian Curriculum Assessment and Reporting Authority) for the purpose of identifying ‘similar’ schools in terms of socio-economics.

18% 3%

69% 85%

13% 12%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

NSW primaries Tuned-Up! schools

Independent

Government

Catholic


12

Figure 2: Breakdown of MCEECDYA geo-location of Tuned-Up! schools compared to the general NSW profile

Figure 3: Breakdown of ABS Remoteness Area of Tuned-Up! schools compared to the general NSW profile

Dividing New South Wales into smaller regions we see there are some differences in in the distribution of participating schools compared with non-participants.11 In particular, the program has higher participation from schools in North-East NSW, and from Sydney Inner regions, and lower participation from schools in South-East NSW.

11 These regions come from the New South Wales state government’s categorisation of the 11 areas of NSW. It is based on grouping together smaller ASGS Statistical Area 4 boundaries.

58%

39%

2% 1%

58%

38%

2% 2%

0%

10%

20%

30%

40%

50%

60%

70%

Metropolitan Provincial Remote Very Remote

NSW primaries

Tuned-Up! schools

54%

43%

2%

56%

42%

2%

0%

10%

20%

30%

40%

50%

60%

Major Cities Regional Remote

NSW primaries

Tuned-Up! schools


13

Figure 4: Spread across regions of NSW of participating schools compared to non-participants

7% 7% 8% 7% 7% 7%

12% 14% 13%

11% 8%

11%

16%

6%

3%

8%

3%

24%

12%

8%

4% 4%

0%

5%

10%

15%

20%

25%

NSW primaries

Tuned-Up! schools


14

4. Alumni Survey We distributed the TunED-Up! alumni survey online in November 2016 to all past participants from 2014 to 2016 and received 53 responses. Questions were not mandatory. We have indicated the number of responses to each question. Most questions received 53 responses.

4.1. Survey sample and margin of error Responses can be read as representative of the entire alumni cohort 2014-16, within a margin of error of+/-11%. Most questions received 53 responses (number of respondents for each question is indicated below). Figure 5: Breakdown of survey respondents by year of participation (n=53)

Year of participation

Number of participants

Number of survey respondents

Margin of error

2014 19 15 +/-12% 2015 49 21 +/-17% 2016 54 17 +/-20%

4.2. Impact on teachers: The effect of TunED-Up! participation Respondents were asked to rate the TunED-Up! program out of 5 for the following effects:

• Professional skills have been improved (1 = not at all, 5 = very much) • Professional confidence has been increased (1 = not at all, 5 = very much)

2014 28%

2015 40%

2016 32%


15

Respondents gave an average rating of:

• 4.4 out of 5 for improvement of professional skills (n=53) • 4.5 out of 5 for increase in professional confidence (n=52)

TunED-Up! is much more positively rated than the typical music professional development program, which past research has shown teachers rate as 3.7 (impact on skills) and 3.6 (impact on confidence).12 Ratings were lower in the 2016 cohort than in the 2014 and 2015 cohorts, but still significantly higher than for the typical music professional development program. The 2016 cohort was the first to include a number of pre-service teachers (17% of the cohort) and it could be that they do not yet have a strong position on their professional skill and confidence, leading to the lower overall ratings.13 Table 1: Breakdown of program impact ratings by cohort

Year of participation

Improvement in professional skills rating (average)

Improvement in professional confidence rating (average)

2014 4.4 4.5 2015 4.8 4.8 2016 4.0 4.2

Respondent’s open-ended text comments demonstrate how highly the TunED-Up! program is regarded by its participants:

Thanks for letting our school be part of this amazing program. Since I did TunEd-Up everything regarding music education for me changed! I was incredibly inspired, refreshed and fully equipped to go back to school and have instant, successful, fun and quality music lessons. In 20 years of teaching, I have never experienced any professional training like TunEd-Up. TunEd Up provided me with skills, new ideas and practical tools to use both in my classroom and on a larger school scale.

Respondent’s open-ended text comments demonstrate the impact of the TunED-Up! program on participants’ confidence to teach music:

More confidence to teach music across the year group. As well as increasing my personal confidence teaching classroom music I am able to support other staff to teach more music in their classrooms.

12 Petrova, 2012, p. 295. (Results converted from 7 to 5-point scale for comparability.) 13 As the survey was anonymous we cannot confirm this by checking individuals’ responses.


16

I LOVED the fact that we all came from different musical backgrounds and experiences and that we all had so much fun, regardless of skill and ability and this meant that we all returned to our classrooms feeling confident, inspired, motivated and ready for an exciting music teaching journey. I know my classroom music teaching would not be the same without TunED-Up. It has been a highlight of my Primary school teaching thank you! More confident overall in teaching music to my class, willing to take music to share with other classes/stage and using the resources. Have dabbled with garage band in the class but don't have enough iPads. Much more confident delivering program and feel much more empowered to be an advocate for the need for music teaching in the timetable.. .. It was the best type of Professional development that I have participated in during my last 18 years of teaching. All primary teachers should be able to access this know how! Thank you

Participants reported a range of positive impacts for their schools:

I was so inspired from participating in TunED-Up. I immediately went back to school and provided in servicing to the whole staff. Without my inclusion in the TunEd Up course I am sure that my voice would not have been heard and support from the school executive to attend would not have been forthcoming Our school is clearing a classroom as a dedicated primary school music space for the first time! Our whole school now has a structured music program.

Networking with other teachers has enriched my program as we share resources and ideas. …. The facebook sharing page has assisted in keeping up with new ideas and resources. Since taking part in TunEd Up, my ability to integrate music into my classes has increased. More importantly I've been able to encourage other teacher to put meaningful music education back into their classrooms and our school now has RFF music classes for all students. The most positive aspect for me has been a re-awakening of the importance of creativity and culture in my own life and also in the lives of the children and in turn the community I live in. It is no cliché to say TunED Up has changed my life.


17

Participating teacher’s comments reflect the findings of the importance of authentic learning by teachers as found in other recent Australian studies into primary teacher education in the arts.14 Participants also reported some difficulties in applying their new learnings in their schools, which are explored below.

4.3. TunED-Up! participants’ current music teaching positions Most survey respondents were still working at the same school as they had been when attending TunED-Up!. Respondents had been in their current position an average of 8 years. One-third of respondents were in casual or temporary positions, while two-thirds were in permanent roles. Figure 6: Breakdown of survey respondents by if they were still working at the same school as when they attended TunED-Up! (n=53)

Table 2: How many years survey respondents have been in their current position (n=53)

Number of years Percentage of survey respondents Up to 5 years 49% 6-10 years 23% 11-15 years 17% 16+ years 11%

14 Hunter et al, 2014; Russell-Bowie, 2012.

Yes 87%

No 13%


18

Figure 7: Breakdown of current type of teaching position of survey respondents (n=52)

4.4. TunED-Up! participants’ views of the status of music teaching at their schools

About half of past TunED-Up! participants reported that their schools’ classroom music was taught across the whole school. The other half reported that classroom music was taught at some but not all years or Stages; or that there was only extra-curricular music teaching. The proportion reporting their schools have classroom music across the whole school is slightly higher for the 2015 cohort, at 62%. Survey respondents were asked how much time was spent on classroom music activity – for those classes that had any time allocated to music at all. The most common amount of time was 45 minutes (32%), or 30 minutes (30%) per week, followed by those saying there wasn’t a regular amount of time devoted to music (17%).

Permanent staff 63%

Casual or temporary

staff 37%


19

Figure 8: Breakdown of the overall situation of music teaching at respondents’ schools (n=53)

Table 3: Breakdown by cohort of the percentage who say classroom music is taught across the whole school

Year of participation Percentage who say classroom music is taught across the whole school

2014 47% 2015 62% 2016 41%

Figure 9: Breakdown of the amount of time per week devoted to music activity in class (n=53)

Classroom music is taught across

the whole school 51%

There is some classroom music,

but it is not in every year (or

Stage) 43%

Music teaching happens only in extra-curricular

activities 6%

There is no regular time devoted to

music activity per week, 17%

30 minutes, 30% 45 minutes, 32%

1 hour, 15%

1 hour 30 minutes, 4%

2 hours, 2%


20

When asked about the types of music activities they had delivered in the past year, the most common response was extra-curricular music activities, and special events, each done by over three-quarters of past participants. A few more than half report doing regular classroom music teaching (57%). A substantial minority report doing music teaching in RFF (Relief from Face-to-Face) classes (34%). Slightly under a third report doing ‘other’ activities (30%). Most commonly these included supporting and training other teachers; and involvement in multi school activities such as the NSW Department of Education and Communities ‘Arts Alive’ Choral Concert series. Figure 10: The types of music teaching activities respondents have personally delivered in the past year (n=53)

In their open-ended text comments, a number of respondents commented on the difficulties of embedding music teaching at their schools. Many linked this to the fact that music was not treated as a mandatory whole-school subject (and sometimes only taught in RFF):

I currently teach RFF in my school, so all classes have 30-60 minutes of music a week, sometimes combined with drama and dance. However when I go back on class very little music will be taught in the classroom without me on RFF to encourage it. I came back from TunEd Up and people really didn't want to know. I think this stems from workload. There is so much expected from teachers that taking it all in can be a problem. After I did the TunEd Up training (which was the BEST training I have ever done in my long career of intensive Music study) I taught music K - 6 with a

57%

34%

79% 77%

30%

0%

20%

40%

60%

80%

100%

Regular classroom

music

Music teaching in RFF (Relief from Face to Face) classes

Extra-curricular music activities,

such as after school

instrumental teaching or a school choir

Special events such as

assembly performances, open days, the school musical

Other


21

Music room. [The following year] the Primary teachers didn't want Music, and I only had Infants. I am so devastated that next year I have been told that there will be no Music program and Musica Viva which I taught in my program to Infants only will be every 2nd year. This is a problem that many Music teachers face, but keep doing TunEd Up!!!! Music is RFF, and RFF is taught by a number of 1,2 or 3 day a week teachers who all have different stages and classes. Some stages have asked to teach it themselves, however my stage refused, and thus I have not been able to teach classroom music since TunEd Up.

4.5. TunED-Up! participants’ ratings of the resources for music teaching at their schools

Allocation of time and access to classroom room resources such as instruments influenced a teacher’s capacity to teach music.15 Respondents were asked to rate resources for music teaching at their schools out of 5 for the following areas:

• Sufficient time in timetable (1 = very poor, 5 = very good) • Sufficient facilities (1 = very poor, 5 = very good) • Sufficient teaching staff (1 = very poor, 5 = very good) • Support of school leadership (1 = very poor, 5 = very good)

Respondents gave an average rating of:

• 3.0 out of 5 for sufficient time in timetable (n=53) • 3.1 out of 5 for sufficient facilities (n=53) • 2.8 out of 5 for sufficient teaching staff (n=53) • 3.6 out of 5 for support of school leadership (n=53)

There were no significant differences in how each cohort of participants rated their school’s resources for music teaching.

15 Power and Klopper, 2011.


22

5. Impact on Students

5.1. Choosing areas of impact to investigate Past research shows learning music and engaging in musical activity can affect students in many ways. It can build emotional and intellectual skills, cause positive changes in attitudes and behaviours, and improve students’ educational results in non-arts spheres.16 In consultation with the SSO we chose two focus areas: students’ school attendance rates, and their educational outcomes. Time series data on both are available.

5.2. Methodology One approach would be to track individual students, for example individual test results before and after their teacher participated in TunED-Up!. But consultation with past TunED-Up! participants17 suggested it was unrealistic to get individual student data from schools retrospectively.18 We therefore analyse whole-of-school data. For educational outcomes, we use data from Australia’s national standardised testing system, NAPLAN. NAPLAN testing in primary schools occurs at Year 3 and Year 5. Tests cover five domains: spelling, grammar and punctuation, writing, reading, and numeracy. The analysis uses a quasi-experimental design, looking at how school outcomes change over time for participating schools compared to non-participating schools. The focus is on relative change, controlling for initial (pre-participation) test scores. The statistical technique is called General Linear Modelling of repeated measures, using a mixed design of one within- and one between-subjects factor. This analytic technique lets us isolate three types of effects:

• A time trend: do outcomes at individual schools change over time? • A group comparison: are participant schools consistently different in their

outcome levels than non-participant schools, both before and after participation?

16 Martin et al, 2013; Schellenberg, 2004; Smithrim & Upitis, 2005, Winner & Hetland, 2008. 17 N=7. 18 Three main difficulties were raised. Firstly, it would require a large investment of time and effort by past participants to extract data from previous years. Secondly, there would be substantial administrative paperwork to complete to meet data confidentiality regulations, and this was not viable within this study’s timeframe. Thirdly, some said it would be difficult to pick out ‘their’ students (as distinct from all the students in the school) because they taught on a casual and relief-teacher basis.


23

• A relative change trend: do outcomes at participating schools change over time in a way that is different from non-participating schools?

The relative change trend is the focus of this report. If TunED-Up! affects outcomes, we will see that in the relative change trend. The findings control for initial variations between the schools, because we are comparing each school’s outcome measures to its own earlier measures – in effect, each subject is its own control.19 Our analysis cannot control for one potential difference between schools: the non-random assignment of schools to the participating and non-participating groups. Schools and teachers had to actively choose to participate in TunED-Up!. They therefore could differ from non-participants in unmeasured ways, for example in levels of teaching motivation. This in turn might affect our outcome measures.

5.3. Data sources and limitations Data came from two sources: ACARA (the Australian Curriculum Assessment and Reporting Authority), and the CESE (the NSW Centre for Education Statistics & Evaluation):

• ACARA is the main source of school data, including school characteristics and NAPLAN outcomes over 2013-2015.

• CESE provide 2016 NAPLAN data, but for NSW Government schools only. As such we do not have 2016 NAPLAN data for Independent and Catholics schools (or non-NSW schools).

The analysis is restricted to NSW schools.20 A few schools do not have their NAPLAN results included in the datasets.21 We have indicated the sample number for each analysis.

5.4. Attendance rates Recent research finds participation in arts learning in school such as through music teaching can lead to students coming to school more often or remaining in school

19 This assumes that the relevant factors about the subjects is static between test points (e.g. for schools: the broad makeup of their student population). The group comparison measure also provides a test of whether the participant and non-participant schools differ consistently prior to participation, and is reported here where relevant. 20 The one participating school from ACT was in the 2016 cohort, and we were unable to obtain 2016 NAPLAN results for ACT. 21 NAPLAN results are supressed if there are too few students at a school who participate in a test in a given year. Special assistance schools can choose to keep results confidential. Schools that are very new (opened in 2015 or later) are not yet included in the datasets.


24

longer.22 We explore this link here using ACARA data on attendance rates (where 100% = attended every school day). ACARA attendance data is only available for 2014 and 2015. This limits us to analysing the effects for the 2015 cohort. We find that attendance rates drop over this period for both participant and non-participant schools. There is no evidence that participation in TunED-Up! affected attendance rates. Anecdotally, several past participants in TunED-Up! reported their impression that music teaching made some reluctant children more likely to want to attend school, and positive feedback from parents on how music teaching improved their child’s attitude to school. Statistically isolating this effect would require individual child-level data and analysis of changes for those with initially lower attendance levels. This is not possible to do with our whole-of-school level data. Table 4: Attendance rates

Attendance rate (%) Significance Participant school (n=50)

Non-participant school (n=2,299)

2014 2015 2014 2015 94.2% 93.4% 94.3% 93.4% Not significant ACARA data. Attendance rates are for Semester 1 (= Terms 1 and 2).

5.5. Educational outcomes Research has found that music teaching can lead to better educational results in non-arts spheres. In particular, there is evidence that music teaching is associated with better scores in mathematics; and that music teaching is associated with better scores in English (for example, reading skills).23 To test whether there is evidence of this among TunED-Up! participants we need to analyse each cohort of TunED-Up! separately. For each cohort, we look at test scores pre- and post-participation. Where possible we use three time points: the year before participation, the year of participation (TunED-Up! takes place in January and NAPLAN is sat in May), and the year after participation. We do three main comparisons:

• Changes in participating schools’ Year 3 NAPLAN scores, compared to non-participating schools


22 Harris Interactive Inc., 2006; Eason & Johnson, 2013; Martin et al, 2013. 23 Hetland & Winner, 2001; Smithrim & Upitis, 2005; Standley, 2008; Johnson & Memmott, 2007; Broh, 2002; Eason et al, 2013.


25

• Relative improvement in NAPLAN scores for students who sat Year 3 in the pre-participation year, and Year 5 in the post-participation year, compared to the relative improvements in non-participating schools.

This section reports the key statistically significant findings. Detailed tables are available in the Appendix.

5.6. Key findings on educational outcomes The 2014 cohort does not show any significant effects.24 That is, scores at participating schools do not change differently over time than those at non-participant schools. But the 2014 cohort was small (17 schools had NAPLAN data), which will have limited our statistical ability to detect any significant changes. We find several positive associations for the 2015 cohort.25 The strongest evidence is for numeracy scores. Their students’ Year 3 Numeracy scores improve more over 2014-2016 than those of non-participant schools. In addition, their students show greater improvements in numeracy scores from Year 3 to Year 5 than do other NSW schools. Figure 11: Changes in Year 3 Numeracy scores (2015 cohort)

24 This analysis uses ACARA data on NSW government, Catholic and independent schools. It covers 17-18 of the schools in the 2014 cohort. 25 This analysis uses CESE data on NSW government primaries, covering 43 of the schools in the 2015 cohort.

380 382 384 386 388 390 392 394 396 398 400 402

2014 2015 2016

Year

3 N

umer

acy

(mea

n sc

ores

)

TunED-Up participants

Other NSW primaries


26

Figure 12: Relative improvements in Numeracy scores from Year 3 to Year 5 (2015 cohort)

The 2015 cohort also shows positive effects for several other NAPLAN domain areas: their Year 3 scores in spelling, grammar & punctuation, and reading all improve more over time than in non-participant schools. Figure 13: Changes in Year 3 Spelling scores (2015 cohort)

Figure 14: Changes in Year 3 Grammar & Punctuation scores (2015 cohort)

350

370

390

410

430

450

470

490

2014 Y3 2016 Y5

Num

erac

y (m

ean

scor

es)

TunED-Up! participants

Other NSW primaries

390

395

400

405

410

415

420

2014 2015 2016

Year

3 S

pelli

ng (m

ean

scor

es)


Other NSW primaries

400

405

410

415

420

425

430

435

440

2014 2015 2016

Year

3 G

ram

mar

& P

unct

uatio

n (m

ean

scor

es)

TunED-Up participants

Other NSW primaries


27

Figure 15: Changes in Year 3 Reading scores (2015 cohort)

For the 2016 cohort, we can only look at changes over 2015-2016.26 The results show several significant effects, but in a negative direction. Year 3 Spelling scores worsened at participating schools, unlike the trend in non-participant schools. Year 3 writing scores also worsened, again unlike the NSW general trend. These results are highly unexpected, and do not reflect past research. Looking at the figures we see the 2016 cohort started with relatively high scores on spelling and writing – higher than the NSW average.27 It may be that the 2015 scores were outlier scores, and the downward trend we observe in 2016 is due to scores moving back to their usual level. Future analysis of the 2017 scores and beyond may clarify the overall trend. Figure 16: Changes in Year 3 Spelling scores (2016 cohort)

26 This analysis uses CESE data on NSW government primaries, covering 29-30 of the schools in the 2016 cohort. 27 In the General Linear Model analysis, the group comparison measure is significantly different for spelling, but not for writing. In a straight comparison of 2015 scores only, both spelling and writing scores are significantly different.

395

400

405

410

415

420

425

430

2014 2015 2016

Year

3 R

eadi

ng

(mea

n sc

ores

) TunED-Up! participants

Other NSW primaries

385

390

395

400

405

410

415

420

425

2015 2016

Year

3 S

pelli

ng (m

ean

scor

es)


Other NSW primaries


28

Figure 17: Changes in Year 3 Writing scores (2016 cohort)

5.7. Interpreting the educational outcomes findings There are some challenges in interpreting the educational outcomes findings. Results for the 2015 cohort suggest participation in TunED-Up! has positive effects on non-arts educational outcomes including numeracy and also spelling, grammar & punctuation, and reading. However, these are not replicated in the 2014 or 2016 cohort, and the 2016 cohort even shows unexpected worsening in Year 3 spelling and writing scores. One reason to have faith in the positive findings of the 2015 cohort is that it has the most complete data available. The sample size is big enough to pick up significant differences, whereas the 2014 cohort has a very small sample. It also has the full set of three time points, unlike the 2016 cohort which has only two time points at this stage. Looking across the different educational domains, the evidence is strongest for an effect on numeracy. There are several positive results for the 2015 cohort, and data limitations could explain the lack of result for the 2014 and 2016 cohorts. By the same logic, there is also reasonably strong evidence of positive effects for reading, and grammar & punctuation. We need to be more cautious about the findings for spelling, due to the negative results in the 2016 cohort. While the negative results may be due to outliers, we will need 2017 data to confirm an overall trend. By the same token, although there is also a negative result for writing in the 2016 cohort, we need 2017 data to check the overall trend. A final point to bear in mind when interpreting the NAPLAN results is that our alumni survey findings suggest many participants encounter challenges in applying their learnings from the TunED-Up! program when they return to work. It is striking that the cohort with the clearest evidence for positive impacts, the 2015 cohort, is also the one with the highest percentage of teachers reporting their schools have

400

402

404

406

408

410

412

414

416

418

2015 2016

Year

3 W

ritin

g (m

ean

scor

es)


Other NSW primaries


29

classroom music across the whole school (62%, compared to 47% of the 2014 cohort and 41% of the 2016 cohort). It could be that, given the range of past research to find positive effects of music education on student outcomes, we would find stronger evidence on the impacts of TunED-Up! if all participants were able to fully implement the TunED-Up! program.


30

6. Recommendations Based on our findings in this evaluation, we have a number of recommendations for the SSO.

1. Continue the TunED-Up! program TunED-Up! was initiated to address the unmet need for primary school teachers to teach music with confidence and consistency, aligned with the state and national curricula. The evaluation shows the program successfully meets this need. It is rated very highly by participants in terms of improving both their professional skills and confidence, and participants report a range of improvements in music activities at their schools as a result of the program. The program is successful at reaching a wide and representative range of schools. There is evidence to suggest the program has positive flow-on effects on students’ outcomes in non-arts educational outcomes. In the cohort for which we have the most complete data, the 2015 cohort, participation in TunED-Up! is associated with significant improvements in Year 3 NAPLAN results across numeracy, reading, and grammar & punctuation, as compared to trends in non-participating schools.

2. Consider further future data gathering and research Our analysis of the program’s impact on student outcomes has been limited by some data constraints. These included the small size of the first participating cohort of 2014, and the lack of a third time point for NAPLAN results for the 2016 cohort. We recommend re-analysing the results for the 2016 cohort once their 2017 NAPLAN results are available. This analysis used whole-of-school data. The SSO could consider future research using data on individual students. This would allow analysis of changes that might affect only some students – for example, improved attendance among previously reluctant students. It would also allow analysis of variables that are not collected in national data sets, such as student behaviour. Collection of individual student data would be a relatively substantial undertaking, as it is time-consuming. The data framework and process would need to be agreed to by participants prior to their TunED-Up! participation. Data would need to be collected at several time points, the first being pre-participation data (which participants could supply at the time of attending TunED-Up!), followed by data from the year of implementation and ideally the following year also. Future research could also collect more detailed information on aspects of the program’s implementation, such as teachers’ self-ranking of motivation, and the school leadership’s perceptions of program benefits. These could shed light on which


31

aspects of program implementation are the most critical to its success on the ground.


32

References Barton, G. M., Baguley, M., MacDonald, A. (2013) Seeing the bigger picture: Investigating the state of the arts in teacher education programs in Australia Australian Journal of Teacher Education v38 (7) pp 75-90. Broh, B. A. (2002) Linking extracurricular programming to academic achievement: who benefits and why? Sociology of Education v75 (1) pp 69-95. Department of Education, Science and Training / Pascoe, R., Leong, S., MacCallum, J., Mackinlay, E., Marsh, K., Smith, B., Church, T., Winterton, A. (2005) National review of school music education: Augmenting the diminished Canberra: Australian Government Department of Education, Science and Training / Perth: Murdoch University. Eason, B. J. A., and Johnson, C. M. (2013). Prelude: Music Makes Us baseline research report Nashville, TN: Metro Nashville Public Schools. Harris Interactive Inc. (2006) Understanding the linkages Between music education and educational outcomes Reston, VA: National Association for Music Education. Hetland, L. & Winner, E. (2001) The arts and academic achievement: What the evidence shows Arts Education Policy Review v102 (5) pp 3-6. Hunter, M., Baker, W., & Nailon, D. (2014). Generating Cultural Capital? Impacts of Artists-in- Residence on Teacher Professional Learning. Australian Journal of Teacher Education, 39(6), 75-87. Johnson, C. M. and Memmott, J. E. (2007) Examination of relationships between participation in school music programs of differing quality and standardized test results Journal of research in music education v54 (4) pp 293-307. Lemon, N. and Garvis, S. (2013) What is the role of arts in a primary school?: An investigation of perceptions of pre-service teachers in Australia Australian Journal of Teacher Education v38 (9) pp 1-9. Martin, A. J., Mansour, M., Anderson, M., Gibson, R., Liem, G. A. D., Sudmalis, D. (2013) The role of arts participation in students’ academic and non-academic outcomes: A longitudinal study of school, home and community factors Journal of educational psychology v105 (3) pp 709-727. Petrova, I. E. (2012) What makes good music programs in schools? A study of school music across Australia and a comparison with England and Russia Thesis submitted to University of New South Wales.


33

Power, B., & Klopper, C. (2011) The Classroom Practice of Creative Arts Education in NSW Primary Schools: A Descriptive Account International Journal of Education & the Arts v12 (11) pp 1-26. Russell-Bowie, D. E. (2012) Developing Preservice Primary Teachers’ Confidence and Competence in Arts Education using Principles of Authentic Learning Australian Journal of Teacher Education v37 (1) pp 60-74. Schellenberg, E. G. (2004) Music lessons enhance IQ Psychological Science v15 (8) pp 511-514. Smithrim, K. and Upitis, R. (2005) Learning through the arts: lessons of engagement Canadian Journal of Education v28 (1-2) pp 109-127. Standley, J. M. (2008) Does music instruction help children to read? Evidence of a meta-analysis. Update: Applications of research in music education v27 (1) pp 17-32. Winner, E. and Hetland, L. (2008) Art for our sake school arts classes matter more than ever – but not for the reasons you think Arts Education Policy Review v109 (5) pp 29-325.


34

Appendix 1: Full Educational Outcomes Analysis

NAPLAN Results for the 2014 Cohort For the 2014 cohort, we look at changes over 2013-2015. 2013 is the year before participation, 2014 is the year of participation (TunED-Up! took place in January, and NAPLAN is sat in May), and 2015 is the year after participation. We do three analyses:



• Relative improvements in NAPLAN scores for students who sat Year 3 in 2013 and Year 5 in 2015

The results show no significant effects. That is, scores at participating schools do not change differently over time than those at non-participant schools. It should be noted that the 2014 cohort was small, which could limit our statistical ability to detect any significant changes.

Table 5: 2014 cohort, Year 3 NAPLAN scores

Average scores Significance of relative change

Participant schools Non-participant schools 2013 2014 2015 2013 2014 2015

Spelling (n=17 / n=1,915)

407.4 397.9 396.6 414.2 414.6 410.0 Not significant

Grammar & punctuation (n=17 / n=1,915)

427.1 417.1 422.6 428.8 427.0 431.7 Not significant

Writing (n=17 / n=1914)

414.2 395.2 408.5 415.0 402.0 417.0 Not significant

Reading (n=17 / n=1914)

421.2 413.6 426.0 419.1 417.4 424.1 Not significant

Numeracy (n=17 / n=1911)

393.1 394.8 391.4 398.5 402.2 396.8 Not significant

ACARA data. All NSW schools. Table 6: 2014 cohort, Year 5 NAPLAN scores

Average scores Significance of relative change Participant schools Non-participant schools

2013 2014 2015 2013 2014 2015 Spelling (n=18 / n=1,901)

482.3 487.6 486.7 496.8 499.2 498.3 Not significant

Grammar & punctuation

492.9 490.4 491.6 501.5 504.1 500.0 Not significant


35

(n=18 / n=1,901)

Writing (n=18 / n=1,900)

467.9 457.9 470.3 477.5 467.4 476.5 Not significant

Reading (n=18 / n=1,898)

490.8 486.7 491.0 502.2 499.1 495.3 Not significant

Numeracy (n=18 / n=1,1897)

476.4 477.5 482.6 487.3 487.1 490.7 Not significant

ACARA data. All NSW schools. Table 7: 2014 cohort, analysis of changes from 2013 Year 3 students to 2015 Year 5 Students

Average scores Significance of relative change Participant schools Non-participant schools

2013 Y3 2015 Y5 2013 Y3 2015 Y5 Spelling (n=19 / n=1,982)

401.6 484.9 413.9 497.4 Not significant


424.6 490.3 428.9 499.5 Not significant

Writing (n=19 / n=1,980)

407.5 469.7 414.5 475.8 Not significant

Reading (n=19 / n=1,978)

418.5 492.4 419.1 494.7 Not significant

Numeracy (n=19 / n=1,978)

393.5 482.5 398.5 490.0 Not significant

ACARA data. All NSW schools.

NAPLAN Results for the 2015 Cohort For the 2015 cohort, we look at changes over 2014-2016. 2014 is the year before participation, 2015 is the year of participation (TunED-Up! took place in January, and NAPLAN is sat in May), and 2016 is the year after participation. In order to get 2016 results we rely on CESE data, which means this analysis includes only Government schools. We do three analyses:



• Relative improvements in NAPLAN scores for students who sat Year 3 in 2014 and Year 5 in 2016

The results show several significant effects:

• Year 3 spelling results at participating schools show more positive improvement over time


36

• Year 3 Grammar & Punctuation results at participating schools show more positive improvement over time

• Year 3 Reading results at participating schools show more positive improvements over time

• Year 3 Numeracy results at participating schools show more positive improvement over time

• Students at participating schools show more positive improvements in their Numeracy scores from Year 3 to Year 5

Table 8: 2015 cohort, Year 3 NAPLAN scores

Average scores Significance of relative change Participant schools Non-participant

schools 2014 2015 2016 2014 2015 2016


405.0 414.8 415.3 404.6 399.2 408.4 Significant at the 5% level (p=0.044)



Writing (n=43 / n=1,547)

395.6 419.8 414.0 391.1 406.8 409.0 Not significant

Reading (n=43 / n=1,546)




CESE data. Government NSW schools (excludes Independent and Catholic). Table 9: 2015 cohort, Year 5 NAPLAN scores


schools 2014 2015 2016 2014 2015 2016


500.9 489.3 488.2 489.1 487.9 482.7 Not significant


502.5 499.2 501.7 491.7 488.6 493.6 Not significant

Writing (n=43 / n=1,530)

466.7 477.4 470.0 454.9 465.2 461.8 Not significant

Reading (n=43 / n=1,529)

495.4 493.9 492.8 489.1 485.8 485.6 Not significant


486.6 491.6 489.4 477.8 482.1 481.1 Not significant

CESE data. Government NSW schools (excludes Independent and Catholic).


37

Table 10: 2015 cohort, analysis of changes from 2014 Year 3 students to 2016 Year 5 Students


schools 2014 Y3 2016 Y5 2014 Y3 2016 Y5


405.1 488.1 404.0 482.7 Not significant


413.5 501.7 414.2 494.3 Not significant

Writing (n=43 / n=1,565)

395.6 470.0 390.2 462.1 Not significant

Reading (n=43 / n=1,564)

408.7 492.8 407.8 486.3 Not significant


393.9 489.4 393.6 481.3 Significant at the 10% level (p=0.099)

CESE data. Government NSW schools (excludes Independent and Catholic).

NAPLAN Results for the 2016 Cohort For the 2016 cohort, we can only look at changes over 2015-2016. 2015 is the year before participation, 2016 is the year of participation (TunED-Up! took place in January, and NAPLAN is sat in May). In order to get 2016 results we rely on CESE data, which means this analysis includes only Government schools. We do two analyses:



The results show several significant effects, but not in the direction one would expect. Year 3 Spelling scores worsened at participating schools, unlike the trend in non-participant schools. Year 3 writing scores also worsened, again unlike the NSW general trend. Table 11: 2016 cohort, Year 3 NAPLAN scores


schools 2015 2016 2015 2016


419.2 414.6 398.7 408.4 Significant at the 10% level (p=0.058) Negative direction


430.1 417.4 419.9 420.4 Not significant


38

Writing (n=30 / n=1,578)

416.8 406.4 407.0 409.1 Significant at the 5% level (p=0.037) Negative direction

Reading (n=30 / n=1,577)

419.2 410.9 412.7 410.8 Not significant


394.8 390.6 387.4 391.5 Not significant

CESE data. Government NSW schools (excludes Independent and Catholic) Table 12: 2016 cohort, Year 5 NAPLAN scores

Average scores: Significance of relative change Participant Non-participant

2015 2016 2015 2016 Spelling (n=29 / n=1,566)

497.3 493.4 487.8 482.6 Not significant


495.4 501.1 488.6 493.9 Not significant

Writing (n=29 / n=1,566)

467.7 463.0 465.4 461.9 Not significant

Reading (n=29 / n=1,566)

491.9 491.2 485.7 485.9 Not significant


493.2 496.5 482.2 481.0 Not significant

CESE data. Government NSW schools (excludes Independent and Catholic)


39

Appendix 2: Distribution of Participants 121 participants have been through the program over 2014-2016 representing between them 102 schools. This match is not exact because:

• some teachers worked at multiple schools • some teachers were in pre-service training • some schools sent multiple teachers in a single year

10 of the 102 schools have participated in the program in more than one year. Table 13: Breakdown of number of participating teachers and schools for each year

Year Number of individual teachers participating

Number of schools participating

2014 19 teachers. Two of these worked across several schools each.

22 schools.

2015 50 teachers. One of these worked across several schools.

50 schools. One of these schools had multiple teachers attending. One of these was a high school and is excluded from this paper’s analysis.

2016 52 teachers. Four of these worked across several schools each. Nine of these were pre-service.

41 schools. Six of these schools had multiple teachers attending. One of these schools was from the ACT and is excluded from this paper’s analysis.

Table 14: Breakdown of geographic distribution of participating schools for each year – by regions of NSW

Region Percentage of 2014 participating schools

Percentage of 2015 participating schools

Percentage of 2016 participating schools

Sydney – North 10% 10% 10% Sydney – Inner 5% 12% 26% Sydney – South 5% 6% 8% Sydney – South West 0% 2% 5% Sydney – West 5% 14% 5% Sydney – North West 0% 4% 5% North East NSW 48% 24% 10% North West NSW 10% 16% 13% South West NSW 14% 6% 8% South East NSW 5% 4% 3% Central Coast and Newcastle

0% 2% 8%


40

About the Authors BYP Group is a specialist research and evaluation consultancy in the government, civil society, arts and creative industry sectors. Our clients are leading national and international cultural institutions, universities, local councils and government agencies. They include the USA National Performance Network, the Australia Council for the Arts, Arts NSW, Creative Victoria, VicHealth, NSW Health, Transport NSW, Aboriginal Affairs NSW, Arts Centre Melbourne, Museum of Contemporary Art, Sydney Symphony, Opera Australia, Melbourne Symphony Orchestra, Creative Partnerships Australia, Macquarie University, University of Western Sydney, Penrith City Council and others.

For more information, please contact: Jackie Bailey Principal BYP Group W: http://bypgroup.com E: [email protected] T: 0428 576 372



mailto:[email protected]

sydney symphony orchestra tuned-up! evaluation report · evaluation report . authors: rachel...

Documents