aes, darwin 10 th september 2014 traversing the interplay of politics and evaluation: education...
TRANSCRIPT
![Page 1: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/1.jpg)
AES, Darwin 10th September 2014
TRAVERSING THE INTERPLAY OF POLITICS AND
EVALUATION: EDUCATION REFORM IN AUSTRALIA
Assoc Prof Janet ClintonDr Amy GullicksonRuth Aston
Edmund MissonPauline Ho
![Page 2: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/2.jpg)
Introduction
Paper 1: Collaborative Evaluation
Paper 2: Evaluation Methodology
Paper 3: Dissemination of Evaluation Findings
![Page 3: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/3.jpg)
Paper 1: Collaborative Evaluation
Janet Clinton & Edmund MissonUniversity of Melbourne & AITSL
![Page 4: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/4.jpg)
Evaluation context - 1
Overview of the Standards
Map progression: Graduate Proficient Highly Accomplished Lead
![Page 5: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/5.jpg)
Evaluation context - 2
Dual purpose: Improvement
Performance and development Professional learning
Career progression Accreditation – Graduate Registration – Proficient Certification – Highly Accomplished and Lead
Support Materials
![Page 6: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/6.jpg)
Why this evaluation? Evaluation embedded in policy implementation to inform
and guide implementation How?
Collaborative Formative Multi-method Multi-year Utilising existing infrastructures
![Page 7: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/7.jpg)
Considerations Stakeholder communication and engagement Sharing information and findings Timing Alignment of requirements/needs Changing political context Evaluation is designed to add value
![Page 8: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/8.jpg)
Paper 2: Evaluation Methodology
Amy Gullickson & Ruth AstonUniversity of Melbourne
![Page 9: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/9.jpg)
Evaluation methodology
WE ARE HERE
![Page 10: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/10.jpg)
Evaluation methodology
Mixed-Methods Informing policy development =
Evidence Based Policy Implementation
![Page 11: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/11.jpg)
Evaluation phases
Phase 1• Build evaluation
foundation• Establish stakeholder
groups and team• Lit review & program
logic
Phase 2
• National Forum• Stakeholder interviews• Collect existing
documents
Phase 3
• National Survey #1• National online
depository
Phase 4
• Case studies• Data collection round
#2
Phase 5
• Stakeholder interviews• Final data collection
round• National Survey #2
Phase 6
• Triangulation of findings
• Revisit program logic • Draw overall
conclusion
Stage 1- Develop and refine design
Stage 2- Collect evidence
Stage 3- Finalise and make recommendations
![Page 12: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/12.jpg)
Levels of influence ACCREDITATIO
N(GRADUATE)
REGISTRATION(PROFICIENT)
CERTIFICATION(HA AND
LEAD)
PERFORMANCE AND
DEVELOPMENT
PROFESSIONAL LEARNING
SUPPORT MATERIALS
AND RESOURCES
STUDENTS
TEACHERS AND SCHOOL LEADERS
SCHOOLS AND COMMUNITIES
SYSTEMS/SECTORS &
AUTHORITIES
STATES/TERRITORIES
NATIONAL
POLICY
![Page 13: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/13.jpg)
National Forum Interviewees
= 82
Workshop participants
= 174
Government teacher employersCatholic teacher employersIndependent teacher employersRegulatory Authorities
Union
Principal AssociationNational
BodiesDeans of EducationAITSL Board MembersAITSL Board
Alumni
![Page 14: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/14.jpg)
National Survey#2 (June 2015)
National Survey #1
Respondents TotalTeacher 4141School leader 1427Teacher educator 214Pre-service teacher 219Combined 6001
![Page 15: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/15.jpg)
Case Studies
![Page 16: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/16.jpg)
Paper 3: Dissemination of Preliminary Findings
Edmund Misson & Pauline HoAITSL
![Page 17: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/17.jpg)
Driving a dynamic communications strategy to evidence change and impact for the Evaluation Edmund Misson, Pauline Ho
In collaboration with Sam Hussein, AITSL Communications, Online & Social Media
![Page 18: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/18.jpg)
Introduction
The Evaluation is a complex and dynamic process of reform implementation
- involves diverse stakeholders across varying contexts, levels, interests and needs.
Engaging the
Profession
Practising Teachers
School Leaders
Pre-service
Teachers
Teacher Educator
s
Members of key
orgs and agencies
System Leaders
![Page 19: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/19.jpg)
AITSL’s Communications & Social Media Strategy Part of the whole of organisation Universal
Analytics Framework (UAF) tracks the promulgation and engagement of AITSL’s tools and resources
Multiple levels of analytics: CEO & Board Teams’ analytics to track engagement with tools and
resources ‘Deep dives’ to explore engagement of specific
requests ‘Personas’ to understand demographics, behaviours
and horizon scanning of AITSL’s audience
![Page 20: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/20.jpg)
Aim of the Evaluation’s Communications & Social Media Strategy Methodological Rigour: Gain awareness and participation in the key data
collection activities of the Evaluation e.g. National Forum, National Surveys, Case Studies.
Engage the Profession: Encourage professional conversations on the Evaluation’s findings, analysed and reported in a variety of data visualisation ways.
Promulgate Findings: Share findings through various communication channels cross stakeholders to value add to the policy implementation.
Ultimately, the key goal is to increase the effectiveness of the implementation of Australian Professional Standards for Teachers in schools and organisations across jurisdictions and sectors.
![Page 21: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/21.jpg)
Acquisition
Behaviour
Outcomes
Digital Marketing & Measurement Model by Avinash Kaushik
What are we doing to gain stakeholders’
attention to the Evaluation?
What are they doing and how are they
accessing the Evaluation’s activities?
What is the impact and outcomes of
their participation?
Objective: Engage stakeholders of the education profession to participate in the Evaluation and to drive implementation of the
StandardsMeasuring Impact
Engage
Agents
Participate
Reimaging Data
Web analyticsSocial media
analyticsCampaigns
Measu
re
Knowledge translation
![Page 22: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/22.jpg)
Acquisition - Using diversity of acquisition channels
Social Media
FacebookTwitter
Print Collaterals
ReportsJournal
publicationsNewslettersInfographicsFact Sheets
Website & Online
MarketingAITSL E-News
Evaluation webpageInteractive
Infographics
Knowledge translation
![Page 23: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/23.jpg)
6233New visi-tors
2140Return-ing Visi-
tors
desktop tablet mobile0
1,000
2,000
3,000
4,000
5,000
6,000
7,000 6,425
1,133 585
Utility of Devices
8,143 unique visitors
E-ne
wsle
tter
Twitt
er
Face
book
Ning
Googl
e+
Wor
dPre
ss
Blogg
er
Pock
et
Slid
eSha
re
010002000
Social Networks
% of audience accessing the survey 70%
898
121
1,780
1,182
3,490
227
1st interaction
2nd interaction
Evaluation page
50.5% 18.7%
Survey page
62.3% 51.8%
Behaviour – Engaging with our content
Avg. Visit Duration: 07:23
![Page 24: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/24.jpg)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
682 763
912
7893
730 325 276 380
971
969
3721
2277
1543
2045
National Survey (Oct-Nov 2013)
Case Studies 2014 (Mar-May 2014)
Monthly Analytics ReportTracking unique visitors
Behaviour – Engaging with our content
![Page 25: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/25.jpg)
Aug 2013
Sept 2013
Dec 2013
Mar 2014
Apr2014
May 2014
National Survey 1
Research
6,001 respondents
across jurisdictions
SR 1 LAUNCH
Disseminatefindings
Case Studies LAUNCH
Case Studies*closed*
140 submissions
across jurisdictions
June 2014
Stakeholder 1 Fact Sheet –Infographics
Disseminatefindings
Targeted Marketing
Ongoing promulgation
Targeted Campaign
Research Research
August
2014
SR 2LAUNCH
Outcomes – Impact of the campaigns
![Page 26: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/26.jpg)
Short-term- Create awareness of the Survey and Case Studies
Mid-term- Participate in ongoing data collection activities- Inform findings and shape the Evaluation
Long-term- Analyse and reporting of results and findings
Investigate the Impact of the implementation of the Standards on improving teacher quality
Outcomes – Impact of the campaigns
![Page 27: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/27.jpg)
Next Steps
Not purely an online/social media strategy Involves other forms of stakeholder engagement
including workshops, symposiums, forums and meetings
Rethinking about how we deepen our understandings of the Evaluation through other ways of working across jurisdictions, sectors, schools and organisations
![Page 28: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/28.jpg)
Paper 4: Bringing it together
Janet Clinton & Edmund MissonUniversity of Melbourne & AITSL
![Page 29: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/29.jpg)
Final Comments
Adding value
Defining boundaries of collaboration
Incorporating knowledge translation
Transparency
Demonstrating worth
![Page 30: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/30.jpg)
Understanding Collaborative Evaluation
2013- defining the approach and how it was going work Difficult– the literature often promotes it but doesn't really tell
us how It about all about:
RELATIONSHIPSUNDERSTANDING
TRUSTRIGOUR
ROLE DEFINITION
![Page 31: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/31.jpg)
Collaborative Evaluation
How do we truly be collaborative and maintain objectivity?
Relationship• How it evolves• Practically how it works• What does it mean for
methods
Understanding the influences
Rigorous methods that are open and transparent
![Page 32: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/32.jpg)
Understanding the Influences for each Party
Collaboration
Historical
Economics
Social
PsychologicalPolitical
Cultural
Contextual
![Page 33: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/33.jpg)
Have the conversations and act! Relationships tensions Strong personalities Different management
styles Variable flexibilities Extended partners
Conversation---• Open & frank• Lose the emotion• Process a solution
![Page 34: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/34.jpg)
Collaboration about Methods
Forums
Case Studi
es
Survey
Conversations
![Page 35: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/35.jpg)
Model of Objectivity
OBJECTIVITY
• Recognised Standard evaluation framework• Rigorous methodology• Mixed methods• Evidence of analysis at every level• Triangulation clarity• Sophisticated social science analysis• High level of dissemination• Evidence base content• Every step transparent & reproducible
DIAMOND STANDARD APPROACH
Re-analyse -- suggest—advise– reflect—promote - use
Analyse –demand—contact--micro-manage-- change content- little dissemination
High
Low
![Page 36: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/36.jpg)
High-stakes EvaluationIf it doesn’t work
![Page 37: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/37.jpg)
But….
HANDLE WITH CARE
![Page 38: AES, Darwin 10 th September 2014 TRAVERSING THE INTERPLAY OF POLITICS AND EVALUATION: EDUCATION REFORM IN AUSTRALIA Assoc Prof Janet Clinton Dr Amy Gullickson](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d6d5503460f94a4e352/html5/thumbnails/38.jpg)
Questions?