INSTITUTIONAL REPORTING L E O N E N U R B A S A R I , A N U
PA M E L A S A R LY , AC U
H I E D I W I L K I N S O N , U S C
Institutional Reporting from the Australian Graduate Survey
Leone NurbasariPlanning and Performance MeasurementJuly 2014
Presentation Overview• Key Performance Indicators• Program (Course) Review reports• Discipline reporting on CEQ/PREQ• Graduate destination reporting• Comments analyses• Other reporting
3
Key Performance Indicators• ANU Strategic Plan• College Operational Plans and KPIs
4
KPI generation• Raw unit record data at 6 digit FOE, assigned to College scores based on
taught disciplines (rolling 5 year average, minimum 10% of load taught by that College)
• Benchmarking against Go8 data that is weighted according to ANU discipline weightings
5
6
Example of some underlying 6 digit FOE data and calculations for KPI generation
Program (Course) Review Reports• Program reviews • Data warehouse
reports – CEQ, Employment & Further study
• Supplemented by additional detailed analyses (including item level results)
• Supplemented by anonymised open-ended comments
7
Discipline reporting – CEQ/PREQ• Pseudo colleges (custom groupings of 2, 4 and 6 digit FOEs)• Pseudo departments (also custom groupings of FOE)• Benchmarking – Go8 and national, based on ANU taught FOE (excludes non
ANU disciplines)• Percentage of individual student-item response in
disagreement/neutral/agreement categories, aggregated to scales
8
9
Graduate destination reporting• Data warehouse
reports – employment rates, sector, industry, salary, employer, job search
• Cohort-specific analysis
• Ad-hoc data requests
10
Comment analyses• CEQuery with charting via Excel• Various student cohorts and org. units• Adopting Geoff Scott’s method of proxies for
importance and quality
11
Other reporting• Topic Papers – comprising data
from various surveys and student feedback sources, eg. HDR Satisfaction Report
• Ad-hoc reporting from internal pivot tables – GDS, CEQ, PREQ
12
AGS Data and Reporting at ACU
Pamela Sarly | Acting Manager, Statistical Analysis & Surveys
2014 Survey Manager Information Forum17 July 2014
A bit about ACU…
1900
1963
1991Good
SamaritanSisters
Teacher Training in NSW, VIC, QLD and
ACT
Merger of education colleges, forming
ACU 2014
One of the fastest
growing universities in
Australia
1857
TeacherTraining
in NSW and VIC
ONE OF THE FIRST
EDUCATION PROVIDERS IN
AUSTRALIA
Data as at 11 July 2014
29,513 Enrolments22,245 EFTSL2,839 international students
12 Schools across 4 Faculties
12,952 commencing students16,561 continuing students
Staff FTE• 1,051 Academic• 982 Professional
490 Higher Degree Research Students6,263 Postgraduate Students21,934 Undergraduate Students826 Non-Award Students
Australia's leading Catholic university which is supported by
more than 2,000 years of Catholic intellectual tradition.
AUSTRALIAN GRADUATE SURVEYData Governance at ACU
• A central contact for surveys: o Office of Planning and Strategic Management
• Publish aggregated data through internal sites:o Staff Site (www.acu.edu.au/opsm)o SharePoint site
• Data updates communicated to relevant staff via email
AUSTRALIAN GRADUATE SURVEYInstitutional Reporting
• Strategic Plan - Traffic Light Report
• Quarterly Report
• TEQSA Risk Indicators
• New Course Application
• Course Review and Renewal
• Published in July and December
• Progress against the University’s Strategic Goals and Key Result Areas
TRAFFIC LIGHT REPORT
STRATEGIC SCORECARD
• Published in January, April, July and October
• High level quantitative data that is linked to Strategic Goal 1 (Student Experience)
QUARTERLY REPORTStrategic Plan Goal 1: Student Experience
• Use of SPSS TAS for qualitative data analysis and reporting
• Better linkage to Student Evaluation on Unit and Teaching surveys
• Future of AGS – to inform our new Strategic Plan 2015+
AUSTRALIAN GRADUATE SURVEYFor the future…
Thank youQuestions?
COMMUNICATING RESULTS FOR EFFECTIVE DECISION
MAKING
Australian Graduate Survey
Note that the figures contained within this presentation are fictitious.
Why do we need to communicate results?
AGS results are used at USC to:Inform strategic decision makingAssess program and institution performanceInfluence improvements to programsReview learning and teachingDevelop internal policies
Why use visualisations?
“Numbers have an important story to tell. They rely on you to give them a clear and convincing voice.”
Stephen Few
Choosing the right medium
Textual
Visual
Oral
Tapping into human nature
Over 50% of the cerebral cortex is involved with the processing of visual inputs.
-Lu & Dosher
There are over 130 million sight receiving cells in the retina.
- Vries 2011Source: Atranik 2011
Tableau
Knowing your audience
USC’s Organisational Chart
Know your goal
Enable insight
"The purpose of visualization is insight, not pictures"
Shneiderman 1999
GDS Dashboards Flowchart
CEQ Dashboards Flowchart
Levels of Analysis
Graduate Destination Survey Benchmark Group Dashboard
Course Experience Questionnaire Longitudinal Dashboard
Graduate Destination Survey Faculty Dashboard
Graduate Destination Survey Program (Course) Dashboard
Course Experience Questionnaire Program (Course) Dashboard
How dashboards are used
Are they being used?
Who is using them?
What are they using them for?
Director - Quick reference point
Program leaders - Program reviews
Executive - Marketing and recruitment - Identifying strengths and
weaknesses - Information sharing
What next?
Reports by Field of Education
Beyond Graduation SurveyUniversity Experience
SurveyInternal SurveysInteractive DashboardsQualitative Data
Visualisations
Bibliography
Atranik.org 2011, http://antranik.org/functional-areas-of-the-cerebral-cortex.
Azzam, T & Evergreen, S 2013, Data Visualization, Part 2: New Directions for Evaluation, Number 140, Google eBook.
Card, S, Mackinlay, J & Shneiderman, B 1999, “Readings in Information Visualization - Using Vision to Think”, Morgan Kaufmann, Massachusetts.
De Vries, J 2011, The Five Senses, Random House, Victoria.
Few, S 2006, Information Dashboard Design: The effective visual communication of data, O’Reilly, California.
Lu, Z and Dosher, B 2013, Visual Psychophysics: From Laboratory to Theory, MIT Press, ISBN: 9780262019453.
University of Leicester (Learning Development) 2012, Presenting numerical data, <http://www2.le.ac.uk/offices/ld/resources>