+

Working with SMART2Workshop for leaders in the use of SMART2

Workshop presentation 20/09/2010

Andrew Fraser and Carmel Kriz

+Objectives

Gain greater understanding of using data from an inquiry frame of reference

Develop greater familiarity with the SMART2 web-based application

Undertake an initial analysis of school SMART2 data

Identify strengths and issues to explore further

Develop strategies for effective leadership in the analysis and use of information

+

Where was I last weekend?

+ Which gives you an accurate answer?

+

Data MattersLeading learning with an inquiry habit of mind

+

Setting the context

A climate of accountability

+Using data to learn

What comes to mind when you hear the word:

DATA

(60 seconds)

+Using data to learn

What comes to mind when you hear the word:

INFORMATION

(60 seconds)

+A commitment to action

(p. 16)

Schools need reliable, rich data on the performance of their students because they

have the primary accountability

for improving student outcomes.

Good quality data supports each school to

improve outcomes for all of the students.

+

What is accountability?

+What is Accountability?

Earl and LeMahieu, 1997

As cited in “Leading learning in a data-rich world” Earl and Katz (2006: 10)

Accountability is

the conversation

about what the information means

and

how it fits with everything else

that we know

and

about how to use it

to make positive changes

+

Opening up productive inquiry

Some principles

Internal motivation: Teachers improve through learning their craft, which happens best in PLCs which have access to valid data about achievement.

Data: the divide…

Judgment

Questions

Inquiry Habit of Mind

Professional Learning Communities Requires

imagination

What happens?

The underlying assumptions

External motivation: Teachers are motivated to improve by the idea that someone is watching over them, judging and visiting consequences on them if the targets aren’t met. Thanks to John DeCourcey

How do you use data in your school?

+

The data, used well, frame the right The data, used well, frame the right questions;questions;Used poorly, they rush to judgmentUsed poorly, they rush to judgment

+Asking the right questions require imagination

+Types of data

Outcome

• External tests (SC, HSC)• A – E Reporting Grades• Retention rates• Enrolment trends• Behavioural data

Demographic

• Gender• LBOTE• ESL• SES• AEDI• Languages• Mobility• Staff age/mobility

Process

• Observation of practice• Descriptive data• Policy• Quality of practice• Professional development• School organisation

Perceptual

• Parent survey• Focus groups• Satisfaction surveys• Student attitudinal data• Community perceptions

Data worth looking at:

“The value of the data emerges only when analysis provides insights that direct decisions for students.”

Stephen White,

Beyond the Numbers, 2005

+From Data to Professional Knowledge

Data are

Making useful information

Developing informed professional knowledge

+Data Information Knowledge

Information Actionable knowledge

Types of Decisions• To identify or clarify a problem • Set and monitor progress towards goals• Address individual and group needs• Monitor and evaluate practices• Validate proposals for change • Assess whether student needs are being met• Strategically allocate resources• Adapt a new practice to fit the situation

+Collaboration

“Data analysis is a team sport.”Doug Reeves

Develops team thinking

Promotes insights that numbers alone can’t produce

Provides a forum for legitimizing practice

A characteristic of “Schools that Learn”

Wonderful learning?

He scored 42/60 on the

midyear testLazy?

Superb teaching?

Dreadful cheat?

Evidence:

What does this indicate?

Construct:

Top mark in the class, student has been studying well, highly motivated…Lowest he’s ever achieved, no sign of any preparation, smart student, easy testWhole class is showing improvement, class average is better than ever gained before…Stole the answer sheet from the teacher’s desk the day before…

Almost any evidence can be an indicator of many different

constructs – finding the most

productive questions is the

art of data analysis.

Thanks to John DeCourcey

+

Using data well

Use it to ask the right questions

‘Triangulate’ the construct – what other way do I have of looking at it?

Decide on a direction – go top-down, or bottom-up

Slice the data different ways: mean, top, bottom, time, individual map

Test the ‘Strength’ of the evidence: is it a sound link to the construct?

Be suspicious of yourself: ask the next question

Thanks to John DeCourcey

+Three Principles of Data Analysis

Exploring and determining the antecedents for success

Collaborating with colleagues

Embracing Accountability - Learning from our data

Connecting Cause and Effect Data at the program/initiative level

+Some questions for consideration

“What does this ask us about how teaching and learning are going in our school?”

“What do we need to do about it?”

“What do we need to learn?”

+

The NAPLAN Scale

Achievement Scale

Band 1

Band 2

Band 3

Band 4

Band 5

Band 6

Band 7

Band 8

Band 9

Band 10

Year 3

Year 5

Year 7

Year 9

Each Year level Student Report shows 6 of the bands.

Year 9 Reports show bands 5 to 10

Year 7 Reports show bands 4 to 9

Year 5 Reports show bands 3 to 8

Year 3 Reports show bands 1 to 6

Students are at the national minimum standard

Achievement Bands

Students are below the national minimum standard

Year 3

Band 1

Band 2

Band 3

Band 4

Band 5

Band 6

Band 3

Band 4

Band 5

Band 6

Band 7

Band 8

Year 5

Year 7

Year 9

Band 4

Band 5

Band 6

Band 7

Band 8

Band 9

Band 5

Band 6

Band 7

Band 8

Band 9

Band 10

Students are proficientStudents working at proficiency would be in the top 2 achievement bands for the respective year level.

Students that are in Band 1 are deemed at operating at below minimum standard.

Students working at minimum standard would be in the second lowest band.

NAPLAN Scales

There are five separate national scales, one each for:

ReadingWriting

Spelling,Grammar/Punctuation

and Numeracy.

Note: There are no more benchmarks. Students that are in the lowest band are deemed to be operating below minimum standard.

Introducing …

1

2

3

4

5 6

7

8

Creating groupsSelect: Manage Groups

+

Using SMART2 to analyse NAPLAN data

+Using Means and Standard Deviations

Good starting point

Gives overall school performance

Helps frame questions for further investigation

Gives difference in mean between the school and the state

Can be found in Means &Standard Deviations

Question : How significant is the difference from the state????

+ Means and Standard Deviations

When student numbers are small, summaries based on mean (average) scores can be misleading. In small groups, means can be affected by a couple of high performing or low performing students.

Means sometimes obscure real differences within a group, even when the numbers are large. For example, a school with concentrations of both educationally advantaged and disadvantaged students may find that the school mean actually describes very few of its students.

For schools with year cohorts less than 5 students no school means and deviations will be displayed.

+ Means and Standard DeviationsSchool Reports

Select aspect – Reading or Numeracy

Compare the mean for the full cohort against the state mean

Compare full cohort with boys, girls, ATSI

Compare means in different strands of literacy (writing, reading etc) or numeracy (number, measurement )

+ Means and Standard DeviationsSchool Report

+Significance of difference in means

School mean – state mean, divided by the state SD.

This calculation is about substantive meaning of any difference. It is not a test of statistical difference.

+Significance of difference in means Rule of thumb method :

> 0.5 Well above state

0.5 > 0.2 Above state

0.2 > -0.2 Within state

-0.2 > -0.5 Below state

< -0.5 Well below state

+Further investigationsIs the Significant difference of the mean:Because of the results of a particular

group? Eg.boys, girlsBecause of the results of a particular class?

Eg. class group that has had 4 different teachers in one year

The same for both year levels (3 & 5, 7 & 9)?

Has this been the pattern in previous years?

+Means and Standard Deviations

Significant difference of the mean allows for comparison between year levels

Means and Standard Deviations allow for creation of means table for special groups

Means and Standard Deviations allow for comparison with school groups

+Trend Data

+Percentage in Bands

Choose Percentages in Bands from main menu.

View for full cohort, boys, girls, ATSI, or custom groups

Record bands where difference from the state is significant.

* less students in top band or top two bands

* more students in the bottom band (below National Minimum Standard)

* more students in the second bottom band (at the National Minimum Standard)

OR of course the reverse

+ Percentage in Bands

+Further Investigations

Check to see if there is there is a significant difference in performance between the strands e.g.

In Literacy - between writing and reading

In Numeracy - between Number/Patterns & Algebra and Data/ Measurement /Space and Geometry

More detailed investigation can be carried out through the Item Analysis.

Questions need to be asked re pedagogy in these areas – this could form part of the School Improvement Plan.

+Percentage in Bands

Record names of students in the bottom two bands

Create a group of these students. (Create /Edit and Delete Groups)

Go to Item Analysis to look at which items all this group have correct /incorrect

+Percentage in Bands

Check names of students in particular bands.

Are there any surprises?

Follow up responses of individual students whose band placement is of concern by going to Student Analysis

+Value added Percentages in Bands may be of use for

schools where they have tried to increase the movement of students into higher bands.

Care must be taken when using this information with schools who have less than 10 students. In schools with small numbers each individual student is worth a large percentage and this may impact greatly on the student numbers in a band.

+ Value added - Percentage in Bands

+Student Growth (Years 5, 7, 9)

Select Aspect – Reading or Numeracy

Select Student Growth

+ Student Growth

+ Student Growth Arrows indicate growth for individual students (3 to 5, 5

to 7, 7 to 9).

Length of arrow indicates amount of growth.

Orange arrows indicates growth that is greater than or equal to expected growth for starting point.

Blue arrows indicate growth that is less than expected growth for starting point.

Downward blue arrow indicates student’s score less than their last score

Click on arrows for students’ names and scores.

Read from left to right – lowest performing students in the cohort’s last NAPLAN on the left – highest performing on the right.

+Student Growth Check which students have made the most

growth.

Is it the lowest third, middle third or upper third based on prior performance?

Is there any relationship between amount of growth and any teaching emphasis or intervention for a particular group?

Is there a group of students who have not made expected growth and who are clustered together?

Make a group of these students and examine their responses in Item Analysis and/or Student Analysis looking for common errors

+ Student Growth

+Student Growth

Check the tables on lower half of screen:

Average scaled score growth

Expected growth

+Student Growth

The tram lines on the main graph indicate the growth made by 75 %, 50 % and 25% of students in the state.

The middle table on the right hand side indicates the % of students in the school in the bottom 25th percentile, and the top 25th percentile and between the 25th and 75th percentile.

Does the school’s growth fit the anticipated 25 50 25 pattern ?

+Item Analysis

Use this table together with Item Analysis Table in the Main Menu.

Print: Report - Analysis by Questions (for required aspect – reading or numeracy and required year level)

Report – Analysis by Questions provides information on % of students choosing correct / incorrect options.

When investigating questions that a large numbers of students answered incorrectly, it is often helpful to ascertain whether a number of them choose the same incorrect answer.

+

+Item Analysis

Go to Item Analysis

Select Year Level

Select Aspect

Sorting of items in different ways.

+Item Analysis

+Item Analysis Sort by School % Correct

Investigate items where school percentage correct is below state by 10% or more.

Using Report – Analysis by Questions to check for common response errors.

+Item Analysis Sort by Syllabus Outcomes

Check: Which Items have/have not been answered correctly by the majority of students in the Year.

Check for common incorrect responses.

+Item Analysis

Select Filter by Substrands (drop-down menu)

Investigate performance on items assessing particular skill

Using Report check for common response errors

Check out the actual item using SCAN.

+

+Item Analysis

can be viewed graphically - choose CHART

+Item Analysis

Select : Filter by Group

Investigate aspects of Item Analysis for various subgroups – boys, girls, custom groups.

+Item Analysis

Compare performance of Year groups in a particular skill area.

Are there any links items in this set of items?

What are the skills year groups are are finding difficult ?

+Leading, Inquiring, TransformingDeveloping an inquiry habit of mind

Being data literate

Creating a culture of Inquiry

+School leadership teams

Familiar with, and competent in, analysing and understanding NAPLAN data

Takes an active role in the analysing and use of the data with staff to bring about measurable improvements in student learning.

+ Towards 20112010

Term 3

Term 4

2011

Analyse NAPLAN data Identify issues to follow up Engage others in the analysis Examine other data

Formulate a whole-school response – School Improvement Plan

Develop specific strategies to implement Determine measureables, milestones,

indicators of improvement

Implement plan Explore changes – cause and effect Work with SMART 2 Teaching Strategies

Monitor and evaluate – evidence of change

Critique impact

Inquire

Plan

Implement

Reflect

+Monitoring change

How will you know change is or has occurred?

What evidence will you use to monitor the transformation of learners?

What evidence will you use to monitor the transformation of learning?

+Acknowledgements

Dr Philip Pettit and the Canberra – Goulburn CEO

Rosemary Vellar and Sydney CEO

Dr John DeCourcy

NSW Educational Measurement and School Accountability Directorate

Australian Government National Partnerships Agreement

+Further assistance

Further support and advice can obtained by contacting the Catholic Schools Office, Broken Bay:

Andrew Fraserandrew.fraser@dbb.catholic.edu.au

Carmel Kriz carmel.kriz@dbb.catholic.edu.au

Or your school’s consultant.

+Reflecting on questions

“What does this ask us about how teaching and learning are going in our school?”

“What do we need to do about it?”

“What do we need to learn?”

Cause and Effect data

Effect Data – outcomes or results in student learning and achievement

Cause Data – professional practices (adult actions) that create specific effects or results

+ Teacher inquiry and knowledge-building cycle to promote valued student outcomes

•What do they already know?•What sources of evidence

have we used?•What do they need to learn

and do?•How do we build on what

they know?

•How effective has what wehave learned and done beenIn promoting our students’Learning and well-being?

•How have we contributed toexisting student outcomes?

•What do we already know that wecan use to promote valued

outcomes?•What do we need to learn to do to

promote valued outcomes?•What sources of evidence/knowledge can we utilise?

What are our students’ learning needs?

What has been the impact of our

changed actions?

What are our own learning needs?

Design of tasks andexperiences

Teaching actionsH. Timperley, A Wilson, H Barrar & I Fung (2007)

Teacher Professional Learning and Development: Best Evidence Synthesis Iteration

Wellington, New Zealand: Ministry of Education

http://educationcounts.edcentre.govt.nz/goto/BES

+ Leader inquiry and knowledge-building cycle to promote valued student outcomes

What knowledgeand skills

do our teachers andstudents need?

What has been the impact of our changed actions

on teachers and students?

What knowledge and skillsdo we as leaders need?

What are our teachers’ learning needs?

What has been the impact of our

changed actions?

What are our own learning needs?

Develop leadershipknowledge

and refine

leadership skills.

Engage teachersIn new

learning experiences

Modified from H. Timperley (2010)

+Data Teams

Guidelines for effective Data Teams1. Have collaborative teams2. Provide adequate time for collaboration3. Engage in collective inquiry4. Focus on the cause and effect data5. Post graphs and charts so they are visible6. Subscribe to action orientation and

experimentation7. React to our data with sound instructional and

curricular decisions8. Implement an effective communication9. Are results driven10. Are devoted to continuous improvement

+

Data literacy – a thinking process

Standing back and thinking about what you need to know and why

Collecting or locating the necessary data

Finding ways to link data sources

Ensuring that data are worth considering

Being aware of their limitations

Thinking about what the results mean

Systematically considering an issue from a range of perspectives so that you feel that you have evidence to explain, support, and also challenge your point of view.

+Data Information Knowledge

ONLY WHEN

Shaped

Organised

Embedded in a context

That gives meaning and connectedness.

Adapted from: van Barneveld (2008) and Earl and Katz (2006)

Which effect data to focus on?“Will this piece of data help a classroom teacher change curriculum, assessment and instruction and thus improve student achievement?”

Douglas Reeves