How Data and Analysis techniques inform effectiveTeaching and Learning ■

Powerful things that schools can now do with their data...

Within School Variation project website at: www.4matrix.org

Should a school ever be surprised

by its Ofsted report?

Question for discussion

Consensus view of group:A school that knows itself well would not be surprised by the findings of a school inspection

“Satisfactory was no longer good enough” – HMCI Gilbert

“Half of schools are failing”- Guardian October 06

11% outstanding, 48% good, 34% satisfactory, 8% inadequate

- Ofsted Annual Report

Are schools doing better or worse than they used to?

Nearly 60% of schools are good or better

Headline figures about ‘standards’ are not always helpful

The data used to judge schools

because:

• School inspections are too brief to add much to these judgements

• A school’s own self-evaluation evidence provided in the SEF tends to be descriptive rather than quantitative

e.g. “We do lots of lesson observations”

RAISEonline (ex-PANDA) data will form the basis of the main judgements about a school’s standards

Issues to do with using National Data to judge schools

• It can come too late in the year to be usefulrather than simply judgemental

• Ofsted global figures are good for Government stats but less useful for school improvement

• Contextual Value Added adjustments are not a precise way to compare one school with another

CVAContextual Value Added adjustment discounts one or more variables (ethnicity, gender etc) so that comparisons can be made on a fairer basis

• It is a statistical technique that really only works with large sample sizes.

• To make CVA accurate enough to make exact comparisons between one school and another it needs to discount every external factor known to affect a pupil’s performance, to leave only those attributable to the influence of the school.

• It has no meaning at class level. A good teacher will aim to do the best for every pupil - and not expect some individual pupils to do less well because of their ethnicity or gender etc.

but…

CVA

Where inspectors say that CVA data shows the school should be doing better, what independent, pupil-level source of measures can a school use to put this judgement into context?

A growing number of schools are able to provide secure alternative evidence arising from their own

pupil-level analysis and research

• There is over 4 times the variation in provision within schools than between schools - source OECD

• Reducing negative variation would raise standards by 10% - source DfES

Within School Variation (WSV)

"We have always known that there is a difference in performance between schools. But what can make a bigger difference is the experience that children have within one school. So a child can do really well in one subject and not do well in another subject. And that can make an even bigger difference to children's life chances than differences between schools." - Jane Creasy, Assistant Director of Research, NCSL

RAISEonline School self analysis

available late in the Autumn term available the day the exam results arrive

inspection could be based on information that is out of date

inspection would be based on the most up-to-date performance information

ownership and control in the hands of Government

ownership and control in the hands of schools

offers an external evaluation of a school’s performance

offers teachers an opportunity to make their own evaluation

evaluates pupils’ performance with national contextual data

evaluates pupils’ performance with school contextual data

forms the basis of Ofsted judgments on the school

provides measures to support the school's own view of its performance

judges leadership on basis of CVA comparisons with other schools

allows a school to show that good leadership equates with successfully tackling negative variation

How would school self-analysis complement RAISEonline data?

What is the better basis on which school leadership should be judged?

High StandardsHigh Achievement and

Low Variation

5 A*-Cs How well every pupil achieves

Some Children Matter Every Child Matters

Headline figures can hide significant pockets of underachievement

Headline figures should show how well every child achieves

Good Leadership = high attainment Good Leadership = evidence of doing the best for every pupil

League table position Measures of Within School Variation

Performance analysis is done by the few and passed to the many

All teachers are involved with analysing their pupils’ performance

Judged by official data Secure school-level evidence of effectiveness

School Improvement = more 5A-CsSchool Improvement = less negative variation

“Schools that are proactive in showing inspectors the evidence of their own

pupil-level analysis and research tend to do better in their inspection.”

- Dr. Mike Treadaway, Fischer Family Trust, Naace ‘Making Information Work’ Conference 27.04.07

Jack is a C level pupil with an F in this subject – should Jack or his teacher try harder?

An analysis of variation can shine a light on the achievement of different groups of pupils

http://www.4matrix.org/example

The ‘Data Confident School’ Toolkit

• The use of pupil-level data analysis and research is relatively underdeveloped at this point in time

• Good schools will be those that can show that they know themselves well, are targeting negative variation, and can show evidence of improvement

http://www.4matrix.org/toolkit/

“The Data Confidence Toolkit is a good starting point for

schools”- Dr Mike Treadaway

Should every school be expected to prove that for them ‘Every Child Matters’ - rather than just those pupils who will get 5 A*-C grades?

Question: