USING CEM SECONDARY DATAPRACTICAL APPLICATIONS IN SCHOOL

APRIL 2011 Bristol Conference

Geoff Davies

CEM Secondary data includes:

• Baseline test data (developed ability)• ‘Predictive’ data including chances graphs• Value-added data• Attitudinal data• Curriculum assessments (Insight/Sosca)• PARIS software programmes

The use of this data needs to allow us to do our best to help every pupil to at least achieve if not exceed their potential.

It may challenge• The culture of ‘my’ school • Accountability policy• Expectations • Staff training in use of data and ability to cope with

data (data overload)• Integrating the data into school procedures,

storage, retrieval, distribution and access • Roles and Responsibilities

Carol Fitz-Gibbon 2001 British Psychological Society

It gradually dawned on me that providing the data to schools was the most important outcome of the effort, far more important than writing research papers…..

The provision of data to practitioners meant that they participated in the research. Indeed they were the only ones who knew the surrounding circumstances for their classrooms, their department, each pupil, each family, etc. They were the major players: the ones who could interpret and learn from the detailed data.

…..there is a need for teacher researcher posts on the Senior Management team with a brief to develop research that is useful. Given time in the timetable thousands of teachers could become active researchers….

Educational research should be a practical reality contacting scientific enlightenment not a mathematical weight lifting exercise

The sense and usefulness of what we are doing induces

creative thoughtfulness.

creative thoughtfulness

QUESTION

IS THIS STIFLED IN THE PRESENT EDUCATIONAL CLIMATE?

I keep six honest serving men

Who taught me all I know

Their names are What and Why and When

And How and Where and Who

Rudyard Kipling The Elephant’s Child

Can school leaders find the time?

Important but not urgent

Urgent but not important

Is it more like? 65%-80% or 15%

Is it more like? 15% or 60%

PRIORITISING

QUESTIONS INSPIRED BY CEM CENTRE DATA• How do we improve our professional judgement?• Do MIDYIS scores tell us more than we think?• Which ability range of pupils gets the best deal from our school?• Which ability range of pupils is served best by a particular subject department?• Can we do action research in our school using historical data?• Can standardised residuals tell us more about a department than just the + or -?• Does learning support work?• What do attitudinal surveys over time give us?• Can we compare standardised scores with comments made by teachers on reports?• Drilling down from SPC charts.• Which pupils are we succeeding with?• Which pupils are we not succeeding with?• What can be done about it?• Is there a pattern?• What are the gender issues?• What are the prior learning issues?• What can we learn from plotting standardised scores over time?• Can SOSCA help with boy/girl issues?• Is key stage 3 teacher assessment sufficiently rigorous?• How can ALIS be used to inform teaching and learning?

WHAT information do I want?WHY do I want it?WHEN do I need it?HOW do I collect it?WHERE can I find it?From WHO do I get it?

The CEM centre ticks many of these boxes for schools

A small selection of questions we will look at

1. Do Midyis scores tell us more than we think?

2. Which ability of students do most or least well in our school (using YELLIS data)?

3. Can we review post 16 pedagogy (using ALIS data)?

4. What did SOSCA teach us about Key Stage 3 assessment?

Do Midyis scores tell us more than we think?

Using Midyis baseline data

Using MidYIS IPRs Booklet.pdf

What do the sections of the test measure?

The Vocabulary component of the test is generally an important element for most subjects. For English, History and some Foreign Languages it is the best. However the Vocabulary score is perhaps the most culturally linked of all the scores. Those who have not been exposed to vocabulary-rich talk or a wide variety of reading material or whose first language is not English are unlikely to have developed as high a vocabulary score as they would have developed in a different environment.

Maths Score

The Maths score is well correlated with most subjects but is particularly important when predicting Maths, Statistics, ICT, Design Technology and Economics.

The Maths section has been designed with the emphasis on speed and fluency, rather than knowledge of Maths. Like the Vocabulary score, the Maths score is a good predictor of later academic performance.

Vocabulary Score

Non-Verbal Score

The Non-Verbal score is composed of the three sub tests: Cross-Sections, Block Counting and Pictures. The Non-verbal score is important when predicting Maths, Science, Design Technology Geography, Art and Drama. It provides a measure of the pupil’s ability in 3-D visualisation, spatial aptitude, pattern recognition and logical thinking. It can give an insight in to the developed ability for pupils for whom English is a second language

Skills Score

In the Proof Reading section pupils are asked to spot mistakes in the spelling, punctuation and grammar of a passage of text. eg mis-spelling of words like ‘there’ and ‘their’.

The PSA (Perceptual speed and accuracy) section asks pupils to look for matches between a sequence of symbols on the left and a number of possible choices on the right. Given enough time most pupils would probably get the answers correct but we are measuring how quickly pupils can find a correct match. The PSA section allows speed to be demonstrated free from the demands of memory.

The Proof Reading and PSA tests are tests for the modern world, and are designed to measure fluency and speed. They rely on a pupil’s scanning and skimming skills, skills that are desirable in examination situations.

Some pupils will display an IPR pattern with significant differences between one or two components of the MidYIS Test.

These can be the most interesting and possibly the most challenging pupils for mainstream classroom teachers.

Scenarios and anecdotal findings

It is when the IPR is placed in the hands of a teacher who knows that pupil that it becomes a powerful tool.

Confidence Limits

The pupil scored 114 on the Vocabulary section. The error bars range from about 105 to 123, about 9 points either side of the pupil’s score. If this pupil was to take this test afresh 100 times, we would expect that 95 of those times the pupil’s score would fall within the range denoted by the error bars

Are the scores significant? Relative to the Average Performance?

Performance in Vocabulary is significantly better than average performance and Maths performance is significantly below average. The error bars for the Non-verbal, Skills and Overall MidYIS scores do cross the line at 100 and hence the pupil cannot be considered to have performed significantly different to the average pupil overall.

Comparing Maths and Vocabulary Scores

The error bars for Vocabulary and Maths do not cross the line at 100 ( av. performance).

A SELECTION OF MIDYIS SCORES FOR ‘WATERLOO ROAD’ !!

Vocabulary Maths   Non Verbal Skills   MidYIS Score

St. Score Band St. Score Band St. Score Band St. Score Band St. Score Band

Surname Sex                    

A F 81 D 110 B 108 B 112 A 94 C

B F 128 A 107 B 105 B 94 C 120 A

C M 106 B 121 A 103 B 90 D 114 A

D F 107 B 84 D 96 C 107 B 96 C

E M 96 C 90 D 130 A 91 C 92 C

F F 86 D 86 D 120 A 74 D 84 D

G F 100 B 115 A 80 D 103 B 108 B

H F 121 A 96 C 114 A 86 D 111 A

I M 92 C 100 C 96 C 123 A 95 C

J M 100 C 105 B 100 C 99 C 102 B

K M 128 A 132 A 114 A 131 A 133 A

L M 76 D 70 D 74 D 73 D 71 D

What do I need to know/do to teach this (difficult) class of twelve pupils

Why would this be a very challenging class to teach?

These are real anonymous scores from a number of schools around the UK

Vocabulary scores significantly lower than other component scoresSecond language? Deprived areas? Difficulty accessing curriculum.? Targeted help does work. Seen in nearly all schools. Worth further diagnosis

Vocabulary scores significantly higher than other component scoresGood communicators. Get on. Put Maths problems in words?

Mathematics significantly higher than other scoresFrom Far East? Done entrance tests? Primary experience?

Mathematics significantly lower than other scoresPrimary experience. Use words and diagrams? Sometimes difficult to change attitude..

Low Mathematics scores with High Non-verbal ScoresUse diagrams. Confidence building often needed

Pupils with non-verbal scores different from others – High Non-verbal ScoresFrustration? Behaviour problems? Don’t do as well as good communicators or numerate pupils?

Pupils with non verbal scores different from others – Low Non-verbal ScoresPeak at GCSE? A level ?Pupils with low Skills scoresExams a difficulty after good coursework?High Skills ScoresDo well in exams compared with classwork?The Average PupilThey do exist!High scores throughoutAbove a score of 130 puts the pupil in the top 2% nationallyLow scores throughoutBelow a score of 70 puts the puil in the bottom 2% nationally

Sharing the MidYIS Information within School

Once you have received your MidYIS feedback you need to decide who will be privy to which information. Some schools decide to keep the data within the senior management team, others with Heads of Department and/or Heads of Year, some share with all staff and what about pupils and their parents?

Use you MIS systems to put the data where it matters

MidYIS data can be useful:

•to indicate reasons for student learning difficulties and may go some way to explain lack of progress, flag up causes for underachievement and even behaviour problems.

•for all teachers and support staff. It can help to support professional judgement and give a better understanding of the progress students make at school and their potential later performance.

•to refer to for pupil reviews, writing reports, meeting parents, monitoring progress and interim assessments.

2. Which ability of students do most or least well in our school (using YELLIS data)

Which ability range of pupils gets the best deal from our school?Which ability range of pupils is served best by a particular subject department?Can standardised residuals tell us more about a department than just the + or -?

Using standardised residuals in a different way

0.2

-0.6 -0.5-0.2 -0.3

-1.3

0.1

-0.1

0.1

-1.1

0.1

-0.1-0.3

0.4

-0.5 -0.5-0.9

-0.5

0.3

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

Art

& D

esig

n

Des

ign

& T

echn

olog

y

Eng

lish

Eng

lish

Lite

ratu

re

Fre

nch

Geo

grap

hy

His

tory

Mat

hem

atic

s

Med

ia S

tudi

es

Mus

ic

Rel

igio

us S

tudi

es

Sci

ence

: G

CS

E

Sci

ence

: G

CS

E A

dditi

onal

SC

Art

& D

esig

n

SC

Geo

grap

hy

SC

His

tory

SC

Rel

igio

us S

tudi

es

Voc

App

lied

Sci

ence

Voc

Hea

lth &

Soc

ial C

are

Ave

rag

e S

tan

dar

dis

ed R

esid

ual

CONTRAST THIS

0.0 0.0

0.4

0.9

0.10.2

1.4

0.4 0.3

-0.1

0.5 0.6

0.2

0.8

-0.1 -0.2

0.3 0.4

0.9

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

Art

& D

esig

n

Bus

ines

s S

tudi

es

Des

ign

& T

echn

olog

y

Dra

ma

Eng

lish

Eng

lish

Lite

ratu

re

Fre

nch

Geo

grap

hy

Ger

man

His

tory

Hom

e E

cono

mic

s

ICT

Mat

hs

Mus

ic

Phy

sica

l Edu

catio

n

Rel

igio

us S

tudi

es

Dou

ble

Sci

ence

Wel

sh

SC

IC

T

Ave

rag

e S

tan

dar

dis

ed R

esid

ual

WITH THIS

yellis exercise.doc

As on the last two slides CEM provides value added charts using average standardised residuals for departments

We are going to show how standardised residuals can be used in a different way in your school

1. Review of post 16 pedagogy (using ALIS data)

SUBJECT A 2008

Why the improvement?

PEDAGOGY….ALIS surveys

SUBJECT A 2005

SUBJECT A 2007

SUBJECT A 2008

DO NOT GET TOO EXCITED!

We have made a comparison of perceived teaching methods as analysed by ALIS in 2004-5 with those in 2007-8. Some subject areas have appeared to change their methods radically. Others have not. Though the samples are small it is an interesting exercise to try to correlate it with the departments statistical process charts over that period. One would like to say that changes in the variety of teaching methods result in improvement but the evidence is a little tenuous so far.

SUBJECT

Exercises

exampapers

essays

Reading

discussions

notes dictated

own notes

handouts

Practical work

Using AV

IT

Researching

Presenting

helping students

original work 2005 to 2008

SIGNIFICANTCHANGE

ART YES L L M L LBIOLOGY NO L LBUSINESS YES M L M LCHEMISTRY NO LCOMPUTING YES L L M M L MDRAMA NO M L MFOOD YES L L M M M MPRODUCT DESIGN YES L L L M LENGLISH NO LFRENCH YES M M M L MGEOGRAPHY YES M M M MHISTORY NO L LICT YES L L L L M M MMATHS NO M M MMUSIC YES M M M M M M MPE YES M M M M M M MPHYSICS YES M M M M M M LRE NO L MWELSH YES M M M M M

YES 11NO 7MORE (M) 54 5 4 1 3 4 5 6 5 2 3 2 3 2 6 3LESS (L) 29 3 3 2 6 3 1 2 1 2 1 0 0 2 3 0

4. What have we learned from SOSCA?

GENDER ISSUES DYFFRYN TAF KEY STAGE 3

BOYS

BOYS

BOYS

GIRLS

GIRLS

GIRLS

OVERALL

OVERALL

-0.4

-0.2

0

0.2

0.4

0.6

0.8

English Maths Science

SUBJECT

AV

ER

AG

E S

TA

ND

AR

DIS

ED

RE

SID

UA

LS

MID

YIS

TO

KS

3

BOYS

95%CONFIDENCE

95% CONFIDENCE

GIRLS

95% CONFIDENCE

95% CONFIDENCE

OVERALL

95% CONFIDENCE

95% CONFIDENCE

GRAPH 1

AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO KEY STAGE THREE TEACHER ASSESSMENT

GENDER ISSUES DYFFRYN TAF SOSCA

BOYS

BOYS

BOYS

GIRLS

GIRLS

GIRLS

OVERALL

OVERALL

OVERALL

95% CONFIDENCE

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

Reading Maths Science

SUBJECT

AV

ER

AG

E S

TA

ND

AR

DIS

ED

RE

SID

UA

LS

MID

YIS

TO

SO

SC

A

BOYS

95%CONFIDENCE

95% CONFIDENCE

GIRLS

95% CONFIDENCE

95% CONFIDENCE

OVERALL

95% CONFIDENCE

95% CONFIDENCE

GRAPH 2AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO SOSCA

GRAPH 1 AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO KEY STAGE THREE TEACHER ASSESSMENT

GENDER ISSUES DYFFRYN TAF KEY STAGE 3

BOYS

BOYS

BOYS

GIRLS

GIRLS

GIRLS

OVERALL

OVERALL

-0.4

-0.2

0

0.2

0.4

0.6

0.8

English Maths Science

SUBJECT

AV

ER

AG

E S

TA

ND

AR

DIS

ED

RE

SID

UA

LS

MID

YIS

TO

KS

3

BOYS

95%CONFIDENCE

95% CONFIDENCE

GIRLS

95% CONFIDENCE

95% CONFIDENCE

OVERALL

95% CONFIDENCE

95% CONFIDENCE

GRAPH 2AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO SOSCA

GENDER ISSUES DYFFRYN TAF SOSCA

BOYS

BOYS

BOYS

GIRLS

GIRLS

GIRLS

OVERALL

OVERALL

OVERALL

95% CONFIDENCE

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

Reading Maths Science

SUBJECT

AV

ER

AG

E S

TA

ND

AR

DIS

ED

RE

SID

UA

LS

MID

YIS

TO

SO

SC

A

BOYS

95%CONFIDENCE

95% CONFIDENCE

GIRLS

95% CONFIDENCE

95% CONFIDENCE

OVERALL

95% CONFIDENCE

95% CONFIDENCE

Using MIDYIS and SOSCA puts the school in a strong position to improve its professional judgment of teacher assessments at Key stage 3. Statutory testing disappeared in Wales some five years ago.Comparing the value added for MIDYIS to SOSCA and MIDYIS to KS 3 teacher assessment shows up some interesting data

Schools who depend on teacher assessment data only to measure value added from Key stage 3 to Key stage 4 need to be aware of the pitfalls. The use of SOSCA data in this exercise highlights that

http://www.tes.co.uk/article.aspx?storycode=6062337see

Subjective views of pupils - as well as pressure from parents - make model unreliable, warns Professor Tymms.

The differences appear to relate to the types of assessment used in the various subject areas. English and Welsh use extended writing for teacher assessment which is more likely to have subjective judgments. What we have learnt from this is that despite a moderation process built on portfolios of work for teacher assessment, it is not sufficient in isolation.Computer adaptive tests such as SOSCA and the resulting value added information from MIDYIS are more informative in a diagnostic sense than levels produced by teachers for statutory assessment.SOSCA has also been used to investigate any changes in reading from baseline testing. A high correlation was found between the London Reading score given to pupils on entry the MIDYIS score and the SOSCA reading test.

PITFALLS

1. Tracking developed ability measures over time.

2. Looking at average standardised residuals for teaching sets.

3. Effect of one result in a small group of students

REGRESSION TOWARDS THE MEAN

Pupils with high MidYIS scores tend to have high SOSCA scores but not quite as high. Similarly pupils with low MidYIS scores tend to have low SOSCA scores, but not quite as low.  It is a phenomenon seen in any matched dataset of correlated and normally-distributed scores, the classic example is a comparison of fathers' and sons' heights.  Regression lines reflect this phenomenon - if you look at the predictions used in the SOSCA value-added you can see that for pupils with high MidYIS scores their predicted SOSCA scores are lower than their MidYIS scores, whereas for pupils with low MidYIS scores their predicted SOSCA scores are higher than their MidYIS scores.  

DIFFERENCE SOSCA-MIDYIS MATHS

-50

-40

-30

-20

-10

0

10

20

50 70 90 110 130 150 170

MIDYIS MATHS STANDARDISED

SO

SC

A-M

IDY

IS D

IFF

ER

EN

CE

DIFFERENCE

Stu

den

ts

Ad

mis

sio

n N

o.

Mat

hs

Tes

t K

3 W

a

Yel

lis

Sco

re

Yel

lis

Ban

d

YE

LL

IS M

AT

HS

BA

ND

PR

ED

ICT

ION

MA

TH

S

YE

LL

IS G

CS

E P

RE

DIC

TIO

N M

A

WJE

C/G

CS

E 0

1840

1 R

esG

F

WJE

C/G

CS

E 0

1840

2 R

esG

F

WJE

C/G

CS

E 0

1840

3 R

esG

F

GC

SE

Sta

nd

ard

Res

idu

al M

a

a 90019 6 62 A A B B A 1.20

b 90090 7 B B

c 90045 6 63 A B B B B 0.10

d 90063 7 64 A A B B B 0.10

e 90166 6 48 B B B C C 0.40

f 90123 7 70 A A A A B -0.40

g 90129 6 47 C C B C C 0.50

h 90146 6 59 B B A B A 1.40

I 90047 7 62 A A B B B 0.20

j 90115 7 67 A A * A A* 1.70

k 90004 6 46 C B B C B 1.50

l 90164 7 65 A A A B A* 1.90

m 90099 7 70 A A A A A* 1.50

n 90011 7 61 A A A B A 1.30

o 90112 7 66 A A B A A 0.80

p 90058 6 70 A A B A A 0.50

q 90150 7 72 A A A A A 0.40

r 90127 6 52 B B B C B 1.00

s 90030 6 58 B B B B B 0.50

t 90050 7 71 A A A A A 0.40

u 90016 6 69 A A B A B -0.40

v 90174 7 74 A A A A A 0.20

w 91165 6 62 A B B B B 0.20

x 90109 7 63 A B B B B 0.10

y 90138 7 47 C B B C B 1.40

z 90122 7 60 A A * B A 1.30

ab 90009 7 60 A A A B A 1.30

ac 90169 7 79 A A * A* A -0.20

ad 90153 6 56 B B B B B 0.70

ae 90010 7 64 A B B B A 1.00

af 90154 7 61 A C B B B 0.30

Total 1323 1868 109 105 201 156 12 190 20.90

Number of Results 31 30 30 30 31 30 2 29 30

Mean 42.68 62.27 3.63 3.5 6.48 5.2 6 6.55 0.70

Mean Grade 6.00 B B B B B B

Marksheet Name : SUBJECT REVIEW

Marksheet Group : 11S1

Export Date : 04/10/2005

CLASS REVIEW

BEWARE PITFALLS

INTERPRETATION

SexScore (Band)

Raw Residual

Standardised

Residual REVISEDM 53 (B) 5.4 (B/C) 6 (B) 0.6 0.5 0.5M 38 (C) 4.5 (C/D) 3 (E) -1.5 -1.1 -1.1F 36 (D) 4.4 (C/D) 3 (E) -1.4 -1.0 -1.0M 48 (C) 5.1 (C) 5 (C) -0.1 -0.1 -0.1F 52 (B) 5.3 (B/C) 6 (B) 0.7 0.5 0.5F 65 (A) 6.1 (B) 7 (A) 0.9 0.7 0.7M 70 (A) 6.4 (A/B) 3 (E) -3.4 -2.5 M 38 (C) 4.5 (C/D) 4 (D) -0.5 -0.4 -0.4F 40 (C) 4.6 (C/D) 5 (C) 0.4 0.3 0.3F 70 (A) 6.4 (A/B) 7 (A) 0.6 0.4 0.4F 44 (C) 4.8 (C) 6 (B) 1.2 0.9 0.9M 56 (B) 5.6 (B/C) 5 (C) -0.6 -0.4 -0.4

5.3 (B/C) 5.0 (C) -0.3 -0.2 0.0

Predicted Grade Achieved Grade

SUBJECT M

creative thoughtfulness

PLEA

DON’T LET THE SYSTEM DESTROY THIS

USING CEM SECONDARY DATAPRACTICAL APPLICATIONS IN SCHOOL

APRIL 2011 Bristol Conference

Geoff Davies