introduction to cem secondary information systems

Introduction to CEM

Secondary Information Systems

Dr Robert Clark

ALIS Project Manager

The Analysis

.

Subject X

0

2

4

6

8

10

4 5 6 7 8

Baseline

Out

com

e

0 2 4 6 8

-ve VA+ve VA

Regression Line (…Trend Line, Line of Best Fit)

Outcome = gradient x baseline + intercept

Correlation Coefficient (~ 0.7)

Residuals

Subject X

Linear Least Squares Regression

Measuring Value-Added – An Example

Low Ability Average Ability High Ability

Baseline Score

A*

U

B

C

D

E

F

G

Res

ult

Alf Bob

Chris+ve (+ 2 grades)

-ve (- 2 grades)

National Trend

‘Average’ Student

The position of the national trend line is of critical importance

Subject A

Subject B

Some Subjects are More Equal than Others….

A-Level (Alis)

E

D

C

B

A

A*

C B A A*

Average GCSE

Gra

de

Photography

Sociology

English Lit

Psychology

Maths

Physics

Latin

> 1 grade

Some Subjects are More Equal than Others….

IB (Alis)

4

5

6

7

C B A A*

Average (I)GCSE Score

Gra

de

Biology

Business and Management

Chemistry

Design Technology

Economics

English_A1

Film

French_B

Geography

History

Mathematics

Music

Philosophy

Physics

Psychology

Spanish_B

Theatre Arts

Visual Arts

F

E

D

C

B

A

A*

Test Score

GC

SE

Gra

des

Art & DesignBiologyChemistryEconomicsEnglishFrenchGeographyGermanHistoryIctMathematicsMedia StudiesMusicPhysical EducationPhysicsReligious StudiesScience (Double)Spanish

Some Subjects are More Equal than Others….

GCSE (MidYIS or Yellis)

1 grade

Standardisation of Residuals

• (Raw) Residuals can be used to examine an individual’s performance

• Standardised Residuals are used to compare performance of groups

• Standardised Residuals are independent of year or qualification type

• For a class, subject, department or whole institution the Average Standardised Residual is the ‘Value-Added Score’

• Standardised Residual = Residual / Standard Deviation (National Sample)

• When using Standardised Residuals then for an individual subject

• 95% Confidence Limit = 2.0 x Standard Error• 99% Confidence Limit = 2.6 x Standard Error• 99.7% Confidence Limit = 3.0 x Standard Error

N

1ErrorStandard where N = number of results in the

group

(for combinations of subjects consult the relevant project)

The Projects

Alis : yr 12/13

Yellis : yr 10/11

Insight : Yr 9

MidYIS : Yr 7/8/9

InCAS : Primary

Pips : Primary

Secondary Age Range Projects

Typical TimelineMeasure BaselineAutumn Term

Prediction

Reports

Collect Results

Value-Added

Feedback

August

September

Baseline Measurement

Baselines

Year 7Year 7

Year 8Year 8(+ additional)(+ additional)

Year 9Year 9

MidYIS Paper test or Computer Adaptive Baseline Test

Year 10Year 10

Year 11Year 11 Yellis Paper test or Computer Adaptive Baseline Test

Year 12Year 12

Year 13Year 13 GCSEAlis Paper test (TDA) or Computer Adaptive Baseline Test

GCSE

A / AS / IB etc

INSIGHTINSIGHT Combines curriculum tests with developed ability (end Y9 & end Y8)

Problems with Key Stage Baselines

1. Not all students have KS baselines

no KS2 / KS3

Foreign Students

(Vocational Students)

(Adult Learners)

2. KS exams do not always represent ‘Start of Course’ ability

Post-16 : Year(s) out or intermediate years

3. Prior Value-Added

Can you add value at every Key Stage ?

Under achievement leading to under expectation

One teacher’s output = another teacher’s input

A level playing field ?

4. Teaching to the test

Does performance represent ability, effort or exam technique ?

5. Aptitude & fluency vs Achievement & knowledge

The Effect of Prior Value Added

Beyond Expectation

+ve Value-Added

In line with Expectation

0 Value-Added

Below Expectation

-ve Value-Added

Average GCSE = 6 Average GCSE = 6 Average GCSE = 6

Do these 3 students all have the same ability ?

• Although Key Stage baselines can be a very good indicator of potential attainment, by themselves they are not sufficient.

• Key Stage baselines are confounded by the effects of prior treatment.

Need for independent, non-curriculum embedded baseline tests

Online Computer Adaptive or Traditional Paper

yrs 12 / 13 +

yrs 10 / 11yrs 7 / 8 / S1 / 9 / S2

The Computer Adaptive Test

• Test performed online – results automatically transmitted to CEM.

• LAN version available if Web access unreliable.• Minimal installation / setup required - if any.• Adaptive – difficulty of questions changes in relation to ability of

student.• Efficient – no time wasted answering questions that are far too

easy or difficult.• Wider range of ability• Less stressful on students – more enjoyable experience than

paper test.• Less demanding invigilation.• Cheaper !

In 2009 / 2010 over 200,000 students across yrs 7-13 sat this test

Try it yourself at www.inturproject.org/demos

Baseline Feedback

Reports, Graphs

& Predictions

IPRs (Individual Pupil Record Sheets)

Look for sections that are

inconsistent

Also available based on MidYIS, Alis & INSIGHT scores

Intake Profiles

Also available based on MidYIS, Yellis and INSIGHT scores

Intake Profiles (Historical)

Intake Profiles – 7 Band for Independent Schools (Midyis)

Predictions

Predictions are available in the following forms:

• Formal Reports (Alis)

• Spreadsheets (All Projects)

• Paris Software (Alis / Yellis / MidYIS)

Feedback is available on Web / CD

Average performance by similar students in past exams

Predictions – MidYIS example

Similar spreadsheets available from Yellis and INSIGHT

Predictions - Alis example

Value-Added Feedback

Reports & Graphs

Value Added Feedback…

Statistical Process Control (SPC) Chart

What is my score ? does it matter ?

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

Year

Subject Summary

Standardised Residual Graph

The Scatter Plot

Baseline Score

Gra

de

Po

ints

Eq

uiv

alen

t

Look for Patterns…

General Underachievement / over achievement ?

Do any groups of students stand out ?

– high ability vs low ability ?

– male vs female ?

Other things to look for…

Why did these students do so badly ?

Why did this student do so well ?

How did they do in their other subjects ?

Independent Schools

Analysis

MidYIS provides…

• Independent schools baseline standardisation

• Independent schools specific predictions & chances graphs

• 7 Band Intake profiles

• Independent schools specific value-added analysis

Alis provides…

• Independent schools specific value-added analysis

Q. Which one should I use – Independent only analysis or all schools analysis ?

A. It depends on what question you are asking…

PARIS Software

PARIS is …..•Software to install and use in school

•An interactive reporting tool

•Included free with Alis / Yellis / MidYIS

PARIS provides …..•Student level reports

•Subject Level reports

•Institution Level Reports

PARIS analyses …..•Potential Performance

•Intermediate Performance

•Actual Performance

Attitudes

There is more to school / college than exams….

• Student attitudes• Student Welfare & Safety• Non-academic activities• Support• Social and personal development• Parental Survey• Induction Survey

Attitudinal MidYIS INSIGHT Attitudinal Yellis Full ALIS

Self Evaluation (Every Child Matters)

Try it yourself at www.inturproject.org/demos

Other Issues

Points to mull over…

• Independence – no agendas

• Transparency of Analysis

• Self Evaluation

• Straightforward and standardised administration

• Prompt Feedback

• Full working hours phone / email support

• Student focus

• Replacement for KS3

• Innovative online adaptive testing available – student experience

• Longitudinal analysis with appropriate error backgrounds

• Non-curriculum embedded baselines available

• Attitudinal surveys available – Every Child Matters…

Dr Robert ClarkAlis Project Manager

robert.clark@cem.dur.ac.uk0191 33 44 193