introduction to cem secondary information systems
DESCRIPTION
Introduction to CEM Secondary Information Systems. Dr Robert Clark ALIS Project Manager. The Analysis. Linear Least Squares Regression. Subject X. -ve VA. +ve VA. 02468. Residuals. Regression Line (…Trend Line, Line of Best Fit) Outcome = gradient x baseline + intercept - PowerPoint PPT PresentationTRANSCRIPT
Introduction to CEM
Secondary Information Systems
Dr Robert Clark
ALIS Project Manager
The Analysis
.
Subject X
0
2
4
6
8
10
4 5 6 7 8
Baseline
Out
com
e
0 2 4 6 8
-ve VA+ve VA
Regression Line (…Trend Line, Line of Best Fit)
Outcome = gradient x baseline + intercept
Correlation Coefficient (~ 0.7)
Residuals
Subject X
Linear Least Squares Regression
Measuring Value-Added – An Example
Low Ability Average Ability High Ability
Baseline Score
A*
U
B
C
D
E
F
G
Res
ult
Alf Bob
Chris+ve (+ 2 grades)
-ve (- 2 grades)
National Trend
‘Average’ Student
The position of the national trend line is of critical importance
Subject A
Subject B
Some Subjects are More Equal than Others….
A-Level (Alis)
E
D
C
B
A
A*
C B A A*
Average GCSE
Gra
de
Photography
Sociology
English Lit
Psychology
Maths
Physics
Latin
> 1 grade
Some Subjects are More Equal than Others….
IB (Alis)
4
5
6
7
C B A A*
Average (I)GCSE Score
Gra
de
Biology
Business and Management
Chemistry
Design Technology
Economics
English_A1
Film
French_B
Geography
History
Mathematics
Music
Philosophy
Physics
Psychology
Spanish_B
Theatre Arts
Visual Arts
F
E
D
C
B
A
A*
Test Score
GC
SE
Gra
des
Art & DesignBiologyChemistryEconomicsEnglishFrenchGeographyGermanHistoryIctMathematicsMedia StudiesMusicPhysical EducationPhysicsReligious StudiesScience (Double)Spanish
Some Subjects are More Equal than Others….
GCSE (MidYIS or Yellis)
1 grade
Standardisation of Residuals
• (Raw) Residuals can be used to examine an individual’s performance
• Standardised Residuals are used to compare performance of groups
• Standardised Residuals are independent of year or qualification type
• For a class, subject, department or whole institution the Average Standardised Residual is the ‘Value-Added Score’
• Standardised Residual = Residual / Standard Deviation (National Sample)
• When using Standardised Residuals then for an individual subject
• 95% Confidence Limit = 2.0 x Standard Error• 99% Confidence Limit = 2.6 x Standard Error• 99.7% Confidence Limit = 3.0 x Standard Error
N
1ErrorStandard where N = number of results in the
group
(for combinations of subjects consult the relevant project)
The Projects
Alis : yr 12/13
Yellis : yr 10/11
Insight : Yr 9
MidYIS : Yr 7/8/9
InCAS : Primary
Pips : Primary
Secondary Age Range Projects
Typical TimelineMeasure BaselineAutumn Term
Prediction
Reports
Collect Results
Value-Added
Feedback
August
September
Baseline Measurement
Baselines
Year 7Year 7
Year 8Year 8(+ additional)(+ additional)
Year 9Year 9
MidYIS Paper test or Computer Adaptive Baseline Test
Year 10Year 10
Year 11Year 11 Yellis Paper test or Computer Adaptive Baseline Test
Year 12Year 12
Year 13Year 13 GCSEAlis Paper test (TDA) or Computer Adaptive Baseline Test
GCSE
A / AS / IB etc
INSIGHTINSIGHT Combines curriculum tests with developed ability (end Y9 & end Y8)
Problems with Key Stage Baselines
1. Not all students have KS baselines
no KS2 / KS3
Foreign Students
(Vocational Students)
(Adult Learners)
2. KS exams do not always represent ‘Start of Course’ ability
Post-16 : Year(s) out or intermediate years
3. Prior Value-Added
Can you add value at every Key Stage ?
Under achievement leading to under expectation
One teacher’s output = another teacher’s input
A level playing field ?
4. Teaching to the test
Does performance represent ability, effort or exam technique ?
5. Aptitude & fluency vs Achievement & knowledge
The Effect of Prior Value Added
Beyond Expectation
+ve Value-Added
In line with Expectation
0 Value-Added
Below Expectation
-ve Value-Added
Average GCSE = 6 Average GCSE = 6 Average GCSE = 6
Do these 3 students all have the same ability ?
• Although Key Stage baselines can be a very good indicator of potential attainment, by themselves they are not sufficient.
• Key Stage baselines are confounded by the effects of prior treatment.
Need for independent, non-curriculum embedded baseline tests
Online Computer Adaptive or Traditional Paper
yrs 12 / 13 +
yrs 10 / 11yrs 7 / 8 / S1 / 9 / S2
The Computer Adaptive Test
• Test performed online – results automatically transmitted to CEM.
• LAN version available if Web access unreliable.• Minimal installation / setup required - if any.• Adaptive – difficulty of questions changes in relation to ability of
student.• Efficient – no time wasted answering questions that are far too
easy or difficult.• Wider range of ability• Less stressful on students – more enjoyable experience than
paper test.• Less demanding invigilation.• Cheaper !
In 2009 / 2010 over 200,000 students across yrs 7-13 sat this test
Try it yourself at www.inturproject.org/demos
Baseline Feedback
Reports, Graphs
& Predictions
IPRs (Individual Pupil Record Sheets)
Look for sections that are
inconsistent
Also available based on MidYIS, Alis & INSIGHT scores
Intake Profiles
Also available based on MidYIS, Yellis and INSIGHT scores
Intake Profiles (Historical)
Intake Profiles – 7 Band for Independent Schools (Midyis)
Predictions
Predictions are available in the following forms:
• Formal Reports (Alis)
• Spreadsheets (All Projects)
• Paris Software (Alis / Yellis / MidYIS)
Feedback is available on Web / CD
Average performance by similar students in past exams
Predictions – MidYIS example
Similar spreadsheets available from Yellis and INSIGHT
Predictions - Alis example
Value-Added Feedback
Reports & Graphs
Value Added Feedback…
Statistical Process Control (SPC) Chart
What is my score ? does it matter ?
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year
Subject Summary
Standardised Residual Graph
The Scatter Plot
Baseline Score
Gra
de
Po
ints
Eq
uiv
alen
t
Look for Patterns…
General Underachievement / over achievement ?
Do any groups of students stand out ?
– high ability vs low ability ?
– male vs female ?
Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?
Independent Schools
Analysis
MidYIS provides…
• Independent schools baseline standardisation
• Independent schools specific predictions & chances graphs
• 7 Band Intake profiles
• Independent schools specific value-added analysis
Alis provides…
• Independent schools specific value-added analysis
Q. Which one should I use – Independent only analysis or all schools analysis ?
A. It depends on what question you are asking…
PARIS Software
PARIS is …..•Software to install and use in school
•An interactive reporting tool
•Included free with Alis / Yellis / MidYIS
PARIS provides …..•Student level reports
•Subject Level reports
•Institution Level Reports
PARIS analyses …..•Potential Performance
•Intermediate Performance
•Actual Performance
Attitudes
There is more to school / college than exams….
• Student attitudes• Student Welfare & Safety• Non-academic activities• Support• Social and personal development• Parental Survey• Induction Survey
Attitudinal MidYIS INSIGHT Attitudinal Yellis Full ALIS
Self Evaluation (Every Child Matters)
Try it yourself at www.inturproject.org/demos
Other Issues
Points to mull over…
• Independence – no agendas
• Transparency of Analysis
• Self Evaluation
• Straightforward and standardised administration
• Prompt Feedback
• Full working hours phone / email support
• Student focus
• Replacement for KS3
• Innovative online adaptive testing available – student experience
• Longitudinal analysis with appropriate error backgrounds
• Non-curriculum embedded baselines available
• Attitudinal surveys available – Every Child Matters…
