Statewide  High  School  Exit  Exams  and    Data-­‐Based  School  Improvement:    

A  European  Story  Esther  Dominique  Klein  

BEAR  Seminar  –  UC  Berkeley  –  February  23,  2016  

Program  

What  is  a  “statewide  exit  exam”?  

Why  are  statewide  exit  exams  so  popular?    

One  size  doesn’t  fit  all  -­‐  Different  statewide  exit  exam  formats  

Statewide  exit  exams  and  school  improvement  

State  intenTons…  

…and  school  responses:  Does  the  message  “hit  home”?    

What  is  a  “statewide  exit  exam”?  

Statewide  exit  exams...  

(Defini'on  adapted  from  Kellaghan  and  Madaus‘,  2003,  defini'on  of  public  exams)  

are  exams  taken  at  the  end  of  a  course  of  studies  that  are…  

mandatory  for  graduaTon  

based  on  prescribed  syllabi  in  curriculum  

taken  in  a  common  test  situaTon  

controlled  by  an  agency  external  to  the  schools  

based  on  tasks  or  assignments  that  are  set  by  an  agency  external  to  the  schools  

usually  administered  by  a  naTonal  or  state  government  

Exit  Exams  in  Europe  (as  of  2010)  

state-­‐wide  exams  only  

state-­‐wide  and  school-­‐based  exams  

school-­‐based  exams  only  

no  exit  exams  or  no  common  procedure  

Germany  before  2005  and  aDer  2008  

Länder  with  statewide  Abitur  exams  (Zentralabitur)  Länder  with  school-­‐based  Abitur  exams  Länder  with  a  mix  of  both  

before  2005   a^er  2008  

Curriculum-­‐based  statewide  exit  exams  2002  

New  York  Texas*  

2008  

Mississipi  New  York  Tennessee  Virginia  

2012  

Alabama*  Arkansas  Indiana  Louisiana  Maryland  Massachusegs*  Mississippi  New  York  Oklahoma  Virginia  

(Zabala  et  al.,  2008)   (McIntosh,  2012)  

Why  are  statewide  exit  exams  so  popular?  

In  recent  years  educa'onal  policy  makers  have  recognized  what  was  known  by  the  Imperial  Chinese  court  over  a  thousand  years  ago  (…):  that  a  

na'onal  external  examina'on  system  can  be  a  powerful  instrument  for  moving  schooling  in  a  desired  direc'on.  

Noah  &  Eckstein,  1992  

 

 

10  

SWEE  as  potenJal  “allrounders”  

Achievement  

Accounta-­‐bility  

InnovaTon  

Professional  learning  

•  Formal:  e.g.,  cerTficates,  comparability  

•  Contents:  e.g.,  standards  assurance,  control  of  the  delivered  curriculum  

For  both  students  and  teachers/schools  

ImplementaTon  of  new  curricula  or  new  methods  of  delivery  and  assessment   •  Teachers:  improving  

diagnosTc  skills,  reduce  uncertainTes,  iniTate  reflecTve  thinking  

•  Schools:  External  validaTon  and  evaluaTon,  improving  collaboraTon  and  other  organizaTonal  rouTnes  

Statewide  exit  exams  are  perceived  as  homogeneous    

SWEE  as  a  tool  for  school  improvement  

Heterogenous  exam  formats  

Research  focuses  on  the  effects  of  student  achievements  and  instrucJon  

Research  Project  „Forms,  FuncJons,  and  Effects  of  Statewide  Exit  Exams“,  funded  by  the  German  Research  FoundaJon  

•  Interview  Study  • Document  Analysis  

State  IntenTons  

• QuesTonnaire  Survey  

School  ImplementaTon  

                                     Review  of  Exam  Procedures  in  OECD  countries  ●  Document  Analysis  ●  Email  Interviews  

Exam  Features  

One  size  doesn‘t  fit  all:  

Different  Statewide  Exit  Exam  Formats  

Analyzing  different  SWEE  formats  

• e.g.,  structure  of  secondary  educaTon,  historical  context,  development  of  exam  tasks,  student  choice,  exam  supervision,  marking  procedures,  use  of  exam  results,  ...  

Exam  procedures  

• e.g.,  specificity,  standardizaTon,  focal  /  core  themes,  ...  

Curriculum  

• e.g.,  task  formats,  complexity  /  difficulty,  curricular  validity,  ...  Tasks  

• e.g.,  specificity,  standardizaTon,  and  bindingness  of  marking  schemes,  actual  compliance  with  marking  schemes,  ...  

Marking  

Document  analysis  

Interviews  

Curriculum  /  Task  analysis  

ObservaTon  of  marking  behavior  

...  

DescripTon  of  statewide  exit  exam  procedures  at  the  end  of  ISCED  3A  

17  OECD  countries  in  Europe**    

→  31  naTonal  or  state  authoriTes  

A.  content  analysis  (Eurybase,  countries‘  relevant  websites,  legal  documents)  

B.  expert  interviews  in  some  countries    

(email,  unstructured)  

**  no  federal  countries  except  Germany  

Cycle  1  Study  Procedures  

(Klein  et  al.,  2009;  Klein  &  van  Ackeren,  2011)  

What  sub

jects   • statewide  vs.  

school-­‐based  • core  vs.  specializaTon    

• students‘  choice?   W

hat  type   • wrigen  vs.  oral  

• students‘  choice?  

How  diffi

cult   • different  

difficulty  levels?  

• role  of  difficulty  level  for  final  grade  

• students‘  choice?  

Differences  in  the  organizaJon  

Who

 sets  th

e  pape

rs  

•  private  vs.  state-­‐run  •  bogom-­‐up  vs.  top-­‐down  

•  unit  sexng  the  papers  

How  are  papers  set  

•  guidelines  and  specificaTons  •  quality  assurance  •  standards  assurance  

•  procedures  

Development  of  exam  tasks  

Who

 doe

s  it   •  marking  stages  

•  role  of  class  teacher  •  role  of  second  marking  •  external  marking  •  marker  choice  •  marker  training  

How  is  it  don

e   •  control  •  marking  schemes  •  Standards  assurance  

Marking  

Handling  of  results  Final  grade

 

•  weighTng  •  role  of  school-­‐based  exams  

•  role  of  conTnuous  assessment  

Feed

back  and

 use  

•  System  monitoring  •  Control  and  Accountability  

•  Benchmarking  and  ProfessionalizaTon  

•  Student  selecTon  

High  or  low  

stakes?  

Statewide  exit  exams  as  a  tool  to  support  improvement?  

Purposes  of  statewide  exit  exams  

Achievement  

Accounta-­‐bility  

InnovaTon  

Professional  learning  

•  Formal:  e.g.,  cerTficates,  comparability  

•  Contents:  e.g.,  standards  assurance,  control  of  the  delivered  curriculum  

For  both  students  and  teachers/schools  

ImplementaTon  of  new  curriculua  or  new  methods  of  delivery  and  assessment   •  Teachers:  improving  

diagnosTc  skills,  reduce  uncertainTes,  iniTate  reflecTve  thinking  

•  Schools:  External  validaTon  and  evaluaTon,  improving  collaboraTon  and  other  organizaTonal  rouTnes  

“Educa@onal  Governance”  –  Managing  the  mulJ-­‐level  school  system  

Macro  level  

Meso  level  

Micro  level  

§  Not  one  single  “governing  unit”,  but  instead  mulTple  actors  with  their  own  logics  

§  Impulses  from  other  actors  are  “recontextualized”  (Fend,  2008)  

§  Interdependencies  between  different  actors  

§  “Actor  constellaTons”  à  acTons  must  be  coordinated  between  different  actors  

(e.g.,  Altrichter  &  Maag  Merki,  2010;  Kussau  &  Brüsemeister,  2007)  

External  actors  

Inter-­‐mediary  actors  

How  can  quality  and  effec@veness  be  assured  and  improved  in  a  system  like  this?  

Actual  Outcome  

Target  Outcome  

Comparison  of  target  and  actual  

outcome  

NaTonal  tests  Statewide  exit  exams  

School  processes  

ReflecTon  Strategic  decisions  

for  school  development  

Standards  

Reinforcement  

If    congruent  

If  not  congruent  

InterpretaTon  RecontextualizaTon  

ModificaTon  

Expected  cause  and  effect  model    (Maag  Merki,  2010;  shortened  version)  

24  

Accountability,  support,  incenTves  

Individual,  school,  society,  and  policyfactors  

“Technology”  of  the  exams  

Use  of  the  exams  (School,  department,  teacher  level)  

OrganizaTonal  rouTnes  (School,  department,  individual  level)  

Intended  and  unintended  effects  

of  the  exams  

FuncTons  and  strategies  

ImplementaTon  process  

Governance  system,  System  of  quality  assurance    

Local  administraTon,  district  

Exam  Use  Model    (Klein,  2013,  adapted  from  Visscher,  2002)  

Research  quesJons  IntenJons  of  educaJon  policy  • How  do  policy  makers  in  different  cultural  and  poliTcal  sexngs  want  state-­‐wide  exit  exams  to  affect  schooling?  

Responses  of  school  actors  • How  do  school  actors  respond  to  the  exams,  and  what  factors  on  system,  school  and  subject  level  influence  their  responses?  

26  

Ø  InternaTonal  comparaTve  case  study  of  statewide  exit  exams  at  the  end  of  upper  secondary  educaTon  (ISCED  3A)  

Ø  MulT-­‐method/mulT-­‐perspecTve  approach  (Yin,  2009)  Ø  Exploratory  approach    

Context-­‐free  research  quesTon  à  DescripTon  à  JuxtaposiTon  and  Comparison  à  Hypotheses  (Philips  &  Schweisfurth,  2007)  

The  three  cases  Finland   Ireland   Netherlands  

Exam   Ylioppilastutkinto   Leaving  Cer'ficate  Examina'ons   Eindexamen  vwo  

Exam  type   Statewide  exit  exams  

Statewide  exit  exams  

Statewide  (50%)  und  school-­‐based  (50%)  exit  exams  

StandardizaTon  of  the  exam  procedures  

moderate   high   SWEE:  high  SBEE:  low  

Governance*  LegiTmacy  type  with  Local  Empowerment  

LegiTmacy  /  Bureaucracy  type  with  School  Empowerment    

LegiTmacy  /  Efficiency  type  with  School  Empowerment  

*Schmid  et  al.,  2007;  Glager  et  al.,  2003  

State  intenJons  

Analyzing  state  intenJons  

QuanJtaJve:  •  e.g.,  correlaTon  between  exam  type  and  demographic  data  or  achievement  data  

QualitaJve  research:  •  Document  analyses  (websites,  press  releases,  educaTon  codes)  

 States  with  statewide  high                                      school  exit  exams  have  a  higher  unemployment  rate  and  more  diversity  among  students  à SWEE  are  a  means  to  counteract  

disadvantages  and  inequiTes  in  the  acquisiTon  of  cerTficates  and  access  to  higher  educaTon  

(Warren  &  Kulick,  2007)  

SWEE  are  used  for  „a  higher  transpa-­‐  rency  of  the  requirements,  a  higher  comparability  of  the  achievements,  and  more  equity  in  the  allocaTon  of  cerTficates.  [...]  Peformance  of  the  educaTon  system  as  a  whole:  [...]  Based  on  the  data  from  SWEE,  improvement  measures  can  be  iniTated.“  (h]p://www.standardsicherung.  schulministerium.nrw.de/zp10/ziele/  ;  transla'on  by  me)  

DescripTon  of    state  intenTons  connected  to  statewide  exit  exams  

Content  analysis    

(countries‘  relevant  websites,  legal  documents,  brochures,  literature)  

Expert  interviews    with  representaTves  of  the  exam  

bodies  (N  =  9)    semi-­‐standardized  interview  

protocol  

Cycle  2a  Study  Procedures  

(Klein  2013)  

FuncJons  of  the  exams  

Finland  •  Formal  qualificaTon  of  students  

•  No  agempt  to  influence  school  processes  

•  No  school  inspecTon  •  Data  feedback  as  offer  •  No  development  expectaTon  

Ireland  •  Formal  qualificaTon  of  students  

•  Completely  external  •  No  agempt  to  control  schools  

•  School  inspecTon  with  low  stakes  for  schools  

•  No  development  expectaTon  

Netherlands  •  50%  statewide,  50%  school-­‐based  exams  

•  Exam  results  as  benchmark  for  student  achievements  

•  MulT-­‐dimensional  internal  and  external  evaluaTon  system  

•  Clear  development  expectaTon  

School  responses:  

Does  the  message  “hit  home“?  

•  Monitoring  

•  Policies  •  Strategic  use  •  Professional  Development  

•  Support  conversaTons  •  ReflecTon  •  Curricular  and  instrucTonal  decisions  

 

(Schildkamp  et  al.,  2012)  

Different  users  with  different  logics  

•  AdministraTon  /  data-­‐wise  Leadership    

•  Teachers  as  the  “deliverers”  of  instrucTon  

•  ...    

Different  types  of  data  use  by  schools  

Use  of  SWEE  data  for  school  improvement  

Perceived  usefulness  and  use  of  data  

•  England:  Perceived  benefit  of  retroacTve  feedback  is  low  (Saunders  &  Rudd,  1999)  

•  Germany:  Perceived  usefulness  of  data  is  higher  for  SWEE  than  for  other  tests  (Wurster  et  al.,  2013),  but  the  actual  use  of  the  data  is  not  higher  and  is  restricted  to  decisions  on  the  curriculum  (Wurster  et  al.,  2013;  Wurster,  2015)  

OrganizaJonal  factors  

•  School  organizaTonal  factors  lead  to  differences  in  uTlizaTon  (DeBray,  2005;  DeBray  et  al.,  2000;  Saunders,  2000;  Saunders  &  Rudd,  1999;  Schildkamp  &  Kuiper  2010;  Schildkamp  et  al.  2012)  

•  Schools  need  support  for  data-­‐based  school  development  (DeBray  et  al.,  2000)  

•  Even  with  training  the  use  of  data  from  exit  exams  is  low  (Tymms,  1995)  

Is  the  school  actors’  percepTon  of  the  exam  funcTons  coherent  with  

the  intenTons  of  the  state?    

How  do  school  actors  uTlize  the  exams  for  school  and  classroom  

development?  

Cycle  2b  Study  Procedures  

(Klein  2013)  

Standardized  quesTonnaire  survey  

35  upper  secondary  university-­‐bound  schools  • 15  schools  in  Finland  • 13  schools  in  Ireland  • 7  schools  in  the  Netherlands  • N  =  35  principals  • N  =  385  teachers  

Data  use  at  school  level  

0%  

25%  

50%  

75%  

100%  Benchmarking  

Monitoring  

Vergleich  

Rechensch_L  

EvaluaTon_L  

EvaluaTon_SL  

PublikaTon  

U-­‐entwickl.  

Kursdiff.  

U-­‐ressourcen  

Finland  

0%  

25%  

50%  

75%  

100%  Benchmarking  

Monitoring  

Vergleich  

Rechensch_L  

EvaluaTon_L  

EvaluaTon_SL  

PublikaTon  

U-­‐entwickl.  

Kursdiff.  

U-­‐ressourcen  

Ireland  

0%  

25%  

50%  

75%  

100%  Benchmarking  

Monitoring  

Vergleich  

Rechensch_L  

EvaluaTon_L  

EvaluaTon_SL  

PublikaTon  

U-­‐entwickl.  

Kursdiff.  

U-­‐ressourcen  

Netherlands  

N  =  15   N  =  12  

N  =  6  

Comparison   Comparison  

Comparison  

Teacher  Acc.   Teacher  Acc.  

Teacher  Acc.  

Teacher  Eval.   Teacher  Eval.  

Teacher  Eval.  

Principal  Eval.   Principal  Eval.  

Principal  Eval.  

PublicaTon   PublicaTon  

PublicaTon  

Instr.  Develop.   Instr.  Develop.  

Instr.  Develop.  

Streaming   Streaming  

Streaming  

Instr.  Resources   Instr.  Resources  

Instr.  Resources  

(Klein,  2013)  

Teacher  collaboraJon  in  the  context  of  the  exams  

2,0  

2,5  

3,0  

3,5  

4,0  

A1   A2   A3   B1   B2   C1   C2   A1   A2   A3   B1   B2   C1   C2   A1   A2   A3   B1   B2   C1   C2  

Finnland   Irland   Niederlande  

Means  and

 95%

 con

fiden

ce  inervals  

 

5  point  Likert  scale;  1  =  not  at  all  true;  5  =  totally  true  

A:  IntegraJng  exams  in  lessons  (exchange  of  experience,  synchronize  teaching  of  exam  content)  B:  InstrucJonal  improvement  (using  exams  and  data  to  improve  teaching)  

C:  StandardizaJon  (use  of  older  exam  tasks,  adjustment  state  standards)  

(Klein,  2013)  

Exam  use  at  individual  level  

3,0  

3,5  

4,0  

4,5  

5,0  

Improvement   Assessment   Improvement   Assessment   Improvement   Assessment  

Finland   Ireland   Netherlands  Finland  

Means  and

 95%

 con

fiden

ce  intervals  

Use  of  exam  tasks  and  results  for...  

5-­‐point  Likert  scale;  1  =  strongly  disagree;  5  =  strongly  agree  

***  p  <  .001;  **  p  <  .01;  *  p  <  .05  

MA-­‐B  =  0,29      t  (128)  =  3.11**;  d  =  .31    

MA-­‐B  =  0,89  t  (148)  =  8.77***;  d  =  .92  

MA-­‐B  =  0,20  t  (53)  =  1.78,  n.s.;  d  =  .26  

(Klein,  2013)  

JuxtaposiJon  and  comparison  

Finland   Ireland   Netherlands  

FuncTon  of  the  exams  

Formal  qualificaTon  and  development  

Formal    qualificaTon  

Formal  qualificaTon,  moTvaTon,  and  development  

Relevance    in  schools   lower   higher   higher  

School    data  use  

Individualized    rouTne  

One-­‐dimensional  purposive  

MulT-­‐dimensional  systemaTc  

(Klein,  2013)  

Discussion  

•  Heterogeneous  Exams  à  Heterogeneous  Inten@ons  à  Heterogeneous  Responses  

Ø The  “governing  effect”  of  the  exam  system  becomes  visible  through  the  internaTonal  comparison  

•  SWEE  are  only  used  for  data-­‐based  school  improvement  under  certain  condi@ons  

Ø Different  funcTons  and  purposes  Ø Data  are  used  at  the  organizaTonal  level,  but  to  a  lesser  extent  at  the  individual  level  

Ø Aspects  of  the  macro  level  might  help  change  collecTve  rouTnes  

•  Variance  within  countries  indicates  school-­‐specific  use  Ø Needs  more  in-­‐depth  analyses  

Thank you!