2010 compass

18
Where youth learn skills to thrive in life. They... find their SPARK learn leadership and tips for HIGH SCHOOL SUCCESS nurture a GROWTH MINDSET hone their GOAL MANAGEMENT SKILLS “I will teach those who didn’t come to Compass, to be a better leader and to show their spark more. And also tell them about other things that they missed.” 2010

Upload: phungbao

Post on 03-Jan-2017

216 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 2010 Compass

 

Where youth learn skil ls to thrive in l i fe. They...

find their

SPARK

learn leadership

and tips for

HIGHSCHO

OL

SUCCESS

nurture a

GROWTH

MINDSET

hone their

GOAL

MANAGEM

ENT

SKILLS

“I will teach those who didn’t come to Compass, to be a

better leader and to show their spark more. And also tell them

about other things that they missed.”

2010

Page 2: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT  

Applied  Survey  Research        2    |    P a g e

The Compass Program

2010 A pilot of Sequoia Union High School District in partnership with Thrive Foundation for Youth

Evaluation Report

Prepared  by:  Thrive  Foundation  for  Youth  and  Applied  Survey  Research  

THRIVE FOUNDATION FOR YOUTH

1010 El Camino Real, Ste 250 Menlo Park, CA 94025 Phone 650.485.2944

Contact: Nicole C. Taylor, President & CEO

www.thrivefoundation.org www.stepitup2thrive.org

Principal  Researcher:  Applied  Survey  Research  

APPLIED SURVEY RESEARCH

991 W. Hedding St., Ste 102 San Jose, CA 95126

Phone: 408.247.8319

Contact: Lisa Colvig-Amir, MA, Director of Evaluation

Vanessa Haug, MS, Senior Research Analyst Patricia Reyes, MPH, Project Manager

www.appliedsurveyresearch.org

Page 3: 2010 Compass

 

Applied  Survey  Research                                                                                                                                                                              3    |    P a g e

Table of Contents

TABLE  OF  CONTENTS  ..............................................................................................................................  3  

1.    INTRODUCTION  ...................................................................................................................................  4  

Program  Description  ................................................................................................................................................  4  

Study  Description  ....................................................................................................................................................  5  Evaluation  Questions  ...................................................................................................................................................  6  Methods  .......................................................................................................................................................................  6  

2.    DEMOGRAPHICS  OF  YOUTH  ............................................................................................................  8  

Characteristics  of  Compass  Students  ........................................................................................................................  8  

3.    STUDENTS’  EXPERIENCES  IN  THE  COMPASS  PROGRAM  ........................................................  9  

Findings  from  the  Student  Survey  ............................................................................................................................  9  Did  Compass  students  feel  there  was  any  value  or  benefit  in  participating  in  the  Compass  program?  ......................  9  Did  Compass  students  feel  that  the  Compass  program  provided  a  supportive  environment?  .................................  10  Did  Compass  students  feel  that  topics  were  meaningful?  .........................................................................................  10  

4.    STUDENTS’  CHANGES  IN  BELIEFS  AND  INTENTIONS  ...........................................................  11  The  Five  Framing  Concepts  ........................................................................................................................................  11  

5.  STUDENTS’  ACADEMIC  PERFORMANCE  IN  9TH  GRADE  .........................................................  12  School  Success  Techniques  ........................................................................................................................................  12  Academic  Outcomes  ..................................................................................................................................................  12  Attendance  Outcomes  ...............................................................................................................................................  15  

6.  SUMMARY  OF  FINDINGS  .................................................................................................................  16  

APPENDIX  A:  THEORY  OF  CHANGE  .................................................................................................  17  

APPENDIX  B:  VIGNETTES  ...................................................................................................................  18  

Page 4: 2010 Compass

ATTENDANCE  OUTCOMES  

 

Applied  Survey  Research                                                                                                                                                                              4    |    P a g e

1. Introduction

Compass  is  a  24-­‐day  summer  program  offered  by  Sequoia  Union  High  School  District  (SUHSD)  in  San  Mateo  County  for  entering  9th  grade  students  who  have  been  below  grade  level  and  are  most  at  risk  of  dropping  out  of  school.  Held  daily  on  four  school  campuses,  each  day  of  the  program  consists  of  five  classes:  Summer  Reading,  Math,  High  School  Success,  Leadership,  and  P.E.  In  2009,  SUHSD  contacted  Thrive  Foundation  For  Youth  (Thrive)  to  assist  in  improving  the  program.  Thrive  promotes  a  research-­‐based  theory  of  change  about  ways  to  redirect  youth’s  life  trajectories  in  order  to  reach  their  full  potential.  The  theory  of  change,  called  Step-­‐It-­‐Up-­‐2-­‐Thrive  (www.stepitup2thrive.org)  has  four  components  that  are  based  on  unique  bodies  of  research:  

(Benson,  2008):  Sparks  are  inner  passions  or  interests  that  catalyze  personal  growth  on  a  Sparks  -­‐  trajectory  towards  thriving.  Youth  who  have  spark(s)  and  have  adult  support  in  growing  them  are  less  likely  to  experience  depression  or  engage  in  acts  of  violence  against  others,  and  are  more  likely  to  have  a  sense  of  purpose,  receive  higher  grades,  and  have  better  school  attendance  rates.  

(Dweck,  2006):  Mindset  is  the  belief  about  one’s  intelligence,  abilities,  and  personality.  Mindset  -­‐  Those  who  believe  their  intelligence,  abilities,  and  personality  are  innate  and  carved  in  stone  have  a  “fixed  mindset,”  whereas  those  who  believe  that  these  qualities  about  themselves  can  be  developed  through  effort  and  practice  have  a  “growth  mindset.”  By  fostering  a  growth  mindset,  youth  are  more  resilient  in  the  face  of  challenges  and  have  a  love  of  learning,  thus  leading  to  greater  life  success.    

:  Reflecting  on  one’s  strengths  and  identifying  risk  factors  with  the  help  Reflect  on  Thriving  Indicatorsof  a  supportive  adult  can  lead  to  more  thoughtful  planning  towards  the  pursuit  of  goals.  Youth  build  their  self-­‐reflection  skills  in  order  to  explore  areas  that  they  want  to  develop  and  identify  factors  that  stand  in  the  way.    

(Lerner,  2006  ):  GPS  (i.e.,  goal  management  skills)  is  a  set  of  skills  that  enable  youth  to:  select  GPS  –  goals,  pursue  strategies  to  attain  goals,  and  navigate  around  obstacles.  When  youth  build  their  goal  management  skills,  they  are  more  likely  to  thrive,  increasing  in  positive  youth  development  outcomes  and  decreasing  in  risk  behaviors.  

Program  Description  Program  Description      The   Thrive   team   partnered  with   teachers   and   consultants   to   design   Step-­‐it-­‐up-­‐2-­‐Thrive   curriculum   that  integrated   the   above   elements.     To   assist   teachers   in   the   delivery   of   their   Compass   course   curriculum,  Thrive   partnered  with   several   Bay   Area   youth   development   groups,   such   as  Math   and   English   Complex  Instruction   Coaches   and   Live   in   Peace   (L.I.P).     Brief   descriptions   of   the   revised   academic   classes   are  provided  below.  

 

 

Page 5: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

5  

 

 –  This  class  trained  students  in  key  study  skills  which  included  Cornell  note  taking,  High  School  Successbinder  preparation,  expository  reading  techniques,  goal  management  skills,  ways  to  seek  help,  test-­‐taking,  memorization  tips,  and  practice  in  classroom  public-­‐speaking.  These  study  skills  were  integrated  into  each  week’s  theme  which  highlighted  a  specific  element  of  brain  development  or  research  on  thriving  in  adolescence.  During  the  first  week  of  class,  students  were  introduced  to  adolescent  brain  growth,  specifically  around  Dr.  Carol  Dweck’s  research  from  Stanford  University  on  Growth  Mindset.  The  second  week  of  class  focused  on  emotions  and  learning  emotional  regulation  techniques.  In  the  third  week,  youth  applied  knowledge  about  the  brain  to  goal  management  practice,  integrating  recent  Dr.  Richard  Lerner’s  research  from  the  Institute  for  Applied  Research  in  Youth  Development  at  Tufts  University.  The  concluding  weeks  of  class  covered  brain  care  with  an  emphasis  on  sleep,  memory,  drugs,  and  plans  for  addressing  challenges  to  high  school  success.  

 –  This  class  introduced  youth  to  the  Search  Institute  research  about  their  personal  Leadership  Classspark.  The  purpose  of  the  class  was  to  help  students  identify  their  personal  dreams  and  catalysts  for  growth,  and  tie  these  interests  to  high  school  opportunities.  The  curriculum  was  designed  to  develop  leadership  skills.  Once  a  week,  Live  In  Peace,  an  East  Palo  Alto  organization  that  works  to  reduce  youth  violence,  presented  a  class  lesson  that  considered  approaches  to  thriving  in  challenging  environments.  The  themes  included  Dreams  and  Dream  Thieves;  Friendship  or  Fearship;  Anger,  Fear  &  Pain;  and  managing  that  "stress"  balloon  inside  each  of  us.  Leadership  practice  included  a  focus  on  styles,  living  by  a  moral  compass,  and  conflict  resolution.

 –  Teachers  assisted  students  in  their  summer  reading  and  writing  skills.  They  were  Reading  Classasked  to  incorporate  concepts  of  thriving  (sparks,  growth  mindset,  indicators  of  thriving,  and  goal  management)  in  the  discussions  of  book  characters.  

–  The  purpose  of  the  class  was  to  give  all  youth  a  sense  that  they  could  succeed  in  math.  Math  Class  The  program  was  designed  to  focus  on  new  algebraic  information  and  problem-­‐solving  skills  that  were  not  dependent  on  a  youth’s  prior  competence  with  math  facts.  In  preparation  for  Compass,  teachers  were  taught  a  method  for  working  with  students  in  groups,  which  takes  advantage  of  the  diverse  strengths  of  all  students.  This  method,  called  Complex  Instruction,  was  developed  by  Dr.  Pope  at  Stanford  University’s  School  of  Education.    In  2010,  SUHSD  pulled  the  lowest  performing  math  students  out  of  this  class  and  they  received  an  alternate  form  of  instruction.    Those  students  had  two  periods  per  day  of  this  alternate  math  approach  and  skipped  the  reading  class.  

Study  Description  Study  Description      Once  the  Step-­‐it-­‐up-­‐2-­‐Thrive  curriculum  was  finalized,  Thrive  reached  out  to  Applied  Survey  Research  (ASR),  a  non-­‐profit  social  research  firm,  to  conduct  a  two-­‐year  evaluation  of  the  Compass  program  (2009  and  2010)  to  determine  the  extent  to  which  students  and  teachers  benefited  from  the  intervention,  and  the  ways  in  which  Thrive’s  theory  of  change  may  hold  true.  The  evaluation  in  2009  was  exploratory  since  the  Compass  Re-­‐Design  was  in  its  first  year  of  inception,  and  considered  to  be  a  pilot-­‐test  of  the  effectiveness  of  a  newly  developed  curriculum.    Findings  associated  with  the  2009  evaluation  (i.e.,  those  obtained  from  student  and  teacher  surveys,  as  well  as  student  and  teacher  focus  groups)  were  therefore  

Page 6: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

6  

 

instrumental  in  determining  the  extent  to  which  Compass’  curricular  activities  were  beneficial  to  both  students  and  teachers.    Ultimately,  a  number  of  programmatic  refinements  were  made  in  2010,  ranging  from  the  content  of  the  curriculum  (e.g.,  expanding  topical  areas)  to  the  delivery  of  the  curriculum  (e.g.,  applying  a  unified  approach  across  all  teachers  and  schools  to  teaching  the  program’s  five  framing  concepts).  This  report  focuses  on  year-­‐two  results  (2010)  ONLY.  

Evaluation  Questions  

Once  the  program’s  theory  of  change  had  been  defined  (see  Appendix  A),  a  comprehensive  list  of  evaluation  questions  was  identified  by  Thrive  Foundation.  These  research  questions  served  to  tap  into  1)  students’  experiences  while  participating  in  the  program,  2)  teachers’  experiences  with  the  enhanced  program,  and  3)  students’  academic  and  behavioral  outcomes  associated  with  their  participation  in  the  program.  

Students’  Experiences  in  the  Compass  Program  

• Did   Compass   students   feel   there   was   any   value   or   benefit   in   participating   in   the   Compass  program?  

• Did  Compass  students  feel  that  the  Compass  program  provided  a  supportive  environment?  

Students  Changes  in  Behavior  and  Intent  to  Behave  

• Did  students  change  their  beliefs  and  intentions  around  Personal  Spark?  

• Did  students  change  their  beliefs  and  intentions  around  Growth  Mindset?  

• Did  students  change  their  beliefs  and  intentions  around  School  Success  Techniques?  

• Did  students  change  their  beliefs  and  intentions  around  Reaching  Goals  &  Dreams?  

 Student  Outcomes  Associated  with  the  Compass  Program      (To  be  answered  in  2011)

• In  what  ways  did  Compass  students  show  evidence  that  they  were  being  successful  in  school?  

• Did  Compass   students  have  better  academic   (e.g.,  GPA  and  number  of   completed  courses)  and  behavioral  outcomes  (e.g.,  disciplinary  incidents)  than  their  peers?  

Methods  

To  answer  the  evaluation  questions  above,  three  research  methods  were  used  in  2010.  These  included  a  student  pre/post  survey,  student  focus  groups,  and  examination  of  school  performance  and  attendance  data.    Each  of  these  methods  is  described  in  more  detail  below.  

Pre/Post  Survey  was  designed  with  the  collaboration  of  Thrive  to  measure  students’  beliefs,  awareness,  and  motivations  around  the  program’s  five  framing  concepts  (i.e.,  Personal  Spark,  Growth  Mindset,  Leadership,  School  Success  Tips,  Reaching  Goals  &  Dreams).    All  Compass  students  were  invited  to  complete  the  survey  in  their  schools’  computer  labs  on  the  first  day  of    

Page 7: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

7  

 

 

Compass  instruction,  and  again  a  day  before  the  end  of  the  program.    To  accommodate  students  with  low  literacy  skills  and  to  ensure  consistency  in  administration  and  interpretation  of  items,  the  survey  was  read  aloud  to  each  Compass  class  by  a  staff  member  from  ASR.    Students  listened  to  each  question  and  marked  their  individual  response  on  their  own  version  of  the  on-­‐line  survey.  In  all,  394  students  completed  a  pre-­‐survey,  and  372  completed  a  post-­‐survey.    

Focus  Groups:  Staff  from  ASR  conducted  focus  groups  with  Compass  students  to  explore  their  impressions  of  the  program  in  general,  as  well  as  to  conduct  a  more  in-­‐depth  discussion  around  the  program’s  five  framing  concepts  (i.e.,  personal  spark,  growth  mindset,  leadership,  school  success  tips,  reaching  goals  and  dreams).  A  total  of  40  students  were  randomly  selected  from  four  different  physical  education  classes  (one  in  each  high  school).        

Academics:  To  assess  the  extent  to  which  the  Compass  program  influenced  youth’s  academic  progress  and  behavioral  referrals  during  their  first  year  in  high  school,  ASR  obtained  specific  student-­‐level  data  from  Sequoia  Union  School  District  for  two  consecutive  academic  years:  2010-­‐2011.  These  data  included  behavioral  and  academic  information  for  both  Compass  and  non-­‐Compass  participants.  Data  requested  from  the  school  district  included  demographic  and  background  information,  such  as  race,  English  learner  status,  GPA,  8th  and  9th  grade  CST  scores,  attendance,  and  misconduct  (n=384  students).  To  compare  Compass  students  to  their  peers,  ASR  used  two  types  of  analyses:  

• “Similarly-­‐situated”  comparison  group:    ASR  created  a  statistically-­‐matched  comparison  group  that  matched  the  cohort  of  Compass  students  in  terms  of  the  prevalence  of  key  characteristics  such  as  8th  grade  CST  performance  levels  in  math  and  English,  ethnicity,  English  Learner  status,  free/reduced  lunch,  and  gender.    Because  the  Compass  population  was  not  itself  homogeneous  in  terms  of  Math  and  English  proficiency,  two  cohorts  were  created:    a  ‘Math’  cohort  of  the  Compass  and  non-­‐Compass  (labeled  as  Cohort  M),  and  an  ‘English’  cohort  of  the  Compass  and  non-­‐Compass  (labeled  as  Cohort  E)(See  Figure  1).    

• “At-­‐risk”  subset  (Compass  vs  non-­‐Compass):  Since  some  Compass  students  were  academically  proficient  or  advanced  in  8th  grade,  and  was  not  reflective  of  the  intended  target  population,  ASR  created  a  second  data  set  with  those  students  removed.    The  resulting  subset  is  Compass  students  and  non-­‐Compass  students  at  the  three  lowest  CST  levels  of  proficiency  (Far  Below  Basic,  Below  Basic,  and  Basic).  

   

Page 8: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

8  

 

Figure  1:  Matched  Cohorts  For  Randomized  Control  Study  

Cohort  Based  on  8th  grade    MATH  scores  (Cohort  M)  

  Cohort  Based  on  8th  grade    ENGLISH  scores  (Cohort  E)  

Compass  students  

 Non-­‐

Compass  students  

  Compass  students  

  Non-­‐Compass  students  

Advanced     Advanced     Advanced     Advanced  

Proficient     Proficient     Proficient     Proficient  

Basic     Basic     Basic     Basic  

Below  Basic     Below  Basic     Below  Basic     Below  Basic  

Far  Below  Basic  

  Far  Below  Basic  

  Far  Below  Basic  

  Far  Below  Basic  

2. Demographics of Youth

Characteristics  of  Compass  StudentsCharacteristics  of  Compass  Students        

Overall,  there  were  455  Compass  students  in  the  populations  (see  Figure  1).    They  had  the  following  characteristics:  

• The  majority  of  students  were  Hispanic/Latino  (78%).    Very  few  were  white  (9%),  Pacific  Islander  (4%),  or  African  American  (3%).      

• There  was  a  fairly  equal  representation  of  males  (52%)  and  females  (48%).      

• Just  over  half  (53%)  were  English  Learners  indicating  that  they  had  limitied  English  Proficiency.  Twenty-­‐five  percent  of  the  Compass  youth  were  Reclassified  Fluent  English  Proficient,  which  means  they  previously  had  limited  proficiency  but  had  become  fluent.  Twenty-­‐one  percent  were  raised  in  househoulds  in  which  the  primary  language  spoken  was  English  (i.e.,  English  

Figure 1 – Compass students’ characteristics

Ethnicity

N=455 (2010) Hispanic/Latino 78%

Pacific Islander (other) 4%

White 9%

African American/Black 3%

Gender

N=455 (2010) Male 52%

Female 48%

English Proficiency

N=409 (2010) English Learner 53%

RFEP 25%

English only 21%

IFEP 1%

Page 9: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

9  

 

only).  Only  one  percent  were  from  households  in  which  a  language  other  than  English  was  used  (IFEP).  

When  examining  Compass  students’  Math  and  English  subject  area  proficiency  from  the  previous  year  (i.e.,  8th  grade),  more  than  one-­‐third  (35%)  did  not  have  a  Basic  level  of  proficiency  in  English,  and  43%  did  not  have  a  Basic  level  of  proficiency  in  Math  (Figures  2  below).    Several  students  were  either  proficient  or  had  advanced  skills  in  English  (21%)  and  Math  (20%).  These  students  were  included  in  the  course  as  potential  role  models.  

Figure  2  –Compass  students  proficiency  levels  in  8th  grade  (2010  cohort)  

 

3. Students’ Experiences in the Compass Program

Findings  from  the  Student  SurveyFindings  from  the  Student  Survey    

Did  Compass  students  feel  there  was  any  value  or  benefit  in  participating  in  the  Compass  program?  

As  indicated  in  the  post  survey  data,  the  majority  of  students  valued  their  time  spent  in  the  Compass  program.  Specifically,  they  thought  that  the  Compass  program  would  support  them  in  high  school  because  it  made  them  feel  more  enthusiastic  about  learning.    They    were  particularly  excited  by  the  subjects:  “How  the  brain  works”  –  A  Thrive-­‐related  concept  -­‐  and  “math  skills.”  Almost  all  of  them  would  recommend  the  program  to  a  friend.  

12%

23%

44%

19%

2%

11%

32% 37%

17%

3%

0%

10%

20%

30%

40%

50%

Far below basic Below basic Basic Proficient Advanced

8th grade English CST

8th grade Math CST

Page 10: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

10  

 

Eighty-­‐nine  percent  of  students  (n=343)  reported  that  what  they  had  learned  at  Compass  was  going  to  help  them  while  in  high  school.    

More  than  three-­‐quarters  of  all  students  (77%,  n=346)  reported  that  the  program  had  helped  them  feel  more  enthusiastic/excited  to  learn  in  high  school.

When  asked  to  list  the  “coolest”  thing  they  learned  while  attending  the  program,  students  most  commonly  reported  discussions  on  how  the  brain  works  (as  it  relates  to  growth  mindset),  as  well  as  discussions  about  the  impact  of  drugs  on  brain  functioning.  The  second  “coolest”  thing  they  learned  was  math  skills.

Ninety  percent  (n=339)  said  that  they  would  recommend  Compass  to  a  friend.    

Did  Compass  students  feel  that  the  Compass  program  provided  a  supportive  environment?  

In  general,  the  majority  of  students  felt  that  their  Compass  teachers  not  only  “cared”  about  them,  but  also  believed  in  their  potential  to  succeed  in  high  school.  For  instance,  three-­‐quarters  of  the  students  (75%,  n=347)  felt  that  their  teachers  “cared”  about  them.    As  for  students’  perceptions  around  their  teachers’  belief  in  their  potential  for  success,  almost  all  students  (88%,  n=347)  felt  that  their  teachers  “believed”  that  they  could  succeed.    

In  addition  to  the  value  placed  on  teachers’  instructional  methods,  students  also  appreciated  specific  content  areas  discussed  in  their  classes.  For  example,  in  2009,  students  most  commonly  talked  about  the  books,  stories,  and  new  vocabulary  they  learned  in  their  Reading  Class.  As  for  their  High  School  Success  Class,  they  most  valued  the  discussions  on  how  the  brain  works,  as  it  relates  to  growth  mindset.    

Did  Compass  students  feel  that  Step-­‐it-­‐up-­‐2-­‐Thrive  topics  were  meaningful?  

Students  were  asked  to  rate  the  meaningfulness  of  all  Compass  topics  on  a  four-­‐point  scale  (ranging  from  1  “not  at  all  meaningful”  to  4  “very  meaningful”).  While  all  eight  topics  resonated  with  students  at  some  level,  students  felt  particularly  drawn  to  Respect  comes  from  within  (78%),  Leadership  skills  (78%),  Friendship  and  fearship  (76%)  and  Standing  up  to  peer  pressure  (73%)  (see  Figure  3).  

 

 

 

“[I learned that] you need sleep because basically your brain needs rest and your brain is basically in charge of everything you do, like thinking,

motion, senses.”

“I will start on all assignments the day they are assigned and I plan on

paying attention and asking teachers for help when I don't understand

something.”

“The coolest thing I learned at Compass is how the brain works and

functions when you take drugs or alcohol.”

“[In my] English class I read a book call "Monster." It was interesting. I

also learned about drugs, fractions, and how to follow the steps to get to

my dreams.”

Page 11: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

11  

 

Figure  3  –  Student  Percentage  Who  Found  Step-­‐it-­‐up-­‐2-­‐Thrive  Topics  Meaningful

   

4. Students’ Changes In Beliefs and Intentions

The  Five  Framing  Concepts  

Pre/post  surveys  showed  that  students  did  not  change  their  beliefs  or  intentions  on  items  related  to  developing  a  personal  Spark,  identifying  Spark  Champions,  or  developing  a  Growth  Mindset.  However,  during  focus  groups,  students  reported  a  different  reality,  stating  that  they  were  more  likely  to:  reach  out  to  their  teachers  after  class  if  they  need  one-­‐on-­‐one  help,  have  a  growth-­‐mindset,  and  try  harder.  These  comments  reflected  specific  lessons  related  to  developing  Spark  Champions  and  Growth  Mindset.    

During  the  focus  groups,  students  were  also  asked  to  talk  about  each  of  the  program’s  five  framing  concepts.  The  objective  here  was  to  determine  the  extent  to  which  students  understood  these  concepts  (and  how  they  intended  to  apply  this  new  knowledge.  Based  on  students’  feedback,  the  majority  of  them  grasped  the  core  concepts.  Specifically,  Compass  students  recognized  the  different  aspects  of  a  leader,  and  that  everyone  can  be  one;  they  understood  the  importance  of  staying  positive  (i.e.,  having  a  growth  mindset);  they  distinguished  friendships  from  “fearships”;  and,  finally,  identified  ways  they  could  further  nurture  their  sparks  and  leadership  skills  while  participating  in  school  activities  (See  Appendix  B).

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Destructive language

Moral compass and bystander behavior

Dealing with anger, fear and pain

Communication and public speaking

Standing up to peer pressure

Friendship and fearship

Leadership skills

Respect comes from within

Percentage of students

Page 12: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

12  

 

5. Students’ Academic Performance in 9th Grade

School  Success  Techniques  

Survey  findings  showed  no  signficant  changes  related  to:  study  skills,  their  ability  to  look  for  leadership  roles  in  high  school,  and  understanding  the  impacts  of    marijuana  on  the  brain.    However,  students’  knowledge  of  two  school  success  techniques  had  significantly  improved  by  the  time  they  concluded  the  program  (i.e,  knowing  how  to  take  notes  and  seeing  themselves  as  a  leader)  (see  Figure  5).      

Open-­‐ended  survey  questions  also  illustrated  that  as  a  result  of  the  program,  students  planned  on  getting  enough  sleep  while  in  high  school  -­‐  another  risk  factor  covered  in  class  -­‐  (87%;  n=349),  on  using  Compass’  study  practices  (84%;  n=347),  and  asking  for  help  when  faced  with  a  problem  at  home  or  school  (83%;  n=346).    Focus  group  findings  also  showed  that  students  felt  more  prepared  academically  as  a  result  of  their  participation  in  the  program.

Figure  5–Student  Pre/Post  Mean  Scores  on  Concepts  Related  to  School  Success  Techniques  

Note: All changes are statistically significant (p < 0.05).

Academic  Outcomes  

SIGNIFICANT  FINDINGS  

Grade  Point  Average  (GPA)  

Compass  students  in  Cohort  E  had  significantly  higher  GPAs  at  the  end  of  9th  grade  than  their  similarly-­‐situated  peers  (2.24  as  compared  to  2.03,  respectively  p<.05).    Most  notable  of  which  is  the  0.28  point  difference  between  “at-­‐risk”  Compass  students  and  their  peers.      

 

2.96 2.84 3.07 3.01

1.00

2.00

3.00

4.00

Students know how to take notes in class Students see themselves as a leader

2010 Pre mean score 2010 Post mean score

Page 13: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

13  

 

Figure  6  –  Students’  average  GPA  by  the  end  of  9th  grade  (Cohort  E)  

 

 

Credits  Earned  

Course  performance  (in  this  case  the  number  of  credits  earned)  is  an  important  factor  taken  into  consideration  when  determining  students’  risk  for  dropping  out  of  high  school.    Ultimately,  for  9th  graders  to  be  on  track  to  graduate,  they  should  have  earned  a  total  of  50  credits  by  the  end  of  their  9th  grade  year.          

Compass  students  earned  an  average  of  58  credits  by  the  end  of  9th  grade,  exceeding  the  50-­‐credit  minimum  (significantly  higher  than  their  peers,  p<.05)  and  illustrating  that  they  were  “on  track”  for  high  school  graduation.    Provided  below  is  the  breakdown  by  cohorts  M  and  E,  and  by  all  students  vs.  “at-­‐risk”  students.    The  widest  gap  was  observed  with  students  in  cohort  E,  where  the  average  number  of  credits  was  higher  by  about  11  points  for  Compass  students.      

Figure  7  –  Credits  earned  by  the  end  of  9th  grade    

 

Note: (*) denotes a statistically significant difference, p<.05. Data based on 384 overall students (Compass and non-Compass), and 237 “at-risk” students (Compass and non-Compass).

2.24   2.09  2.03   1.81  

0.00

1.00

2.00

3.00

4.00

All students* "At-risk" students*

Compass students

Non-Compass students

58  56  

58   57  54  

52  48  

46  

30

40

50

60

All students* "At-risk" students* All students* "At-risk" students*

Compass students

Non-Compass students

Note: (*) denotes a statistically significant difference, p<.05.

Page 14: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

14  

 

CST  Proficiency  Levels  

Compass  students  were  more  likely  than  their  peers  to  increase  their  proficiency  in  English.    For  instance,  30%  of  Compass  students  had  increased  their  proficiency  in  English  by  at  least  one  level,  as  compared  to  19%  of  non-­‐Compass  students  (a  statistically  significant  difference,  p<.05).  

Figure  8  –  Percentage  of  students  who  improved  their  English  CST  proficiencies  in  9th  grade    

 

Note: (*) denotes a statistically significant difference, p<.05.

Statistically  significant  improvements  were  also  observed  with  the  “at-­‐risk”  group  of  Compass  students,  in  that  more  Compass  students  experienced  an  increase  in  CST  level  (37%,  as  compared  to  21%  of  their  peers),  and  fewer  Compass  students  experienced  a  decrease  in  CST  level  (13%,  as  compared  to  36%  of  their  peers).  

 

Figure  9  –  Percentage  of  “at-­‐risk”  students  who  improved  their  English  CST  proficiencies  in  9th  grade    

 

Note: (*) denotes a statistically significant difference, p<.05.

30%  

19%  

51%  

19%  

39%   42%  

10%

20%

30%

40%

50%

60%

Increased by 1 level* Decreased by 1 level* Remained the same

Compass students

Non-Compass students

37%  

13%  

49%  

21%  

36%  

43%  

10%

20%

30%

40%

50%

60%

Increased by 1 level* Decreased by 1 level* Remained the same

Compass students

Non-Compass students

Page 15: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

15  

 

 

Attendance  Outcomes  

The  district’s  data  file  included  two  variables  pertaining  to  students’  attendance:  number  of  days  “present,”  and  number  of  days  “possible.”    To  calculate  the  number  of  days  missed,  the  researchers  subtracted  each  student’s  number  of  days  present  from  number  of  days  possible.    For  example,  if  a  student  had  180  days  possible  and  was  present  on  170  days,  this  student  had  missed  a  total  of  10  days.    Provided  next  are  findings  pertaining  to  students’  attendance  while  in  9th  grade.  

SIGNIFICANT  FINDINGS  

On  average,  Compass  students  missed  12  periods  in  9th  grade,  as  compared  to  14  periods  for  their  non-­‐Compass  peers.    Significant  differences  were  noted  for  Compass  students  in  cohort  E,  as  seen  in  the  figure  below.          

Figure  10–  Students’  average  number  of  missed  periods  (Cohort  E)

 Note: (*) denotes a statistically significant difference, p<.05. The overall sample size of Compass and non-Compass students was 384. The sample size of the Compass and non-Compass “at-risk” subset was 237.

     

12  13  

16  17  

5

8

11

14

17

20

All students* "At-risk" students*

Compass students

Non-Compass students

Page 16: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

16  

 

   

6. Summary Of Findings

Thrive  Foundation  supported  a  Compass  re-­‐design  anchored  to  the  Quaglia  Institute  research  (www.qisa.org)  that  students  drop  out  of  high  school  when  they  do  not  feel  a  sense  of  self-­‐worth,  a  sense  of  purpose,  and  are  actively  engaged  in  classroom  learning  linked  to  their  life  experiences  and  direction.  The  Compass  curriculum  strived  to  improve  youth’s  skills,  attitudes,  and  perceptions  about  themselves  so  that  they  would  enter  high  school  with  increased  motivation  and  skills  to  reach  their  potential.  

Surveys  and  focus  groups  demonstrated  that  Compass  students  had  a  very  positive  learning  experience.  Students  felt  that  the  program  was  of  value,  helping  them  to  feel  more  enthusiastic  about  high  school.    They  also  felt  that  the  program  had  provided  a  supportive  environment,  believed  that  teachers  cared  for  them,  and  believed  in  them.    During  the  Compass  program,  students  also  began  to  change  some  of  their  beliefs  about  themselves  and  reported  that  they  might  behave  differently  in  high  school.  Specifically,  they  reported  that  they  would  ask  their  teachers  for  help,  use  Compass  study  practices,  have  a  growth  mindset,  try  harder,  and  get  enough  sleep  (All  topics  covered  in  the  curriculum.)  The  question  was  then  asked:    “Would  this  shift  lead  to  reduced  risk  factors  for  graduation  that  can  be  measured  by  better  attendance,  improved  GPA,  and  reduced  delinquency  than  a  similarly  matched  control  population?”    Even  lacking  evidence  of  consistent  reinforcement  during  the  following  school  year,  data  from  the  short  Compass  intervention  suggests  “yes,”  the  methodology  adds  value.  The  data  also  suggests  a  need  for  continued  reinforcement  of  Compass  learnings  throughout  the  school  year  in  order  to  sustain  students’  early  positive  gains.  

To  answer  the  overarching  research  question  of  whether  students'  participation  in  Compass  translates  into  improved  school  outcomes,  the  researchers  analyzed  three  domains  of  school  success:  academic,  attendance  and  behavioral.    The  findings  revealed  that  the  Compass  program  showed  significant  impact  on  students'  academic  performance  and  attendance,  but  no  impact  was  noted  on  students'  behaviors.      

Key  take-­‐aways  include  the  following:  

GPA:    Compass  students  had  significantly  higher  GPAs  by  the  end  of  9th  grade  than  did  non-­‐Compass  students.    The  most  “at-­‐risk”  students  registered  the  greatest  difference  with  their  similarly-­‐situated  non-­‐Compass  peers.      

CREDITS  EARNED:    The  overall  population  of  Compass  students  had  earned  significantly  more  credits  as  compared  to  non-­‐Compass  students  by  the  end  of  9th  grade.  

CST  SCORES:    With  regard  to  students'  CST  proficiencies  in  9th  grade,  significantly  more  Compass  students  moved  up  one  CST  level  in  9th  grade  than  their  similarly-­‐situated  peers.      

ATTENDANCE:    Finally,  Compass  students  had  significantly  better  9th  grade  attendance  than  non-­‐Compass  students.      

 

       

Page 17: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

17  

 

Appendix A: Theory of Change

   

Students  also:  

• Feel  supported  to  achieve  goals  

• Develop  a  sense  of  belonging  at  school  

At-­‐risk  students  participate  in  a  24  day  program  with  four  classes  each  day:    

• High  school  success  

• Summer  reading  

• Math  

• Leadership    

Teachers  receive  :  

• Kickoff  training  to  Thrive  concepts  

• Time  to  design  the  summer  curricula  that  infuses  Thrive  concepts  into  the  four  core  classes  

• On  site  coaching  

• Follow  up  group  work    

Students  increase  

AWARENESS  about:  

• School  competencies:  “I  know  what  it  takes  to  be  successful  in  school”  

• Growth  mindset  

• Personal  spark  

• Reaching  Goals  &  Dreams  

• School  success  techniques  

• Leadership  &  Social  emotional  skills  

Students  change  

BELIEFs  about:  

• School  competencies:    “I  have  lots  of  strengths  and  I  can  apply  them”  

• Growth  mindset  

• Reaching  Goals  &  Dreams  

• School  success  techniques  

• Leadership  &  Social  emotional  

skills

Students  increase  

INTENTIONs  about:    

• School  competencies:  “  I  would  like  to  follow  my  HS  success  plan”    

• Growth  mindset  

• Reaching  Goals  &  Dreams  

• School  success  techniques  

Students  show  EVIDENCE  of  change:  

• School  competencies:    “I  have  achieved  my  HS  Success  goals””  

• Growth  mindset  

• Reaching  Goals  &  Dreams  

• School  success  techniques  

• Leadership  &  Social  emotional  skills  

 

• Reduced  risk  of  drop  out  at  the  end  of  9th  grade  (no  Fs,  at  least  50  units,  better  than  80%  attendance)    

 

Students  also  

• Develop  a  love  of  learning

Program  Activities                                                        Immediate  Outcomes                                                                                                                                                                                                              Longer  Term  Outcomes  

Page 18: 2010 Compass

COMPASS  2010  –  PROGRAM  EVALUATION  REPORT    

18  

 

Appendix B: Vignettes

Personal  Spark  

What  does  it  mean  to  students?  

“Something that gets you up in the morning and something that you love to do. Something you are good at.”

“Your inspiration – what you like to do.”

“Your sparks will be diminished if you don’t overcome your fearships.”

“Something you can always fall back onto.”

[A spark] is something that makes you shine.”

Personal  Spark  

How  will  students  use  that  knowledge?  

“Become a soccer player, and practice every day to get better and better.”

“Help my community.”

“It will help me select my classes.”

“I will attend school clubs.”

Growth  Mindset  

What  does  it  mean  to  students?  

“Try to stay positive and do your best, and never have a fixed mindset.”

“If you can’t do something you have to keep on trying until you get it right. If you can’t get it right, ask somebody. Even though you have rough times, you can get over them.”

“It means having an open mind, and being open to new things.”

Growth  Mindset  

How  will  they  use  that  knowledge?  

“Find positive ways to solve problems.”

“[I will] not say ‘no’ the first time someone gives me a new opportunity.”

“If you fail a test you know that you need to study harder. [It does not mean] that you are always going to fail or that you are not smart enough.”

“It’s better to face reality than to hide all the time. Most likely, I’m the kind of girl that will face every problem whether they help me, so I know what I did wrong, and so I can do better next time.”

Leadership  

What  does  it  mean  to  students?  

“When you are a leader you have to take responsibilities.”

“At first I thought that a leader was someone loud and in charge, but it’s someone who really listens to what everyone else is saying.”

“A leader helps people.”

Leadership  

How  will  they  use  that  knowledge?  

“I learned in leadership class not to let anyone take control of you. Don’t let drama get in the way of your goals.”

“I learned what it takes to be a leader and when to let your balloon air out before it could explode.”

“How to make better friends and lead people in the right direction.”

School  Success  Techniques  

What  does  it  mean  to  students?  

“How to take Cornell notes and be able to keep organized.”

“Grades depend on behaviors. Pay attention and don’t play around.”

“[I learned] how to control my emotions.”

“Try to manage your time with your homework.”

School  Success  Techniques  

How  will  they  use  that  knowledge?  

“How to ask questions, take notes, be open minded, have a growth mind, sleep well.”

“To ask for advice from others and to go and look for help when struggling on a subject.”