Gaining Options for College Collaborative

Testing New Models for Talent Search: Using Data, Empowerment Evaluation

and Systems Collaboration

Dr. Nicole NorflesDr. Margaret CahalanDr. Stephanie R. MillerCouncil for Opportunity for Education

Topics• Overview of the pilot project and Go-

College (i3)project (Stephanie)• Project components

• Using data (Stephanie)• Collaboration (Nicole)• Traditional and empowerment evaluation

(Maggie)• Discussion and Questions (all)

Pilot Project (GE Project) Project support by the GE Foundation Launched in 2006 Student-level intervention (60 students) Located in four sites

› Louisville, KY (1 school, n ~= 300 students)› Erie, PA (3 schools, n ~= 720 students)› Connecticut (1 school, n ~= 240 students)› Harlem, NY

1 school, n ~= 350 students 1 school (whole school approach), n ~= 800 students

GE Pilot: Project Components Existing College Access Program (CAPs) serving

students in school Academic and College Coaching Services

› Academic advising (quarterly sessions)› Weekly/bi-monthly group sessions› Limited tutoring services› College exploration› Summer programming› One embedded college coach (serving 60 - 80 students per

grade)› Limited whole school effort

Base data driven decision-making Learning communities

GE Pilot: Preliminary Findings from 1st Graduating Cohort in Louisville

69 students served › 32 students enrolling at 2-year school› 30 enrolling at 4-year school› 3 joining the military› 4 not graduating or transferred

Financial Aid› 18 students received some form of scholarship› 6 students received full scholarships

Majors› Animal science› Biology, chemistry› Math education› Nursing› Graphic design› Pre-pharmacy› Business

GO College: Investing in Innovation

• COE 1 of 49 selected from 1,700 applicants• $20 million grant from Dept. of Education, $4

million match from the GE Foundation • Builds on Talent Search model and GE pilots • Whole school model with intensive learning

communities• Implemented in 2 cities: Erie, PA and Louisville,

KY• Rigorous external evaluation required

(Educational Testing Service - ETS)

GO College: Project Components, Add on Model

base Talent Search vs. GE Pilot vs. GO College Selection of students/learning communities

(intensive services) Services Outcomes Using data/data process (system) Collaboration Evaluation (internal and external)

GO College: CollaborationGO College provides one model where TRIO pre-college programs can meet the rigorous curricula and collaboration requirements of the Higher Education Opportunity Amendments of 2008The project could be replicated locally by collaboratives of TRIO programs and high school districts.

GO College: Multiple Communities of Learning Stakeholders and Collaborators

Students

Parents &

community

Funders (ED OII-OPE, GE)

COE, PellExternal evaluato

rs

College Access

Providers

School & District

staff

GO College: CollaborationEngaging the Community

•Community events

•Churches, community leaders, businesses, parents

•Marketing materials

•Four press events per year

Press Conference and Launch GO College - Erie

GO College: The EvaluationA Personal Journey

Contractor Project Director (National Evaluation of Student Support Services, National Evaluation of Talent Search, TRIO performance reporting support contracts)

Department of Education as Technical Monitor › Did review of evaluation studies from last decade—(not often viewed as

useful or valid by practitioners----not often find positive impacts --are we asking the right questions?; how can we make studies more useful and still provide input into policy decisions )

› Has evaluation research –overpromised in terms of validity of results and in terms of usefulness---what does lack of effects mean???)

› Came to see need for taking a more participatory approach involved and began Designing Next Generation of GEAR UP studies that were developed by grantees using technical assistance from RTI –use traditional models of evaluation

GO College: Internal or External Evaluation?

External evaluation required—for I-3 Validation studies using traditional methods meeting What Works Clearinghouse criteria as much as possible—model of validate and then scale up if find positive effects

“The dilemma of whether to use external or internal evaluation is as false as that between qualitative and quantitative methods. The solution is always to use the best of both, not just one or the other” (Michael Scriven)

COE-I3—Go-College Collaborative grant is using both approaches— working collaboratively with ETS and their sub-contractor Brown—ABT is technical assistance provider—(for example, Brown just completed random assignment of rising 9th graders in the 6 schools based on data COE compiled and processed)

Traditional vs. Empowerment Evaluation

Traditional EvaluationExternalExpertDependencyIndependent judgmentDeveloped when data not available to all— elite with resources to collect and skill to analyze

Empowerment EvaluationInternalCoach or Critical friendSelf-determination & capacity buildingCollaborationMakes use of Data Revolution—internet, web, real time interactive sharing of knowledge—all publish—face book, blog, twitter

Internal and External Evaluation Enhance Each Other

Can provide richer data set that enables more complete external examination

External reality check and quality control—keep on track

External—help question shared bias Coordinate data needs Mixed methods Evaluators—co-equals—not superior or

servant

Logic/Theory of Action/Change and Assessment Plan Goal: I ncreased college preparation and entrance for low- income students in high schools in project

Outcomes and I mpact Assessment Whole School --light intervention--expectation is will be accompanied by small upward changes in rates when compared to 6 caref ully matched schools—case study ---comparative time series monitoring pre and post—also ethnographic context & demographics change –note current context severe system staff cutbacks overall Outcomes—measures 1. SAT/ ACT taking-12th 2. FAFSA completion-12th 3. College entrance cohorts f rom 2000-2015 (depending

on district data availability) to National Student Clearinghouse

4. Limited survey data change over time on pre-college measures

Student Communities of Learning—Existing grade cohorts in 6 schools diff er in method of recruitment & selection & intervention; 2 of 6 schools not in GE Pilot. Rising prior to 2011 grades 10 to 12—Matched comparison using propensity and instrumental variables regression to control f or not eliminate selection eff ects—goal higher than expected in comparison to similar students Rising 9th grade 2011 cohort will mostly be selected using random assignment within academic and demographic stratum to create diverse group of low income students to be invited to learning communities Outcome measures 1. Retention in the learning communities program 2. Measures of student engagement and expectations in

surveys 3. Normal progress through high school (GPA and

attendance) 4. Selection & completion of rigorous courses 5. SAT/ ACT taking-12th 6. FAFSA completion-12th 7. College entrance at graduation & plans

I ntervention Actions

Whole School Services: 6 schools –3 in ERIE and 3 in Louisville 1. District wide media launch of initiatives 2. I nf ormational brochure to all students 3. Embedded college coaches in building—available f or

drop in counseling 4. At least 2 college wide events or workshops provided

by college coaches per year 5. Tracking FAFSA completion f or 12th graders 6. Data base college planning tool all students and staff

can use for planning 7. Prof essional development with f ocus on data use f or

planning 8. Math and science enrichment activities

Strategies & high level intermediate

objectives

1. Provide structure for increased collaborative & coordinated initiative across multiple community stakeholder groups

2. Provide Data Access

Tool for empowerment of stakeholders and real- time planning & monitoring & tool f or and increase skill in data use among students

3. Provide increased

level of college access services to whole school

4. Establish small

student and stakeholder learning communities

5. Foster culture of

valuing diversity, asset based and leadership service learning approach

Student Communities of Learning 9. Small groups receive invitation to participate

in community—given opportunity not provided to all (60 per grade)

10. Summer pre-college program—1 to 2 weeks 11. Academic year regular meetings of the group

either during class day or af ter school a. Service learning & leadership &

team building, empowerment, effi cacy curriculum

b. Focus on data—project based learning

12. Personal f ollow up & coaching throughout 4 years of high school—early intervention if academic or attendance troubles manifest

13. Rigorous course taking encouragement and support-use of data tool to plan pathway

14. Tutoring or mentoring services available 15. College visits and mentoring

GO College: Null-Hypothesis & MethodsNull Hypothesis1. Implementation of the GE-Pilot and

the I-3 enhancement strategies will not be accompanied by measurable change in college readiness and college going rates over the period of study for the whole schools

2. Rising 9th graders randomly assigned to participate in the more intensive learning communities will not differ in outcomes observed from those not invited to participate

3. Diverse students selected at random from ability quartiles will not differ in outcomes observed from students who volunteer for the learning communities

Associated Methods1. Observational study of pre and

post –and comparison schools outcome trends in schools over 15 year period–Requesting data from 2000—2015—Case History—–use systems concepts—community asset development study

2. Experimental study for purpose of understanding best way to select students and to give equal chance for scarce treatment. Departure from usual method of TS of selection based on student and teacher interest

3. Observational study of differences in outcomes observed in GE pilot in which students volunteered or teachers selected—method used for 10 to 12 graders

GO College: Confounders, Challenges and Opportunities

o Dynamic Data Driven Focus---implies using the data to improve services on an on-going basis—intervention changes –we are validating a dynamic context driven adaptive reflective process not a static intervention

o Community Initiative and Collaborative data use by all stakeholders lends project to involve internal reflection and self evaluation

o Fast Changing Context---examples --Severe cutbacks to schools in districts, US presidential challenge to be first in degrees, data use change, no community college in ERIE and recently voted down, 55 000 degrees-Louisville, ACT collaboration

o Whole School and Previous Other services received before and during—Talent Search—GE-Pilot—I-3—Others-limits contrasts and meaningful counterfactuals

Go-College: Collaborative –Process of Participatory Action & Change

Using Empowerment Evaluation As Internal Evaluation Tool

Empowerment evaluation is the use of evaluation concepts, techniques and findings to foster improvement and self determination. It employs both qualitative and quantitative methodologies.

Empowerment evaluation: Knowledge and Tools for Self-Assessment and Accountability Fetterman, Kaftaraian, and Wandersman 1996 Widely-used

Joint Committee for Education Program Standards have included concepts

American Evaluation Association and AERA workshops Public Health Field International Development work Education school systems

10 Key Principals of Empowerment Evaluation(Wandersman et. Al 2005)

1. Community Ownership— primary responsibility with organization and not outside evaluator

2. Inclusion— involves representation of key stakeholders3. Democratic participation— highly collaborative—

opportunity to voices questions—every stakeholders voice is heard

4. Community knowledge- promotes growth of knowledge in community—stakeholders are considered to be in the best position to understand the issues and generate solutions to problems

5. Evidence Based strategies- promotes use of strategies with high quality evidence—research evidence of effectiveness—evidence strategies contextualized to fit community

10 Key Principals of Empowerment Evaluation(cont.)6. Accountability- provides data that can be used to

determine whether a strategy has achieved its goals—negative results are used to inform change in a strategy or the selection of a new strategy for the purpose of producing better outcomes

7. Improvement— Helps organizations improve strategies so that they are more likely to achieve stated goals—process and outcome evaluation (Rossi 1999)

8. Organizational learning—fosters a culture of learning—view positive and negative feedback as valuable information and believe that all strategies can be improved

9. Social justice— Increase capacity to reduce disparities that affect marginalized by persecution, discrimination, prejudice and intolerance

10. Capacity building— builds capacity of organizations to conduct their own evaluations, understand results and use them to continuously improve organization

Preliminary Plan of Topics for Stakeholder Collaborative Report

1. Ethnographic school histories: Quantitative and qualitative, outcomes assessment

2. Assessment of the strategies effectiveness and recommendations for improvement (collaboration, data use, whole school, learning communities, diversity/asset based)

3. Implications for Talent Search and College Access Programming

4. Modeling Meeting the 2020 College Attainment goals—student contributions –data use (international, national, state, local, individual)

5. Reflections on use of innovative empowerment evaluation tool—evaluate the evaluation tool

Questions and Answers