From Sea to Shining Sea: Evaluating Title II TQE Partnership

Strategies to Ensure Quality

Susan Ann Tucker, Evaluation & Development Associates LLC

Nancy Shapiro, University System of Maryland

Letitia H. Fickel, University of Alaska, Anchorage

Shirley Lal, California State University, Dominguez Hills

Roy Weaver, Ball State University

Tomas Morales, College of Staten Island

Agenda Partnership challenges Core TQE assumptions and charges 5 questions of focus today Overview of case study projects Findings: causal network analysis Recommendations & follow-up strategies Small group discussion Open Q&A

Core TQE assumptions Partner approach is a vehicle for human and institutional

development Facilitates knowledge creation and diffusion Moves the NCLB agenda forward Makes possible better cost-sharing Contributes to teacher education capacity development

TQE charges Creating collaborative partnerships and learning communities in

high need P-12 schools to enhance student achievement

Improving quality teacher preparation—integrating standards-based assessment strategies and interdisciplinary training and expanding clinical experiences

Articulating a continuum of services from recruitment to induction that serves preservice to beginning teachers in collaborative (and sometimes not so collaborative) settings of school districts, community colleges, and universities

Partnership challenges How to identify premises, tacit and overt, embedded in

“partnership”? What are sources & methods to document partnering

contexts, processes, outcomes? What model(s) to use for evaluation design, data

collection, and analysis? What evidence do you have that partnership is “improving”

partners? How is data being used?

Evaluation Questions

Contexts and Inputs: - How did your partnership begin? Were there clear rules for membership? - What is the purpose of this partnership? - What is the range of partner members? Any linkers? - How do organizational structures support/inhibit partnering? - What resources exist for partnering: Incentives? PD? Resources? IT? Are they used? Processes and Hard Talk: - What do you talk about? Any patterns? How regularly? Any networks? - Who decides what to talk about? How are agendas formed? - How does data guide your talk? How do you collect this data? - How satisfied are members with partnership? How do you know when it is working? - What are methods of handling conflict? Outputs: Decisions and Actions: - Identify significant decisions. Who participated in this? - How did your decisions relate to grant objectives? - Who made these decisions and by what method? - How are actions of partnership coordinated? - What were benefits & successes of partnership? Success story? - Constraints you could not fix? Challenges? ? Surprises? - Recommendations for follow-ups

Discussion questions today…

1. Who are the partners? What is the setting where partnerships operate?

2. What were moderators & mediating variables?3. What partnering actually occurs?4. What are the outcomes of the partnerships? 5. What are needed follow-ups and recommended

replication strategies?

Five TQE projects in study

Ball State University 6 districts (23 PSD school network), 2 museums, Apple

Cal Poly Pomona 5 districts, 3 CC’s, Southern Edison, Boeing

California State University- Dominguez Hills

1 district, 2 CC’s, 2 other CSUs, NSF SCALE, Boeing, AgileMind

University of Alaska-Anchorage 9 districts, 3 CC’s,

GCI (telecom company)

University System of Maryland 1 district, 1 CC, 3 USM universities, Maryland Business Roundtable

Mixed methods design & strategies

Utilization-oriented evaluation model Mixed data sets

Qualitative:• observations, interviews and self-report surveys (eg, candidates, faculty, district

staff, leaders, mentors), critical friend protocols, context analysis of district/building/classrooms, electronic portfolios, collaboration analysis using NSDC tools, NCATE PDS standards, participant action research,

Quantitative: • performance-based and standardized achievement tests, teacher & administrator

retention and turnover rates, costs and budgeting allocations, GIS data, candidate enrollment and completion rates, policies formalized by partners, publications and grants generated, new partnerships

Contexts to consider The five case studies represent a convenience sample unified by

having the same external evaluation Three are in their fourth year of operation Two fundees have received a second cycle of TQE five-year funding

and are referred to as exemplars by the federal staff Each grantee has at least the following partners:

IHE where both SCDEs and colleges of arts and sciences participate public school districts serving high need, diverse populations for-profit private sector organizations

Examples of partners: SEA, PDS networks, State K-16 Committee, Chancellor’s Office of a state’s university

system, Superintendents’ Roundtable, Provost/Academic Affairs Office, Tech/telecom companies, Museums, Business Roundtable, CC’s

Analysis methods Demographic analysis Content analysis

Open-ended survey items, interviews, extant docs, portfolios Performance analysis of content & pedagogy

Multifactor matched pair method Pre-post score analysis

Management and partnership analysis Data team protocol analysis Group development rubric (Wellman & Lipton, 2004) NSDC IC rubrics Datawise step analysis (Boudett et al, 2005) Causal network analysis

Learnings & Recommendations

Causal network analysis Contextual variables Mediating variables Outcome variables

Contextual variablesExternal influence

No systematic STEM faculty training in teaching; no PD facilitator training;

Awareness of state and national standards

Internal structure

COE programs separate from A&S programs; TED programs divided from Admin programs; demanding faculty workload, RTP policies

Leadership verbally “supports” TQE reform; value of COE for field service vs A&S; mid-career and senior IHE faculty participation

Interactions & resources

STEM/COE faculty interaction competitive; lack 360 degree involvement in grant prep

Networks of discipline based faculty; PDS networks vs “partners”; leveraged resources by PI’s over time

Shared meanings

Lack negotiated definitions by A&S/COE/CC/LEA re: COP, PDS,inquiry, partnership,assmt

Verbal support for diversity and culturally responsive teaching, student centered instruction by COE/LEA

Individual

beliefs

Instructional practice not consistently conscious at all levels

Some A&S individuals interested in pedagogy improvement

Practice Improvements in instruction not guided by data; teacher & admin turnover in high need schools

Role of data enhanced decision making valued “verbally”

Mediating variables

External influence Logic maps tied to $ and SIP; Research-based PD for LEA teachers & STEM IHE faculty, NCATE, shifting state $

Internal structures Workload reduced by grant monies; RTP polices and workload reviewed by administration; LEA release time

Interactions & resources

Grant & LEA interactions become driven by data and action agendas; systematic attention to trust building; partnership networks created/enhanced, national & state standards; leveraging multiple grants/partners across grant

Shared meanings Intentionally built by trained facilitators: CRT, PDS, COP, inquiry, pedagogy techniques, protocols for partnering

Individual beliefs Facilitators uncover tacit assumptions and make them overt

Practice Use interdisciplinary teams of facilitators, A&S go beyond “expertise”, shifting mental models, trained data teams; trained IT integration across COE & A&S, curric mapping

Outcome variablesExternal influence

Admin PD models elusive Developed IHE & LEA faculty PD models;

Internal structure

Workloads unchanged Institutionalized novice IHE faculty PD; partnerships between COE/A&S/LEA faculty in PD/research

Interactions & resources

Dependency on grant funding

Personality referent leadership

Transcending match difficult

New networks created; operationalized for partnering and “divorce”; RTP policies mitigated by personal intervention,strengthened principal involvement; enabled new grants

Shared meanings

A&S discipline currency discourages pedagogical improvement across depts

Changes in A&S/COE views of one another,actions tied to SIP’s; codified in action plans and logic maps

Individual

beliefs

Short term visions predominate Conscious connecting of beliefs with practice; increased valuing of research & pedagogy & alignment with practice

Practice Private sector involvement limited; Changes in IHE/LEA instruction; datawise cycles increase; cohorts/COL develop across partners; Increased research-based instructional practice & assessment

LINC Project-wide Spending 6 years - $10,732,825 total spending

PGCPSS, $5,747,950,

53%

UMCP, $1,566,082,

15%

TU, $793,677, 7%

BSU, $497,895, 5%

MBRT, $40,000, 0%

USM Office, $1,572,042,

15%

PGCC, $515,179, 5%

Partnering strategies Write the original proposal with adequate collaboration among all partners Plan for sustainability & institutionalization across partners from year 1 as reflected in logic

maps Tie budgets for PD and collaborative activities to SIP and “real” school needs Deal with the politics of cost-sharing

Expect incentives to partnering to evolve over time in terms of specificity, collaboration, and resourcing

Overtly plan for changing leadership in IHE & districts & districts Be open and/or actively recruiting new partners Tie partner debriefings to evidence-based action agendas Make funding contingent upon a review process that involves specific requirements for

collaboration across multiple stakeholders and evidence of successful collaboration Invest in “facilitation” training across partners

Example: train STEM faculty in pedagogy using Understanding by Design Plan preservice & induction collaboratively & use cross disciplinary teams

Success depends on… Incentives to partners evolving over time

Genuine collaboration between COE and A&S

Recognizing patterns of resistance & constraints you can’t fix

Leveraging resources: umbrella enabled moderating and mediating that could not be done

at single partner level and harnessed competition to be productive

Collaborations between R-1 and comprehensive universities

Using partners and enhanced resource capacity to solicit new monies

Customizing partnership to harness unique contexts & inputs

Investing in developing networks of facilitators teachers as leaders, data teams, COL, mini-networks, PDS networks

MENU of topics to explore

1. Testing range of certification options Redesigned innovative undergraduate programs Producing STEM graduates Piloted 2+2 program between university and CC

2. Developing Mentoring & Induction to improve retention rates

Created Orientation Program as part of new teacher contract Trained Job-Alike mentors working with new teachers

Developing PDS, mini-networks and communities of practice 3. Leveraging resources with partners

Dealing with increasing match requirements Embedding IT and Culturally responsive teaching Soliciting and maintaining partners--building shared meanings

4. Institutionalization and dissemination strategies1. Short-term2. Long-term

Interactive discussion

Discuss 1-2 topics of most import to your group Notetakers summarize main points to share out in

large group Identify clarifying questions for panel to discuss in

more detail

Panel discussion Open Q&A