From Sea to Shining Sea: Evaluating Title II TQE Partnership
Strategies to Ensure Quality
Susan Ann Tucker, Evaluation & Development Associates LLC
Nancy Shapiro, University System of Maryland
Letitia H. Fickel, University of Alaska, Anchorage
Shirley Lal, California State University, Dominguez Hills
Roy Weaver, Ball State University
Tomas Morales, College of Staten Island
Agenda Partnership challenges Core TQE assumptions and charges 5 questions of focus today Overview of case study projects Findings: causal network analysis Recommendations & follow-up strategies Small group discussion Open Q&A
Core TQE assumptions Partner approach is a vehicle for human and institutional
development Facilitates knowledge creation and diffusion Moves the NCLB agenda forward Makes possible better cost-sharing Contributes to teacher education capacity development
TQE charges Creating collaborative partnerships and learning communities in
high need P-12 schools to enhance student achievement
Improving quality teacher preparation—integrating standards-based assessment strategies and interdisciplinary training and expanding clinical experiences
Articulating a continuum of services from recruitment to induction that serves preservice to beginning teachers in collaborative (and sometimes not so collaborative) settings of school districts, community colleges, and universities
Partnership challenges How to identify premises, tacit and overt, embedded in
“partnership”? What are sources & methods to document partnering
contexts, processes, outcomes? What model(s) to use for evaluation design, data
collection, and analysis? What evidence do you have that partnership is “improving”
partners? How is data being used?
Evaluation Questions
Contexts and Inputs: - How did your partnership begin? Were there clear rules for membership? - What is the purpose of this partnership? - What is the range of partner members? Any linkers? - How do organizational structures support/inhibit partnering? - What resources exist for partnering: Incentives? PD? Resources? IT? Are they used? Processes and Hard Talk: - What do you talk about? Any patterns? How regularly? Any networks? - Who decides what to talk about? How are agendas formed? - How does data guide your talk? How do you collect this data? - How satisfied are members with partnership? How do you know when it is working? - What are methods of handling conflict? Outputs: Decisions and Actions: - Identify significant decisions. Who participated in this? - How did your decisions relate to grant objectives? - Who made these decisions and by what method? - How are actions of partnership coordinated? - What were benefits & successes of partnership? Success story? - Constraints you could not fix? Challenges? ? Surprises? - Recommendations for follow-ups
Discussion questions today…
1. Who are the partners? What is the setting where partnerships operate?
2. What were moderators & mediating variables?3. What partnering actually occurs?4. What are the outcomes of the partnerships? 5. What are needed follow-ups and recommended
replication strategies?
Five TQE projects in study
Ball State University 6 districts (23 PSD school network), 2 museums, Apple
Cal Poly Pomona 5 districts, 3 CC’s, Southern Edison, Boeing
California State University- Dominguez Hills
1 district, 2 CC’s, 2 other CSUs, NSF SCALE, Boeing, AgileMind
University of Alaska-Anchorage 9 districts, 3 CC’s,
GCI (telecom company)
University System of Maryland 1 district, 1 CC, 3 USM universities, Maryland Business Roundtable
Mixed methods design & strategies
Utilization-oriented evaluation model Mixed data sets
Qualitative:• observations, interviews and self-report surveys (eg, candidates, faculty, district
staff, leaders, mentors), critical friend protocols, context analysis of district/building/classrooms, electronic portfolios, collaboration analysis using NSDC tools, NCATE PDS standards, participant action research,
Quantitative: • performance-based and standardized achievement tests, teacher & administrator
retention and turnover rates, costs and budgeting allocations, GIS data, candidate enrollment and completion rates, policies formalized by partners, publications and grants generated, new partnerships
Contexts to consider The five case studies represent a convenience sample unified by
having the same external evaluation Three are in their fourth year of operation Two fundees have received a second cycle of TQE five-year funding
and are referred to as exemplars by the federal staff Each grantee has at least the following partners:
IHE where both SCDEs and colleges of arts and sciences participate public school districts serving high need, diverse populations for-profit private sector organizations
Examples of partners: SEA, PDS networks, State K-16 Committee, Chancellor’s Office of a state’s university
system, Superintendents’ Roundtable, Provost/Academic Affairs Office, Tech/telecom companies, Museums, Business Roundtable, CC’s
Analysis methods Demographic analysis Content analysis
Open-ended survey items, interviews, extant docs, portfolios Performance analysis of content & pedagogy
Multifactor matched pair method Pre-post score analysis
Management and partnership analysis Data team protocol analysis Group development rubric (Wellman & Lipton, 2004) NSDC IC rubrics Datawise step analysis (Boudett et al, 2005) Causal network analysis
Learnings & Recommendations
Causal network analysis Contextual variables Mediating variables Outcome variables
Contextual variablesExternal influence
No systematic STEM faculty training in teaching; no PD facilitator training;
Awareness of state and national standards
Internal structure
COE programs separate from A&S programs; TED programs divided from Admin programs; demanding faculty workload, RTP policies
Leadership verbally “supports” TQE reform; value of COE for field service vs A&S; mid-career and senior IHE faculty participation
Interactions & resources
STEM/COE faculty interaction competitive; lack 360 degree involvement in grant prep
Networks of discipline based faculty; PDS networks vs “partners”; leveraged resources by PI’s over time
Shared meanings
Lack negotiated definitions by A&S/COE/CC/LEA re: COP, PDS,inquiry, partnership,assmt
Verbal support for diversity and culturally responsive teaching, student centered instruction by COE/LEA
Individual
beliefs
Instructional practice not consistently conscious at all levels
Some A&S individuals interested in pedagogy improvement
Practice Improvements in instruction not guided by data; teacher & admin turnover in high need schools
Role of data enhanced decision making valued “verbally”
Mediating variables
External influence Logic maps tied to $ and SIP; Research-based PD for LEA teachers & STEM IHE faculty, NCATE, shifting state $
Internal structures Workload reduced by grant monies; RTP polices and workload reviewed by administration; LEA release time
Interactions & resources
Grant & LEA interactions become driven by data and action agendas; systematic attention to trust building; partnership networks created/enhanced, national & state standards; leveraging multiple grants/partners across grant
Shared meanings Intentionally built by trained facilitators: CRT, PDS, COP, inquiry, pedagogy techniques, protocols for partnering
Individual beliefs Facilitators uncover tacit assumptions and make them overt
Practice Use interdisciplinary teams of facilitators, A&S go beyond “expertise”, shifting mental models, trained data teams; trained IT integration across COE & A&S, curric mapping
Outcome variablesExternal influence
Admin PD models elusive Developed IHE & LEA faculty PD models;
Internal structure
Workloads unchanged Institutionalized novice IHE faculty PD; partnerships between COE/A&S/LEA faculty in PD/research
Interactions & resources
Dependency on grant funding
Personality referent leadership
Transcending match difficult
New networks created; operationalized for partnering and “divorce”; RTP policies mitigated by personal intervention,strengthened principal involvement; enabled new grants
Shared meanings
A&S discipline currency discourages pedagogical improvement across depts
Changes in A&S/COE views of one another,actions tied to SIP’s; codified in action plans and logic maps
Individual
beliefs
Short term visions predominate Conscious connecting of beliefs with practice; increased valuing of research & pedagogy & alignment with practice
Practice Private sector involvement limited; Changes in IHE/LEA instruction; datawise cycles increase; cohorts/COL develop across partners; Increased research-based instructional practice & assessment
LINC Project-wide Spending 6 years - $10,732,825 total spending
PGCPSS, $5,747,950,
53%
UMCP, $1,566,082,
15%
TU, $793,677, 7%
BSU, $497,895, 5%
MBRT, $40,000, 0%
USM Office, $1,572,042,
15%
PGCC, $515,179, 5%
Partnering strategies Write the original proposal with adequate collaboration among all partners Plan for sustainability & institutionalization across partners from year 1 as reflected in logic
maps Tie budgets for PD and collaborative activities to SIP and “real” school needs Deal with the politics of cost-sharing
Expect incentives to partnering to evolve over time in terms of specificity, collaboration, and resourcing
Overtly plan for changing leadership in IHE & districts & districts Be open and/or actively recruiting new partners Tie partner debriefings to evidence-based action agendas Make funding contingent upon a review process that involves specific requirements for
collaboration across multiple stakeholders and evidence of successful collaboration Invest in “facilitation” training across partners
Example: train STEM faculty in pedagogy using Understanding by Design Plan preservice & induction collaboratively & use cross disciplinary teams
Success depends on… Incentives to partners evolving over time
Genuine collaboration between COE and A&S
Recognizing patterns of resistance & constraints you can’t fix
Leveraging resources: umbrella enabled moderating and mediating that could not be done
at single partner level and harnessed competition to be productive
Collaborations between R-1 and comprehensive universities
Using partners and enhanced resource capacity to solicit new monies
Customizing partnership to harness unique contexts & inputs
Investing in developing networks of facilitators teachers as leaders, data teams, COL, mini-networks, PDS networks
MENU of topics to explore
1. Testing range of certification options Redesigned innovative undergraduate programs Producing STEM graduates Piloted 2+2 program between university and CC
2. Developing Mentoring & Induction to improve retention rates
Created Orientation Program as part of new teacher contract Trained Job-Alike mentors working with new teachers
Developing PDS, mini-networks and communities of practice 3. Leveraging resources with partners
Dealing with increasing match requirements Embedding IT and Culturally responsive teaching Soliciting and maintaining partners--building shared meanings
4. Institutionalization and dissemination strategies1. Short-term2. Long-term
Interactive discussion
Discuss 1-2 topics of most import to your group Notetakers summarize main points to share out in
large group Identify clarifying questions for panel to discuss in
more detail
Panel discussion Open Q&A