assessing for program improvement presented at the university of arizona february 11, 2009 victor m....
TRANSCRIPT
Assessing for Program Improvement
Presented at the University of ArizonaFebruary 11, 2009
Victor M. H. BordenAssociate Professor of Psychology (IUPUI)
Associate Vice President for University Planning, Institutional Research, and Accountability (IU)
Overview
What I think you think I might talk about
What I think you need to think about
How to Assess Programs for Improvement
A range of methods from simple to complex
The Core IdeaSimple modelsQuality improvement modelsProgram reviewMore complex models
The Core Idea: The Planning-Evaluation-Improvement
CyclePlan
Implement
Assess
ImprovePlan
Check
Do
Act
Adapted from Norman Jackson
Toward a Spiral of Improvement2. ENGAGE WITH THE
PROBLEM CALLED HOW DO WE IMPROVE PROGRAM?
5. EVALUATE IMPACT ON OUTCOMES
* did it work as I intended?
* how did people respond?
* what were the results?
6. PLAN TO IMPROVE
1. THINK ABOUT PROGRAM ISSUES
3. DEVELOP RESOURCES/ STRATEGIES TO IMPROVE
4. IMPLEMENT CHANGES
* experiment
Back
to t
he d
raw
ing b
oard
On
to s
om
eth
ing e
lse
Core Evaluation Cycle Questions
What are you trying to achieve?
Needs assessment
What are you doing to achieve it?
Process assessment
How will you know when you get there
Outcomes assessment
What can you do with the results?
Improvement
Why the Fixation on Outcomes?
We haven’t paid sufficient systematic attention to this in the past
We look at inputs (resources) and processes (curricula and programs) fairly systematically
We tend to look at outcomes one student at a time
The link to accountability
Simple Models of Assessment
AdvantagesEasy to communicate, use, and learn from
Can be built into everyday work
Helps build and maintain culture of evidence
ModelsThe evaluation cycle (or spiral)
The assessment matrix template
The Assessment Matrix
The Limits of Simple Models
Often overly simplistic relative to problems
Actual measures can be misguided
Implementation can be inconsistent across units
Not always easy to link outcome measures to “responsible” processes
doing the right thing vs. doing it right
Quality Improvement Models
AdvantagesFocus on process provides best chances for identifying points of improvementCollaborative teams empower staff and help improve communication across unitsFormulaic method and external staff support help guide and keep on track
Sample methodsPenn State’s FAST TRACKU of Wisconsin Accelerated Improvement
PSU Fast Track
Team # 526 -- Food Sciences Measures Task Force College of Agricultural Sciences December 2002 ObjectiveDevelop and implement a centralized system for collection and reporting of key performance indicators and departmental reports.
Action Plan: 1. Evaluate current processes and data sources for gathering data for performance indicators and department reports.2. With information from #1, work with Dept. Head to define "key performance indicators" fromDepartment's Strategic Plan.3. Working with Dept. Head, define "departmental reports and other measures".4. Develop feasible solutions for a collection process for data/information defined in #2 and #3.5. Present solutions to the Sponsor with cost/benefit analyses.6. Sponor to disseminate solution and plan to faculty.7. Assist in implementation of solutions including initial collection of information (test cycle for newprocess), revising process, flowcharting and writing procedures and training stakeholders.
Results Achieved to DateTeam was disbanded at the end of 2003 after achieving action plans #1 - 7. Most of the expected outcomes were achieved for a fundamental centralized data collection system. Given the current resource and budget constraints, the sponsor decided not to pursue further automation of the centralized data system at the time. The Food Sciences Task Force was disbanded at the end of 2003.
http://www.wisc.edu/improve/improvement/accel.html
UWisc Accelerated Improvement
Define Goals and measures of success
Document process
Understand customer needs
Check/refine goals
Design Develop potential solutions
Analyze solutions/options
Finalize solution develop implementation plan
Implement Inform affected people
Conduct training, if needed
Execute action plans w/timeline
Follow-up Collect data to track improvement
Review and refine process changes
Issue final report with results
Limits of QI Models
Academicians wary of “business” models
Focus on process emphasizes doing it right over doing right thing
Can be episodic rather than continuous
Program Review
Program self-study, site visit by “peers”Common method for academic programs
Increasing use for administrative programs
Fits well with accreditation frameworkGuidelines shape tone and tenor
Content standardsReview team composition
Flexibility accommodates range of inquiry orientations
Limits of Program Review
Expensive and time-consuming
Can be done with little participationOr with a lot
Results not always directly useful for change
Memorandum of understanding helpful
Episodic nature not responsive to changing environment
More Complex Models
AdvantagesHandle true complexity
Provide in-depth insight into context
Academicians respect the scholarship (although not necessarily the particular approach)
Examples (from WMU’s evaluation center)
CIPP
Constructivist Evaluation
More Complex Models
The CIPP Model
1. Contractual Agreements2. Context Evaluation3. Input Evaluation4. Process Evaluation5. Impact Evaluation6. Effectiveness Evaluation7. Transportability Evaluation8. Sustainability Evaluation9. Metaevaluation10.The Final Synthesis Report
Constructivist Evaluation
Guba & Lincoln (2001)Two-stage process
Discovery - effort to describe “what’s going on here,” the “here” being the evaluand and its context Assimilation - effort to incorporate new discoveries into the existing construction or constructions …so that the new construction will fit, work, demonstrate relevance, and exhibit modifiability.
Limits of Complex Models
Too complex to be practical
Expensive
They require an…“evaluation unit as a staff operation at a high level of the organization in order to help insulate the unit from inappropriate internal influences and enhance its influence on decision making .”
Take Home Points
There are many approaches to assessing for improvement
Virtually any method of inquiry can be accommodated
The point of all of them is to determine how well you are doing things and how they might be done better; and to then try doing better and to see if that improves the outcomesEach can be done well or poorly
Doing Assessment Well
Being “data-” or “evidence-driven” is not, in and of itself, a good thing
e.g., “selective use” of evidence to support a foregone conclusion
Torture numbers long enough and they’ll confess to anything
Effective use of data requires sharing diverse and often divergent perspectives
It’s not what the data say, it’s what you say about the dataSome disagreement and dissent is important to learning and innovation
Further Heresy
Building effective programs requires some level of irrationality and disorder
To learn from what we do requires that we unlearn some things that we often don’t want to unlearn
As if doing this by ourselves were not difficult enough, we must do this together
From Data- to Learning-Driven
Data-driven implies…Rational, systematic testing of ideas through inspection of facts
sequential, often individual decision-making process
Learning-driven implies…Going beyond what we already know and can do to gain new competencies
Deconstruction and reconstruction of ideas and beliefs
Becoming irrational to become re-rational
Single- and Double-Loop Learning
Learning is the detection and correction of error (unintended consequences)“Governing Variables” are those things what we feel are important to keep within acceptable limits“Action Strategy” is what we do or plan to do to keep the governing variables within limits“Consequences” are the intended and unintended outputs and outcomes
Intended: confirm our theory in useUnintended: suggests error in our theory in use
Single-Loop Learning
Governing variables not called into question
Adjustments made to action strategies at best
Defense mechanisms can readily arise to maintain single-loop learningGoverningVariables
ActionStrategies
Conse-quences
Double-Loop Learning
Questioning the role of the framing and learning systems which underlie actual goals and strategies
Reflection is fundamentalBasic assumptions are confronted
Hypotheses publicly tested
Falsification is sought
Ego is laid aside
GoverningVariables
ActionStrategies
Conse-quences
Model I and II Org Learning
Single- and double-loop learning at the organizational level
Model I: Organizational members prescribe to a common theory in use
Organizational policies and practices inhibit change
Model II: Governing values, policies, and practices promote double-loop learning
A Model I Learning Organization
Governing VariablesTow the lineWin at all costsSuppress negative feelingsEmphasize rationality
Action StrategiesControl environment and task unilaterallyProtect self and others unilaterallyDiscourage inquiry
ConsequencesDefensive relationshipsLow freedom of choiceReduced production of valid informationLittle public testing of ideas
A Model II Learning Organization
Governing VariablesValid information is most importantFree and informed choiceShared internal commitment
Action StrategiesShared controlParticipation in design and implementation of action
ConsequencesMinimally defensive relationshipsHigh freedom of choicePublic testing of ideas
Participatory Action Research/Inquiry
Systematic inquiry process Can use any of aforementioned methods
Stakeholder empowerment through active and on-going participation Dialog throughout process promotes collaborationActive learning and discovery fostered by critical reflection process Action plans create shared responsibility for doing something with the resultsFollow-up to action (checking results) maintains relationships and commitments
Participatory Action Research/Inquiry
Quotes from Handbook of Action Research by Peter Reason http://www.bath.ac.uk/~mnspwr/Papers/HandbookIntroduction.htm
The aim of participatory action research is to change practices, social structures, and social media which maintain irrationality, injustice, and unsatisfying forms of existence.
(Robin McTaggart)
Participatory research is a process through which members of an oppressed group or community identify a problem, collect and analyse information, and act upon the problem in order to find solutions and to promote social and political transformation.
Daniel Selener
We must keep on trying to understand better, change and reenchant our plural world.
Orlando Fals Borda
Participatory Action Research
Who does what?Decides what actions are taken?Is responsible for effective implementation?Can devise appropriate evaluation protocols?Has access to or can collect appropriate evidence?Reviews the results and decides what to do?
What can be done to get these people to work together and in concert?
Example: Evaluation of New Student Orientation
Research Question and Evaluation Focus reassessment of goals; incoming students’ needs; impacts on knowledge, attitudes, and behaviors
Data Collectionfocus groups and questionnaires, sought perspectives of all major stakeholders
Data Reporting and Feedbackmeetings with orientation leaders and faculty stakeholders
Development of Action Plansfacilitation of dialogue and data-driven proposals
Actionimplementation of proposed changes
Assessment – on-going formative evaluation; re-administration of process and outcome instruments
Example: Indiana Project on Academic Success (IPAS)
Research-based inquiry for enhancing academic successFour-stage method
AssessmentOrganizingAction InquiryEvaluation
Supported by use of state and institutional student tracking records
Stage 1: Assessment
Compare campus assessment information to statewide assessment results; identify possible challengesCollect additional information from campus sources, such as prior reports and studies and focus group interviewsOrganize teams of administrators, faculty, professional staff, and students to identify critical challenges on the campusPrioritize the challenges, identifying two or three that merit special attention at a campus level
Stage 2: Organizing
Coordinate the assessment and inquiry process with campus-level planning and budgeting; integrate the challenges with strategic plans; coordinate budgeting to provide necessary support. Appoint workgroups to address critical, campus-wide challenges; consider providing release time to team leaders to work on tasks for the campus. Coordinate the inquiry process (activities of the workgroups) with campus planning and budgeting.
Stage 3: Action Inquiry
Build an Understanding of the ChallengeWhat solutions have been tried in the past, and how well did they work? What aspects of the challenge have not been adequately addressed? What aspects of the challenge require more study? Develop hypotheses about the causes for the challenges using data to test the hypotheses. Do the explanations hold up to the evidence?
Look Internally and Externally for SolutionsTalk with people on campus about how they have addressed related challenges. Consider best practices for retention and how they might be adapted to meet local needs. Visit other campuses that have tried out different approaches to the problem. How well would these alternatives address the challenge at your campus?
Assess Possible SolutionsConsider alternatives in relation to the understanding of the problem developed in Stage 3, step 1. Will the solutions address the challenge at your campus? How can the solution be pilot tested? If you tried out the solution, how would you know if it worked? What information would you need to know how well it worked?
Stage 3: Action Inquiry (cont.)
Develop Action PlansAction plans should address the implementation of solutions that should be pilot tested. Consider solutions that can be implemented by current staff. If there are additional costs, develop budgets for consideration internally and externally. (Remember, seeking additional funds can slow down the change process.) Develop action plans with time frames for implementation and evaluation
Stage 4: Evaluate
Implement Pilot Test and EvaluateProvide feedback to workgroups and campus coordinating team. Use evaluation results to refine the solution. Also, evaluation can be used as a basis for seeking additional funding from internal and external sources, if needed
Building Trust – Lowering Resistance to Change
Do…• Evaluate program
effectiveness• Provide incentive for
using information (regardless of results)
• Raise expectations regarding quality and use of evidence
• Be patient with the learning curve
• Raise expectations for learning (for students and colleagues)
Don’t…• Evaluate individual
effectiveness• Tie resource
allocation directly to results
• Beat people over the head with findings
• Confuse anecdotes with evidence
• Keep changing direction based on initial findings
• Lower expectations for learning
What’s the Point?
Assessment and evaluation are means not endsOther important ingredients include:
Bringing the right people togetherA climate of trust and experimentationIncentives and support
It’s not rocket scienceAn imprecise answer to the right question is much more useful than a precise answer to the wrong question