Download - Cindy Willey Washington Institute for Mental Illness Training and Research [email protected]
Building and Maintaining Consumer Organizations:
Using Evaluation to Support Consumer Delivered Services
Cindy WilleyWashington Institute
for Mental Illness Training and [email protected]
Eric J. BrunsUniversity of Washington
Division of Public Behavioral Health and Justice [email protected]
Description of the session Introductions
Why did we attend this session? Setting the stage
A brief overview of research and evaluation Why is it relevant for consumers and families?
Consumer-led evaluations History and examples
Brainstorming and group discussion Evaluating your program in Washington State
Getting Ideas and Feedback The Washington State Consumer and Family
Evaluation Mini-Grant Program
Consumers, family members, and researchResearch has led to many breakthroughs
in treatment and rehabilitationSuch breakthroughs are only possible
through teamworkSo why is research sometimes not seen as
relevant? Research as a mystery Research as something consumers don’t
participate in Research as something not relevant to everyday
life
Research Questions from consumers What factors help people to recover?
Or, what factors hinder recovery? What kinds of jobs supports can help people with
psychiatric disabilities succeed? What factors make treatment most useful and
meaningful? How does spirituality affect recovery? What is the relationship between mental health and
physical health? How do budget cuts affect the mental health of
consumer/survivors? What is the impact of involuntary commitment on
someone’s mental health? What are the benefits or risks of taking a specific
medication?
Program Evaluation A growing emphasis on outcomes
(Outcomes = Documented result or impact of a program, intervention, or process…)
A positive development for consumers? All programs will have to show how services helped to
improve the lives of their clients (i.e., not just how many people they served or how many
units of service were provided) A negative development for consumers?
For consumer-run programs, more responsibility for conducting research or evaluation that shows whether these programs are beneficial to their participants.
How to make research and evaluation more relevant More useful products – “real-world” information focused
on daily living (employment, transportation, housing) Research reports in more simple language and in varied
formats that are user- friendly Better understanding by researchers of cultural
differences and needs. Recruit, train and hire more researchers with disabilities
and from minority backgrounds. More public education should be provided. Consumers should be full participants in planning and
doing research Build greater trust between researchers and consumers.
From the National Center for the Dissemination of Disability Research and Research Utilization Support and Help (RUSH) project
Evaluation Methodology, Design and Analysis:
An Overview
Process and Outcome Evaluation Process Evaluation (a.k.a. implementation assessment, formative
evaluation, intervention fidelity) Is the program being delivered as intended? Is the program being delivered to who it was intended?
Outcome Evaluation (a.k.a. impact assessment, summative evaluation) What is the effect of the program on the program recipients? Is the program beneficial? For whom is the program beneficial? More for some than
others?
Process evaluation examplesConsumer satisfactionConsumer-oriented valuesService delivery and practice
Availability General assessment of service quality Treatment quality / treatment fidelity
System development Service coordination and collaboration Assessment of community capacity
Why process evaluation?Testing assumptions in an evaluation of respite care
Information about respite availability and respite’s benefits will reach the target audience
When parents and providers hear of the program, they will refer/be referred to the program
Respite workers will be well-trained to deal with problem behaviors Respite workers will be able to offer activities during respite hours that
are attractive and beneficial to the child and the parent Parents will be able to use allotted respite care time in a way that
reduces stress Respite will be of adequate intensity and duration to be of benefit to
children and parents Parents and children will perceive the respite care program to be helpful Respite provides a “value-added” to other services and supports
Outcomes Evaluation:Experimental Evaluation Design
Random assignment into intervention and control groups
Group comparabilityMinimize threats to internal validity
• Internal validity: Are the outcomes associated with the intervention; Can we say that the intervention causes the outcome?
Can only be used to evaluate programs that do not target entire populations
Experimental Designs: The Realities
Offer best assessment of cause and effect
Offer best linkage between program and outcomes
The “Gold Standard” in determining “Evidence Based Treatments”
May not be ethical May not be legal May not be feasible
Examples of Quasi-Experimental Designs (when program does NOT target entire population)
Post-test only comparison group designs Two groups (intervention and comparison) observed following the
presentation of intervention No random assignment to groups No pre-testing
Pre-test post-test comparison group designs This is the most frequently used design in social science evaluation
research. Involves a pre and post intervention measurement for the two groups (intervention and comparison), no random assignment to groups.
Examples of Quasi-Experimental Designs (can be used when program targets entire population)
Repeated measures designs Can be used with just the intervention group or with two groups
(intervention and comparison). Several measurements over time (e.g., baseline, post intervention, one year subsequent, etc.)
Time series designs Expanded repeated measures, include many observations of
either just the intervention group or two groups (intervention and comparison)
Intervention may be introduced and removed and reintroduced At least 50 observations points required
Data Collection Methods Self-report: Individuals report/describe their own
behavior/state of mind/feelings/etc. OR some other informant provides information about the
individual of interest (e.g, parent reports on child behavior)
Observation: Researcher observes and records (in some way) the behavior of interest.
Archival Data/Existing Records/MIS
Evaluation Principles to considerTheory-based
What are the expected outcomes? Why do we think we will get them?
Utilization-focused What are the priorities for our evaluation resources?
• Participatory• How do we make sure the evaluation is informed and used by all the
relevant stakeholders?
Theory-based: Clubhouse model
Clubhouse
This example courtesy of Dori Hutchinson, USPRA Research Committee
Utilization focused:Asking questions about “What to measure?”
Types of questions: Evaluation use What decisions, if any are the evaluation’s
findings expected to influence? When will the decisions be made? By whom? When, then, must the evaluation findings be
presented to be timely and influential?
Utilization focused:Asking questions about “What to measure?”
Types of questions: What data is needed? What data and findings are needed to support
decision-making? Are the data available through existing
means? …Or, are new data collection approaches
needed?
Participatory Evaluation Broadly identify stakeholders and include more non-
evaluators than evaluators in the evaluation (e.g., consumers, families, youth, program directors, community groups, etc.). A “stakeholder” = Anyone who has a stake in
the program that is being evaluated!! The group of stakeholders of a program and its evaluation may
not be static. Regularly update stakeholder groups on the data collection
process. Learn how it can be improved, refined, enhanced. Convene stakeholder group to design data dissemination plan. Disseminate results regularly to and with stakeholders
HISTORY AND EXAMPLES OF CONSUMER-LED EVALUATION
CINDY WILLEYWASHINGTON INSTITUTE
In 1979 Prager and Tanaka reported to the Ohio Department of Mental Health on the results of involving mental health consumers in evaluation. They concluded: “Representing the consumer’s perspective on the meaning of mental illness and the correlates of ‘getting better,’ the process of client involvement in evaluation design and implementation is not only realistic and feasible; it is, we feel, a professional necessity whose time is overdue.”
THE WELL-BEING PROJECTA LANDMARK STUDY
1989Funded by the California Department of
Mental Health Office of PreventionFirst consumer research project:
Conducted by the California Network of Mental Health Clients Jean Campbell, Principal Investigator Ron Schraiber, Co-Investigator
Discovering the Consumer Perspective Asking New Questions
negative outcomes identified self-management skills revealed importance of personhood established effects of prejudice and discrimination
quantified incongruity of values, perspective, and
identified needs between consumers, family members, and mental health professionals found
Findings 40% of mental health clients surveyed felt that “all”
or “most of the time” people treated them differently when they found out they have received mental health services:like they are violent (16%)like a child (21%)like they don’t know what is in their own best interest (31%)like they are incapable of caring for children (20%)like they are incapable of holding a job (33%)
Findings
More than half of the clients surveyed indicated that they “always” or “most of the time” recognize signs or symptoms of psychiatric problems and that they can take care of these problems before they become severe.
Findings
48% indicated that they have avoided treatment due to fear of involuntary commitment.
30% reported that they had “little” or “no” control over the kind of mental health services they receive.
HOW CONSUMERS BECAME PART OF THE MHSIP COMMUNITY
MHSIP = Mental Health Statistics Improvement Program
First consideration of adding consumers & family members came in the early 90’s.
On the local and state levels consumers were beginning to be involved in policy and evaluation
FIRST STEPS
STATE PLANNING COUNCILS INCLUDE CONSUMERS
CONSUMER REPRESENTATIVES JOIN MENTAL HEALTH BOARDS
ANN LODER PRESENTS PLENARY “HOPE WITH A CAPITAL H” AT NATIONAL MENTAL HEALTH STATISTICS CONFERENCE (1991)
JEANNE DUMONT BECOMES FIRST CONSUMER MEMBER OF MHSIP AD HOC ADVISORY GROUP (1992)
MOVING FORWARD
OFFICES OF CONSUMER AFFAIRS ARE ESTABLISHED ACROSS THE NATION
THE CONSUMER/SURVIVOR MENTAL HEALTH RESEARCH AND POLICY WORKGROUP IS FORMED (1991)
THE MHSIP MENTAL HEALTH CONSUMER-ORIENTED REPORT CARD (1994-PRESENT) CONSUMERS AND MENTAL HEALTH PROFESSIONALS COLLABORATE TO PRODUCE AN OUTCOME TOOL GROUNDED IN CONSUMER VALUES
CONSUMER OPERATED SERVICES PROGRAM (COSP) MULTI-SITE
RESEARCH INTITIATIVEhttp://www.cstprogram.org/cosp/
Consumer and non-consumer interviewers were employed from 1998-2002 for this SAMHSA project. The study took place in 7 different states. (Connecticut, Florida, Tennessee, Missouri, Illinois, Maine, Pennsylvania)
Among 6 of 7 study sites providing information about consumer hiring, the composition of their interviewing staff, it was reported that 2/3 of the project interviewers were consumers.
CURRENT INFORMATION ON CONSUMER-LED EVALUATION
MHSIP - http://www.mhsip.org/index.aspNational Empowerment Center -
http://www.power2u.org/SAMHSA - http://www.samhsa.gov/CONTAC - http://www.contac.org/Directory of Consumer- Driven Services –http://www.cdsdirectory.org/Self-Help Clearinghouse -
http://www.mhselfhelp.org/index.htmCenter for Psychiatric Rehabilitation Boston U. -
http://www.bu.edu/cpr/index.shtml
New York Association of Psychiatric Rehabilitation Services (NYAPRS)
PEER BRIDGER PROJECTPeer Bridger Project Outcomes
A participatory evaluation of the project prepared by Cheryl MacNeil, Ph.D. identified and examined several
areas where the project benefited those involved:
The development of uniquely different relationships than one typically experiences in a psychiatric hospital
The temporary relief from a state of social isolation and physical enclosure often experienced by those hospitalized
The ability to share with each other the experiential wisdom and survival skills necessary for the process of recovery
Rehospitalization Rates:
"The most substantial finding is that the follow-up rehospitalization rate of Matches while enrolled in the Peer Bridger Project was
significantly less than the baseline hospitalization rate (i.e.. the 2-year period prior to enrollment).
That is, during the 2-year baseline period, the Matches were hospitalized an average of 60% of the time, while enrolled in the
program, however, they were rehospitalized only 19% of the time. That's an improvement of 41%!". (National Health Data Systems,
December 1998)
For further information about the Peer Bridger Project please fill out our online request form or contact Tanya Stevens,
Peer Bridger Project Director, by phone at 518-436-0008 x18 or by e-mail at [email protected].
EDITORIAL
“I believe the real mental health transformation will come when our practice is guided by people first values. Values fundamental to our culture, such as self determination, freedom, opportunity and choice, must be fundamental to our practice. “
“When I look at the field of mental health services research, I see a field in which significant research typically follows significant changes in practice.”
EDITORIAL CONTINUED“From my perspective, what currently is contributing
most to delaying the creation of recovery-oriented, people first practices is not the absence of critical science underlying our practices, but the absence of the critical values underlying our practices. And if our practices were grounded in these people first values, subsequent research could study how to best incorporate these values into our program and system policies, procedures, and documentation. Science can certainly help close the gap between values and practice.”
William A. Anthony Summer 2005 Psychiatric Rehabilitation Journal
Group discussionEvaluation questions What does your organization have to
offer to improved health and wellness of clients or the community?
What would you like to evaluate with respect to those impacts?
What else could your organization learn in order to improve its effectiveness?
What evaluation questions would be related to the above questions?
Group discussionEvaluation methods How might you gather data on those
evaluation questions? What would you need to gather, analyze,
report, and use the data? What kind of support would you need?
Examples: A list of existing instruments, how to conduct a focus group, how to create a measure, how to analyze data, how to do a report
Washington State Consumer and Family Evaluation Grant ProgramHow best to provide support to consumer
organizations and consumer evaluators? Ideas:
Conferences and educational opportunitiesAn RFP and Grant process for consumer and
family organization evaluation and consumer-led evaluations
Consultation and technical assistance to support consumer and family-led evaluation projects