how do measures measure up?. what is the centre? knowledge bringing people and knowledge together to...
TRANSCRIPT
How do measures measure up?
What is the Centre?What is the Centre?
Knowledge
Bringing people and knowledge together to promote the best mental health and
well-being for every child and youth
Capacity Partnerships
Full continuum of effective and accessible mental health services for children and youth.
Brief IntroductionsBrief Introductions
Name and affiliation Program to be evaluated One outcome of interest
ObjectivesObjectives
To provide an overview of concepts on measurement
To provide guidelines for assessing and selecting measures
To reduce fear on the topic of measures and indicators
OutlineOutline
A. What is measurement?
B. What are the different sources of data and types of measures?
C. What affects the quality of measures?
D. How can we select appropriate measures for our evaluation?
A. What is measurement?A. What is measurement?
Getting Ready for Getting Ready for EvaluationEvaluation
Evaluation Heaven!
Evaluation Heaven!
Develop Logic Model
Identify Priority Outcomes
Review and Select Measures
Develop Data CollectionProcedures
What is measurement?What is measurement?
Measurement refers to the process of “operationalizing” the evaluation question
Evaluation question
Information needed
Sources of information
Types of data
Evaluation Measure
Sample Evaluation Question: What are clients’
perceptions of our services? Has client satisfaction with
our services improved? What areas of the program
do we need to improve on?
Evaluation question
Information needed
Sources of information
Types of data
Evaluation Measure
Information from parents and/or youth on:
Satisfaction with location Ease in accessing the services Perceptions of outcomes as a
result of the service Appropriateness of the
services Perceptions on therapeutic
alliance or relationship
Evaluation question
Information needed
Sources of information
Types of data
Evaluation Measure
B. What are the different B. What are the different sources of data and sources of data and types of measures?types of measures?
Levels of measurementLevels of measurement
Nominal Ordinal Interval Ratio
Sources of informationSources of information Questionnaires Interviews or focus groups Observation Administrative data and/or Health records
Sources of information from parents and/or youth on how they experience our services:
Interview Focus groups Administrative data: number and
type of complaints Self-administered survey Standardized questionnaire
Evaluation question
Information needed
Sources of information
Types of data
Evaluation Measure
Qualitative data: verbal and pictorial Numeric scores:
• Basic units such as frequency, duration, length• Scores from rating scales• Standardized scores (z scores)• Age- or grade-equivalent or adjusted scores
Criteria for evaluating scores:• Standardization: procedural comparability• Psychometric properties
Types of DataTypes of Data
Type of data on clients’ satisfaction or perception of care
Scores on a standardized measure on client perception of care: • Average of ratings on all
items• Changes in average ratings
every 6 months
Evaluation question
Information needed
Sources of information
Types of data
Evaluation Measure
C. What affects the quality C. What affects the quality of measures?of measures?
Cultural and Historical ContextCultural and Historical Context Use of tests and measures as a
“cultural tool” Cultural appropriateness of the
measures Revisions or updates to
measures to accommodate changes in scores in the population
Psychometric Properties: ValidityPsychometric Properties: Validity
Does the measure REALLY measure what it is supposed to measure?
Face validity
Content Validity
Construct, Construct, Criterion Criterion
and and FactorialFactorial
PredictiveConcurrent
Discriminant or Divergent
Psychometric Properties: ReliabilityPsychometric Properties: Reliability
= Dependable
= Trustworthy
= Same old, Same old
Cost and availability Time for administering, scoring
and analyzing Staff involvement Information management
Psychometric Properties: FeasibilityPsychometric Properties: Feasibility
Clinical utility Culturally appropriate Administrative uses for decision-making Usefulness for improving program Usefulness for information on public reports
Psychometric Properties: Psychometric Properties: RelevanceRelevance
Other measurement considerations Other measurement considerations when involving childrenwhen involving children
Developmental level Educational level Health status Family
D. How can we select D. How can we select appropriate measures for appropriate measures for
our evaluation?our evaluation?
Criteria for selecting measuresCriteria for selecting measures
Evidence-based
Feasible
Relevant and meaningful
Evidence-basedEvidence-based
Sound psychometric properties: valid and reliable
Used in similar settings Recommended by experts, if no previous
literature
Sample Summary TablesSample Summary Tables
Sample Summary TableSample Summary Table
Name of Measure
Reference or Developer
Areas measured
Evidence on reliability and validity
Feasibility
(cost, training, who administers)
Relevance Comments
Evaluation question
Information needed
Sources of information
Types of data
Evaluation Measure
General Information
Developer/ Authors Children’s Indicator Workgroup of Sixteen State Study, Centre for Mental Health Services.
Date of publication, versions available
2001.Separate forms for parents and youth.
Constructs measured The YSS is used as a measure of youth service usage and satisfaction with services.The YSSF is used as a measure of parent report of youth service usage and satisfaction with services.
Population for which designed
Adolescents ages 13 and up.
Method of administration
Self-report questionnaire, paper or telephone interview.
Subscales and number of items
25 items.Five scores based on a factor analysis can be obtained:Good access to servicesParticipation in treatmentCultural sensitivity of staffAppropriateness of servicesPositive outcome of service
Estimated time to administer
Not mentioned.
Costs, availability and permission to use
In public domain. See online resources below.
Examiner qualifications and training requirements
Information not available at this time.
Youth Services Surveys (YSS, YSSF)
Technical Information
Sample for development of norms
The YSS was created as part of the Mental health Statistics Improvement Program for the Sixteen State Study.
To date 430 youth have completed the survey.
Reliability Reliability indices for the 5 factors of the survey are good: Access = .713, Participation in Treatment = .776, Cultural Sensitivity = .863, Appropriateness = .863, Outcome =.893 (Brunk et al 2000)
Validity This indicator was developed by the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) for the National Mental Health Performance Measures, and feasibility was assessed across 16 states (Lutterman et al, 2003). The Virginia Department of Mental Health, Mental Retardation, and Substance Abuse Services coordinated the development of the surveys for children and youth (Brunk et al 2000).
The survey is widely used in the U.S. and reports by various states are available in the grey literature.
Uses The YSS may be used for the following:(1) As a data collection method for the following: Client perception of care, consumers
linked to physical health services, children in family-like arrangements and other 24-hour residential care programs.
(2) As a clarification method for findings from other indicators.(3) As a measure of penetration (when providers serve a higher than average
percentage of youth with a particular problem than occurs in the general population.
Review and Comments
Applicability for Ontario
Information not available at this time.
Other considerations
Information not available at this time.
Resources
Key References Brunk, M. (2001). Youth Services surveys Guidelines for administration. [Power Point presentation].
Brunk, M., Koch, J.R., McCall, B. (2002). Report on Parent satisfaction with Services at Community Services Boards. Virginia Department of Mental Health, Mental Retardation and Substance Abuse Services.
Online Resources http://www.mhsip.org
Contact information
E-mail : [email protected].
Youth Services Surveys (YSS, YSSF) http://www.mhsip.org
Evaluation question
Information needed
Sources of information
Types of data
Evaluation Measure
Measures of Therapeutic Alliance
Blatt & Zuroff 2005: Importance of including therapeutic alliance as a factor moderating treatment outcomes science direct link
Elvins & Green 2008: Review of concept and measures on therapeutic alliance science direct link2
Green 2006: Measures on therapeutic alliance for child and youth mental health interscience link
Blatt & Zuroff 2005: Importance of including therapeutic alliance as a factor moderating treatment outcomes science direct link
Elvins & Green 2008: Review of concept and measures on therapeutic alliance science direct link2
Green 2006: Measures on therapeutic alliance for child and youth mental health interscience link
Evaluation question
Information needed
Sources of information
Types of data
Evaluation Measure
Measures on Cultural Competency
Cultural Competency Self-Evaluation Questionnaire friends link
Some common issuesSome common issues
To translate or not Picking and choosing items from various
measures Making sense of data: indicators,
benchmarks and standards
Decision process for identifying, Decision process for identifying, selecting and using measuresselecting and using measures
Specifytopic
of inquiry
Identifying Selecting Using
Conductliterature
review
Identifyexisting
measures
IdentifyGap:
NO existingmeasures
Examine evidence,
feasibility &relevance
Usemeasure
Revisemeasure
Developnew
measure
Collect &AnalyzeData (as
Developed)
Gather newevidence onvalidity andreliability
Steps in Translating MeasuresSteps in Translating Measures
Translateto secondlanguage
Back translate
to firstlanguage
Pilot-testwith
experts &end-users
Gather newevidence
on validityand reliability
Compareresults with
originalversion
Measures MatrixMeasures Matrix Evaluation question Outcome or process variable Indicator or measure Where data comes from How data is collected data & how frequently Who collects data When data is collected How data will be analyzed
Sample Measures MatrixSample Measures MatrixEvaluation question Do clients perceive outcomes as a result of the
care/ services they receive?
Process or Outcome variable
Client perception of care
Indicator/ Measure Youth Services Survey
Source of data Youth 16 to 18 years, and
Parents of children and youth below 16 years
How data is collected Self-administered questionnaire, in-person and by mail
Who collects data Receptionist
When data is collected At termination of the program, at least 3 months in program
Plan for analysis Compare means of total scores for those in Program A with those in Program B
Questions?Questions?
Key insights?Key insights?
SummarySummary
Measurement involves specifying or “operationalizing” the evaluation question.
Process or outcome variables can be assessed using numerical or qualitative data.
Sources of information include questionnaires, interviews, focused groups and administrative data.
Summary, Cont. Summary, Cont.
Summary tables assist in organizing information about various measures, and help in making informed decisions on the most appropriate measure for the evaluation.
A matrix of the indicators and measures to be used in the evaluation summarizes how, where and when data is to be collected.
Next StepsNext Steps
Review logic model and identify evaluation questions
Specify process and outcome variables Summarize literature review Review and select measures Create indicators and measures matrix Develop protocol for collecting data Develop plan for analyzing data
Future webinars?Future webinars?
February 2009: Using Excel and SPSS for basic statistics
March 2009: Writing the Final Report
For more informationFor more information
Evangeline Danseco, PhDHead, Evaluation and Research
[email protected], ext. 3319
www.onthepoint.ca