development of survey from an item bank for counselors motivational interviewing performance...
TRANSCRIPT
DEVELOPMENT OF SURVEY FROM AN ITEM BANKFor Counselors Motivational Interviewing Performance Management and Vocational Rehabilitation Program Evaluation
By Ifeoma Chika Iyioke, PhD.
PERFORMANCE MANAGEMENT(PM)
Process by which an agency plans and monitors organizational practices and progress towards goals
PM involves assessment and measurement
Assessment and measurement are activities for collecting, analyzing and/or reporting information about the performance of an individual, group, organization, system or component.
PM IN VOCATIONAL REHABILITATION: CONDITION Emphases on Program Outcome Measures Standards and Indicators Outcome-Based Indicato
r No.Description RSA Criteria
PI 1.1: Change in number of employment outcomes
≥ 0
PI 1.2: Percent employed > 55.8
PI 1.3: Competitively employed > 72.6
PI 1.4: Significant disability > 62.4
PI 1.5: Earnings ratio > 0.52
PI 1.6: Self-support > 0.52
PI 2.1: Minority ratio > 0.80
PM IN VOCATIONAL REHABILITATION: GAPS
Many confounding factors with outcome measures
Difficult to compare performance of state –agencies and counselors
Lack of focus on the program process and the primary function of the vocational rehabilitation
PURPOSE OF PRESENTATION Introduce a strategy for vocational
rehabilitation counselor PM by monitoring program process
An Item Bank for development of surveys for management of counselor performance of counseling
Connection with vocational rehabilitation program evaluation
FOOD FOR THOUGHT Why measure?
Who or object to measure?
What to measure?
How to measure?
COUNSELOR FUNCTIONS
Consultant
Career/Vocational Counseling
Service Coordinator
COUNSELING FRAMEWORKS Directive Counseling
Trait-Factor (Parson, 1909, 1989) Person-Environment Fit (Dawis & Lofquist, 1984;
Lofquist & Dawis, 1991) Motivational Interviewing (Miller & Rollnick,
1991)
Non-Directive Counseling - Client-Centered (Carl Rogers, 1951)
WHAT IS AN ITEM BANK? A repository for survey/test questions
Survey/test questions organized and catalogued in a similar way to books in a library by topics
Contains also measurement characteristics of the survey/test questions (e.g., their reliability and validity)
EXAMPLE OF AN ITEM BANKItem No
Construct Source Stem Response Scale
Item Statistics
1 Value clarification
Intellectual property
The counselor asked about what I really want from life
Yes, No, I don't know.
2 Develop discrepancy
Intellectual property
The counselor highlighted inconsistency between my actions and the things I consider most important in life
Every meeting, Half or more than half of the meetings, Fewer than half of the meetings, Never, I don't know
See
WHY AN ITEM BANK
An assessment tool for standardized performance management
Provides program administrators access to a variety of survey/test questions to cover a whole variety of situations
Can be used to develop effective and efficient surveys/tests for meeting the needs of criterion referenced evaluation
Facilitates assessing the performance of individual employees, monitoring changes, and for evaluating program innovations
DEVELOPING ITEM BANK FOR MOTIVATIONAL INTERVIEWING PM Started with Literature Review - of
Motivational Interviewing
What, Why, and How
Item Bank Design
Populating Item Bank - Creation of Items
MI: DEFINITION
An Assessment Method Interview on motivation via open-ended
questions)
An Intervention (stage-based and tailored) A counseling method (directive and person-
centered ) A method of communication (collaborative
conversation and partner-like relationship)
MI: WHY
Motivation
Readiness or priority
Willing or Importance
Ambivalence - Approach-avoidance conflict
Able or Confidence
MI: WHYImportance/Confidence
Low High
Low Group A Group B
High Group C Group D•Group A: Low Importance, low confidence - These people neither see change as sufficiently important nor believe that they could succeed in making such a change if they tried•Group B: Low importance, high confidence – these people are confident that they could make the change if they thought it were important to do so but are not persuaded that they want to change•Group C: high importance, low confidence – here the problem is not in willingness to change, for these people express desire to do so. The problem is low confidence that they could succeed if they tried•Group D: High importance, high confidence – these people see it is important to change and also believe that they could succeed
MI: PRINCIPLES
Explore motivation
Express empathy
• Develop discrepancy
Roll with resistance
Support self-efficacy
MI: METHODS
Enhance Importance
OARS Open-ended questions Affirmations Reflections Summaries Eliciting change talk
MI: METHODSEnhance Confidence
Review and affirm his or her own past successes in changing behavior
Give a choice of strategies to try, it is more likely that a person may find one that is acceptable as a starting point
DESIGN OF AN ITEM BANK
Define the purpose – formative or summative performance management and evaluation
Define content domain – Question classification and organization Scheme of an item bank
Motivational Interviewing: Why, Components, Principles and Methods
MI: ITEM BANK DESIGN Why MI Component MI Principles MI Methods
Importance
[Ambivalence ]
Assessment Explore Open-ended questions
Confidence Assessment Explore Open-ended questionsImportance
[Ambivalence
]
Assessment Express empathy Reflective listening
Confidence Assessment Express empathy Reflective listeningImportance
[Ambivalence]
Intervention Develop discrepancy
Eliciting change talk
Importance
[Ambivalence ]
Intervention Develop discrepancy
Summarizing
Importance
[Ambivalence ]
Intervention Develop discrepancy
Affirmation
Confidence Intervention Support self-efficacy
Elicit confidence talk
Confidence Intervention Support Self-efficacy
Affirmation
HOW TO POPULATE THE ITEM BANK
Compile and share previous questions
Create new survey questions
HOW TO USE THE ITEM BANK FOR COUNSELOR PM
Build the Plan for performance management into the customer satisfaction survey
The counselor should keep a record of the customers they serve and the level of motivation presented at the time they began to be counseled and at the end of the planning process.
HOW TO USE THE ITEM BANK FOR COUNSELOR PM
Question sampling and survey delivery procedures Administration electronic vs. paper
The surveys are to be administered to the customers served by the counselor at intervals to monitor the counselor delivery of counseling services.
HOW TO USE THE ITEM BANK FOR COUNSELOR PM
Grading – Criterion-Referenced vs. Norm-referenced
Temporal Feedback on areas of strengths and weaknesses
Follow-up with training and remediation of areas of weaknesses
MI AND VOCATIONAL REHABILITATION PROGRAM EVALUATION: MODEL
Engagement & performance of motivational interviewing consumer motivation consumer engagement consumer outcomes (e.g. employment, satisfaction)
MI PROGRAM EVALUATION QUESTIONS
To what extent did the counselor deliver the intended counseling style of motivational interviewing to the consumers?
Does relationship exist between the level of consumer perception of counselor engagement and performance of MI with their motivation, engagement, and outcomes (e.g. employment, satisfaction)?
EXERCISE
The counselor asked about the things I consider most important in life - ?
The counselor highlighted inconsistency between my actions and the things I consider most important in life?
The counselor listened to what I had to say -?
The counselor asked about the justifications for my actions-?
The counselor praised my efforts towards accomplishing my goal(s) -?
QUESTIONS ?
Thank you for listening