evaluation team evaluator: nancy amodei, ph.d. – dept pediatrics evaluation coordinator: danielle...
TRANSCRIPT
Evaluation Team•Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics•Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics•Data Manager: Kyle Kozlovsky, M.S. – Dept Pediatrics•Qualitative Expert: Suyen Schneegans, M.A. – Dept Pediatrics
®
Special thanks to: Rasheem BattleAlejandro BocanegraMeghan CrabtreeMerced DoriaDestiny RamosDrew Russell
Process Evaluation: How is the program being implemented?
S-START Process Goals
1. Train UTHSCSA medical residents and residents from other participating South Texas programs to use evidence based SBIRT procedures for patients who have or are at risk of substance abuse disorders.
2. Promote systems change in targeted residency programs by integrating the SBIRT model into the curriculum on a long-term basis.
Process Goal #1: Train UTHSCSA medical residents and residents from other participating South Texas programs
Key Activities and Measures
1) Develop/implement a comprehensive curriculum
2) Train UTHSCSA and other Faculty Demographics, type of training, satisfaction (GRPA & qualitative
findings)
3)Train UTHSCSA residents & residents from other programs Demographics, type of training, satisfaction (GPRA)
SBIRT Curricular Strategies by Specialty
Pediatrics
OB-GYN
Psychiatry
Internal
Medicine
IM - ERAH
C
FCM
FM –McAlle
n
FM -Fort Hood
FM -Santa Rosa
Surgery
Large Group Didactic Lecture
+ + + + + + + + +
Small Group Discussion
+ + + + + + +
Skill-Building Workshops
+ + + +
Reading Assignments
+ + + +
Screening Questions In Medical Records
+ + + + + + + + + + (likely but not in place
yet)
Reminder Pocket Cards
+ + + +
Screening Assignments*
+ + +
Inpatient Clinical Supervision
+ + + + + +
Outpatient Clinical Supervision
+ + + (child psych only)
+ + + + + +
Independent Study Module (Blackboard)
+ + + +
OSCE (Observed Standardized Clinical Exams)
+ +
What is the core SBIRT event?
Faculty Training
Who? How Many?
Method/Approach
Pediatrics, UTHSCSA 8 Train-the-trainer
Family & Community Medicine, UTHSCSA
9 Train-the-trainer
Family Medicine, CHRISTUS Santa Rosa
9 Train-the-trainer
Family Medicine, Fort Hood 5 Train-the-trainer
Internal Medicine, UTHSCSA 1 Chief Residency Immersion Training (CRIT)
Internal Medicine, ERAHC 1 Train-the-trainer; Chief Residency Immersion Training (CRIT)
Other UTHSCSA Departments and external Departments
18 Personal consultation, Sharing resources via website, Newsletter, Email
Demographics of Faculty Completing GPRAs
Demographics or Respondents at Baseline
N = 17 (%)
UTHSCSA Family and Community Medicine Pediatrics
8 (47%)
9 (53%)
Male
9(52.9%)
Hispanic/Latino 7(41.2%)
White
12 (70.6%)
Asian 3(17.6%)
African-American 0(0%)
* Wilcoxen Signed Ranks Test
Baseline vs. 30-day Faculty GPRA ratings (N=17)
GPRA Item Mean Baselin
e Rating
Mean 30-Day Rating
Z * P (2- tailed
)
Overall how satisfied with your training experience?
2.12 1.93 -.53 NS
Material presented useful to me in dealing with substance abuse
1.82 1.94 -.51 NS
Training enhanced my skills in topic area 2.00 1.94 .00 NS
Training relevant to my career 1.76 1.88 -.63 NS* Wilcoxon Signed Ranks Test
Qualitative Study of Faculty Perceptions of S-START
Purpose: Gain an in-depth understanding of the experience and perceptions of S-START faculty
Methods
16 training faculty from 5 specialties invited
15 accepted (12 from UTHSCSA; 1 FM program in McAllen, 1 FM CHRISTUS Santa Rosa, 1 FM from Fort Hood)
Mean age (43.93 years); 72% female; 73% MDs, 1 Ph.D., 1 PsyD, 1 M.A.
Average yrs of experience = 15
Qualitative Study of Faculty Perceptions of S-START
Data Collection
Data collection: ≈ 22 months after S-START began
45 to 60 minute interviews using scripted but open-ended questions
14 of the interviews taped to facilitate transcription
Topics: How S-START implemented in the program, barriers and challenges, impact of potential clinical service reimbursement in facilitating program; suggestions for improvement
Qualitative Study of Faculty Perceptions of S-START
Data Analysis
Evaluation team hand-coded transcribed interviews
Thematically coded them to correspond to each question
Collapsed materials thematically into 10 emergent or preset categories
Qualitative Study of Faculty Perceptions of S-START
Results3 Thematic categories accounted for > 50% of interview
responses Critical components Barriers Motivation
Critical Components Faculty training
Barriers Lack of Leadership
Motivation Buy-in from faculty and residents
Resident TrainingDepartment Trained to
date Expected trained by Year 05 Old estimates
Total Expected Trained by Year 05
Pediatrics - UTHSCSA 80 107 112
Family & Community Medicine
•UTHSCSA•McAllen (South Texas)•CHRISTUS Santa Rosa•Fort Hood (Military)
71292517
8424----
95414031
Internal Medicine•UTHSCSA•ERAHC (South Texas)
11316
163--
16326
OB-GYN -UTHSCSA 29 41 41
Psychiatry - UTHSCSA 71 120 107
Surgery- UTHSCSA 58 --- 96
Nurse Practitioners-UTHSCSA
39* --- 39
Total 548 539 679
Demographics of Residents Completing GPRAs
Demographics or Respondents at Baseline
N = 409 (%)
Family Medicine Pediatrics Internal Medicine OB/GYN Psychiatry
138 (33.7%)80 (19.6%)
121 (29.6%)27 (6.6%)43 (10.5%)
Male
155(37.9%)
Hispanic/Latino 133(33.8%)
White 259 (63.3%)
Asian 81(19.8%)
African-American 17(4.2%)
* Wilcoxon Signed Ranks Test
Baseline vs. 30-day Resident GPRA ratings (N=409)
Item Mean Baselin
e Rating
Mean 30-day Rating
Z * P (2- tailed)
Overall how satisfied with your training experience?
1.75 1.71 -4.11 .00
Material presented useful to me in dealing with substance abuse
1.68 1.64 -3.98 .00
Training enhanced my skills in topic area
1.75 1.67 -3.09 .01
Training relevant to my career 1.59 1.50 -2.92 .01
Process Goal #2: Promote Systems Change in Targeted Residency Programsby integrating SBIRT model into curriculum on long-term basis
Key ActivitiesCouncil of Residency SBIRT Trainers Meetings
Elicit support of key personnel
Changes to Electronic Medical Record
Pocket Cards
SBIRT resources (including key modules) on the S-START website
iPad Project
Progress towards Goal 2:Council of Residency SBIRT Trainers Meetings
Date Pediatrics
BAMC
FCM
Internal Medicine
Psychiatry
OB-Gyn
FM – Fort Hood
FM – Santa Rosa
Trauma
Nursing
1.29.09 2 1 1 - 2 1 - - 1 -
1.30.09 2 - 2 2 - - - - - -
9.29.09 4 2 1 2 2 - - - 3 -
12.08.09
5 1 2 1 1 1 - - - 1
3.09.10 6 1 - - 1 - 1 1 - 1
8.17.10 4 2 2 2 1 1 - - - -
2.15.11 5 - 2 1 - - - - 1 -
6.28.11 2 2 - - - 1 - - - -
7.19.11 5 - 2 1 1 - - - 3 -
Process Goal #2: Promote Systems Change in Targeted Residency Programsby integrating SPIRT model into curriculum on long-term basis
Progress re other ActivitiesSupport of change leaders- e.g. UTHSCSA President, Residency
Program Directors; PD and Co-PD have high profile positions
Changes to Electronic Medical Record- UTHSCSA – DFCM, Peds; (Psychiatry and Surgery planned)
Pocket cards McAllen FM: Part of every patient visit paperwork
SBIRT resources (including core modules, resource directory) on the S-START website
iPad Project-proposed for UTHSCSA Pediatrics
S-START Outcome Goals:
1. Enhance residents’ knowledge of evidence-based
SBIRT practices.2. Enhance residents’ readiness and perceived
confidence to implement SBIRT with their patients
3. Increase residents’ implementation of SBIRT practices with their patients
4. Enhance Faculty Participants’ knowledge and confidence in ability to teach SBIRT practices to future physicians
What is the program’s impact?
3 x 2 Repeated MeasuresThree data collection methodsMeasurement Occasions for Surveys:
Pre-Test30-Day Follow-UpAnnually up to 36-month follow-up
Measurement Occasions for Pocket CardsVaries by department
Measurement Occasions for Chart Reviews12-month period prior to first core SBIRT module
implementation12-month period following the first year of SBIRT module
implementation12-month period following the third year of SBIRT module
implementation
Outcome Design
Evaluation MeasuresDOMAIN
Measure Description Source
Knowledge
Core SBIRT knowledge every resident should know
Local
Residency-specific
Department-specific SBIRT knowledge
Attitudes Readiness to use SBIRT
Readiness to screen patients for use, assess readiness to change, perform intervention/referral, & documentation
Alcohol Education Survey (D’Onofrio et al., 2002)
Confidence to use SBIRT
Confidence to screen patients for use, assess readiness to change, perform intervention/referral, & documentation
Current practice
Self-reported current use of SBIRT
Self-reported current practice of screening patients for use, assessing readiness to change, performing intervention/referral, & documentation
Alcohol Education Survey (D’Onofrio et al., 2002)
Pocket cards Family Medicine programs documenting SBIRT behavior w/patients
ASSIST (WHO, 2002)
Chart review Review of charts in Ped. & FCM inpatient clinics to see change in use & documentation of SBIRT
Local
Tool Pre-test
Baseline
30-Day F/U
12- mos F/U
24- mos F/U
36 –mos F/U
GPRA X XAlcohol
Education Survey
X X X X X
Brief Substance Abuse Attitude
Survey
X X X X X
Core knowledge
X X X X X
Residency-specific
knowledge
X X X X X
Incentive $20 $10 $10 $20
Timeline of Self-Administered Instruments & Incentives
Methods of Survey Data Collection
Web-based surveys (i.e., SurveyMonkey)Emails to UTHSCSA and private email addressesUnique web links provided to residency
coordinators
Hard copy surveys Pass out at grand rounds and conference periodsIntra-office mail for fellows, facultyMail to home and clinic physical address
Strategies for survey follow-upCollected contact information using a comprehensive tracking
formText reminders to cell phone numbersPhone calls to (1) cell, (2) home, (3) significant others, (4) clinicContact residency coordinators for updated contact informationEnlist authoritative support of facultyLook up information using White Pages, AMA DoctorFinder,
respective state medical board websites (usually Texas)Peer-to-peer contact updates
Future strategies for survey follow-upReminder postcards sent twice before each
annual surveyBring surveys to end-of-year gatherings for
graduating residentsInclude surveys in residents’ exit processing
before graduation I pity the fool who doesn’t take
the survey.
Resident survey ratesPre-
test**Baseline
**30-day 12-
month24-
month
Total residents & students
153.4% 128.7% 89.9% 56.6% 67.0%
Response rates for similar populations (e.g., students, medical professionals) tend be 60% or lower on follow-up surveys (Asch et al., 1997; Kaplowitz et al., 2004; Kaspryzyk et al., 2001; McMahon et al., 2003; Porter & Whitcomb, 2007)
Analyses of resident survey dataDemographic data (pre-test)Measured changes from pre-test to 12-
month follow-up in:
Selected departments for analysis: Pediatrics, Family and Community Medicine, Internal Medicine
Resident DemographicsPre-test results
TotalN=46
5
UT Ped
n= 97
UT OBn = 29
UT Psy
n = 83
UT Surn = 49a
FMn = 137
IMn = 130
Gender (freq. male)
38.3% 20.6% 10.7% 38.6% - - - 44.9% 51.6%
Race (freq.)
White44.1% 55.2% 67.9% 61.3% - - - 32.6% 31.5%
Black 3.7% 2.1% 3.6% 7.5% - - - 1.5% 4.8%
Hispanic 31.1% 28.1% 21.4% 18.8% - - - 35.6% 38.7%
Asian 17.9% 12.4% 7.1% 10.0% - - - 27.4% 19.4%
Other 3.2% 2.1% - - - 2.5% - - - 3.0% 5.6%
Age (mean; sd)
30.3(4.8)
28.2(2.6)28.8(2.4
)31.4(5.9) - - -
32.5(5.7)
29.1(3.6)Note. UT=University of Texas Health Science Center at San Antonio; Ped.=Pediatrics; FM=Family
Medicine; IM=Internal Medicine; OB=Obstetrics/Gynecology; Psy.=Psychiatry (adult & child); Sur=Surgery.aSurgery residents began the SBIRT curriculum on August 15, 2011.
Resident Demographics (cont.)Pre-test results
UT FCM
n = 72
UT McAn = 23
SRn = 24
FHn = 18
UT IMn = 109
UT ERAHCn = 21
Gender (freq. male)
40.0% 47.8% 33.3% 90.9% 48.6% 66.7%
Race (freq.)
White22.5% 9.1% 54.2% 72.2% 35.0% 14.3%
Black 2.8% 0% 0% 0% 1.9% 19.0%
Hispanic 28.2% 77.3% 45.8% 0% 36.9% 47.6%
Asian 42.3% 13.6% 0% 22.2% 19.4% 19.0%
Other 4.2% 0% 0% 5.6% 6.8% 0%
Age (mean; sd)
33.5(6.2) 35.3(4.6) 28.2(2.4)30.5(4.4
)28.8(3.5) 30.8(3.9)
Note. UT=University of Texas Health Science Center at San Antonio; FCM=Family and Community Medicine; McA.=McAllen Family Medicine; SR=CHRISTUS Santa Rosa Family Medicine; FH=Fort Hood Family Medicine; IM=Internal Medicine; ERAHC=Edinburgh Regional Academic Health Center Internal Medicine.
Outcome goal 1Enhance residents’ knowledge of evidence-based SBIRT practices.
Core SBIRT knowledge12 items developed locally by the SBIRT project
directorsKnowledge that residents across all
departments should know after training
Residency-specific SBIRT knowledge7-17 items developed locally by the SBIRT faculty
in the respective programsItems designed for specific residency program
SBIRT knowledge and patient populations
Sample core knowledge item:“How many ‘standard drinks’ are considered at-risk alcohol
use by a healthy 40-year-old man?”
Sample Pediatrics knowledge item:“________ exposure is the leading known preventable cause of
mental retardation.”
Sample Family and Community Medicine knowledge item:“Hepatitis B, hepatitis C, HIV and AIDS are strongly
associated with abuse of…”
Sample Internal Medicine item:“Alcohol withdrawal treatment on the inpatient medical
service is best accomplished by…”
Outcome goal 1Enhance residents’ knowledge of evidence-based SBIRT practices.
Outcome goal 1 cont.Enhance residents’ knowledge of evidence-based SBIRT practices.
Core SBIRT knowledgeAll residents increased
SBIRT knowledge, F(1, 167) = 32.1, p < .001.
No differences found between residency programs
Outcome goal 1 cont.Enhance residents’ knowledge of evidence-based SBIRT practices.
Residency-specific SBIRT knowledge
Outcome goal 2Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients.
Readiness to use SBIRT (D’Onofrio et al., 2002)Subscale of AES comprised of 7 10-point Likert scale itemsRange: 0-100Sample item: “How ready are you to change your practice
behavior to ask patients about quantity and frequency of their alcohol use?”
Confidence to use SBIRT (D’Onofrio et al., 2002)Subscale of AES comprised of 7 5-point Likert scale itemsRange: 0-100 Sample item: “I am confident in my ability to
discuss/advise patients to change their drinking behavior.”
Outcome goal 2 cont.Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients.
Readiness to use SBIRT: No significant change in
readiness from pre-training to 12 months post-training, F(1, 161)
= .87, p = .353.
FCM reported higher readiness than IM overall, F(2, 161) = 4.7, p
= ..010.Pediatrics was not
significantly different than the other two programs
Outcome goal 2 cont.Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients.
Confidence to use SBIRT: Residents overall
reported higher confidence at 12-month, F(1, 161) = 27.3, p < .001.
FCM reported higher confidence overall than IM and Pediatrics, F(2,
161) = 8.1, p < .001.Pediatrics was not
significantly different than the other programs
Outcome goal 3Increase residents’ implementation of SBIRT practices with their patients.
Self-report of current SBIRT practice (D’Onofrio et al., 2002)Subscale of AES comprised of 7 5-point Likert
scale itemsRange: 0-100Sample item: “How often do you formally screen
patients for alcohol problems using brief screening tools (e.g., T-ACE, AUDIT, CAGE)?”
Pocket cardsChart reviews
Current practice of SBIRT skills:
Residents overall reported higher current SBIRT practice at 12-month, F(1, 161) = 35.2, p
< .001. Significant interaction,
F(2, 161) = 19.7, p < .001.Both Pediatrics and
FCM improved self-reported current practice .
Internal Medicine declined in self-reported current practice.
Outcome goal 3 cont.Increase residents’ implementation of SBIRT practices with their patients.
Summary of Resident Survey Data Findings
SBIRT core knowledge improved from pre-test to 12-month follow-up
Readiness to implement SBIRT did not change, but was high at pre-test
Confidence to use SBIRT improved from pre-test to 12-month follow-up
For self-report of SBIRT practice, residents overall improved from pre-test to follow-upHowever, when departments were analyzed
separately, Internal Medicine decreased from pre-test to 12-month
Settings:Family Medicine inpatient service at University
Hospital in San Antonio, Texas
Subjects:285 adult patients, from July 2009 to May 2011.Average Age: 47Gender Distribution: 71.3% Male
Outcome goal 3 cont.UTHSCSA Family Medicine Pocket Cards
Patients were interviewed with a 4-step pocket card
Step 1: Pre-screening questions for substance useStep 2: WHO ASSIST (Alcohol, Smoking, and
Substance Involvement Screening Test)Step 3: ASSIST score to assess the level of risk and
determine need for interventionStep 4: checklist describing the intervention,
patient response, and future plan.
Residents were asked to complete 12 per year26 out of 26 trained residents participatedResidents completed 11 total on average
UTHSCSA Family Medicine Procedures
Step 1: Pre-screening Results
95.8% of patients screened positive for at least 1 substanceAvg. ASSIST Score was 19 indicating a moderate risk of substance abuse
Step 2-3: ASSIST Results
When the ASSIST Score recommended a brief intervention, residents reported some form of brief intervention 69.4%(over two thirds ) of the time
Residents most likely to discuss consequences of use if ASSIST Score recommended brief intervention (79% of the time)
8% of patients declined to discuss their response to screening
Brief Interventions
Brief Intervention Actions Taken
When the ASSIST Score suggests a referral to treatment, residents referred a patient to treatment 71.8% of the time
Residents were most likely to contact an LCDC (Licensed chemical dependency counselor) when ASSIST Score recommended a referral to treatment (46.5% of the time)
Referrals to Treatment
Referrals to Tx
Referrals to Treatment Actions Taken
Patients were more likely to report some sort of action (cut back, quit or seek outside help) when the resident documented a brief intervention (80% of the time compared to 71.4%)
Patients were more likely to report some sort of action when the resident documented a referral to treatment (80% of the time compared to 68.8%)
Step 4: Patient Plans
The ASSIST Pocket Card was distributed to other departments with some participation
Participation in Other Departments
Residency # of Cards Completed
# of Residents
Participating
# of Residents Trained
UTHSCSA McAllen Family Medicine
103 13 29
Fort Hood Family Medicine
25 6 11
Outcome goal 3 cont.Overall chart review study design
Location 12-month period prior to first core SBIRT module implementation
12-month period following the first year of SBIRT module implementation
12-month period following the third year of SBIRT module implementation
Pediatrics May 4, 2008-May 4, 2009
May 4, 2010-May 4, 2011
May 4, 2012-May 4, 2013
Family & Community Medicine
April 22, 2008-April 22, 2009
April 22, 2010-April 22, 2011
April 22, 2012-April 22, 2013
Goal 3 cont.Overall Chart Review Study Design
1st Chart Review*
# of cases for
sample1. Pediatrics outpatient continuity clinic 4001. Family & Community Medicine outpatient
continuity clinic 400SUBTOTAL 800
2nd Chart Review^
# of cases for
sample1. Pediatrics outpatient continuity clinic 4001. Family & Community Medicine outpatient
continuity clinic 400SUBTOTAL 800
3rd Chart Review
# of cases for
sample1. Pediatrics outpatient continuity clinic 4001. Family & Community Medicine outpatient
continuity clinic 400SUBTOTAL 800
TOTAL 2400
*Completed^In Progress
Setting: Pediatrics Outpatient Continuity Clinic
Chart selection criteria: (1) seen at least once within the review period by a
resident(2) at least one visit within the appropriate review
period (e.g., one year prior to implementation of core SBIRT intervention and medical record changes, one year after, and three years after)
(3) age of patient 11 and upVisits included in the analysis are acute visits and
annual well child visits
Pediatrics Chart ReviewBaseline findings
Patient Demographics
Mean age was 13 (1.98)
49.5% Female
Ethnicity known to be majority Hispanic, but documentation in charts is rare
ScreeningOut of 967 visits, 511 (just over half) screenings
were documentedVisits included Annual Check ups and Acute Visits
Some acute visits were sports physicals
Screening Cont.
9 positive screenings documented for tobacco0 positive screenings documented for alcohol
or other drugsThe HEADSS was documented as a
screening tool in 149 visitsOnly one CRAFFT
screening was documented
Brief Interventions and Referrals to Treatment
Of 967 visits, 204 brief interventions were documented4 of 9 positive tobacco screenings were
followed by a documented brief intervention“Anticipatory guidance” for drug use is often
used with pediatric patients and was considered a BI in our design
Only 2 referrals to treatment were documented
Conclusions from Pediatric Chart Review
Documentation is a likely contributor to the lack of SBIRT practices foundOngoing changes to the medical record and
routine SBIRT training will likely increase SBIRT practices
An increase in screening might lead to an increase in positive screenings (and, in turn, increase opportunities for brief interventions and referrals to treatment)
Outcome goal 4Enhance faculty participants’ SBIRT knowledge and confidence in ability to teach SBIRT practices to future physicians.
SBIRT knowledge Developed locally by the S-START project directorsSame core knowledge that the residents receive
Confidence to teach SBIRT to residents Adaptation to the resident scale (D’Onofrio et al.,
2002)Range: 0-100Sample item: “I am confident in my ability to train
residents in advising patients to change drinking behavior. “
Outcome goal 4 cont.Enhance faculty participants’ SBIRT knowledge and confidence in ability to teach SBIRT practices to future physicians.