creating an evaluation process by dr. kathryn rugen

Creating an Evaluation Process Kathryn Rugen, PhD, FNP-BC, FAAN, FAANP

VETERANS HEALTH ADMINISTRATION

• Describe the structure of the VA Centers of Excellence in Primary Care Education NP Residency Programs

• Explain the development of the NP Residency competency tool

• Appraise the outcomes of the NP Residency• Identify next steps

1

Objectives


VA Centers of Excellence in Primary Care Education: NP ResidencyEligibility and Recruitment• New graduates of adult-gerontology primary care or family nurse

practitioner program from an accredited master’s or doctor of nursing practice (DNP) program within the past year

• Obtain board certification and state APRN licensure in 90 days of starting • All the CoEPCEs have been successful in recruiting NP students who have had

clinical practicums in CoEPCE to continue in their NP residency program. Program Attributes• 12 months long supported with a trainee stipend which includes benefits for

healthcare, vacation and sick leave• Majority of time in primary care with own panel, specialty and inpatient

rotations• Focus on interprofessional collaborative practice, leadership, scholarship, QI• Co-precepting in last 6 mo

2


• Demonstrate program effectiveness

• Standardization across 5 sites

• Document competence in 7 domains

• Prepare for site accreditation

NP Competency Tool


– AACN/CCNE Masters and DNP Essentials– AACN/NONPF Adult-Gerontology Nurse Practitioner

Core Competencies – NCQA PCMH Standards– Core Competencies for Interprofessional

Collaborative Practice (IPEC)– ACGME competencies– VA top outpatient diagnoses– COE education core domains – Entrustable Professional Activities

Development


• Iterative process – VA NP experts at each site and MD education

consultant

– Post-graduate NP trainee reviewed and offered suggestions

– Solicitied input from experienced and new NPs throughout VA Primary Care

Content validity


• Clinical competency in planning and managing care• Leadership• Interprofessional team collaboration• Patient-centered care• Shared decision making• Sustain relationships• Quality improvement and population management

Domains


• Assess, diagnose, treat and manage health conditions commonly seen in primary care

• Conditions prevalent in the Veteran population - Post-Traumatic Stress Disorder (PTSD), Traumatic Brain Injury (TBI), military sexual trauma, suicidality, and hepatitis C virus

• Perform a comprehensive history and physical exam, construct pertinent differential diagnoses, order appropriate screening diagnostic tests and appropriate medications, perform comprehensive medication review and reconciliation, present clear, concise and organized patient case

Clinical Competency


• Lead team huddles, case conferences, team meetings, quality improvement projects, shared medical/group appointments

• Apply leadership strategies that support collaborative practice and team effectiveness

Leadership Competency


• Develop own professional identity and ability to explain one’s role to patients, families and other professions

• Use respectful language, understand and appreciate contribution of other team members

• Function as a resource to other professions, maintain open communication with team members

• Safely transition patients among team members • Seek feedback from team members, constructively

manage disagreements with team • Engage in continuous professional and interprofessional

development to enhance team performance

Interprofessional Collaboration Competency


• Communicate with patients between office visits by telephone, secure messaging and telehealth monitoring

• Elicit patient’s values, preferences, and cultural beliefs • Identify, accommodate, and customize care for patients

with language, cognitive, functional or cultural barriers • Assess and provide education to empower patients to

self-manage their chronic conditions• Track and coordinate care by ensuring follow-up on

messages, tests, consults, and care at outside facilities • Engage other health professionals in shared patient-

centered problem-solving • Use motivational interviewing to help change health

related behaviors

Patient Centered Care Competency


• Use active listening skills and open-ended question during a patient visit

• Counsel and support patients in their self-management of chronic diseases

• Facilitate patient’s participation in healthcare decisions using decision aids

• Engage patient in advanced care planning • Activate community resources for patients or population

needs • Engage patients as care team members in tracking and

coordinating care • Share accountability with other professions, patients and

communities for outcomes relevant to prevention and health care

Shared Decision-Making Competency


• Devise, follow, review and adjust a longitudinal care plan to meet the patient’s needs

• Develop and sustain a respectful and trusting relationship with the clinic staff, the faculty, their peer learners and their patients/families

• Give timely, sensitive instructive feedback to others about their performance on the team

• Respond respectfully to feedback from others

Sustained Relationships Competency


• Access and interpret clinic performance data • Improve care through Plan-Do-Study-Act cycles • Perform root cause analyses and reflect upon critical

incidents (medical error, near miss, preventable emergency room visits or readmissions)

• Query registries to determine the health status and care needs of the entire practice and/or specific populations of interest i.e. all diabetic patients

• Reflect on individual and team performance and introduce strategies for improvement

QI/Population Health Competency


• NP resident and mentor complete competency tool at 1, 6, and 12 months (total 69 items)

• Rate on 0-5 scale– 0= not observed or not performed– 1= observes task only– 2= needs direct supervision– 3= needs supervision periodically– 4= able to perform without supervision– 5= able to supervise others- aspirational!NP resident responds to open ended questions

Methods


• Evaluation questions: – identify items and domains NP residents are strongest and

weakest – determine how NP residents progress over time – determine agreement between trainee and mentor ratings

• Descriptive statistics to evaluate the distributional characteristics of each item and domain, the impact of the time on trainee and mentor

• T-test and general linear models to assess relationship between NP resident and mentor ratings over time

Analysis


Number of NP Post-Graduate Trainees

2011-2012

2012-2013

2013-2014

2014-2015

2015-2016

2016-2017

Boise - 1 2 4 2 2Cleveland - - 2 4 4 4SanFrancisco - 3 3 5 3 5West Haven 3 4 4 5 6

(1post DNP)6

Seattle - - 1(1 post-DNP)

1 2(1post DNP)

2

Houston - - - - - 2Los Angeles - - - - - 2

Total 3 8 12 19 17 23


CharacteristicAge

n reportingMean (years)SDRange

1034.19.4

27-59Gender (n, %)n reportingFemale Male

3832 (84.2%)6 (15.8%)

Prior CoE student (n, %) n reportingYesNo

3819 (50.0%)19 (50.0%)

Retained in VA After Training (n, %)n reportingYesNo

5924 (40.6%)

35 (59.4%) (2 Yale geriatric fellowship, 1 VAQS)Retained in VA Primary Care (n, %)n reporting 8/24 (33.3%)

Cardiology pain clinic VAQS Women's Health

Demographics


CharacteristicAverage Years of RN experience prior to NP ResidencyNaverageRangeMedianMode

385.4 years

0-31 years51

Type of NP programNgraduate entryBSN to MSN (traditional)

3817 (44.7%)21(55.3%)

Demographics


Subscale

Trainee Ratings Faculty Ratings

1 month6

months12

monthsp-value 1 month

6 months

12 months

p-value

Clinical Competency in Planning/Managing Care

nMeanSDRange

372.75.56

1.71-3.85

343.41.46

2.28-4.25

353.751.430 -

5.00

<.000137

2.94.60

1.86-4.59

343.68.49

2.89-5.00

364.42.50

3.50-5.00

<.0001

Clinical Competency


Clinical Competency

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Mea

n

Mentor_1m

Mentor_6m

Mentor_12m

Trainee_1m

Trainee_6m

Trainee_12m


Subscale


1 month6

months12


6 months

12 months

p-value

LeadershipnMean SD Range

371.451.35

0-4.85

342.411.58

0-5.00

353.131.56

0-5.00

<.0128

2.641.231.00-4.33

293.63.67

2.00-5.00

364.44.55

3.20-5.00

<.0001




0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

2.1 Lead PACTteam huddle

2.2 Lead caseconference

2.3 Lead teammeeting using

conflictmgmt/resolution

2.4 Lead groupeduc activities for

pts/fam, PACTteam, peers

2.5 Lead PACTteam qualityimprovement

project

2.6 Leadshared/groupmedical appts

2.7 Applyleadership

strategies tosupport

collaborativepractice/teameffectiveness

Mea

n

Mentor_1m

Mentor_6m

Mentor_12m

Trainee_1m

Trainee_6m

Trainee_12m


Subscale


1 month6

months12


6 months

12 months

p-value

Interprofessional Team Collaboration

nMean SD Range

373.48.65

2.33-4.77

344.06.57

2.66-5.00

354.101.53

0-5.00

<.0236

3.65.70

2.00-5.00

334.13.42

3.44-5.00

364.59.42

3.89-5.00

<.0002




0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Mea

n

Mentor_1m

Mentor_6m

Mentor_12m

Trainee_1m

Trainee_6m

Trainee_12m


Subscale


1 month 6 months12

monthsp-value 1 month 6 months

12 months

p-value

Patient Centered-CarenMeanSD Range

373.21.68

1.71-4.71

343.94.52

2.85-5.00

354.041.51

0-5.00

<.00236

3.43.69

2.29-5.00

334.08.51

2.86-5.00

364.61.46

3.71-5.00

<.0002




0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

4.1 Communicatewith pt between

office visits byphone, secured

email,MyhealtheVet

4.2 Elicit pt values,preferences, and

cultural beliefregarding care

4.3 Identify,accommodate, andcustomize care forpts with language,

cognitive,functional or

cultural barriers

4.4 Assess/provideeducation to

empower the ptsto self-manage

chronic conditions

4.5Track/coordinate

care for ptsensuring follow-upon messages, tests,

consults, care atother facilities

4.6 Engage healthprofessionals,

appropriate to thespecific care

situation, in sharedpt centered

problem solving

4.7 Usemotivationalinterviewing

Mea

n

Mentor_1m

Mentor_6m

Mentor_12m

Trainee_1m

Trainee_6m

Trainee_12m


Subscale


1 month 6 months 12 months p-value 1 month 6 months 12 months p-value

Shared Decision Making

nMean SD Range

373.17.71

1.28-4.57

343.86.55

2.28-5.00

353.961.52

0-5.00

<.00337

3.42.72

2.14-5.00

343.99.53

2.71-5.00

364.63.46

3.71-5.00

<.0001

Shared Decision Making Competency


Shared Decision Making Competency

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Mea

n

Mentor_1m

Mentor_6m

Mentor_12m

Trainee_1m

Trainee_6m

Trainee_12m


SubscaleTrainee Ratings Faculty Ratings

1 month 6 months12

monthsp-value 1 month 6 months

12 months

p-value

Sustained Relationships

nMean SD Range

373.46.69

2.00-4.83

344.01.43

3.16-5.00

354.041.53

0-5.00

<.0437

3.62.70

2.00-5.00

344.15.44

3.40-5.00

364.70.44

3.67-5.00

<.0001




0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

6.1 Devise, follow,review, adjust

longitudinal care planto meet assisgned pt

panel needs

6.2 Develop/sustainrespectful and trustingrelationship with clinic

faculty, preceptor,mentor

6.3 Develop/sustainrespectful and trustingrelationship with peer

trainees

6.4 Develop/sustain arespectful & trusting

relationship with PACTteam (includes clinic

staff)

6.5 Develop/sustain arespectful & trusting

relationship withpts/families

6.6 Give timely,sensitive, instructivefeedback to others

about theirperformance on team

Mea

n Mentor_1m

Mentor_6m

Mentor_12m

Trainee_1m

Trainee_6m

Trainee_12m


SubscaleTrainee Ratings Faculty Ratings

1 month 6 months 12 mos p-value 1 month 6 months 12 mos p-value

Quality Improvement/Pop Management

nMean SD Range

312.071.51

0-5.00

342.821.59

0-5.00

353.261.54

0-5.00

<.000127

2.531.201.00-5.00

313.43.72

1.75-5.00

344.2669

2.40-5.00

<.0001

QI/ Population Management Competency


QI/ Population Management Competency

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

7.1 Access/interpretperformance data

7.2 Improve care via Plan-Do-Study-Act cycles

7.3 Perform root causeanalyses and reflect upon

critical incidents

7.4 Query registries todetermine the healthstatus/needs of entirepractice/population of

interest

7.5 Reflect onindividual/team

performance & introducestrategies for improvement

Mea

n Mentor_1m

Mentor_6m

Mentor_12m

Trainee_1m

Trainee_6m

Trainee_12m


• At 1 month, 24 out of 28 items were rated between 2 and 3 (2= needs direct supervision; 3=needs supervision periodically) only four items were rated greater than 3 by the NP Residents.

• Four items rated higher than 3 were “perform comprehensive history and physical exam” (3.48), “perform medication reconciliation” (3.54) and “management of hypertension” (3.13) and “management of obesity” (3.35).

• At the 12 month time point all items were rates higher than 3 and seven items out of 28 were rated higher than 4 (able to perform without supervision) by the NP Residents

• The seven items rated 4 or higher were “perform comprehensive history and physical exam” (4.17), “order appropriate consults” (4.11), “perform medication reconciliation” (4.14) “management of hypertension” (4.08), “management of obesity” (4.11) “management of gastroesophageal reflux” ( 4.02), and “management of osteoarthritis” (4.00).

• At the 12 month time point the mentors ratings were all above 4 (4=able to perform without supervision) except for two items, “management military sexual trauma” (3.58) and “ management of traumatic brain injury” (3.66).

Item Analysis -Clinical Competence


Item Analysis: Leadership Competency

• NP Residents rated themselves between 1 and 2 (1= observes only; 2= requires direct supervision) on each item within the leadership competency domain at 1 month.

• At 12 months, NP Residents ratings were between 2.5 and 3.5 (2= requires direct supervision; 3= needs partial supervision); two items that remained low at the 12 month time point were “lead PACT huddles” (2.68) and “lead shared medical appointments” (2.80).

• Mentors ratings at 12 months on the leadership domain were 4 (4=able to perform without supervision) or above except on one item, “lead team meeting using conflict management/resolution” (3.41).

34


Item Analysis: QI/Population Mgmt

• Both NP Residents and mentor ratings were low at 1 month; between 1 and 2 (1= observes only; 2= requires direct supervision) on each item except NP Residents higher rating on “reflect on individual/team performance and introduce new strategies for improvement” (2.43).

• By 12 months, both NP Residents and mentor ratings were 3 (3= needs partial supervision) and above, with the exception of “perform root case analyses and reflect upon critical incidents” (NP residents= 2.94; mentors=2.80)

35


• At 1 month highest ratings for both NP residents and mentors were interprofessional collaboration, sustained relationships, patient centered care, and shared decision making domains

• These domains remained the highest rated at the end of 12 months by both NP residents and mentors

• Domains that were rated lower at 1 month by both NP residents and mentors were clinical competency, leadership, and quality improvement.

• All domains improved significantly over the 12 month period (p< .002 - .0001)

Findings


• At 12 months, mentor ratings were at a 4 or higher on each domain

• Interprofessional collaboration remained the highest rated domain at 12 months for the NP residents and sustained relationship was the highest rated by the mentors

• Quality improvement/population management was the lowest rated domain at 12 months

• Ratings were comparable at 1 and 6 months, but statistically different at 12 months in several domains

• NP residents generally rated themselves lower than their mentors

• COE general program evaluation findings show that NP residents are highly satisfied with the program

Results


Psychometrical Analysis

• Internal consistency calculated by NP resident and mentor for each domain and each time point

• Cronbach’s alpha ranging from 0.82-0.96

38


• List 2 things you do well• List 2 things you would like to improve• Set 2 short term goals that you can achieve in the next 3

months• Set 1 long-term goal that can be achieved by the end of

the residency• Describe how you will know if you achieved these goals• Describe any potential opportunities/obstacles you

might encounter as you try to reach these goals• Describe your strategies for achieving these goals• Other comments

Qualitative Questions


• Mapped to 7 competency domains plus personal attribute category• Most comments in clinical and personal attribute categories

– Clinical: comprehensive care, clinical guidelines, medication management, behavior modification skills, pathophysiology, care planning, diabetes management

– Personal attributes: time management, manage full load of patients, confidence, comfortable, work-life balance, organized, seeing more patients in less time, competence, knowledgeable primary care provider

• Overall agreement (between 3 raters) was .54 (Kappa statistic, this value indicates Moderate agreement) after first pass rating

• 100% after second pass rating

Qualitative Analysis


Final Program Evaluation• What did you like best about the NP residency program?• What did you like least about the NP residency program?• Now that your NP residency is completed, how confident and proficient do you feel in

your role as an advanced practice provider? Why?• Tell us the top 3 things we can do to improve the NP residency.• Were the program expectations for your performance, too high, too low, or just right?

How?• What specific experience or skill during your residency helped to advance your

transition towards a confident, proficient advanced practice provider?• What additional specific experience or skill that was NOT provided in the NP residency

would have helped to advance your transition towards a confident, proficient advanced practice provider?

• Thinking back, tell us why you selected to participate in a NP residency.• If you had to do it over again, would you participate in a VA Primary Care NP

residency?• Tell us if you would like to seek employment in VA, Why? Why not?

41


Final Program Evaluation• Please rate the following: 5= excellent, 4= good, 3= fair. 2= poor, 1 = very poor• Your preceptors• The staff• Your learning experience with trainees from other professions• Your ability to lead interprofessional teams• Your ability to work in an interprofessional collaborative practice setting• Your ability to implement quality improvement strategies• Your ability to manage chronic diseases• Your ability to develop differential diagnoses• Your ability to use shared decision-making with the patient and family• Your ability to develop sustained relationships with the patient and family

42


• Expand the use of the tool to the 2 new VA Centers of Excellence (Houston and West Los Angeles) and other VA and private sector sites

• Publish quantitative and qualitative outcomes• Conduct further psychometric analysis

Next Steps


[email protected]

Questions