2/15/2017

1

Continuous Quality Improvement as a Strategy to Improve NCLEX Scores

Cheryl L Mee MSN, MBA, RN

Susan Sportsman PHD, RN, ANEF,

FAAN

| 2

• Analyze aggregate results of faculty made tests, standardized examinations, and past NCLEX RN aggregate scores to identify weaknesses in curriculum and individual student performance.

• Use data to revise curriculum to improve identified weaknesses.

• Develop student remediation tactics that foster positive outcomes.

• Implement teaching learning actions that prepare students to successfully pass NCLEX exam.

Objectives

| 3

First time, USA Education NCLEX Pass Rate, Percent

Passing

https://www.ncsbn.org

Category 2016 2015 2014 2013

Total RN 84.57 84.53 81.78 83.04

Diploma 85.39 85.77 83.28 83.42

BSN 87.88 87.49 84.93 85.18

ADN 81.68 82 79.26 81.43

PN 83.73 81.89 82.16 84.63

24,236 novice nurses

graduated from a pre-

licensure RN program but

were unable to practice after

taking the NCLEX for the first

time!

1863 PNs did not pass on the

first attempt

In 2016….

2/15/2017

2

| 5

Plan

Do Study

Act

W Edwards Deming

Aguayo, Rafael (1990), Dr. Deming, The American Who Taught the Japanese About

Quality, Simon & Schuster, p. 76

| 6

Plan

Do Study

Act

A path to performance improvement Plan

Do Study

Act

Plan

Do Study

Act

| 7

Methods to Predict Students who will be

successful

• Grades in Previous Academic Programs

• Admission Scores

• Student Desire

• Essays

• Past Experiences

• Recommendations

| 8

Traditional Assessment Methods

• High School GPA

• GPA in other college courses

• Placement Tests

• Critical Thinking Tests-Writing Tests

College Success Predictors

2/15/2017

3

| 9

• Delayed enrollments

• Attends part time

• Works full time (35 hrs or more /week)

• Financially independent (financial aid)

• Has dependents other than a spouse

• Single parent

• GED

Non-Traditional Students are on a continuum

Non-Traditional Students

| 10

It is easier to predict

who will

pass than it is to

predict those

that fail

“Just admit the smart

students!”

An anonymous University President

| 11

Are you using

Admission

Assessment

Testing?

| 12

• Analysis - 5 years – first to compare different admission assessment exams

• Determine which of three preadmission examinations best predicted

success in an associate degree in nursing program. (NLN, HESI, ATI)

• HESI Admission Assessment examination scores can explain success in the

nursing program better than the other tests included in this study.

Testing for success: A logistic regression

analysis to determine which pre-admission

exam best predicts success in an associate

degree in nursing program

Manieri et al. (2015). Teaching and Learning in Nursing. 10, 25-29

2/15/2017

4

| 13

• There is no one criteria or tests score that will guarantee

choosing the successful student.

• Success in various disciplines vary depending upon the

competencies needed in the discipline.

• Concern about personal biases

The Problem with Admission Criteria

| 14

• Clarifying the “at risk” profile, using admission criteria

(A2 and other measures)

• Identifying students who meet the “at risk” criteria

• Identifying support strategies for “at risk” throughout

the curriculum that go beyond the support given to all

students

• Determine a process for identifying students who

demonstrate at risk behaviors throughout the semester

• Are we “singling out” students for failure?

“At Risk” Students

| 15

• Nursing Process

• Caring

• Communication and

Documentation

• Teaching/Learning

• Cultural/Spiritual

2016 NCLEX-Test Plan: Integrated Processes

| 16

https://www.ncsbn.org/RN_Test_Plan_2016_Final.pdf

Client Needs Categories and Distribution of Content

unchanged

16

2010 2013 2016

Safe and Effective Care:

- Management of Care

- Safety and Infection Control

16-22%

8-14%

17-23%

9-15%

17-23%

9-15%

Health Promotion and Maintenance 6-12% 6-12% 6-12%

Psychosocial Integrity 6-12% 6-12% 6-12%

Physiological Integrity

- Basic Care and Comfort

- Pharmacological and Parenteral

Therapies

- Reduction of Risk Potential

- Physiological Adaptation

6-12%

13-19%

10-16%

11-17%

6-12%

12-18%

9-15%

11-17%

6-12%

12-18%

9-15%

11-17%

2/15/2017

5

| 17

Should we teach to the test?

| 18

Curriculum

Policies

Product Integration

Student Preparation

Remediation

Data Analysis

Steps to

Success

| 19

• What research is available that demonstrates that your

exams are reliable and valid?

• Are exams questions piloted to ensure that questions are

psychometrically sound?

• Are the exams base on the NCLEX blueprint?

• How often is exam content updates?

• Qualifications of the item writers?

• Use of data?

• Security?

Considerations

| 20

• Cohort Data – one point in time (cohort specific

variables can affect data)

• Trending Cohort Data – identify ongoing patterns

Data Analysis Focus

•

2/15/2017

6

| 21

Are you reviewing and using data after each

standardized exam?

| 22

Summary of Cohort Data

| 23

Look for normal distribution patterns or bell curve

| 24

Sample data – student data

Student name

2/15/2017

7

| 25

• Does the mean cohort score make sense?

• Does the distribution of individual scores make sense?

• Does the individual student score correlate with students

performance?

Does the data make sense?

| 26

Review cohort score in categories and number of items

| 27

Item detail – Institution score vs. National score

| 28

Are you trending exam data across time?

2/15/2017

8

| 29

• Potential consistent curriculum weaknesses

• Curriculum gaps

• Curriculum strengths

Trending vs Individual Exam Scores

| 30

Trends and item details….

| 31

| 32

• Who is reviewing the cohort data?

• Who is responsible for trending data and identifying

concerns in cohorts?

• What happens to identified cohort concerns/weaknesses?

• Are all faculty involved and responsible for data analysis?

• Is exam data correlating with other assessment data?

• Is exam data trended to determine ongoing concerns?

• Are data analysis activities documented?

• Are identified interventions documented and follow up data

analysis should reflect improvement

Exam Data -- CQI

2/15/2017

9

Develop student

remediation tactics that

foster positive outcomes.

Integrative Review, multiple databases,

2000-2014 - 25 articles

Remediation

Culleiton, 2009

“The process of identifying

the need to take action to

remedy a situation that, if left

unresolved, will result in

unfavorable outcomes,

whereas implementing

intervention strategies will

successfully address the

situation.”

| 35

2003 - 28% of schools in a national

sample required remediation for nursing

students when assessment benchmarks

were not achieved (Nibert, Young, &

Britt, 2003)

2010 - 64% of schools surveyed required

nursing students to remediate after the

initial failure to meet a benchmark and of

these 37% required proof of remediation

completion (Barton, Willson, Langford, &

Schreiner, 2014).

Remediation on the Rise!

| 36

Test Performance can be affected by non academic and personal

factors affect test performance

Remediation –

frequently based on test performance

Test performance

Test Anxiety

Self Efficacy Remediation

2/15/2017

10

Individual or group remediation?

| 38

• Individual remediation impacts performance

• Standardized exams help identify deficiencies

• Multiple strategies impacted NCLEX success

• Student contracts increased ownership and

elevates the importance of remediation

Individualized Remediation

| 39

• Targets the group’s weaknesses

• Target test strategy and test anxiety

• Credited course

• Built into curriculum

Cohort Remediation Remediation for All?

2/15/2017

11

| 41

Should remediation be for only high risk students?

How do you define the high-risk student?

• Course scores and scores on standardized

exams

• Groups participating in enhanced remediation

had significantly higher NCLEX scores.

Build “value” of remediation early

Consider program long attention to remediation

Who needs remediation and when?

Remediation is a form of

relearning and “relearning not

only enhances retention, it

facilitates reacquisition of

forgotten information.”

(Rawson and Dunlosky, 2013)

| 43

Quantifying the amount of remediation?

Faculty expert opinion

Time

Measurable activities

Retesting post remediation

Measuring remediation?

| 44

Remediation strategies vary

Barton, et al. (2014). Administrative Issues Journal, 4(2), 68-76

HESI Validity 10

2/15/2017

12

| 45

• Type of remediation interventions varied and progression

and remediation policies were effective. “No

consistency in type of remediation is likely an

indication that the progression and remediation

policy motivated the students to study.” (Morrison et

al. 2002)

• Mandatory remediation

HESI exit scores were significantly higher in

programs that required remediation after failing to

achieve benchmark. (Barton, Willson, Langford, and

Schreiner, 2014)

Remediation Policy Research

| 46

New Grads at work given standardized exams,

remediation and retest. Students reported that in nursing

program the test and remediation was not explained to

them. They were more receptive when understood

potential.

Instilling the “Value” of Remediation

| 47

Curriculum Revision

| 48

Plan

Do Study

Act

Plan

Do Study

Act

Plan

Do Study

Act

2/15/2017

13

| 49

• Represents practice

environment

• Meets professional

standards

How can you tell? • Review NCLEX Test

Plan

• Review faculty made

tests (test blue print)

• Review aggregate

results of standardized

tests against

benchmark

Curriculum Validity

| 50

Client Need Categories

Safe and Effective Care

Management of Care

Safety and Infection Control

Health Maintenance and

Promotion

Psychological Integrity

Physiological Integrity

Basic Care and Comfort

Pharmacology and Parenteral

Therapy

Risk Reduction

Physiological Adaptation

Integrated Processes

Nursing Process

Caring

Communication and

Documentation

Teaching and Learning

Culture and Spiritualty (new)

What Data is Available?

NCLEX Test Blue Print (Detailed, Educator)

| 51

NCLEX Test Design

• Activities

• Related Content

Standardized test data (Specialty, Exit)

Results from faculty made tests

What Data is Available?

| 52

Teacher made

• Most common

• Less expensive

• More control of content

• Match chosen teaching

materials

Teacher-made and standardized exams

comparison

Standardized Tests • Nationally normed

• Diverse expert item writers

• Emphasis on higher order thinking

• Multilogical thinking

• Peer reviewed and edited

• Decrease legal risk

• Mapped to national or nursing organization content outline

/

2/15/2017

14

| 53

• Not Linked to NCLEX Test plan

• Not Peer Review

• No analysis of distractor discrimination 1

• No assessment of cultural bias 1

• Frequent low cognitive level 2

• Violates good item writing protocol 2

• Items too easy 3

• Distractors often not chosen 1,3

• Stimulates student anxiety 4

Research on teacher-made exams

1. Killingsworth (2013); 2. Tarrant (2006); 3. Clifton & Schriner (2010); 4. Bailey (2012)

| 54

• Map curriculum to NCLEX-test plan

• Tag Test questions to:

NCLEX-Category

Cognitive level

Peer Review tests, using testing protocol

• Cognitive level

• Analyze test results

• Compare student results in faculty made

exams to scores in standardized tests

Improving Faculty-made tests

| 55

1. Analyze aggregate HESI data to

identify weaknesses, particularly

in NCLEX categories.

2. Compare poor performance on

NCLEX Categories in HESI with

low aggregate scores in test

items tagged in identified

NCLEX category.

3. Is there congruence between

the two? If so, evaluate:

• Objectives

• Teaching-Learning activities

• Serendipity

Test Data Analysis: CQI Process

How might this CQI Process be integrated into the Program Evaluation? (Table Discussion)

2/15/2017

15

Facilitated Discussion: Using the Group

Process to Make Change

| 58

• A guide or discussion

leader for the group

• Facilitation provides

leadership without taking

the reigns

• Get others to assume

responsibility and take

the lead

Definition of facilitator

| 59

Should the facilitator have a “dog in the fight”?

| 60

Process Facilitator: How Content Leadership : What

Methods and Procedures The subjects for discussion

How relationships are maintained The Task

Tools being used The problem being solved

The roles or norms set The decision being made

Groups The agenda Items

Dynamics The goals

Climates

2/15/2017

16

| 61

Characteristics Facilitator Facilitative leader

Group Membership Third Party Group Leader

Involvement in

Substantive Issues

Substantially neutral Deeply Involved in

issue

Use of expertise Process Expert

May have general

content knowledge

Process and content

expert

Decision-making power No Yes

Differences between Facilitators and Facilitative Leaders http://www.virginia.edu/processsimplification/resources/Facilitator.pdf

Facilitation in Difficult Situations

| 63

Has the group made every

effort to be collegial and

supportive?

Stop wishing the

difficult folks would stop

being difficult.

You may be able to

control the environment

to reduce disruptions,

but you can not control

other people’s

behaviors.

Getting your own house in order

| 64

Participant interrupts

frequently: Allowing the

disrupter to manipulate the

group in order to “keep the

peace” rewards the

troublemaker for

objectionable behavior.

Action: Redirect the

speaker: “Ok Mary, let’s

hear from Jane now.”

The Environment is disrupted

2/15/2017

17

| 65

Participant verbally attacks

facilitator or another

member of the committee

Action: Maintain self-

control. Identify

inappropriate behavior:

“Let’s all take a deep

breathe, We want to remain

civil in our discussion.”

“Mary, Let’s discuss this

concern at another

time.”

Rude or Attacking

| 66

Assign unlikely people

to discuss both sides—

assign the devil’s

advocate to the

opposite side

Acknowledge the

perspective, then ask if

there are other

reactions,

“What solution would you

have to resolve this issue?

The Chronic Devil’s Advocate

| 67

Person won’t give up on an

idea.

Acknowledge the

importance of this to him.

State clearly that others do

not support this idea

The “Broken Record”

| 68

• Ask for intent, versus

positions

• Clarify needs and

beliefs-Understanding

drivers helps move to

win-win

• Identify points of

agreement, then

determine how

differences will be

resolved and who

gets to decide

Different Philosophies or Beliefs

2/15/2017

18

| 69

• Not all decisions are majority vote.

• Be honest about who makes the final

decision, before the work begins

• Set time limits, in some cases

What if there is no agreement?

| 70

| 71

NCLEX Test Plan

Class Simulation/Lab Clinical

Teaching-Learning Process to integrate

NCLEX Categories

| 72

Management of Care:

Activity Statement

Maintain client confidentiality

and privacy

Related Content

Participate in cost effective care*

How can activity statements or content

examples help us plan teaching-learning

activities?

2/15/2017

19

| 73

Activity Statement

Use ergonomic principles when providing care (e.g., assistive devices, proper lifting)

• Assess peer’s actions which moving, listing patients

Content

Reporting of Incident. Event, Irregular, Occurrence, Variance

„

• Identify need/situation where reporting of incident/ event/ irregular occurrence/variance is appropriate

Safety and Infection Control

| 74

Activity Statement

Provide information for

prevention and treatment of

high risk health behaviors

(e.g., smoking cessation,

safe sexual practices, drug

education)

Content

Identify risk factors for

disease/illness (e.g., age,

gender, ethnicity, lifestyle)

Health Promotion and Maintenance

| 75

• Mapping for each syllabus

• Learning activities in class

• Mapping of Test Items

• Test Items written at the application level or higher

• Analysis of faculty-made and standardized test

Remediation of individuals

Remediation of Class

Tracking and Revision of Course

CLASS: Use of NCLEX Test

Plan

| 76

Specific Objectives based on NCLEX Test Plan congruent with class

objectives

Simulation Lab

2/15/2017

20

| 77

In your typical Clinical experience---

• Which categories do you typically emphasize?

• Which categories are not likely to be

emphasized?

• Do you correlate assignments related to

weaknesses in testing?

Question:

| 78

Most clinical experiences (and pre- and post-clinical preparation) should feature learning objectives related to:

1. Patient- Centered Care

Nursing Process

Physiological (Basic Care/ Comfort; Pharmacological/Parenteral Therapies; Reduction of Risk)

Communication

2. Learning Objectives related to:

Caring

Health Promotion & Maintenance

Psychological Integrity

Management of Care

Physiological Adaptation

Safety/Infection Control

Cultural & Spiritual

3. Post-Clinical Conference and/or Debriefing

Intentional Framework for Planning Clinical

Experiences

| 79

1. Implement an appropriate

focused assessment,

including VS. (Physiological

Integrity: Reduction of Risk)

2. Implement, evaluate &

document morning care

required for selected patients

(Physiological Integrity:

Basic Care/Comfort,

Communication)

3. Educate client about

medications (Physiological

Integrity: Pharm,

Teaching/Learning

4. Assess skin and

implement measures to

maintain skin integrity

(Physiological Integrity:

Basic Care/Comfort)

5. Provide non-

pharmacological comfort

measures and evaluate

patient response. (Physiological

Integrity: Basic Care/Comfort)

6. Use therapeutic

communication techniques to

provide client support

(Psychological Integrity;

Communication)

Clinical Objectives : Nursing Process and Caring is integrated throughout

| 80

Management of Care

• Utilize information resources to enhance the care provided to a

patient with a Fx hip (e.g., evidenced- based research, information

technology, policies and procedures)

Health Promotion and Maintenance

• Determine potential area of health risk. Assess client’s readiness

to learn, learning preferences and barriers to learning (Also risk

reduction, teaching learning)

Physiological Integrity: Physiological Adaptation

• Relate the physiological effects of a fx hip to the best practices of

care.

Safety /Infection Control

• Scavenger Hunt: Strategies to insure safety

Other Clinical Objectives

2/15/2017

21

Choose a Clinical

objective and

plan a clinical

experience

Group Activity:

| 82

• Barton, et al. (2014). Administrative Issues Journal, 4(2), 68-76

• Mee, C. L., & Hallenbeck, V.J. (2015). Selecting Standardized Tests in Nursing Education. Journal of Professional Nursing. 31(6), 493-497.

• Mee, C.L. and Schreiner, B. (2016). Remediation in Nursing Education Today: Review of the Literature and Considerations for Future Research. Journal of Nursing Regulation. 7(1), 37-45.

• Morrison, S., Free, K., & Newman, M. (2002). Do progression and remediation policies improve NCLEX-RN pass rates? Nurse Educator, 27(2), 94-96.

• Culleiton, A. L. (2009). Remediation: A closer look in an educational context. Teaching and Learning in Nursing, 4 (1), 22–27.

References