Incorporating Standardized Tests and Modifying Policies to Measure Program Outcomes

Bernice Brennan MSN,RN Andrea Schapire MSN,RN

PRESENTATION OBJECTIVES

Historical perspective of standardized testing in nursing education

Overview of recent literature related to standardized testing and policies

Modification of policies/measurements Incorporation of standardized testing

from a student and faculty viewpoint Measurement of SLO(student learning

outcomes) and overall NCLEX results

KEY TERMS

NCLEX-RN scores Standardized Tests Student Learning Outcomes (SLO) External Assessment Programs Non-traditional students Computerized Adaptive

testing(CAT) Benchmarks

Historical Perspective A little bit about nursing history….. https://

www.youtube.com/watch?v=GQ2ni7VOobQ

Professional roots of nursing in the United States since 1870’s when the first school was organized in New York City

Scope of nursing practice for new graduates: Dispensing medications Providing physical and psychologic comfort

to the ill and infirm(Holstein 2006)

Historical Perspective Nursing care was based on standards

recognized across the country and on more stringent educational requirements.

Prior to 1982 in the US, the State Board Test Pool Examination was administered by individual states, then the NCLEX-RN replaced the Test Pool Exam.

The NCLEX exam was the accepted method of mandating that a nurse has met the minimum requirement for safe nursing practice

THE BOARDS

Where did you take them?

Was it paper and pencil?

Who were your proctors?

Historical Perspective 1941-1982 The State Board test pool

Examination (SBTPE)one exam used for all students (paper and pencil)

1978 NCSBN(National Council Licensure Exam) established. This organization renamed exam NCLEX in 1982

1986-1994- pilot studies conducted to determine sound, valid and legality of the NCLEX

Historical Perspective

First NCLEX on-line- April 1st 1994

(155,00 candidates took the NCLEX via computerized testing (CAT)

Review of the Literature An eight year review of the

literature utilizing CINAHL/Medline data bases were retrieved from : (2006-2014)

Temple University Ginsberg Health Science Library

Thanks to Rebecca LloydEducation Services Librarian

Review of the Literature Highlights (2006)Holstein- Standardized Testing

Tools to Support Quality Educational Outcomes- many schools of nursing integrated standardized computerized testing into the curricula to decrease the failure rate in the NCLEX-RN examination. Student’s at risk for failure in NCLEX-RN would have time for remediation

Predictors of success in NCLEX-RN was based on GPA’s in nursing courses, NLN Comprehensive test scores, Mosby’s Assess Test, E2

Review of Literature

Cont’d Holstein: Multiple researchers conducted

studies to identify predictors for NCLEX success but data not consistent, so schools of nursing turned to commercial companies for assistance through computerized testing that simulates the NCLEX-RN

Review of the Literature

(2008)-Richards and Stone: Student Evaluation of a Standardized Comprehensive Testing Program-report high level of satisfaction with testing, favorable SLO’s

Faculty gave points for classes with standardized tests Students given remediation and felt it was helpful

Review of the Literature

(2008) Spurlock,D,R. and Hunt,L,A. A study of the usefulness of the Hesi exam at predicting NCLEX-RN Failure. Many schools of nursing have implemented progression policies based on a single end-of-program predictive test.

Review of the Literature

(2010)- Harding, M. Predictability Associated with Exit Examinations: A Literature review

Only 16 studies have been done on Standardized testing. HESI most widely used standardized test with 86.8% predictability for success in NCLEX. Determined more research needed in this area.

Review of the Literature

(2011)- DeLima, L. Retrospective Study of AD Programs-Standardized test results and pass/fail rate in nursing courses to be the most significant variables identified in profiling students most likely to fail NCLEX. Males and minorities (African-American, Latino and Asian) had lower pass rate than their White counterparts.

Review of the Literature

(2011) Carr- NCLEX-RN pass rate peril: One school’s journey through curriculum revision, standardized testing, and attitudinal change.

Faculty are reluctant to use external exams to evaluate student’s ability and knowledge. Self-made standardized exams (faculty prepared) utilized in each clinical course. Remedial courses implemented for student’s not passing the exams

Review of the Literature

(2011) Alameida, M.D- et.al, Predicting NCLEX-RN Success in a Diverse Student Population- 589 students taking ATI Predictor from 2003-2009. Standardized testing had positive outcomes with first time attempt of NCLEX. Age not a predictor. Men failed exams at higher rates. Cumulative GPA’s and increased grades in nursing had impact on NCLEX success

*MH Pharm and Fund. predicted success.

Review of the Literature (2012) Staykova, P. Community college education through the

looking glass of Associate Degree Nursing- numerous barriers affected students success: inflexible schedules, busy curriculum, and insufficient socioeconomic status, minority students, and immigrants with limited English language skills.

Competent educators must be skilled at integrating assessment tools throughout the curriculum to help students pass NCLEX-RN on the first attempt.

Review of the Literature

(2013)-Ayers,V,D.- Finding the balance in testing-Improving performance and setting consequences: Survey 2013-Deans,Directors and Faculty

Questionaire looked at the barriers/benefits to standardized testing.

Review of the Literature

(2013)Santos,L.T et.al: The use of Standardized Exit Examinations in Baccalaureate Nursing education: looked at ethical, legal, cultural, socioeconomic and technological considerations of standardized testing

Each school needs to determine if one predictor can restrict student from graduating or taking the exam

Legality-high stakes testing Cultural-English as a second language (ESL

students). Socioeconomic Technological (space/students with disabilities)

Review of the Literature (2013) Halstead, J, The NLN’s

fair testing imperative and implications for faculty development:

20% of nursing schools are required to achieve a minimum score on ST to graduate from program and take the licensure examination, regardless of how well they mastered their required nursing courses

Con’td (2013) Halstead Many faculty, even experienced

faculty,find the development of valid and reliable means of evaluating student learning to be a challenge

Faculty must have access to comprehensive testing, administration, and evaluation information before they administer or write policies regarding use of standardized testing

Review of the Literature

(2013) Zweighaft,E. Impact of HESI Specialty exams: The 9th HESI Exit Exam Validity Study- 379 students all random schools and types of programs, HESI E2 can be confidently used to evaluate students for licensure exam readiness

Implementing as the final exam to students, the higher the grade the more successful the student.

Review of the Literature

(2014)- Coons, I,V,C. : Use of standardized tests within nursing education programs-Use of standardized exams has increased over several decades. May be related to satisfy various accreditation requirements (including annual state board of nursing pass rates).

Controversy due to tests being detrimental to minority students

Review of the Literature Cont’d Coons

Experts in the field of measurement need to be consulted by nursing faculty in relation to policies concerning standardized testing

Expectations and goals with implementation of standardized testing must be in line with nursing schools mission ,philosophy and frameworks

Incorporating Standardized Testing Policies

NEHSON: Standardized Testing (In Student Handbook) (2010-2011)

Standardized Testing Complete and achieve the minimal Proficiency

Level 2 for all exams at each nursing level. Remediation with faculty members after all

unsuccessful exams. Remediation of 100% on unsecured exams(Form

B) before repeating the secured exam a second time

Graduation Completion of all standardized tests with a

minimal Proficiency Level 2 Student must achieve 91% predictability on

Comprehensive Predictor Exam Then attend Live Review

Incorporating Standardized Testing Policies

(2012-2013) Standardized Testing Complete and achieve the minimal Proficiency Level 2 for

all exams at each nursing level. Remediation with faculty members after all unsuccessful

exams. Remediation of 100% on unsecured exams(Form B)

before repeating the secured exam a second time Graduation Completion of all standardized tests with a minimal

Proficiency Level 2 Student must achieve 91% predictability on

Comprehensive Predictor Exam Live Review

FEBRUARY 2012

Fair Testing Guidelines implemented (NLN) Feb. 2012

Students notified of testing policies at meet and greet, prior to admission

Reviewed during orientation Freshmen Year

In each course syllabi where standardized testing is to be completed

Remediation plan developed and implemented

Incorporating Standardized Testing Policies

(2013-2014) (Provisional Status) Standardized Testing Complete and achieve the minimal Proficiency

Level 2 for all exams at each nursing level. Remediation with faculty members after all

unsuccessful exams. Remediation of 100% on unsecured exams

(both A and B Forms)before repeating the secured exam a second time.

Focused assessments to be completed for each individual student on unsuccessful proctored exam before taking it the 2nd time.

 

2013-2014 Focused assessments to be completed for each individual

student on unsuccessful proctored exam before taking it the 2nd time.

No more than one ATI proctored exam per day- can not remediate and take the same day.

Fundamentals assessment unsecured tests done at holiday break- a minimum of 3 hours of remediation to be done prior to taking proctored exam.

A fee to be charged for any exam after the 2nd attempt  Graduation Completion of all standardized tests with a minimal

Proficiency Level 2 Student must achieve 94% predictability on Comprehensive

Predictor Exam A readiness program and individualized remediation

developed for each student and detailed report signed prior to graduation.

Live Review Capstone Implemented 

2014-2015 (Full Approval-off Provisional)

Standardized Testing Complete and achieve the minimal Proficiency

Level 2 for all exams at each nursing level. Remediation with faculty members after all

unsuccessful exams. Remediation of 100% on unsecured exams (both

A and B Forms)before repeating the secured exam a second time.

Focused assessments to be completed for each individual student on unsuccessful proctored exam before taking it the 2nd time.

No more than one ATI proctored exam per day- can not remediate and take the same day.

2014-2015 (Full Approval-off Provisional)

Fundamentals assessment unsecured tests done at holiday break- a minimum of 3 hours of remediation to be done prior to taking proctored exam.

A fee to be charged for any exam after the 2nd attempt   Fair Testing Guidelines implemented (NLN)Feb. 2012 Students notified of testing policies at meet and greet , prior to

admission Reviewed during orientation Freshmen Year In each course syllabi where standardized testing is to be completed Remediation plan developed and implemented   Graduation Completion of all standardized tests with a minimal Proficiency Level 2 Capstone Student must achieve 97% predictability on Comprehensive Predictor

Exam A readiness program and individualized remediation developed for

each student and detailed report signed prior to graduation.  

Comparative Analysis using Standardized testingClass of 2014-2015

Fundamentals Assessment

CLASS OF 2014 N=42

Individual Score Required

CLASS OF 2015 N=38

Adjusted group score 68.4%

Level 3 80-100%Level 2 68.3-78.3%Level 1 58.3-66.7%Below Level 1 <58.3%

Adjusted Group Score 68.6%

Proficiency Level 3 N=1

Proficiency Level 3 N=1

Proficiency Level 2 N=22

Proficiency Level 2 N=21

Proficiency Level 1 N=17

Proficiency Level 1 N=15

Below Level 1 N=2 Below Level 1 N=1

% of group Above Ind. Mean program54.8%

% of group Above Ind. Mean Program57.9%

Comparative AnalysisClass of 2014-2015

Fundamentals AssessmentCLASS OF 2014 MAJOR

CONTENT AREAS

CLASS OF 2015

Sub-Scale Group Score

Subscale Group Score

Safety Inf. Control 61.2%

Safety Inf. Control 61.8%

Health Pro&Maint 64.3%

Health Pro&Maint 64.3%

Psychosocial Integrity 58.1%

Psychosocial Integrity 58.1%

Basic Care and Comfort 58.7%

Basic Care and Comfort 59.4%

SLOCLASS OF 2014 CLASS OF 2015

THINKING SKILLS THINKING SKILLS

CLIN.JUDGEMENT/CRITICAL THINKING IN NURSING 67.2%

CLIN.JUDGEMENT/CRITICAL THINKING IN NURSING 69.5%

PRIORITY SETTING GROUP SCORE 68.2%

PRIORITY SETTING GROUP SCORE 68.8%

NURSING PROCESS NURSING PROCESS

PLANNING GROUP SCORE 74.8%

PLANNING GROUP SCORE 76.3%

EVALUATIONGROUP SCORE 60.5%

EVALUATIONGROUP SCORE 63.4%

EVALUATION OF DATAClass of 2015

SAFETY AND INFECTION CONTROL INCREASED BY 0.4%

HEALTH PROMO AND MAINT INCREASED BY 0.7%

PSYCHOSOCIAL INTEGRITY INCREASED BY 8.7%

BASIC CARE AND COMFORT INCREASED BY 0.7%

Class of 2015 adjusted score increased 0.2% from 2014 class

Class of 2015 2 less students with Proficiency Level score 1

Class of 2015 Individual Mean for Program increased 3.1%

STUDENT LEARNING OUTCOMESGroup Scores Class of 2015

Clinical Judgment/Critical Thinking

2.3% increase

Priority Setting 0.6% increase

Planning 1.5% increase

Evaluation 2.9% increase

FACULTY VIEWPOINT

All faculty needs to be committed to the process

Why were faculty non-supportive• Lack of technical skills• Unknown concepts• Threat to autonomy of teaching• Unsure how to remediate • Unsure how to evaluate results and

read individual/group reports

INTERVENTIONSFACULTY ORIENTATION

Live in-service from testing vendor representative Faculty supporting other faculty

SCHEDULED WEBNARS FOR ALL FACULTY Monday afternoons (7 topics) One hour in length TOPICS: Anatomy/Physiology Learning Strategies Active Stack Pharm Flash Cards Nursing Touch Manager Certificate Program Dosage Calculations Reporting/Remediation

STUDENT VIEWPOINTSCLASS OF 2014

More work, More stress Did not take it seriously No grade or value to it in their

mindset Timely burden Unable to understand their results Anger

Class of 2015 Implementation Strategies…

Early introduction to test taking skills Test reviews (theory )and how it is

looked at from a test taking strategy Positive reinforcement to students

explaining standardized testing is a skill for many

Faculty cheerleading students to success

Remediation for those identified early on with difficulty

Class of 2015 Implementation Strategies…

Students oriented to standardized testing in the classroom

Emphasis placed on test taking strategies Prior to taking proctored assessments

students required to do practice assessments A & B only once. A focused review will be done for each assessment minimum 3 hours for each total 6 hours.

Scheduled for proctored assessment, needed to have transcript with time spent on focused review as ticket to take proctored assessment.

ATI Capstone Course IntegrationCLASS OF 2014-New

During the senior semester and the Leadership and Management in Nursing Course, all 33 students were enrolled in the ATI Capstone Course.

Students were required to complete all components of the Capstone Content Review.

On 2/5/14, students were given the 2010 Comprehensive Predictor

Then they followed with nine weeks of content review. (Capstone)

On 3/26/14, the students completed the repeat of the 2010 Comprehensive Predictor

Capstone Guidelines

Successful completion of the Capstone Content Review will include:

completion of each content assessment

focused review quality post-assessment

assignments. Post-Capstone Predictor

Dr. Michelle M. Murphy-Rozanski, PhD, MSN, RN, CRNP

Name M/S Nursing 2nd M/S Post Pharmacology 2nd Pharm Post Comp Pred #12nd Predictor Post-Capstone

Increase in Proficiency

Proficeincy Level Proficeincy Level Percentage Percentage 76=97% Post CapstoneJ.A. 1 1 1 2 57.3 65.3 8S.A. 1 1 1 1 68 74.7 6.7A.B. 3 3 2 3 76 82.7 6.7M.B. 3 3 2 2 58.7 64 5.3B.C. 2 3 2 3 76.7 77.3 0.6M.C. 1 1 2 1 62 80 18B.C. 1 1 2 2 70 78 8S.E. 2 2 2 2 65.3 72.7 7.4N.E. 3 3 2 2 76.7 80.7 4B.G. 3 2 3 2 78.7 77.3 -1.4T.G. <1 1 1 2 60 58 -2A.J. 2 2 2 2 69.3 78 8.7D.J. 1 2 1 2 66.7 76 9.3

E.J. 2 3 2 2 76 70 -6D.L. 3 3 2 2 74.7 84 9.3T.M. 1 3 2 3 68 72.7 4.7E.M. 1 1 1 1 53.3 64.7 11.4D.M. 3 3 2 2 76.7 74 -2.7R.N. 2 2 2 2 66.7 62 -4.7W.N. 2 2 2 2 62.7 72 9.3A.P. 2 2 2 2 70 75.3 5.3T.R. 2 2 2 2 70 73.3 3.3S.R. 1 2 2 2 62 76 14F.R. 2 2 2 2 72.7 75.3 2.6E.R. 1 2 1 1 56.7 68 11.3D.S. 2 3 2 2 74 79.3 5.3K.S. 1 3 2 2 78 70 -8V.S 3 2 2 3 74 72.7 -1.3K.S. 2 2 2 2 68.7 76 7.3A.S 1 2 2 1 70.7 67.3 -3.4D.T. 2 2 1 1 70 73.3 3.3E.T. 1 2 1 1 62 71.3 9.3T.W. 3 3 2 2 73.3 75.3 2

STUDENT LEARNING OUTCOMES POST

CAPSTONE

Overall total N=25 179.12 points increased after Capstone on 2nd Predictor attempt

Overall mean 7.16 points per student

Range 0.6-14 points per student

6 students that failed NCLEX-RN on the First Attempt

All failed on the FIRST ATTEMPT

Medical-Surgical*** Pharmacology Comprehensive Predictor

NEHSON NCLEX-RN Pass Rates

Class of 2014 had 33 Students

27 out of 33 Passed on the first attempt

Overall Pass Rate = 81.81%

Off Provisional!!!!

https://www.youtube.com/watch?v=29fdVOqraQs&list=RDEN5zt7c7jHM&ind

ANY QUESTIONS?????????????????????

Bernice.Brennan@tuhs.temple.eduAndrea.Schapire@tuhs.temple.e

du