using nursing exam data effectively in preparing nursing accreditation

45
Using nursing exam data effectively in preparing nursing accreditation Ainslie T. Nibert, PhD, RN, FAAN Consultant Email – [email protected] 1

Upload: examsoft

Post on 20-Mar-2017

52 views

Category:

Education


2 download

TRANSCRIPT

Page 1: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Using nursing exam data effectively in preparing nursing

accreditation

Ainslie T. Nibert, PhD, RN, FAANConsultant

Email – [email protected]

1

Page 2: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Objectives• Identify the steps needed to establish a systematic method

for documenting the reliability and validity of teacher-made tests used throughout the curriculum.

• Discuss analyses of student performance data collected from assessments used throughout the program that establish the success of the curriculum in meeting nationally-recognized standards and promoting achievement of program outcomes.

• Describe ways of incorporating assessment-related findings in the self-study prepared for the accreditation visit that illustrate how the testing program assists faculty in identifying student knowledge gaps, and evaluates the effectiveness of strategies employed to close these gaps.

•  

2

Page 3: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Nursing Accreditation

3

Source: http://www.aacn.nche.edu/ccne-accreditation

Source: http://www.acenursing.org/mission-purpose-goals/

Source: http://www.nln.org/accreditation-services/the-nln-commission-for-nursing-education-accreditation-(cnea)

Page 4: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Teacher-made tests & Standardized Exams in Nursing provide essential data in our outcomes-oriented academic world• Nursing has moved from process-oriented standards to

program outcome-oriented standards• The increase in use of computerized, standardized exams that are

produced under rigorous psychometric standards provides reliable data and more opportunities for deeper analysis of results.

• As standardized testing resulted in ‘raising of the bar’ with nursing assessment, teacher-made testing has also moved increasingly into computerized applications that allow for psychometric analysis.

Page 5: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

CCNE’s Accreditation Standards• Standard I: The mission, goals, and expected program outcomes

are congruent with those of the parent institution, reflect professional nursing standards and guidelines, and consider the needs and expectations of the community of interest. Policies of the parent institution and nursing program clearly support the program’s mission, goals, and expected outcomes. The faculty and students of the program are involved in the governance of the program and in the ongoing efforts to improve program quality.

• Standard IV c Licensure and certification pass rates demonstrate program effectiveness.• The NCLEX-RN® pass rate for each campus/site and track is 80% or higher for first

time takers for the most recent calendar year.

• http://www.aacn.nche.edu/ccne-accreditation/Standards-Amended-2013.pdf

5

Page 6: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

ACEN’s Accreditation Standards• STANDARD 1 Mission and Administrative Capacity

• The mission of the nursing education unit reflects the governing organization’s core values and is congruent with its mission/goals. The governing organization and program have administrative capacity resulting in effective delivery of the nursing program and achievement of identified program outcomes.

• STANDARD 6 Outcomes • Program evaluation demonstrates that students and graduates

have achieved the student learning outcomes, program outcomes, and role-specific graduate competencies of the nursing education unit.

Source: http://www.acenursing.net/manuals/SC2013.pdf

6

Page 7: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

NLN CNEA’s Accreditation Standards• Standard I: Culture of Excellence – Program Outcomes

• Establishing clear program outcomes is an essential first step in benchmarking and evaluating a nursing program’s success in achieving and sustaining a quality educational environment for faculty and students. Program outcomes can be defined as the results achieved in response to goals set by the program….

• 1a - Examples of faculty and staff engaging in regularly scheduled review of identified program outcomes with documentation of review outcomes and resulting decisions Examples of data-driven decisions based on the review analysis of achievement of identified program outcomes

• 1d -Three calendar years of trended data on licensure and certification pass rates. Documentation of implementation and evaluation plans to achieve set benchmarks related to licensure and certification examinations and a plan for intervention if needed.

• Source: http://www.nln.org/accreditation-services/the-nln-commission-for-nursing-education-accreditation-(cnea)

7

Page 8: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Documenting program achievement

Sources of outcome measuresInternal and external curriculum evaluation plans

Data collection and analysis Preparation of the self studyProduce evidence to support self study

Internal curriculum evaluation measures – teacher made testsExternal curriculum evaluation measures – standardized testsLicensure exam results

8

Page 9: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Relationship betweenTesting & the Curriculum

9

Internal and ExternalCurriculum Evaluation

Outcome Predictors

Page 10: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Where do I start?• What can I do now to start preparing for reaffirmation of

accreditation self study and visit• Specific recommendations for strengthening testing program

• Teacher made tests• Standardized tests• Item analysis reports• Exam analysis reports• Testing policies, procedures, and guidelines• Closing the loop – changes to curriculum based on these analyses

• Best Practices• Follow best practices for managing your program’s assessments• Strengthening the testing and evaluation component of your program

NOW will result in a sound dataset for analysis of the curriculum, and a stronger self-study

10

Page 11: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Best Practice #1: Establish Testing Committee• Design and revise the program’s testing policies

• Publishes Guidelines for Exam Development • Writing style protocol – apply to all tests• Critical thinking items – increase Application level

• Exam Administration Policies• Number of items/exam; length of time allotted/exam• Proctoring Guidelines• Dissemination of Grades (When? How?)• Test Review – to do or not to do; how to do it

Page 12: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

…more Testing Committee Activities

• Systematic Item Analysis (see also #3 below)• Use statistical parameters for analyzing overall test and individual test items

• Adhere to published psychometric standards• Summarize analysis and include action plan• Review test blueprint as needed before next test• Review individual items earmarked for attention

• Compile/analyze standardized test results• Make recommendations to curriculum committee for changes based on findings

12

Page 13: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Relationship betweenTesting & the Curriculum

INTERNAL Curriculum Evaluation(Teacher-made Tests)

Page 14: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Recommended Item Analysis Standards

14

Parameter Level of Acceptance

Item Difficulty 30%-90%

Item Discrimination Ratio 25 % and above

PBCC (point biserial correlation coefficient)

0.20 and above

KR20 (Kuder-Richardson 20) 0.70 and above

Morrison, Nibert, Flick, J. (2006). Critical thinking and test item writing (2nd ed.).Houston, TX: Health Education Systems, Inc.

Page 15: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Establish the use of a Systematic Item Analysis used for all teacher-made tests

1. Review Difficulty Level

2. Review Discrimination Data

• Item Discrimination Ratio (IDR)

• Point Biserial Correlation Coefficient (PBCC)

3. Review Effectiveness of Alternatives

• Response Frequencies

• Non-distracters

Page 16: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Best Practice #2: Writing Style ProtocolEstablishing clear guidelines for faculty leaves little room for ambiguity; helps insure uniformity in the presentation of exams throughout the curriculum:

Will present or past tense be used for test items? Will options end in periods, whether or not there are

complete sentences? Will all options begin with a capital letter? When stressing a word in the stem, will it be highlighted,

boldfaced, italicized, underlined? Will the term patient or client be used?

Page 17: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Best Practice #3: Increase proportion of Application-and-above Items

17

Which intervention is most important?

Which intervention, plan, assessment data is/are most critical to developing a plan of care?

Which intervention should be done first?

What action should the nurse take first?

Which intervention, plan, nursing action has the highest priority?

What response is best?

Page 18: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Best Practice #4: Use Uniform Statistical Parameters for Item

AnalysisWhat is the acceptable number of mastery itemsto include?

What is the minimum acceptable difficulty levelfor a test item?

What is an acceptable discrimination level (PBCC) for a test item?

What is an acceptable reliability coefficientfor the exam?

18

Page 19: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Key Testing ‘Best Practices’

• Improve Test Blueprints • Adopt a uniform item style/format; publish the guidelines• Invert the proportion of Application-and-above test items

as compared to Knowledge/Comprehension items• Use a Systematic Item Analysis methodology• Include 10% alternative item formats – across all tests• Remove items that no longer align with NCLEX ® items

• Obsolete nursing diagnoses• Trade names for most medications (use generic only)

• Include 10% med calculations – across all tests• Revise length/time allotments – across all tests

19

Page 20: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Maintaining Consistent Testing and Grading Policies• Establish the Use of Best Practices with Testing• Update Policies and Re-educate annually• Discuss Testing Practices in a Faculty Forum

• Identify areas of inconsistency• Review test outcomes for impact of inconsistency

20

Page 21: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Relationship betweenTesting & the Curriculum

EXTERNAL Curriculum Evaluation(Standardized Tests)

Page 22: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Best Practices: Standardized Exams• Establish scoring benchmarks (evidence-based)• Determine weight of standardized exam in overall grade• Identify consequences for failure to achieve benchmarks

• Offer additional (parallel) versions of the exam – mix these up!• Allow enough time for remediation to occur between attempts.• Assign specific remediation activities (multiple strategies)

• Date/time for submission of these should be specified in the learning contract. Monitor remediation usage for student adherence to contract.

• Selections should be based on weak areas identified on scoring report.• Focus on building confidence – this is not a punitive activity. Model

effective study habits and test taking strategies.

• Use proctoring guidelines; take test security measures • Investigate suspected breaches and enforce published policies.

22

Page 23: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Teacher-made Tests and Standardized Tests: Vigilance with Test Security

23

1. Encourage moral behavior (Academic honesty program at your school with clear language placed in handbooks)

2. Discourage cheatinga. Before testing

1. Minimize access to exams and viewing of exam content2. Use highest levels of security available in learning mgmt.

systems for unit tests and all security features available in standardized testing platform - protect logins & access codes; proctors should initiate active dashboarding

3. Train proctors for all types of live proctoring activitiesb. During testing

1. Establish secure environment2. No deviations to test procedures or breakdown of

environmental security allowed. Ex: leaving room equates to the test being over for that student regardless of reason

3. Vigilant proctoring3. Detect cheating with Data Forensics and take actions as needed

Page 24: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Teacher-made Tests – Analyze/Document

24

Page 25: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Using an electronic test blueprint

25

Page 26: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation
Page 27: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Every test….Every test administrationConduct a Systematic Review of Teacher-made tests Each exam administered in the program should be evaluated and blueprinted using standard criteria agreed upon by the faculty; for example, level of item (knowledge/comprehension vs application/analysis) along with the percentage of alternative items. Look carefully at each test’s strengths & weaknesses•Consistent item analysis will be provided with all tests for better comparison and test improvement. •Faculty should complete the review on each test following administration, including item analysis data and any changes made based on the review that led to rescoring.

27

Page 28: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation
Page 29: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation
Page 30: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Teacher-made Test – Analysis results• Observations and Suggestions for NURS 101• Exam 2 and Final Exam were evaluated • Both exams lack predominance of questions written at the

application-and-above level on the cognitive taxonomy.

• Analysis of Leveling• K/C – Knowledge and Comprehension• A/A – Application and Analysis

30

Page 31: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Item Strength Evaluation: Nursing 101

31

Page 32: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Teacher-made Tests: Evaluation resultsIdentifying Testing Weaknesses•Review the item bank for consistency. If styles of items vary greatly from one exam to the next, edit items to conform to style guidelines adopted by the faculty.•Seeing the same problems with different tests used within a particular course? Authored by a particular faculty? •Consistently strive to increase the overall percentage of items written at the Application and Analysis levels of the cognitive taxonomy. Too often, the item bank predominantly reflects items written at the Knowledge and Comprehension levels.

32

Page 33: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Test Review Summary

33

Page 34: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Course Coordinator Report

34

Page 35: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Course Coordinator Report: Tests

35

Page 36: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Standardized Tests – Analyze/Document

36

Page 37: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Standardized Test Score Patterns

37

0

100

200

300

400

500

600

700

800

900

1000

1100

1 2 3 4 5 6 7 8 9

Sco

res

Student

Page 38: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

Licensure Exam Results - Analyze

38

Page 39: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

NCLEX® Test Plan Report: Client Needs

39

Page 40: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

A closer look at Client NeedsSafe and Effective Care Environment•Coordinated/Management of Care•Safety and Infection ControlHealth Promotion and MaintenancePsychosocial IntegrityPhysiological Integrity•Basic Care and Comfort•Pharmacological (Parenteral) Therapies•Reduction of Risk Potential•Physiological Adaptation

40

Page 41: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

NCLEX-RN® Client Needs – 2016 Changes

Percentage of Items from each category did not change from 2013 planSource: https://www.ncsbn.org/2013_NCLEX_RN_Test_Plan.pdf

Page 42: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

NCLEX-RN® 2016 Test Plan Details

42

Page 43: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

NCLEX® Test Plan Report: Performance

43

Page 44: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

NCLEX® Test Plan Report: Percentages

44

Page 45: Using Nursing Exam Data Effectively in Preparing Nursing Accreditation

References

Morrison, S., Nibert, A., & Flick, J. (2006). Critical thinking and test item writing (2nd ed.). Houston, TX: Health Education Systems, Inc.

National Council of State Boards of Nursing. (2016) 2016 NCLEX-RN test plan. Chicago, IL: National Council of State Boards of Nursing. https://www.ncsbn.org/RN_Test_Plan_2016_Final.pdf

Nibert, A. (2010) Benchmarking for student progression throughout a nursing program: Implications for students, faculty, and administrators. In Caputi, L. (Ed.), Teaching nursing: The art and science, 2nd ed. (Vol. 3). (pp.45-64). Chicago: College of DuPage Press.

Schroeder, J. (2013). Improving NCLEX-RN pass rates by implementing a testing policy. Journal of Professional Nursing, 29(2), S43-S47.

Sewell, J., Culpa-Bondal, F., & Colvin, M. (2008). Nursing program assessment and evaluation: Evidence based decision making improves outcomes. Nurse Educator, 33(3), 109-112.

45