chapter 14 ppt eval & testing 4e formatted 01.10 mo checked

© 2013 Springer Publishing Company, LLC.

Chapter 14Clinical EvaluationMethodsOermann & GabersonEvaluation and Testing in Nursing Education4th edition


Steps in Clinical Evaluation Process

♦ Decide on purpose of evaluation–why are you evaluating student?

♦ Formative or summative?♦ Establish standards for evaluation.– Letter grade or P/F?

♦ Select evaluator. – Faculty? Preceptor? Self? Multiple?

2


Steps in Clinical Evaluation Process

♦ Decide on evaluation method based on competencies to be achieved– Use a variety of appropriate methods

♦ Collect and interpret data♦ Use results to make decisions about students♦ Evaluate your process—need for changes?

3


Selecting Clinical Evaluation Methods ♦ Select based on clinical competencies♦ Vary the methods♦ Select realistic evaluation methods to use♦ Decide if formative or summative♦ Review purpose and number required of each

assignment ♦ Consider your time

4


Clinical Evaluation Methods

• Observation• Notes about

performance• Checklists• Rating scales• Simulations

• Standardized patients• Objective Structured

Clinical Evaluation• Media clips • Written assignments• Journal

5


Clinical Evaluation Methods (cont’d)

• Nursing care plans• Concept maps• Case method,

unfolding cases, case study

• Papers• Portfolios• Conferences• Group projects• Self-assessment

6


Observation

♦ Predominant method of assessing clinical performance

♦ Should be guided by clinical outcomes or competencies– Help the teacher focus the observation

7


Observation: Threats to Validityand Reliability♦ Teacher’s values and biases may influence:– What is observed– Inferences and judgments about quality of

performance♦ Overreliance on first impressions– Should observe over time and in different situations

before drawing conclusions

8


Observation: Threats to Validityand Reliability (cont’d)♦ Focus of observation may differ with each

instructor and time period♦ Teacher may make incorrect judgments

about the observation– Discuss observations with students, obtain

their perceptions, and modify inferences when indicated

9


Observation: Threats to Validityand Reliability (cont’d)♦ Every observation in the clinical setting is only

a sample of the student’s performance– Observation of the same student at another time

may reveal a different level of performance – Same is true for observations of teacher

performance

10


Documenting Observations

♦ Methods for recording observations and their context– Notes about performance – Checklists – Rating scales

11


Notes About Performance

♦ Descriptions of observed student performance– May include a judgment about the quality of

the performance♦ Should be recorded as soon after the

observation as possible ♦ Methods of recording– Handwitten notes, flow sheets, narratives– Using technology such as tablet computer or iPad

12


Notes About Performance (cont’d)

♦ Share with students frequently to give them feedback about their performance– Incorporate student input when indicated

♦ Use conferences with students to review pattern of performance over time

♦ Serve as documentation for ratings on the clinical evaluation tool

13


Checklist

♦ List of specific behaviors or activities to be observed and means of indicating whether or not they were present

♦ Often lists steps of a procedure or skill♦ Facilitate teacher’s observation of

performance♦ Facilitate learners’ self-assessment prior to

assessment by the teacher

14


Developing a Checklist

♦ List steps or actions in order– Focus on critical items and sequencing

♦ List errors that learners often make – Alerts the assessor to observe for these

♦ Develop into a form to check steps or behaviors as they are performed– Allow flexibility of sequencing if multiple ways of

performing a procedure safely

15


Rating Scales

♦ Also called clinical evaluation tools or instruments

♦ Means of recording judgments about observed performance of learners

♦ Two parts– List of outcomes or competencies learner is to

demonstrate in clinical practice– Scale for rating students’ performance of them

16


Rating Scales (cont’d)

♦ Most useful for summative evaluation of performance

♦ Also may be used to evaluate specific clinical activities – e.g., student’s presentation of a case in

postclinical conference♦ Useful for giving specific feedback to students

about their performance

17


Types of Rating Scales

♦ Two-point scales– Pass-fail– Satisfactory-unsatisfactory

♦ Multiple-level scales– Letters (e.g., A, B, C, D, F)– Numbers (e.g., 5, 4, 3, 2, 1)

18


Types of Rating Scales

♦ Multiple-level scales– Qualitative labels (e.g., “excellent,” “very good,”

“good,” “fair,” “poor”)– Frequency labels (e.g., “always,” “often,”

“sometimes,” “never”)♦ Matrix combining different qualities of the

performance

19


Issues With Rating Scales

♦ Interrater consistency (reliability)– Do all users agree on meaning of ratings?• e.g., difference between “good” and “excellent”

– Problem even when using descriptors with letters, numbers, or labels used to rate outcomes or competencies

20


Issues With Rating Scales (cont’d)

♦ Scales based on frequency labels– e.g., “always,” “sometimes”– Limited opportunities for students to practice and

demonstrate a high level of some skills– How to rate a student who has had only one

opportunity to practice a skill?

21


Common Errors With Rating Scales

♦ Leniency error – Tendency to rate all students toward the high end of

the scale

♦ Severity error – Tendency to rate all students toward the low end of

the scale

♦ Central tendency error– Hesitancy to mark either end of the rating scale– Tendency to use only the midpoint of the scale

22


Common Errors With Rating Scales (cont’d)♦ Halo effect– Basing judgment on a general impression of the student

♦ Personal bias– Teacher’s personal values or preferences influence ratings

♦ Logical error– Giving similar ratings to items that are logically related to

one another

23


Common Errors With Rating Scales (cont’d)♦ Rater drift – Definition or interpretation of competencies to be

observed and assessed changes over time♦ Reliability decay – Over time, teachers may become less consistent in

their ratings

24


Improving Use of Rating Scales

♦ Teachers should have regular discussion of outcomes or competencies to be rated– Meaning of each outcome or competency– Describe what student performance would look

like at each rating level– May use simulation to facilitate discussion• Observe a performance, assess it with the rating scale,

and discuss rationale for the ratings

25


Clinical Evaluation Tools

♦ Same tool for all courses or course-specific tool?– Most faculties use one tool for all courses• Competencies adapted to each course

♦ Two-level or multilevel scales?– Most faculties use pass-fail or satisfactory-

unsatisfactory rating scales

26


Simulation

♦ Activity that allows learners to experience a clinical situation without the risks and constraints of a real-life situation

♦ Does not replace actual experiences with patients

♦ Useful for clinical evaluation as well as instruction

27


Use of Simulation for Clinical Evaluation♦ Case scenarios that students analyze♦ Computer simulations♦ Models and manikins♦ Standardized patients♦ Human patient simulators

28


Incorporating Simulation Into Evaluation Protocols♦ Identify clinical outcomes to be assessed with

simulation♦ Identify types of simulations needed to assess the

designated outcomes♦ Determine if simulations are available or need to be

developed by the faculty♦ Determine if simulation is for formative or

summative evaluation or both

29


Incorporating Simulation Into Evaluation Protocols (cont’d)♦ Determine need to develop or obtain

checklists or other methods for rating performance

♦ Decide when the simulations will be used in the course

♦ Decide how the effectiveness of the simulations will be evaluated

30


Standardized Patients

♦ Individuals trained to accurately portray the role of a patient with a specific diagnosis or condition

♦ Provides consistency in performance evaluation– Recreate the same patient condition and clinical situation

with each student

♦ Provide written and oral feedback to the students on their performance

31


Objective Structured Clinical Examination (OSCE)♦ Students rotate through a series of stations♦ At each station they may complete an activity

or perform a task♦ Performance evaluated with checklists or

rating scales♦ Usually used for summative evaluation

32


Games

♦ Involve competition, rules (structure), and collaboration

♦ Individual games and games played against other students either individually or in teams

♦ Many require props or equipment♦ Not appropriate for grading; use only for

instruction and formative evaluation

33


Media Clips

♦ Short segments of a digital recording, DVD, video from YouTube, other forms of multimedia

♦ May be viewed by students as a basis for:– Discussions (e.g., postclinical conference, online discussion

boards)– Small-group activities– Critique in a written assignment

♦ Students can visualize the patient and context

34


Written Assignments

♦ Use: Assess problem solving, higher level thinking, understanding content relevant to clinical practice

♦ Misuse: Students complete the same assignments repetitively throughout a course after outcomes have been met

35


Written Assignments: Journals

♦ Opportunity for students to describe and reflect on events and experiences in clinical practice

♦ Means of engaging in dialogue with the teacher♦ Print or electronic format♦ Students need to be clear about the purpose of the

journal activity

36


Written Assignments: Journals (cont’d)♦ Written guidelines for journal entries– How many entries?– What types of entries? – How frequently?

♦ Teacher gives prompt and meaningful feedback♦ Decide if graded or ungraded– Grading inappropriate if journal is used to express feelings,

values, beliefs

37


Written Assignments:Nursing Care Plans♦ Learn components of the nursing process ♦ Weaknesses when used for clinical evaluation– Linear process—does not facilitate assessment of

complex thinking necessary for clinical practice– Students often paraphrase from textbook– Teachers should be cautious about the number of

care plans required in a course

38


Written Assignments: Concept Maps♦ Tool for visually showing relationships among

concepts ♦ Help students organize knowledge as they

plan for clinical activities– Can “see” how aspects of care relate

♦ Best used for formative evaluation – Can be graded with criteria established for

evaluation

39


Written Assignments: Case Method, Unfolding Cases, and Case Study♦ Methods for assessing higher-level learning ♦ Individual or group activities♦ Useful for formative evaluation and student

self-assessment♦ Short cases, unfolding cases, and case studies

can be graded – Establish scoring criteria for responses to the

questions with the case

40


Written Assignments: Papers

♦ Short papers about clinical practice– Useful for assessing higher level thinking and

other cognitive skills – Often better than long papers in which students

may summarize textbook and other literature– Can be assessed formatively or summatively

41


Written Assignments: Papers (cont’d)♦ Term papers– Critique and synthesize relevant literature and

relate it to patient care – Assess drafts formatively to improve writing skills– Grade final product

42


Portfolio

♦ Collections of products that document student achievement– Demonstrate clinical competencies– Show work completed over period of time

♦ Types—can be combined for clinical evaluation– Best work (graded)– Growth and development (formative)

43


Using Portfolios for Clinical Evaluation♦ Identify purpose of the portfolio– Best work or growth and learning– Formative, summative, or both

♦ Identify the type of entries and content to be included in the portfolio– Content to be chosen by student, faculty member or both?

♦ Decide how portfolio will be evaluated– Individual entries and the portfolio overall– Scoring rubric?

44


Electronic Portfolio

♦ Developed and stored electronically– Facilitates updating and revision

♦ Can be easily sent to others for feedback or scoring

♦ Limited or no cost ♦ Can be modified for job search♦ Can include a variety of multimedia

45


Conferences

♦ Method for evaluating competency in oral communication skills

♦ Can be used to assess higher-level thinking and problem solving

♦ Can be used for formative or summative evaluation or both– If summative, need specific criteria and

scoring rubric

46


Conferences

♦ Types of conferences– Preclinical– Postclinical– Interprofessional

♦ Format– Face-to-face– Online

47


Criteria for Evaluating Conferences

♦ Ability of students to:– Present ideas clearly and logically– Participate actively in group discussion– Offer relevant ideas – Demonstrate knowledge of content – Offer different perspectives or share reflections – Assume a leadership role, if relevant, in group

process

48


Group Projects

♦ Short- or long-term work♦ Can assess products developed by the group

and ability of students to work cooperatively♦ Rubrics should be used for assessing– Should be geared to the specific project – Should reflect the goals of group work • Students can assess participation and contributions

of peers

49


Grading Group Projects

♦ Group grade– Each member of the group receives the same grade– Does not account for individual student effort and

contribution to the group product♦ Combination of individual and group grades– Students identify their contributions to the overall

product; teacher or self-assessment– Students prepare individual and group projects

50


Self-assessment

♦ Ability of students to evaluate their own clinical competencies and identify need for further learning

♦ Difficult for some students to assess their own performance– Need supportive environment, guidance, practice

♦ Appropriate only for formative evaluation

51


Clinical Evaluation inDistance Education♦ Preceptors may observe and evaluate individual

students♦ Adjunct or part-time faculty members may evaluate a

group of students at remote site♦ Intensive on-campus or regional site skill acquisition

workshop♦ Students may independently demonstrate clinical

skills and procedures in simulation setting– Record performance for evaluation by teacher

52


Clinical Evaluation inDistance Education (cont’d)♦ Critical decision: Which clinical skills and

competencies need to be observed and the performance rated?– Determines evaluation methods to use

♦ Should use consistent methods across all clinical sites♦ Need to orient all evaluators of student performance– Promotes reliability

53

chapter 14 ppt eval & testing 4e formatted 01.10 mo checked

Documents