chapter 18 ppt eval & testing 4e formatted 01.10 kg edits

© 2013 Springer Publishing Company, LLC. 1

Chapter 18Program Assessment

Oermann & GabersonEvaluation and Testing in Nursing Education4th edition


Program Assessment

♦ Process of judging the worth or value of an educational program

♦ Purposes – Provide data on which to base decisions about the

program– Provide evidence of educational effectiveness in

response to internal and external demands for accountability


Program Assessment Models

♦ Many models available♦ Three commonly used models in nursing

education programs– Accreditation– Decision oriented– Systems oriented


Accreditation Models

♦ Examples– NLNAC– CCNE– CASN– The Joint Commission

♦ Assess whether the program meets external standards of quality

♦ Use combination of internal and external assessment methods


Decision-Oriented Models

♦ Examples– CIPP model– CQI model

♦ Provide information to decision makers for program improvement purposes– No guarantee that corrective action will be taken if it is

indicated

♦ Usually focus on internal standards of quality, value, and efficacy


CIPP Model

♦ Context: What needs to be done? – Program needs, problems, strengths, weaknesses

♦ Input: How should it be done? – System capabilities, work plans, budgets

♦ Process: Is it being done? – Monitor implementation, make changes as needed

♦ Product: Did it succeed? – Impact, effectiveness, sustainability


CQI Model

♦ Based on assumption that those who are closest to a process know it best

♦ 4 phases– Plan– Do – Study– Act


CQI Model

♦ Plan– Define problems and opportunities for

improvement– Collect and organize data– Decide on improvement initiatives

♦ Do– Identify needed resources– Implement planned initiatives

(cont’d)


CQI Model (cont’d)

♦ Study– Monitor progress of improvement initiative– Observe the effects of the change

♦ Act– Modify improvement as needed – Incorporate change into program structure and

policies– Educate stakeholders about the change


Systems-Oriented Models♦ Assessment of inputs into the program – Characteristics of students, teachers, administrators, other

participants – Program resources

♦ Assessment of program processes, operations– Program implementation – Context within which the program is implemented

♦ Assessment of program outcomes – Intended outcomes being achieved– Stakeholder satisfaction– Program quality and cost-effectiveness


Stakeholders

♦ Judge worth or value of an educational program– Affected directly or indirectly by the decisions

made – Key stakeholders of nursing education programs • Students• Faculty and staff members• Partners (health care and community agencies)• Consumers


Stakeholders ♦ Assessment purpose determines stakeholders to involve

– Formative assessment (program improvement)• Students• Teachers• Administrators

– Summative assessment (whether program should be continued, revised, closed) • Program participants• Graduates and their employers• Prospective students• Health care and community agencies• Consumers• Legislative and regulatory bodies


External vs. Internal Evaluators

♦ Some program assessment models use both♦ External evaluator– May be more objective– May not know program or its context well enough – Additional expense

♦ Internal evaluator – Better understanding of program operations and context – Can provide ongoing feedback– May be biased


Program Outcome Assessment Tools♦ Teacher-made or standardized?♦ Must possess qualities of effective

measurement instruments♦ Teachers must be able to make valid and

reliable interpretations of standardized test results

♦ Should be chosen based on the outcomes to be measured


Limitations of Outcomes Assessment♦ Some intended outcomes may not be important or

related to stakeholders’ needs♦ May provide feedback only at the end of a program ♦ To contribute to program improvement, program

assessment also should examine– program goals– structure– process – “side effects”


Curriculum Assessment

♦ Focus is more narrow than program assessment– Focus on elements central to the course of studies

taken by students– Program assessment includes additional elements• Institutional support for the program• Administrative structure• Faculty qualifications and productivity• Student support services


Curriculum Assessment

♦ Focus of curriculum assessment has changed over time

♦ Classification of assessment models – First generation– Second generation– Third generation– Fourth generation


Focus of Curriculum Assessment

♦ First-generation model– Measure to obtain student scores representing curriculum quality

♦ Second-generation model– Describe congruence between student performance and curriculum

objectives

♦ Third-generation model– Judge the merit of performance in relation to standards

♦ Fourth-generation model– Develop holistic understanding of needed improvements based on

input from stakeholders


Curriculum Assessment Elements

♦ Curriculum outcome statements– Congruent with professional and institutional standards? – Appropriate to program level?– Relevant to healthcare context?

♦ Curriculum design– Good fit of components? – Reflect curriculum foundation and goals? – Congruent with program environment? – Courses logically sequenced?

(cont’d)


Curriculum Assessment Elements (cont’d)

♦ Curriculum outcomes– Evidence of program outcome achievement? – Success of program graduates?

♦ Courses– Course goals congruent with curriculum goals?– Learning activities consistent with the philosophical

framework and goals? – Course content current, evidence-based, logically organized?

(cont’d)



♦ Teaching-learning methods– Congruent with philosophical framework?– Help students to achieve outcomes? – Respect student diversity? – Student satisfaction with the quality of teaching?

(cont’d)



♦ Student assessment methods– Assess all relevant types of learning? – Accommodate student input, diversity, feedback

needs? – Accommodate faculty expertise, preferences,

workload?

(cont’d)



♦ Resources (human, physical, fiscal)– Sufficient numbers of appropriately credentialed faculty

members? – Adequate number and type of staff members to support

the curriculum? – Classrooms, offices, labs—adequate, comfortable,

appropriately equipped? – Library holdings appropriate and sufficient? – Sufficient quality and quantity of clinical placements?

(cont’d)



♦ Learning climate– Effect of social, emotional, intellectual atmosphere on

faculty, student, staff quality of life? – Satisfaction of faculty and students with relationships,

academic freedom, sense of community? – Promote diversity of perspective and foster

responsibility and accountability? (cont’d)



♦ Policies– Appropriate, reasonable, applied consistently? – Well understood by faculty and students? – Need for additional policies?


Assessment of Online Courses

♦ One element of curriculum assessment ♦ Involves many of the same criteria used to assess

courses offered in traditional classrooms♦ Additional elements specific to online environment – Technology– Accessibility– Instructional design– Content– Interactive activities


Assessment of Online Courses

♦ Challenges– Online courses hidden from view of faculty

members and administrators who are not directly involved in teaching them• Limits the role that these colleagues can play in course

evaluation• Difficult for outside reviewer to assess elements of

asynchronous learning– Pace of the learning activities – Timing of instructor feedback to students


Assessment of Teaching

♦ Focuses on quality of classroom and clinical teaching ♦ Also focuses on other dimensions of the faculty role– Depend on goals and mission of the nursing education

program– Examples

• Scholarship and research• Service to the academic institution, nursing program, community,

nursing profession• Clinical practice



♦ Research in nursing education: Qualities of effective teaching in nursing– Knowledge of the subject matter– Clinical competence– Teaching skill– Interpersonal relationships with students– Personal characteristics

♦ Consistent with studies about teacher effectiveness in other fields



♦ NLN’s "Core Competencies of Nurse Educators with Task Statements" (2005)– Knowledge, skills, and attitudes required for

effectiveness in the role of nurse educator– Each competency statement followed by

description of tasks that exemplify it– Three of the core competencies focus on effective

performance in the role of teacher


Core Competencies of Nurse Educator1. Facilitate learning– Uses variety of teaching strategies– Communicates effectively– Shows enthusiasm for teaching and nursing– Demonstrates interest in learners– Demonstrates personal attributes that facilitate

learning– Maintains professional practice knowledge base


Core Competencies of Nurse Educator2. Facilitate learner development and

socialization– Identifies individual learning needs and styles– Provides appropriate learning resources– Creates learning environments that promote

socialization to the role of nurse


Core Competencies of Nurse Educator 3. Use assessment and evaluation strategies– Uses a variety of methods to assess learning– Gives timely and constructive feedback to learners– Demonstrates skill in design and use of

assessment tools


Qualities of Effective Teaching

♦ Knowledge of subject matter– Content area expert– Understands theories and concepts relevant to

nursing practice• Can assist students to apply to patient care

– Keeps current with nursing and other interventions, new developments in areas of expertise, and research

– Necessary but not sufficient; must be able to communicate knowledge to students



♦ Clinical competence– If teaching in the clinical setting – Make sound clinical judgments– Expert clinical skills– Can guide students in skill development



♦ Teaching skill– Identify students’ learning needs– Plan instruction– Present material effectively– Explain concepts and ideas clearly– Demonstrate procedures effectively– Use sound assessment practices• Especially important in clinical evaluation



♦ Positive relationships with learners– Shows respect for and confidence in students– Is honest and direct– Supports students– Is approachable– Treats students fairly– Creates climate of mutual respect between

teacher and student



♦ Personal characteristics– Enthusiasm– Patience– Sense of humor– Friendliness– Integrity– Perseverance– Courage– Willingness to admit mistakes


Teaching Effectiveness Data Sources ♦ Student ratings– Students can make valid and reliable judgments about

• teaching methods.• Fairness.• interest in students.• enthusiasm for the subject.

– Class size, course format can affect ratings


Teaching Effectiveness Data Sources♦ Student ratings – Necessary but insufficient data source • Students not the best source of data about accuracy,

depth, and scope of the teacher’s knowledge

– Can be obtained in classroom or online• Assure anonymity and confidentiality


Teaching Effectiveness Data Sources♦ Peer review– Perspective of another teacher who knows the content

and who has experience working with similar students – Best used for formative evaluation

• Reviewer may not be a content or clinical expert in same area as teacher• Observations can be influenced by personal feelings about

the colleague(cont’d)


Teaching Effectiveness Data Sources♦ Peer review (cont’d)– Should take place within a context of continuous

improvement of teaching–learning process – Must be supported by adequate resources • Faculty development• Mentoring• Modeling by master teachers

– Observation in classroom, clinical setting, laboratory(cont’d)


Teaching Effectiveness Data Sources♦ Peer review (cont’d)– Online environment• Visit as guest user, review course materials• Evidence of effective instruction

– Promotes interaction among students and teacher, active learning, student engagement

– Communicates appropriate performance expectations; gives prompt, rich feedback

– Respects diversity of views, learning styles, abilities – Uses effective instructional and graphic design


Teaching Portfolio

♦ Collection of materials related to teaching activities in classroom, clinical practice, online environment, simulation center, or other settings

♦ Materials selected by the faculty member should be highly selective and organized to create a cohesive professional narrative

♦ Variety of formats


Teaching Portfolio

♦ May reflect specific purpose– Teaching improvement (formative use)– Promotion and tenure review (summative use)– Seeking faculty position (career portfolio)

♦ Variety of formats– Print– Electronic


Teaching Portfolio

♦ Contents– Teaching materials

• e.g., syllabi, sample tests, student assignments

– Philosophy of teaching• Should be reflected in the teaching material

– Additional content for instructional improvement• Areas of teaching that need improvement • Description of efforts to develop teaching skills

– Peer and administrator evaluations of teaching– Self-evaluation of strengths and weaknesses

chapter 18 ppt eval & testing 4e formatted 01.10 kg edits

Documents