measurung clinical competennce: problems and solutions
TRANSCRIPT
Measuring clinical competence:
Problems and solutions
Roger Watson
Faculty of Health Sciences
University of Hull, UK
23/08/2017
• Background
• Issues
• Measurement
• Research
23/08/2017
BACKGROUND
A North American (USA) concept
• An alternative to intelligence testing
• Aimed at manual and not professional occupations
23/08/2017
ISSUES
• What is the definition of competence?
• How does it relate to other similar concepts:
– Capability?– Expertise?– Performance?
• How is competence measured?
23/08/2017
23/08/2017
COMPETENCE IS NOT ‘VALUE NEUTRAL’
‘…competence in practical matters must be preserved against the encroachment of the intellectuals’ (Eraut 1994)
• ‘…competence may be preferred to excellence if it result(s) in quicker and cheaper service’ (Eraut 1994)
23/08/2017
WHAT IS THE DEFINITION OF COMPETENCE?
• There is no consistent definition of competence
• Common words:
– Level– Agreed– Standard– Performance– Competency– Ability– Capability
23/08/2017
HOW DOES IT RELATE TO OTHER SIMILAR CONCEPTS?
23/08/2017
COMPETENCE
COMPETENCE AND RELATED CONCEPTS
competency
capability
23/08/2017
COMPETENCYcompetence
capability
COMPETENCE AND RELATED CONCEPTS
23/08/2017
CAPABILITYcompetence
competency
COMPETENCE AND RELATED CONCEPTS
23/08/2017
knowledge skills
attitude
23/08/2017
knowledgeskills
attitude
23/08/2017
knowledge skills
attitude
COMPETENCE
23/08/2017
knowledgeskills
attitude
COMPETENCE
23/08/2017
MEASUREMENT
• If we cannot define competence…how can we measure it?
• If we can define it…what do we measure?
• If we can measure it…how do we measure it?
23/08/2017
PROBLEMS WITH MEASUREMENT
• WHAT?
• WHY?
• WHERE?
• WHEN?
• WHO?
• HOW?
23/08/2017
PROBLEMS WITH MEASUREMENT
• WHAT?
– Performance– Knowledge– Skills– Attitudes
23/08/2017
PROBLEMS WITH THE ‘WHAT’ OF MEASUREMENT
• CONTENT
– Authenticity– Directness
Samuel Messick (1994) The Interplay of Evidence and Consequences in the Validation of Performance Assessments Educational Researcher 23, 13-23
23/08/2017
PROBLEMS WITH MEASUREMENT
• WHY?
– …do we want to know that?*
• * = knowledge, skills, attitude
23/08/2017
PROBLEMS WITH MEASUREMENT
• WHERE?
– Classroom– Hospital– Both
23/08/2017
PROBLEMS WITH THE ‘WHERE’ OF MEASUREMENT
• Classroom
– Not ‘real world’
• Hospital
– Risk to patients
• Both
– Ideal but time-consuming
23/08/2017
PROBLEMS WITH MEASUREMENT
• WHEN?
– …and how often?
• At the start
• In the middle
• At the end…of placements; years, programmes, units?
23/08/2017
PROBLEMS WITH THE ‘WHEN’ OF MEASUREMENT
• Under-assessment
• Over-assessment
– ie ‘getting it right’
23/08/2017
PROBLEMS WITH MEASUREMENT
• WHO?
– The student– The teacher– The clinician– The patient
23/08/2017
PROBLEMS WITH THE ‘WHO’ OF MEASUREMENT
• The student
– Not objective (but students tend to be hard on themselves)
• The teacher
– Not familiar with the clinical area
• The clinician
– Socialisation processes at work
• The patient
– Find it hard to be honest
23/08/2017
PROBLEMS WITH MEASUREMENT
• HOW?
– Self-assessment– Observation– Questionnaires/inventories– Qualitatively– Examination
23/08/2017
PROBLEMS WITH THE ‘HOW’ OF MEASUREMENT
• PSYCHOMETRICS
– Reliability– Validity
23/08/2017
RELIABILTY
• Test-retest
• Intra-rater
• Inter-rater
23/08/2017
VALIDITY
• Content (including Face)
• Criterion (Concurrent & Predictive)
• Construct (Convergent/Divergent, Discriminant, Factorial)
23/08/2017
ECOLOGICAL VALIDITY
• Ecological Validity is the degree to which the behaviours observed and recorded in a study reflect the behaviours that actually occur in natural settings.
23/08/2017
IN ANY CASE – WHAT ARE WE GOING TO DO WITH THE
INFORMATION?
Students rarely fail on the grounds of incompetence
FACT!
23/08/2017
23/08/2017
A clear finding from this study is that no
single method is appropriate for assessing
clinical competence. A multimethod
UK-wide strategy for clinical competence
assessment for nursing and midwifery is
needed if we are to be sure that assessment
reveals whether or not students have
achieved the complex repertoire of
knowledge, skills and attitudes required for
competent practice.
23/08/2017
23/08/2017
A lack of consistency in the training of
student assessors in the clinical areas
was identified. Some of these issues may
be resolved with the development of a
national instrument for competence
assessment.
23/08/2017
23/08/2017
The conclusion is that a multimethod
approach enhances validity and ensures
comprehensive assessment of the complex
repertoire of skills required of students in
nursing.
23/08/2017
23/08/2017
23/08/2017
Three categories of paper emerged:
• Instrument development and testing (n=4)
• Approaches to testing competence (n=7)
• Assessment and related factors (n=12).
23/08/2017
23/08/2017
CONCLUSIONS ON COMPETENCE MEASUREMENT
• Competence measurement is problematic
• Competence measurement is controversial
• Very few good quality studies
• Competence inventories tend to be:
– Long– Multidimensional
23/08/2017
SOLUTIONS TO COMPETENCE MEASUREMENT
• Derive professional NOT political competencies
• Agree national/international standards
• Develop psychometrically sound instruments
• Conduct longitudinal studies of utility and efficacy
Capability
The concept of capability
Capability can be observed when we see people with justified confidence in their ability to:
• Take effective and appropriate action
• Explain what they are about
• Live and work effectively with others; and
• Continue to learn from their experiences as individuals and in association with others, in a diverse and changing society.
• Stephenson (1992 in Stephenson & Yorke 1998; p2)
Capability and competence
‘Capability includes but goes beyond the achievement of competence in present day situations to imagining the future and contributing to making it happen.’
Stephenson (1998; p3)
Unfamiliar
Problems
Familiar
Problems
Unfamiliar
Contexts
Familiar
Context
Unfamiliar
Problems
Familiar
Problems
Unfamiliar
Contexts
Familiar
Context
Y
Z
Competence
Capability
Unfamiliar
Problems
Familiar
Problems
Unfamiliar
Contexts
Familiar
Context
Y
Z
Higher
education
23/08/2017
52
