![Page 1: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/1.jpg)
Assessing Practice: OSCE Adapted for Social Work
Marion Bogo
King’s College, London May 2015
![Page 2: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/2.jpg)
Holistic Competence
Meta-competencies
Procedural competencies
![Page 3: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/3.jpg)
Meta-competencies Matter
Meta-competencies are higher order, overarching qualities, abilities and capacities:
• Cognitive • Affective • Reflective
Related to ability to use
Procedural competencies – operational, behavioral techniques and performance skills needed to carry out professional tasks. (Bogo et al., 2006; 2010; 2013; 2014).
![Page 4: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/4.jpg)
A Model of Holistic Competence in Social Work
Skills
Self-regulation Emotions, reflection,
self-awareness
Knowledge Generic and specialist
Theoretical and empirical
Judgment Assumptions, critical thinking,
decision making
Complex Practice Behavior
PROFESSIONAL CONTEXT – VALUES (Bogo, Rawlings, Katz, & Logie, 2014)
ORGANIZATION AND COMMUNITY CONTEXT
![Page 5: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/5.jpg)
The OSCE Adapted for Social Work
Simulated Interview
Assess Procedural Competencies
-Relationship building
-Assessment
-Interviewing
Scenarios depicting social work situations
Trained actors simulating clients
Rated by trained instructors
Reflection
Assess Meta-competencies
-Conceptualize practice
Assess/Judgment
-Relationship/Use of self
-Learn from reflecting on their practice
Questions
Reflective dialogue
Written reflections
Course assignment
![Page 6: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/6.jpg)
What are OSCE’s
• An Objective Structured Clinical Exam • 15 minute interview with a standardized client directly
evaluated by an evaluator using a rating scale. • Standardized Client: An actor or actress trained to
enact the role of a client scenario consistently each time.
• Objective: Skills directly observed • Structured: A set scenario and format • Clinical: Assesses practice skill implementation. • Exam: Performance is evaluated on a set scale.
![Page 7: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/7.jpg)
Post Interview Reflection
• Traditional OSCE only measures behaviors • A post-interview reflection with standardized
questions taps into meta-competencies: – what and how students are thinking and feeling
about their practice • Link theory to practice, critical thinking, judgment,
decision making • Subjective experience • Self assessment and learning
![Page 8: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/8.jpg)
Rationale: Why OSCE’s
• Limitations of self-report and field evaluations for assessing direct practice skill.
• Need for reliable and valid evaluation methods in social work that directly assess skill.
• Post interview reflection questions demonstrate students’ ability to: – Accurately recall their practice – Use theory in practice – Think critically about the way they make decisions,
use of self in practice.
![Page 9: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/9.jpg)
OSCE in Health Professions
• Extensive research literature exists • Method has acceptable validity and reliability • Accepted by students and faculty members, as
fair and authentic • Frequently used for episodic performance-based
assessment in undergraduate medical education, specialty areas, and in licensing examinations in some countries.
(Hodges, 2006)
![Page 10: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/10.jpg)
OSCE in Social Work Critical Appraisal
• Found 14 research studies reporting intervention and observational data
• Used standardized clients/actors; trained actors, faculty or drama students.
• Educational interventions included individual, group or classroom interactions with a SC.
(Logie, Bogo, Regehr & Regehr (2013).
![Page 11: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/11.jpg)
Limitations and Needs
• Need • replication – build on each other’s work
• standardized, reliable and valid tools
• studies focusing on evaluation of performance rather than satisfaction
• include diversity and cultural competence
![Page 12: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/12.jpg)
Toronto Team Studies
• Reliability and validity of OSCE adapted method of performance and reflection, scales, scenarios
• Variation in student scores: – On performance – On reflection
• (Bogo et al., 2011; 2012; 2013)
![Page 13: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/13.jpg)
Variation OSCE Performance
![Page 14: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/14.jpg)
Variation OSCE Reflection
![Page 15: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/15.jpg)
Toronto Team Studies
• OSCE correlated with field performance – to some extent (Bogo et al., 2012)
• Reflective dialogue highlighted needs that informed curriculum design.
![Page 16: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/16.jpg)
Extremely well received by students
– Fair method for assessing competence
– Contributes to learning through practice
– More confident re: entering practicum
![Page 17: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/17.jpg)
Azusa Pacific Studies: Summary
• Reliability – Internal Consistency strong for 5 point scale and
strong inter-rater reliability. – Overall item is strongly correlated between raters,
and strongest predictor of field performance.
• Validity – Potential for predicting field. – Good correlation with EPAS items, however, EPAS not
predictive of field.
• Rater bias risk and considerations.
![Page 18: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/18.jpg)
The Process of Developing an OSCE
Identify competencies and practice behaviors
Map competencies, issues and content for
scenarios
Design scenarios
Construct measures
![Page 19: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/19.jpg)
The Process of Developing an OSCE • Conceptualizing competence
What practice behaviors and meta-competencies do you want to measure
• Mapping competence Identify issues and abilities in potential scenarios
• Designing scenarios Practice situations with material for students to demonstrate competence
• Defining specific behaviors for a rating scale What should the student be able to do?
Conclusion… an iterative process….
![Page 20: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/20.jpg)
Challenges to Conceptualization and Definition of Competence
• Context: Diverse settings and a range of practice approaches - general descriptions
• To assess practice behaviors need a level of specificity • Reduction to numerous discrete items - as if practice
were mechanistic • Need definitions and methods that capture the holistic
nature of practice: what type of knowledge is used and how it is used (internal cognitive and emotional processes that contribute to decision making and performance)
![Page 21: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/21.jpg)
Selected References • Bogo, M., Rawlings, M., Katz, E., & Logie, C. (2014). Using Simulation in
Assessment and Teaching: OSCE Adapted for Social Work (Objective Structured Clinical Examination). CSWE: Alexandria, VI.
• Bogo, M., Regehr, C., Woodford, M., Hughes, J., Power, R., & Regehr, G. (2006).
Beyond competencies: Field instructors' descriptions of student performance. Journal of Social Work Education, 42(3), 191-205.
• Bogo, M., Katz, E., Regehr, C., Logie, C., Mylopoulos, M., & Tufford, L. (2013). Toward understanding meta-competence: An analysis of students’ reflection on their simulated interviews. Social Work Education 32(2): 259-273. DOI: 10.1080/02615479.2012.738662
• Bogo, M., Shlonsky, A., Lee. B., & Serbinski, S. (2014). Acting like it matters: A scoping review of simulation in child welfare training. Journal of Public Child Welfare 8(1)70-93. DOI: 10.1080/15548732.2013.818610
• Bogo, M., Regehr, C., Katz, E., Logie, C., Tufford, L., & Litvack, A. (2012). Evaluating the use of an objective structured clinical examination (OSCE) adapted for social work. Research on Social Work Practice. 22(4), 428 - 436. DOI: 10.1177/1049731512437557
![Page 22: Assessing Practice: OSCE Adapted for Social Work](https://reader033.vdocument.in/reader033/viewer/2022051521/5852a7351a28abfa398dde3f/html5/thumbnails/22.jpg)
Selected References
• Bogo, M., Regehr, C., Logie, C., Katz, E., Mylopoulos, M., & Regehr, G. (2011). Adapting objective structured clinical examinations to assess social work students’ performance and reflections. Journal of Social Work Education (47)1, 5-18.
• Katz, E., Tufford, L., Bogo, M., & Regehr, C. (2014online). Illuminating students’ pre-practicum conceptual and emotional states: Implications for field education. Journal of Teaching in Social Work 34, 96-108.
• Logie, C., Bogo, M., & Katz, E. (2015online). “I didn’t feel equipped: Social work students’ reflections on a simulated client ‘coming out’. Journal of Social Work Education.
• Logie, C., Bogo, M., Regehr, C., & Regehr, G. (2013). A critical appraisal of the use of standardized client simulations in social work education. Journal of Social Work Education, 49(1): 66-80. DOI: 10.1080/10437797.2013.755377
• Tufford, L., Bogo, M., & Asakura, K. (2015). How do social workers respond to potential child neglect? Social Work Education 34(2): 229-243.DOI: 10.1080/02615479.2014.958985