Evaluating Learning Using Vignettes

Gillian MacIntyre Pam Green Lister

Glasgow School of Social Work

Introduction

Our use of vignettes was:

Based on pilot work in Scotland Evaluation of the social work

degree qualification in England

Why use vignettes?

Vignettes are used to test how students’ demonstrate the necessary skills, knowledge and values informed by research

The use of vignettes in previous research

Developing the vignettes

Two vignettes Represented children and families,

adult services Consultation with service user and

carer advisory group Students to provide written

answers at time one and time two

Development of the vignettes (2)

Vignettes were designed to “test” certain areas What are the most significant factors for

you in this situation? (knowledge) How might you explain what is going on in

this situation? (application of knowledge) What should happen now in this situation?

(Assessment/ analysis) As a social worker what would you do in

this situation? (Action/ Skills) Why would you do this? (linking theory to

practice)

Data Collection

Pilot response rates Administered to PG

students prior to going out on placement and again after placement

22 students, time one 20 students, time two 14 direct comparisons

at time one and time two

DH Evaluation response rates

Administered at start and end of programme

222 students, time one

131 students, time two

79 direct comparisons at time one and time two

Developing a Framework for Analysis of vignettes

Development of proforma and coding form

Refined using pilot data at a training day

Content of framework for analysis Scale (emphasised - not

mentioned) Extent to which the account is

descriptive, analytic or reflective

Criticality

Describing: answer focuses on stating what is happening and what practical tasks are necessary without giving any reason or undertaking them

Analytic: speculates on what is happening and why; no reason for their relevance is offered

Reflection: draws on theories and knowledge to explain what is happening; consequences of any intervention is discussed

Themes from vignette analysis

Attention to process of relationships Non-pathologising/ pathologising continuum Emphasis on helping Task/ service oriented Language Understanding social work role Use of theory Personal/ situation continuum Awareness of social/ structural/ political

issues Attention to risk and prioritising Added at time two - service user and carer

involvement

Changes in students’ approach to vignette one

Not at all Some evidence

Very

% Tim

e One

% Tim

e Two

% Tim

e One

% Time Two

% Tim

e One

% Time Two

N

Descriptive 8.1 12.2 48.6 64.9 43.2 23.0 74

Analytic 51.4 21.6 45.9 54.1 2.7 24.3 74

Reflective 77.0 50.0 23.0 28.4 0.0 21.6 74

Changes in approach to vignette one Time one:

33 % of answers mentioned social exclusion or normalisation

68% mentioned independence and ascertaining David’s wants

80% made no mention of establishing a relationship

Time two: 78% mentioned social exclusion or

normalisation Little change in mention of establishing a

relationship

Example of change between time one and time two Theme: emphasis on helping T1 - rush to resolve problems and

helping, less attention to wider context, less recognition of potential complexities

T2 - shift in students’ emphasis in recognising potential impact of learning disabilities, recognition of complexity of situations

Discussion

Evidence of change over time in development of skills

Movement from reactive helper to a more discerning, reflective practitioner

Shown by: more consideration to wider context, greater appreciation of implications of intervention, more awareness of policy and legislation

Reflections: Advantages

Enables research to be carried out with relatively large numbers of students

Allows students’ to reflect on their learning

Less intrusive than observing practice Thought-provoking exercise, cant get

enough practice, needs more time to do it justice

I found the task helpful for myself to give me a set time to think about the issues and to come up with ideas for intervention

Reflections: Challenges

Time consuming for students Attrition rates Inter-rater reliability Subjectivity and complexity in

terms of isolating and attributing factors that contribute to learning

Considerations for future research

Incorporation of self-evaluation

Service user and carer involvement

Further discussion in workshop this afternoon…!