Workshop: Evaluation Framework for Assistive Technology pilots 13 November 2019

Lead boroughs: Newham, Camden and Hackney

London Office of Technology & Innovation

LOTI_LDNmedium.com/loti

#LOTI

Welcome and Introductions

10:00

Priya Javeri

1. To learn from boroughs’ experiences of evaluating previous AT trials

2. To create a template evaluation framework for ATs

Objectives for today:

10:05

Agenda

9:30 Arrival

10:00 Welcome and Introductions - Priya

10:05 Objectives - Priya

10:10 Exercise 1: Sharing of Experiences - LOTI

10:40 Exercise 2: Creating Personas - LOTI

11:10 Break

11:20 Recap of LOTI AT evaluation research - LOTI

11:30 Exercise 3: Creating a prototype evaluation framework - Priya

12:25 Wrap up and next steps

12:30 Workshop Close

Exercise 1

10:10

Sharing of experiences● What AT pilots have you run / are

you running now?

● What has been your approach to evaluation?

● What have been the strengths and weaknesses of your evaluations?

Exercise 2

10:40

Creating personas● From your experiences with AT, what

factors about a person, their life and context are most important to understand before implementing a new AT intervention?

● Which factors are common to almost any person and scenario?

● Which factors are unique to certain types of person and scenario?

Break

11:10

AT evaluation - Research Summary

11:20

Key points from interviews with Sutton and Hackney

Learnings from existing projects and considerations for future evaluations● create a baseline

● Set up control groups

● Involve end users

● Gather qualitative and quantitative data

● Consider: a) the technology and any processes in place aspects before commissioning, b) the technical practicalities, c) borough context, d) the purpose

Existing tools: LGA’s Digital Social Care Programme and Global Disability Innovation Hub

What we’ve learned from our desk research

From other frameworks● NICE framework

● The Nesta Standards of Evidence

● SETT Framework

From case studies:● RNIB - highlights the need to provide training including takeaway factsheets /

user manuals to users of AT devices.● CQC - highlights that different ATs (independent living, condition

management etc.) might require bespoke qualitative and quantitative evaluation.

How LOTI can add value

Develop a consistent methodology for evaluating AT pilots that can enable LOTI member boroughs to compare like for like technologies and make better investment decisions.

LOTI’s outcomes-based methodology could be used to help boroughs frame their hypothesis about the intended change they believe an AT might make to help someone live independently.

PROBLEM SOLUTION

Phase

2 - Discover 3 - Define 4 - Develop 5 - Deliver

Generate Focus Generate Focus

Key Question

What are the real problems and what do we know about

them?

What specific problem(s) will we

try to solve?

What hypotheses do we want to test

by prototyping?

Which prototypes perform best?

Tech & Data

What problems relate to technology and data?

What role can technology and data play in enabling the desired outcome?

Non Tech & Data

What problems relate to people, processes and conditions?

Beyond technology, what is needed to achieve the desired outcome?

OUTCOME

What real-world

outcome do we want to

enable?

(This is not the solution, tech or

data!)

1 - Motivate

LOTI draft AT Evaluation Framework template

Desired Real-World Outcome (applying slide 5):

For each of the below, we need to capture: 1) How are things now? What’s our baseline? 2) What’s are the specific problems the AT will address / what are the user needs?

3) What’s our hypothesis for change and how it will we measure it? 4) What results and insights have we captured?

INDIVIDUAL ENVIRONMENT

Goal: Assess the individual’s current abilities, unique needs, and functional areas of concern.

Goal: Assess the individual’s environment (including the context of any public service support being delivered), how the proposed AT integrates, and the individual’s current capabilities within those

environments in comparison to other individuals.

TASKS TOOLS

Goal: Assess what actually happens in the environment (including the context of any public service support being delivered) and the

impact of current demands on the individual.

Goal: Assess the devices, services, resources and strategies that are currently in place to support the individual. Assess whether the

proposed tools are user-centred and/or task orientated.

Consider what Standard of Evidence you need to make the desired decision:● What are the variables and which can you control?● What are the limitations of your sample size?● What qualitative and quantitative data can you realistically capture?

Exercise 3

11:30

Creating a prototype evaluation framework● What are the objectives of AT evaluations

/ what needs must they serve, individually and collectively?

● Against what headings do we need to capture information?

● What is realistic to measure?

Wrap up

12:25

Next steps● Agreeing and prototyping the evaluation

framework

● Refining it for more systematic testing of ATs

Workshop close

12:30

London Office of Technology & Innovation

LOTI_LDNmedium.com/loti

#LOTI