independent evaluation of choice of choi

Sca

Independent Evaluation

of CHOI

July 2021

Independent Evaluation

of CHOICE

2021

Independent Evaluation of CHOICE 2

EVALUATION TEAM

Andrew Liles, Strategic Advisor, Wessex Academic Health Science Network

Philippa Darnton, Associate Director, Wessex Academic Health Science Network

Dr David Kryl, Director of Insight, Wessex Academic Health Science Network

Dr Andrew Sibley, Programme Manager, Wessex Academic Health Science Network

Dr Jackie Chandler, Programme Manager, Wessex Academic Health Science Network

Sophie Barton, Evaluation Programme Analyst, Wessex Academic Health Science Network

Tim Benson, Director, R-Outcomes Ltd

Ann Robertson, Project Assistant, Wessex Academic Health Science Network

CORRESPONDENCE

Philippa Darnton, Associate Director - Insight

Wessex Academic Health Science Network,

Innovation Centre, 2 Venture Road, Southampton Science Park, SO16 7NP

DISCLAIMER

This report presents the findings of an independent evaluation of CHOICE.

The findings of this independent evaluation are those of the author and do not necessarily represent

the views of the CHOICE project team.

ACKNOWLEDGEMENTS

We would like to thank the CHOICE project team, cochlear implant users and implant centre staff for

their participation in this evaluation. We express our gratitude to Professor Helen Cullington,

University of Southampton Auditory Implant Centre and Chief Investigator for CHOICE, for engaging

us to undertake this evaluation and to Dr. Maria Kordowicz, Evaluation Advisor to The Health

Foundation, for her guidance and support in the evaluation design and reporting.

This project was part of the Health Foundation’s Scaling Up Programme. The Health Foundation is an

independent charity committed to bringing about better health and health care for people in the UK.


CONTENTS

1. INTRODUCTION…………………………………………………………………………………………………………………….. 5

The CHOICE scaling up improvement programme 5

CHOICE explained 5

Evaluation design 7

Evaluation methods 8

Implementation timeline 12

2. SCALING UP CHOICE: THE EXTENT OF ADOPTION AND SPREAD…………………………………………… 14

2.1 The extent of the spread and use of CHOICE 14

2.2 The impact CHOICE had on users 16

How users engaged with digital technology 17

Whether CHOICE has improved equity of access for users 18

2.3 The impact CHOICE had on staff 20

2.4 The impact CHOICE had on implant centres 21

Summary 22

3. SCALING UP CHOICE: UNDERSTANDING THE FACTORS WHICH INFLUENCED ADOPTION AND SPREAD……….………………………………………………………………………………………………………………. 23

3.1 How users and staff perceived the value of CHOICE 23

3.2 How useability was perceived in practice 25

3.3 How CHOICE was integrated into practice 29

3.4 How preferences and culture influenced implementation 34

3.5 How users were recruited 35

3.6 How Covid-19 impacted implementation 38

Summary 39

4. Lessons for future scale up programmes ……………………………………………………………………………. 40

A final word from the innovator 43

References 44

APPENDICES

1. Non-adoption, abandonment, scale-up, spread and sustainability (NASSS) framework……….. 45

2. Normalisation Process Theory and the NoMAD instrument 47

3. Activity data matrix: planned and actual data received 48

4. Implementation constraints and enablers of CHOICE by theme ……………………………………… 51

SUPPLEMENTARY INFORMATION …………………………………………………………………………………………. 53


List of figures

1. Timeline showing major milestones …………………………………………………………………………….………. 13

2. R-Outcomes summary scores at baseline and follow up 16

3. Changes to R-Outcomes scores due to CHOICE 17

4. Users’ digital readiness scores 17

5. Users’ product rating for CHOICE 18

6. Convenience of getting implant support as a result of using CHOICE 19

7. Identification and resolution of implant-related problems as a result of using CHOICE 19

8. Travel to clinic appointments as a result of using CHOICE 19

9. Staff summary R-Outcomes scores 20

10. Outpatient attendances at the Manchester site 21

11. NoMAD baseline and follow-up scores for all centres 29

12. NoMAD scores comparing Southampton with the other centres 30

13. NoMAD domain scores for all centres ………………………………………………………………………………... 30

List of tables

1. CHOICE Functions ……………………………………………………………………………………………………….………... 6

2. CHOICE scale-up evaluation questions 8

3. Quantitative data planned and delivered 9

4. R-Outcomes measures and responses 9

5. Qualitative fieldwork undertaken 10

6. The number of users and how many times they logged in, by centre 14

7. Use made of the CHOICE functions 15

8. Staff participants and logins by centre 15

9. Recruitment phase by centre and total users recruited 35

10. Recruitment strategies used by centres ……………………………………………………………………………….. 37


1. INTRODUCTION

THE CHOICE SCALING UP IMPROVEMENT PROGRAMME

The CHOICE Scaling Up Improvement Programme (Cochlear Implant Home Care) is an innovative approach to

supporting adults with cochlear implants to manage aspects of their care at home. Each year around 1,500

people in the UK have a cochlear implant fitted - an electronic medical device that replaces the function of a

damaged inner ear, and each patient requires lifetime support from specialist centres (20 in the UK) for

maintenance and rehabilitation purposes. The nearest centre for some patients may be a considerable

distance from their home, meaning there could be significant financial costs, travel time and inconvenience if

attending in person. Remote care offered through the CHOICE platform includes a home hearing check,

auditory training, stock ordering, uploading a photo of the implant site for a clinician to check, and other

resources to allow patients to care for their hearing at home.

CHOICE was developed by a highly experienced audiologist specialising in the management of cochlear

implants, to offer an alternative to routine clinical appointments, for the benefit of patients and clinicians. The

intention was that face-to-face appointments would only be arranged if requested or if indicated to be

required, e.g., for a clinical need or access to specialist resources identified through the CHOICE platform. It

was not assumed however that remote care would suit all patients.

CHOICE was designed to respond to a policy and practice context in which advances in implant technology and

an increase in numbers of people likely to benefit from an implant meant that demand for post-implant care

was likely to grow. Just prior to the implementation of CHOICE, NICE published a review of the evidence for

cochlear implants for children and adults with severe to profound deafness, and recommended new eligibility

criteria that would make cochlear implants an option for more people with hearing impairments than before1.

The Health Foundation awarded £500,000 funding and expertise to support the scaling up of CHOICE from one

site, the University of Southampton Auditory Implant Service, where it was developed and trialled, to other

cochlear implant centres across the UK. Whilst the randomised controlled trial (Cullington et al, 2018)2 of the

pathway indicated the potential benefits of the innovation (increased patient empowerment, improvements in

hearing, and better ability to keep hearing stable), it was untested in other environments (other clinical

centres) and with a larger number of patients.

The programme was designed to run as a research project supported by a detailed research protocol that

describes the roles and responsibilities of the team leading the CHOICE programme, the standard clinical care

pathway for cochlear implant users, what CHOICE offered and for whom, how CHOICE would be rolled out and

how it would be evaluated.

Comprehensive governance was designed to support the programme. The CHOICE Leadership and

Management Team would take overall responsibility, based at University of Southampton Auditory Implant

Service, centred around Professor Helen Cullington, the Chief Investigator, and a new Project Manager post

funded from The Health Foundation budget. A Steering Group made up of representatives of the key

stakeholders would meet every four months to monitor delivery and advise and guide the CHOICE Leadership

Team. An Evaluation Advisory Group focused on the co-design and delivery of this evaluation. Each centre

would appoint their own site champion.

CHOICE EXPLAINED

Functions

CHOICE is a web-based application for implant users who wish to manage their own hearing health at home.

Once registered, users can access the following functions:

1 Technology appraisal guidance (TA556), published 07 March 2019


Table 1 CHOICE Functions

Monitoring your hearing

Hearing check Based on the Triple Digit Test (TDT), participants listen to sets of three digits in

background noise and type in the numbers they hear. An overall score is generated at

the end of the test. Designed to be repeatable so users can regularly repeat the test

and potentially detect any deterioration or improvement.

General check up This questionnaire asks about users' hearing health and their implant. Users complete

this check-up when they first log-in to CHOICE and can then repeat it when they want

to.

Check your

implant site

The ability to upload and store photos of the user’s implant site (behind their ear) and

share this with their implant centre. It is recommended that a baseline photo is taken

at an early stage to provide comparison for later images.

Maintaining your implant

Order spares Enables users to order spares.

Hours of implant

use

An opportunity to log how many hours the users are wearing their processors per day.

Rehabilitation tools

RealSpeech

listening practice

RealSpeech for CHOICE is an auditory training app, designed to empower people with

cochlear implants to improve their ability to understand speech in everyday situations.

Music practice The mixer app allows the user to control the sound of a song or a piece of music, by

adjusting how many instruments (tracks) they are listening to and how loud each of

them are. They can turn the vocals on and off and adjust the level of them. They can

adjust the overall volume and also control the pitch of the music.

Telephone

training

Used to learn and practise techniques for phone usage with cochlear implant/s and

provides tips for all experience levels. Users can download test scripts and access

training scripts to try with family and friends.

Signposting

Wellbeing

support

Signposting to organisations for information relating to wellbeing support e.g. wellbeing services who support people with hearing loss specifically and more general sources, such as Samaritans help line.

The hearing check

The hearing check was a fundamental component of CHOICE and initially the CHOICE team recommended that

users should not continue with CHOICE if they were unable to register a score, or if the score was too poor. It

was considered that if people with implants could not do a hearing check at home, then they would need to

come into the clinic anyway for hearing testing. This was relaxed later in the project, around the time of the

start of the Covid-19 pandemic, as people felt that patients could not come into the clinic anyway, so they

should at least be able to access the other beneficial features of CHOICE. CHOICE used an online hearing in

noise test based on the Triple Digit Test (TDT). Some people with implants were unable to obtain a score on

the TDT. As guidance, the team recommended that if users hadn’t previously been able to register a sentence

score in clinic of 50% or greater, they would not be eligible to use CHOICE. Clinic speech perception testing is

usually done using the Bamford-Kowal Bench (BKB) sentence test. Issues with the usability and reliability of the

TDT hearing check were reported throughout the lifecycle of the project and many changes were made in an

attempt to rectify these issues, including updates in December 2019 and April 2020, prior to its removal in

January 2021.


Staff interface

Staff have access to a dashboard that provides updates on their patients’ use of CHOICE, to which they

respond. The app flags to the member of staff when their implant users upload a photo, have a worse result

on the hearing check, require stock, or indicate they need help from answers to the general check-up

questionnaire (including reporting that they use their sound processor less). Users who have not logged into

the app for 90 days are also flagged. Staff use the app to record their responses and actions.

Recruiting users

Scale up and spread of CHOICE was reliant on both its successful integration into participating centre pathways

and the take up of CHOICE by implant users. Participating centres were advised to adopt shared decision

making when recruiting implant users. Recruiting clinicians were instructed not to make assumptions about

users’ abilities and motivations and to take into account their care needs, routine maintenance of equipment

and access to technology. The original plan had been to recruit users once they had worn their implant for 12

months or longer. This was revised in April 2020 in response to the Covid-19 pandemic to allow users to be

recruited sooner if deemed appropriate by the recruiting clinician. The study protocol provides a full list of

inclusion and exclusion criteria (see Supplementary material 5).

Cochlear implant suppliers

There are four suppliers of cochlear implants in the UK. Centres decide which suppliers they offer and there

are differences in their preferences. The four suppliers are Advanced Bionics, CochlearTM, MED-EL and Oticon

Medical. CochlearTM created their own remote care app and this was rolled out to many of the same centres

during the period of this study and is referred to as Remote Check.

EVALUATION DESIGN

This evaluation was commissioned in September 2017 to understand if the ‘desired effects’ of CHOICE,

observed during the original trial (Cullington et al, 2018), would be reproduced when rolled out to more ‘real-

world situations’. This would be determined by understanding the extent to which the programme was

successfully scaled up, the impact of the new pathway of care on users (staff and people using the care

programme), and the factors which enabled or hindered the scaling up of the programme. Seven sites were

identified by the CHOICE project team to participate in the roll out of CHOICE and the evaluation.

The evaluation design is informed by a growing research base on the challenges associated with the adoption

and spread of digital programmes. Technological innovations are increasing their presence in health care and

are now widely viewed as a significant potential contributor to supporting patients’ health (Maguire et al.,

2021). However, the experience of implementing technology programmes, especially ones that require major

changes in team working, organisations, or at the system level is poor, because of the combined problems of

non-adoption and abandonment by individuals and difficulties with scale-up and spread (Greenhalgh et al.

2017).

It is increasingly recognised that the more complex an innovation or the setting in which it is introduced, the

less likely it is to be successfully adopted, scaled up, spread, and sustained (Creswell & Sheikh, 2013).

Furthermore, it is not just the influence of commonly sought individually identified barriers and facilitators, but

the dynamic relationship between these factors, that creates the digital innovation adoption context.

Telehealth services, of which CHOICE is one, have received considerable attention in the research community

and are often supported at the policy level, but research has indicated that they are rarely successfully

sustained (Standing et al. 2016).

Two conceptual frameworks were particularly important to the design of this evaluation. The Non-adoption,

Abandonment, and Challenges to the Scale-Up, Spread and Sustainability of Health and Care Technologies

(NASSS) framework (Appendix 1) provides a theoretical base about what is important when implementing a

technological innovation, such as CHOICE. It has a very broad focus, across seven domains, including socio-

political issues, external and inner context issues, the nature of the innovation and the adopters. Appendix 1


sets how these domains were applied and interpreted for the CHOICE Scaling Up Programme. This framework,

in particular, aids the understanding of complex technology and its implementation. The other framework is

Normalisation Process Theory (NPT) (Appendix 2), which provides a theoretical base to help understand the

dynamics of implementing, embedding and integrating new technology or complex interventions into local

work environments. The NoMAD survey, completed by implant centre staff, is based on NPT and helped to

measure normalisation of CHOICE (see Supplementary material 1).

The CHOICE Scaling Up Programme had a clear set of evaluation questions (Table 2). These were co-designed

with the CHOICE project team to explore both the impact of the roll-out and the process of implementation.

Table 2 CHOICE scale-up evaluation questions

What is the impact of the roll out of the new care pathway on people with cochlear implants and

staff?

1.

a)

b)

What is the extent of spread of the new care pathway?

What has facilitated adoption?

What has hindered adoption?

2.

a)

Does the new care model improve patients' confidence to self-manage their cochlear implant as

measured by patient reported outcomes of health confidence, health status and personal wellbeing?

Do patients initiate review appointments with the service rather than rely on or wait for appointments

scheduled by the service?

3.

a)

Does the new care model improve patients' experience of follow-up care?

Do patients engage with the technology as measured by patient reported outcomes of digital confidence

and perceived value of the tool?

4. Does the new care model improve equity of access to follow-up care?

5.

a)

b)

Does the new model of care improve the experience of staff working in the service as measured by

staff reported outcomes of job confidence and work wellbeing?

Do staff have confidence in the new care model as measured by staff reported outcomes of digital

confidence and perceived value of the tool?

Do they recommend it?

6. Does the new care model improve use of resources through reducing the need for follow-up

appointments and enabling the service to be delivered by a different skill mix?

What lessons can be learned from the implementation process that will benefit spread and

adoption of this model?

EVALUATION METHODS

Quantitative data

The collection and analysis of activity data over time was key to measuring the extent of the spread and

adoption of CHOICE by centres, staff and people with cochlear implants. A set of measures and sources of

quantitative data were designed and agreed with the Steering Group at the beginning of the evaluation. In

practice, the data delivered to the evaluation team was variable and this is summarised in table 3. The

quantitative data used in this evaluation is explained in more detail in Appendix 3.


Table 3 Quantitative data planned and delivered

Planned quantitative data

Data delivered

CHOICE App data

A range of activity measures built into the App

including number of users and staff registering and

logging in and use of the different features - by centre.

Mostly delivered and described in section 2.1 of this

report.

Outpatient data

Tracking outpatient attendances for CHOICE users to

understand if they change. Mapping outpatient activity

by truncated postcode to support travel time analysis.

Unable to deliver because of Information Governance

constraints. 3 of 7 centres supplied aggregate before

and after data, but with low CHOICE activity it wasn't

possible to see an effect - described in 2.4 of this

report.

Site data

A range of manually collected measures from each of

the centres to understand in more detail the impact on

users (e.g. users contacting the centre with problems

with CHOICE) and the impact on Centres (e.g. adverse

events, clinician caseload ratio).

Most of these measures were not collected

consistently across the centres and were not able to

support this evaluation.

Patient questionnaire

To understand the potential time and cost benefits to

patients (travel cost, time, work, childcare etc.).

25 (10%) users completed the survey and its findings

are included in section 2.2.

User and Staff self-reported measures R-Outcomes (Benson, 2020) provide a wide-range of self-reported outcome measures using easy to

understand and complete surveys (see Supplementary material 2). Each measure typically has four questions

that the person responds to by selecting where their feelings lie on a four-point scale. For this evaluation, the

R-Outcomes surveys were built into the CHOICE web app, with users being prompted to complete them when

they first login (baseline) and again once they have had experience of CHOICE (follow-up). The following R-

Outcomes measures were used and completed:

Table 4 R-Outcomes measures and responses

Cochlear Implant Users Staff

Health and wellbeing:

▪ The Health Confidence Score measures what

people think about their health literacy, self-

efficacy, access to care and shared decision

making.

▪ The Personal Wellbeing Score is a simplified

version of the Office of National Statistics

measure, covering life evaluation,

worthwhileness, happiness and anxiety.

▪ The Health Status Score is a short patient

reported outcome measure to track and compare

patient perceptions of how they feel and what

they do

Experience at work:

▪ The Job Confidence Score measures staff

confidence to do their job, addressing their

knowledge, self-efficacy, access to help and

involvement in decisions.

▪ The Work Wellbeing Score measures staff views

of their job satisfaction, worthwhileness,

happiness and anxiety.

Digital innovation:

▪ The Digital Readiness Score self-rates users’

digital literacy and confidence to use digital

products, along with their openness to innovation.

Digital innovation:

▪ The Digital Confidence Score self-rates users’

digital literacy and confidence to use digital


▪ The Product Rating Score provides a user

assessment of a digital product in terms of

usefulness, ease of use, support and satisfaction.

products, with dimensions of familiarity, social

pressure, support and digital self-efficacy.

▪ The Innovation Readiness Score rates how much

users are open to and up to date with new ideas,

and whether their organisation is receptive to and

has innovation capabilities.

▪ The Product Rating Score provides a user

assessment of a digital product in terms of

usefulness, ease of use, support and satisfaction.

Responses:

▪ 236 baseline

▪ 96 follow-up

Responses:

▪ 5 baseline

▪ 8 follow-up

Additional information on all data collection instruments, surveys and interview schedules can be found in the

Supplementary materials.

Qualitative methods The qualitative investigation of CHOICE, as part of a planned mixed method approach, sought to provide

explanatory information on the implementation of CHOICE. Key aspects to explore were related to desired

changes in knowledge, awareness and attitude and changes in the behaviour of those involved in receiving or

implementing CHOICE. The evaluation team were guided by the NASSS framework in structuring the interview

schedules for users and staff (see Supplementary information 3) to help understand the issues around the

scale up of CHOICE (Appendix 1). In addition, the NPT NoMAD questionnaire was used with staff to

understand the dynamics of implementing, embedding and integrating CHOICE in their centre (see

Supplementary material 1).

In July 2018, the evaluation team supported beta-testing of the CHOICE web app through two focus groups

with nine participants, and email and survey responses from a further 21 participants. The aim was to identify

technical and usability challenges. In total, 74 issues were identified for potential improvements to the web

app.

The intention had been to undertake a two-day site visit to each of the seven participating centres, comprising

a focus group with staff, a focus group with users, individual interviews with users where preferred, and an

interview with the CHOICE site champion. In response to the limitations imposed by the Covid-19 pandemic,

the fieldwork was instead carried out using video calls. Table 5 summarises the qualitative fieldwork data that

was collected between June and December 2020. The number of user and staff participants was lower than

intended, due to a more limited uptake of CHOICE than anticipated.

Throughout the report, participant quotes from implant users and staff members are linked to their respective

interview identification codes, demonstrating the range of contributions to the qualitative findings.

Table 5 Qualitative fieldwork undertaken

Data source

type (see

Supplementary

information 3)

Recruitment Recruitment notes and sample demographics

Implant centre

staff

semi-structured

interviews

16 The aim was to recruit a total of 70 staff across seven implant centres,

approximately 10 per implant centre. However, due to low use of CHOICE

across the sites, 16 staff were interviewed - between 1 and 4 per centre.

Staff job titles included Audiologist, Clinical Scientist (Audiology), Senior

Audiologist, Principal Audiologist, and Advanced Specialist Audiologist.


Implant centre

Champion

semi-structured

interviews

7 All 7 centre champions were interviewed. These staff had the following job

titles: Audiologist, Clinical Scientist (Audiology) and Advanced Specialist

Audiologist.

Implant users

semi-structured

interviews

35 The aim was to recruit between 10-20 implant users per implant centre,

however, due to low uptake of CHOICE, 35 implant users in total were

interviewed about CHOICE across the 7 participating implant centres. Of

these, 18 were female and 17 male. The mean age of implant users was 58.7

years. The majority had either CochlearTM (48.6%, n=17) or Advanced Bionics

(31.4%, n=11) implants. Six implant users had MED-EL.

Of the 35 implant users interviewed, information was not available to

determine the experience level (years since surgical implant) of 4 implant

users. Of the remaining 31 implant users, 25.7% (n=9) had more than 10

years’ experience of implant use, a further 40% (n=14) had between 4 and 9

years’ experience, 14.3% (n=5) had between 1 and 2 years implant

experience and 8.5% (n=3) users had less than 1 year of experience with

implant use.

Case studies

(completed by

staff)

59 The aim was to obtain 10 case studies per implant centre, however, 1

implant centre was not able to provide case studies. The mean age of

implants users in the case studies was 54.1 years. Of the 59 case studies, 34

(57%) were female implant users and 25 male implant users (43%).

NoMAD implant

staff survey

responses

19

(Baseline)

11

(Follow- up)

Almost all responses were from audiologists, apart from 2 baseline responses

from administrative staff. Most responses were from four implant centres,

the remaining three centres only completed the survey once at baseline and

once at follow up. Most staff had been working at their implant centre for

many years, 63% for at least 6 years. Between-group comparisons were

made.

CHOICE Steering

Group - Focus

group

1 group (4

people)

4 members of the steering group participated in 1 focus group.

Chief

Investigator

interview

1 The study Chief Investigator was interviewed for 2 hours about their

perception of CHOICE and its implementation journey.

Limitations

As part of the quantitative data, the intention had been to collect row-level outpatient data for all eligible

implant users (both those who had registered to use CHOICE and those that had not). This would have allowed

us to track changes in the use of outpatient appointments among CHOICE patients over time and compare this

to any observed change among the non-CHOICE group over the same period. Information Governance

concerns were raised at the start of the project regarding the possibility of being able to personally identify

users, given that they are a relatively small group. This meant that we were unable to analyse row-level data

for all patients as originally planned, instead being limited to the use of aggregated data for non-CHOICE

patients and for CHOICE patients up until the point they registered to use the web app. Without the ability to

ascertain how many appointments had been attended on an individual patient level, we were not able to draw

any conclusions regarding the impact of CHOICE on outpatient activity.

In addition to this, the Covid-19 pandemic led to changes in the way that centres delivered their outpatient

appointments and reduced their capacity to provide any data for the project. While centres were able to

provide some of the planned metrics, they were not collected consistently enough across all sites to able to

quantify them as planned.


In August 2020, it was realised that follow-up R-Outcomes data was not being collected as expected. This was

traced to an issue in the CHOICE web app that was not fixed until December 2020. The large majority (77 of 96)

of follow-up responses were collected in January 2021 during the second UK national lockdown and very close

to the end of the project (end of January 2021). This is likely to have distorted results negatively. The number

of staff responses was less than originally expected and these were largely confined to the site champions at

each site.

The qualitative fieldwork adequately represented the key stakeholders involved in CHOICE, however, four

limitations were evident.

i) Due to the limited rollout fewer implant centre staff used CHOICE and fewer implant users registered on

CHOICE than expected. This reduced the population and diversity of staff and implant users from which to

recruit. Wider involvement may have gleaned more learning from this study. Importantly, all centre

champions were interviewed, which ensured implementation issues and perceptions of impact were

obtained for each centre.

ii) Due to the limited involvement of implant centre staff, fewer NoMAD surveys were completed than

expected. The low number of surveys meant no inferential statistical analyses comparing baseline and

follow-up responses could be conducted.

iii) Views were gathered from the CHOICE Chief Investigator and CHOICE Steering Group, however, a key

member of the CHOICE project team, responsible for project management, left the project in mid-2019 and

was unable to be interviewed.

iv) Interviews with the CHOICE software platform developers, in hindsight, would have been a productive

avenue of inquiry. This was not part of the study protocol, but CHOICE project team members highlighted

the importance of that relationship to the spread of CHOICE.

IMPLEMENTATION TIMELINE

The Health Foundation awarded funding for CHOICE to be scaled up to eight centres in September 2017. At

that time the intention was that the sites would begin to 'go live' in April 2018, with evaluation fieldwork visits

happening between July 2018 and February 2019 and this report being issued in July 2020.

In practice, the programme experienced significant delays in gaining approvals. Unexpectedly, CHOICE

required a CE Mark, and this wasn't granted until August 2018. Ethics approval from the University of

Southampton had been expected in January 2018 but wasn't granted until November 2018.

This meant the first site opened for recruitment in June 2019 and all seven weren't open until March 2020.

Unfortunately, this was quickly followed by the first national lock-down for the Covid-19 pandemic and site

champions reported that recruitment to CHOICE stalled in six of the seven centres as centres did not have the

opportunity to introduce CHOICE face-to-face. In June, they began to re-start recruitment and by October all

were recruiting again. The following chart (figure 1) describes the major milestones.


Figure 1 – Timeline showing major milestones

Sep 2017

Dec 2017

Mar 2018

Jun 2018

Sep 2018

Dec 2019

Mar 2019

Health Foundation Scaling Up funding approved.

Wessex AHSN appointed evaluation partner

Programme governance established

CE Mark granted (unforeseen requirement)

University of Southampton ethics approval (plan had been January 18)

Training on how to use CHOICE for all participating centres.

Health Foundation agree extension to programme.

Jun 2019

Sep 2019

Dec 2020

Mar 2020

Jun 2020

Sep 2020

Jan 2021

1st Southampton patient recruited

1st Manchester patient recruited

1st Cambridge patient recruited

1st North East patient recruited

1st Guys’ & St Thomas’s patient recruited

1st Royal National ENT and Eastman patient recruited

1st Nottingham patient recruited

Ethics approval to include 16 & 17 yr olds

Hearing check removed from CHOICE

End of CHOICE project

1st NATIONAL LOCKDOWN

2nd NATIONAL LOCKDOWN

3rd NATIONAL LOCKDOWN


2. SCALING UP CHOICE: THE EXTENT OF ADOPTION AND SPREAD

This section considers whether CHOICE achieved the intended plans for scaling up. It describes the extent to

which CHOICE was taken up by cochlear implant centres (its adoption) and its reach and use within the target

groups (its spread). The findings are drawn from web app data about who used CHOICE, what they used it for,

their use over time and self-reported outcomes on its impact. An explanation of these findings is provided in

section 3.

2.1 THE EXTENT OF THE SPREAD AND USE OF CHOICE

There are 18 Cochlear Implant (CI) centres for adults in the UK, supporting more than 11,000 implant users

aged over 16 who might benefit from a remote care option.

The scaling up ambition for CHOICE was that eight centres would offer CHOICE. The protocol for this study set

out the assumption that these eight centres would care for approximately half of the c.11,000 adults with

cochlear implants in the UK and that 40% of their adults would enrol for the home care pathway - this would

be 2,200 users.

In practice, this scaling up project attracted seven centres to participate and through them 312 users

registered, of whom 240 logged in to CHOICE at least once. The following table compares the extent of the

spread and use of CHOICE by implant users, across the seven centres:

Table 6 The number of users and how many times they logged in, by centre. Jun 2019 – Jan 2021

Centre Users registered

Users logging in at least once

Number of logins by centre

Average logins per person per

month*

Southampton 89 63 750 0.9

North East 79 65 346 0.6

Manchester 46 35 354 1.1

St Thomas' Hospital 35 29 171 1.0

Royal National ENT and Eastman Dental Hospital

25 21 128 1.1

Nottingham 17 10 38 0.4

Cambridge 21 17 212 1.7

Total 312 240 1,999 0.9

* Average logins per person per month are the number of logins (summed), divided by time spent registered on the web

app (summed). This accounts for different users being registered on the app for different periods of time.

This shows that users logged in at most once a month on average. Many registered users did not proceed to

logging in following registration.

41% of users were male and 58% were female (1% were not recorded). As the left-hand chart below shows,

there were users of all ages with the largest groups being for people in their 60s and 70s. The chart on the

right shows most users had sound processors implanted that were produced by either CochlearTM or Advanced

Bionics (AB).


Once logged in, the features available from CHOICE were used as follows:

Table 7 Use of the CHOICE functions, Jun 2019 – Jan 2021

Function Number of times this

function was used

Number of Users,

using this function

% Users that have

used this function (of

those who logged in

at least once)

Triple Digit Test hearing check 575 190 79%

RealSpeech hearing check 161 15 6%

Ordering replacement parts 152 60 25%

Photograph of implant site 131 125 52%

Record of implant use 111 82 34%

There are approximately 10 members of staff per centre, and it was anticipated that all of the staff in the

participating centres would be involved with CHOICE - approximately 70 staff from 7 centres.

In practice, while 43 registered, only 23 staff used CHOICE, with a total of 2,046 logins. The following table

compares the extent of the spread and use of CHOICE by staff, across the seven centres:

Table 8 Staff participants and logins by centre, Jun 2019 – Jan 2021

Site Staff

registered

Staff logging in

at least once

Number of

logins by centre

Average logins

per person per

month

Southampton 3 2 1,505 37

North East 11 8 242 3

Manchester 5 5 97 2

St Thomas' Hospital 17 4 63 1

Royal National ENT and Eastman

Dental Hospital

1 1 56 4

Nottingham 2 1 42 3

Cambridge 4 2 41 2

Total 43 23 2,046

While most staff only logged in a handful of times a month, the number of log ins at Southampton was much

larger. Some sites shared generic login details to CHOICE; the number of individuals logging in would be slightly

higher than the data suggests in these centres.

0 10 20 30 40 50 60 70

11 - 20

21 - 30

31 - 40

41 - 50

51 - 60

61 - 70

71 - 80

81 - 90

91 - 100

User age groups


Key findings

This scale up project achieved a substantially smaller adoption and spread of CHOICE than planned. The

number of users who registered and logged into CHOICE between June 2019 and January 2021 in seven

centres was 240, compared with a target of 2,200 in eight centres.

The rest of this section explores the impact of this limited scale up on users, staff and centres.

2.2 THE IMPACT CHOICE HAD ON USERS

Health, Health Confidence and Wellbeing

R-Outcomes surveys were completed by 236 individual users when they registered to use CHOICE and 96

individual users once they had been using it for some time. An important issue with the follow up data is that

59 (63%) were submitted in January 2021, during the third Covid-19 lockdown, a context that is likely to have

had an effect on how people report how they feel. R-Outcomes scores are out of 100, the higher the score the

better - the more positive the respondent feels about the four questions that make up the score. Scores over

80 are recognised to be high and positive, scores between 60 and 80 are moderate and below 60 are low and

negative. These are descriptive statistics, meaning the before and after differences shown are not statistically

significant.

This chart presents the summary scores for the three measures of health and wellbeing.

▪ All of the scores got

slightly worse during

the period using

CHOICE.

▪ Users describe feeling

less confident with

managing their own

health and lower

personal wellbeing.

▪ These changes are

not statistically

significant.

Figure 2 R-Outcomes summary scores at baseline and follow-up

The follow-up survey also gave users the opportunity to think about and record the impact that CHOICE has

made on how they feel about these measures. Figure 3 lists the four questions that make up each measure

and the average score given by the 96 follow-up respondents.


▪ Collectively, users

reported that using

CHOICE may have

made a modest

impact on their

wellbeing and

health status.

▪ The health

confidence

questions, which

focus on self-

management, are

slightly more

positive.

Figure 3 Changes to R-Outcomes scores due to CHOICE

HOW USERS ENGAGED WITH DIGITAL TECHNOLOGY

We asked users to complete a Digital readiness score to help us understand their confidence to use digital

devices and their openness to innovation in health care:

▪ They reported that

they have high digital

confidence (first two

questions) and that

this increased while

they used CHOICE.

▪ They were open to

innovation in

healthcare.

▪ These changes are

not statistically

significant.

Figure 4 Users' digital readiness scores


Users also responded to a Product rating score to give their assessment of CHOICE in terms of its usefulness,

ease of use, support and satisfaction.

▪ Their score for

whether CHOICE

helps them do what

they want was fairly

low.

▪ They were fairly

positive about the

ease of use and

getting help, and

expressed

moderate

satisfaction with

CHOICE.

Figure 5 Users’ product rating for CHOICE

WHETHER CHOICE HAS IMPROVED EQUITY OF ACCESS FOR USERS

The evaluation plan had been to access user level data that would enable us to track and compare clinic

attendances prior to and while using CHOICE. We planned to use this information to map and compare travel

times and travel frequency from people's homes to their centre over time. This was not possible because of

Information Governance restrictions.

In the absence of this data, we collected and analysed self-reported data from a separate travel survey to

provide insights into whether CHOICE had an impact on equity of access to cochlear implant follow support.

25 people completed the survey, representing 10% of users. The findings are presented below.

→ Most were aged over 60 and travelled by car

→ None of them needed to arrange child-care

→ 28% brought someone else with them to their appointment

→ Between 1 and 1.5 hours was the most common journey time

→ Seven people had costs of less than £4 and four people had costs of over £30

Which modes of transport do you use to travel to your appointments?

Car only

Train only

Train & bus

Car, train & walk

Train & taxi

Underground


→ The following chart shows that 71% of respondents in the 0-5 month group felt CHOICE made support

a lot more convenient, compared with just 11% of those respondents who had been using it for more

than a year.

Figure 6 Convenience of getting implant support – as a result of using CHOICE

→ A similar pattern was reported when asked whether CHOICE improved identification and resolution of

implant-related problems. People using CHOICE longer were less positive about how much it sped

things up.

Figure 7 Identificaton and resolution of implant-related problems – as a result of using CHOICE

→ Most people felt that they had travelled to clinic appointments less as a result of using CHOICE, but

again this reduced with time.

Figure 8 Travel to clinic appointments – as a result of using CHOICE

0 1 2 3 4 5 6 7 8

£0-£4

£5-£9

£10-£14

£15-£19

£20-£24

£25-£29

£30-£34

£35-£39

Not recorded

How much money do you spend travelling to your appointments?

0 1 2 3 4 5 6 7 8 9

Less than 30 mins

30 to 59 mins

1 hour - 1 hour 29 mins

1 hour 30 mins - 1 hour 59 mins

2 hours - 2 hours 29 mins

2 hours 30 mins - 2 hours 59 mins

4 hours+

How much time do you spend travelling to your appointments?


From the interview data (described in detail in section 3), implant users found CHOICE easy to use. It promoted

both independence and reassurance. Not travelling to clinic was beneficial, however, for a variety of reasons

some preferred face-to-face appointments, and these reasons are described in section 3.4. CHOICE was also

less useful for experienced implant users. For many, CHOICE needed further development with a desire for a

more informative and interactive site that is compatible with all digital devices and provides greater

functionality on the performance of their implant.

2.3 THE IMPACT CHOICE HAD ON STAFF

Figure 9 shows the results for 5 staff when they registered to use CHOICE and 8 staff once they had been using

it for some time. Whilst these numbers appear low, decisions at the implant centres to centralise CHOICE

activity around the centre champions meant it was commonplace for a very limited number of staff to manage

CHOICE related work. Five of the seven centres only had one ‘main user’ (routinely logging into CHOICE), with

two other centres having three and five ‘main users’ of CHOICE. This means that these surveys were

completed by the staff who were the most involved in CHOICE.

Scores over 80 are recognised to be high and positive, scores between 60 and 80 are moderate and below 60

are low and negative. All of the staff measures improved between their baseline and follow-up responses,

which were mostly recorded in Summer 2020. Again, Covid-19 may be a determinant here.

Figure 9 Staff summary R-Outcomes scores

• None of the before and after differences shown in Figure 9 are statistically significant (two-tailed

independent samples t-test). The sample size was too small for this to be likely.

• Job Confidence is fairly high to start with and improves. The baseline includes a low score (64) for "I am

involved in decisions that affect me". The highest question score was for "I can get help if I need it"

which at 84 rising to 91 at follow-up is very positive.

• Work Wellbeing is positive with very high scores for job satisfaction (88) and maximum scores for feeling

their job is worthwhile (100).

"I want to make the best of my implant and I’m very conscious of making sure that I try and listen and I try and contribute,

pay attention to what I’m doing and making sure I’m still educating myself. I think sometimes I need that information, I

need that baseline or something to see and then compare it to. I think that’s not in CHOICE at the moment.” Implant user

01_ICR


• Digital Confidence was very high at baseline and improved further. These staff are very confident with

using digital devices at work.

• Innovation readiness improves, with the highest score being for "New ideas are needed in my field" (88).

• The overall Product Rating for CHOICE improved from a low score (60) to a better figure (69). At baseline

there was a very low score (49) for "CHOICE helps me do what I want" which improved (58). A higher

baseline score of 67 for "I can get help if I need it" rose to a good score (84). Staff initial impressions of

CHOICE were low, but these improved with experience.

The interview data (see section 3) found that some staff (including site champions) found CHOICE impacted on

their work by lessening administration, however, it also created more work when trying to link CHOICE activity

with information in their patient management systems. There were also concerns about its impact on patient

pathways. There was a benefit to information sharing between themselves and implant users. Some staff

preferred either their own hearing tests or the CochlearTM Remote Check app (see section 3.3) so the

availability of the hearing check within CHOICE did not have an impact on their working practice. Also, similarly

to some implant users, staff wanted additional improvements particularly to their dashboard (see

Supplementary information 4).

2.4 THE IMPACT CHOICE HAD ON THE IMPLANT CENTRES

An area of interest for the scaling up programme and this evaluation was whether CHOICE helped improve the

use of resources in centres, by reducing the need for follow-up appointments or enabling the service to be

delivered by a different skill mix.

Unfortunately, a mixture of lack of available data and the low uptake of CHOICE meant that it was not possible

to quantitatively evaluate this.

Three of the centres did supply their outpatient attendances each month, and these did not show any

evidence of change following the launch of CHOICE. The example below is for Manchester where there an

average of 466 outpatient attendances per month. 35 users logging in to CHOICE on average 1.1 times per

month did not have a noticeable impact on clinic attendances. The chart does show a reduction mirroring the

Covid-19 lockdowns.

Figure 10 Outpatient attendances at the Manchester site during CHOICE and following lockdown

None of the sites reported changing the skill mix of their workforce as a result of adopting CHOICE.


Summary

This scale up project achieved a substantially smaller adoption and spread of CHOICE than planned. The

number of users who registered and logged into CHOICE between June 2019 and January 2021 in seven

centres was 240, compared with a target of 2,200 users in eight centres.

There was no evidence that users’ self-reported health, confidence and wellbeing improved while using

CHOICE. In fact, their scores went down, though we suspect this was due to many of the follow-up responses

being made during Covid-19 lock-down. When asked specifically about the difference CHOICE had made, users

responded that it had had a moderate improvement on their ability to manage their own health. When rating

CHOICE as a digital product they were not positive about it helping them to do what they want, and were only

moderately satisfied with it overall.

There were differences in how users feel about CHOICE depending on how long they had been using it. In the

travel survey (n=25), people who had been using CHOICE for less than 6 months were noticeably more positive

than people using it longer than a year. We were not able to explore the reasons for this as part of this

evaluation.

A survey of staff found they were moderately satisfied with CHOICE. It improved with use but remained

moderate.

With low numbers of users recruited, centres did not experience a noticeable reduction in face-to-face

outpatient attendances, nor did they change their staff skill-mix.

The next section seeks to describe the main themes that explain why CHOICE did not scale up as planned.


3. SCALING UP CHOICE: UNDERSTANDING THE FACTORS WHICH

INFLUENCED ADOPTION AND SPREAD

Section 2 described a substantially smaller adoption and spread of CHOICE than was planned and limited

impact on users, staff and centres. This section seeks to understand why this was the case. A range of factors

which contributed to the adoption and spread of CHOICE are described. These are derived from user and staff

experiences of using CHOICE and their reflections on how CHOICE was implemented.

When considering all the qualitative findings together, the evidence indicates that CHOICE did not realise its

predicted impact in ‘real world’ settings and that scaling up was premature. In total, 50 different constraints

to implementation were identified (see Appendix 4 for full detailed list) and these outnumbered the 28

enablers identified. Six themes were identified from this long list of constraints and enablers:

1. How users and staff perceived the value of CHOICE

2. How useablity was perceived in practice

3. How CHOICE was integrated into practice

4. How preferences and culture influenced implementation

5. How users were recruited

6. How Covid-19 impacted on implementation

Findings have been organised thematically based on a synthesis of data from the different qualitative sources.

The six themes are applicable to all study centres and any disconfirming case or variation is discussed where

appropriate. Both theoretical frameworks applied to this evaluation (NASSS and NPT) facilitate an

understanding of these findings.

3.1 HOW USERS AND STAFF PERCEIVED THE VALUE OF CHOICE

For any innovation expecting to become part of routine care, its perceived value to those that both deliver and

receive the innovation is a critical success factor for its adoption. This perceived value is formalised in the

NASSS framework (Appendix 1) as the ‘value proposition’. This value proposition domain seeks to explain both

the value to the developer or supplier (supply side) and the value to those who will use the innovation

(demand side) (Greenhalgh et al 2017). The original value proposition presented a new clinical pathway for

implant users that provided personalised support using a range of online tools adopting a patient-centered

approach that would identify any hearing performance issues earlier than routine clinic appointments. In

providing access to a variety of tools, information and support the expectation was to provide individual users

with greater autonomy and independence, whilst remaining connected to their implant centre and audiologist.

This initial value proposition was strong and clear, CHOICE showed promise as a remote care opportunity.

However, as evidenced in section 2, it was unable to meet many expectations for both implant staff and users.

Some of the initial programme assumptions led to the targeting of users with many years of implant

experience and potentially missed newer users who might have valued it more. CHOICE was not “sold” widely

enough to have the projected impact for a scale up initiative, and over time there was less emphasis on

promoting CHOICE due to issues with functionality, described as a theme in 3.2. A key factor was the selection

and identification of suitable implant users, described as a theme in 3.5. The Covid-19 pandemic saw an

increased reliance on remote care in the NHS, but this was not the experience of CHOICE because

opportunities to meet with implant users face-to-face to recruit them to CHOICE were restricted.


Despite this, perceptions of the potential value of CHOICE were mainly positive. Users and staff described it as

providing a ‘safety net’ between clinic visits for implant users. Implant users felt they could try and sort a

problem through CHOICE before needing to approach implant staff thus allowing some independence.

In addition, members of the Steering Group described CHOICE as a catalyst for the likelihood of future market

development by the implant companies.

However, the ability of CHOICE to deliver on a key component, the Triple Digit Test hearing check (TDT),

proved to be critical. For both implant users and staff, access to the hearing check to monitor implant user

hearing performance was regarded to be an important potential benefit of the remote system. Staff were of

the opinion, initially, that the implementation of CHOICE as an option in the care pathway would reduce

hospital visits, lessen their increasing workload and replace annual reviews. They also saw this as an

opportunity to encourage implants users to take greater ownership of care. However, staff had initial anxieties

about the transition to remote care and whether users would accept this care model. They also had concerns

about CHOICE itself, increasing staff workload.

A broadly cited view, held by users and staff, was CHOICE was targeted at users with too much experience of

implant use, a group which did not need many of the rehabilitation support tools on CHOICE. 17 of 35 users

interviewed had over five years implant experience, for example, and described the following:

“It also told me that I could do my hearing

checks and that I could do lots of things, which

make you feel like you're in control. I think

that's a big thing.” Implant user 01_ICB

“I'll use it to order my spares as well. Stuff I need; batteries,

microphone cover etc. Actually, it allows me to be more

independent rather than relying on my mum to phone up the

hospital. It allows me to be a lot more independent." Implant user

06_ICK

"I use it on my smartphone. I know there isn't an actual app to download it on, so I go on the website to logon. I use it now

and then. I know there's a questionnaire on there. I use it just to check my implants, and now I need to think about stuff I

wouldn't usually think about." Implant user 06_ICK

"Personally, I just love the concept of it. Sometimes people come for

review, and you do the same thing again and again. Nothing has

changed in their programming. I just don't want to see those

people in my life either. I just say, "Stay at home, do it from home".

I'd rather see difficult patients who really are having hearing

difficulties. I'm all for the CHOICE pathway." Staff member 01_ICR

"... we need to have a bit more faith in our

patients so that they can do these things.

Initially, I think we were a bit worried. I

think that the word remote worried us a

little bit" Staff member 03_ICG

"Obviously, now 16 years later, I feel that I've got maximum benefit anyway from the experience with it. In the early days,

it would have been really, really helpful to have those additional app and capabilities" Implant user 07_ICP

"I genuinely believe this is going to be the case, as each of

the major manufacturers will be developing this themselves.

It will be available they will fund it, we will have access to it

.... they will probably do this with knobs on, they’ve got all

the money." Steering group member 3

"I think they’re all desperate to get the market moving

forward, though well of course, they always have

been. Therefore, they want to get this solution right"

Steering group member 4.

“…its main impact was indeed to set the agenda….it succeeded in that." Steering group member 4


Targeting the introduction of CHOICE and its applicability for different users was a nuanced picture based on

its purpose, whether for rehabilitation and/or monitoring between user and the implant centre. So, whether

length of time post implantation was a factor or not:

In summary, from the perceived value of CHOICE at its outset, the experience of staff and users over time

revealed errors in its initial assumptions (e.g. targeting experienced users). Despite the value proposition of a

remote care system for implant users to encourage self-care and provide an additional pathway for implant

centres, a range of challenges led to questioning of the initial value proposition and its future potential.

3.2 HOW USEABILITY WAS PERCEIVED IN PRACTICE

Dependability of technology in real world conditions relies on its ability to function as intended or its ability to

adapt as required (see NASSS domains 2 and 4 in Appendix 1). User and staff feedback about each of the core

functions of CHOICE revealed benefits and disbenefits of each which are described below.

CHOICE remote care provided three core functions:

• Monitor hearing and implant site: (TDT hearing check, questionnaires, photo)

• Maintain the implant: (spares)

• Provide rehabilitation tools: (e.g. music and telephone training)

Some users and staff reported they were able to communicate via email easily through CHOICE. Staff sent

reminders and received requests for action from users. This enabled communication between staff and users

in between annual review clinic appointments, possibly replacing them. It presented an additional option in

the care pathway to manage an increasing population of implant users who will always require support from

their centre.

Monitoring of hearing and implant site

For implant users

Both early and experienced implant users welcomed the opportunity to monitor their hearing and were highly

motivated to do the hearing check.

“I'm glad the hearing test is there, because it's essential for people long term to keep an eye on things." Implant user

01_ICP

"Definitely 100%, yes. There's something that in hindsight, we realised that it was important. Initially, we thought

CHOICE is just replacing long-term care, but really because it has a lot of rehabilitation tools, it needs to be introduced

early on because many of the tools, such as practicing speech, listening, or music training, really are mainly relevant in

the very beginning. By telling them after nine months you can have this, and we've not talked about it before, it would

be like, it could have been useful then". Staff member 02_ICG

Key messages

1. Initially, the value perception of CHOICE by implant users and staff was strong which provided a

“safety net” for ongoing care.

2. Eligibility and selection of users who would benefit was misaligned from the outset.

3. Subsequent challenging experiences with use undermined the value of CHOICE and its future

potential.


However, many experienced issues with this check, which raised levels of anxiety and frustration as expressed

in the interviews. This included the requirement to complete the hearing check to complete registration,

leading to some not continuing to full registration (this requirement was later relaxed). Users wanted to

understand their hearing check scores to enable greater independence in monitoring their own hearing

performance.

Other users did not seem overly concerned due to their experience and confidence in noticing problems with

their implant.

For some implant users, taking photos of their implant site, especially if they lived alone, presented a

constraint to the use of this component.

For staff

The TDT hearing check was an important incentive for using CHOICE and is not routinely used in implant

centres. It is a test of hearing in the background of real-world sounds. Staff preferences were variable on its

use and whether it could replace other hearing checks they use. In addition, some staff were concerned that if

they were not seeing patients in the clinic there would be no tests of the internal device electrode function

(impedances). Nevertheless, the TDT in CHOICE was widely welcomed as a hearing check to identify those

starting to develop problems with their hearing.

Unfortunately, reliability issues with theTDT persisted throughout implementation leading to its withdrawal in

December 2020.

The project team identified that the test’s unreliability was probably due to a combination of issues including

differing sound environments, test anxiety, user attention and equipment. Staff reported that problems were

establishing a baseline and unreliable hearing test scores which incorrectly notified implant centres of a

problem. Implant users were kept abreast of emerging concerns and asked to contact the centre with any

ongoing hearing worries. Therefore, despite the evidence supporting the potential reliability of the TDT

(Cullington et al, 2018), real world testing in the remote care context revealed limitations in its useability.

CHOICE also allows users to upload a photo of the implant site. Staff welcomed this facility, however they

recognised that implant users might struggle if living alone. The platform also has a questionnaire that

monitors the status of the implant user and their implant. Staff found this a useful tool to triage implant users

and assess whether a clinic appointment was required.

“I've not been in this field that long, but personally, I didn't find it very helpful. I just kept getting lots of warnings when

someone was away from their baseline and then I've have to email and they've done it wrong or maybe it wasn't the

right test to do. I think it would be better if there was just a sound field or something in a pure tone, something simpler.

That would be better really for our patient group." Staff member 04_ICR

“The questionnaire itself is quite good because it flags up any answers that might need intervention." Staff member

01_IC_ICY

“The score came out as minus 0.7 which left me a bit confused. I

didn't know what that meant. I think it meant my hearing was

a little bit better, but I wasn't sure and so that was a bit

confusing." Implant user 02_ICG

“... it would come back saying, "This test is

probably not for you." I'm putting the right

figures in aren't I?." Implant user 07_ICG

"Now, not all our patients can do a hearing check. You have to have a reasonably good speech perception score to do

this particular test. [….] I think what would have been really nice is to maybe have two different tests in the app, or even

maybe three, so all patients can actually do some kind of hearing check. I think from an audiologist point of view, some

of us actually struggled with the hearing check. […..] The speech test used by CHOICE was quite new to us. [….] If the

patient did badly on a hearing check on CHOICE, we were not always sure is it because it's a different test, or is it

because they're really struggling in their life?" Staff member 01_ICR


Maintenance of implant

CHOICE offers a dedicated section for ordering implant spares. This also includes reminders, for example, to

change microphone covers. The interviews indicated the complexity of ordering replacement parts due to

different implant manufacturers and their different types of implant models along with upgrades and

improvements. Also, implant users wanted to have the option to specify the colour when ordering certain

parts.

For implant users

There was some confusion for implant users in using the spares option because some were asked to go directly

to their implant company. Some implant users welcomed reminders to change parts, however, some had

issues with getting the wrong part. There was some discussion by both implant users and staff on the best

approach to ordering parts online via the CHOICE platform.

For staff

The added value of ordering via CHOICE rather than directly from the centres’ spares and repairs service was

to provide a service with greater efficiency to implant centres with easy access to implant users. However, the

online ordering service struggled given the technical complexity of implants and implant manufacturers.

Overall, for some centres it was not good enough.

However, there were benefits post Covid-19, for repairs, to illustrate the potential of an online service.

Provision of rehabilitation tools One key value-added component of CHOICE was the provision of rehabilitation tools to encourage implant

users to improve their hearing performance. Although timing of accessing these tools in the user implant

journey, as mentioned previously, was important.

For implant users

Use of the rehabilitation tools was variable. Generally, this was based on whether they were very experienced

users with up to 10, or even 20, years post implantation. Those with less experience might choose to use other

rehabilitation software. Some remarked CHOICE would have been more useful earlier following implantation.

However, some implant users did value the availability of the rehabilitation tools. They were either early

implant users or those who wanted to improve certain aspects of their hearing performance, for example

music or telephone training.

“Our repairs department is very, very busy. Completely overwhelmed every day to the point where staff are just burying

their head in the sand a bit. We need something like CHOICE, but it's not just quite good enough for what we currently

need. We couldn't swap. That's what I wanted to be able to do. I want to be able to say let's drop our system and let's

pick up CHOICE. It's not good enough." Staff member 06_ICR

“It's created efficiencies, I think in terms of we've had appointments in normal times prior to COVID, I think these patients

would have asked me before an appointment to come in to change a T mic, to change a cable, whereas there's now, and I

know personally I've emailed a video link from YouTube telling them how to do things and they've managed to do it at

home, whereas before we probably wouldn't have thought to do that, we would have just brought them in and changed it,

so I think it's been more efficient in terms of repair appointments.” Staff member 03_ICR

“It's been quite strange that you click on something, I needed some new microphone covers, and you go down and you

click on it. The microphone covers arrived through the post. It feels strange almost, but it's so reliable." Implant user

02_ICG


There were problems for some who wanted to use RealSpeech, one of the rehabilitation tools, if they were

accessing CHOICE with Apple devices because of its Windows 10 requirement. Finally, for some implant users

accessibility would be improved if it were an app. However, certain aspects functioned well for some users,

e.g. the music training.

It should also be noted that many implant users did not need to use CHOICE frequently.

For staff

One site champion expressed the opinion that some implant users have low expectations of their implant and

limit training to improve their hearing performance. CHOICE was seen as a means of encouraging these users

to improve their hearing performance by having the option of using the rehabilitation tools provided.

In addition, staff would welcome some feedback on implant user progress or whether they were accessing

these tools on CHOICE.

Function of the staff dashboard Staff raised limitations with the dashboard of CHOICE. They felt it required smoother navigation and

improvements to drop down menus with more response options to clarify follow up actions. Lack of

interconnectivity with the implant centres’ patient management systems created more work for some implant

centres. CHOICE used NHS numbers to identify implant users whereas audiologists identified users by name, so

cross referencing to their patient information systems was necessary and added another step in the process.

In summary, the functionality of CHOICE to monitor, maintain and rehabilitate implant users showed potential

for both implant users and staff, affording some utility. However a crucial tool, the TDT hearing check, did not

fulfil this potential and led to frustration for many staff and implant users.

“The real speech listening practice ranges from easy to difficult where the

audio is played. This is perhaps a lady or gentleman reading some poetry

or a story but then the background sound is propped with street noises or

cafeteria or general hubbub. You have to guess or attempt to say which

particular bubble with speech in it presents what you think is the answer

for each particular word. I found that particularly helpful. It was a bit of

a trial at first but I'm getting more successful the more I use it. That's

helping." Implant user 01_ICG

“I have actually tried your music app and I

rather liked it. The fact that you could

move the gauge in the various

instruments and the vocalist. Thought I'd

give it a go with something I hadn't heard

before, and I found it quite fun actually."

Implant user 06_ICP

“Not that much, I think I try to use it. I just check in every couple of months or so. It's not something that is embedded into

my routine, if that makes sense. It's not something that I think about. Just because, I think, I'm used to having one

appointment every year or eighteen months.” Implant user 01_ICR

“Some people are just happy that they can hear.

Whereas if only we could persuade them to use

those rehab tools on CHOICE, they have potential

to actually increase their abilities. It's amazing

how many people don't. They're just satisfied

with the here and now." Staff member 06_ICR

“I think a lot of people are using those rehabilitation tools. I'm not

sure if you know, correct me if I'm wrong, but I think we can't

necessarily see when they're using it. That would be quite useful

as well for us to know. "Oh, this person has been doing the music

training, and this is how they're doing it". At the moment, I think

it's just an open forum, in a way, so we don't get any feedback."

Staff member 02_ICG

" I think the clinician portal…[when] you're dealing with it there's not as many options. It's having more options would

actually be super helpful. I think it would just give you more of an idea about what has actually happened. Whereas it's

got all of the outcomes are like it's been done but it might have been stock is sent or just a few more options from what

you can record has happened". Staff member 01_ICG


3.3 HOW CHOICE WAS INTEGRATED INTO PRACTICE

The integration of an innovation can be dependent on the stakeholders involved in adoption, the

characteristics of the adopting organisation, and the processes undertaken to embed the innovation (see

NASSS domains 4, 5 and 7 in Appendix 1). Furthermore, integration requires a range of implementation work

to embed the innovation (see NPT domains in Appendix 2).

Staff completed the NoMAD survey about the implementation of CHOICE within their implant centre. The

findings provide an important overview of implementation activity.

Figure 11 highlights three key questions from the NoMAD survey. The combined view across all implant

centres indicated a slightly improved sense of familiarity with CHOICE after it began. However, there was only

a moderate sense that CHOICE was currently part of their normal work, and a decreased sense about whether

CHOICE would become part of routine care in the future.

Figure 11 NoMAD baseline and follow-up scores for all centres

These views were stable across all centres apart from the Southampton implant centre (home of CHOICE and

the project team) where CHOICE was more familiar. Staff at Southampton expected to routinely use CHOICE

more in the future compared to all other centres combined (see Figure 12).

The NoMAD survey is designed to measure the four domains of NPT (Coherence, Cognitive Engagement,

Collective Action, Reflexive Monitoring) and is described in Appendix 2. These domains assess staff ability to

mobilise, organise and engage in changing practice as well as appraise and reflect on changes made with the

ambition to embed change as a routine practice. Figure 13 highlights little change over time and some small

differences between domains.

Key messages

4. CHOICE offered a range of tools to monitor and improve hearing performance, as well as provide

a portal to order spares, which had some success with both implant users and staff.

5. CHOICE enabled easier communication between staff and users, however, limitations with the

staff dashboard created additional work for some implant centres.

6. A key component for the success of CHOICE, the Triple Digit Test, did not fulfil expectations

because of its unsuitability for many implant users and its unreliability for those that were able to

use it leading to its eventual withdrawal.


Figure 12 NoMAD scores comparing Southampton with the other centres

Coherence and Cognitive engagement generally scored higher than Collective Action and Reflective monitoring

domains. This suggests staff had a fair idea of ‘what CHOICE was and was for’ (Coherence) and ‘who they

needed to involve to use it’ (Cognitive Engagement), but they had less clarity/confidence about ‘how to

integrate CHOICE into their context’ (Collective Action) and ‘limited feedback/monitoring mechanisms to know

if CHOICE was working’ (Reflexive Monitoring).

Figure 13 NoMAD domain scores for all centres

In interviews, staff and users described a range of issues related to the integration of CHOICE at implant

centres. These contribute to explaining the limited impact described in section two. The integration of CHOICE

at implant centres can be understood at two levels: firstly, the level of implementation support from the

CHOICE project team leading the study, and secondly, independent decisions made by implant centres on how

to integrate CHOICE.

Level of implementation support from the CHOICE project team Considerable delays between study approval (almost a year until the first centre, Southampton, started

recruiting and 18 months until all seven implant centres starting recruiting to CHOICE) meant the decision to

provide early training on CHOICE to centre staff was mistimed and led to a loss of staff enthusiasm.

"The problem was, for me, is that we did all these training sessions, but there were quite a few delays for CHOICE to be

launched. By the time we launched, you almost forgot about the basics of it, and then you had to go back to those

resources…not everybody had the time to do that." Staff member 02_ICG

"The problem was, for me, is that we did all these training sessions, but there were quite a few delays for CHOICE to be


Several mechanisms existed that could have supported integration of CHOICE at centres, such as using the

experience of the Southampton team and the breadth of experience in the Steering Group. However, most

centre staff reported limited support with integrating CHOICE. They indicated the Steering Group was a

‘sounding board’ and did not provide detailed guidance to support integration. Mixed views on the value of

the site champion meetings were reported by site champions, many describing the forum as unstructured and

not supporting integration. However, the Chief Investigator from the Southampton team was reported as an

enabler for implementation and provided highly responsive replies when approached by centres.

Importantly, the study protocol stated each centre could choose how to offer CHOICE to implant users. This

decision by the project team to devolve the work required to integrate CHOICE into centres constrained

implementation. An implementation gap existed, from which several challenges emerged.

Integrating CHOICE into centres became heavily reliant on CHOICE site champions. Most champions were

allocated clinical time, albeit not a lot – approximately half a day a week – to manage all aspects of CHOICE and

did so diligently. Many site champions were solely responsible for all activity related to CHOICE and this

bottleneck of implementation work relied on site champions’ confidence to integrate and manage CHOICE,

their creation of local operating procedures, their development of local training on CHOICE, and their

prompting of other centre staff to engage with CHOICE. Where this happened, recruitment to CHOICE was

higher.

Integration relied on the availability of duty audiologists and administrative staff to recruit and manage actions

from the CHOICE platform. Where this was supported, recruitment to CHOICE and management of CHOICE

actions was better.

Similar to the implementation gap between the CHOICE project team and implant centres, was the

implementation support gap between the CHOICE project team and platform developers. Integration of

CHOICE was affected by delays to fix IT issues, the different nomenclature of audiology and platform

development staff, and a failure to manage the interdependencies between the stakeholders, as described

below.

Decisions made by implant centres As the responsibility for integration was predominantly in the hands of individual implant centres, contextual

factors and local ways of working were highly influential and centre-orientated decisions affected how CHOICE

was implemented.

Considerable delays, usually many months, occurred in all implant centres – related to initially agreeing to

participate in CHOICE, obtaining centre approval, obtaining local ethics approval, signing the contract to

"We had some meetings early

on, but it would have been nice

if that was maybe a bit more

continued throughout." Staff

member 04_ICG




continued throughout."















"We've probably recruited more

in the last couple of months

because what we've done is

targeted our annual review

waiting list." Staff member

07_ICG





waiting list."





waiting list."





waiting list."





waiting list."

"We have duty audiologists, so everyone is on a duty for two hours of the day in the morning, and one hour in the

afternoon and it's their job to deal with all patient queries." Staff member 01_ICP


afternoon and it's their job to deal with all patient queries."









"I found it very difficult at the beginning ... to make decisions about things and I didn't even know what the words meant.

Very kind of techy words. I just had no idea what they were talking about, but it was things that I had to make decisions

on." (CHOICE Chief Investigator)

"I found it very difficult at the beginning ... to make decisions about things and I didn't even know what the words

meant. Very kind of techy words. I just had no idea what they were talking about, but is was things that I had to make

decisions on." (CHOICE Chief Investigator)


meant. Very kind of techy words. I just had no idea what they were talking about, but is was things that I had to make

decisions on." (CHOICE Chief Investigator)


"We have two companies that have done the development. One did the hearing check and the other one did everything

else. That's caused some difficulties along the way. Obviously the company that didn't do the hearing check still needed

to integrate that information from the hearing check into the cohesive CHOICE app. It was difficult having two

companies involved, but that's the way it happened because we already had a hearing check ready to go" (CHOICE Chief

Investigator)

"We have two companies that have done the development. One did the hearing check and the other one did

everything else. That's caused some difficulties along the way. Obviously the company that didn't do the hearing check

still needed to integrate that information from the hearing check into the cohesive CHOICE app. It was difficult having

two companies involved, but that's the way it happened because we already had a hearing check ready to go" (CHOICE

Chief Investigator)

"I was more or less left to my own devices.

To get up to speed, I just had meetings with

our local site and asked questions. I did a lot

of self-teaching, reading the protocol, things

like that." Staff member 04_ICY





like that."





like that."





like that."





like that."


proceed, and recruiting implant users onto CHOICE. These delays negatively affected staff enthusiasm for

CHOICE and slowed the recruitment of users.

As mentioned in 3.2, each implant centre had existing systems to manage, order and deliver component parts

to users. Staff and users reported confusion about who to contact about repairs, either CHOICE or their centre.

The lack of clarity about how this should be integrated at centres affected the perceived value and workability

of the CHOICE platform. This led to several centres taking the decision to ask their users to only use the

existing repairs service at their centre.

None of the implant centres attempted to interoperate the CHOICE platform with their existing patient

management systems. The two systems operated separately and thus required human work to link

information that was already on, generated from, or required to be on, both systems. This additional work was

generally considered a hindrance to implementation and the perceived value of CHOICE.

All implant centres decided to prioritise their existing care pathways whilst adopting CHOICE. This was largely

due to CHOICE being considered ‘a research project’ and not a fundamental or mandated change in how

centres operate. Staff held the general opinion that CHOICE ‘should or would’ save annual review

appointments. However, many staff were unclear when or how CHOICE should save annual reviews,

particularly as many staff took the view that the implant user had a say in that decision as well. It appeared

some centres engaged with CHOICE as an alternative to face-to-face first or second annual review

appointments, but there were considerable staff concerns about the value of all the required CHOICE work to

avoid one or two review appointments – which they perceived might need to happen anyway for legitimate

reasons.

Many implant centres promoted several remote platforms, in particular Remote Check for users with certain

devices made by the company CochlearTM. Whilst CHOICE and Remote Check are different, the former patient-

led and the latter clinician-led with more technical capabilities for checking/testing a device, it was clear

implant centres were reluctant to have CochlearTM users engaged with both platforms. Generally, it was

reported that people using devices made by CochlearTM were better served by Remote Check and one centre

purposefully promoted Remote Check before CHOICE as most of their caseload were CochlearTM users.

“A problem I've got at the moment is that cell rack isn't working…that prompted my question, "Should I go through you,

or should I go through [CHOICE project team]?" My audiology department said, "We'd prefer if you go through us." That

was a blurred line for me because I don't know if I should also be contacting [CHOICE project team] as well…I'm still not

clear about where [CHOICE project team] fit in with my life in comparison with my centre.” Implant user 02_ICG




clear about where [CHOICE project team] fit in with my life in comparison with my centre.”





















"We haven't phased out existing pathways because of, obviously, the COVID situation as well. We're keen to just make

sure we don't lose anybody. I think down the line, it will be looking as if CHOICE is going to replace some of our normal

review systems. It will be making sure that we’ve got a way of identifying which system each patient is following. We

can run reports to do that, but we may need to keep a separate list or something, which will be just an admin

person…it’s going to take a bit of time to set up." Staff member 03ICY


sure we don't lose anybody. I think down the line, it will be looking at if CHOICE is going to replace some of our normal



person…it’s going to take a bit of time to set up."




















"This is an important reason why the project didn't work in our department. We decided to introduce CHOICE only after

12 months from implantation (of the cochlear implant). The current pathway is for patients to then come in for annual

reviews. They will still come for their first review. They still need to come for their third annual review because they need

to see the surgeon. They still need to come for their five, ten, fifteen, twenty year reviews, because that is when they are

'upgraded' with new equipment. I think if I was a manager, I probably would not be committed to CHOICE because of all

of this extra work .. you are just dropping one appointment. I can see why (staff) didn't buy into it." Staff member 01_ICP



reviews. They will still come for their first review. They still need to come for their third annual review because they

need to see the surgeon. They still need to come for their five, ten, fifteen, twenty year reviews, because that is when

they are 'upgraded' with new equipment. I think if I was a manager, I probably would not be committed to CHOICE

because of all of this extra work .. you are just dropping one appointment. I can see why (staff) didn't buy into it."



reviews. They will still come for their first review. They still need to come for their third annual review because they

need to see the surgeon. They still need to come for their five, ten, fifteen, twenty year reviews, because that is when

they are 'upgraded' with new equipment. I think if I was a manager, I probably would not be committed to CHOICE

because of all of this extra work .. you are just dropping one appointment. I can see why (staff) didn't buy into it."

"CochlearTM have got that Remote

Check…we’re signposting patients with

CochlearTM devices towards that rather

than CHOICE just because of the other

functionality involved with that." Staff

member 02_ICY

"Cochlear have got that Remote

Check…we’re signposting patients with

“In our centre the majority of patients are CochlearTM and we have fewer

AB and MED-EL. Cochlear's Remote Check is always the first option…if

they have the correct internal device, have the correct processor, we will

talk about Remote Check first because it’s suitable for them and it can

give us a little bit more information. In the beginning of CHOICE, we

would only be offering CHOICE to our AB MED-EL patients.” Staff member

05_ICG

"“In our centre the majority of patients are Cochlear and we have fewer


Furthermore, one centre took a decision that all but stopped integration and recruitment to CHOICE. Due to a

national request to replace faulty electrodes for Advanced Bionic devices, CHOICE recruitment was put on hold

for several months and then only partially re-started after the first national Covid-19 lockdown, with

replacement appointments continuing to be the priority for the centre.

The outcome of these various pathway decisions was a reduced number of users eligible to be introduced to

CHOICE at many implant centres.

Despite these constraining implementation factors, CHOICE was eventually adopted by all seven centres, albeit

not recruiting at the expected levels and with a range of integration challenges to manage as recruitment

progressed. CHOICE was integrated to varying degrees and this was due to the enabling factors of senior staff

support for CHOICE, site champions’ decisions to manage most CHOICE introductory conversations and

actions, and implant centre decisions to be responsive when they received user queries via the CHOICE

platform.

Some integration of CHOICE into centre processes was observed, including linking to repairs staff, introducing

CHOICE at 3-month reviews and discussing CHOICE at routine staff meetings. However, a range of integration

issues remain unaddressed (see Appendix 4).

"You may or may not be aware of the electrode concern they had earlier this year, [surgical replacements] are

dominating our schedule…we have such limited clinical capacity, where possible we’re using Remote Check to track

functional hearing over time. It’s not worth it [introducing CHOICE alongside Remote Check]…if a patient only has an hour

in the month that they’re able to do this for, we would prefer them to do Remote Check because it gives us much more

clinically useful information compared to CHOICE." Staff member 04_ICY

" You may or may not be aware of the electrode concern they had earlier this year, [surgical replacements] are

dominating our schedule. For our patients who either show early signs of an electrode fault, or who we’ve reset their

settings so that their performance is good again, we’ve asked them to enrol with Remote Check…we have such limited

clinical capacity, we’re using that to track functional hearing over time. It’s not worth it [introducing CHOICE alongside

Remote Check]…if a patient only has an hour in the month that they’re able to do this for, we would prefer them to do

Remote Check because it gives us much more clinically useful information compared to CHOICE."

14.

























Key messages

7. Familiarity with CHOICE improved over time, but due to a range of challenges with

implementation, it was never strongly considered to be routine work.

8. The ability of implant centres to fully embrace CHOICE as a clinical pathway option was affected by

limited implementation resources, a long gap between training and recruitment and heavy

reliance on local 'site champions'.

9. Lack of involvement of the platform software developers in strategic decisions limited the ability

to adapt and improve CHOICE in a timely way.

10. Integration was also hindered by the lack of interoperability between CHOICE and patient

management systems and by not incorporating CHOICE into routine clinical pathways.

11. CHOICE enabled email communication between staff and users, however, limitations with the staff

dashboard created additional work for some implant centres.


3.4 HOW PREFERENCES AND CULTURE INFLUENCED IMPLEMENTATION

The uptake of an innovation can be dependent on the stakeholders involved in adoption and the adopting

organisation culture (see NASSS domains 4 and 5 in Appendix 1).

It was apparent from the staff and user interviews that users had considerable influence on the success of

CHOICE, as uptake required their consent and engagement. As seen in previous sections, uptake of CHOICE by

users was low and this can be partly explained by users’ preferences for audiology consultations. Mixed views

were reported about remote care. Many reported a preference for face-to-face appointments, including for

better technical and emotional support, confidence building support, avoidance of misunderstanding in email

communications, and better social connections with the implant community.

However, many implant users reported a preference for a balance in consultation options.

An interesting contrast to this position, reported as an enabler to implementation, was Southampton centre

users’ perceived ‘duty’ to participate in CHOICE. Considerable loyalty to their centre, and innovation promoted

by their centre, was expressed by these implant users. This helps explain why their registrations to CHOICE was

highest and usage one of the highest (see table 6).

Staff and implant centres also reported mixed views about remote care and this likely explained low uptake of

CHOICE. Interviewed staff reported ‘younger’ centre staff were more engaged with remote care generally and

with CHOICE. ‘Older staff’ were considered less engaged, less willing to integrate CHOICE into routine clinic

ways of working and expressed a preference for face-to-face consultations. This indicative finding would

warrant further exploration.

It was reported that the prevailing culture across many audiology settings was for face-to-face contact.

However, with the recent changes in NICE guidance which widened eligibility for implants and will increase

activity at audiology centres, staff did acknowledge the need for remote care options to release the pressure

of demand for services.

A less discussed area of influence was the commissioning model for cochlear implant care. Only two staff

referred to it but indicated its negative influence on CHOICE recruitment.

"I don't think I'm at the stage where this [CHOICE] would

replace an appointment for me…so it [CHOICE] would be

in addition to. I mean, it’s great, but obviously it’s not

got everything that you would normally have at the

appointment, I would still prefer to have the

appointment." Implant user 01_ICR

"I don't think I'm at the stage where this [CHOICE]

would replace an appointment for me…so it [CHOICE]

would be in addition to. I mean, it’s great, but obviously

it’s not got everything that you would normally have at

the appointment, I would still prefer to have the

appointment."






appointment."






appointment."

"I don't think I'm at the stage where this [CHOICE] would

replace an appointment for me…so it [CHOICE] would be

in addition to. I mean, it’s great, but obviously it’s not

got everything that you would normally have at the

appointment, I would still prefer to have the

appointment."




"I seriously consider the best care and attention for me is a

personal one-to-one consultation with the excellent staff I

meet at [implant centre]. This involves a helpful discussion

addressing problems and avoids misinterpretation. The

personal contact can also be a boost to one’s confidence,

which is sometimes required." Implant user 04_ICR






which is sometimes required."





















"If everything is working properly, then I’m quite happy to do this [CHOICE] kind of remote checkup. If I’ve got a query or I

know that there’s something not right I would want to go to my implant centre. It’s nice to see them, they’re a very

friendly bunch and it is pleasant to actually just go and be seen properly and they look at all the mechanics of [implant]

and check it thoroughly and tell me that it’s fine. They can’t do that remotely so easily. I think it’s a bit of a mix, I like the

remote [CHOICE] to a point, I like going there to a point, I think both complement each other and we’ve got the option.”

Implant user 05_ICG

"“If everything is working properly, then I’m quite happy to do this [CHOICE] kind of remote checkup. If I’ve got a query

or I know that there’s something not right I would want to go to my implant centre. It’s nice to see them, they’re a very



remote [CHOICE] to a point, I like going there to a point, I think both complement each other and we’ve got the option ".











"If everything is working properly, then I’m quite happy to do this [CHOICE] kind of remote checkup. If I’ve got a query or I

know that there’s something not right I would want to go to my implant centre. It’s nice to see them, they’re a very




"If we can get any proportion of them using it, that then means we don't have those patients who need those

appointments...any reduction is helpful." Staff member 03_ICY


appointments...any reduction is helpful."


appointments...any reduction is helpful."

"We really need CHOICE to fit with the commissioner’s

point of view…if you’re not offering users an annual review,

we need to make sure that users fill in the review

questionnaire on CHOICE, because that shows you have

had contact and can charge the commissioners for

it…that’s why in the beginning we did a lot of

deregistration of users." Staff member 01_ICP

" We really need CHOICE to fit with the commissioner’s

"We need to really think about is how we bill any

interaction with CHOICE because normally, you see

a patient face-to-face, you can bill a hospital for

that and you can get money for that

appointment…it’s something in the future will need

to be discussed and it will need to be agreed on."

Staff member 02_ICR

"We need to really think about is how we bill any

interaction with CHOICE because normally, you see


Similarly, a less discussed but relevant aspect of broad implant centre culture was the use of

platforms/support tools developed by major implant suppliers. The wide availability of online rehabilitation

support also reduced CHOICE usage, as described by this implant user.

Clearly, the proliferation of other support platforms cannot be predicted nor controlled, however, the value

proposition of any new platform should consider the market gap it would fill. Many staff reported they see

individual device manufacturers developing their own, like Remote Check, online remote care platforms. The

implications for CHOICE could be to re-evaluate its value proposition, improve its remote hearing check, and

target its support at users with rehabilitation needs.

3.5 HOW USERS WERE RECRUITED

Constraints to recruiting CHOICE users outweighed the enablers, contributing to lower recruitment than

planned in all centres (NASSS Domain 4, Appendix 1). A range of key factors impacted on recruitment. The

programme had a set of recruitment criteria, and individual centres and staff added their own. There was a

lack of consensus as to when users should be recruited and centres differed in the recruitment strategies they

deployed (e.g., in-person recruitment or mail outs). Not all users invited to use CHOICE took up the offer, in

line with the mixed views on remote care described in the previous section. Inevitably the Covid-19 pandemic

affected recruitment and this is explored in the next section.

The following table indicates that active recruitment for each centre was between 10 and 19 months duration,

including the periods in which recruitment to CHOICE stalled as a result of Covid-19. There is no relationship

between recruitment time period and number of users recruited, indicating other factors were more relevant.

Table 9 Recruitment phase by centre and total users recruited3

Centre First User

registration

Number

recruited by

26/1/214

Months

recruiting

Months unable

to recruit due to

Covid-19

Southampton 11/06/19 65 19 2.5

Manchester 29/10/19 44 15 3

North East 11/02/20 74 11 0

Cambridge 11/12/19 19 14 4

Nottingham 11/06/19 16 19 4

Royal National ENT and Eastman Dental Hospital 04/03/20 31 10 5

St Thomas’ Hospital 09/03/20 25 10 6

3 First user registration dates sourced from the CHOICE app data. 4 26/1/21 was the date that data collection stopped

"If I wanted to support, as part of the [implant social group], I confess I’d send them towards the Advanced Bionics rehab

software, which does tend to be quite detailed…because they have such a wide suite of options in the rehab software

that Advanced Bionics run." Implant user 04_ICR

Key messages

11. Implant users held mixed views on the general desirability of remote care. Not all, but many, reported

a preference for face-to-face appointments for better support and social connection.

12. Younger centre staff were reportedly more engaged with remote care generally and with CHOICE.

Older staff were considered less engaged, less willing to integrate CHOICE into routine clinic ways of

working and expressed a preference for face-to-face consultations.

13. Less cited factors such as the commissioning model for cochlear implant care and wide availability of

online rehabilitation support were likely to have impacted the scalability of CHOICE.

"If I wanted to support, as part of the [implant social group], I confess I send them towards the Advanced Bionics rehab


that Advanced Bionics run."

Key messages








17.

Key messages








"If I wanted to support, as part of the [implant social group], I confess I send them towards the Advanced Bionics rehab


that Advanced Bionics run."

Key messages





Key messages

12. Implant users held mixed views on the general desirability of remote care. Not all, but many,

reported a preference for face-to-face appointments for better support and social connection.

13. ‘Younger’ centre staff were reportedly more engaged with remote care generally and with

CHOICE. ‘Older’ staff were considered less engaged, less willing to integrate CHOICE into routine

clinic ways of working and expressed a preference for face-to-face consultations.

14. Less cited factors such as the commissioning model for cochlear implant care and wide

availability of online rehabilitation support were likely to have impacted the scalability of

CHOICE.


Recruitment inclusion and exclusion criteria

The study protocol set out eligibility criteria (see Supplementary material 5). Not all people with a cochlear

implant could score on the hearing check. Although it was not a definite relationship, it appeared that those

who scored 50% or more on the BKB sentence test were more likely to be able to obtain a score on the hearing

check. Some staff reported that this additional requirement limited the number of cochlear implant users for

whom CHOICE was considered suitable. In addition, there was an initial criterion to target users at one year

post implantation as the most relevant time to introduce remote care into their pathway. This one-year post

implantation criteria was removed in response to the Covid-19 pandemic, to enable recruitment any time

post-surgery.

However, individual staff and centres used a range of other criteria when recruiting that were in addition to

the original programme criteria. The most prominent of these, used across at least four centres, was the

identification of ‘tech savvy’, computer literate users. Another was selecting those that were ‘good

performers’ (BKB score of 50% or more) at utilising the implant to its full advantage. In section 2.2, it is noted

that those completing the R-Outcomes survey had a high level of digital confidence. Selection of technically

confident users by staff presents a possible explanation for this finding. In addition, users were excluded for

multiple reasons which included level of anxiety, visual impairment, and users who were known to limit the

use of their implant.

Nevertheless, the 50% BKB scoring for the Triple Digit Test hearing check remained a key constraint on

deciding when users were appropriate for CHOICE.

When to introduce CHOICE Staff deliberations also involved determining the right time in the user's post implantation pathway to invite

them onto CHOICE. This was a key decision staff and implant centres needed to make. Views varied between

staff as to whether early as possible post implantation was better or to wait until implant users were

comfortable in their ability to hear and were implant confident. This was expected at 9 -12 months post

implantation. One staff member summed up the complexity of timing decisions in response to when to

introduce CHOICE to the implant user.

Due to NICE (March 2019) broadening the eligibility criteria for a cochlear implant, those with more residual hearing were able to have an implant. The implication of this is that those with residual hearing may be able to access CHOICE earlier,

Recruitment strategies

Different recruitment strategies were undertaken by implant centres. These ranged from via clinic

appointment, email or letter mail outs, and an in-person user group training session. Recruitment could either

be quite active by recommending CHOICE as an option or passive by passing information to the implant user

"This is something that I was surprised about. It was initially sold as something that we could use for any and all patients

with and implant, and its very much not. It was an unwritten exclusion criteria, that people had to have a certain score

when they were doing their testing. That is surprisingly hard to get ... in actual fact, it needs be for us, something along

the lines of a 70% BKB score". Staff member 04_ICY




the lines of a 70% BKB score".














"Since we’ve opened it up to more people, there are people signing up to CHOICE who have only had their implant for a

couple of months, where they’re not really at the point where we would even be expecting much from doing hearing

checks, and they also might not need the spares. If they’ve not had it for very long, the use of CHOICE is limited to what

they’re going to need it for. Having it for once you’ve passed the year mark, which is what the original intention was of

how we were using it, was people from 12 months onwards, it shouldn’t then, from that point onwards, affect how

they’re using it." Staff member 01_ICY






they’re using it."






they’re using it."

" …because the criteria for who is suitable for a cochlear implant or who will get funding for a cochlear implant has

become less strict. We are seeing people, particularly, we're seeing adults that have more hearing now, but are still

eligible for a cochlear implant than we were in the past. What that means is that normally, some of them will have

some good residual hearing before they're implanted. Often for them, their adjustment period is much quicker and they

will gain more benefit overall and they will get to that level quite quickly." Staff member 01_ICG

" …because we’re only implanted one ear and they have good hearing…on the other ear, even after one month, they’re

able to communicate well fact to face”.

" …because we’re only implanted one ear and they have good hearing…on the other ear, even after one month, they’re


when the opportunity arose. The Covid-19 pandemic inevitably influenced recruitment strategies because in

person attendance at clinic was not available to most and was used for urgent cases only. So, some centres

moved to mail outs from previous in-person recruitment.

Identification of eligible implant users was hampered for some by their patient management systems and

required a level of effort further constrained by busy workloads. See Table 10 for examples of strategies used

to recruit implant users to CHOICE.

Table 10 Recruitment strategies used by centres

Centre Strategies used by centre

Southampton Recruited in clinic and through a mail out (post Covid-19), either by email or

post.

A range of recruitment strategies used to engage staff in recruitment e.g.

reminders and stickers on monitors.

Manchester Identifying eligible users on patient management system, in clinic and by

emailing.

North East All eligible users invited by two single bulk mailshots, either by email or post.

Cambridge Sent out pre-made email templates with CHOICE information once there was

an initial interest at an appointment.

Sent out regular email reminders with patient criteria to staff team.

Nottingham Recruitment poor so undertook a mailshot (letter) approach to recruitment –

this had some initial success followed by a poor response to subsequent

mailshots.

Royal National ENT and

Eastman Dental Hospital

Initial recruitment invited 10 people to come to centre for a training session.

Recruited 9 of these successfully but following Covid-19 unable to do face to

face training session. Developed CHOICE packs for staff to distribute to users.

St Thomas’ Hospital Recruited by face-to-face only and deliberately decided against a bulk mail out

approach.

Recruitment strategies were either targeted, opportunistic, persuasive or passive. Implant centres adapted

strategies, following Covid-19, from in-person recruitment to email invite. Different active approaches to

getting implant users onto the app were initially successful for some sites, including mass mailout, however,

overall recruitment remained poor. This suggests other factors such as appropriate targeting of implant users

in their pathway may have stalled effective recruitment. Site champions were able to gain advice on

recruitment from a champion support forum. Feedback from the champion interviews suggests there was no

formal implementation guidance disseminated beyond the primary site at Southampton to facilitate

recruitment.

Research project effect on recruitment CHOICE was introduced to implant centres as a research project and this influenced how it was implemented.

Staff reported mixed effects related to the ‘starting status’ of CHOICE. A small number of staff felt that

implementation of CHOICE benefitted by being a research project because this gave a level of priority to its

delivery. However, some staff were less engaged because as a research project it did not have the same

motivating influence as the roll out of a mandated service. Moreover, the status of CHOICE as a research

project meant considerable ethical and governance arrangements were required. This complexity at the start

of CHOICE was a burden to implementation, as described by this staff member:

"From a research side, you have to deal with every individual hospital's internal research department and information

governance for that, but then the Trust-wide ones are also much more strict depending on where you are. We have a

particularly difficult one where it requires a lot of paperwork, to update the things, requires paperwork, and months of

preparation even if it's just a minor amendment. If this is something that needs to go through hospital governance, then

it's going to be a challenge to make sure that we're all up and running on the same servers at the same time.” Staff

member 04_ICY

"From a research side, you have to deal with every individual hospital's internal research department and information

governance for that, but then the trust-wide ones are also much more strict depending on where you are. We have a


Overall, when considering the nearly year long (January to November 2018) timeframe to gain ethical approval

for CHOICE (see section 1) and the fact that CHOICE was not a mandated service transformation initiative, it is

clear the impetus for implementation was constrained.

3.6 HOW COVID-19 IMPACTED IMPLEMENTATION

Successful implementation of an innovation can be dependent on favourable conditions surrounding the

innovation rollout site. The wider context can often be an important and difficult to manage set of influential

factors (see NASSS domain 6 in Appendix 1).

The impact of Covid-19 on CHOICE could not have been predicted and several effects were reported by staff

and users. Covid-19 initially acted as a logistical constraint to CHOICE recruitment but later acted as a co-

enabler, alongside CHOICE, for cultural change within audiology settings toward remote care.

Following the first national lockdown from March 2020, staff initially reported that Covid-19 increased the use

of the CHOICE platform by the users who were already registered. However, having to stop face-to-face

appointments had the effect of stopping the recruitment of new implant users to CHOICE in six of the seven

centres for between 2.5 and 6 months (see table 9 above). The compounding impact on the CHOICE

programme was that it had only achieved all seven centres being open for recruitment a couple of weeks

before the first lock-down. Unfortunately, this meant that at the time remote models of care were

accelerating across the NHS, recruitment of new CHOICE users stalled.

However, over time the need to remotely manage audiology activity influenced centres to work in new ways.

Recruitment activity was adapted at one centre, and some centres expressed a change in approach to offering

CHOICE as a ‘must have’.

It was apparent from staff and user interviews that methods for remote care were being considered and

welcome, as a necessity to manage centre activity during the first Covid-19 national lockdown.

"We know that historically, similar projects that weren't usually research projects, have been very difficult to put through.

That definitely was an advantage for CHOICE." Staff member 02_ICG


That definitely was an advantage for CHOICE."











"Everyone started using it by October time, and then of course in March, we went into lockdown. The moment we went

into lockdown everyone stopped using CHOICE." Staff member 01_ICP


into lockdown everyone stopped using CHOICE."





"Covid-19 probably worked in our favour…we had a period

of three days where we weren’t seeing patients because

we were waiting to get the PPE that we needed. I had a

block of time where I could sit and print out invite letters. I

think that’s why our recruitment improved so much, we

just did a bulk mail out to everybody who met the criteria."

Staff member 03ICP

"COVID probably worked in our favour…we had a period of

three days where we weren’t seeing patients because we

were waiting to get the PPE that we needed. I had a block

of time where I could sit and print out invite letters. I think

that’s why our recruitment improved so much, we just did

"To start off with, it was more, ‘Would you be interested

in this?’ As Covid-19 and the lockdown happened, we

were then ‘We’d like you to do this.’ We were changing

what we were saying because it would give us more

information than we could get whilst we were unable to

see patients." Staff member 05_ICR

"To start off with, it was more, ‘Would you be

interested in this?’ As COVID and the lockdown

happened, we were then ‘We’d like you to do this.’ We

were changing what we were saying because it would

give us more information than we could get whilst we

were unable to see patients."

Key messages

15. Recruitment of users to register on CHOICE was well below programme expectations.

16. Decisions on when to target users in their pathway and other selection criteria used by staff,

along with the stalling of recruitment due to the COVID pandemic, were major constraints on

registering users to CHOICE.

17. The impetus for implementation at implant centres was constrained by its status as a research

project.


Later in 2020, staff reported an enabler to CHOICE implementation was changing NHS Trust policies on

digital/remote care. As six of the seven participating centres are hosted by NHS Trusts, these policies are highly

influential.

In response to the Covid-19 pandemic and its impact on access to routine clinic appointments, ethical approval

was obtained in June 2020 to offer CHOICE to all adult CI centres in the UK. Six additional sites expressed

interest in adopting CHOICE but due to concerns about the hearing check component, these sites did not

proceed and were unable to be included in the evaluation.

Summary The scalability of CHOICE was significantly affected by low perceptions of its value to users and staff in

practice, problems with the functionality of the technology and lack of integration into clinical pathways at

implant centres. Influential factors were the culture of implant centres and staff practice preferences towards

digital innovation, recruitment decisions and strategies, and inevitably the impact of the COVID 19 pandemic.

Constraints outweighed enabling factors and both were largely similar across implant centres.

These factors were interdependent and are supported by the assumptions that underpin the NASSS

framework. They provide the best explanation for the low impact of CHOICE on implant centres and its take up

by implant users. In summary, firstly, some users were excluded primarily due to poor hearing performance,

and this impacted significantly on the eligibility of the user population of some centres. Second, following the

promise of the original value proposition, the experience of CHOICE in practice revealed flaws in its initial

assumptions which led to questioning of the initial value proposition and its future potential. Third, CHOICE

scale up was impaired by its poor function, in particular the crucial hearing check that was a key device to

monitor hearing performance remotely. Fourth, CHOICE adoption and integration into implant centre

pathways struggled due to mistiming of training, limited support from the CHOICE project team on integration,

reliance on single staff members (centre champions), and long delays in getting underway. The lack of

integration of CHOICE with patient management systems also created issues for centres. Finally, both staff and

users retained some preference for face-to-face appointments and recruitment practices. This constrained

users’ opportunities to register for CHOICE or simply meant they chose not to register.

Use of the NASSS framework to understand the factors that affected the scalability of CHOICE suggests greater

complexity than anticipated when it was considered for scale up. Although easy to use for many users, the

real-world implementation of CHOICE presented many challenges to scalability, some predictable constraining

factors and others less predictable (Appendix 4). Affordability for ongoing support and adaption also proved

challenging.

“More people are seeing the need that you have to change, which is what’s happened in general with lockdown isn’t

it…you couldn’t do anything remotely and all of a sudden everyone’s had to do it remotely and they have been able to. I

think we’ve just had our eyes open that there are different ways of working, where people were maybe resistant before.”

Staff member 01_ICG

"“More people are seeing the need that you have to change, which is what’s happened in general with lockdown isn’t









“More people are seeing the need that you have to change, which is what’s happened in general with lockdown isn’t












"We have a mandate, this is post Covid-19, that 8% of our clinics have to be online. This is a hospital mandate, it’s not

audiology or cochlear implant related, and obviously that’s completely unmanageable audiology or cochlear implant-wise

because we can’t have telephone conversations with many patients, but it tells you about the pressures that we’re facing

in terms of seeing patients remotely. I think CHOICE tapped onto that.” Staff member 02_ICR

Key messages

19. COVID-19 halted recruitment in all but one implant centre in the first lockdown in March 20. Centres re-

opened at different times, and it wasn't until October 2020 that all seven centres were recruiting again.

20. Indications from the study are that COVID-19 and CHOICE have enabled the start of a cultural shift

towards more remote care in audiology settings.

"“We have a mandate, this is post COVID, that 8% of our clinics have to be online. This is a hospital mandate, it’s not



in terms of seeing patients remotely. I think CHOICE tapped onto that.”

Key messages





23.

Key messages









Key messages









Key messages

18. Covid-19 halted recruitment in all but one implant centre in the first lockdown in March 2020.

Centres re-opened at different times, and it wasn't until October 2020 that all seven centres were

recruiting again.

19. Indications from the study are that Covid-19 and CHOICE have enabled the start of a cultural shift



4. LESSONS FOR FUTURE SCALE UP PROGRAMMES

The first ever randomised controlled trial (RCT) of remote care for adults using cochlear implants was

published by the team behind CHOICE in January 2018 (Cullington et al, 2018). Sixty adults who had had an

implant for at least six months were recruited, with half randomly allocated to the remote care group, and half

randomly allocated to the standard pathway acting as a control group. The study reported positive results in

that the remote care group experienced greater patient activation, improved hearing and more convenient

care. However, the RCT showed that the Triple Digit Test (TDT) was too difficult for some participants, with

only 14 of the 30 being able to record their score. Two other limitations were described by the authors as a

limited assessment of clinician preference and of implementation challenges.

The team behind CHOICE then made an application to the Health Foundation Scaling Up Improvement

Programme. This was successful, with £500,000 awarded to scale up this innovative care model. The ambition

was to roll it out to 2,200 users - 40% of the adult implant users in eight centres. In the event, just 240 users

registered and logged in to the CHOICE Web app. Issues with the TDT were reported throughout the life of the

project and updates were completed in December 2019 and April 2020 prior to the removal of the hearing

check completely in January 2021.

Our evaluation found little evidence of CHOICE having a consistent positive impact on users, staff or implant

centres. This is disappointing for the CHOICE programme. However, despite this, there are important

positives for the people dedicated to its wider use.

• It broke new ground, introducing clinical and cultural innovation in a highly specialised area of patient care

• Its potential value was strong and clear to users and staff, aided by committed and visionary leadership

• Whilst its impact was less than expected, the concept of CHOICE was welcomed as a catalyst for more

work in this area

• The programme has delivered important lessons for the cochlear implant community and others about

when and how to scale innovations.

Two evidence-based frameworks, NASSS and NPT, supported the analysis to explain why CHOICE did not scale

as intended and inform the lessons below. Our evaluation revealed a range of operational lessons that could

have helped the roll-out and we have summarised these as a set of 48 factors that constrained the programme

and 28 factors that enabled it (Appendix 4). However, for this last section we want to draw out six lessons that

could apply to all programmes seeking to scale digital remote care models:

1. Have a clear value proposition for each stakeholder

Greenhalgh's NASSS framework has a value proposition domain, concerned with understanding whether a new

technology is worth developing in the first place - and for whom it generates value. Value means different

things for different stakeholders. For demand side stakeholders (adopters), it is important to establish

whether the innovation is desirable to its target users and will bring them sufficient benefits. On the supply

side (developers) there should be a plausible business case, the relevant approvals in place, a well-defined

customer base, and realistic assessment of the challenges of implementing at scale. Ideally, all stakeholders in

the innovation's value chain will gain some value.

Our evaluation demonstrated that the aims of CHOICE had intrinsic value with users, they thought it was a

good idea and could improve their experience of having a cochlear implant. Similarly, many staff could see its

potential value and that it was a development in the right direction.

This evaluation found that for many reasons, this value wasn't realised, and these are covered in the remaining

lessons below. We also found that the nature of the value of CHOICE was not pinned down. A key objective

was replacing face-to-face appointments in order to confer benefit to users who didn't need to travel for their

annual review and to centres through reduced appointments. In interviews, users and staff described the


value of continuing face-to-face appointments. Users and staff felt CHOICE had more value as a

supplementary to annual reviews in person rather than a replacement.

The supply side value of CHOICE is less clear, in particular the value to the implant suppliers, the four

companies with a commercial interest in this market. CHOICE was developed by one of the implant centres as

a generic offer covering all four commercial implant devices. We heard that one of these companies offer

their own remote support packages and that in some cases these are preferred. We heard that while CHOICE

had not succeeded in its scale up ambitions, it had made the case for a remote care option to be further

developed which is positive. It will be important to determine whether this is best delivered by the NHS

centres, the commercial suppliers or a combination.

2. Ensure the technology meets expectations

Technology is also a domain in the NASSS framework. In their research for the framework, the authors found

that many technologies are insufficiently prototyped and that aesthetics and 'clunkiness' could have a

significant impact on useability. Technology needs to be well-designed and easy to use with support. And it

needs to generate data that are well understood and accepted by its intended users. Relationships with

suppliers are important to the sustainability of technology and the ability to adapt and customise it as things

change.

This evaluation found many issues with the technology underpinning CHOICE and how it functioned in the real

world. The biggest issue was the reliability of the TDT hearing check, which for many was the most important

incentive for using CHOICE. Reliability issues eventually led to its withdrawal. The rehabilitation tool

RealSpeech, received good feedback from users but was only available to people using Windows software (not

Apple devices) which ruled some users out. Users gave CHOICE a moderate rating as a digital product and

some described it as less professional in its look and utility to other downloadable apps they use. CHOICE was

not able to interoperate with existing patient information systems in the centres, which led to a lot of

duplicate data entry and cross-checking.

A large part of The Health Foundation funding was invested in the development of the CHOICE web-based app,

using an app developer. What was 'scaled up' was a newly developed web app that hadn't been prototyped or

extensively tested. It appears that the commercial relationship with the developer didn't include enough time

or budget for the ongoing adaptation of CHOICE based on its use in the real world. The evaluation team was

aware that requests to make changes to CHOICE took a long time to action and weren't a priority for the

developer. Users and staff described a desire for a more sophisticated tool, like those now used in everyday

life (e.g. banking, shopping, booking).

3. Provide the right type and level of support to adopters

The management team with overall responsibility for the CHOICE programme was based in Southampton, the

home of CHOICE, and comprised the Chief Investigator and a project manager and small amount of

administration support funded from The Health Foundation. A steering group made up of representatives of

the key stakeholders was due to meet every four months and each centre signing up to participate in CHOICE

planned to appoint their own site champion allocated half a day a week to support its implementation.

In interviews, site champions described being left to their own devices, without much guidance and having to

interpret the study protocol to inform their local implementation. The Health Foundation have reported that

successful scale up of innovations requires an adequate description (codification) of how to implement it -

covering technical, contextual, social and dynamic forces. There are choices on how tightly the innovation is

codified for adopters:

‘'Tight' descriptions attempt to draw social and contextual factors into the intervention protocols, though in doing so tend

to highlight the capabilities required for successful implementation. 'Loose' descriptions, by contrast, focus on helping

adopters adapt the intervention to fit their own context, though in doing so make them own the constraints within which

they need to operate". The spread challenge (2018)

"'Tight' descriptions attempt to draw social and contextual factors into the intervention protocols, though in doing so tend

to highlight the capabilities required for successful implementation. 'Loose' descriptions, by contrast, focus on helping


In practice, the CHOICE programme didn't provide enough codification of how to implement it locally. Some of

it could have been tight (e.g. who to recruit) and some could have been looser (e.g. how to approach local

integration into workflow, including culture). Site champions talked a lot about the importance of the

prevailing culture in their centre, that ‘younger’ staff were more engaged with remote care options and ‘older’

staff less engaged and preferring face-to-face consultations. Cultural challenges to implementing innovation

are predictable and should form part of the support package for adopters.

Getting the timing right is also important for successful implementation. Delays with approvals at the

beginning of the programme put it nearly a year behind schedule. There were then local delays as each centre

needed to negotiate local information governance approvals. This meant that by the time recruitment started

the training that staff had received on CHOICE felt like a long time ago. The Steering Group did not meet as

regularly as planned.

It is common to plan to spend some of the requested funding on a project manager and to rely on this post to

assure delivery. In practice, these posts can be hard to recruit to - there is a shortage of good project

managers. The CHOICE project manager was new to digital health innovation, left early in the implementation

programme and was only partially and for a short time replaced. This meant an over-reliance on the Chief

Investigator and an unsustainable level of pressure on one person. Adequate and distributed resource for

management support to implementation may have helped to mitigate these risks.

4. Expect to adapt the innovation as it scales and plan to support this Scaling up programmes and innovations need to be able to adapt and evolve to the real-world context that

they are operating in. The process of adoption requires new sites to make adaptations to meet their own local

context. Feedback loops from adopters to the programme team, and supporting adopters to network with

each other, are important elements of scaling and embedding. These were in place to a degree in this

programme but could have been more developed and proactive. There was a forum for the site champions to

meet and they in turn described the Chief Investigator as always being helpful and responsive when contacted.

The long delays to the roll-out, national lockdowns and loss of the project manager all acted against the

programme proactively supporting sites to adapt their local implementation and to learn from each other.

As discussed above, the CHOICE web app was newly developed before this scale up programme and hadn't

been extensively trialled. A scale up programme of a new app should expect and plan for ongoing

development and adaptation based on its real-world use in a range of settings. The CHOICE developer was not

able to provide as much support to this as desired by the Southampton team, and this was likely due to

original contract arrangements not anticipating the need for ongoing requirements.

‘Normalisation’ of a change in practice requires opportunities for staff to reflect on how this is working, and to

do this regularly over time (Normalisation Process Theory). Staff did not have routine access to data about

CHOICE uptake, experience and implementation processes that would have helped them to adapt its

implementation. This data was only made available during the last few months of the programme when it was

too late to support reflective monitoring.

5. Be sure that the innovation is ready for scaling up Perhaps the greatest lesson from CHOICE was that it was not ready to scale. In practice, it proved to be too big

a step to go from the small RCT that indicated CHOICE was effective (Cullington et al, 2018) to scale to a

remote care model for 40% of adults in eight centres. With hindsight (and evaluation), it would have been

better to continue to develop and test CHOICE with a small number of centres. This could have identified and

resolved many of the real-world functionality problems ahead of a more significant scale up. It could also have

co-designed the best form of codification and implementation support that new centres would need when

adopting CHOICE.

Returning once more to the NASSS framework and the importance of a value proposition that supports the

scale up of a technological innovation. This is described as including a plausible business case, with a strong

case for a return on investment and key approvals in place. CHOICE had not reached this bar when it moved to


scale up - but had shown the potential to be able to. It appears that an intermediate step would have been a

better approach.

This is a lesson for innovators, but even more so for funders.

6. Getting the most from evaluation

The Health Foundation’s supported programmes include their own independent evaluation which adds

positively to our collective understanding of what is required for successful scale up of innovation in health

and care. Wessex AHSN has a lot experience working as an independent evaluation partner and has learnt

how important it is to develop the right relationship with the programme and the people leading it who are

passionate about spreading their innovation and delivering benefits. We had a good working relationship with

the CHOICE leadership team, which included early co-design of our evaluation plan, a dedicated Evaluation

Advisory Group and a place at the programme Steering Group.

Long delays in implementation like those experienced in CHOICE are a challenge for evaluation. It requires

flexibility from both sides to agree new timelines for data collection and site visits. In the future, we intend to

include discussion and agreement of contingency planning for long delays with the programmes. It is

important that the scope includes both an evaluation of the impacts a programme delivers and the process of

implementation. As it became clear that CHOICE was not going to deliver the impacts it had intended, our

evaluation of the implementation process became more important. We missed opportunities to observe

CHOICE implementation at sites due to restrictions as a result of Covid-19 and feel this would have enhanced

the qualitative insights from this evaluation.

A common challenge for evaluation is access to data. We only had about half of the quantitative data that we

had planned to use for this evaluation. As well as specifying and agreeing the data needed to answer

evaluation questions, it is important to stress-test who and how data will be delivered. In practice, we

experienced two significant constraints to accessing CHOICE data. There was a significant reliance on a third-

party app developer with whom we didn't have a direct relationship and this led to gaps, errors and delays.

There was an expectation that each of the individual sites would take responsibility for supplying outpatient

and activity data, but in practice most did not. A lesson for evaluation activity is to more tightly specify and test

how data will be delivered to evaluators.

A Final Word from the Innovator – Professor Helen Cullington:

CHOICE was launched with the aim of offering an option of home-based care to adults with cochlear

implants. Seven clinics started using CHOICE, but the patient take-up was low. 312 implant users registered

during the life of the project. 277 implant users remained registered at the close of the project. Change within

the NHS is difficult and introducing a significant change to clinical practice across multiple centres was always

going to be challenging. However in this case, the main issue is the very significant ongoing commitment to

maintenance and development that an app requires. It now seems unrealistic for a research project with time-

limited funding led by an academic individual to sustain a permanent digital addition to UK care options. It

seems that going forward this would be better offered by a large company with significant resources to invest,

including dedicated app support staff.

The value proposition of a home-based care option remains (person-centred responsive needs-led care,

reduced travel costs and convenience for patients, and clinic resource prioritisation). It is hoped that the

awareness and mindset shift that CHOICE raised in patients, clinicians and funders will enable a smoother

transition to a remote care option in future.


REFERENCES

Benson T. Measure what we want: a taxonomy of short generic person-reported outcome and experience

measures (PROMs and PREMs). BMJ Open Quality 2020; 9 (1): e000789.

Cresswell K, Sheikh A. (2013) Organizational issues in the implementation and adoption of health information

technology innovations: an interpretative review. International Journal of Medical Informatics, Vol 82(5): e73–

86.

Cullington, H., Kitterick, P., Weal, M., & Margol-Gromada, M. (2018). Feasibility of personalised remote long-

term follow-up of people with cochlear implants: A randomised controlled trial. BMJ Open, 8(4), e019640.

https://doi.org/10.1136/bmjopen-2017-019640

Finch, T.L., Girling, M., May, C.R., Mair, F.S., Murray, E., Treweek, S., Steen, I.N., McColl, E.M., Dickinson, C.,

Rapley, T. (2015). Nomad: Implementation measure based on Normalization Process Theory. [Measurement

instrument]. Retrieved from http://www.normalizationprocess.org.

Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A’Court, C., Hinder, S., Fahy, N., Procter, R., &

Shaw, S. (2017). Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption,

Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies.

Journal of Medical Internet Research, 19(11), e367. https://doi.org/10.2196/jmir.8775

Maguire, D., Honeyman, M., Fenney, D., & Jabbal, J. (2021) Shaping the future of digital technology in health

and social care. The King’s Fund. An independent report commissioned by the Health Foundation.

May, C., & Finch, T. (2009). Implementing, Embedding, and Integrating Practices: An Outline of Normalization

Process Theory. Sociology, 43(3), 535–554. https://doi.org/10.1177/0038038509103208

Standing C, Standing S, McDermott M, Gururajan R, & Kiani Mavi R. (2016) The paradoxes of telehealth: a

review of the literature 2000-2015. Systems Research and Behavioural Science, Vol 35(1): 90-101.

Telemedicine for people with cochlear implants in the UK: empowering patients to manage their own hearing

healthcare, protocol V1.9, 17,12,2020.

https://doi.org/10.1136/bmjopen-2017-019640

http://www.normalizationprocess.org/

https://doi.org/10.2196/jmir.8775

https://doi.org/10.1177/0038038509103208


APPENDIX 1: NONADOPTION, ABANDONMENT, SCALE UP, SPREAD

AND SUSTAINABILITY (NASSS) CONCEPTUAL FRAMEWORK

The evaluation team used the Nonadoption, Abandonment, Scale up, Spread and Sustainability (NASSS)

conceptual framework to structure the interview schedules using its seven key domains to inform our

understanding of what happened and why. This framework articulates the necessity for clarity on the

expectations of the technology, and whether the innovation and its implementation is knowable and

predictable. If not, whether its implementation is dynamic and emergent (Greenhalgh et al 2017) thus

potentially inhibiting scale up. The table below provides the key features of the NASSS framework with an

explicit statement on how this was translated to this study’s context and innovation. It should be noted that

ideally use of the NASSS framework is recommended earlier in the development pathway. However, the study

protocol predates the publication of NASSS.

NASSS Domain (ref) Summary of domain (ref) For the purpose of this study

1. The condition or

illness

Addresses the clinical,

comorbidities and sociocultural

aspects of the condition.

1. Type of deafness, e.g. whether profoundly

deaf from an early age or hearing deterioration

overtime.

2. Other related factors such as learning

disability, mental health issues.

3. Implantation also relates to condition within

the context of implant technology and the

CHOICE platform.

2. The technology

Addresses material and technical

features of the technology and its

dependability in challenging

conditions. Asks:

• what knowledge is generated

or made visible by the

technology?

• What is needed to support the

use of the technology by those

using it?

• How is the technology

procured?

• What is the relationship

between developer and user.

• What other products could be

substituted for the technology

should it become unavailable?

CHOICE has several key features as a remote

care system for people who need to manage

their implant independently, whilst also

permitting communication between implant

user and the implant centre.

It provides several rehabilitation tools (speech,

music, telephone), a real world hearing check

to test performance of the implant (Triple Digit

Test), and access to a system for ordering

spares.

The system provides the opportunity for

notifications and reminders as well as user

questionnaires.

Other commercial products for rehabilitation

and hearing checks also exist. Designed by staff

at the Southampton implant centre and

developed by an independent software

company.

3. The value

proposition

Addresses whether a new

technology is worth developing in

the first place—and for whom it

generates value and asks:

• What preliminary testing and

evidence as well as business

planning has occurred?

• What testing has occurred to

provide evidence of benefit to

CHOICE provides a universal (for all types of

implant) remote care system enabling implant

staff and implant users to communicate and

judge whether a clinic visit is required. Implant

users can improve their hearing performance

and order spares in their own time.

Increased demand on implant centres as

numbers of implant users grows allows an

additional option in the clinical pathway to

keep in touch with implant users and does not

http://revistamagisterioelrecreo.blogspot.com/2012/05/el-implante-coclear.html



https://creativecommons.org/licenses/by-nc-sa/3.0/












NASSS Domain (ref) Summary of domain (ref) For the purpose of this study

users and real-world

affordability?

constrain users to wait for their annual reviews

should an earlier clinic visit be needed.

4. The adopter

system

Addresses whether the technology

is adopted and its use continued by

staff or whether it is subsequently

abandoned and whether this is

understood to be with regard to

the technologies attributes or

concerns of staff seeking to embed

the technology. Asks:

• Is the technology acceptable to

users and the efforts required

to use it.

• What assumptions were made

about its use and impact on

wider social networks, e.g.

computer literacy.

Adoption by both implant users and implant

staff was key to successful implementation and

scale up of CHOICE. Data considers the pathway

adoption and nonadoption factors.

5. The organisation Addresses the organisation’s

capacity (to

embrace any service-level

innovation) and readiness (for a

specific technology), To support

scale-up requires understanding

antecedent conditions and

organisational readiness. This

includes senior level adoption

decision making and resourcing,

levels of disruption to routines and

the level of implementation effort

required.

Implementation efforts to embed CHOICE into

implant user care pathways in implant centres

are described.

6. The wider context Asks what other wider institutional

and sociocultural context factors

are relevant to explaining whether

the technology moves from a

demonstration to a mainstream

service (scale up), widely

transferable (spread) and sustains

overtime.

Sought to identify whether there were factors

beyond the control of those seeking to adopt

CHOICE that either enabled or hindered

adoption of CHOICE. For example, the unique

and widespread impact of the Covid-19

pandemic.

7. Embedding and

adaption overtime

Relates to the medium and long-

term feasibility of ongoing

technology adaptation. This

includes adaptation of staff roles

and care pathways. Organisational

resilience is required to manage

expected multiple

interdependencies, nonlinear

effects, and unintended

consequences.

Successful adoption is likely to require

adaptation to the CHOICE Remote care system

to facilitate its fit into implant centre pathways,

as well as meet implant user needs.

Organisation of care and care pathways at

implant centres are also likely to need to adapt

to fit the introduction of CHOICE

This

Phot

o by

Unkn

own

Auth

or is

licen

sed

unde

r CC

BY-

SA

This

Phot

o by

Unkn

own

Auth

or is

licen

sed

unde

r CC

BY-

SA

This

Phot

o by

Unkn

own

Auth

or is

licen

sed

unde

r CC

BY-

SA

This

Phot

o by

Unkn

http://picpedia.org/highway-signs/a/adoption.html



https://creativecommons.org/licenses/by-sa/3.0/



















APPENDIX 2: NORMALISATION PROCESS THEORY AND THE NOMAD

INSTRUMENT

Normalisation Process Theory via the NoMAD instrument was used in this evaluation. The NoMAD instrument

questions are tailored to the specific project and for this study a copy of the survey is available in

Supplementary material 1.

Normalisation Process Theory (NPT) (May and Finch 2009) is a validated instrument developed to understand

how innovations in healthcare are implemented in practice – and in particular how new ways of working

become embedded and sustained. It is an Action Theory, concerned with explaining what people actually do,

rather than how they describe their attitudes or beliefs. The focus is on factors (beliefs and behaviours) that

promote or inhibit (enablers and barriers) the implementation of an innovation. The factors are divided into

four constructs:

i. Coherence: the mobilisation of a practice – how it is conceptualised and held together in action

ii. Cognitive participation: participation in a practice – how members decide to engage and actually engage

iii. Collective action: enacting a practice – how the work is organised and activities structured and constrained

iv. Reflexive monitoring: the appraisal of a practice – how it is appraised and the effects of appraisal, i.e. how it

is ‘understood’ and what changes the team make.

The NoMAD instrument was devised as a way of measuring the presence of these four components in teams

working together to deliver an intervention. It is made up of 20 questions across the four components.

Further information: http://www.normalizationprocess.org/

http://www.normalizationprocess.org/


APPENDIX 3: ACTIVITY DATA MATRIX: PLANNED VS ACTUAL DATA RECEIVED

Data Type Description of data required Rationale Questions

Answered

Comments

App Data

Numbers of patients registered to use the new care

model An indicator of staff uptake/confidence in the system Q1

Received

Number of staff registered on clinician portal To understand spread of use amongst clinicians Q1 Received

Numbers of patients using the new care model An indicator of patient uptake Q1 Received

Actions arising from use of the remote care package,

e.g. orders for replacement parts

An indicator of how patients are using the App; self-management

actions Q1

Received

Home hearing check results Research measure Research Q Received but not used by the

evaluation team

Number of logins To understand overall use of the system Q1 Received

Uses of self-device adjustment (if appropriate) An indicator of how patients are using the App; self-management

actions Q1

Not available in the app data

Uses of home hearing check An indicator of how patients are using the App; self-management

actions Q1

Received

Mapping of activity by truncated postcodes or distance

from the clinic

To understand whether distance from the clinic is a factor influencing

uptake of CHOICE and therefore contributes to improving equity of

access

Q4

Received

Number of patients changing back to clinic pathway An indicator of patient dissatisfaction/inappropriateness Q1 Received

Patients using the remote care pathway, as a

percentage of the total clinic caseload To understand spread of use amongst patient Q1

Not consistently available across all

sites

Number of patients who register to use the tool but do

not log in subsequently To understand consistency of use by patients Q1

Received

NHS Friends and Family Test Over-arching measures of satisfaction included at the request of the

project lead Q5

Received



Answered

Comments

Outpatient data

Change in use of outpatient appointments as a

consequence of using the remote care pathway,

including numbers and types of clinic attendances and

other contacts and reasons for clinic attendance/type of

appointment

To understand if there is a change in behaviour of patients using the

system and in their interactions with clinicians Q6

Baseline data received from 5 out of 7

sites and post-launch data received

from 3 sites. IG restrictions allowed

for the use of aggregated data only for

non-CHOICE patients and CHOICE

patients before they signed up.

Did Not Attend rate To understand if there is a change in behaviour of patients using the

system and their interactions with clinicians Q6

Same as above

Mapping of activity by truncated postcodes or distance

from the clinic

To understand whether distance from the clinic is a factor influencing

uptake of CHOICE and therefore contributes to improving equity of

access

Q4

Same as above

Patient

Questionnaire

Patient travel cost, time, hours off work, childcare

(including accompanying person) To understand the potential time/cost benefits for patients Q6

Received

Site data

Numbers of self-initiated versus service-initiated

(scheduled) appointments

To understand if there is a change in behaviour of patients using the

system and their interactions with clinicians Q2

Not available

Number of errors in online resources, adverse events,

missed issues An indicator of clinical effectiveness of the system Q1

Not consistently available across sites

Patients wanting to continue remote care after the end

of the evaluation To understand whether level of engagement is likely to be sustained Q1

Not available due to CHOICE being

taken offline at the end of research

project

Contacts from patients having difficulty with remote

tools To understand patient adaptation to new system Q1


sites

Additional appointments to train in remote care To understand patient adaptation to new system Q1,6 Not consistently available across all

sites

Additional appointments for patients concerned about

results from remote tools To understand patient adaptation to new system Q1,6


sites

Clinician caseload ratio To understand whether the acuity of the clinicians’ caseloads changes

as a result of introducing the CHOICE pathway Q1

Not available

Follow-up activity costs To understand if there is a change in use of outpatient services that

realises any economic impact Q6

Not pursued due to being unable to

understand change in outpatient

appointments



Answered

Comments

Workforce

To understand if the introduction of CHOICE has impacted the

workforce e.g. do clinical staff organise themselves in a different way as

a result of CHOICE

Q6

Received from 5 out of 7 sites

Chief

Investigator

Clinics wanting to stop offering remote pathway To understand spread of use amongst clinicians Q1 Received

% clinics wanting to participate To understand spread of use amongst clinicians Q1 Received


APPENDIX 4: IMPLEMENTATION CONSTRAINTS AND ENABLERS OF CHOICE BY THEME

Themes Constraining the implementation of CHOICE Enabling the implementation of CHOICE

The value of CHOICE

1. Considered an optional service by staff

2. CHOICE not suitable for all implant users

3. Staff concerns about increase workload led to restricting recruitment

4. Many experienced implant users did not feel they needed CHOICE

5. Staff and users anxiety that face-to-face consultations important

6. Expected value of CHOICE impaired by unreliable hearing check

7. CHOICE did not fulfil some users desire for remote impedance checks (not its

intention)

8. CHOICE was considered a ‘trial’, so optional work for sites as to whether they

registered or not as oppose to a formal roll out.

1. CHOICE perceived as a ‘safety net’

2. Loyalty and reputation to CHOICE innovator

3. Perceptions about general increase in clinical activity

4. Perceptions about its value to monitor hearing

performance

5. ‘Younger’ staff perceived more value

6. Site champions confidence supported CHOICE

The useability of

CHOICE

9. Confusion at registration

10. Unable to access RealSpeech

11. Ordering spares: Superiority of clinic spares and repairs service

12. Ordering spares: Difficulties in specifying and receiving the right parts

13. Ordering spares: Confusion, i.e. via clinic, specific implant company or via CHOICE.

14. Hearing check issue: Difficulty establishing baseline

15. Hearing check issue: Comparability with other hearing checks used

16. Hearing check issue: Unreliability of feedback scores and notifications

17. Hearing check issue: Results confusing, frustrating and anxiety inducing for users

18. Hearing check issue: Limited explanation of results for users

19. Hearing check issue: Taking photos of implant site hard to do solo

20. Created new work: Limited linkage between CHOICE and NHS user identifying codes

21. Created new work: Limited linkage between CHOICE clinician portal and patients

management system

22. Created new work: Limited dropdown menu user status options in clinician portal

23. Limited necessary adaptations to the CHOICE platform functions

7. Access to music training

8. Access to telephone training

9. Easy access to ordering parts

10. Provided a system for reminders and notifications

11. Importance of monitoring motivated staff and users

12. Questionnaire triaged users

13. Easy communication between user and staff


24. Difficulty engaging software developers to operationalise changes to CHOICE

platform functions

The integration of

CHOICE

25. Long duration of profound deafness problems, e.g. unable to use the Triple Digit Test

26. Users who’ve had multiple implantations described anxiety about remote care

27. Learning difficulties / mental health difficulties

28. Users inability with technology

29. Obtaining permission to start CHOICE took time at sites

30. Limited support offered by CHOICE team at the day-to-day level to sites

31. Limited opportunity for CHOICE due to unclear clinical pathway fit at sites

32. Considerable delays by sites in starting to recruit to CHOICE led to a loss of staff

enthusiasm

33. Considerable reliance on site champions to drive CHOICE forward led to slow

recruitment

34. Site champions had limited clinical time to do CHOICE work

35. Infrequent steering group meetings, particularly during pandemic period

36. Site champion forum: some sites reported they were not solution-focused and did

not support integration of CHOICE

37. Retention of existing pathways limits opportunity for CHOICE to valued

38. Unclear boundaries between implant centre responsibilities and CHOICE team

responsibilities

39. Limited processes in place to manage the CHOICE clinician portal outputs

14. Site champions led locally tailored training

15. Guidance document given to Southampton users

16. Site senior staff support for CHOICE

17. Some site champions wrote operating procedures to

integrate CHOICE

18. Responsive implant centres to CHOICE platform queries

from implant users

19. Chief Investigator at Southampton site was responsive to

requests for support from other sites

20. Southampton users perceived it was their ‘duty’ to

participate

21. Users’ general technical competency

22. No user concerns about privacy

Influential factors

beyond the control of

CHOICE

40. Staff and user preference for face-to-face consultations

41. Staff and user preference for other support platforms

42. Covid-19 lockdown stopped 6 sites recruiting to CHOICE for several months

43. Covid-19 lockdown reduced engagement about CHOICE between site champions and

site staff

44. Some staff deregistered CHOICE users if they didn’t use CHOICE platform, due to

need to demonstrate chargeable clinical activity to specialised commissioners

45. Alternative implant support options available to implant users, e.g. Remote Check

46. One site prioritised a surgical repairs service for many months which stopped CHOICE

recruitment

23. Covid-19 lockdown increased use of CHOICE by implant

users already registered on CHOICE

24. Covid-19 lockdown allowed one site time to review

caseload for all eligible users and conduct a mass

recruitment mailout

25. Covid-19 encouraged more remote care consultations

26. Emerging NHS Trust policies for more remote care

legitimised CHOICE and increased use


Cross cutting theme:

Factors affecting

recruitment to

CHOICE

47. Staff recruitment criteria limited registered users e.g. users need to be “tech savvy”

and “good performers”

48. Face-to-face recruitment was slow

49. Complexity of timing CHOICE into user pathway limited decisions as to when to

recruit

50. Bulk emails to users did not work for all centres

27. Success of bulk mail outs to all eligible centre users

28. Reminder strategies to staff to recruit to CHOICE

SUPPLEMENTARY MATERIAL

Supplementary material 1 – CHOICE NoMAD survey

Supplementary material 2 – R-Outcomes surveys

Supplementary material 3 – Qualitative field work

Supplementary material 4 – Lessons

Supplementary material 5 – CHOICE Protocol

These materials can be found as separate resources in the Wessex AHSN Innovation Insight library.

independent evaluation of choice of choi

Documents