MAKING SENSE OF RISK BY VISUALIZING

COMPLEX HEALTH DATA

Nicholas Tenhue

MSc ICT Innovation

University College London, 2014

MAKING SENSE OF RISK BY VISUALIZING

COMPLEX HEALTH DATA

Nicholas Tenhue

MSc ICT Innovation & Data Vis & Design Intern, Intel Health & Life Sciences

Author and Principal Investigator

Ann Blandford

Professor of Human-Computer Interaction

Thesis Supervisor

Chiara Garattini

Anthropology & UX Research, Intel Health & Life Sciences

Industry Supervisor

This report is submitted as part requirement for the MSc Degree in ICT Innovation,

at University College London. It is substantially the result of my own work except

where explicitly indicated in the text.

The report may be freely copied and distributed provided the source is

explicitly acknowledged.

September 2014

University College London, 2014

1

MAKING SENSE OF RISK BY VISUALIZING

COMPLEX HEALTH DATA

Methods for visualising patient data in a way that supports sensemaking

may help clinicians to understand risk factors at the individual patient level.

This thesis uses sensemaking theory and visualisation techniques to de-

velop a tool and test it with clinicians in the healthcare domain. This is an

exploratory study into how information visualisation techniques can help cli-

nicians make sense of risk in a patient. We present a chronological account

of the approach taken to build and assess a visual tool for sensemaking.

We present two main findings (i) making sense of risk is a multifaceted pro-

cess that entails complexity beyond just using research evidence and clini-

cal expertise (ii) we have preliminary evidence that the visual tool supports

by creating externalisations that facilitate to make the implicit processes

that they use frequently in their work, explicit.

Keywords: information visualisation, sensemaking, design

2

3

I owe this journey of discovery and learning to a number of people.

Academic Supervisor, Ann Blandford, who gave sound advice throughout.

Industry Supervisor, Chiara Garattini, for advising me with her deep

knowledge of anthropology and UX.

Mentors, Mario Romero and Connor Upton, who shared their expertise in

information visualisation.

My peers Misha Patel, Hanna Schneider and David Pribil for their constant

encouragement.

This thesis is written and reported, in entirety, by the Author. However,

employees at the Author’s internship company provided invaluable

assistance:

Chiara Garattini – risk calculator research & persona creation; partici-

pants’ recruitment;

Marisa Parker – assisting in design of Stage 2 & 3 prototypes;

Reese Bowes – final screenshots for use in external publication.

4

5

TABLE OF CONTENTS 1 Introduction .......................................................................................... 7

2 Background ....................................................................................... 11

2.1 Sensemaking .............................................................................. 11

2.2 Visualisation for sensemaking .................................................... 14

3 Methods............................................................................................. 21

3.1 Participants ................................................................................. 21

3.2 Apparatus & Materials ................................................................ 22

3.3 Ethical considerations ................................................................. 25

4 Design, study & analysis .................................................................... 27

4.1 Patient personas......................................................................... 27

4.2 Stage 1 ....................................................................................... 29

4.3 Stage 2 ....................................................................................... 33

4.4 Stage 3 ....................................................................................... 48

5 Results .............................................................................................. 61

5.1 Data complexity & the implicit ..................................................... 61

5.2 Clinical workflow ......................................................................... 64

5.3 Thinking about risk ..................................................................... 66

5.4 Dealing with data ........................................................................ 68

5.5 Visualising risk ............................................................................ 74

6 Limitations ......................................................................................... 83

6.1 Design failures in the tool ........................................................... 83

6

6.2 Limitations of the study ............................................................... 85

7 Conclusions ....................................................................................... 87

8 Bibliography ....................................................................................... 89

9 Appendices ........................................................................................ 97

9.1 Appendix A – Participant Information .......................................... 97

9.2 Appendix B – Consent Form ..................................................... 100

9.3 Appendix C – Interview Plan (Stage 2) ..................................... 101

9.4 Appendix D – Interview Plan (Stage 3) ..................................... 104

9.5 Appendix E – Requirements Statement .................................... 105

7

1 INTRODUCTION When dealing with risk the clinician must search for and decide what data is rele-

vant to the individual patient. This complex task involves judging the integrity of

the data, the relevance of that data to the individual patient, the particulars of the

circumstances, patient wishes, and a host of other variables. Few tools support

this process because they downplay the role of clinical expertise for judging par-

ticular circumstances, instead they rely only on empirical population studies that

may or may not apply to the individual in question and leave the rest to the clini-

cian.

Clinicians are forced to deal with an excess of data in their work (Feblowitz,

Wright, Singh, Samal, & Sittig, 2011). This issue will continue to grow as a sur-

plus of noisy, multivariate, homogeneous data is generated from a number of dis-

similar sources. Clinicians already deal with patient reported data and hospital

generated data. Also, the proliferation of wearables and mobile devices lead us to

believe that doctors are increasingly exposed to self-generated health data (My-

natt, 2011), providing new insight into patients’ lives that population studies. Most

of the existing risk algorithms and evidence-based research do not presently take

data generated from these new types of eHealth self-monitoring devices into ac-

count. In addition, recent developments in whole genome sequencing and ge-

nomic science are making personalised healthcare possible. There is potential to

integrate this wealth of data with traditional health data, to present a tailored rep-

resentation of patient. Nevertheless, the question arises of how the clinician

makes sense of all of these factors to form an overarching understanding of risk

in an individual patient.

Patient information is highly complex with data intervals ranging from minutes to

decades (Shneiderman, Plaisant, & Hesse, 2013). The best possible care is de-

livered when clinicians can, without difficulty, consolidate and make sense of this

patient information in a way that matches their mental model (Johnson-Laird,

1983). Clinicians must seek out data and organize it internally to form a unified

8

understanding of the patient’s current condition (Faiola & Hillier, 2006), this in-

creases cognitive load and the time spent foraging for relevant information.

Possibly more problematic is the lack of externalisations that show relational or

context-based data (Faiola & Hillier, 2006) that would allow clinicians to recog-

nise trends and relationships between co-variables. This leads to the question

‘How can we inform, rather than overwhelm clinicians when they are faced with

these problems?’

One way to tackle this issue is through the use of digital, interactive, visual repre-

sentations of data (Card, Mackinlay, & Shneiderman, 1999). Through the effec-

tive use of visualisations it is possible to deliver what Spence (2007) calls an ‘A

Ha!’ moment, providing insight to task-specific problems that clinicians face. Yet,

information visualisation alone is not the answer. The solution must also support

clinicians’ sensemaking whilst focusing on the quality of the fit between the user

and system models.

This thesis expands on existing knowledge within information visualization and

sensemaking literature and attempts to apply it to creating a tool to support clini-

cians who deal with complex health data when assessing risk.

The aims of this thesis are twofold:

Firstly, we aim to provide a domain specific account of the way in which clini-

cians make sense of complex health data when assessing risk in a patient.

Secondly, by understanding clinicians’ needs and practices we aim to itera-

tively develop a visualisation tool with representations that reduce the gap

between the data and the clinicians’ mental model.

The goal of this thesis is to stimulate discussion in the human-computer interac-

tion (HCI) and information visualisation communities by contributing to the under-

standing of how visualization of complex health data can support clinicians in

making sense of a patient’s risk. To further this goal, this thesis presents findings

from an exploratory study where a tool was designed to support clinicians’ needs

during the assessment of risk.

9

Chapter 2 begins with a review of relevant literature in sensemaking and infor-

mation visualisation. We discuss how visualisations can be powerful tools for

making sense of a domain, and then look at literature on how to design visualisa-

tions. In Chapter 3, the methods used in the study are presented along with the

rationale behind using them. Chapter 4 is a chronological account of the project.

The design process of the tool created for the study, study procedure, and ap-

proach to analysis of each stage are described. The HCI and information visuali-

sation methods that were used are reflected upon. Findings and discussion are

combined in Chapter 5, where we first talk about how clinicians’ understand and

think about the domain, and then give examples of where the tool supported this.

We discuss the limitations of the study and the tool in Chapter 6, along with pos-

sible areas for further investigation. Finally, we draw conclusions in Chapter 7.

10

11

2 BACKGROUND This chapter provides an outline of the existing work relevant to clinicians making

sense of risk and visualising complex health data. Also, the motivations behind

using that work are given. Firstly, literature on sensemaking and existing models

are explored. Secondly, we look at how visualisation can play a role in making

sense of a domain. Finally, information visualisation design considerations and

techniques required to create the tool in this thesis are discussed.

Sensemaking is regarded as the process of information seeking and interpreta-

tion, it is about how people make sense of and understand a domain or topic, in

this case clinicians making sense of risk in a patient. Sensemaking research is

employed in a number of disciplines (e.g. decision-making, organisational re-

search), here we use it specifically to look at how the individual clinician makes

sense of risk. Klein, Moon, & Hoffman (2006) describe sensemaking in modern

research as a continuous effort to comprehend connections between individuals,

places, and events in order to anticipate their trajectories and act accordingly.

There have been a number of efforts to

formalise this process in sensemaking re-

search. In this section we will delve into

some of the leading theories in sense-

making and discuss their relevance to this

thesis as well as how looking through the

sensemaking lens might be beneficial in

understanding the problem domain.

Russell, Stefik, Pirolli, & Card (1993)

describe the sensemaking process

through what they call the Learning

Loop Complex model (Figure 1).

Figure 1 the Learning Loop Complex.

Image taken from Russell et al.

(1993).

12

This model follows a pattern where information in encoded in representations to

reduce the cost of operations. This involves four sensemaking phases:

Search for good representations – representations are created to track

regularities that are significant to the sense maker. This is the generation loop.

Instantiate representations – significant information is identified and encoded

in a suitable representation. Encodons are created in the data coverage loop.

Shift representations – data (residue) that does not fit the existing schema

force a change in representations by moving up through the representational

shift loop, leading to merging, division and generation of schema.

Consume encodons – encodons are used in task-specific information

processing.

The Learning Loop Complex model shows how sense makers use a top-down

(representation instantiation) and bottom-up (representation search) process to

form a mental model of a domain.

Pirolli & Card (2005) identify sensemaking in terms of two loops; the foraging and

sensemaking loops. The foraging loop includes three processes; exploring, en-

riching and exploiting. Exploring is about searching a space to gain new infor-

mation. In the setting of healthcare, this could mean retrieving doctor’s notes and

health record data. Through enriching, a clinician might order new investigations

or drill deeper into information to come up with a higher precision account of a

patient. Exploiting items in a set could mean going through patient information

and making inferences and detecting patterns. The sensemaking loop involves a

recurring process in which a mental model that matches the evidence is created.

However, these models do not take into account interaction effects that can occur

in sensemaking. Weick (1996) notes that schemas do not shift easily as residue

goes unnoticed by the sense maker. People to people interactions can cause

representational shift through exchange of ideas. In the context of this study, this

refers to how clinician-clinician and clinician-patient communication affect the un-

derstanding of risk in the patient. Healthcare is very much a human-centred do-

main where interactions affect the outcome. Sharma (2006) shows how theories

13

can be reconciled to provide a richer understanding of sensemaking. Interper-

sonal interactions help the sense maker notice residue, and consequently change

schemas. In addition, patients themselves can be the information source and col-

laborate with the clinician in the data coverage loop. This suggests that rather

than simply interpreting newly discovered data, sensemaking is about creation

and invention between the various actors.

Klein, Phillips,

Rall, & Peluso

(2007) propose

a sensemaking

theory, which

successfully

condenses the

characteristics of

the previously

discussed models.

Data-frame theory (Figure 2)

states that the sense maker

places data into a frame about

what that data represents; pre-existing

frames (results of previous experiences) influence how the new data is framed.

Three cycles make up the process of sensemaking; elaborating, preserving and

reframing. There are a total of seven steps in the data-frame model:

Data and frame connection –data set is connected to a frame.

Questioning the frame – unexpected or surprising data is encountered and a

frame is questioned.

Elaborating a frame – a frame is elaborated but not changed due to new data.

Preserving the frame – data is disregarded or ignored, preserving the frame.

Seeking a frame – recalling or constructing a fitting frame.

Comparing multiple frames – numerous frames are compared.

Reframing – a frame is either replaced or combined with another.

Figure 2 Data-frame theory of sensemaking.

Redrawn from Klein, Phillips, Rall,

& Peluso (2007).

14

Regardless of whether we talk about schemas, or frames, they both refer to the

way that individuals subjectively look at, filter, and sort the data that they encoun-

ter. A number of questions can be raised when making sense of how clinicians

think about risk in a patient. How do clinicians deal with inconsistencies and

anomalies in data? How do they judge the plausibility and quality of data? When

do clinicians seek and infer or disregard data? How do they seek and infer new

relationships in data? In later chapters, we will attempt to explain our findings

through the sensemaking lens by using the data-frame theory as a framework for

understanding how clinicians think about complex data and risk.

As we can see from exploring these models, sensemaking is a cyclical and itera-

tive process where data is collected and assimilated into pre-existing frames, or

frames are modified based on previous experiences. Sensemaking is about gen-

erating new internal frameworks based on new data. In the context of this thesis,

these sensemaking models provide a way to explain how clinicians deal with the

complex data that they are presented with. The visualisations created for this the-

sis are informed by sensemaking literature and focus on supporting the pro-

cesses clinicians go through when making sense of risk.

There are a number of existing applications of visualisation techniques in

healthcare. Rind et al. (2013) explore effective ways of visualising electronic

health records. Bui & Hsu (2010) discuss systems for adaptive visual interfaces

that integrate clinical information necessary to users’ aims. Faiola & Hillier (2006)

show how complex clinical datasets can be transformed into contextual

knowledge using visualisations, improving the quality of clinical decision-making

and decreasing the time wasted foraging for information by organising it in a con-

text-related format in a single location. Others have looked at the applications of

visualisation for classification and assessment of risk in chronic heart disease

(Harle, Neill, & Padman, 2012) and diabetes (Harle, Neill, & Padman, 2008).

However, when it comes to research into visualisation for healthcare, few studies

look through the sensemaking lens.

15

Faisal, Blandford, & Potts (2013) identify potential ways that information visuali-

sation can assist both clinicians and patients in making sense of health data, but

conclude that more work needs to be done in order to incorporate the sensemak-

ing processes into the design of these tools.

Each person makes sense in his or her own way; sensemaking does not occur

externally, but by definition, inside the mind of the user. As users engage with vis-

ual representations, they also interact with the interface itself, in order to do so

they rely on the mental model that they develop (Sarah Faisal, Cairns, & Bland-

ford, 2007). This internal creation of concepts happens through interaction with

the external world.

Kirsh (2010) identifies a number of ways external representations help sense-

making and allow us to ‘think more powerfully’. As described in the cyclical pro-

cesses of sensemaking theory, when one experiences externalisations, the inter-

nal conceptualisations of a domain are generated, updated and used (Russell et

al., 1993). These externalisations can be in the form of visualisations. Spence

(2007) recognises visualisation as a cognitive activity; when designed well, visu-

alisations can amplify cognition and in turn amplify the sensemaking process

(Card et al., 1999).

Card et al. (1999) state that the purpose of visualisation is insight, as opposed to

just being ‘pictures’ to look at. Insights can be gained when data is represented in

a visual manner, thus supporting the user through visual sense making. Pirolli &

Card (2005) talk about insight being engrained in the sensemaking tasks; infor-

mation gathering, re-representation of data in schema, creation of insight through

manipulating representations, creating a knowledge product or direct action. In-

sight is but a single step in the sensemaking process, but sensemaking may not

be the only way to gain insight (Yi, Kang, Stasko, & Jacko, 2008).

16

Yi, Kang, Stasko, & Jacko (2008) propose four ways in which users gain insight

through information visualisation. Firstly, provide overview is about understanding

the big picture; it informs the user of what is known and what is not known about

a data set. Although it is not directly related to gaining insight, it leads to an un-

derstanding of what parts need further investigation. Adjust is about changing the

level of abstraction or range of selection, this can be done by filtering or grouping.

Detect pattern is about finding trends, relationships, outliers etc. During this pro-

cess users may not only discover what they were looking for but also discover the

unexpected. Match mental model is about decreasing the gap between the data

and the mental model (Johnson-Laird, 1983) of the user, thus reducing cognitive

load.

These processes are not separate and can be used together to gain insight, they

are cyclical and iterative, much like sensemaking. These processes are relevant

to the design of visualisations in this thesis; however, we found no concrete

guidelines in the literature for designing visualisations to provide users with in-

sight. Insight is a qualitative process (Saraiya, North, Lam, & Duca, 2006) making

it well suited to exploration with the methods used in this study.

The previous sections in this chapter focused on sensemaking theory and the

way in which visualisations can move away from simply communicating known

insights in the data toward an exploratory process of iterative understanding that

supports the sensemaking process.

This section will give an overview of information visualisation principles and ap-

propriate concepts from pedagogues of information visualisation (e.g. Mazza,

2004; Spence, 2014) that were used in the design of visualisation in this thesis.

17

The data that clinicians deal with comes from plethora of sources; verbally re-

ported data, sensors, health monitors, clinical tests etc. Card et al. (1999) men-

tion a number of points to consider before information visualisation visual repre-

sentations of data can be made:

Data measurements – Nominal data is categorically discrete data such as

(e.g. behavioural, genetic, social, demographic). Ordinal data has a natural

ordering but the intervals between values are not the same (e.g. high, me-

dium, low risk). Interval data is numerical data (e.g. integers or real numbers).

Data dimensions – univariate (1 dimension), bivariate (2 dimensions), trivari-

ate (3 dimensions), and multivariate (4 or more dimensions).

Data structure – linear (made up of arrays, tables, lists etc.) temporal, spatial

or geographic (maps), hierarchical (taxonomies, genealogies etc.), network

(graph structures).

Interaction type – static (print), transformable (user can manipulate)

Visualisations can be an effective way of representing information if designed

well. People assimilate information much more rapidly through visualisations than

they do through text (Ware, 2013). This section will cover the ways in which vis-

ual elements can be used to facilitate this.

Visual variables create mappings and structures; these should pull out interesting

features from the data. It is possible to take advantage of pre-attentive pro-

cessing to design effective visualisations. In this case, defined as the term as-

signed to objects that are processed faster than 10ms (Treisman, 1998):

Form – line direction, size, curvature, grouping, marking, and luminosity

Colour – hue and intensity

Motion – flicker and direction of motion

Spatial position – position, stereo-depth, convexity and concavity

18

Figure 3 Visual types. Image by

Krygier & Wood (2005).

Bertin (2010) identified attributes

that he called ‘retinal variables’

in his 1967 work, Sémiologie

Graphique. Each of these

variables were identified as best

used to show either or both

quantitative and qualitative data.

Krygier & Wood (2005) expanded

on these ‘retinal variables’ – size,

colour value, texture, orientation,

and shape – by representing

them in points, lines, and

areas (Figure 3).

Visual properties refer to the way in which we are able to create differentiation in

the visualisation and effectively show representations. Fry (2004) identifies con-

trast as the most fundamental visual property. Gestalt principles (Wagemans et

al., 2012) explain how we notice visual elements as being contrasting. Pre-atten-

tive features are all ways to differentiate or contrast visual elements. Hierarchy is

about the order of importance of elements; visualisations should emphasise ele-

ments important to the task and de-emphasise those that are not, this can be

achieved through creating a hierarchy. Grouping is about clustering elements to

imply a relationship or shared meaning. Grouping creates patterns; dissimilar ele-

ments that are grouped together can also highlight differences or contrast.

Weight of elements such as the size or thickness of lines can show relative im-

portance or differentiation. Prominence should always be on ‘showing the data’

(Tufte, 1995), distracting with design runs the risk of data representations being

missed. Use of borders must be carefully thought out as not to increase the

amount of ‘non-data ink’.

19

Figure 4 Hue,

value, and

saturation.

On a screen, colour is represented by a combination of red, green, and blue.

When referring to colour, the model of hue, brightness, and value

is better understood by the human mind (Figure 4). The hue

is what would usually be meant when colour

is mentioned (for example green or

magenta) value is the range

of black to white, and

saturation is the intensity

of the colour. Colour is useful

for contrast and mapping data.

Placement conveys hierarchy by ordering elements. Contrast can be shown

when an outlier is placed away from a group of similar elements. Grouping is the

principal use of placement.

A problem with displaying complex data is that it cannot be easily displayed in

one view. The user must be able to transform the view in order to use externali-

sations to forage for information.

A number of taxonomies have been proposed by researchers such as ‘overview,

zoom, filter, details-on-demand, relate, history and extract’ from Shneiderman

(1996), and ‘zoom, pan, scroll, focus+context and magic lens’ by Spence (2014).

These taxonomies describe low-level interaction techniques. We refer to the in-

teraction techniques proposed by Shneiderman (1996) when we talk about con-

crete operations in the visualisation that do not imply the cognitive aspect of user

intent.

20

Yi, ah Kang, Stasko, & Jacko (2007) present a taxonomy based on user intent, or

what the user aims to do by interacting with the system, thus adding a cognitive

dimension to interaction. In this thesis, we adopt this taxonomy to refer to user in-

tent and the user tasks the visualisations were intended to support. The following

are the seven user intent interaction techniques proposed by Yi, ah Kang,

Stasko, & Jacko (2007):

Select – mark something as interesting

Explore – show me something else

Reconfigure – show me a different arrangement

Encode – show me a different representation

Abstract/Elaborate – show me more or less detail

Filter – show me something conditionally

Connect – show me related items’

Armed with a toolbox of sensemaking and information visualisation knowledge

we can move toward creating a tool that supports the clinician in understanding

risk in a patient.

21

3 METHODS This chapter describes the participants and recruitment, apparatus and materials,

data gathering methods, and ethical considerations of the study.

The study was comprised of a total sample of 10 participants. Eight of whom

were male and two were female. All participants were doctors from primary or

secondary care. Participants were either general practitioners or specialists. Nine

participants were from the UK and one participant was from the USA.

The study was divided into three stages; six participants took part in Stage 2 and

ten participants took part in Stage 3 (including the six from Stage 2). Sessions

were performed face-to-face (F2F) where possible, but some sessions had to be

performed remotely for pragmatic reasons.

Participant summary:

Number of participants: 10 (8 male, 2 female)

Inclusion criteria: clinicians in primary or secondary healthcare who need

to about ‘risk’ of developing diseases when dealing with patients

Demographic: 9 United Kingdom, 1 United States of America

Purposive sampling (Jupp, 2006), a form of non-probability sampling, was the

main method for recruitment. A variety of specialists and general practitioners

were selected as the sample of they matched the inclusion criteria. A range of

specialists and general practitioners were chosen in order to gain an understand-

ing of the similarities and differences in the way various clinicians think about risk.

Recruitment was carried out through email and word of mouth using industry and

academic connections.

22

Figure 5 (A) Lenovo Thinkpad

with Stage 1 Visualisation 1 on

screen. (B) Samsung 700T

with Stage 2 visualisation

on screen.

The following software, hardware, data gathering tools, printouts & documents

were used in the study:

Adobe Creative Cloud for creating designs; Stage 1 & Stage 2

Axure RP for creating the Stage 3 visualisation

Camtasia Studio® for capturing audio and screen activity

Voice recorder as a back-up for capturing audio

Lenovo Thinkpad 3680K16 laptop, Windows 8 64-bit, Intel® Core™ i5 CPU

M540 2.53GHz, 4096MB RAM, 1280 x 800 resolution, seen in Figure 5(A)

Samsung 700T tablet, 1366 x 768, seen in Figure 5(B)

10 x Participant information (Appendix A)

10 x Consent form (Appendix B)

2 x Interview plan: 1 x Stage 2 (Appendix C) & 1 x Stage 3 (Appendix D)

Notebook for taking notes during interviews and think-aloud

Pink and yellow Post-it® notes for creating the affinity diagram

Coloured pens for colour-coding the affinity diagram

A B

23

A semi-structured interview (Gillham, 2005) format was used – rather than struc-

tured or open interviews – in order to strike a balance between structure and

openness. This enabled a wide scope of questioning whilst remaining on topic.

This method was useful for both gathering requirements for re-design of the visu-

alisations and understanding how clinicians think about risk using complex data.

Semi-structured interviews made it possible to cover important questions while

also allowing for the pursuit of unanticipated themes as they arose. Interview

guides were used to guide the researcher’s line of questioning. Audio recording

of the semi-structured interviews was used to transcribe the interviews.

The purpose of the initial part of the interview was to make the participant com-

fortable, and learn about the clinical work they are involved in. This was useful to

understand the context of the problems they face, since participants came from a

variety of specialisations and medical fields. The middle of the interview was ded-

icated to constructing a deep understanding of participants’ work and sensemak-

ing activities. The aim of this part of the interview was to understand how clini-

cians make sense of complex health data in order to assess risk in a patient,

leading to a better informed understanding of how it might be possible to design

tools to support this. The end of the interview was used to bring up any lingering

points that the participants felt had not been covered.

Focus was always on framing the questions in real-life incidents that the partici-

pants had encountered, they were encouraged to talk about specific incidents ra-

ther than the general. The interviews were also a good way of identifying possible

biases within the sample (e.g. differences between medical fields), helping to mit-

igate those biases when analysing the data.

Nevertheless, interviews do not always elicit all interesting information from par-

ticipants; things that are obvious to the participant but not to the researcher may

be overlooked and remain unmentioned, therefore it was also beneficial to use

24

think-aloud protocol. Conversely, participants will not mention every part of their

thinking during the think-aloud either because they do not think it is important, or

because they are not consciously aware of the particulars of their thought pro-

cesses. Interviews are a good way to extract the information that might be omit-

ted in the think-aloud session.

Think-aloud protocol (Boren & Ramey, 2000) was used in Stages 2 and 3 of the

study, whereby participants verbalised their thoughts as they completed a task.

Verbalising their thoughts helped to highlight differences in the user’s mental

model and the system image. Think-aloud data contributed to the iterative devel-

opment of the visualisations used in the study by fixing the limitations of the pre-

ceding designs (Ericsson & Simon, 1993).

Before the session started, participants were provided with detailed instructions

of how to think aloud and were encouraged to speak freely as they noticed things

in the visualisation. Whenever participants stopped thinking aloud, due to becom-

ing too involved in the task or forgetting to keep reporting verbally, they were

prompted (by the moderator) to continue. Care was taken to remain sensitive to

when the participant needed prompting, to prevent interruptions in the partici-

pants’ thought processes.

Unnecessary questioning was avoided, as users who are asked for information

about something they are not attending to in the think-aloud are forced to infer ra-

ther than recall their mental processes (Anders & Simon, 1980), leading to inac-

curate reporting.

The advantage of having participants verbalise their thoughts over merely ob-

serving their activity, was that it enabled the articulation of their understanding of

the activity.

25

A risk assessment was completed and ethical approval for this study was gained

through University College London Interaction Centre.

Informed consent was given by all participants after they read the participant

information and signed the consent form. The participant kept one signed copy of

the consent form and the researcher kept another. Participants were all healthy

adults and did not belong to vulnerable or dependent groups.

The study adheres to the Data Protection Act 1998. Data was gathered with

consent, kept confidentially and securely. All participant data was anonymised

and made unidentifiable in reports and other shared materials.

26

27

4 DESIGN, STUDY & ANALYSIS This chapter presents a chronological account of the steps taken to

generate the results of this thesis. We describe the details of the ob-

jectives, study design used with participants, visualisation design,

analysis, and outcomes of each stage. The motivations behind the

approach taken, as well as reflections on the strengths and weak-

nesses of those approaches are also presented throughout.

To begin with, we cover the development of the patient personas for

use in the visualisations. Then, the three stages of the study are pre-

sented. Firstly, the iterative design and evaluation of the two visuali-

sations in Stage 1 are described. Secondly, we explain the parallel

design approach to the two Stage 2 visualisations, the interview and

think-aloud procedure with participants 1-6, and then Stage 2 analysis

& requirements generation. Thirdly, an account of the final design of

the single Stage 3 visualisation, the study protocol with participants 1-

10, and Stage 3 analysis is given.

Three patient personas were created for the purposes of the study.

Unique patient information was generated for each of them. These

personas were not based on real patients or real patient data, the pa-

tient persona information was synthesised from research into particu-

lar conditions.

The personas were created with characteristics that would not clearly

place them at exceptionally high or low risk of a condition, this was

done in the hope that it would tease out the way in which participants

thought about risk.

28

Figure 6 - Patient

personas for use

in the think aloud

scenario and to

populate the tool

with data.

In order to verify the personas, we consulted with a nurse in the

healthcare industry to review them; the feedback was used to update

and improve the persona data to be more representative of a ‘typical

patient’.

Persona A was Diego Blanco, a 35-year-old male with potential type

2 diabetes risk. Persona B was Deirdre Maguire, a 64-year-old female

with potential melanoma risk. Persona C was John Smith, a 75-year-

old male with potential lung cancer risk. These personas had thirty or

more pieces of patient information each. Each piece of patient infor-

mation (ethnicity, BMI, diet etc.) was placed into a category (social

history, clinical stats, behavioural etc.). The information was then as-

signed a value (Hispanic, 29, high calorie etc.), metadata (eats out

with clients, low activity due to family life etc.), and risk severity (gen-

eral patient information, reduced risk, low risk, high risk etc.). The pro-

file photographs and information for patient personas can be seen in

Figure 6.

There were two uses for the personas; first, the data set of each per-

sona was used to populate the visualisations with data, and second,

the personas were used in the think-aloud sessions to introduce the

scenario and the task where participants evaluated a patient persona

for risk of developing a specific condition in the near or distant future.

29

The aim of Stage 1 was to use the persona data to produce a concept

design that informed the visualisations in Stage 2. An iterative ap-

proach (Nielsen, 1993) was adopted to improve the Stage 1 visualisa-

tions. Two visualisations were created in Stage 1. Multipage PDF

documents were created as the artefacts for each of the two visuali-

sations in Stage 1. Each page of the PDF had one screen of the inter-

face on. Both Stage 1 visualisations relied on existing information vis-

ualisation literature (Section 2.2.2) as a reference for design. Expert

evaluation with an information visualisation professional was used to

evaluate and provide recommendations for the next iteration.

The first Stage 1 visualisation consisted of two views; Compound

View and Category View. A risk severity number on an interval scale

of 1-8 was assigned to each risk factor; the designer assigned this ar-

bitrarily, but it can assumed that real software would use relative risk

from population studies to assign severity. Patient information was

plotted in circles along the x-axis according to their risk severity. A

search bar was present in both views to perform a query on the da-

taset to reduce the amount of data that is visible on screen. For ex-

ample, filtering by category or individual risk factor. Clicking on an in-

dividual circle would bring up information associated with that piece of

patient data, thus providing details-on-demand to the user upon re-

quest.

The Compound View, illustrated in Figure 7(A), gave an overview of

the whole dataset. In this view, the circles were pushed outward on

the y-axis subject to the amount of other circles already in that area,

making the concentration of circles larger with the intention of making

30

Figure 7 Stage 1 iteration 1.

(A) Compound View.

(B) Category View.

the distribution of risk more apparent pre-attentively. An overall risk

value was assigned based on the average distribution of circles. The

circle colour was mapped to patient information categories. A different

set of colours was also assigned to risk factors that were changeable

(can change) and risk factors that that were not changeable (can not

change) through intervention; a key was placed in the bottom right

hand side to act as a reference.

In the Category View, illustrated in Figure 7(B), the categories (ge-

netic, medical history etc.) were divided along the y-axis and as-

signed to lines along the x-axis. The average risk for categories was

displayed, showing the distribution of risk within individual categories.

31

The second Stage 1 visualisation (Figure 8) mapped changeable/non-

changeable to shapes in order to decrease the amount of colours

used; circles showed ‘can change’ whilst triangles showed ‘can not

change’. The 1-8 interval scale was divided into an ordinal scale of

protective, neutral, and low, medium, med/high and high risk. Filters

were added to show what the patient could change and what the clini-

cian could change through intervention, allowing the user to adjust.

Figure 8 Stage 1 iteration 2.

Category names and colours

updated from previous version.

Filters added to search types

of data. Triangles and circles

differentiate between ‘can’ and

‘can not change’. Risk has

been split into protective,

neutral & low-high relative risk

for each data point.

(A) Compound View.

(B) Category View.

32

By filtering, the user was able to see something conditionally, for ex-

ample ‘show me things that the patient ‘can change’ that have been

proven to influence the risk of type 2 diabetes’.

Expert evaluation was practical because it could be done at any time

and with minimal resources, providing a satisfactory cost-benefit ratio

(Nielsen, 1994). In contrast, issues can be missed (validity) and differ-

ent experts can find different issues (reliability). A data visualisation

expert was consulted and a ‘simplified think-aloud’ was carried out to

identify and provide suggestions for re-designs. This type of evalua-

tion is no replacement for real users; this method was used in order to

resolve basic usability and design issues before sessions with real

participants, so that the focus of testing would be on how the visuali-

sation supports clinicians’ sensemaking. Below are some examples of

the issues that were identified through the expert evaluation:

Categories are a nominal type of data. In the visualisations, the mis-

take of representing the data in a way that implicitly suggested an or-

der to it was made. The colour choices in both Stage 1 visualisations

were sequential in nature, using diverging schemes of colour led to

confusion. Using progressive variations transitioning between hues

suggested continuity, something that is not present in the categorical

or nominal data (Silva, Sousa Santos, & Madeira, 2011).

Since we assumed that actionable data is what makes clinical inter-

vention possible, we wanted to see if visualising the factors that could

be changed through intervention would support clinicians thinking

about risk. The first visualisation used colour to differentiate changea-

ble factors, while the second visualisation used shape to represent

the same thing. Although the latter was more effective, it was not use-

ful for clearly seeing how the risk is weighted among changeable and

non-changeable factors.

33

Table 1 Visualisations and patient

personas matrix showing the six

variations.

Horsky et al. (2012) state that ‘poor usability is one of the core barri-

ers to adoption and a deterrent’ to use of clinical decision support

systems. Upon reflection, using the iterations in Stage 1 to identify

and fix usability and design problems provided a good foundation for

designing better visualisations in the sessions with real participants.

Following Stage 1 visualisations, work begun on Stage 2 visualisa-

tions where a parallel design approach (Nielsen & Faber, 1996) was

used. The rationale behind using parallel design was that less time

was required to explore designs than if they were produced sequen-

tially. Parallel design was useful for testing and comparing visualisa-

tion types, presenting the same data set in different visual structures.

Two designers worked simultaneously in this stage; each worked in-

dependently on different visualisations. The Principal Investigator

worked on Visualisation 1 and a different designer worked on Visuali-

sation 2. Two separate designs were created for Stage 2; the three

personas and their data were used to populate each of the designs.

As before, the designs were exported to multipage PDFs, this time

the PDF documents were created with interactivity in the form of click-

able parts of the interface that were hyperlinked to other parts of the

document. A total of six variations were created; these can be seen in

Table 1. The artefacts created in Stage 2 were used on a laptop with

participants during the think-aloud part of the study protocol.

Persona A Persona B Persona C

Visualisation 1 1A 1B 1C

Visualisation 2 2A 2B 2C

34

In Visualisation 1, illustrated in Figure 9, the colour scheme for cate-

gories was changed in an attempt to avoid implying magnitude differ-

ences between categories. Differences in hue with only slight differ-

ences in the lightness were used to differentiate categories, but using

nine colours to represent the categorical data made it hard to discrim-

inate between categories. As a consequence, the ability for the user

to memorise the meaning of each block in the visualisation was di-

minished, MacDonald (1999) suggests using seven or less colours to

show data of this kind. In Stage 2 Visualisations, the name of the per-

sona was added to the title, along with the condition they were sus-

pected of being at risk of. In addition, a frame at the bottom-centre of

the screen was added with the patient name, photograph and dummy

text for notes on the patient.

Figure 9 Stage 2

Visualisation 1 with

Compound View

selected.

35

Figure 10 Compound View

with ‘early osteoporosis’

selected in the main

visualisation, thus

changing the bottom-

centre frame content to an

image of the patient’s

DEXA scan with notes.

The content of this frame changed as the visualisation was interacted

with, as can be seen in the Compound View (Figure 10). The selected

block was highlighted with an orange outline and a line connected the

selected block to the bottom-centre frame, inferring a relationship. We

also included a maximise button in the top right hand of the frame to

expand the window and zoom into the data subset. This interactivity

was also present in in the Category View, illustrated in Figure 11.

In the Compound View, each item of patient data was represented by

a solid block of a fixed size. These blocks stacked upon each other

additively in columns out from the central line that divided factors that

‘can’ and ‘can not change’. This allowed spatial grouping to be used,

instead of colour or shape, to differentiate between the entities ‘can

change’ and ‘can not change’. This made it possible to distinguish the

36

Figure 11 Category

View with ‘nevi checked’

selected, notes and an

image of the patient’s

nevi is displayed in the

bottom-centre frame.

weighting of risk in each risk severity group through the height of the

block stacks from the distance the stacks protruded from the

mid-section.

In the Category view, ‘can change’ and ‘cannot change’ were still rep-

resented with shape. However, instead of using triangles and circles,

as was the case in the second Stage 1 visualisation, squares repre-

sented ‘cannot change’ and squares with rounded edges represented

‘can change’. The user is able to encode the data in a different repre-

sentation by shifting views from Compound view to Category view

and vice versa.

37

Figure 12 Visualisation 2.

Three columns (left to

right) show factors that

reduce risk, patient

information and factors

that increase risk.

In Stage 2 Visualisation 2, illustrated in Figure 12, three separate col-

umns were used to visualise data. The central column contained the

total information that was available for that patient. The left column

represented factors that reduce risk and the right column represented

factors that implied increased risk for the condition being assessed.

In Visualisation 1, the severity of risk of an individual data point

relative to a condition was shown by separating data into columns;

low, medium and high risk. In Visualisation 2, the severity of risk was

represented by the size of a block in the ‘Increase Risk’ column. Lines

were drawn from the central ‘Patient Information column to show and

infer relationships between factors that reduced, increased, or had no

direct correlation to risk.

38

Figure 13 Individual data point (A) Exposure to

radon. Map from ukradon.org (B) Lung X-rays.

Clicking on a data point would enlarge the

patient information bar and move it to the

left hand side of the screen and let the

user look closer at the data subset for that

patient data (Figure 13); what Tufte (1995)

would refer to as a micro view or what

Shneiderman (1996) refers to as zoom.

Zooming added deeper contextual text to

the enlarged patient information column

for each data point. A red line on the right

side of a data point in the patient

information column meant that it was a

risk factor, while a green line on the left of

the data point meant it was a risk reducing

factor.

The additional properties related to a data

point were displayed on the right hand

side of the zoomed in patient information

bar. Some of the screens displaying

zoomed in patient information, such as in

Figure 13(A), had the design error of

heavy use of thick lines as borders in the

design placing less prominence on

‘showing the data’ (Tufte, 1995). This led

to users becoming distracted with the

design rather than with the actual data the

design was trying to display; excluding

this from the design would have reduced

the amount of ‘non-data ink’.

A

B

39

In advance of the actual study, the protocol was piloted with a USA

based doctor in the healthcare industry in order to modify and im-

prove interview questions and think-aloud procedures. The feedback

from the pilot session has not been included in the sample. Prior to

conducting the sessions, participant information and consent form

documents were sent in an email to participants 1-6 (Table 2). The

body of the email confirmed the time and place of the interview (con-

ference call details if session was remote).

Upon commencement of the session, the Principal Investigator – ac-

companied by his Industry Supervisor – asked participants if they had

any questions about the study and if they understood everything in

the participant information. Upon confirmation that the terms of the

study were understood, consent forms were signed and collected (a

signed and scanned version of consent form for remote sessions).

Participants were given a short introduction to the procedure that

would follow; firstly, an interview involving current workflow, decision

making processes, the actionability of data, how they make sense of

the information they deal with, the various sources of data, trustwor-

thiness of data and communicating risk to others. Secondly, a think-

aloud session that would involve looking at two different visualisa-

tions, where the participants would explore a patients risk of develop-

ing a condition.

The researcher then started audio recording on the voice recorder,

and audio & screen recording on the laptop using Camtasia Studio®.

The researcher also took notes using a notebook and pen during both

interview and think-aloud.

The semi-structured interview, described in Section 3.2.1.1, took

place. The researcher asked the main questions and followed up with

40

Table 2 The column ‘Vis’ shows the visualisation order and patient persona used during the think-aloud. For

example, participant 3 saw visualisation 1 with patient persona A followed by visualisation 2 with persona C.

additional questions to probe further. Clarifying questions were asked

when a point was unclear or confirm of what a participant meant want

needed. The interview took around 30 minutes per participant.

Once the interview part of the session was over, the researcher ex-

plained think-aloud protocol (Section 3.2.1.2). Participants were told

that they would explore two visualisations representing complex

health data about a patient, and were asked to talk about what they

understood from the interface and how it might relate to their task of

assessing risk. The scenario was explained; the hypothetical patient

in question was sitting outside in the waiting room and that this was

the first time that the participant was viewing the patient’s data. Their

task was to discern whether the patient was at risk of developing a

condition based on what was understood from the interface. The

think-aloud session took around 30 minutes per participant. Around

15 minutes was spent thinking aloud about each visualisation.

Upon completing the interview and think-aloud the session was con-

cluded; participants were de-briefed and asked if they would be will-

ing to be re-contacted for Stage 3 of the study. Once all six Stage 2

sessions had been completed, analysis took place.

Participant Gender Medical field Location Vis F2F

P1 M Cardiology UK V1A, V2A Yes

P2 M Cancer genomics USA V2B, V1A No

P3 M Clinical Pharmacology & General Medicine UK V1A, V2C Yes

P4 M Psychiatry UK V2B, V1B No

P5 M General Practitioner UK V1B, V2C Yes

P6 M Paediatric Pathology UK V2C, V1C Yes

41

This section describes the approach to analysis, findings from the

Stage 2 study with participants 1-6, and requirements statements.

The main purpose of analysing data after the Stage 2 study was to in-

form the design of the Stage 3 visualisation. However, findings from

Stage 2 were also integrated into results in Sections 5.1-5.4.

To begin Stage 2 analysis, all interview and think-aloud data was

transcribed word-for-word using the audio from the voice recorder.

When it was not clear which part of the interface a participant was

talking about in the think-aloud recordings, the screen recording was

used as a reference.

Interviews were annotated in a word processor with approximate

codes. Recurring patterns in the way clinicians think about risk, prob-

lems they face, work practices, work environment, and attitudes to-

ward data helped to form initial interpretations.

The majority of analysis in Stage 2 was on the think-aloud data. The

think-aloud transcripts were printed onto A4 paper and a highlighter

was used to mark substantive statements for each participant.

Post-it® notes were then attached to these substantive statements,

the theme of the statement was summarised and an identifying code

was written on the Post-it® to recognise where the data came from in

the transcript. The letters and numbers in square brackets in Section

4.3.5.1 follow the same order as these codes; participant number, vis-

ualisation number, patient persona. For ease of visual differentiation,

Pink Post-it® notes were used for visualisation 1 and yellow Post-it®

notes were used for yellow visualisation 2.

42

The following pattern was used to identify where data came from:

Once all of the Post-it® noted has been added to the transcript high-

lights, an affinity diagram (Hartson & Pyla, 2012) was used to aid

analysis (Figure 14).

Large sheets of A1 paper were taped together and the Post-it® notes

were taken from the transcripts, placed in clusters with similar

themes, and given topical labels on the A1 paper. These clusters

soon formed groups within hierarchies; these were all labelled. Even-

tually a structure began to emerge.

Figure 14 Affinity

diagram created

during analysis.

42

43

Through the creation of the affinity diagram, both interface specific

observations and abstract findings about clinicians needs began to

emerge:

Although participants said Visualisation 1 was a tool with a structured

presentation [P1V1A], they had to deal with various data at the same

time [P1V1A].

There were also usability issues; for example, a participant was con-

fused about the horizontal relationship of data points in the Com-

pound View, even though there was no meaningful relationship in-

tended in the design of the visualisation [P4V1B].

It was noted that this tool might be good for a specialist consultation

[P1V1A] rather than for a busy general practitioner, since it took a

while to digest all of the information on the screen.

43

44

Visualisation 1 had too much data to process at once, leading to par-

ticipants asking for more ‘black and white’ and ‘yes and no’ [P1V1A].

In Visualisation 2, participants said that the interface was putting all

variables for assessment in one place [P6V2C], which gave a nice

sense of the data collected and how it influences risk up or down

[P2V1B]. The simpler information structure of the overview in Visuali-

sation 2 helped participants to understand the three columns

[P2V2B]. However, it was not always apparent that the columns were

not equivalent; the size of the data points relating to severity of risk

was not a connection that was easily made [P4B2B], also having a

smaller pixel area for low risk factors made interaction a challenge

[P1V2A].

In both visualisations, users generally had trouble interpreting the sig-

nificance or meaning of category colours [P6V1C] [P6V2C] [P1V2A]

[P4V2B]. This led us to believe that using colour to differentiate be-

tween categories is not essential to the task of assessing risk. Using

colour that does not provide further insight for the user can be per-

plexing as they try to understand its meaning and, hence, should be

avoided (MacDonald, 1999).

The filters in both visualisations were not visually apparent to all users

[P4V2B] and the terminology was not well understood [P3V2C]. The

readability of Visualisation 2 was generally better than Visualisation 1;

only one participant noted the readability of the text was poor in Visu-

alisation 2 [P4V2B], in Visualisation 1 participants complained about

small boxes, end-of-line hyphenation, differentiation between square

and rounded edge squares & small text size [P1V1A] [P2V1A].

Although both visualisations succeeded in showing complex data

from disparate sources in one interface, an understanding of whether

the patient was at risk or not was missing. Information about how a

single data point correlates to risk (why it is placed in high, medium,

45

low or protective) [P6V2C] was not present. Adding context about

where a patient fits into a risk population would have helped.

Participants pointed out that an overall risk score was missing from

both visualisations, something the majority of participants noted

[P2V1B&2A] [P3V1A&2C] [P1V1A&2A] [P6V1C&2C]. Due to the vari-

ety of data represented in the visualisations, finding an existing risk

calculator or algorithm that considered all factors was not possible.

Nevertheless, including existing risk calculators to be applied on a

subset of data emerged as an option. In the end, risk is complex, but

in a clinician’s daily work, a summary is needed [P1V2A].

The think-aloud protocol used with the two Stage 2 visualisations

highlighted both strengths and weaknesses in their respective de-

signs. The interviews also revealed the way in which clinicians think

about trusting data, actionability, workflow, attitudes, understanding,

and communicating risk.

Following the completion of the affinity diagram, the designers dis-

cussed possible design ideas and any unanswered questions or

holes in the data that required further investigation in Stage 3. The

learnings were merged from both the think-aloud and semi-structured

interviews into a requirements statement document (Appendix E).

When moving from codes in the affinity diagram toward requirements,

focus was on matching the internal mental model of clinician through

externalisations. The rationale behind the requirements were dis-

cussed and design recommendations were made. Care was taken by

the designers to avoid bias toward their own competing design ideas

by looking objectively at the findings in the data when creating the

requirements document.

46

Table 3 Requirement

Statement structure.

The requirements statements follow the structure shown in Table 3. A

traffic light metaphor was used to show priority, red (top) being high-

est priority and green (bottom) being lowest. Textures have been re-

dundantly mapped to the colours for colour-blind readers.

Priority was assigned requirements according to the severity of the

problem, which was derived from, to how frequently the topic or con-

cept in question came up in interviews and think-aloud data and the

amount of design work required to implement the change.

An assortment of requirements statements have been taken out of

the ‘Requirements Statement’ document (Appendix E) to show

as examples. These can be found in Table 4.

Requirement Statement Priority

#: Name of feature/category

Second-level feature/category

Requirement statement [place in Affinity Diagram]

Rationale (if useful): Rationale behind requirement

Design recommendation (optional): Commentary about requirement

47

Table 4 Selected examples from

Requirement Statements

document

5. User input

Editing risk category

If there is no/low evidence for a data point, allow the user to assign a risk category. But, do not al-

low the strongly evidence based data points to be moved. Track all changes and show if an item

has been moved through the interface [information/supporting interpretation &

‘objective’ vs. ‘subjective’ data]

Rationale: ’Solid’ data has its limitations [P1V1A]. For example, occupational history is useful

[P5C1B], but interpretation is subjective. In clinical work evidence & subjective opinions are mixed,

this tool gives an objective view of both that can be reviewed [P6V1&2]

Design recommendation: The data points that aren’t used in validated scales can be re-assigned

to another risk category, but these changes must be tracked for later review

9. Individual data point view

Relative risk

Let the user know how a single data point correlates to risk [P6V2C] [risk type>relative risk]

Rationale: Relative risk is not known for all of the data points, but it would be helpful is it was avail-

able for those points that are known [P2V2B]

Design recommendation: Display relevant patient data on classic graphs & scales within the indi-

vidual data point view. Let user know the high/med/low in the overview is based on the relative risk

of that specific measurement in order to ground the perspective [P2V1A]

17. Overview

Overall risk calculation

The user needs an overall risk calculation that gives a quantitative measure [interface specific]

Rationale: Overall risk was missing from both visualisations, something the majority of participants

noted [P2V1&2] [P3V1&2] [P1V1&2] [P6V1&2]

Design recommendation: Individual conditions have their own risk calculators (i.e. risk of diabetes

in 5 years is X [P3V1A]). Include a risk calculator to display overall risk

48

After defining the requirements, the design and implementation of the

Stage 3 Visualisation (also referred to as ‘the tool’) started. This in-

volved merging the best parts of the two versions produced in Stage

2 into one design.

In order to do this, the two designers used the requirements state-

ment document created from Stage 2 learnings. In Stage 2, the de-

signers worked separately, but in Stage 3, they worked collaboratively

on the visualisation.

The merged design aimed to apply findings about how clinicians think

about and make sense of risk in a patient by creating tool that re-

duced the gap between the clinicians’ mental model and the data in

the externalisations that they currently work with.

Before the design work begun, personas ‘A’ and ‘B’ were improved

based on feedback from participants in Stage 2. Persona ‘C’ was not

included in Stage 3 design due to temporal constraints in the develop-

ment period; this persona was discarded because it produced the

least interesting data during the think-aloud sessions.

Two artefacts were produced; one visualisation populated with per-

sona ‘A’ data and another with persona ‘B’ data. The Stage 3 visuali-

sation was created in the rapid prototyping tool, Axure RP, and then

exported to HTML and JavaScript for use on a Samsung 700T tablet

in Stage 3 of the study.

49

Figure 15 The overview shows

the title bar, tool bar, patient

information, risk factors.

This section will discuss the design of each part of the tool and the ra-

tionale behind the design. The interface was interactive but some

parts such as ‘Go To Investigation’ were not functional. Data showed

that they were important to clinicians’ needs and understanding, but

beyond the scope of what could be implemented. They were included

in the interface to hint at what their functionality might be.

49

50

The overview screen (Figure 15) was made up of a number of parts;

the header, patient information and risk factors. In Shneiderman's

(1996) mantra, ‘overview first, zoom and filter, then details-on-de-

mand’, overview refers to the act of looking at ‘the big picture’. The

overview screen in the tool does this by presenting all of the patient

data in a macro view (Tufte, 1995). Spence (2007) speaks of over-

view as the ‘qualitative aspect of some data’ – in this case the pa-

tient’s risk of developing a condition – that is ideally ‘acquired rapidly

and, even better, pre-attentively’. This screen attempted to achieve

this by showing the clinician factors that are known to contribute to

risk, how severe they are, and whether those factors can be changed.

Since user attention is first attracted to visually strong (big, colourful,

prominently placed) objects, the design attempted to lead users’ gaze

towards high risk factors. The user can then search for details among

less prominent elements in patient information. This was key because

if information is not organised in an optimal manner overview (Yi et

al., 2008) clinicians could potentially be stuck in the foraging loop (Pi-

rolli & Card, 2005) for longer than necessary.

The header, illustrated in (Figure 16), contained the title bar, serving

as a reminder of the purpose of the tool. Under that, basic patient in-

formation and photograph helping identification of the patient, useful

when a clinician deals with many patients. On the bottom, the condi-

tion the patient was being assessed as being at risk for, risk calcula-

tor selection, search bar, and buttons to lead onto next steps (which

were beyond the scope of this thesis) within the clinical workflow.

Figure 16 Header at the top of

the tool. Different conditions

could be selected from ‘Risk of

developing’ and risk calculators

can be applied from the

dropdown ‘Apply calculator’.

51

Figure 18 Risk factors.

Figure 17 Patient Information.

Hierarchies and structures refer to elements within elements. They

also refer to an element that has a pointer to another element. In the

‘Patient Information’ (Figure 17) section of the overview the patient in-

formation was ordered into categories using lines to divide them. This

was designed in a structure that follows the clinical workflow that par-

ticipants described in Stage 2 interviews and think-aloud.

The coloured strips on the right hand side of the boxes relate to the

level of risk that piece of patient data is thought to have relative to an

empirical population study. All information available about a patient is

displayed here.

There is a danger that, if a part of the interface that contains less im-

portant data draws attention, features that are more important might

be overlooked. We tried to make sure that the most prominent fea-

tures of the interface were also the most significant parts of the data.

By placing all factors that are known to contribute directly to the risk

of a condition in one place (Figure 18) the clinician is able to conserve

mental resources that would otherwise be spent searching for risk

factors.

The location of elements also affects how a screen is viewed. For this

reason we grouped changeable factors separately from non-changea-

ble, indicating to the clinician which data is actionable. High risk fac-

tors are represented with highest saturation red and low risk factors

with the lowest saturation red.

52

Figure 19 If the risk

calculator runs when

information is missing,

a prompt appears

requesting missing

information.

Selecting a risk calculator from ‘Apply Calculator’ would bring up an

overlay that superimposed a yellow hue on the factors that the calcu-

lator took into account. As a reference, the NHS (National Health Ser-

vice, 2013) calculator was used with the persona information to cre-

ate a more realistic outcome. If patient information that the calculator

algorithm needed was missing, as shown in (Figure 19), the tool dis-

played an alert prompting the clinician to gather that information.

52

53

Figure 20 The completed

risk score calculation.

NHS Diabetes Risk

Calculator available from:

www.nhs.uk/Tools/Pages

/Diabetes.aspx.

The factors that were present and contributed to risk had a line drawn

across from ‘Patient Information’ to ‘Risk Factors’. By connecting two

elements, it is possible for a relationship between the two to be

shown. This feature was meant to allow the user to differentiate be-

tween factors taken into account by the calculator that did contribute

to risk and those that did not contribute to risk in the patient.

53

54

Figure 21 Individual

data point view.

The advantage to overlaying the cal-

culator on top of the interface is that it

is plain to see which factors are iden-

tified as contributing to risk by other

evidence sources, but not taken into

account by that particular calculator.

Providing all the required information

was present, the risk calculator ran

and displayed an overall risk score

(Figure 20). This was the absolute

risk of developing a condition within a

timeframe. A segmented coloured

scale as well as descriptive text rep-

resented this.

Clicking on a piece of patient

information on the overview screen

would lead the user to an individual

data point view (Figure 21). This view

effectively zoomed into the patient

information bar, showing more detail

about each data point on the left and

providing extensive properties about

the patient information that was

selected. The individual data point

view displayed various types of

information depending on the type of

data; risk severity, relative risk (with

embedded visualisations to show it),

and contextual information.

55

Figure 22 Examples of the

embedded visualisations

within the tool.

Individual data points had embedded visualisations in order to repre-

sent the patient data in a way that was easy to make sense of it. Only

data that was better represented visually was represented this way,

for example natural language notes were better displayed as textual

information. A number of the embedded visualisations are shown in

Figure 22. Examples of these representations included the following:

Tables – these were used to show two-dimensional data. How-

ever tables are only useful when there are a limited number of en-

tries for the columns and rows, otherwise they get too crowded.

Line graphs – these are a number of data points connected by

lines, showing continuity across the values. Line graphs were

used to show data such as hemoglobin A1c levels over time.

Bar graph – were used to display series data where there

was no continuity between values. Bar graphs were used to show

data such as steps taken per day.

Geographical map – these were used to show environmental ex-

posure and post code information.

Matrix – wo dimensional sets such as the measures for BMI were

plotted in a matrix.

Tree – used when hierarchically ordered data is used. This was

useful for showing family heart disease history.

56

The bottom of the individual data point view had a series of buttons:

Notes – these were added to allow easy access

to notes when searching for information about a

data point or allowing the clinician to add notes

when they gain insight.

Guidelines – these were guidelines that link

directly to the National Institute for Health and

Care Excellence. This was added to support

clinicians through guidelines, advice and

appraisals.

Evidence – these were added for the times that

guidelines do not apply. Clinicians are able to

search for available evidence and latest literature

concerning a risk factor.

57

More – there were three buttons in this submenu.

‘Go to Investigations’ and ‘Go to Diagnosis’ were

placeholder to hint at following phases in the clini-

cal workflow. As they were beyond the scope of

this work, they were not made interactive. Change

risk classification opened a pop-up window when

tapped.

Change risk classification – allowed the user to

flag a piece of patient information with a risk factor

(e.g. if the clinician finds out that the patient

spends too much time in the sun that contextual

factor can be flagged). Marking helps users make

sense of a domain on their own terms and track

their own developing understanding (Huang &

Eades, 2013).

Confirming the action – The risk classification

change is flagged for review if the user continues,

this acts as a safeguard for rogue actions. The

user is prompted before the change is submitted

for review; adding an extra step to confirm in an at-

tempt to minimise error.

58

Following the completion of the tool, Stage 3 sessions with partici-

pants 1-10 were scheduled. Participants 7-10 were sent participant

information and consent form documents attached to an email, since

they had not taken part in Stage 2. Participants 1-6 simply confirmed

that they were still willing to take part in the study. Table 5 shows the

participants, order in which they saw the personas, and whether the

session was F2F or remote.

The protocol for Stage 3 sessions followed the same structure as

Stage 2 sessions, but with one main difference; the think-aloud was

carried out before the semi-structured interview. Sessions lasted

around 60 minutes, however this time the think-aloud sessions took

approximately 45 minutes and the interviews took approximately 15

minutes.

This time, when the think-aloud was explained, the participants were

asked to perform a number of tasks that were designed to test the as-

sumptions made about the visualisation being able to support clini-

cians thinking about risk:

Use risk calculator to see a risk summary for this patient.

Look through patient information and explore some of the data.

Flag a piece of social information as a risk factor because you

know (persona specific scenario) might be putting them at risk.

Participants thought aloud with persona A and persona B. About half

of the 45 minutes was spent viewing the visualisation with each pa-

tient persona.

The interview plan (Appendix D) in Stage 3 sessions was also fo-

cused on evaluating the tool and the way complex health data was

visualised, as well as confirming preliminary evidence from Stage 2

about how clinicians make sense of risk in the patients they deal with.

59

Participant Gender Medical field Location Vis F2F

P1 M Cardiology UK A, B Yes

P2 M Cancer genomics USA B, A No

P3 M Clinical Pharmacology & General Medicine UK B, A Yes

P4 M Psychiatry UK A, B Yes

P5 M General Practitioner UK A, B Yes

P6 M Paediatric Pathology UK A, B Yes

P7 F Ophthalmology (specializing in genetics) UK A, B Yes

P8 F Radiology UK B, A Yes

P9 M General Practitioner UK B, A Yes

P10 M General Practitioner UK B, A Yes

Table 5 Participants for Stage 3 study. ‘Vis’ shows the order that the patient personas were presented

to each participant during the think-aloud. For example, participant 10 saw the visualisation with

persona B followed by persona A.

60

A thematic analysis (Braun & Clarke, 2006) was used to identify, ana-

lyse and report themes in the data. This approach was used to give a

rich description, reflecting the predominant themes that arose within

the data set. The multi-stage approach to the study meant that Stage

2 was focused on understanding users’ needs and practices to yield

requirements for Stage 3 design, but the overall aims were to under-

stand how clinicians make sense of risk. As was the case with Stage

2 analysis, interviews and think-aloud sessions from Stage 3 were

transcribed word-for-word by the researcher. This helped with becom-

ing familiar with the data and forming initial interpretations (Riessman,

1993).

Learnings from Stage 2 analysis were carried over to Stage 3. The in-

itial themes and approximate codes were reviewed as new data was

incorporated from the ten Stage 3 sessions. Notes made by the Prin-

cipal Researcher during the sessions was also used to support the

other sources of data. Data from research notes, interviews and think-

aloud sessions was triangulated (Guion, Diehl, & McDonald, 2011),

increasing validity of results.

At first, notebooks and paper were used when the focus was more on

the exploration of the data. Preliminary codes were first added to the

transcriptions, these codes were sorted by recurrent patterns in the

data. This process was done in an iterative manner, themes were cre-

ated for similar codes, and themes were merged and adapted in light

of new data. The main tool for analysis then shifted to a word proces-

sor, making it easier to organise data in a proper structure. This was

done systematically whilst building up a narrative about each theme,

reviewing themes when inconsistences in the data emerged. This

was done iteratively until a clear narrative emerged within the data.

The findings from this process are presented in the next chapter.

61

5 RESULTS This section contains the findings that arose from the data. Firstly, the main find-

ings are stated and discussed in Section 5.1. Sections 5.2 to 5.5 present and dis-

cuss secondary findings, which include the way in which clinicians systematically

organise their work, think about risk, arrive at a decision, and communicate risk.

Finally, Section 5.6 contains findings related to the tool in terms of its efficacy in

using visualisation to support clinicians dealing with complex health data.

The data collected from think-aloud and interviews and subsequent analysis

revealed two main findings:

I. Clinicians use a number of disparate sources of information to make sense of

risk in a patient – this entails complexity. There is a discrepancy between how

clinicians talk about their work, and what they actually do when assessing risk

in a patient.

II. We have promising preliminary evidence from this exploratory study that

tools, such as the one described in this thesis, can support clinicians by

creating externalisations that facilitate the implicit processes that they use

frequently in their work.

Clinicians routinely use complex data from a number of different places to try

to form a unified understanding of a patient’s risk. This diverse data is usually

presented in a way that makes the relationships between potentially significant

information difficult to perceive.

Our findings show that clinicians speak of the data that they use when assessing

risk as being ‘limited to validated clinical data’. They routinely use studies based

on specific populations and tools such as risk calculators. However, clinicians ac-

cept that research evidence is an abstracted generalisation that does not always

represent the actual risk of the individual patients they deal with. Patients do not

62

always fit into the population of the epidemiological studies that they are com-

pared to. In reality, the patient may be found anywhere within the vast landscape

of risk. Similarly, guidelines are a common point of reference; however, they are

not always sufficient, as there are always outliers that do not fit the typical case.

The difficulty arises when it comes to making sense of what all of this actually

means for the patient that the clinician is attempting to assess.

Our participants described how they use studies, calculators, and guidelines in

conjunction with patient data related to risk of a certain condition, in order to as-

certain whether or not an individual is at risk and what – if any – intervention

should be taken.

Current health policy and evidence based medicine (Straus, 2011) state that clini-

cal practice is a scientific discipline where the science base is derived from ra-

tional, universal, and objective evidence. The rationalistic attitude of only utilising

validated empirical evidence to assess risk is not completely reflected in the way

that clinicians seek and use information in practice. We found that clinicians use

empirical tools and validated data alongside contextual factors specific to the pa-

tient they are dealing with when assessing risk.

In terms of data-frame theory (Klein et al., 2007), the empirical tools present data

that cause a connection to an existing frame about what the risk score or sum-

mary means. The clinician questions the frame by asking whether these

measures apply to the patient in front of them. The frame may be preserved if the

evidence is adequate enough for the clinician to confidently say that the risk data

about the patient matches their existing frame. If there are remaining questions

about whether the patient is at risk, the frame may elaborated as clinicians seek

and infer data in the current context (specific to the individual patient), through

adding and filling slots to build up a more comprehensive frame of the current sit-

uation. These contextual factors are not currently accommodated for by existing

externalisations that come in the form of risk calculator algorithms, guidelines and

population studies.

63

Clinicians apply their internal knowledge, the unstated, that when applied usually

presents itself through skilful performance in order to gain insight. This is what

Schon (1983) refers to as reflection in action. As part of the decision-making pro-

cess, clinicians apply expertise, reflect on their old model, and change their think-

ing to fit the new task. This process is internalised and unique to the individual. In

the same way, Polanyi (1983) speaks of this phenomenon as tacit knowledge.

This leads us to believe that the sensemaking process is more complex than

practitioners are able to verbalise. Henry (2010) supports this finding by saying

appreciation of the tacit dimension of knowledge ‘will help clinicians to build a

more accurate critical framework for evaluating what kinds of information are im-

portant for particular clinical decisions’ (p.296).

When elaborating a frame, clinicians do not simply amass data and elaborate

frames based on best fit, for this would result in erroneous frames being pre-

served. Instead, past experience, clinical expertise, and critical thinking are used

to inform this process. Novices can end up relying on anchors that are not correct

leading distortions or flawed interpretations of what risk data means, leading to

preservation of incorrect frames (Sieck et al., 2007). More experienced clinicians

are able to rely on their larger knowledge base to avoid such pitfalls:

P6 That pattern [some aspect of patient data] can either be caused by only

one thing or it can be caused by 100 things, and that's what comes from

basically learning and doing the job.

Understanding the epistemological aspects of how clinicians think about the data

they use for decision-making and assessment of risk, as well as the environment

and workflow that these processes are embedded in was crucial to the develop-

ment of the tool.

64

Through testing the tool on participants, we have found preliminary evidence that

it supports clinicians by making the implicit (knowledge that they utilise but do not

directly express), explicit. This allowed participants to gain insight and infer rela-

tionships between data through use of the tool.

One participant verbalised the way in which the tool attempts to support the way

in which clinicians apply general rules to specific cases using their internal

knowledge:

P4 I suppose experienced clinicians will have these gestalts, you know. You

see someone and you recognize a certain sort of configuration of features,

and if you then focus on bringing out important risk factors in this patient…

and here's the evidence base behind it... You are facilitating that process.

Risk assessment sits deeply in the context of clinical workflow. This process is

shown in Figure 23. Of course, this is an oversimplification of what actually hap-

pens in clinical practice; however, we found that the general process remained

the same regardless of whether the participant was talking about primary/second-

ary preventative/reactive care. This may have resonated with opinion throughout

partly because it reflects the way in which medicine is taught.

The evidence based medicine process (Gronseth, Woodroffe, & Getchius, 2011)

is aligned with the desired clinical workflow that clinicians expressed. As we can

see, the process is comparable. However, as highlighted by Mynatt (2011), fac-

tors such as diet, activity levels, social conduct other types of intervention, are

generally not viewed in their relation to disease until it presents itself.

In this model, stages may be skipped depending on the availability and confi-

dence that the clinician has in the information they have to work with. For exam-

ple, a clinician may move from Assess Info to Diagnosis if there is strong enough

evidence, or may go from Investigation to Patient monitoring if a test comes back

negative.

65

Figure 23 Model of

desired clinical

workflow expressed

by participants which

aligns with the

evidence based

medicine process

adapted from

(Gronseth,

Woodroffe, &

Getchius, 2011)

The workflow described by participants is as follows:

Assess info – reviewing existing information establishes a mental model of

the patient, there can be varying levels of data available to the clinician at this

point. This is where the tool developed for this study attempted to target the

needs of clinicians.

Examination – gathering medical history and taking an account of the pre-

senting complaint through physical examination of the patient. The tool only

emulated this part of workflow where the user had to enter missing blood

pressure information for the patient during the think-aloud.

Investigation – ordering or performing tests that are not part of the examina-

tion in order to find out something that the current information does not offer.

Diagnosis – establishing that the patient is suffering from the condition

Intervention – taking measures to improve decrease risk in a patient.

Patient monitoring – looking for changes in patient condition. A change may

result in the retrieval of schemas due to a new combination of data elements.

Patient-clinician communication and consultation – the constant exchange

between patient and clinician that may occur at all times during the clinical

workflow. Patients can be sources of data and collaborate with the clinician in

the data coverage loop ((Russell et al., 1993).

Clinician-clinician communication – consulting another clinician in order to

gain further information. Communication can help the sense maker become

aware of residue and change schemas (Sharma, 2006). Multidisciplinary

team meetings and second opinions about a patient can lead to reframing.

66

This section contains findings of the way in which clinicians think about risk.

The participants expressed a need for a quantitative measure when they think

about risk. Regardless of whether it was for an overall summary of risk or the risk

related to a single data point that contributes to the risk of a condition, the partici-

pants wanted the numbers. The probabilistic view of risk has the clinician antici-

pating an empirical measure.

One way that clinicians think about risk is absolute risk. This refers to the risk of

being affected by a condition over a period. For example, a 1 in 5, 20%, or 0.2

risk of developing of developing diabetes in the next 10 years.

Relative risk is the ratio of the probability of the patient who is part of a risk group

that is affected by a condition compared to the probability of somebody outside

the risk group being affected. This allows the clinician to put the risk in context

based on the two groups within the population.

The way in which clinicians think about risk is a complex process. The risk of de-

veloping a disease or condition is not the only one that affects the decision-mak-

ing process. Aspects of risk include shared risk factors of related diseases, tem-

poral risk, risk from intervention or medications, the risk of failure to act, risk from

co-morbidities, and economic risk etc. These factors, as well as others, are taken

into account when thinking about risk.

Discussing an exhaustive list of these factors is beyond the scope of this thesis,

so the risk versus benefit of intervention is used as an example. Lifestyle

changes, clinical tests, medication, surgery and other interventions all carry risks

in themselves. When the risk of an intervention gets closer to outweighing the

67

benefits of reducing the risk of developing a condition, clinicians become less

likely to go ahead with that intervention. In primary preventative care, such as in

the scenario used in this study, the tolerance to risk of developing a condition di-

minishes. For example, if the patient has a 1 in 5 chance of developing diabetes

II within 10 years and evidence has shown that an intervention reduces the rela-

tive risk by 50%, the absolute risk goes down from 1 in 5 to 1 in 10. However, a

hypothetical intervention might carry an increased risk of heart attack that out-

weighs the benefits of this risk reduction and therefore the clinician would not tol-

erate the risk of prescribing that intervention. This obviously changes in a situa-

tion where the benefits on intervention are higher than risks of doing nothing. One

participant expanded on this point:

P1 The person who is well, but sort of just in a prevention strategy, we are

not going to give them very major things that carry high risks. Whereas the

individual who I mentioned before whose heart disease condition is so

severe that he's going to be dead in the next three to four months, an

operation with a 25% mortality is balanced by a 100% mortality without it.

Individuals make sense in their own way; sensemaking is inherently an activity

that occurs in the mind of the individual. The way a person makes sense is influ-

enced by the history they have had in medicine, the patients they have seen, and

any other experiences affect a clinician’s understanding of a situation. Therefore,

it should not come as a surprise that participants talked about clinicians having

varying interpretations of risk from the same data:

P3 Different individuals and different clinicians would have different

interpretations of qualitative risks. A small risk or a tiny risk or a moderate

risk... and that's why clinical practice is not consistent between different

doctors, they all have slightly different interpretations of what's going on

and the risks of different things.

68

P6 What I often find is that different clinicians seem to remember risks a bit

differently. Sometimes one will say ‘oh, this [a risk factor] is more important’

and the other would say ‘this [the same risk factor] is less important’.

This inconsistency between clinicians is due in part to the nature of clinical prac-

tice, which is a subjective activity, despite the aspiration to remain objective when

assessing risk. Our findings echo Sutherland & Dawson (2002) who state, ‘in the

doctors’ worlds, new information is received and interpreted on the basis of past

experiences, cognitive structures, and social context’.

In some cases the threat of accountability, may create a lower tolerance toward

risk. It is possible for lawsuits to occur due to ‘greed, or simply because the pa-

tient or family members did not like the outcome or the doctors involved’ (Noland

& Carl, 2006, p.88). As a participant said, underestimating risk can result in nega-

tive consequences for the clinician, but overestimating may cause negative con-

sequences to the patients:

P1 I think for medical legal reasons, the world gets a little more defensive

and therefore we don't want to underestimate risk of a treatment procedure

and risk the patient or colleagues saying 'you only quoted 10%, it was

clearly 25%'. […] Maybe then there's a worry that we overestimate the risks

to be defensive. Much better to have been conservative than provide

someone with a higher risk, but if you start doing that [i.e. overestimating]

we might get more people saying no and declining important care.

Findings show that the participants deal with an excess of data from multiple of

sources. Extracting the most important information from seemingly irrelevant ma-

terial remains a challenge. The presentation of data, availability of information,

and trust placed in information all have severe implications on clinicians’ deci-

sion-making.

69

Data from interviews and think-aloud sessions revealed that trust is assigned to

data in different ways. The weighting attached to data in the decision-making pro-

cess surrounding risk is connected to how much they trust it.

Trust in self-reported data from patients tends to be lower than other types of

data because of the expectation of a higher margin of mistake in them: patients

may lie for their own reason, misreport, forget, or lack the appropriate knowledge.

P1 People lie and in consultations, not everyone tells the truth the whole

time, but then it's possible that you get an incorrect value based on an

incorrect piece of information […] You get a lot of people that say 'ah,

everything's fine' and then you scratch below the surface and it’s not. Trust

is a judgement, but anything that involves a person […] what I'm saying

there is a higher risk versus sending a blood sample to a laboratory and

getting a number back.

There is also a chance of misinterpretation during communication between pa-

tient and clinician. This happens when the mental model of the patient does not

match that of the clinician. Coping strategies to mitigate these issues include ask-

ing the patient in a number of different ways and approaching witnesses (e.g.

asking friends, family and co-workers) to confirm what the patient reports. In our

study, clinicians regarded expertise as the key factor in being able to know how

much weight to place on patient reported data.

The quality of the tools used to gather data is another factor clinician’s take into

account when assigning trust. For example, a participant was describing a patient

using their personal sphygmomanometer to measure their own blood pressure

(BP) from home. The sphygmomanometer is not quality assessed like the ones in

the hospital; therefore, trust in the patient’s equipment is lower. With the increase

of consumer eHealth devices, trust in the quality measurement tools will likely be-

come a prevailing issue. One possible way to mitigate the uncertainty created by

this lack of trust is by triangulating data from a number of sources.

70

Clinicians do not only assign trust to the tool itself, but also the way in which it is

used. The quality of the tool may be fine, but the measurement method may be

flawed:

P5 The issue that some patients take the BP at the wrist as opposed to

taking it on their arm. The risk reading is going to be a different reading to

the arm. The data that the patients sometimes present might not be as

accurate as the data that we have.

Other ways that clinicians assign trust are:

Volume of data – more usually means more confidence as trends over time

are revealed. For example, P5 said that BP readings from a patient might be

preferable if there are many data points making anomalies easier to track.

Age of data – depending on the type of data, older data can be less trusted

than newer data, for example blood chemistry tests from years ago have little

bearing on the patients current condition.

The findings suggest that clinicians say that they assign trust to data is partly due

to the amount of uncertainty surrounding a data point. Since the limitations are

known for clinically generated quantitative data, clinicians assign higher trust:

P1 [the highest trust is assigned to] hospital generated data, which is hard

data. Results from the laboratory, results from a scan, results from a

procedure where we will review that, we will know […] some of the

limitations of that.

Although there was a consensus that all data should be viewed with scepticism, a

number of participants spoke of quantifiable hard data as the basis of clinical

knowledge, and therefore overall more trustworthy. At the opposite spectrum, we

find ‘soft’ qualitative data. The tendency (due in part to medical training) is to as-

sign to numerical, clinical and quantified data (i.e. ‘objective’ data) a higher

amount of trust than qualitative, textual and contextual ones (i.e. patient reported

and ‘subjective’ data).

71

However, as observed through the study via interviews and in the think-aloud

session, the way that clinicians claim they assign trust to and use data to reach a

decision does not fully draw a parallel to the way in which this is applied in prac-

tice. Much of the time, the soft data is invaluable for gaining insight or recognising

further information needs. Clinicians showed varying levels of acceptance to the

idea of formalising this soft data as part of clinical practice, because they viewed

the data through an evidence-based medicine lens.

Some participants welcomed the idea of a tool that supported the process of

making the implicit, explicit. In contrast, others resisted the idea of including this

data in an interface. There seems to be a disconnect between the way that some

clinicians verbally and rationally assign importance to hard data versus soft data.

Some said the only data they need to see is ‘validated’ ‘hard’ data, but when

asked to think of which data they use in practice; it included ‘soft’ data:

P5 In terms of the patient information, knowing that she is widowed is fairly

important. Her husband might have just died last week, and if her husband

died last week you don't really want to necessarily be talking to her about

her risk of melanoma, she won't care…

We are familiar with the problem of noisy, multivariate, homogeneous data gener-

ated from dissimilar sources. It is good to have all information available, but pre-

senting it in the wrong way can easily increase cognitive load. As we know, risk is

not only the sum total of all of the available data, some data points will trump oth-

ers in terms of urgency or severity.

Things get even more complex when we start to think about types of data that

most clinicians are not used to looking at. We found that the majority of clinicians

have limited understanding of genetic risk except for monogenic, or single abnor-

mal gene, diseases (i.e. Mendelian disorders), unless they were specialists in the

field. Participants expressed a desire for supporting explanation and summarisa-

tion of genetic risk, although participants who were not genetics experts lacked

the confidence to interpret genetic information.

72

Rosas-Blum, Shirsat, & Leiner (2007) observe that there is a lack of essential

knowledge and comprehension of genetics; this was reflected in our findings:

P4 The more you understand these things the more you accept them. They

were talking about doing genome sequencing on the radio this morning, I

don't want to know to be honest. I'm a head in the sand person. Unless of

course it's something very very preventable.

Genetic risk is still in the early stages and also susceptible to the same problems

of trust, in terms of the strength of the study and supporting evidences, rele-

vance, and of the patient being outside the population of the study being used as

reference:

P2 Most data about genetics [specifically referring to congenital heart

defects] are very basic research data on genes and gene pathways, which

may or may not have anything to do with the patient you're looking at, or

they're based on large population averages, so they're a gross estimate of

risk.

In order to move from assessing risk to making a decision, a clinician must have

enough information and enough confidence in it. This means clinicians work in

conditions of great uncertainty and within a probabilistic paradigm (e.g. establish-

ing relative plausibility) (Hammersley, 1995).

Encountered information that brings the possibility of a patient being at risk to

their attention usually creates a latent need for more information and thus, more

information seeking. Sometimes the clinician is not aware of available information

and so the need does not arise. If it does arise later, it can be a source of great

frustration.

73

One participant said that it is common to do many investigations and then find out

some unrelated contextual data:

P6 I would say the biggest problem with information not being useful is

when information is left out […] Often I find when I'm reviewing notes, what

is missing, is the most frustrating “Oh, why did they not ask about this”,

“Why did they not check this?” Purely because people forget when they are

in a rush.

Frames can be elaborated when new data that was missing before is presented,

leading to new inferences and relationships being made. In this example, the

small biopsy and uncertainty of diagnosis stimulates thinking about missing data.

We explain the following process in the context of Pirolli & Card's (2005) sense-

making model; information, scheme, insight, product:

P6 …with this very tiny biopsy that was taken […] we could not make a

diagnosis so we asked the surgeon whether he could go back and take a

larger chunk of tissue […] when that tissue came through we took a frozen

section and within five minutes we had a definitive diagnosis. […] While the

diagnosis of infantile fibrous sarcoma is by and large very easy it can still be

impossible if the tissue is not representative or insufficient, however you

don't know what you're diagnosing in advance so you have that source of

uncertainty.

The clinician did not have enough data to make sense of the situation, and there-

fore asked the surgeon for a bigger sample because of an information need. The

new data was re-represented in the appropriate schema, through manipulation of

the representations of about what the patient’s tumour actually was, insight was

gained. Following this newfound understanding, a product was created in the

form of a definitive diagnosis.

Although this example seems very straight forward, it is impossible to know if

substantial data is missing unless the relevant frame is elicited through the appro-

priate anchors. There is a constant strive to ‘completeness’ and a desire to be

aware of what relevant information is not present, and to seek it out.

74

Tools can assist in flagging missing information used in validated scales and

other models. However, it is not often known what other data are needed as clini-

cians use so many contextual factors as anchors in so many different combina-

tions that it is probably unrealistic to compile an exhaustive list in any externalisa-

tion. It seems that there is a misapprehension (or at least an unachievable desire)

that data about a patient can be all encompassing. Usually, when a gap in

knowledge is identified, a way to fill that gap needs to be present. Filling a gap in

knowledge will then often lead to further information needs. The sensemaking

process is cyclical by nature.

Visualisation was used in the study to display, interact with and explore infor-

mation in a graphical manner with the hope that it would aid the participants’ un-

derstanding of risk. By mapping data to a visual form, the tool attempted to sup-

port visual sensemaking. By making sense of externalisations, internal models of

the domain are formed (Ware, 2013), preliminary evidence shows that the tool

supports this.

The tool presented patient information (information that is usually gathered

and stored in the clinicians’ mental model of the patient) whilst steering clini-

cians’ attention to risk factors (risk factors for a particular condition with an ev-

idence base as to why it is a risk factor) that were relevant to their assess-

ment of risk through an appropriate information structure. The way that partic-

ipants interacted with the tool showed that presenting both validated/clinical

scales/data and contextual information as part of the externalisation was rele-

vant to their needs.

It also provided the relevant evidence to support why a data point had risk at-

tributed to it, along with access to guidelines and standards. This allowed the

clinician to gauge the data quality.

The data that was and was not taken into account by quantitative risk scales

(risk calculator) was plainly highlighted, therefore showing how comprehen-

sive that empirical measure was.

75

Figure 24 Seeking a frame.

The condition identified by the

tool and the available patient

data act as anchors.

The goal of the clinician, when assessing risk, is to make sense of the information

they have about an individual patient and then decide on the next steps to take in

order to modify the patient’s life. Here we will present findings about the tool de-

veloped for this thesis. This section will cover the things that worked well to sup-

port making sense of risk, and cover some of the limitations and shortcomings of

the tool.

The overview page was found to provide a well-

structured representation of information to help

the clinician in thinking about risk.

P4 Linear clearly presented demographic

versus risk factor information, so that's fairly

clear.

P5 I like the fact that its patient information

and risk factors, it's absolutely clear.

Presenting all of the patient information in the

same interface helped to provide the contextual

factors that clinicians use to think about risk in

their work, but are not taken into account by tra-

ditional tools. Provide overview (Yi et al., 2008)

helped clinicians see what parts were most rele-

vant for further inspection by providing a ‘con-

cept-oriented and task-specific’ summary that

may help clinicians ‘recognize new problems, im-

plement preventive care and formulate care

plans’ (Feblowitz et al., 2011).

Providing the patient information (and risk factors) along with the risk condition

served as anchors to stimulate thinking about risk, thus connecting the data to a

frame about patient risk of developing a condition (Figure 24).

76

Figure 25 Data

categories.

The patient information was organised into categories that followed the pattern

that clinicians are used to seeing it, making it easier for users in the sensemaking

loop to search and filter information. This was part of supporting the clinical work-

flow, generally moving from broad to specific (Figure 25). The designs for the vis-

ualisation attempted to decrease the gap between users mental model – match

mental model (Yi et al., 2008) – and data in order to decrease cognitive load.

Essential to the task, risk was represented by the visualisation in a frame on the

right hand side with most significant information that clinicians use when thinking

about risk given priority. Participants expressed that the most severe risk factors

are the most essential to take into account because they will lead to insight about

the patient’s potential future condition:

P1 If the risk is low; you wouldn't be delving into why it's low. Whereas if

it's elevated then you'd start to look into what you can do to modify that.

Of these high risk factors, the changeable ones were the most important, since

they are the ones that would allow for intervention:

P1 You could argue that, the most important ones are the changeable ones

because there's something you can do about them.

Non-changeable risk factors are also important for educating patients about their

risk, even if it is not possible to mitigate that risk through intervention:

P4 One of the worst things is in fact if your chance of 1 in 10 is all to do

with your male, ethnicity and your genes. Then I suppose what would you

do? But, it's worth knowing that

77

The we found that the tool supported this information need.

Figure 26 shows elaborating a frame by satisfying the queries

that arose when questioning the frame about a patient being at

risk of developing a condition. This is how the tool design tack-

led the need:

Risk severity – The ordinal scale of high/medium/low risk was

used to represent severity, clinicians deemed high risk factors

most important and low risk factors least important. A colour

mapping of differing saturation was used to let the clinician

know which risk factors were more severe.

Changeable risk factors – important since they were also the

factors that were directly actionable through intervention. Vis-

ual differentiation was achieved by grouping of the modifiable

factors.

Non-changeable risk factors – secondary in importance as

they are useful for educating the patient about their risk, even

if it is not possible to change them. These were grouped in the

same way at changeable factors, but contrasted from the

changeable through placement and a separating line.

Most western audiences natively read left to right and top to bottom. With hind-

sight, the structure of information may have been better if changeable factors had

been positioned above non-changeable factors, and within the two respective

sections, high/medium/low risk factors ordered from top to bottom, thus, giving

visual priority to the most important information to the western user.

Figure 26 Elaborating

a frame.

78

Providing a risk summary was found to be a need for clinicians. A risk number is

useful for giving an empirical measure of the information. Some participants

noted that some clinicians rely on calculators a lot:

P1 There will be some clinicians who go 'I don't care about that [points to

patient information], I care about that [points to risk calculator score]'.

As mentioned earlier, calculators, evidence and even guidelines were developed

with a certain population, not all patients fit within these populations:

P4 I suppose sometimes it would depend on what population you're dealing

with. Sometimes ranges depend on the population that you're dealing with.

If we were to take a range of haemoglobin in blood, our range in western

society is a particular range. But you translate that range to an Indian

society, it would be completely different. Because we live in a multicultural

society sometimes, you can't necessarily believe the ranges that are

applicable to this population. Sometimes you have to be sensitive to that.

We look at many western calculators of risk, which is not necessarily always

appropriate to minorities.

Here we highlight an example of how data-frame theory

forces a change in the clinicans understanding of the world.

In light of the new information about the patient not fitting into

the risk population that the evidence is based on,

the clinian finds that the situation is not what

was anticipated in the original frame. Once

the clinian realises that the patient

does have the risk that the evidence

would suggest, the clinician’s world

view must be adjusted; this is done

through reframing. In Figure 27 we

can see that when anomalies or

Figure 27 Reframing

cycle in progress.

79

inconsistencies occur, the competing data forces a change in understanding.

Before the understanding was that the patient was at risk, due to the evidence

presented in the original anchors. However, the frame no longer fits the new data

no longer fits; the patient is outside the population for the evidence base and

therefore, reframing occurs.

Participants said that using other patient information they are able to explain and

communicate to the patient why the risk score given by the calculator may not

apply to them:

P5 You're just giving them an idea of their risk... you adapt your risk

calculator. You have the conversations which says 'this might be the case'...

you put in a caveat. It would be good to have a bit of narrative around that,

in the risk calculator say. This is a risk calculator, it's from this particular

population and the narrative might be that if you’re thinking about a

different population, you might just need to just take that into account. You

might not be able to completely calculate that, but it's a narrative, it's

qualitative.

In addition, participants noted that risk calculators vary in quality:

P4 There are lots of different risk calculators, some of them are more

evidence based than others.

The tool successfully showed where the risk calculator algorithm was lacking by

displaying factors that research studies have identified as risk factors but were

not taken into account by risk calculators thus, highlighting the limitations of the

summary and laying emphasis on the need for further clinical judgement in the

areas not covered by the risk calculator. When running the calculator the user

was prompted to fill in information. This helped to make participants aware of

missing data.

Zooming into the individual data point view allowed the participants to explore

why something contributes to risk. This operation had the intent of enabling the

80

user to abstract/elaborate one aspect of the patient information to show more de-

tail by interacting with the interface. The user was also able to explore the other

patient information from this view, supporting the information foraging loop. Infor-

mation for individual data points was embedded within visualisations, these

aimed to impart knowledge about the distribution of risk for that factor within a

population, as well as where the patient sits within the population.

P4 I like the fact that you've got, two things, you've got the knowledge bit,

the overall prevalence. Then you've identified which group the patient is in.

Think-aloud data revealed that the inclusion of supporting sources within the tool

was perceived as useful to all participants. However, some participants men-

tioned that certain clinicians might like to use guidelines and find viewing evi-

dence is too much work due to the sheer volume of available evidence and time it

takes to go through it. One participant mentioned that it is not common to view

evidence during a consultation. Whereas others would like to go through evi-

dence to aid their decision-making process. Further improvement of evidence

source and clinical trial research visualisation for sensemaking could aid clini-

cians in searching for evidence to support risk assessment. An approach similar

to that of Faisal, Cairns, & Blandford (2006), but for the healthcare research evi-

dence domain may be one possible way to approach this problem.

In summary, the tool supported the clinicians in making sense of risk. All availa-

ble patient information was make available, without drawing the user attention

from the task. Information was presented in a structured manner by grouping re-

lated items and contrasting dissimilar entities such as changeable/non-changea-

ble and data contributing to risk/data not contributing to risk. Segmentation of

lists, blank space and lines were used to avoid overloading the screen with con-

tent. The tool was integrated into the clinical workflow by providing an appropriate

order of screens, information and alerts. Missing information was highlighted by

prompting the clinician to enter information that was needed for the risk calculator

81

at a suitable moment. However there was also issues with the tool, such as use

of colour and consistency of terminology and wording, most of which could most

likely be fixed through adding further iterations to the design process. Further

consultation on the correct clinical terminology throughout the tool would have

avoided the need for the clinicians to switch regimes of language during use of

the tool.

82

83

6 LIMITATIONS In this chapter, we first cover some of the design limitations of the tool. Then

weaknesses of the study as a whole as discussed, and then we propose ways in

which the tool could be improved to further meet the needs of clinicians.

There were a number of cases where the tool failed to support the clinicians’ task

of assessing risk. There were also instances where the system model failed to

match that of the user. This section will look at some of the shortcomings of the

tool.

Results show that the option of manually changing a risk of a data point in the

tool was a controversial point. Some participants were opposed to the implica-

tions of defying a validated data point, even if it was flagged for review in the sys-

tem. It was preferable for most participants to change the value of a data point ra-

ther than the risk score attached to it:

P3 If the data's correct you should use them as they are […] You're not

going to change the classification of the weight, you want to change the

weight to make it correct. I'm not sure how useful this is unless it's very

inherent as part of what the actual data item is.

This feature is obviously not useful for things such as weight, but maybe there is

potential for use with more contextual such as stress at work:

P3 Stress at work could be a risk factor for other things, so... It's a question

of 'how is that best represented?' You could have 'stress at work' and assess

that... the actual job doesn't matter […] then again job is demographic

information. It would be relevant for lots of other kind of clinical uses apart

from risk assessment. You can't only go by the job description, because two

jobs could have the same description but could have a very different work

84

environment. I'm not sure about this 'changing a risk classification... This is

a case where it probably does make sense, but this kind of thing wouldn't

work for other things. Like medical conditions.

Some participants thought it was best to ‘override the system’ not through a tool,

but by talking with the patient directly and having a conversation about why the

risk is different for them and the risk the evidence says that they have does not

apply to them. However, some participants felt that it was essential to allow the

clinician to flag items for review, stating that feedback is essential, both to the de-

velopers/maintainers of the tool, as well as to fellow clinicians for second

opinions:

P4 I think the more you can collect feedback from people... Feedback about

not understanding what the system is doing […] Just ongoing feedback

about whether it's producing [useful results]. Whether it is actually doing

something useful basically.

Using colour to represent risk was found to be rather haphazard, some partici-

pants understood the use of colour, and some did not. A few participants did not

understand that the three gradations of colour represented low, medium and high

risk. The fine distinctions in increasing saturation levels was not directly mapped

to increased risk in all participants’ mental model. Although this method was more

effective than using size (Stage 2, Visualisation 2), it was less effective than us-

ing placement (Stage 2, Visualisation 1) as a signifier for low, medium and high

risk. Using a key to make the differences in saturation obvious may be an option,

but further visual design for effective use of colour (Silva et al., 2011) is required

for an optimal solution.

Using the red and green in the same interface to ascribe meaning to risk severity

was a problem. One participant was colour-blind, resulting in the risk severity fail-

ing to hold meaning for them. However, most participants understood red as be-

ing risky and green as being safe.

85

Showing risk in colours other than red and green may not have been semantically

resonant (Lin, Fortuna, Kulkarni, Stone, & Heer, 2013) to clinicians looking at risk.

Some participants even called for the use of those colours when exploring the in-

terface:

P8 Make the others [the reduced risk factors in the risk calculator] green

and the risk factors red and that would be more obvious.

Clinicians have a common way of talking about risk. Being educated in the same

manner has led to a shared language; the language of science is specific. Partici-

pants mentioned a similar phenomenon of regime switching where patients have

different terms for the same concept. Similarly, the differences in nomenclature

were the cause behind some of the misunderstandings in the interface that may

have been avoided with better choice of language:

P4 Changeable and non-changeable, those are static and dynamic really

aren't they? Which are terms I'm more familiar with.

A limitation of the recruitment method used in the study is that the clinicians

who were recruited through industry connections were inherently interested in

technology in healthcare. This may have led to a bias in prevailing attitudes

toward the tool.

Although we found that the general principles of assessing risk using complex

health data is relevant to both primary and secondary care, the tool used in this

thesis focused on a primary preventative setting. However, much of the way

healthcare is delivered today is reactive rather than preventative. This may have

influenced the way in which participants thought about the tool. That said, we

believe the tool used is both relevant and timely since primary preventative

healthcare is high on the agenda, both in the United Kingdom (Department

of Health, 2009) and the United States of America (Francis Collins, 2014).

86

Since the personas were created for the purposes of this study, the data set is

not based on real patient data. Although a doctor and a nurse reviewed the per-

sonas for validity, the use of fabricated data may have influenced the way that

participants thought about and understood the tool. Testing of the tool was limited

to using scenarios with personas and a prototype tool. Testing using a real pa-

tient in a clinical setting would have been preferable put was not feasible due to

pragmatic reasons such as temporal constraints and ethical concerns. These lim-

itations in the experimental design have implications on the transferability of find-

ings to real clinical practice. Clinical trials would be needed to validate the useful-

ness of this tool in an environment with actual patients and working clinicians.

Ideally the tool would not only be able to support clinicians in deciding if a patient

is at risk, but also to help determine which prevention measures would provide

the greatest reduction in risk. Integration of the tool into the entire clinical work-

flow would have been ideal, something that was beyond the scope of this study.

Integrating existing guidelines such as those proposed by Horsky et al. (2012) to

fit existing standards would be useful for this process.

Workplace factors also played a part in the perceived utility of the tool; partici-

pants mentioned that they do not have access to tablets or computers during

their consulting work. The ability to output, or what Shneiderman (1996) calls ex-

tract, the relevant sub-collection of data would be necessary to integrate the tool

into a real work environment. The data showed that, this tool would only be useful

if it pulled data from existing sources and if clinicians could input data back to the

Electronic Heath Record. Although these issues are important for the successful

integration of tools into the healthcare system (such as the one proposed in this

study), there are policy, data ownership, security, and authorisation considera-

tions that need to be taken into account. These topics are multi-layered and com-

plex and therefore far beyond the scope of this study. The study design did not

take into account the important role that society and organisations in healthcare

play when it comes to clinicians dealing with complex data and thinking

about risk.

87

7 CONCLUSIONS By looking through the sensemaking lens, we have provided an account of the

way in which clinicians make sense of complex data when assessing risk in a pa-

tient. We have shown that assessing risk is not merely about knowing the rules,

but about deciding which rule is most relevant, or when the rules do not apply.

When a representative sample of a population is available, clinicians are able to

understand the implications of risk for the abstract patient. However often the sit-

uation is more nuanced, the average for the population might not fully apply to

the patient in front of them. By using an array of information – derived from both,

what clinicians consider hard and soft data – they consciously or subconsciously

determine which patient characteristics influence their decisions.

It was found that some clinicians tend to downplay or be unaware of factoring

‘soft’ or non-clinical data into their judgments. The processes behind these judg-

ments are tough to express explicitly but are important for personalised, patient-

centred work. Through this process clinicians are able to interpret when general-

ising the abstract to the individual patient (Smith & Egger, 1998), what is and is

not appropriate.

This thesis presented the design process taken to develop a visualisation to sup-

port the clinicians’ needs was presented. We have shown preliminary evidence

for a tool that supports the sensemaking process, by reducing the gap between

the representation of data and the clinicians’ mental model. However, further in-

vestigation is required to assess the utility and safety of such a tool in actual

clinical settings.

88

89

8 BIBLIOGRAPHY Anders, K., & Simon, H. A. (1980). Verbal reports as data. Psychological Review,

87(3), 215–251. doi:10.1037/0033-295X.87.3.215

Bertin, J. (2010). Semiology of graphics: diagrams, networks, maps (1st ed.).

Redlands, Calif: ESRI Press : Distributed by Ingram Publisher Services.

Boren, T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice.

Professional Communication, IEEE Transactions on, 43(3), 261–278.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative

Research in Psychology, 3(2), 77–101. doi:10.1191/1478088706qp063oa

Bui, A., & Hsu, W. (2010). Medical Data Visualization: Toward Integrated Clinical

Workstations. In Medical Imaging Informatics (pp. 139–193). New York:

Springer US.

Card, S. K., Mackinlay, J. D., & Shneiderman, B. (1999). Readings in information

visualization: using vision to think. San Francisco, Calif: Morgan Kauf-

mann Publishers.

Department of Health. (2009). NHS 2010-2015: from good to great: preventative,

people-centred, productive. London: Stationery Office.

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: verbal reports as data

(Rev. ed.). Cambridge, Mass: MIT Press.

Faiola, A., & Hillier, S. (2006). Multivariate relational visualization of complex clini-

cal datasets in a critical care setting: A data visualization interactive proto-

type (pp. 460–468). IEEE. Retrieved from http://ieeex-

plore.ieee.org/xpls/abs_all.jsp?arnumber=1648300

90

Faisal, S., Blandford, A., & Potts, H. W. (2013). Making sense of personal health

information: Challenges for information visualization. Health Informatics

Journal, 19(3), 198–217. doi:10.1177/1460458212465213

Faisal, S., Cairns, P., & Blandford, A. (2006). Developing User Requirements for

Visualizations of Literature Knowledge Domains. In Information Visualiza-

tion, 2006. IV 2006. Tenth International Conference on (pp. 264–269).

IEEE. Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?ar-

number=1648271

Faisal, S., Cairns, P., & Blandford, A. (2007). Challenges of evaluating the infor-

mation visualisation experience. In Proceedings of the 21st British HCI

Group Annual Conference on People and Computers: HCI... but not as

we know it-Volume 2 (pp. 167–170). British Computer Society. Retrieved

from http://dl.acm.org/citation.cfm?id=1531451

Feblowitz, J. C., Wright, A., Singh, H., Samal, L., & Sittig, D. F. (2011). Summari-

zation of clinical information: A conceptual model. Journal of Biomedical

Informatics, 44(4), 688–699. doi:10.1016/j.jbi.2011.03.008

Francis Collins. Fiscal Year 2015 Budget Request, § Senate Subcommittee on

Labor – HHS – Education Appropriations (2014). Maryland. Retrieved

from http://www.nih.gov/about/director/budgetrequest/fy2015testi-

mony.htm

Fry, B. J. (2004). Computational information design. Massachusetts Institute of

Technology. Retrieved from http://dspace.mit.edu/handle/1721.1/26913

Gillham, B. (2005). Research interviewing: the range of techniques. Maidenhead ;

New York: Open University Press.

91

Gronseth, G. S., Woodroffe, L. M., & Getchius, T. S. (2011). Clinical Practice

Guideline Process Manual. St. Paul, MN: American Academy of Neurol-

ogy. Retrieved from http://curebraindisease.net/globals/axon/as-

sets/9023.pdf

Guion, L. A., Diehl, D. C., & McDonald, D. (2011). Triangulation: Establishing the

validity of qualitative studies. Retrieved from http://edis.ifas.ufl.edu/fy394

Hammersley, M. (1995). Ethnography: principles in practice (2nd ed.). London ;

New York: Routledge.

Harle, C. A., Neill, D. B., & Padman, R. (2008). An information visualization ap-

proach to classification and assessment of diabetes risk in primary care.

In 3rd INFORMS Workshop on Data Mining and Health Informatics (DM-

HI 2008) (pp. 1–6).

Harle, C. A., Neill, D. B., & Padman, R. (2012). Information Visualization for

Chronic Disease Risk Assessment. Intelligent Systems, IEEE, 27(6), 81–

85.

Hartson, H. R., & Pyla, P. S. (2012). The UX Book: process and guidelines for

ensuring a quality user experience. Amsterdam ; Boston: Elsevier.

Henry, S. G. (2010). Polanyi’s tacit knowing and the relevance of epistemology to

clinical medicine: Tacit knowing and clinical medicine. Journal of Evalua-

tion in Clinical Practice, 16(2), 292–297. doi:10.1111/j.1365-

2753.2010.01387.x

Horsky, J., Schiff, G. D., Johnston, D., Mercincavage, L., Bell, D., & Middleton, B.

(2012). Interface design principles for usable decision support: A targeted

92

review of best practices for clinical prescribing interventions. Journal of Bi-

omedical Informatics, 45(6), 1202–1216. doi:10.1016/j.jbi.2012.09.002

Huang, W., & Eades, P. (2013). Handbook of human centric visualization. Re-

trieved from http://dx.doi.org/10.1007/978-1-4614-7485-2

Johnson-Laird, P. N. (1983). Mental models: towards a cognitive science of lan-

guage, inference and consciousness. Cambridge [Cambridgeshire] ; New

York: Cambridge University Press.

Jupp, V. (2006). The Sage dictionary of social research methods. London ; Thou-

sand Oaks, Calif: SAGE Publications.

Kirsh, D. (2010). Thinking with external representations. AI & SOCIETY, 25(4),

441–454. doi:10.1007/s00146-010-0272-8

Klein, G., Moon, B. M., & Hoffman, R. R. (2006). Making Sense of Sensemaking

1: Alternative Perspectives. IEEE Intelligent Systems, 21(4), 70–73.

Klein, G., Phillips, J. K., Rall, E. L., & Peluso, D. A. (2007). A data-frame theory of

sensemaking. In R. R. Hoffman (Ed.), Expertise out of context: proceed-

ings of the sixth International Conference on Naturalistic Decision Making

(pp. 113–155). New York: Lawrence Erlbaum Associates.

Krygier, J., & Wood, D. (2005). Making maps: a visual guide to map design for

GIS. New York: Guilford Press.

Lin, S., Fortuna, J., Kulkarni, C., Stone, M., & Heer, J. (2013). Selecting Semanti-

cally-Resonant Colors for Data Visualization (Vol. 32, pp. 401–410). Wiley

Online Library. Retrieved from http://onlineli-

brary.wiley.com/doi/10.1111/cgf.12127/full

93

MacDonald, L. W. (1999). Using color effectively in computer graphics. Computer

Graphics and Applications, IEEE, 19(4), 20–35.

Mazza, R. (2004). Introduction to Information Visualisation. Faculty of Communi-

cation Sciences, University of Lugano. Retrieved from

http://www.dti.supsi.ch/~mazza/infovis_introduction.pdf

Mynatt, E. D. (2011). IT in healthcare: a body of work. Interactions, 18(3), 22–25.

National Health Service. (2013, August 4). Type 2 diabetes self-assessment. Re-

trieved from http://www.nhs.uk/Tools/Pages/Diabetes.aspx

Nielsen, J. (1993). Iterative user-interface design. Computer, 26(11), 32–41.

Nielsen, J. (1994). Guerrilla HCI: Using discount usability engineering to pene-

trate the intimidation barrier. Cost-Justifying Usability, 245–272.

Nielsen, J., & Faber, J. M. (1996). Improving system usability through parallel de-

sign. Computer, 29(2), 29–35.

Noland, C., & Carl, W. J. (2006). “It’s Not Our Ass”: Medical Resident Sense-

Making Regarding Lawsuits. Health Communication, 20(1), 81–89.

doi:10.1207/s15327027hc2001_8

Pirolli, P., & Card, S. (2005). The sensemaking process and leverage points for

analyst technology as identified through cognitive task analysis. In Pro-

ceedings of International Conference on Intelligence Analysis (Vol. 5, pp.

2–4). Mitre McLean, VA. Retrieved from http://vadl.cc.gatech.edu/docu-

ments/2__card-sensemaking.pdf

Polanyi, M. (1983). The tacit dimension. Gloucester, Mass.: Peter Smith.

Riessman, C. K. (1993). Narrative analysis. Newbury Park, CA: Sage Publica-

tions.

94

Rind, A., Wang, T., Aigner, W., Miksch, S., Wongsuphasawat, K., Plaisant, C., &

Shneiderman, B. (2013). Interactive Information Visualization to Explore

and Query Electronic Health Records. Foundations and Trends® in Hu-

man–Computer Interaction, 5(3), 207–298. doi:10.1561/1100000039

Rosas-Blum, E., Shirsat, P., & Leiner, M. (2007). Communicating genetic infor-

mation: a difficult challenge for future pediatricians. BMC Medical Educa-

tion, 7(1), 17. doi:10.1186/1472-6920-7-17

Russell, D. M., Stefik, M. J., Pirolli, P., & Card, S. K. (1993). The cost structure of

sensemaking. In Proceedings of the INTERACT’93 and CHI’93 confer-

ence on Human factors in computing systems (pp. 269–276). ACM. Re-

trieved from http://dl.acm.org/citation.cfm?id=169209

Saraiya, P., North, C., Lam, V., & Duca, K. A. (2006). An insight-based longitudi-

nal study of visual analytics. Visualization and Computer Graphics, IEEE

Transactions on, 12(6), 1511–1522.

Schon, D. A. (1983). The reflective practitioner: how professionals think in action.

New York: Basic Books.

Sharma, N. (2006). Sensemaking: Bringing theories and tools together. Proceed-

ings of the American Society for Information Science and Technology,

43(1), 1–8.

Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for in-

formation visualizations. In Visual Languages, 1996. Proceedings., IEEE

Symposium on (pp. 336–343). IEEE. Retrieved from http://ieeex-

plore.ieee.org/xpls/abs_all.jsp?arnumber=545307

95

Shneiderman, B., Plaisant, C., & Hesse, B. (2013). Improving health and

healthcare with interactive visualization methods. IEEE Computer Special

Issue on Challenges in Information Visualization, 58–66.

Sieck, W. R., Klein, G., Peluso, D. A., Smith, J. L., Harris-Thompson, D., & Gade,

P. A. (2007). FOCUS: A model of sensemaking. DTIC Document. Re-

trieved from http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPre-

fix=html&identifier=ADA469770

Silva, S., Sousa Santos, B., & Madeira, J. (2011). Using color in visualization: A

survey. Computers & Graphics, 35(2), 320–333.

doi:10.1016/j.cag.2010.11.015

Smith, G. D., & Egger, M. (1998). Incommunicable knowledge? Interpreting and

applying the results of clinical trials and meta-analyses. Journal of Clinical

Epidemiology, 51(4), 289–295.

Spence, R. (2007). Information visualization: design for interaction (2nd ed.). New

York: Addison Wesley.

Spence, R. (2014). Information visualization: an introduction. [Place of publication

not identified]: Springer.

Straus, S. E. (2011). Evidence-based medicine: how to practice and teach it. Ed-

inburgh: Elsevier Churchill Livingstone.

Sutherland, K., & Dawson, S. (2002). Making sense in practice: Doctors at work.

International Studies of Management & Organization, 51–69.

Treisman, A. (1998). Feature binding, attention and object perception. Philosophi-

cal Transactions of the Royal Society of London. Series B: Biological Sci-

ences, 353(1373), 1295–1306.

96

Tufte, E. R. (1995). Envisioning information. Cheshire, Conn.: Graphics Press.

Wagemans, J., Elder, J. H., Kubovy, M., Palmer, S. E., Peterson, M. A., Singh,

M., & von der Heydt, R. (2012). A century of Gestalt psychology in visual

perception: I. Perceptual grouping and figure–ground organization. Psy-

chological Bulletin, 138(6), 1172–1217. doi:10.1037/a0029333

Ware, C. (2013). Information visualization: perception for design (Third edition.).

Waltham, MA: Morgan Kaufmann.

Weick, K. E. (1996). Sensemaking in Organizations. The Academy of Manage-

ment Review, 21(4), 1226. doi:10.2307/259169

Yi, J. S., ah Kang, Y., Stasko, J. T., & Jacko, J. A. (2007). Toward a deeper un-

derstanding of the role of interaction in information visualization. Visuali-

zation and Computer Graphics, IEEE Transactions on, 13(6), 1224–1231.

Yi, J. S., Kang, Y., Stasko, J. T., & Jacko, J. A. (2008). Understanding and char-

acterizing insights: how do people gain insights using information visuali-

zation? (p. 4). ACM. Retrieved from http://dl.acm.org/cita-

tion.cfm?id=1377971

97

9 APPENDICES

98

99

100

101

102

103

104

105

Requirements Statement Priority

1. Actionability

Investigations

Allow the user to know if there is any potential high risk factor information that is miss-ing [information>data gathering]

Rationale: It’s usually part of the consultation to gather information [P6V2C]. For high risk factors, do I need to gather data or do tests [P2V2B]? Users want to understand if info is missing, especially that which is known to affect risk in that condition

Design recommendation: If there is information missing for a specific risk calculator, make it apparent to the user through the interface that the information should be gath-ered. For example, if a user is trying to use a particular risk calculator and there is missing data, it could be represented by empty boxes in the patient info section. In the actual prototype the ‘order investigations’ section can be a placeholder letting the par-ticipant know that there would be more interface to allow for investigations (a full mock-up of this part is beyond the scope of this project). We can assume that this part of the interface would allow adding new data points during the consultation [P6V1C]

2. Actionability

Diagnosis

Assuming the user has enough information to make a diagnosis he wants to know what to do about risk [P2V2B]. Allow the user to move toward a diagnosis [infor-mation>actionability]

Rationale: Nothing tells you about treatments to change the course [P5V2C]. It would be nice to know what to do to change, such as surveillance or intervention [P2V1A]. What can I do about this for the patient [P6V1C]

Design recommendation: Similarly to Requirement Statement #1, have a part of the interface that leads to diagnosis with a placeholder for next steps in the traditional di-agnosis work a clinician does

3. Text

Understanding purpose of visualisation

Explicitly let the user know what the visualisation is for [user & system model mis-match]

Rationale: Users thought the visualisation was showing co-morbidities of diabetes [P3V1A], thought the patient had the condition already [P1V2A], did not know if the data was from a patient or just generic/textbook data showing all risk for a particular disease

Design recommendation: The text should be changed to support user understanding. ‘Risk of developing X’

4. Text

Patient information

The user understands data that does not contribute to the risk factor as neutral or not contributing to risk [information>categories]

Rationale: All data is considered ‘patient information’, even the data that infers risk [P2V1A]

Design Recommendation: Refer to ‘patient information’ that does not contribute to risk as ‘neutral patient info’ or ‘patient info not contributing to risk’

106

5. User input

Adding data to individual data points

There should be the possibility for the user to add contextual info & notes to data points [information>supporting interpretation]

Rationale: Everyone makes sense of data in their own way. By allowing the user to populate data points with contextual information & notes [P1V1A], the thought process behind decisions can be documented and referred to later. Patients are individual and therefore some will be more at risk than others [P1V1&2] when it comes to the risk vs. benefit of side effects from intervention (or doing nothing) [P5V2C]

Design recommendation: Allow user to input notes, that are then added to a data point

6. User input

Editing risk category

If there is no/low evidence for a data point, allow the user to assign a risk category. But, do not allow the strongly evidence based data points to be moved. Track all changes and show if an item has been moved through the interface [information/sup-porting interpretation & “obj” vs. “subj” data]

Rationale: ‘Solid’ data has its limitations [P1V1A]. For example, occupational history is useful [P5C1B], but interpretation is subjective. In clinical work evidence & subjective opinions are mixed, this tool gives an objective view of both that can be reviewed [P6V1&2]

Design recommendation: The data points that aren’t used in validated scales can be re-assigned to another risk category, but these changes must be tracked for later re-view

7. Data Categories

Category order

Present categories in the order clinicians are used to looking at it in their workflow [in-formation>categories]

Rationale: Although the categories cover relevant things [P1V1A], they are not in the order that clinicians would be used to looking at them [P5V2C]

Design recommendation: Order by patient history, social history, family history, occu-pational history, medication, examination data, genetic data (genetic is not so con-crete about where it should be because only specialists are likely to understand)

8. Data Categories

Genomic information

The user must be able to understand genetic information [information>categories]

Rationale: Genetic information is difficult to interpret [P5V1B]. As it means nothing to most people [P1V2A]

Design recommendation: Summarise genetic information on the information page, but allow zooming into details for the specialist

9. Individual data point view

Relative risk

Let the user know how a single data point correlates to risk [P6V2C] [risk type>relative risk]

Rationale: Relative risk is not known for all of the data points, but it would be helpful is it was available for those points that are known [P2V2B]

Design recommendation: Display relevant patient data on classic graphs & scales within the individual data point view. Let the user know the placement on high/med/low in the overview is based on the relative risk of that specific measure-ment in order to ground the perspective [P2V1A]

107

10. Individual data point view

Contextual data

The user must have access to:

Temporal factors if they are relevant (how long has he smoked, how long has he lived here, how long did he live at his previous residence etc.)

Metadata if it is relevant (where was this is the bone density at the hip, what is the bone density at the spine etc. for DEXA scan)

[information>contextual data]

Rationale: If a person has only smoked for 5 a day for 2 weeks vs. 40 a day for 20 years and the 2 week patient still smokes & the 20 year stopped last week, it’s point-less to call him a non-smoker when assessing risk [P6V1&2]. If bone density is mas-sively low in hip but normal in spine the DEXA scan should have metadata attached to it so the non-specialist can easily process that info [P5V1B]

Design recommendation: Include temporal factors & metadata where relevant when zooming in on the individual data points

11. Individual data point view

Doctors notes

Allow the user to view notes about each individual data point [information>contextual data]

Rationale: Having access to notes about to a data point would be useful for user. The user would check notes of other specialist when looking at data they might not under-stand [P2V2B]

Design recommendation: Have button to doctors notes in the individual data point view which allows the user to read any notes that are attached to a specific data point

12. Individual data point view

Literature & official information portals

Allow the user to view evidence about the risk associated with each data point [infor-mation>supporting sources]

Rationale: User wants to see all factors and be able to assess evidence behind it, useful for things with low clinical utility [P2V2B], for example if a patient asks about e-cigarettes [P6V2C]. Different clinicians remember risk differently, literature should be there to be able to check [P6V2C].

Design recommendation: Have a ‘view evidence’ button that shows evidence & litera-ture for data point

13. Individual data point view

Guidelines

Allow the user to view NICE, NCCN (for oncology) or other guidelines relevant to the individual data point where releavant [information>supporting sources]

Rationale: NCCN guidelines for patient surveillance would be helpful [P2V2B]. NICE guidelines for a range of options & treatments [P5V2C]

Design recommendation: Include a button that links a specific data point to the rele-vant official guideline (simple example: what does NICE say I should do about high blood pressure?)

14. Individual data point view

Back button

Make it easier for the user to navigate back to the overview [interface specific]

Rationale: User doesn’t know how to get back to overall risk [P2V2B]

Design recommendation: Make the back button more visually apparent/allow the user to pinch zoom out to the overview

108

15. Individual data point view

Border design

Keep the interface design clean & consistent [interface specific]

Rationale: User was surprised when the border (purple above and below data) ap-peared [P1V2A]. Visually nice, but found it unnecessary for task and distracting

16. Individual data point view

Patient information bar

Make sure the user understands whether an individual data point contributes to risk, protective or it neutral in the individual data point view [risk type>overall risk]

Rationale: Some most users did not comment on it, some users understood [P3V2C], and some did not notice but mentioned the point [P6V2C] of seeing the risk severity

Design recommendation: Make the zoomed in patient information bar red & green lines more prominent

17. Overview

Overall risk calculation

Give the user an overall risk calculation that changes if modifiable factors are changed in the patient [P3V1&2] [interface specific]

Rationale: overall risk was missing from both visualisations, something the majority of participants noted [P2V1&2] [P3V1&2] [P1V1&2] [P6V1&2]. Considering the disparity of the data sources that are represented in the visualisations, there are no risk calcu-lators that take all factors into account. A solution to this is to include existing risk cal-culators (whichever is preferred by clinician) to display a summary number, because in the end it’s all complex but clinicians want a summary [P1V2A]

Design recommendation: Individual conditions have their own risk calculators (i.e. risk of diabetes in 5 years is X [P3V1A]), so a number, percentage, slider bar etc. [P2V2B] representing overall risk should be implemented

18. Overview

Patient verification

Keep identifying information about the patient clearly visible on the overview screen [interface specific]

Rationale: The name and photo of patient is good for verifying it’s the right person [P5V1B] [P5V2C] [P6V2C]

19. Overview

Risk factors

Make high/medium/low risk factors visually apparent to the user [interface specific]

Rationale: It would be good to highlight risk factors that raise risk immensely [P2V2B]. Thought Visualisation 2 didn’t have high/med/low scale [P2V1A]. The user needs to be primed to the scales to see the size of box is relevant to the scale of the issue [P1V2A]. Not sure if size of colour represents importance in columns [P4V2B]

Design recommendation: Show risk level with different shades of colour [P6V2C] with high risk given the highest visual priority

20. Overview

Colours

Rethink the colours used to distinguish categories [interface specific]

Rationale: Participant expects red to signify risk [P6V2C]. Some category colours clash with colour scales in the single data point view [P1V2A]. Red-green next to each other is bad for universal design (not colour-blind friendly) [P1V2A]

109

21. Overview

Size of data point

Make it possible for the user to navigate to specific single data point view without error [interface specific]

Rationale: Pixel area makes it hard to find and navigate to each data point [P1V2A], this could result in errors where the user accidentally navigates to a data point he wasn’t trying to reach

Design recommendation: Make each data point large enough to tap on a touch inter-face (or allow pinch zoom)

22. Differentiation

Can/cannot change

These factors are of high importance to the clinician as it relates to what data points are actionable [interface specific]

Rationale: Modifiable & unmodifiable, can change & can’t change, static & dynamic were terms for the same concept that clinicians said were useful to the working doctor [P3V2C] [P1V1&2] [P2V2B] [P4V1B]

Design recommendation: Give changeable risk factors the most visual priority

23. Differentiation

Clinically validated

Change ‘clinically validated’ to something that is better a better conceptual fit for clini-cians [interface specific]

Rationale: Clinicians work with ‘soft’ & ‘hard’ data/qualitative & quantitative data, but the term ‘clinically validated’ was poorly understood; ‘if it’s not validated why is it there?’ [P1V2A], ‘why show factors that are not validated?’ [P4V2B]. In the words of one participant ‘we don’t necessarily trust the data, but we still like to include it in our calculations’ [P2V2B], with this in mind the visualisations were meant to give, what participant 6 called ‘an objective view of evidence based & subjective data’ [P6V1&2]. Thus, we should focus on showing the data points that contribute to an evidence based calculation of risk

Design recommendation: Have evidence based risk factors on the right, and patient data/contextual factors on the left (but keep colour codes to show the risk factors within patient information).

24. Filters

Search

Allow the user to search for a data point [external factors/observation notes]

Rationale: The large amount of patient data [P1V1A] may lead to particular data points being hard to find. Some participants found it hard to find particular data points when asked to interact with the concept

Design recommendation: Add a search bar to find specific data points