2/16/10

1

Development and evaluation of clinical IT systems

Clinical information systems, tdt 4213

Gry Seland, 16. Feb. 2010

TDT4213 – 16.02.10

Agenda:

• 1215-1300: Lecture: Development and evaluation of clinical IT systems Guest lecture by Gry Seland, researcher, IDI, NTNU

• 1315-1400: Project presentation/

background (The COSTT project) Arild Faxvaag, MD, PhD, Associate professor and Leader of Norwegian Research Centre for Electronic

Patient Records (NSEP)

Goal of lecture

Knowledge of some user-centered approaches of relevance for the project in

this course, and for design and evaluation of clinical IT system in general.

The challenge

2/16/10

2

2/16/10

3

2/16/10

4

2/16/10

5

2/16/10

6

2/16/10

7

2/16/10

8

Put focus on usability: Effectiveness, efficiency and

user satisfaction for specific users with specific goals

in specific contexts

Efficiency (effektivitet)

”All physicians at St. Olav should be able to find their patient

list after attending an introductory course on the system”

Effectiveness (anvendbarhet)

- lett for brukeren å utføre en oppgave med tilgjengelige hjelpemidler

Effectiveness of DocuLive to:

not effective very effective

Example 2: Request blood test by IT system or nurse

User satisfaction (Subjektiv opplevelse)

DocuLive at St Olav:

2/16/10

9

User involvement? Methods to understand and

involve users of clinical IT system

• Interviews and discussions

• Observation studies

• Role play

• Usability testing

Methods

Interviews and group discussions

• From structured to unstructured discussions

• Useful information about practice and needs

• Memory constraints

• Demand characteristics: Tell what is expected

• Don’t know about technological solutions

• Not everything is easy to articulate (tacit knowledge)

• What about mobility?

Methods

Observation studies

• Video, present in the same room, shadowing

• Observe things people don’t talk about

• Must wait with questions,

cannot interrupt

• Difficult to transform

observations to design

suggestions

Methods

• Some times it is easier to show than to explain

• A role play can be “frozen” and replayed

• Design of mobile technology needs methods that

make visible the mobility of the situation

Methods

Role play

• Technology change practice and vice versa

Technology

Practice

Present

Present

Future

Future

Observation

studies Future workshops,

BPR

Task analysis,

Use case (RUP)

Interviews

Role play and

low-fi prototyping

Methods

Role play and prototyping

2/16/10

10

1. Act out a scenario until

someone identifies an

information need

2. ”Freeze the scenario”

3. Choose a prototyping

model. Sketch the functionality on paper.

4. Continue acting

Design in action

Methods

The nurse:

1. acts out the scenario as it is today

2. explains what current practice is like

3. chooses a prototyping model and sketches the functionality on paper.

4. explains the functionality to the ”developer” sitting in the corner

Design in action

Methods

Limitation of role-play workshop

• The situation i too complex to be reduced to a

simple scenario

• The use of role-play puts demands for new skills

on the facilitator

Methods

Usability test in the IT development

• Formative – input into process

• Summative – final evaluation of system

Usability test report

DIPS vs. DocuLive

Usability tests (brukbarhetstest)

• A test subject solve specific tasks while ”thinking aloud”

Paper prototype or computer system Roles: Test subject, test ”co

leader, Scenario observ,

Task observers

EVAlab, Lille (http://www.univ-lille2.fr/evalab/presentation_evalab.htm)

2/16/10

11

The usability lab

Kilde: Dag Svanæs/TV Trøndelag

What is usability testing good for?

- Identifying breakdowns:

Watch people struggele with ”intuitive solutions”

- speed up design process

- Input to design

- find problems during development, not after implementation

Weaknesses - find solutions

- test short time use

- in lab, taken out of context

Conventional “thinking aloud”

Laboratory usability testing:

How to (traditional guidelines) 1: Introduce yourself

2: Explain the purpose of the test

3: Tell the participants that they do not have to complete the test

4: Describe equipment in the room, limitations of the prototype

5: Teach how to “think aloud”

6: Explain you cannot offer help during the test

7: Describe the task and introduce prototype

8: Ask if everything is clear, and start the test

9: End test, let the test user explain his/her feelings about the system. Ask questions about unclear issues

10: Use the results

Thinking aloud Thinking aloud

OK, so this is my office?

Kilde: Ole Alsos

It seems like there is a message here on my cell phone! I think I will be

able to read it if I press this button. Yes, there is the message. It says…

2/16/10

12

Where was patient Adams room in this lab? I think it is behind this wall. Huh? The cell phone is blinking. Aha, here comes some information

about patient Adams. It is a little bit difficult to read while I a moving.

My phone is calling again! I’ll better check what the problem is. I’ll press

the button in the middle. It seems like it is the right button to press.

Message again? Yes, it is about the patient I started with.

Simulation-based usability testing

Instructions, simulation-based usability test

Nurse instructions for situation 2: Ward round

Patient room 250 and 262

Time: 9.30 AM

Present: Physician and nurse

Artifacts: Laptop on trolley, bar code reader, paper with

information on the patients

Anne Olsen has severe pain

Kristin Hansen is a new patient in the ward

2/16/10

13

Pre-round meeting Ward round

Administration of medications Focus group discussion

Simulation-based testing

• Knowledge of clinical processes, more realism

• Requires extensive preparations to ensure a realistic lab setting

Combination of methods

2/16/10

14

Project challenge: How do you develop and evaluate a

product based on some of the COSTT-ideas?