IVR Caller Experience Metrics World IA Day 2013

March 2

What is Caller Experience?

2

• Processes and Design Principles that put

the caller at the center of attention, to

ensure that an IVR is:

– Useful

– Usable

– Aesthetically Pleasing

3 Dimensions of Design

3

Determine Caller Needs

Design To Fit Needs

Engaging

Artful

Clever

Pleasant

Make Obvious

Reduce Errors

Save Time

Equation for Caller Experience Measure

P(Successful Caller Experience) = P(Useful) x

P(Usable) x P(Aesthetically Pleasing)

• Equation is more about people than technology

• Even if the caller experience is useful, it will not

succeed if people cannot use it or do not like it.

3 Key Processes for Caller Experience

1 Know the Caller

3 Evaluate

Appropriately

2 Employ Iterative Design

Caller Needs Analysis 1

Very

Detailed

More

Specific

General

Design

Requirements

• Demographics - age, education, location…

• Call Environment - home, car, office…

• Concurrent Activities - walking, driving,

working

• Needs – get something done

• Wants – quickly and reliably complete

• Enjoys – no “what ifs”

• Exact procedures - step-by-step

• Automation delegation - the caller acts, the

system responds

• How to think - mental models

Iterative Design & Analysis 3 2

Requirements

Compare to needs analysis, standards,

lessons learned, best practices, etc.

High Level Design

Building the functional pieces, placement, order

Detailed Design

Dialog design matches persona, error recovery, transitions, natural traversals

Heuristic Evaluations

UI peers reviewing the design – different peers find different issues

Usability Testing

Compare caller performance & perceptions/preferences to CE Thresholds

Caller Experience Design Principles

• Differentiate confusable functions

and commands

• Know human cognitive and auditory

limits

• Allocate tasks that people are

better at to people, and tasks

machines are better at to machines

• Fit technology to people,

not visa versa

• Minimize what must be learned

• Build in performance support

including reminders & examples

• Keep it simple

• Consistency

• Error recovery

• Cultural norms

• Obviousness

• Menu option grouping

• Caller-controlled pace

• Flexible order of actions

• Consider caller’s environment

• Consider caller’s other activities

• Familiar language & terminology

Measuring Caller Experience

Performance

(Success on

tasks)

Best Success

Best Rating Perceptions/Preferences

(Opinion Ratings)

Caller Experience

Score

Measuring Performance

•Performance Success - metric for how usable a product is.

•Behavioral Data - measuring speed and accuracy

collected by having callers do tasks.

•Scenario-Based Tasks - information-gathering or problem-

solving descriptions are commonly used.

•Design Performance Data - compared to target thresholds

for performance.

Performance Examples

Task Scenario Examples

Task 1: Take money from your checking account

and deposit it into your savings account.

Task 2: Report a death in the family and identify

what needs to be done to collect benefits.

Threshold Examples

Gold Medal Performance: 90% of

participants perform each task error-free.

Silver Medal Performance: 80% of

participants perform each task error-free.

Summarizing Performance

• Performance is summarized for each task.

• Performance on each task can be compared to acceptance thresholds.

• Overall performance is measured by averaging across all measured tasks. This number summarizes how usable the product is.

100

90

80

70

60

Task 1

Task 2

Task 3

Overall

Gold Medal

(outstanding)

Silver Medal

(acceptable) P

erf

orm

an

ce

Su

cc

es

s

(%)

Measuring Perceptions/Preferences

•Opinions form the metric for callers’ Perceptions and

Preferences.

•Questions are asked to determine opinions about

usefulness, usability, and aesthetics.

•A Numerical Scale with anchoring verbal labels is used to

make the opinions quantifiable.

•Deductive interviewing is used to probe callers on

opinions and why is that data?

Perceptions/Preferences Examples

“Useful” example

The system was missing some

important features.

1 2 3 4 5

Strongly

Disagree

Strongly

Agree

“Usable” example

I think most people would learn to

use this system quickly.

1 2 3 4 5

Strongly

Disagree

Strongly

Agree

“Aesthetics” example

The system voice was pleasant like

an agent that I might speak with.

1 2 3 4 5

Strongly

Disagree

Strongly

Agree

Threshold Examples

Gold Medal Preference:

Average rating is 4 or higher.

Silver Medal Preference:

Average rating is 3 or higher.

Summarizing Perceptions/Preferences

• Perceptions/Preferences are summarized for the three dimensions of design, usefulness, usability and aesthetics.

• Ratings for each dimension can be compared to acceptance thresholds.

• An overall rating can be obtained by averaging across the three dimensions. This number is the Perceptions/Preferences opinion rating.

5

4

3

1

Useful Usable Aesthetics Overall

Gold Medal

(outstanding)

Silver Medal

(acceptable)

2

Perc

ep

tio

n/p

refe

ren

ces R

ati

ng

s

Caller Experience Score Matrix

Performance Success (%) (Combined results

for all measured tasks)

Qualitative Ratings

(Combined opinions about usefulness, ease of use and aesthetics)

The Caller Experience Score

Matrix combines the

quantitative data

(callers’ performance) with

the subjective data (callers’

perceptions/preferences)

into a single summary

statistic about the overall

Caller Experience.

GOLD MEDAL CALLER EXPERIENCE

SILVER MEDAL CALLER EXPERIENCE

100

90

80

70

60 1 2 3 4 5

Caller Experience Score Matrix Examples

Performance Success (%) (Combined results

for all measured tasks)

Qualitative Ratings

(Combined opinions about usefulness, ease of use and aesthetics)

The Caller Experience Score

Matrix combines the

quantitative data

(callers’ performance) with

the subjective data (callers’

perceptions/preferences)

into a single summary

statistic about the overall

Caller Experience.

GOLD MEDAL CALLER EXPERIENCE

SILVER MEDAL CALLER EXPERIENCE

100

90

80

70

60 1 2 3 4 5

Auto & Home

Auto & Home

Dental

Dental

Life

Life

Disability

Disability

Questions ?