managing participant satisfaction

Managing Participant Satisfaction

Elisabeth LangIntangible Asset Manager, Senior Consultant, CFI Group

+46-8-562 800 00, Box 70373,107 24 Stockholm, Sweden Thursday October 9th, 2008

© 2008 CFI Group - 2

Content

– The point of satisfaction

– The CFI Group method

– Describing IFL from the participants’ perspective

– Analyzing results

– Identifying focus areas

– How to leverage the information to achieve IFL’s goals

– The IFL perspective

Feel free to interrupt with questions whenever you likeNB! Results have been altered, data is not real


Is there a point to focusing on quality?

IFL Overall

80

77

81

80

91

0 10 20 30 40 50 60 70 80 90 100

Investment

Activities

Personal development

Loyalty

RecommendExam

ple

Increased satisfaction, loyalty and willingness to recommend

Increased participation in alumni activities, strengthening the alumni network

Stronger brand Higher revenue


What’s the effect of satisfied customers?

The cost of retaining customers is much smaller than the cost of acquiring new ones

Satisfied customers are much more likely to consume more and again, and are also much more likely to recommend the company, generating new business at a lower cost

Source: The Satisfied Customer,Claes Fornell


Satisfaction drives profitability and shareholder value! ACSI Fund vs. S&P 500: April 2000 (start) – May 26th, 2008

Source: S&P 500 from msn.com-100%

-50%

0%

50%

100%

150%

200%

Cu

mu

lati

ve P

erfo

rman

ce

S&P 500

ACSI Fund


The reasons why IFL focuses on quality and satisfaction

1. Identify relevant areas from the customers’, i.e. participants’, perspective

2. Identify the present situation, i.e. the participants’ assessment of IFL as a supplier. This is achieved by:

– Asking questions in relevant areas

– Calculating the scores for the individual questions that comprise the areas, as well as for the areas as a whole

3. Identify key areas that affect participant satisfaction and desired behaviors. This is achieved by:

– Statistical analyses, e.g. factor analyses and structural equations modeling (PLS regression)

– Combining scores and impacts derived from the statistical analysis

4. Initiate actions for improvement

– Maintain current position on local and international market

– Improve position on local and international market


Content









The method captures cause and effect relationships, and identifies the driving factors

Example:

PerceptionGut

feelingAction

Cause and effect

Faculty? Satisfied? Recommend?


The model is a conceptual version of the participants’ relationship with IFL

The model includes relevant questions and performance areas (e.g. Image) that describe the participants’ opinion on IFL

Each performance area consists of one or more questions

The questions are weighted together to form the performance area scores, thus the results are not percentages

PSI consists of three separate questions, also weighted together

If a respondent answers less than 2/3 of the questionnaire, the answer is not included in the analysis

If a respondent leaves a question unanswered, it is left blank

Information

Effects

Applicationprocedure

Facilities &accommodation

Program director & program admin.

Faculty's inspiration

Faculty’s pedagogy

Faculty's competence

Literature & materials

Interaction

Participant group

Program structure

Investment

Activities


Loyalty

Recommend

PSI

Exam

ple

© 2008 CFI Group - 10

A value is calculated for each performance area on both the left hand side…

IFL Overall

78

88

86

85

83

81

86

77

84

85

76

85

81

0 10 20 30 40 50 60 70 80 90 100

Information

Application procedure

Facilities & accommodation






Interaction

Participant group

Program structure

Effects

PSI

Exam

ple

© 2008 CFI Group - 11

…and the right hand side of the model

IFL Overall

80

78

81

79

88

0 10 20 30 40 50 60 70 80 90 100

Investment

Activities


Loyalty

Recommend

Exam

ple

© 2008 CFI Group - 12

Each performance area consists of one or more questions, weighted together to minimize the error term

Literature &Materials

PSI

Litterature & materials consists of four questions, for example:– The format of the material that was distributed (compendia, hand-

outs etc.) was user friendly– The literature and material used was up-to-date– The literature and material used complimented the teaching

process– The literature and material used will be helpful to me in my work

PSI (Participant Satisfaction Index) is not an average score of all other questions, PSI is an index based on three separate questions:

– How satisfied are you with the program?– To what extent has the program met your expectations?– Imagine a program which is perfect in every sense. How satisfied

are you with the program compared to this ideal program?

It is necessary to have a separate PSI to be able to do the cause and effect analysis

Example:

Literature &Materials

PSI

© 2008 CFI Group - 13

The questions are answered on a scale from 1-10, in the analysis, the answers are converted to a score on a scale from 0-100

In the questionnaire, the questions are answered on a scale from 1-10

“Don’t know” is an option

To make it easier to interpret results and detect differences between segments and/or changes between years, the answers are converted to a score on a scale from 0-100

If the respondent answers “7” on a question, this becomes a score of “67”

Please note that this is a score, not a percentage

If a respondent answers less than 2/3 of the questionnaire, the answer is not included in the analysis

1 02 113 224 335 446 567 678 789 89

10 100

1 02 113 224 335 446 567 678 789 89

10 100

© 2008 CFI Group - 14

Scores can be more or less precise, depending on the method used

Using optimal weights all the available information is used, including the relationship to all other questions and performance areas

Calculating a performance area value based on several questions increases the reliability of the value, decreasing the width of the confidence interval

A scale with ten points allows for more response alternatives and increases the possibility of detecting actual changes

In a “top box” approach only a small part of the information is put to use (typically responses 4 and 5)

Top Box approach: % 4 and 5

Single item, five point scale

Single item, ten point scale

Multiple item scale, equal weights

Multiple item scale,optimal weights

PRECISION: Width of Score Confidence Interval

PO

WE

R:

Ab

ility

to

det

ect

chan

ge

Pred

ictio

n Er

ror

inte

rval

© 2008 CFI Group - 15

Content








© 2008 CFI Group - 16

External benchmark indicates performance compared to other industries, but internal benchmark will prove most useful in day-to-day management

Source: SKI 2007

Satisfaction benchmark

8278

71 7168 67 67 65 63 61

0

10

20

30

40

50

60

70

80

90

100

IFL O

vera

ll

US MBA P

rogra

m (F

T Top

30)

Banki

ng, B

2C-m

arke

tM

ortg

ages

, B2C

Insu

rance

com

panie

s, B

2CTel

ecom

/bro

adba

nd

Retai

l cha

ins

College/

univ

ersi

ty e

ducatio

n

Trans

ports

(airl

ines

/trai

n/bus

)Pow

er d

istri

butio

n, B2C

* Full time MBA

*

Exam

ple

© 2008 CFI Group - 17

21%

56%

23%

Up to 35 years old

Between 36 and 45years

Older than 45years old

There is no significant difference in PSI scores between age groups

PSI

80 81 82

0

10

20

30

40

50

60

70

80

90

100

Up to 3

5 ye

ars

old

Betwee

n 36

and 4

5 ye

ars

Older

than

45

year

s old

PSI

PSIscore

Exam

ple

© 2008 CFI Group - 18

42%

58%

Female

Male

PSI

83 80

0

10

20

30

40

50

60

70

80

90

100

Femal

e

Mal

e

There is only a slight difference in PSI scores between genders

PSI

Exam

ple

© 2008 CFI Group - 19

69%

31%

Private sector

Public sector

PSI

87

73

0

10

20

30

40

50

60

70

80

90

100

Privat

e se

ctor

Public s

ecto

r

The private sector is significantly more satisfied

Exam

ple

PSI

© 2008 CFI Group - 20

Identify programs with abnormally low or irregular scores, rather than just focusing on low scores

EGMP-FLHT07

88 (+3)

83 (+6)

88 (+3)

89 (+5)

75 (-12)

93 (+4)

84 (+3)

70 (0)

96 (+9)

88 (+2)

98 (+15)

85 (+7)

86 (+5)

0 20 40 60 80 100

PSI

Information







Litterature & materials

Interaction

Participant group

Program structure

Effects

(* Difference compared to IFL Overall indicated in parentheses)

Exam

ple

Ask yourself: – Why is this program so much better/worse in

these areas?– Is there some key learning that can be

applied when comparing to other programs?

© 2008 CFI Group - 21

Natural score variations occur between types of questions

When analyzing score levels it is unwise to compare scores across performance areas

Performance scores should be compared with relevant benchmark instead

1 10

© 2008 CFI Group - 22

Compare PSI scores across programs and participant groups

PSI

8284

68 67

86 87

7680

82 8277

86

0

10

20

30

40

50

60

70

80

90

100

IFL O

veral

l 1 2 3 4 5 6 7 8 9 10 11

© 2008 CFI Group - 23

Focus on important performance areas and compare to benchmark

Program structure

7983

6157

83 82

69

7470

81

70

82

0

10

20

30

40

50

60

70

80

90

IFL O

veral

l 1 2 3 4 5 6 7 8 9 10 11

© 2008 CFI Group - 24

Identify programs which deviate from the overall pattern

0 10 20 30 40 50 60 70 80 90 100

Information




Interaction

Effects

Program structure


Participant group




PSI

EXEXVT07 IFL Overall

Performance areas will always have varying levels

Identify “normal” pattern Identify programs that deviate from this

pattern

© 2008 CFI Group - 25

New programs could have uneven performance – it is important to identify weaknesses quickly and take action!

New Program (HTXP 06/07)

68

68

94

84

86

70

63

70

72

75

84

55

74

0 10 20 30 40 50 60 70 80 90 100

PSI

Information








Interaction

Participant group

Program structure

Effects

© 2008 CFI Group - 26

Established programs that run smoothly should have relatively even scores – if not, what is the quality issue?

Program on 50th run (LY 1-07)

87

78

88

87

92

88

88

90

85

91

88

82

91

0 10 20 30 40 50 60 70 80 90 100

PSI

Information








Interaction

Participant group

Program structure

Effects

© 2008 CFI Group - 27

Find patterns and identify underlying reasons

Private vs Public sector

87

90

82

86

93

83

73

82

74

78

86

76

0 10 20 30 40 50 60 70 80 90 100

PSI

Effects

Program structure

Participant group

Interaction


Private sector Public sector

Exam

ple

Participants from the private sector give notably higher scores

What is the reason (background, prior knowledge, levels of expectation, quality issues on targeted courses)?

If found, can something be done to affect participants from the public sector in the same way?

© 2008 CFI Group - 28

Find changed patterns and identify underlying reasons

Exam

ple Private vs Public sector

94

87

60

87

89

93

85

76

64

92

87

68

72

71

0 10 20 30 40 50 60 70 80 90 100




Program director & programadmin.



Information

Private sector Public sector

Why is the pattern broken for two performance areas?

Which steps need to be taken to prevent quality issues in the future?

What is the cost of implementing these steps – and what is the cost of refraining from implementation?

© 2008 CFI Group - 29

Content








© 2008 CFI Group - 30

Where should improvement efforts be focused?

Resources (e.g. time, money) are always limited

Improvements should be focused on areas that will have a higher impact on PSI and the desired behaviors

These areas are identified in the cause and effect analysis

Information

Effects

Applicationprocedure

Facilities &accommodation






Interaction

Participant group

Program structure

Investment

Activities


Loyalty

Recommend

PSI

© 2008 CFI Group - 31

How to identify the drivers of Satisfaction

The graphs show the impact of Information and Faculty’s competence on Satisfaction– The slope of the line represents the impact– The steeper the line, the higher the impact on Satisfaction

In this example:– If the score for Information increases with five (5) units, the score for PSI will increase with 0,2– If the score for Faculty’s competence increases with five (5) units, the score for PSI will increase with

1,5

The impacts are calculated using PLS-analysis (an advanced type of regression analysis) using PSI as the dependent variable

Scores for Information

Scores for PSI

Low impact

+5

+0,2

Scores for Faculty’s competence

Scores for PSI

High impact

+5

+1,5

Illustration

© 2008 CFI Group - 32

To be able to rely on the results we need a significant amount of answers

The analysis can in theory be carried out with a few answers, but…– The results would be very unreliable as an individual respondent would influence the results

heavily– The results might be due to chance, rather than certainty

As the number of responses increases so does the stability of the results

n=2 n=20 n=100 n=250

Examples

© 2008 CFI Group - 33

Example comparing high impact vs. low impact performance areas

© 2008 CFI Group - 34

Working with the PSI results – prioritizing actions

To prioritize actions, focus on those areas that will have the greatest impact on PSI and recommendation

– Performance areas with high impacts will affect PSI and recommendation more than performance areas with low impacts

– It is easier to improve performance areas with lower scores

The Priority Matrix combines these two aspects:

– Focus primarily on performance areas within the Improve-quadrant – these have higher impact and lower scores

High

Low

Impact on PSI

4. Exploit/Save 2. Improve/Maintain

3. Monitor 1. Improve

1

2

Score

© 2008 CFI Group - 35

Example of a priority matrix

Exam

ple

High

Low

Score

Impact on PSI


Application procedure Faculty's competence


Information Room and food

© 2008 CFI Group - 36

Content








© 2008 CFI Group - 37

Working with PSI results to achieve the goals

PSI results

Get all co-workers involved

Action plansExecution

Follow-up

Planning:Internal

coordination groupRevision

© 2008 CFI Group - 38

How do other companies do it?Example from a real estate firm

A strong vision dictates all actions Actions are value-based and firmly

established on all levels– Strong endorsement from top

management Humor

– Welcome to a better world (landlord)!

Exam

ple

© 2008 CFI Group - 39

Consistency is key

The vision to become Sweden's premier real estate company – and the conviction that this is achieved through customers – guides all actions

The company wants to be number one…– Employer– Landlord– Investment

© 2008 CFI Group - 40

Symbolic and systematic

The customer is always in focus All processes have been…

– Analyzed– Documented– Implemented as a part of daily lives

© 2008 CFI Group - 41

An example taken from a telecom firm

Symbolism – King Customer! Clear, attainable goals Champions are appointed to each area Everybody is involved in quality assurance

and customer delivery (e.g. billing, complaint handling, technicians, sales personnel, PR, HR…)

Endorsed by management– Management actively ensures that the whole

company sees King Customer as important

Exam

ple

© 2008 CFI Group - 42

The ultimate goal is clear – and customer satisfaction is just a stepping stone

The goal with ”Kung Kund” is a larger customer base and increased revenue!

More paying customers

More paying customers

More satisfied customers

More satisfied customers

© 2008 CFI Group - 43

Company culture focuses on the customers

Use of humor to focus on customers, e.g. using known sayings and proverbs, “adapted” to the situation

– For Customer and country– The best with this company is the customer– Don’t wait until tomorrow what you can give

to the customer today– A good customer makes your life longer

Priorities are set straight by management

© 2008 CFI Group - 44

Everybody influences customer satisfaction

HR: potential employees are also customers

Billing: how can we best meet customer needs?

PR: marketing influences customer expectations

Product development: pricing solutions have a great impact on customers

© 2008 CFI Group - 45

Specific actions and deliberate work leads to positive results

Structure is everything– Without proper organization results will not

be achieved

SMART actions and goals– (Specific, Measurable, Attainable, Realistic,

Timely)

Use technology!– E.g. a database to keep track of actions and

that sends out automatic reminders

© 2008 CFI Group - 46

Working with the PSI results – setting goals and increasing scores

When results are low it is easier to increase them and they will have a higher effect on both CSI and Loyalty

CSI

Loyalty

High returns oninvestments in CSI

Low returns oninvestments in CSI

40%

60%

80%

100%

50 100807060 90

Example

© 2008 CFI Group - 47

Working with the PSI results – Positive and hygiene factors

The priority matrix consists of both positive factors of satisfaction as well as hygiene factors. Normally, the positive factors have a high impact on the participants’ satisfaction with IFL. The hygiene factors don’t.

However, hygiene factors have a greater impact on PSI when the scores are low – once the scores have reached the hygiene level the impact on PSI decreases. If IFL would underperform in an area that is considered hygiene it may start to become relatively more important for the overall satisfaction.

Positive factors are qualifiers – you need to maintain a certain level to even be considered. Once this level is achieved increases will lead to higher CSI scores.

Score

Hygienefactors

Positivefactors

CS

I le

ve

l

© 2008 CFI Group - 48

Living, breathing and thinking PSI is a key factor

PSI results

Get all co-workersinvolved


Follow-up

Planning:Internal


PSI results

Get all co-workersinvolved


Follow-up

Planning:Internal


Set goals!E.g. today PSI is 81 – next year we aim for 84

Include the co-workers in the creation of action plans

Use internal benchmarks and best practice – show success cases

Inspire through workshops and seminars

Integrate with internal systems and processes

Follow up!

© 2008 CFI Group - 50

Based on key areas, plan actions and set targets

Activity/actionPerformance areaTarget score

Score today

DoneResponsible Target date

© 2008 CFI Group - 51

Take the drivers seat and control the process!

PSI results2009!

Get all co-workers involved


Follow-up

Planning:Internal


Managing Participant Satisfaction

Elisabeth LangIntangible Asset Manager, Senior Consultant, CFI Group

+46-8-562 800 00, Box 70373,107 24 Stockholm, Sweden Thursday October 9th, 2008

managing participant satisfaction

Documents

performance areas

relevant questions

participants assessment

performance area scores

participants relationship

participants opinion

individual questions

separate questions