“evolved” mystery shopper - lanla · by the mid-1980s the mystery shopper methodology begins to...

“Evolved” Mystery ShopperA VERY PERFORMANT METHOD TO MEASURE

THE CUSTOMER EXPERIENCE!

Measure With the Efficient Method! by Christian Watier, MBA, Ph.D.

In today’s economy, where the services industry is so important, consumer sensitivity MUST be a major

concern for business managers operating in a B2C business context. Several authors assert that the North

American economy has moved beyond the service economy towards an experience based economy.

Although everyone recognizes the importance of measuring the customer experience, many still confuse

the vastly different concepts of service quality, customer satisfaction and customer experience.

Consequently, several companies barely or poorly measure the customer experience deemed so important.

Here are the latest developments in this science that benefits to managers who can master it.

Contents / Segments identification:

MASTER THE PRINCIPLES1. Quality of service, customer satisfaction and customer experience

2. “Traditional” vs “Real Consumer” Mystery Shopper

3. Developments in measuring methods

CONSIDER THE OPTIONS4. Advantages

5. Disadvantages

TAKE ACTION6. Methodology to Successfully Implement a Mystery Shopper Program

6.1 Program Creation

6.2 Communication

6.3 Management Values

6.4 Questionnaire Design

6.5 Program Management

6.6 Analysis and Recommendations

MASTER THE PRINCIPLES

Very few research methodologies effectively measure the real customer experience. The “real consumer

“type of mystery shopper; also called “improved” mystery shopper, is one such methodology. Often confused

with the so-called “traditional” mystery shopper approach; or commonly called the “robotic” mystery shopper,

the mystery shopper has acquired a bad reputation. Yet, when the “real consumer” type of mystery shopper

is used in a rigorous fashion, it enables the efficient measure of the customer experience and substantially

improves the profitability of businesses that use it through increased sales and/or a decrease in operating ex-

penses.

1. Quality of Service, Customer Satisfaction and Customer Experience

Until the beginning of the 21st century, the measurement of quality of service (also called the measurement

of conformity) and customer satisfaction were performed separately. Indeed, several businesses throughout

the world use standardized tests to measure the quality of services (e.g.: SERVQUAL, designed by

Parasuraman, Zeithaml and Berry in 1988), mainly using the “traditional” mystery shopper method or audit

(e.g.: in hotels). It is important to note that this approach, which consists essentially in measuring the gap between the standards set by management and the performance of the business (right portion of

Figure 1) IS NOT a measure of the customer experience.

In addition to these measures of quality of service, the survey is the most widely used marketing research

method to measure customer satisfaction (very popular in the 80 and 90s). However, once again, it should be

noted that the approach consists mainly in measuring the gap between customer expectations and the latter’s perception (left portion of Figure 1), IT IS NOT a measure of the customer experience.

In the early 2000s, a new research trend emerges. Several authors suggest that due to methodological issues

associated with the separate measures of the concepts of quality of service and customer satisfaction,

customer service must be defined by the simultaneous measure of both concepts (Shaw, 2002; Riechheld,

2001, 2006, 2009 and 2011). This gives birth to the Customer Experience concept. It is worth noting here that

performance indicators measured separately (satisfaction measure vs. conformity measure) are at the root

of endless debates between marketing and operations departments. Indeed, this explains why many

businesses observe that the customer always appears unsatisfied (satisfaction measure) in spite of all the

efforts made for the customer (conformity measure). Those businesses would benefit from the integration of

the customer experience concept, namely the simultaneous interaction between what the business does for

its customer and what the customer perceives of the business’ efforts. The customer experience (lower

portion of Figure 1) consists in the simultaneous measure of the quality of service and customer satisfaction.

The preferred method of marketing research firms that is the most efficient to measure the customer

experience is the “real consumer” type mystery shopper method. What is a “real consumer” type mystery

shopper? In what way is this shopper different from the “traditional” mystery shopper (robotic)? What are the

strengths and weaknesses of using “real consumer” mystery shoppers? What are the key success factors in

implementing a program to measure customer experience with this type of “improved” mystery shopper?

These questions should be looked at in order to shed some light on the topic.

First of all, it should be noted that the mystery shopper, no matter the type, is achieved by an anonymous

paid provider (either by salary as is the case for a “traditional” mystery shopper or by compensation in the

case of a “real consumer” mystery shopper), who performs his/her evaluation without revealing his/her

identity and role to the manager of the point of sale. It differs totally from the audit, in which the goal and the

necessary means are announced, such as in the hospitality industry, where professionals (internal or

external) are brought in to review the hygiene and to take samples of raw materials.

Moreover, a “traditional” (or robotic) type mystery shopper is a fake customer sent by a specialized service

provider to measure the conformity of a point of sale and/or of the individuals in contact with customers.

Mystery shoppers, in no way representative of the business being evaluated, are employees (this is their job)

sponsored by their employer to audit mainly the financial, transportation and public service sectors and

government departments. Thus, they perform hundreds of evaluations every year. In spite of the fact that

they do not have real consumers’ motivations, their evaluations are still a type of participatory observation.

Mystery shoppers are normally mandated to evaluate in a methodical manner, according to quality standards

predefined by the business (known to mystery shoppers). Therefore, the emphasis is placed on the service

experience as it unfolds, while noting which activities or standards are or are not met, rather than gathering

perceptions with respect to the actual experience.

Unlike the “traditional” mystery shopper, the “real consumer” (or improved) mystery shopper, as the name

suggests, is a true customer with a real intent to purchase. He is rigorously selected and professionally

trained to perform mystery shopper evaluations, while maintaining his “naïve” perception of a real consumer.

This type of mystery shopper must perform a minimum of evaluations on a voluntary basis without being

informed of the business’ standards, thus ensuring maximum representativeness of the real customers of

the business under evaluation. “Real consumer” mystery shoppers are normally instructed to behave in a

natural way, evaluating not only what was done for them but also reporting their perception of what was

done for them (their real experience). The emphasis is thus placed on the consumer’s perception rather than

on the staff’s normative delivery of service. By rigorously applying the established statistical methods, the

analysis of collected data will shed light on the customer’s perceptions and level of satisfaction, and will help

in prioritizing the important elements, thus enabling the business to improve those high added value

elements that will have maximum impact in the eyes of the customer.

The graph below shows where the “traditional” mystery shopper and “real consumer” mystery shopper

should be used.

The participatory observation method goes back to early anthropological studies. The mystery shopper

concept per se is more than 80 years old and was very popular in the sixties. From the 1940s to the 1960s,

banks used mystery shoppers to monitor the integrity of their employees. At that time, there were few

mystery shoppers, and they followed a very structured approach (a list of binary norms of the yes/no type) to

collect and assess specific information on the performance of a specific service. The results of mystery

shopper evaluations were then used to meet three objectives:

1) As a diagnostic tool enabling the identification of weak points in the delivery of a service;

2) Penalize service personnel based on unsatisfactory mystery shopper evaluations;

3) Assess the competitiveness of an organization’s service by comparing it to certain competitors in the same industry.

In most organizations at that time, managers gathered information, interpreted it and intervened with staff

to show them what should be improved in order to meet management’s expectations. In that process,

employees were not very involved, and deemed this evaluation process as inappropriate, punitive, inefficient,

non-representative and distorted by observation errors, etc. In addition, due to poor communication

between management and front-line staff, the latter often misunderstood performance expectations.

Throughout the 1970s, mystery shopper programs were mainly used in the financial, fast food, and hotel

industries. Notwithstanding the fact that mystery shoppers largely remain focused on formally verifying

conformity with standards, we observe the emergence of the first objectives with a more positive connotation

such as the building of team spirit (employee mobilization) and the association of program performance to a

staff reward system.

By the mid-1980s the mystery shopper methodology begins to show signs of fatigue. As they are often

recognized as false customers (e.g. a single mystery shopper, without children, dressed in a formal manner

evaluating the service of a business selling children’s shoes, or else the same mystery shopper often

evaluating the same businesses), they are progressively replaced by satisfaction surveys. In fact, we observe

a trend towards more concern for consumers, as the development of a stronger customer relationship is

deemed to provide a competitive advantage to the business. Many researchers at that time agree on the

importance of customer satisfaction and its impact on the survival of businesses.

However, at the beginning of the 21st century, managers recognize that customer satisfaction surveys are

unable to provide the detailed information required to identify the strengths and weaknesses of a given

service’s processes, since respondents only have an overall view of the service. Thus, achieving service

excellence through customer focus no longer relies on satisfaction surveys. Managers recognize that the

focus is not only about achieving excellence in service delivery, but rather in achieving excellence in “customer

experience”. Organizations must therefore shift the focus on consumer expectations and on the

implementation of a set of methods to measure the customer experience. It is at this point with the full

development of the Internet, that “consumer” type mystery shoppers emerge to meet this demand through

the simultaneous data collection of facts, and their perception of such facts.

To ensure the reliability of this new type of mystery shopper who performs very few evaluations (3 to 5 per

year compared to several tens or even hundreds annually in the traditional approach), firms rely on a careful

selection (memory tests, writing capacity, motivation analysis, etc.) and on an efficient training of mystery

shoppers, in addition to a continuous, credible and representative recruitment of new mystery shoppers. The

importance of having real consumers with a neutral attitude, and performing only one business assessment

per year is emphasized. Efficient training requires the implementation of a clear script outlining questions to

be asked as well as the behaviours to be adopted. The whole process should be performed in total anonymity

of the mystery shopper as the latter may well return to the establishment being evaluated.

Today, the mystery shopper (traditional and consumer types combined) is a worldwide industry worth several

billion dollars. Indeed, the interest for mystery shoppers seems well established, particularly in the retail

industry. This interest is confirmed by the fact that the mystery shopper industry in North America alone has

generated revenues estimated at 1.5 billion US dollars in 2015 (Mystery Shopping Providers Association,

2016). “Real consumer” type mystery shoppers participate in motivating local teams and are no longer simple

surveillance mechanisms. They have become an analytical, motivational and strategic evolution tool.

According to the author, it has become a tool used in all sectors with a commercial nature (franchise network,

hospitals, administrative services, transportation, etc.). An increasing number of managers have shifted from

a disciplinary to a supportive approach in employee relationships, for instance through more continuous

feedback. However, it is difficult for managers to find an approach that is credible with employees. Thus,

many call upon “real consumer” type mystery shoppers. The authors underline the importance of feedback

to employees to capture the relationship between what they do and what customers perceive they are doing

for them.

CUSTOMERSATISFACTION

CUSTOMER EXPERIENCE

STRATEGY

CONFORMITY

4Customers’

expectations

3Customers’perception

2Business

performance

1Standards

set bymanagement

THECUSTOMER

EXPERIENCE S Q U A R E

THE CUSTOMER THE BUSINESS

5. Disadvantages

6.2 Communication

penses.

should be used.

for them.

CUSTOMERSATISFACTION

CUSTOMER EXPERIENCE

STRATEGY

CONFORMITY

4Customers’

expectations

3Customers’perception

2Business

performance

1Standards

set bymanagement

THECUSTOMER

EXPERIENCE S Q U A R E

THE CUSTOMER THE BUSINESS

5. Disadvantages

6.2 Communication

penses.

should be used.

for them.

5. Disadvantages

6.2 Communication

penses.

should be used.

for them.

5. Disadvantages

6.2 Communication

penses.

should be used.

for them.

CONSIDER THE OPTIONS

4. Advantages

The mystery shopper methodology has many advantages compared to other methods, regardless of the type

used. Indeed, it provides a clear and specific picture of the service received (traditional and consumer mystery

shopper approaches) as well as of the perception of the service by the real consumer at a given time

(consumer type mystery shopper approach only). Both approaches also enable a comparative analysis of

certain competitors. Finally, they make feedback to employees possible; this feedback can be associated to a

reward and/or consequence according to the performance.

The Advantages of the Traditional Approach

Notwithstanding the fact that the “traditional” mystery shopper approach is in decline in many industries, it

remains practical when one does not wish to measure the real consumer perception and when no qualitative

feedback is required (e.g. measurement of the normative service delivery in binary mode yes/no without

comments). Therefore, it effectively measures conformity with processes. It is also a less costly approach,

both for the business purchasing the service as well as for the firm providing the service, providing the

evaluations are performed in large urban areas. Indeed, as the number of mystery shoppers required to

perform the evaluations is very small (one same mystery shopper can perform several evaluations for the

same business in each measurement period), recruitment efforts are less demanding. However, it should be

noted that this financial advantage is rapidly lost when the evaluations are performed in non-urban settings

where the service provider normally has to bill travel expenses to compensate for the limited number of

mystery shoppers. Finally, it should also be noted that this type of approach has the advantage of measuring

a larger number of elements since the measurement of qualitative information is very limited if not totally

absent.

The Advantages of the “Real Consumer” Approach

Without a doubt, the main advantage of the “real consumer” type mystery shopper methodology is the

possibility of measuring in real time the consumer’s perception of the service offered i.e. the measurement

of the consumer’s real experience. In addition, this approach enables the measurement of 5 major indexes of

the consumer experience, namely:

• The average customer experience score;

• The customer experience consistency score (see Insights Serie #2 for more details);

• The gap between business performance and the expectations of its customers;

• The customer loyalty score (the “Net Promoter Score” (NPS), see Insights Serie #1 for more details);

• The moments of truth of the customer experience.

In addition to the aforementioned main advantage, it should be said that since a different consumer

performs each evaluation, they are always carried out much more rapidly. Finally, the quality and richness of

the qualitative information provided by the consumer type mystery shopper is also a substantial benefit.

Indeed, studies have shown that a questionnaire filled out by a consumer type mystery shopper consistently

included more comments and that these were longer (more words per comment). Therefore, programs

implemented following this approach constitute powerful management tools enabling business owners as

well as their staff a better understanding of their customers’ expectations.

5. Disadvantages

The total number of evaluations per point of service assessed is the main weakness of the mystery shopper

methodology, no matter the type of approach. Indeed, despite the fact that the global sample size within the

measured network may be statistically interesting, it is often difficult to perform statistical analyses that

would be significant at per point of service. Consequently, due to the limitations of this method, many

organizations strive to increase the frequency of evaluations in order to be able to provide better feedback

(some go up to 10 per month). Costs per evaluation are higher for a mystery shopper program than for a

survey, but it is a much more profitable method to collect reliable information for certain managerial

decisions.

The Disadvantages of the “Traditional” Approach

Obviously, the impossibility of measuring the real customer experience is the principal drawback of the

“traditional” mystery shopper methodology. It does not allow for the measurement of the main real customer

experience aforementioned indexes. Consequently it is only the process that is being evaluated and not its

results. Also, studies have shown that this type of mystery shopper is much more easily identifiable, since he

is not representative of the business being evaluated, or because he frequently evaluates the same business.

The quality and richness of the qualitative information provided by this type of mystery shopper is another

significant disadvantage. Users of this approach often complain of the paucity of qualitative data due to the

fact that mystery shoppers must often fill out too many questionnaires each week, and even in one evening.

Finally, it should be mentioned that this approach simply cannot be used in cases where a real consumer is

required (e.g. a same mystery shopper cannot open several bank accounts, or subscribe to several sports

centres within a network).

The Disadvantages of the “Consumer” Approach

Two disadvantages of this approach are worthy of mention. Recruitment is the first one. Indeed, firms using

this method must constantly recruit and maintain a database of credible mystery shoppers with different

profiles (tens, hundreds and even thousands) to ensure adequate representativeness and to avoid the

duplication of missions or too many evaluations performed by the same mystery shopper. This can often be

difficult and costly.

In addition, it is necessary to carry out a careful selection and to provide an effective training to a large

number of mystery shoppers. Finding the right consumers based on the anticipated scenario or scenarios,

ensuring that they have the proper motivation (trying not to find faults) and providing an effective training,

constitute major challenges.

The second disadvantage of this method is the maximum amount of information gathered by this type of

mystery shopper. Indeed, often dealing with several questions or wishing to make detailed comments, users

of this method occasionally complain about the lack of quantitative information (about 40 questions, rather

than 80, 100 or even 120 questions or more). However, it is reported in the literature that shorter

questionnaires produce better results in mystery shoppers programs in general.

TAKE ACTION

6. Methodology to Successfully Implement a Mystery Shopper Program

There are certain problems concerning the

implementation of the mystery shopper program,

which should be noted. First, it should be said that

humans fear being evaluated. They fear being

“labelled” as not doing their work properly on the

basis of a negative evaluation from a mystery

shopper. As a result, when this method is poorly

implemented, it often leads to confrontations

between franchisees and employees, who will do

their utmost to sabotage the program and lead to its

withdrawal.

The literature first states the importance of establishing a structured process for the establishment of a

mystery shopper program. Indeed, several businesses recognize the importance of the customer experience,

but without allocating the proper resources. Marketing, operations or human resources departments often

implement these programs. Therefore, it is important to set up a customer experience committee constituted

of several employees from the business’ various departments headed by a decision maker, whose unique

responsibility will be to improve the business’ profitability through the customer experience. It is also highly

recommended that one or several employees (franchised, district manager, manager etc.) sit on this

committee at least occasionally if not on a permanent basis. Employee participation is considered essential

to the success of the program, at each step of the program’s development. This prevents employees from

perceiving the program as one of headquarters’ “weapons”, and from viewing it as a threat, a spying

technique or a disciplinary tool.

6.2 Communication

Next, it is very important to adopt a transparent attitude towards the entire staff regarding the program.

Everyone should be informed, from the start, of the existence of the mystery shopper program and to

impress upon each one what is expected of them. Employees need to know the points on which they will be

evaluated and on the business’ standards. Consequently, it is important to clearly spell out the program’s

objectives and ensure that all employees are aware of the assessment tool and the expected standards prior

to the arrival of the first mystery shoppers. Often, a speech during an annual event used as an “official launch”

of the program is successful.

An important point in the success of the mystery shopper program: the program must be constructive and

not punitive! As mentioned earlier, employees fear being evaluated. It is thus of utmost importance to

frequently remind them that the program is not designed as a disciplinary measure, but to improve the

overall customer experience. The mystery shopper program must indeed be used to establish a diagnostic of

the customer experience, and not as an individual staff assessment (it is possible to conduct targeted

evaluations to meet staff assessment needs, but these should be performed outside the yearly evaluation

and management of the customer experience program). Studies have shown that employee reactions to the

approach is sometimes negative at first, but that their behaviour always changes when the program is

managed in a positive and constructive way. It is also much easier to implement a “consumer” type mystery

shopper program since the objective is not to measure what is done for customers but rather what

customers perceive as being done for them. This approach is seen as much less threatening by the assessed

employees..

Psychometric properties of the mystery shoppers’ questionnaires and scenarios also play an important role

in the success of an evaluation and management of the customer experience program. The questions need

to be well formulated so as to avoid biasing the judgement of respondents (e.g.: how bad was the

welcoming?) It is important to ask specific questions to elicit clear and complete information from the mystery

shopper. Also, the scenarios must be relevant and mirror the natural transactional behaviour of the

consumer (hence the advantage of a program with real consumers). This will contribute to the reinforcement

of the credibility of the program among the staff.

It should be mentioned that once the evaluations have begun, employees will try to identify the mystery

shoppers, thus the importance of using different ones each time and to ensure that they are credible and

representative of the business being evaluated. In addition, completed evaluations must be quickly

submitted to the people involved (front-line employees, managers, franchisees, etc.). Some businesses share

the information on a bulletin board in a staff room, others meet employees in small groups, etc. It is

very important to depersonalize the evaluation so that assessed employees and mystery shoppers cannot be

recognized. It is also important to reinforce the positive points and to take action on points that need

improvement. The use of customized “dashboards” for each user level is highly recommend so that progress

along the program’s main indexes can be tracked in real time. In this respect, it is also important to establish

comparisons within one’s own network and not only with respect to individual results. Finally, it is worth

mentioning that the frequency of evaluations also influences the credibility of the program: the more regular

the mystery shopper visits, the more employees are aware and sensitized and the more the program

becomes an efficient operational management tool.

After several evaluations, a report should be produced and submitted to management. It is worth reminding

that results should be interpreted at an overall level and that the anonymity of employees must be protected.

Observations must be written in an appropriate manner, they should also be accessible and communicated

to the appropriate people. Next, the results must be shared with operators (and employees if possible)

insisting on the evolution through time, strengths and weaknesses, actions taken and those that were

successful, the comparison with similar points of sale (competition), etc.

In order to maximize the success of the program, training programs should take into consideration the

results of these evaluations, highlighting the elements deemed important by the real consumer and for which

the network was seen as less performing. Front-line employees should be trained as well as immediate

managers and franchisees.

Studies show that employee acceptance of the mystery shopper program is crucial if we expect them to

accept the results. It is therefore important to spare no effort to be well structured and transparent in

informing all parties of the program on its role, the assessed elements and the results of the evaluations. It is

also noteworthy that such programs have considerable impact at the beginning, but that impact diminishes

over the long term, hence the importance of updating the program year after year.

THE “CONSUMER” MYSTERY SHOPPER: IT’S PROFITABLE!

Quality of the customer experience is a

determining factor in industries where

products are very similar to one

another. If the experience provided is

not up to expectations, customer will

not hesitate to go see the competitor.

This is the reason why mystery

shopper programs, especially those

using real consumers, are very popu-

lar. Moreover, when successfully

implemented, these programs

contribute significantly to the

long-term survival of the business.

Over the last 15 years especially,

“consumer” type mystery shopper programs have clearly demonstrated their value. Indeed, studies have

shown that the most performing businesses, which maximize the potential of their “consumer” type mystery

shopper program, see their profitability increase by 1% to 4% per point of service.

However, it remains that no method is perfect. It would be wrong to claim that the mystery shopper

methodology is self-sufficient. While it is true that it enables a more specific analysis of the services provided,

it must be used in conjunction with other methods, such as surveys, focus groups, or even with detractor

management programs.

List of relevant references

Websites:• Mystery Shopping Providers Association (2016). http://www.mspa-global.org/

Books :• Owen, Richard; Brooks, Laura L. (2009). Answering the Ultimate Question: How Net Promoter can Trans-

form your Business. Hoboken, NJ.: John Wiley & Sons Inc. ISBN 978-0-470-26069-2. • Parasuraman, A., Zeithaml, V. A., Berry, L. L. (1988). « SERVQUAL: A Nultiple-Item Scale for Measuring

Consumer Perceptions of Service Quality ». Journal of Retailing. • Shaw, Colin; Ivens, John (2002). Building Great Customer Experiences. New York, N.Y.: Palgrave Macmil-

lan. ISBN 978-0-230-55471-9. • Reichheld, Fred (2001). Loyalty Rules!: How Today’s Leaders Build Lasting Relationships. Boston, Mass.:

Harvard Business School Press. ISBN 1-57851-205-0. • Reichheld, Fred (2006). The Ultimate Question: Driving Good Profits and True Growth. Boston, Mass.:

Harvard Business School Press. ISBN 978-1-59139-783-0.• Reichheld, Fred; Markey, Rob (2011). The Ultimate Question 2.0: How Net Promoter Companies Thrive in

a Customer-Driven World. Boston, Mass.: Harvard Business Review Press. ISBN 978-1-4221-7335-0.

4. Advantages

absent.

5. Disadvantages

decisions.

TAKE ACTION

withdrawal.

6.2 Communication

employees..

4. Advantages

absent.

5. Disadvantages

decisions.

TAKE ACTION

withdrawal.

6.2 Communication

employees..

4. Advantages

absent.

5. Disadvantages

decisions.

TAKE ACTION

withdrawal.

6.2 Communication

employees..

4. Advantages

absent.

5. Disadvantages

decisions.

TAKE ACTION

withdrawal.

6.2 Communication

employees..

4. Advantages

absent.

5. Disadvantages

decisions.

TAKE ACTION

withdrawal.

6.2 Communication

employees..

“evolved” mystery shopper - lanla · by the mid-1980s the mystery shopper methodology begins to...

Documents