demystified cem-optimization sponsoredby tealeaf

creating an integrated view of

your online customers’ behavior customer experience management and the web site

optimization ecosystem

Eric T. Peterson, Web Analytics Demystified, Inc.

Sponsored by

Tealeaf, Inc.

45 Fremont Street, Suite 1450

San Francisco, CA 94105

415.495.8000

www.tealeaf.com

© 2008 Tealeaf, Inc. All rights reserved. 2

Customer Experience Management And the Web Site Optimization Ecosystem

Executive Summary

The single most common question executives ask about their web sites is “Why?” “Why don’t more shoppers

complete purchase transactions? Why don’t more customers use our less expensive online help? Why don’t

readers click more deeply into our web site?” The “why” question makes perfect sense, especially given the

supposed “infinite measurability” of the Internet and the widespread use of key performance indicators—

metrics that often describe in painful detail the vast number of site visitors who don’t complete transactions,

don’t use efficient online help systems, and don’t generate valuable page views.

Unfortunately the systems most commonly used to attempt to understand visitor behavior are extremely limited

in their ability to answer seemingly simple questions. Web Analytics applications from vendors like Omniture,

Coremetrics, and WebTrends all do an excellent job at processing, aggregating, and rendering the quantitative

minutiae collected from visitor click-stream data. But quantitative data alone never accurately describes subtle

human behavior; qualitative inputs are required.

Fortunately there are excellent sources of qualitative data available via Customer Experience Management

(CEM) systems like Tealeaf and Voice of Customer (VOC) systems like ForeSee Results, iPerceptions, and

OpinionLab. Both technologies gather a variety of qualitative data about exactly the same visitor interactions

recorded in click-stream data, thereby providing tremendous incremental value when used in conjunction with

pure Web Analytics systems.

These three systems—Web Analytics, Customer Experience Management, and Voice of Customer technologies—

make up the Web Site Optimization Ecosystem and collectively are able to resolve the “Who, What, Where,

When, Why, and How?” of visitor interaction on the Internet. Individually, each of these systems fills a

fundamental niche in data and information gathering:

• Web Analytics systems focus on “where and when” questions by aggregating, reporting and

visualizing large volumes of data, by reporting on online marketing and visitor acquisition

efforts, by summarizing page-level visitor interaction data, and by summarizing visitor flow

through defined multi-step processes.

• Customer Experience Management systems focus on “what and why” questions by detecting

web application issues and problems, by tracking and resolving business process and usability

obstacles, by reporting on site performance and availability, by enabling real-time alerting and

monitoring, and by supporting deep-diagnosis of observed visitor behavior.

• Voice of Customer systems focus on “who and how” questions by gathering and reporting

direct feedback from site visitors, by benchmarking against other sites and offline channels, and

by supporting predictive modeling of future visitor behavior.

Yet none of these systems exclusively answer any one set of questions. Clearly, Voice of Customer systems are

appropriate for answering “why” questions like “why did you abandon your shopping cart?” and “why did you

come here today?” while Web Analytics systems can be used to determine “what” visitors are doing. But our

central thesis is that there is a most appropriate use for each of these Ecosystem technologies.

Used properly, Web Site Optimization Ecosystem technologies are able to answer critical business questions

like:



• When online shoppers abandon shopping carts without making a purchase, did they abandon

because of performance issues, cart-related errors, usability issues, or because they were only

browsing and fully intend to either return later and complete the purchase or complete the

purchase in an offline channel?

• When customers look for help online but ultimately pick up the phone and generate a more

costly support transaction offline, did they do so because they weren’t able to find what they

were looking for or because they didn’t understand the answers they were getting? And if they

had problems using the web site, what impact does the failed support transaction have on their

overall satisfaction with the product and brand?

• When readers don’t click more deeply into the web site, generating additional page views and

ultimately CPM-based revenue, is it because the site is not responding quickly enough, because

they’re looking for something specific and not finding it, or because of how the site is rendering

in their browser or on their particular browsing device?

Experienced readers will immediately recognize that answers to the questions posed above require both

quantitative and qualitative inputs. This document describes in detail each component of the Web Site

Optimization Ecosystem, including the core competencies of each, as well as outlines specific use cases where

the combination of technologies will provide significant benefit to executives tasked with answering the

seemingly unanswerable “why” questions.

We believe that the most successful companies will recognize the value of investing in the combined

technologies we describe in this document and diligently work to leverage both qualitative and quantitative

data in their ongoing efforts to better understand visitor and customer behavior. Given the continual shift of

advertising dollars, resources, and most importantly, customers coming to the online channel, our belief is that

executives who aggressively pursue convergent validity and a more holistic view of their customers will have a

substantial advantage over those who insist on a simplified view of visitor behavior on the Internet.



Table of Contents

Executive Summary .............................................................................................................................................. 2

Table of Contents ................................................................................................................................................... 4

Introduction............................................................................................................................................................. 5

The Web Site Optimization Ecosystem ............................................................................................................. 6

Web Analytics....................................................................................................................................................................7

Primary Value Proposition: Web Analytics............................................................................................................ 8

Customer Experience Management........................................................................................................................ 8

Primary Value Proposition: Customer Experience Management ................................................................11

Voice of Customer .......................................................................................................................................................... 11

Primary Value Proposition: Voice of Customer ..................................................................................................12

Additional Inputs in the Optimization Ecosystem.............................................................................................. 12

The Integrated View of the Visitor..................................................................................................................... 12

A Sudden Decrease in Conversion Rate ...............................................................................................................14

A Dramatic Increase in Negative Feedback........................................................................................................14

Web Site Optimization Ecosystem Integration Examples............................................................................15

Track Critical Key Performance Indicators ......................................................................................................... 15

How Art.com Tracks Critical Key Performance Indicators Using Tealeaf...............................................17

Create a Robust View of Visitor Interaction ........................................................................................................ 17

How Esurance Creates a Robust View of Visitor Interaction using Tealeaf ...........................................19

Actively Monitor Critical Visitor Activities ............................................................................................................19

How Bluefly Actively Monitors Visitor Activities Using Tealeaf ................................................................ 20

Create a Feedback Loop with Online Customers.............................................................................................21

How Airlines Reporting Corporation Creates a Positive Feedback Loop using Tealeaf ...................23

Perform “Uncontrolled” Experiments ...................................................................................................................23

How a Major Online Travel Hub Aggressively Conducts “Uncontrolled” Experiments using Tealeaf ...............................................................................................................................................................................24

Conclusions ........................................................................................................................................................... 25

About the Author.................................................................................................................................................. 25

About Web Analytics Demystified.................................................................................................................... 25

About Tealeaf........................................................................................................................................................26



Introduction

The Internet is a seemingly infinitely measurable medium—every click can be recorded, every view can be

captured, and every visit can be analyzed—all in an effort to continually and automatically optimize the online

experience. Because so many user activities in the online channel can be measured, there has been a great deal

of recent interest in technology that supports measurement. Especially in a declining economy, the need to

understand which marketing efforts and technology initiatives are working and which are failing to provide the

necessary return on investment becomes paramount.

Unfortunately, the notion of “infinite measurability” and “automatic optimization” in the online channel is far

more complex than most realize. The assumption that any single application or data source will provide the

necessary depth and breadth required to deeply understand web site visitor behavior is fallacious at best and

dangerous at worst. The best technology available has been designed to perform a specific set of tasks, and

despite assurances from sales and marketing, trying to do too much with too little usually results in incomplete

analysis, unconvincing results, and recommendations unlikely to be followed.

In reality, the inputs for all optimization efforts can be classified along two axes describing the type of input

and how the input is fundamentally designed to be used. On one axis we have data and information -- data

being primarily quantitative and information being primarily qualitative. On the other axis we have measures

and actions -- measures being reports, analysis, and recommendations all designed to serve to drive actions, the

actual changes being made in the ongoing process of optimization. Each quadrant created by these axes

describes what member technologies are fundamentally designed to support: the gathering of data and

information and acting on those inputs.

Web Site Optimization Ecosystem framework, showing the relationship between quantitative and qualitative data and between measures and actions.

This document describes the measurement components of the Web Site Optimization Ecosystem and their

practical use, with a specific focus on the technology sold by the document’s sponsor, Tealeaf.



The Web Site Optimization Ecosystem

Many software vendors would have you believe that the Web Site Optimization Ecosystem is defined by the

ability to log, parse, and report on the click-stream behavior of site visitors. Nominally referred to as “Web

Analytics”, these applications are distributed by small vendors like Omniture, WebTrends, and Coremetrics and

industry giants like Google, Yahoo, and Microsoft. For the most part, the vendors would have you believe that

“all that can be known” about web visitors can be gleaned through their data collection strategy and reporting

interface. However, this is far from the truth: understanding visitor behavior is as much a function of

qualitatively determining interests and intent as it is quantifying clicks from page to page.

Fortunately there are applications designed to provide a more qualitative view of online visitor behavior

designed to report on the overall visitor experience and to quantize direct feedback given by visitors and

customers. Customer Experience Management (CEM) applications provided by vendors like Tealeaf (this

paper’s sponsor) are able to record tremendous detail about individual visitor interactions with the web site,

granular to the level of the actual links clicked and text entered but also including relevant performance-related

information to help site operators understand “what happened?” Voice of Customer (VOC) application

provided by vendors like ForeSee Results, OpinionLab, and iPerceptions take the gathering of qualitative

feedback to the logical extreme by directly surveying a sample of visitors and using responses to build complex

models of visitor and customer satisfaction with the site.

Yet Customer Experience Manegement and Voice of Customer applications on their own are no more likely to

provide a complete view of visitor behavior than Web Analytics applications. Each application plays a valuable

role in the Web Site Optimization Ecosystem. Smart business owners have already learned how to take

advantage of each application in their ongoing efforts to simultaneously optimize the online channel in order to

maximize profits while minimizing costs. The most common error made by online businesses is failure to

recognize that each of these systems exists. The real challenge is understanding the true capabilities each

system provides and where the three systems converge to create a more accurate view of visitor behavior and

customer experience.

Web Analytics, Customer Experience Management, and Voice of Customer systems are the foundation of the

Web Site Optimization Ecosystem that supports the online business’s ability to positively influence desired

outcomes. These similar-yet-distinct systems each contribute to a site operator’s ability to recognize, react, and

respond to the ongoing challenges faced by every web site owner. Fundamental to the optimization process is

measurement—the data and information gathering measures that can be transformed into recommendations for

action. When properly used, these systems allow for convergent validation—combining different sets of data

collected for the same audience to provide a richer and deeper understanding of audience behavior.



The fundamental measurement technologies underlying the Web Site Optimization Ecosystem: Web Analytics, Customer Experience Management, and Voice of Customer systems.

Web Analytics

Web Analytics systems like those provided by Omniture, WebTrends, and Coremetrics are designed to

quantify visitor actions—the number of visits to a web site, the number of pages viewed, the number of

downloads requested, the number of forms started and completed, the number of visits from a particular site

or marketing campaign, etc. The best systems support segmentation of these counts based on other criteria

contained in the click-stream data such as the count of page views from a visitor responding to a specific

campaign or the list of search terms used by visitors who had viewed a particular set of pages.

Example web analytics application from Coremetrics highlighting this class of applications ability to summarize large volumes of information.



While a great deal of information about visitor behavior can be measured through these counts, traditional Web

Analytics systems contain little information about a visitor’s motivation, why they failed to complete a critical

action, or what ramification that failure may have on the relationship between the visitor and the business. Even

complex multi-dimensional analysis fails to do anything more than highlight which combination of countable

events were more likely to create negative outcomes without providing meaningful information about the

underlying cause.

Of course, being able to quantify visitor actions is tremendously important to online business. Web Analytics

systems do an excellent job of supporting the creation and deployment of digital dashboard and key

performance indicators (KPIs) that provide business-critical views of site activity. They also do an excellent job

of providing data inputs to the marketing performance management ecosystem, a suite of related applications

designed to help online business owners manage and optimize their online marketing investment. In a nutshell,

Web Analytics systems do an excellent job of helping business owners visualize the where and when of visitor

behavior on their web sites.

Primary Value Proposition: Web Analytics

Aggregating, reporting and visualizing large volumes of data; reporting on online marketing and visitor

acquisition efforts; summarizing page-level visitor interaction data; summarizing visitor flow through defined

multi-step processes.

Customer Experience Management

Customer Experience Management systems provided by vendors like Tealeaf and others bridge the gap

between purely quantitative data collected by Web Analytics systems and largely qualitative data collected by

Voice of Customer systems. At their core, these systems are able to record information about the visitor

experience, including data about usability and business process obstacles, page performance and availability,

application errors, and granular visitor interactions to the level of the individual visitor experiences on the site.

Systems like Tealeaf provide the ability to replay the entire session, allowing analysts to see exactly what the

user saw and did during their visit to the site.



Example of Tealeaf’s replay, which shows the exact experience of individual visitors to the site—including any problems, dynamic content, and data input or output.

A common perception about CEM systems is that they provide “too much information” and the ability to

examine the experience of every site visitor is too challenging to provide benefit. Considered in a vacuum, this

assessment is undoubtedly true; any effort to determine anything meaningful about the visitor experience by

watching individual sessions and/or the gross performance of the interaction between visitors and web serving

technology will almost certainly fail.

© 2008 Tealeaf, Inc. All rights reserved. 1 0


Successful Customer Experience Management relies heavily on the ability to segment visitors into meaningful

groups and then track those groups in aggregate, an ability that fully depends on CEM’s ability to track all

visitor interactions without sampling or summarization at the point of data collection. Put another way, the “too

much information” criticism arises from a misunderstanding about the use of CEM systems—the depth of

information that solutions like Tealeaf provide is a clear benefit in the context of the Web Site Optimization

Ecosystem.

Example depth of data collected using Tealeaf, allowing experience replay for any visitor session on the site and supporting very robust session segmentation.

The ideal use of a Customer Experience Management technology is to support the deep diagnosis of issues

surfaced through either Web Analytics or Voice of Customer systems as well as through traditional mechanisms

© 2008 Tealeaf, Inc. All rights reserved. 1 1


such as customer service groups or system monitoring tools. CEM applications allow analysts to more easily

determine whether site performance and availability, the occurrence of unexpected errors, or other failures of

technology are responsible for visitor abandonment as well as the projected revenue impact of each. In the

Web Site Optimization Ecosystem , CEM applications are also most appropriate for providing real-time

alerting and monitoring, not only for site functionality issues but for business-related issues such as a dramatic

decline in conversion rate, an increase in bounce rate, or a sudden change in average order value on a retail site.

In comparison to Web Analytics systems, Customer Experience Management applications are most appropriate

for getting to the what and why behind failed visitor and customer interactions.

Primary Value Proposition: Customer Experience Management

Detecting web application issues and problems; tracking and resolving business process and usability obstacles;

reporting on site performance and availability; real-time alerting and monitoring; deep-diagnosis of observed

visitor behavior.

Voice of Customer

The phrase “voice of the customer” or VOC is usually used to describe systems that enable direct feedback from

customers, such as those provided by ForeSee Results, iPerceptions, and OpinonLab. Highly qualitative, these

systems typically survey a sample of site visitors and ask a series of questions about their intent, experience on

the site, and overall satisfaction. These systems often leverage an established methodology such as the

University of Michigan’s American Consumer Satisfaction Index (ACSI) to produce both composite and

component scores designed to identify opportunities to positively impact visitor satisfaction. Often these

systems serve a similar role as email and customer relationship management systems, gathering open-ended

feedback that can then be mined for direct feedback and associated with individual visits to the web site.

Example overview from ForeSee Results, helping target who had unsatisfactory customer experiences and more importantly how they are likely to behave in the future.

Like Customer Experience Management, Voice of Customer data is often overlooked when considering web site

measurement, often because the data collection process is considered intrusive, the small samples may be

misleading, or it is difficult to effectively align click-stream and VOC data. Like any application, if improperly



used there is potential for issues of accuracy and utility to arise; that said, the market leading vendors in the

VOC sector today all have a great deal of experience working with their customers to get reliable, accurate, and

useful data.

Voice of Customer technology should be used in concert with Web Analytics and Customer Experience

Management to develop a more robust qualitative model of visitor behavior, focusing quantitative

measurements and filling gaps regarding visitor intent, personal experience, and future behavior. And while

both CEM and Web Analytics provide some level of qualitative data, neither delivers explicit feedback from

visitors nor are based on a statistically valid and well-vetted methodology like the ACSI. Voice of Customer

technology used properly is the most appropriate component of the Web Site Optimization Ecosystem to

measure visitor satisfaction, evaluate the potential impact of site and marketing changes recommended by

data coming from CEM and Web Analytics systems, and predict future visitor behavior both online and off.

Compared to other technologies in the Web Site Optimization Ecosystem, “voice of customer” applications are

broadly appropriate for understanding the wwho and how of visitor interactions, primarily when considering

the technology’s ability to gather direct feedback from individuals and predict future behaviors.

Primary Value Proposition: Voice of Customer

Gathering and reporting direct feedback from site visitors; benchmarking against other sites and offline

channels; supporting predictive modeling of future visitor behavior.

Additional Inputs in the Optimization Ecosystem

Astute readers may note the lack of discussion about A) those technologies supporting the “Action” axis in the

Web Site Optimization Ecosystem like multivariate testing platforms, behavioral targeting networks, and

personalization engines or B) related measurement and analysis techniques like usability studies, eye tracking,

competitive analysis and reputation management.

The former group of technologies are complex enough to fall outside of the scope of this document but will be

treated elsewhere by Web Analytics Demystified.

The latter group clearly includes important information inputs, usually serving as an adjunct source providing

very small samples of highly relevant information, designed to help analysts better understand some aspect of

the visitor experience with the site, products, or brand. The authors recommend each of these techniques as

necessary in the design and optimization process but believe that primary attention should be given to the core

measurement technologies described in this document.

The Integrated View of the Visitor

The convergent validation model—one where multiple sources of data describing the same population are

integrated to increase the depth and richness of the resulting analysis—provides a foundation for the Web Site

Optimization Ecosystem. By combining qualitative data from Voice of Customer applications with quantitative

data from Web Analytics and using Customer Experience Management to bridge the gap, we are able to

create a rich integration view of visitor behavior.



As discussed previously, each of the three systems provides a slightly different view of visitors, varying by type

of data they provide and their general scope of insight. The following diagram broadly describes the

relationship between type and scope of data for these systems, although the scope of insights possible certainly

expands as each technology evolves.

The Web Site Optimization Ecosystem, showing the relationship between the measurement technologies and highlighting the Integrated View of the Visitor.

When properly implemented, all three systems sample data from the same audience providing for convergent

validation. Historically speaking, the integration of these technologies has been arduous at best and oftentimes

impractical. Fortunately in the past year the market leaders in all three categories have started working more

closely together and now support more direct data transfer between systems through APIs and application

platforms like Omniture Genesis, Coremetrics Connect, WebTrends OpenExchange, and Google Analytics’ APIs.

When combined, typically in the web analytics platform with the data sharing truly bi-directional between all

systems, an audience sample is created based on the smallest whole population available to the integration. The

limiting factor is usually Voice of Customer data owing to the fact that most survey platforms only sample small

portions of the visiting audience. Depending on what type of segmentation has been applied and how much

customer experience data is collected, Web Analytics and Customer Experience Management platforms can

also limit the size of the integrated data set.

The most important things to remember when combining these data are:

• Don’t assume statistical validity. When in doubt, consult a statistician, especially since a

perfectly valid sample in a VOC system may be inadequate when segmentation is applied in

Web Analytics;

• Don’t assume the interface supports the integration. Most companies doing this type of work

still have to experiment and learn from trial-and-error to create truly integrated views of the

visitor.



Best of breed integrations still largely rely on homegrown tools and manual statistical validation, although this

situation is rapidly changing. As each of the tool sets become more flexible and support better API-based data

exchange, Web Analytics Demystified expects integration support to become part of each application’s core

package—the rule rather than the exception.

Despite the occasional difficulty associated with creating the integrated view of the visitor, most companies

find the value immediately clear. Consider the following examples:

A Sudden Decrease in Conversion Rate

Session conversion rates on a retail site suddenly decline, as reported by Web Analytics. With only Web

Analytics, site owners are able to identify the location of the problem, likely to the level of the page where

abandonment has increased, but have little more to indicate the reasons for increased abandonment.

• Data from Customer Experience Management systems allows site owners to determine the

reason for increased abandonment. Some CEM systems (Tealeaf for example) allow operators

to replay sessions abandoned at the page identified by Web Analytics, pinpointing the true

source and business impact of the problem (i.e. a site error, a usability glitch, an inventory issue)

and shortening the time to correction.

• Data from Voice of Customer systems allows site owners to directly analyze feedback from

potential customers lost, better quantifying the impact on customer satisfaction and brand

value.

Both CEM and VOC systems provide far greater visibility into the real source of the problem. Customer

Experience Management is highly diagnostic in this case, allowing site operators to determine whether the

abandonment was audience or application related and to observe actual user sessions to look for behaviors,

trends, and commonalities. Voice of Customer is highly predictive in this case, allowing site owners to estimate

the impact of the problem on both future purchase behavior and the overall brand.

Web Analytics alone allows analysts to report “Conversion rates are down 15 percent and we’ve isolated the

problem to a sudden increase in abandonment at step three in the checkout process.” Adding Customer

Experience Management allows them to add, “The problem was apparently caused by a number of checkout

sessions that suddenly went into an error loop where visitors were asked for a critical piece of information that

our form wasn’t able to collect.” Adding Voice of Customer allows them to finish with, “Unfortunately these

customers were extremely upset, on average reporting customer satisfaction scores less than half our site-wide

average. What’s worse, 50 percent of impacted shoppers reported that they’ll never shop at our site again and

will tell their friends to avoid us as well.”

A Dramatic Increase in Negative Feedback

Customer satisfaction scores begin to decrease and are apparently tied to an increase in negative feedback.

With only Voice of Customer data, site operators are able to respond to individuals who have given poor

feedback but do not have sufficient information to uncover, diagnose, or resolve problems that customers have

experienced.

• Data from Web Analytics systems allows site operators to mine for commonalities in the

browsing paths of unhappy customers, looking for pages or groups of pages that were



common to each session. Web Analytics would potentially allow site owners to better quantify

the magnitude of the problem based on the total volume of traffic through problem pages.

• Data from Customer Experience Management systems allows site operators to find the actual

experiences of the user providing feedback, determine whether it was application errors,

performance problems, or even user confusion causing the problem, plus compare that

experience to all other sessions in order isolate the problem to a very granular level (i.e.

geographic region or technographic segment such as browser, operating system, cookie

acceptance, etc.)

Web Analytics allows for rapid scoping of the problem across all site visitors and Customer Experience

Management is highly diagnostic in nature. Without Web Analytics, site operators may respond inappropriately

to a small number of complaints that are unlikely to have any material impact. Without Customer Experience

Management, operators will likely struggle to accurately determine the root cause of the problem.

To effectively leverage the Web Site Optimization Ecosystem site owners must know what data is available in

each system and, given the inevitable overlap, have a refined understanding of the relationship between each to

prevent inevitable data conflict. Put another way, there is a tendency for companies to struggle to reconcile

similar data from disparate systems -- a struggle that more often than not becomes political. Our

recommendation is to avoid this problem altogether by carefully establishing business processes that guide

the use of web site measurement technologies.

Web Site Optimization Ecosystem Integration

Examples

While the integrated use of Web Analytics, Customer Experience Management, and Voice of Customer

technologies is still in a relatively nascent state, there are a handful of good use cases described in this

document. They focus preferentially on processes best suited for Customer Experience Management systems

such as Tealeaf, addressing each in the order we recommend to maximize the value of both individual systems

and the entire Ecosystem of technologies.

Track Critical Key Performance Indicators

Key performance indicators and digital dashboards are very much in vogue these days, especially with senior

management. Key performance indicators (KPIs) are used to simplify the Web Analytics reporting process and

provide a business-focused view of visitor behavior and online marketing efforts. Some of the most commonly

used key performance indicators include:

• Average page views per visit

• Order and buyer conversion rates

• Average cost per keyword click

Web Analytics systems like Omniture, WebTrends, and Google Analytics are all very good at generating traffic

and count-based indicators. And while the reporting infrastructure provided with most good Web Analytics is

very appropriate for the delivery of KPI reports, traffic-based indicators only describe part of the visitor



experience. Customer Experience Management systems like Tealeaf are able to provide critical CIO/CTO key

performance indicators such as:

• Site-wide application error rate

• Overall site-wide performance scores

• Percent of abandonments that include site issues

Additionally, Customer Experience Management systems are able to provide granular diagnostic KPIs for

critical visitor and site segments. These KPIs are particularly critical for managers responsible for visitor flow

through the web site:

• Form completion rates on critical pages

• Average response times on key pages (landing pages, home page, etc.)

• Checkout path error rate by segment:

Browser version

Campaign

Visitor Geography

• Percent sessions with known usability obstacles

Example key performance indicators reported by Tealeaf, covering response times, application errors, unexpected issues, and overall site usability.



How Art.com Tracks Critical Key Performance Indicators Using Tealeaf

In addition to a variety of click-stream-based key performance indicators like bounce rate, conversion rate, and

revenue per site visitor, Art.com actively monitors key performance and error rates in their regular business

health dashboard. Art.com added error and performance monitoring using Tealeaf to support more rapid

diagnosis when issues were observed in other critical site KPIs. By closely monitoring error rates across the site

and through key site processes, Art.com is able to immediately determine whether an unexpected decline in

conversion rate or revenue per visitor is likely being caused by conditions such as application errors or usability

issues.

One example of how this reporting helps Art.com answer specific “why” questions comes from a recent decline

in a critical site KPI. In the middle of ongoing customer acquisition efforts, the company’s conversion rate

suddenly dropped. Normally, this decline would often trigger the need for time consuming analysis efforts to

examine marketing campaigns, product mix, and site functionality. But thanks to Tealeaf, management was

immediately alerted to a marked increase in site-wide errors. Problem sessions were isolated and reviewed

using Tealeaf, and the root cause was quickly diagnosed and corrected, telling Art.com both why the decline

was occurring and what they needed to do about it. In certain cases, Art.com could proactively reach out and

re-engage customers who experienced errors and did not complete transactions. In other cases, site copy and

error notifications were altered to provide a more friendly flow for visitors who ran into the discovered error

conditions. In both situations, insights gained from Tealeaf enabled Art.com to recover a substantial number of

otherwise “lost” customers.

According to Michael Marston, VP of E-Commerce and Product Management at Art.com, “Without Tealeaf as

part of our Web Site Optimization Ecosystem, Art.com would be tremendously less efficient in our ability to

track and diagnose experience-related issues on the site.”

Create a Robust View of Visitor Interaction

For the web measurement professional, one common complaint about Web Analytics systems is the lack of

visibility they provide into real visitor interactions. Especially when those interactions appear to be

problematic—as is the case all too frequently—pure Web Analytics tools fall woefully short when attempting to

describe exactly what went wrong during the visit. Unfortunately this lack of visibility oftentimes leads to an

odd form of “analysis paralysis”—an organizational inability to make a decision because there is simply not

enough of the right kind of data to guide the decision making process.

The need for deep-dive analysis is perhaps the number one reason that Customer Experience Management and

Voice of Customer systems must be considered as part of the Web Site Optimization Ecosystem: During click-

stream analysis you are inevitably left asking yourself the important questions “why did so many people

abandon the process?”, “why exactly did the problem occur?” and “who did this affect?” If you’ve carefully

integrated Web Analytics and Voice of Customer systems you’re far more likely to be able to answer the “who”

question. But your ability to answer “why” questions will be largely limited by your existing survey design—you

can’t answer questions you’re not asking.

Fortunately Customer Experience Management technologies are designed to support very deep analysis of

visitor behavior on the site, going so far in the case of Tealeaf to be able to replay entire visitor experiences,

errors and all. Full-experience replay to the level of the individual, combined with powerful search functionality



that allows analysts to determine exactly how many visitors experience the same type of problem, allow site

owners to very quickly diagnose failed interactions and answer questions like:

• If the problem was on a page with a form, did the failure have something to do with the

information the visitor was entering into the form fields? If so, what were they entering?

• If the problem had to do with a dynamically generated offer or some other dynamically

generated content, was the problem isolated to a single offer or type of content? If so, which

offer or content?

• If the problem occurred on a page relying on a “Web 2.0” object such as a Rich Internet

Application (RIA) generated in AJAX, Flash, or Flex, was there a common usage pattern that

precipitated the failure? If so, what?

Example AJAX application being tracked using Tealeaf to record and replay each individual event on a single, HTML page.

The best Customer Experience Management applications available today are by-and-large far better at

measuring Web 2.0 technologies like AJAX, RSS and XML than most commonly used Web Analytics

applications. Because CEM systems are integrated into web deployment architecture somewhere between the

client and the server, their ability to view both intra-page activities (as in the case of RIAs) and non-human user

agents (as in the case of RSS feeds and XML) is different and arguably superior. This level of granularity is

important to creating a robust understanding of visitor interaction on a site.



How Esurance Creates a Robust View of Visitor Interaction using Tealeaf

Esurance, a prominent online insurance provider, used Tealeaf to diagnose and discover a major cause of failure

for their customers. Web Analytics systems identified a high level of abandonment at a critical step in the

insurance quoting process but were unable to resolve the problem any further given the amount of information

being requested on the page in question. By studying a segment of customer experiences identified as having

abandoned at this step in the process, Esurance observed people struggling to provide their vehicle

identification number in the format the online application expected.

The problem’s cause was simply that while many state registration forms show the VIN with spaces between

certain digits, Esurance’s site was expecting it without spaces—a classic “why” question that would be

impossible to glean using traditional Web Analytics. Without requiring complex tagging and analysis, the

solution was quickly identified and fixed using Tealeaf’s experience replay functionality. The company saw an

immediate increase in successful quote completions reflected in their Web Analytics system and, more

importantly, in the number of new customers Esurance signed up month over month.

According to Marj Hutchings, Director of Internet Operations at Esurance, “We were frustrated with the high

rate of bail-outs on this particular page in our purchase process, but we were unable to determine why

people left our Web site. When Tealeaf showed us exactly what our visitors were struggling with, we

immediately corrected the problem and dramatically improved our application completion rate.”

Actively Monitor Critical Visitor Activities

One of the benefits of the online channel is that it is always on. Sales and engagement occur independent of

staffing and operational expenses. Unfortunately, this benefit is also a curse since problems sometimes happen

in the middle of the night, leaving embarrassing issues for all to see until detected and resolved – in the best

cases the following day. To address issues of system availability and site performance, a variety of technologies

ancillary to web site measurement exist—largely synthetic measurement applications designed to simply ping

web pages and report on their status.

But sometimes business problems don’t manifest as easily as slow-loading pages or full-blown application

errors. Sometimes critical site paths appear fine when measured using simple criteria like load time and

availability but paint an entirely different picture when real customer experiences are observed. Consider the

following situations:

• All pages in a checkout path load normally but an unexpected change to the application

prevents visitors from part of the world from progressing past a critical step, instead forcing

them into a infinite (and frustrating) loop

• Search results pages load just fine, but a change to the relevancy algorithm causes totally

irrelevant results to be rendered

• All pages in a critical process load and render normally in Internet Explorer but all text fails to

render properly in non-IE browsers

• A lack of product availability is noted per normal, but the application is not directing the user

to alternate choices as designed



In instances like these, all three ecosystem applications would report some type of issue: Web Analytics would

report a decrease in conversion or click-through, Voice of Customer would likely detect a decrease in visitor

satisfaction, and Customer Experience Management would have a wealth of data to report for each problem.

But in the online channel, visitors are less forgiving when problems occur given low switching costs and the

frequent duplication of products and services.

Site operators owe it to their visitors and customers to actively monitor and quickly diagnose and correct

problems when they occur. And while all three ecosystem technologies support some level of alerting and

monitoring functionality, Customer Experience Management systems like those provided by Tealeaf are

extraordinarily well suited to provide real-time feedback against complex scenarios.

Example of an alert monitor in Tealeaf, showing the number of visitors that have abandoned a business process after running into site issues.

How Bluefly Actively Monitors Visitor Activities Using Tealeaf

As a leading online retailer, Bluefly is open twenty-fours a day, every day of the year. It is therefore essential

that the company has a firm handle on what the customer is experiencing during any given buying or browsing

session. To that end, Bluefly was particularly interested to monitor all critical processes on their website,

including their checkout process.

Through the use of Tealeaf, it didn’t take long for Bluefly officials to notice a pattern—many of the customers

who had recently abandoned their shopping cart were those who had opted to pay with BillMeLater. Instead of

simply wondering why this was happening, the company was able to rapidly identify the cause of the problem

by replaying real-user sessions. As it turned out, a large number of users hadn’t been able to complete their

transactions because they had failed to check a box indicating their agreement with certain terms and

conditions. Bluefly typically uses a red exclamation point to alert customers to required input fields, and

although that red mark was showing up next to the terms and conditions box, it was appearing in a minuscule



font, which customers were missing. The vast majority of customers who hadn’t checked the box, and therefore

could not complete their transactions, were forced to give up.

For Bluefly, the ability to immediately see what users were experiencing on the web site, along with the ability

to capture and correct the errors as they happened, proved invaluable. “We don’t want to lose one customer to

a technical problem,” says BlueFly vice president of technology Matt Raines. “If they have a bad experience — if

something goes wrong — they aren’t going to give you another chance. Tealeaf gave us the ability to detect

and resolve these issues before they became widespread.”

Create a Feedback Loop with Online Customers

Few things are less satisfying than not being able to complete a task you’ve started, especially when the

obstruction is due to a failure of technology. Web site measurement applications are often able to identify

issues that have occurred—usually by measuring abandonment in a critical site process such as booking an airline

ticket or executing a stock trade. But most often the important news is not that the abandonment happened;

rather, “why” the event happened and “who” the event happened to.

Voice of Customer applications are appropriate for helping site operators understand who was affected by

specific service problems and the resulting impact on overall customer satisfaction. But as soon as you know

who has been affected, the next important step is to quickly move to diagnose and correct the problem—both

by conducting triage with the individuals affected and by accurately reporting the nature of the problem to

internal teams assigned to maintain the failed application or process. Being able to fully understand the exact

nature of any failed visitor interaction can have a profound impact on the customer’s willingness to continue to

conduct business with your organization.

Given appropriately robust Customer Experience Management technology, individual visitor interactions can be

flagged for examination and follow-up. This information can be reported:

• Via a key performance indicator report (“percent failed purchase sessions from high value

customers”)

• As part of a digital dashboard (in a report explicitly naming known customers who have had a

transactional issue)

• From within the organization’s customer relationship management (CRM) consoles to allow

front-line resources to directly review immediately recent customer interactions to diagnose

and correct problems

Additionally, based on inputs from Web Analytics and Voice of Customer solutions, managers responsible for

the overall customer experience can use CEM technology to regularly review customer interactions with the web

site. Guided by changes in customer feedback from VOC or declines in critical click-stream key performance

indicators reported by Web Analytics, managers can spend an hour a week following the same best practice

widely used in offline call centers and “listen” to customer interactions, looking for common problems and

opportunities to improve the overall online experience.



Example of Tealeaf’s customer experience replay embedded within a CRM application used by customer service agents. By showing exactly what a customer saw or did, this capability enables front-line reps to effectively and efficiently serve the online customer.



How Airlines Reporting Corporation Creates a Positive Feedback Loop using

Tealeaf

Processing over $77 billion annually for airlines, travel agencies, corporate travel departments, railroads, and

other travel suppliers, Airlines Reporting Corporation (ARC) is the financial backbone of travel distribution.

Understanding how these organizations use their online products and services is vital to ARC for maintaining and

fostering positive relations. When users need assistance, the company delivers the most efficient, effective

service possible by providing their customer service teams with immediate access to complete user

interactions. This full-experience replay, which has become part of ARC’s customer service process, enables

reps to address customer issues immediately – resulting in a 25% improvement in the team’s First Call

Resolution Rate.

The aforementioned dialog, based on customer experience, has also extended to internal communications

between ARC’s service, development, and product teams. When resolving issues or discussing product

enhancements, ARC relies on user interactions to identify opportunities for improvement, drive requirements,

and set priorities. In fact, Tealeaf, and the capabilities that it provides, has become a corporate standard for the

delivery of all online products and services at Airlines Reporting Corporation.

Perform “Uncontrolled” Experiments

A great deal has been written about the need for controlled experimentation and multivariate testing as a

fundamental component of Web Analytics and web site measurement. The authors could not agree more:

Controlled experimentation is critical to every business’s success using Web Site Optimization Ecosystem

technologies. Most of the tools and applications described in this document are designed to help people

analyze visitor behavior—going so far as being predictive in the case of Voice of Customer. But the analysis and

predictions still need a platform on which ideas and alternatives can be tested. Experimentation platforms like

Omniture Test&Target, SiteSpect, and Google Website Optimizer fill this need well and are a required

investment for any site operator serious about being truly successful with web site measurement.

An important part of the analysis process is the ad hoc generation and testing of hypotheses regarding

problems and potential improvements and innovations on the site. In sufficiently powerful web site

measurement systems this ad hoc exploration is usually done by segmenting visitors into easily compared

groups based on some aspect of their session or experience. This session-level segmentation essentially becomes

a type of “uncontrolled experimentation”—allowing for the comparison of different groups against a set of

defined key performance indicators. “Uncontrolled experimentation” allows for the exploration of leading

questions like:

• Do we get higher-quality search-referred traffic from Google, Yahoo, or Microsoft?

• Are visitors more likely to purchase after viewing our shipping policy?

• What is the impact on customer satisfaction when a visitor cannot complete a critical site

process?

• Does site performance have any material impact on a shopper’s likelihood to purchase?

• Do application errors occur more frequently on some browsers or operating systems?



Obviously this list is nearly endless and forms the core of conducting good web site measurement and analysis.

But the ability to answer most of these questions absolutely depends on a robust system supporting visitor

segmentation. While popular Web Analytics systems provide increasingly robust segmentation capabilities, a

great deal of valuable information is contained in the document proper – error response codes, error messages,

and a whole host of application metadata that can be used to create ad hoc segments for examination. To

access this information, site operators need a CEM solution capable of recording the entire visitor transaction,

looking beyond the typical variables the site operator expects to find into the “uncontrolled” events, errors, and

visitor attributes that may be indicative of significant opportunities that would otherwise go unnoticed.

Example of how Tealeaf enables “uncontrolled experimentation” based on information gathered from the document response buffer, including application errors and other performance problems, text reporting that “flight status has changed” and the need for the user to re-enter their password.

How a Major Online Travel Hub Aggressively Conducts “Uncontrolled”

Experiments using Tealeaf

Managers at a major online travel provider spent a great deal of time trying to determine why some customers

failed to complete transactions. The working hypothesis was that certain performance and pricing related

conditions were having a substantial negative impact on conversion rates; unfortunately they had no data in their

Web Analytics system that allowed them to test these hypotheses. Given the complex nature of pricing in the

airline industry and great variation in demand observed on a daily and weekly basis driven by extrinsic factors,

it was unlikely that the company’s Web Analytics solution would be able to even begin to support the complex

multi-dimensional analysis required to determine why these transactions failed.

Using Tealeaf, analysts were able to deeply explore the behavior of real visitors using segments defined in real-

time by pricing and performance conditions. The company explored the impact of slow search results and

forced “repricing” on the visitors likelihood to complete the transaction. In doing so, analysts saw a surprisingly

high correlation between these complex conditions and abandonment – with abandonment rates often twice as

high among affected users than non-affected users. By making several relevant user experience and business

logic changes to the site, this impact was dramatically minimized, recovering several million dollars of

previously foregone revenue.



Conclusions

Throughout this document we have described the measurement and analysis tier of the Web Site Optimization

Ecosystem, the relationship between measures and actions, and the fundamental need for both quantitative and

qualitative data. While the majority of business web sites already have a pure Web Analytics system deployed

and are actively gathering quantitative click-stream data, the same cannot be said for Customer Experience

Management and Voice of Customer technologies. In our experience, few sites that do have two or three

ecosystem technologies deployed are effectively using them in combination to better answer the “who, what,

where, when, why, and how” questions.

Our prediction, bold as it may be, is that the best companies will take our guidance to heart and actively begin

to explore these necessary solutions that make up the Web Site Optimization Ecosystem and the processes

involved in successfully using these technologies together. Given the continual shift of advertising dollars,

resources, and most importantly, customers into the online channel, our belief is that executives who

aggressively pursue convergent validity and a more holistic view of their customers will have a substantial

advantage over those who insist on a simplified view of visitor behavior on the Internet.

About the Author

Eric T. Peterson, CEO and Principal Consultant at Web Analytics Demystified, has worked in web analytics since

the late 1990's in a variety of roles including practitioner, consultant, and analyst for several market-leading

companies. He is the author of three best-selling books on the subject, Web Analytics Demystified, Website

Measurement Hacks, and The Big Book of Key Performance Indicators, as well as one of the most popular web

analytics bloggers at www.webanalyticsdemystified.com. Mr. Peterson has committed much of his life to the

betterment of the web analytics community, so much so that Jim Sterne, President and co-founder of the Web

Analytics Association says "Eric's leadership in the industry in unparalleled, his devotion to the community is

legendary and his years of experience translate immediately into strategic and tactical competitive advantage

for everybody who works with him."

About Web Analytics Demystified

Web Analytics Demystified, founded in 2007 by internationally known author and former JupiterResearch

analyst Eric T. Peterson, provides objective strategic guidance to companies striving to realize the full potential

of their investment in web analytics. By bridging the gap between measurement technology and business

strategy, Web Analytics Demystified has provided guidance to hundreds of companies around the world,

including many of the best known retailers, financial services institutions, and media properties on the Internet.

For more information on Eric T. Peterson and Web Analytics Demystified, please visit

www.webanalyticsdemystified.com, email [email protected], or call (503) 282-2601.



About Tealeaf

Tealeaf is the leading provider of online customer experience management solutions. Tealeaf's CX family of

solutions provides unprecedented enterprise-wide visibility into every user's unique online interactions. This

"360-degree view" of the online customer experience enables a clear and consistent understanding of the

customer for Ebusiness, IT, customer service and legal and compliance executives and their organizations across

a wide range of vertical industries including retail, banking, travel, insurance, telecommunications,

pharmaceutical and transportation. Founded in 1999, Tealeaf is headquartered in San Francisco, California, and

is privately-held. For more information, visit www.tealeaf.com.

demystified cem-optimization sponsoredby tealeaf

Documents

voice of customer systems

pure web analytics systems

web sites

web analytics applications

voice of customer voc

stream data

quantitative data

site performance