002343.online testing vendor landscape

8/8/2019 002343.Online Testing Vendor Landscape

1/12

Making Leaders Successul Every Day

Octoe 20, 2009

The Olie Testig VeoLscpe Joh Lovett

o Cstoe Itelligece Poessiols
http://www.forrester.com/


2/12

2009, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Inormation is based on best availableresources. Opinions refect judgment at the time and are subject to change. Forrester, Technographics, Forrester Wave, RoleView, TechRadar,and Total Economic Impact are trademarks o Forrester Research, Inc. All other trademarks are the property o their respective companies. Topurchase reprints o this document, please [email protected]. For additional inormation, go to www.orrester.com.

Fo Cstoe Itelligece Poessiols

ExECuTIVE Summary

Online testing is an underused technology solution in the burgeoning eld o Web site optimization.

Numerous methodologies and testing disciplines exist, each with evidence o success in improved Web

site usability, increased relevance, and elevated revenue. Forrester surveyed 250 Web site decision-

makers to determine current levels o adoption as well as the barriers and benets o online testing. Tis

report uncovers our ndings, untangles a conusing vendor market, and helps customer intelligence

proessionals use testing to improve their online initiatives.

TabLE OF COnT EnTSOli Tsti Still Ss A Adi

Propr Oli Tsti Iprovs Bsiss

Prora

Forrstrs Dirtor O Oli Tsti

Vdors

rECOmmEndaTIOnS

Iplt Tsti As Part O A Opratioal

Pross

nOTES & rESOurCESFoeste iteviewe 14 veo se

copies o this epot.

Rlatd Rsarh Dots

Testig a Optiiztio dieetitos

novee 7, 2007

mltivite Testig age

Octoe 22, 2007

Octoe 20, 2009

The Olie Testig Veo Lscpea mte disciplie Oes The net Fotie I Site Optiiztio

b Joh Lovttwith Clto a. dot, Sesh Vittl, Eil mph

2

3

7

9
mailto:[email protected]://www.forrester.com/http://www.forrester.com/go?docid=53062&src=53637pdfhttp://www.forrester.com/go?docid=52881&src=53637pdfhttp://www.forrester.com/go?docid=52881&src=53637pdfhttp://www.forrester.com/go?docid=53062&src=53637pdfhttp://www.forrester.com/http://www.forrester.com/mailto:[email protected]


3/12

2009, Foeste resech, Ic. repoctio PohiiteOctoe 20, 2009

The Olie Testig Veo Lscpe


2

OnLIne TeSTIng STILL SeekS An AuDIence

Online testing reers to the discipline o using technology to present varied messages, images, or

promotions to consumers in hopes o determining which version resonates best or a specied

audience and purpose. Te practice is as old as dirt, but the prolieration o the Web and digital

channels is giving testing a second wind enabling it to evolve into a science in its own right.

Despite the implicit value and documented success stories, adoption among marketers remains

woeully low. Only 26% o survey respondents currently use testing on their primary Web sites.1

Marketers are still skittish about testing or the ollowing reasons:

Testing methods conuse marketers. esting relies on complex algorithms like aguchi,orthogonal arrays, optimal design, and ull versus ractional actorial. Tese statistical methods

are derived rom design o experiments (DoE) principles and have been applied to digital

marketing over the past decade.2 While some le-brained marketers deploy these methods with

great success, most dont know which techniques to use or what to make o the results, so theyavoid complex testing technologies and treat them with irreverence.

Common challenges continue to stave of adoption. Ironically, the biggest challenge citedby 48% o marketers is proving the return on investment, which testing is inherently designed

to do (see Figure 1). Yet, marketers struggle to adopt online testing because deterrents such as

prioritizing test initiatives, taking action upon test results, and understanding testing outcomes

inhibit action.

Lack o ownership muddies accountability. Online testing lacks a true home in manyenterprise organizations. echnology teams dont want it because it requires creative types

to populate ideas, marketers shy away rom it because it involves page tagging, and analyticsteams dont sign o because most are overwhelmed with their own issues. Furthermore, a urry

o market consolidation in recent years conuses vendor relationship managers.3 As a result,

ownership and accountability o online testing is obscured, thereby limiting its success.


4/12

2009, Foeste resech, Ic. repoctio Pohiite Octoe 20, 2009



3

Fir 1 Top Opetiol Chlleges Fo Testig

Source: Forrester Research, Inc.53637

Please indicate your top three challenges in operating your companys

site optimization or A/B or multivariate testing application.

We havent had any challenges in operatingour A/B or multivariate testing application

Attaining enough traffic to achieve statisticalsignificance

Creating test elements (e.g., copy, creative)

Securing funding to execute tests

Establishing testing scenarios (e.g., concepts,areas to test)

Ability to segment/target test participants

Usability of the testing interface

Coding test within administrative interface

Lack of corporate buy-in

Understanding test results

Modifying Web pages for testing

Acting upon testing results

Prioritizing testing initiatives

Demonstrating return on investment (ROI)

Base: 65 Web site decision-makers who currently use A/B or multivariate testing applications

48%

25%

25%

23%

20%

20%

19%

19%

15%

14%

14%

12%

8%

2%

Source: JupiterResearch/e-Rewards Website Spending Executive Survey (12/08)

(multiple responses accepted)

PROPeR OnLIne TeSTIng ImPROVeS BuSIneSS PeRFORmAnce

Despite low adoption and pronounced challenges, 48% o marketers surveyed who currently employ

testing technologies report increases in online conversions as a top benet. We also ound that 46%

o marketers using testing cited increased customer satisaction as another key benet. Overall, the

majority o organizations that are testing preer managing it entirely in-house, ollowed by agency

managed services (see Figure 2). Yet, analysis shows that organizations that leverage managed

testing services produce better results than those that manage in-house. Forrester ound that:

Recurring programs are successul. On average, testers deploy nearly eight distinct A/B ormultivariable tests each month. Our research showed that processes like registration and orm

completion improved with ewer tests, while overall conversions and customer satisaction

benet rom more testing (see Figure 3). Survey results also showed that running more than 12


5/12




4

monthly tests could produce diminishing returns, yet sites that test on a recurring basis exhibit

the greatest success.

Marketers prioritize testing on high-impact areas. Organizations using testing technologiestypically gain the best results when ocusing on specic initiatives such as: streamlining usability,

increasing relevance, or elevating prots (see Figure 4). Heavily trafcked pages or specic

campaign landing pages are eective places to start optimizing Web sites. Why? esting relies

on visitor trafc to determine which test scenarios (sometimes reerred to as recipes) are most

eective and thus requires a certain volume o trafc to reach statistical signicance.

Employing a testing partner pays dividends. While most organizations use sel-service testingmethods ollowed by managed services, our research reveals that organizations working with

managed service vendors, agencies, or consulting partners achieve greater li in most categories

(see Figure 5). While evidence shows that testing can result in improvements regardless o the

method employed, organizations that leverage a managed service model achieve greater gains.

On average these gains are two percentage points higher, which could equate to millions o

dollars or conversion-based Web sites.

Fir 2 Peee metho Fo Testig mgeet


Allow vendor to managethe entire process

2%

Work with our vendorsprofessional services

group7%

Allow external agency tomanage the entire process

9%

Work with an externalagency or consultancy

38%

Manage entirelyin-house

45%

Base: 128 Web site decision-makers who currently use or plan to use A/B or multivariate testing applications


What is your preferred method of performing site optimizationand/or A/B or multivariate testing on your public facing Web site(s)?


6/12




5

Fir 3 Test Feqec a We Site beets


Approximately how many A/B or multivariate tests (on average) do you perform each month?

Base: 128 Web site decision-makers who currently use or plan to use A/B or multivariate testing applications


0%

10%

20%

30%

40%

50%

60%

20 ormore

18-1915-1712-149-116-83512Zero

Number of tests

*Increasedregistrations

*Increasedconversions

Number of tests

*Increased conversions

*Increased registrations

5%

0%

0%

18%

13%

4%

23%

19%

57%

13%

13%

9%

19%

19%

9%

10%

23%

4%

2%

7%

0%

1%

0%

0%

9%

7%

17%

Did you know?

Testers deploy an average of

7.6 tests each month.

What are the benefits your company has realized through siteoptimization or A/B or multivariate testing?*


7/12




6

Fir 4 aes O Focs Fo Olie Testig


Off-site display advertising

Product cross-sell/upsell

Product placement/merchandising

Internal promotions/offers

Segment targeting

Site or commerce search results

Email marketing

Landing pages

Multistep processes (e.g., checkout,booking)

Forms (e.g., registrations, surveys)

Site navigation

Site design 66% 19%

65% 21%

56% 29%

52% 20%

59% 23%

56% 20%

51% 23%

45% 32%

56% 21%

53% 27%

46% 25%

43% 26%

Currently use Plan to useUsability

Relevance

Revenue

Base: 128 Web site decision-makers who currently use or plan to use A/B or multivariate testing applications(multiple responses accepted)


In which of the following areas do you currently use site optimization or A/B or multivariate

testing technologies and in which do you plan to start using it in the next 12 months?


8/12




7

Fir 5 avege Lit Copiso Fo I-Hose Vess Otsoce mgeet


0%

3%

6%

9%

12%

Work with an external agency or consultancy

Manage entirely in-house*

Conversionrate

RegistratonCustomersatisfaction

Formcompletion

rate

Averageordervalue

Sitesearch-basedconversions

Landingpage

conversion

4%4%

4%4%

7%

6%

8%

3%

5%5%

7%

5%

8%

10%


Base: 22 Web site decision-makers who work with an external agency or consultancy*Base: 31 Web site decision-makers who manage entirely in-house

Averagepercentageincrease

Metrics

FORReSTeRS DIRecTORy OF OnLIne TeSTIng VenDORS

Marketers considering testing opportunities have a daunting task o understanding the testing

methodologies available, justiying development o testing programs, and selecting vendor partnersto supply the technology.

o provide insight into vendors oering technology solutions or online testing, we created a

directory o solution providers (see Figure 6). Tis is not a comprehensive list o vendors, nor is

it a ormal evaluation all inormation is sel-reported. Some vendors were not included, while

others oering technologies and services in this area, such as platorm providers, agencies, and

consultancies are not included here.4


9/12




8

Fir 6 Olie Testig Veo mti


Methodology Serviceoffering

Fullfactorial

Partialfactorial

Proprietary

Self-service

Managedservice

Fullservice

Maxymiser

Omniture (Test&Target)

Autonomy (Interwoven Optimost)

SiteSpect

SiteTuners

Vertster

Webtrends (Optimize)

Yeartestingfirstoffered

Staffdedicatedtotest

ing

Amadesa

Google Website Optimizer

Averageannualcost

Targetindustries

Conversions,landing pages,

lead generation

Retail,subscription-based

services

Financial services,retail, travel

Financial services,media, retail

Financial services,retail, travel

Agencies, media,retail

Form completions,lead generation,

retail

B2B, B2C

CPG, financialservices, media

YouTube,

Quicken Loans,Obama PresidentialCampaign

Petco, General Mills,New England

Journal of Medicine

ASDA (part of the Wal-Mart group), Santander

Bank, Virgin Media

3M Corp, TDAmeritrade, Real

Networks

Allstate, Avis,Delta Air Lines

Walmart.com,Staples.com,NYTimes.com

CBS Interactive,Facebook, Symantec

The Planet, Gore,Verizon

Microsoft

Marqueeclients

2006

2006

2004

2001

2004

2004

2005

2005

2006

30

Y Y*

Y

Y

Y

Y Y

YY

Y

Y Y

Y Y

Y

YY

Y

Y

Y

Y

Y

Y

Y Y Y

Y

YYY

Y

YYYY Y

Y

Y YY Y

Y

Y

$$

N/A (free)

55 $$$

N/A $$$

N/A $$$

30 $$

15 $$

5 $

28 $$

www.amadesa.com

www.google.com/websiteoptimizer

www.maxymiser.com

www.optimost.com

www.omniture.com

www.sitespect.com

www.sitetuners.com

www.vertster.com

www.webtrends.com/Products/Optimize.aspx

*For AdWords clients.Self Service: Refers to vendors that offer technology and supporting documentation, yet require clients to

develop, manage, and execute tests using their own internal resources.Managed Service: Refers to vendors that aid clients by working collaboratively in activities such as:1) technical support and implementation, 2) testing strategy, and 3) analysis.

Full Service: Refers to vendors that wholly support, design, manage and execute tests on behalf of their clients.Full Service offerings typically include but are not limited to: 1) technical support and implementation, 2)testing strategy, 3) test design, 4) creative development, 5) execution, and 6) analysis.


10/12




9

r E C O m m E n d a T I O n S

ImPLemenT TeSTIng AS PART OF An OPeRATIOnAL PROceSS

Olie testig is citicl copoet o efciet ketig optiiztio soltio. The

coitios o viles possiilities eqie tht ketes levege techolog to

coetl est wht woks olie i tiel e. as sch, testig st ecoe

itegl pt o e-to-e pocess. mketes seekig cotios ipoveet thogh

testig shol:

Bi with a tsti strat. mketes shol evelop iivil test pls tht pck to ovechig testig stteg. Test pls sttegies shol e socilize og

stkeholes to vet gols, epecttios, itios eoe ekig o the st test. a

goo testig pl shol icle: 1) specic hpothesis; 2) gols o the test; 3) sccess

etics. Tal th basis bor oi alorithi. m ketes who ek o testig

eoeosl elieve tht ltivite testig is the ost efciet etho. While mVT oes

epeset the picle o testig sophistictio, ketes shol stt with a/b tests to

ow the el o possiilities eoe eig etils with ltiple viles i pl. This

ppoch eles ketes to ehst low-hgig oppotities with a/b testig eoe

ovig o to oe sophisticte lgoithic optios.

choos tsti aras jdiiosl. all too ote testig eecises ecoe politicl ttleswithi ogiztio. Ogiztios tht stt with visios o ge o optiizig thei

hoe pges ote eet with sttio ile. becse the hoe pge is sch highl

visile ipott piece o olie el estte, eveoe iclig o CEO hs veste iteest i its esig. Whe sttig ot, choose test es tht e less visile t still

iti high-ipct oppotities. Lig pges o olie os ke ecellet povig

gos o testig pogs to illstte sccess eoe tkig o the HiPPOs (highest pi

peoples opiios) thei hoe pges.5

Tst draatiall difrt opts or bst rslts. We ecoe tht ketestke cotol o testig, ct qickl, epeiet o est eslts. Tke isks testig

ticll ieet cocepts the th ce vitios o ies. Wh? big ieeces

i test cocepts will evel oe ot cose setiet th ite vices, which

ote eslt i icoclsive eslts.

Istill tsti as part o a oplt sit optiizatio l. Leig ogiztios viewtestig s st opetig pocee o ew ketig iititive. Fo othe s,

this will eqie cltl chge. Cetive evelopes ee to the hilit to ecogize

tht coses hve gete s i the esig o thei We sites igitl essgig

th the o. yet, s coses pli thei olie itectios, testig povies es to

llow ketes to keep pce with shitig itetios ehvio.


11/12


12/12

Forrester Research, Inc. (Nasdaq: FORR)

is an independent research company

that provides pragmatic and orward-

thinking advice to global leaders in

business and technology. Forrester

works with proessionals in 20 key roles

at major companies providing

proprietary research, customer insight,

consulting, events, and peer-to-peerexecutive programs. For more than 26

years, Forrester has been making IT,

marketing, and technology industry

leaders successul every day. For more

inormation, visit www.orrester.com.

Australia

Brazil

Canada

Denmark

France

Germany

Hong Kong

India

Israel

Japan

Korea

The Netherlands

Switzerland

United Kingdom

United States

Headquarters

Forrester Research, Inc.

400 Technology Square

Cambridge, MA 02139 USA

Tel: +1 617.613.6000

Fax: +1 617.613.5000

Email: [email protected]

Nasdaq symbol: FORR

www.orrester.com

m k i g L e e s S c c e s s l E v e d

For a complete list of worldwide locations,

visit www.forrester.com/about.

Research and Sales Ofces

5363

For inormation on hard-copy or electronic reprints, please contact Client Support

at +1 866.367.7378, +1 617.613.5730, or [email protected].

We oer quantity discounts and special pricing or academic and nonproft institutions.
mailto:[email protected]:[email protected]://www.forrester.com/

002343.online testing vendor landscape

Documents