vab dwivedi - real world e-commerce and customer experience optimization practices from dell.com

Driving Customer Experience

Through Testing, Research &

Analytics

Vab Dwivedi

Dell | Director, Testing & Research

@vabdwivedi

linkedin.com/in/vabdwivedi

2 of Y

What’s our approach?

3 of Y

Our tenets

We Are Customer Obsessed1

Act as Subject 2

Deliver 3

Guide 4

Serve 5

Be Agile6

4 of Y

Our tenets

We Are Customer Obsessed1

Act as Subject 2

Deliver 3

Guide 4

Serve 5

Be Agile6

5 of Y

Our tenets

We Are Customer Obsessed1

Act as Subject Matter Experts2

Deliver 3

Guide 4

Serve 5

Be Agile6

6 of Y

Our tenets

We Are Customer Obsessed1

Act as Subject 2

Deliver Advanced Analytics In A Consumable Manner3

Guide 4

Serve 5

Be Agile6

7 of Y

Our tenets

We Are Customer Obsessed1

Act as Subject 2

Deliver 3

Guide Decision Making Through Insights And Experimentation4

Serve 5

Be Agile6

8 of Y

Our tenets

We Are Customer Obsessed1

Act as Subject 2

Deliver 3

Guide 4

Serve As A Channel of Knowledge5

Be Agile6

9 of Y

Our tenets

We Are Customer Obsessed1

Act as Subject 2

Deliver 3

Guide 4

Serve 5

Be Agile6

10 of Y

Our tenets

We Are Customer Obsessed1

Act as Subject 2

Deliver 3

Guide 4

Serve 5

Be Agile6

11 of Y

Holding ourselves accountable

Influencing Agenda/Roadmap1

Delivering Actionable Insights2

Delighting 3

Managing Stakeholders4

Executing With Operational Excellence5

Driving An Innovative Vision6

12 of Y

Holding ourselves accountable

Influencing Agenda/Roadmap1

Delivering Actionable Insights2

Delighting 3

Managing Stakeholders4

Executing With Operational Excellence5

Driving An Innovative Vision6

13 of Y

Holding ourselves accountable

Influencing Agenda/Roadmap1

Delivering Actionable Insights2

Delighting 3

Managing Stakeholders4

Executing With Operational Excellence5

Driving An Innovative Vision6

14 of Y

Holding ourselves accountable

Influencing Agenda/Roadmap1

Delivering Actionable Insights2

Delighting Our Customers3

Managing Stakeholders4

Executing With Operational Excellence5

Driving An Innovative Vision6

15 of Y

Holding ourselves accountable

Influencing Agenda/Roadmap1

Delivering Actionable Insights2

Delighting 3

Managing Stakeholders4

Executing With Operational Excellence5

Driving An Innovative Vision6

16 of Y

Holding ourselves accountable

Influencing Agenda/Roadmap1

Delivering Actionable Insights2

Delighting 3

Managing Stakeholders4

Executing With Operational Excellence5

Driving An Innovative Vision6

17 of Y

Holding ourselves accountable

Influencing Agenda/Roadmap1

Delivering Actionable Insights2

Delighting 3

Managing Stakeholders4

Executing With Operational Excellence5

Driving An Innovative Vision6

18 of Y

Holding ourselves accountable

Influencing Agenda/Roadmap1

Delivering Actionable Insights2

Delighting 3

Managing Stakeholders4

Executing With Operational Excellence5

Driving An Innovative Vision6

Setting our team up for success

20 of Y

21 of Y

22 of Y

23 of Y

24 of Y

That was ‘how’…let’s look at some projects…

Dell.com Home Page

26 of Y

27 of Y

28 of Y

29 of Y

30 of Y

Iterative testing over 18 months…

31 of Y

Iterative testing over 18 months…

32 of Y

We measured EVERYTHING!

33 of Y

We measured EVERYTHING!

34 of Y

We measured EVERYTHING!

35 of Y

We measured EVERYTHING!

36 of Y

We measured EVERYTHING!

37 of Y

We measured EVERYTHING!

38 of Y

…And then some more…

39 of Y

…And then some more…

40 of Y

…And then some more…

41 of Y

…And then some more…

42 of Y

…And then some more…

43 of Y

…Look into device type performance…

44 of Y

…Look into device type performance…

45 of Y

…Look into device type performance…

46 of Y

…Look into device type performance…

Drive Big Change:A new Dell.com

48 of Y

49 of Y

50 of Y

Not just a new visual identity…

51 of Y

Not just a new visual identity…

The new Dell.com was new in every way….platform,

architecture, back end services

52 of Y

Not just a new visual identity…

The new Dell.com was new in every way….platform,

architecture, back end services

Before going live, test it to ensure a win for customers

and the business

53 of Y

Not just a new visual identity…

The new Dell.com was new in every way….platform,

architecture, back end services

Before going live, test it to ensure a win for customers

and the business

Expecting a quick win….

54 of Y

Not just a new visual identity…

The new Dell.com was new in every way….platform,

architecture, back end services

Before going live, test it to ensure a win for customers

and the business

Expecting a quick win….

… -33% revenue/visit

55 of Y

OK, no big deal…we know what to do

56 of Y

OK, no big deal…we know what to do

Let’s introduce UI enhancements

57 of Y

OK, no big deal…we know what to do

Let’s introduce UI enhancements

24 of them…through various parts in the site

58 of Y

OK, no big deal…we know what to do

Let’s introduce UI enhancements

24 of them…through various parts in the site

Recipe C will be our super recipe that will bridge the

gap when we re-run the test

59 of Y

OK, no big deal…we know what to do

Let’s introduce UI enhancements

24 of them…through various parts in the site

Recipe C will be our super recipe that will bridge the

gap when we re-run the test

… -45% revenue/visit

60 of Y

Maybe its not the UI…

61 of Y

Maybe its not the UI…

It is a new platform…

62 of Y

Maybe its not the UI…

It is a new platform…

We need re-run and this time look at the metrics that

can point to any issues with the platform

63 of Y

Performance metrics

64 of Y

Performance metrics

65 of Y

Performance metrics

66 of Y

Performance metrics

67 of Y

Performance metrics

68 of Y

Performance metrics

69 of Y

Performance metrics

70 of Y

Well that explains it…

71 of Y

Well that explains it…

Now we can work with engineering and provide

guidance on where the issues are

72 of Y

Well that explains it…

Now we can work with engineering and provide

guidance on where the issues are

Emergency sprints scheduled to address live issues

73 of Y

Well that explains it…

Now we can work with engineering and provide

guidance on where the issues are

Emergency sprints scheduled to address live issues

Re-test now….

74 of Y

Well that explains it…

Now we can work with engineering and provide

guidance on where the issues are

Emergency sprints scheduled to address live issues

Re-test now….

… +3% revenue/visit!!!

Find something small that can make an impact: Continue shopping

76 of Y

Should ‘Continue Shopping’ be smarter?

77 of Y

Should ‘Continue Shopping’ be smarter?

78 of Y

Should ‘Continue Shopping’ be smarter?

79 of Y

Should ‘Continue Shopping’ be smarter?

80 of Y

Did it work?

81 of Y

Did it work?

82 of Y

Are you sure?

83 of Y

Are you sure?

84 of Y

Are you sure?

85 of Y

Are you sure?

86 of Y

Now iterate, globalize and future proof

87 of Y

Now iterate, globalize and future proof

88 of Y

Now iterate, globalize and future proof

89 of Y

Now iterate, globalize and future proof

90 of Y

Now iterate, globalize and future proof

91 of Y

Now iterate, globalize and future proof

92 of Y

Now iterate, globalize and future proof

93 of Y

Now iterate, globalize and future proof

Rapid fire: Everything doesn’t have to be a ‘project’…

95 of Y

Cycling through order codes

vs

96 of Y

Cycling through order codes

vs+9.1% Conversion

+11.9% Revenue/Visit

+91.6% Button Engagement

-1.49% Exit rates

+.6% Compare usage

97 of Y

Adding payment methods to cart

vs

98 of Y

Adding payment methods to cart

vs+4.3% Conversion

+10.5% Revenue/Visit

+2.2% Payment Pg > Confirm Page

99 of Y

Results sorting method

vs

100 of Y

Results sorting method

vs

+6.7% Conversion

+15% Revenue/Visit

+51% Conversion for highest price

band

-64% Engagement with sort option

101 of Y

Category page filter layout

vs

102 of Y

Category page filter layout

vs-7.5% Conversion

-6.4% Revenue/Visit

-40% Engagement

-17% Conversion for

engagers

-8% Next page

103 of Y

Sequencing options for systemsBased on business rules

and availabilityHardware, Software,

Services, Misc.

vs

104 of Y

Sequencing options for systemsBased on business rules

and availabilityHardware, Software,

Services, Misc.

vs+13.8% Conversion

+15.5% Revenue/Visit

+12.7% Margin/Visit

105 of Y

Clickable tabs and ‘Add to Cart’ CTA

vs

106 of Y

Clickable tabs and ‘Add to Cart’ CTA

vs+11% Conversion

+52% Engagement w/tabs & CTAs

+13% Movement through funnel

-2% Exit rate

107 of Y

Sign in page titles

vs

108 of Y

Sign in page titles

vs

+3% Conversion

-4% Guest checkout

+3.7% Movement through funnel

+1% Authentication

109 of Y

Navigation title change

vs

110 of Y

Navigation title change

-3.5% Conversion

-5.8% Revenue/Visit

-3% Engagement

No Change Movement to

browse

vs

If you only remember three things from today…

112 of Y

Our daily mantra

113 of Y

Our daily mantra

You want iterative testing & a LOT of data1

Winners and Losers are good2

Keep it simple for your customer3

114 of Y

Driving Customer Experience

Through Testing, Research &

Analytics

Vab Dwivedi

Dell | Director, Testing & Research

@vabdwivedi

linkedin.com/in/vabdwivedi