Tim Burgess Justin Bridegan

Web Analytics Manager Senior Marketing Manager

Air Canada MECLABS

Case Study: Air Canada

The Biggest Optimization Mistake I Ever Made

Presenters Tim Burgess

Web Analytics Manager

Air Canada

Justin Bridegan

Senior Marketing Manager

MECLABS

2

3

Where I Came From…

Agile

Selling primarily to new visitors

Experimental with both software and marketing

Failure was accepted as part of the learning process

Don’t ask about the product quality

4

Testing at Air Canada

Unknown or brand new to most individuals and groups

Measurement is done through old database architecture

Company structure is separated into silos

Web analytics and optimization are positioned in IT

5

Web Analytics Maturity Model

2

1.5

2 2

2

3

Resources

Methodology

Tools Management

Objectives

Scope

Tools

1

Management

1

Objectives

0

1

Scope Resources

1

Methodology

1

Jan 2011 Mar 2012

***This has been stolen from Stephan Hammel

6

7

Before I tell you about the worst test I ever had….

…don’t feel bad for me…

• We are installing an enterprise-level testing tool

• We are gaining ground in the marketing community

• My boss still likes me (as far as I know)

8

5 steps for developing a test within Air Canada

1. Develop a strategy – where is the lowest-hanging fruit for profitability?

2. Use the MarketingExperiments concepts of Friction and Value to develop mockups

3. Find a stakeholder whose interest I will work in, and sell them on the concept

4. Set up and run the test

5. Report to stakeholders

1

2

3

4

5

9

Step 1: Strategy

• Test to improve seat selection

– Air Canada’s largest source of ancillary revenue

Develop a strategy – Where is the lowest-hanging fruit for profitability? 1

10

Use the MarketingExperiments concepts of Friction and

Value to develop mockups

Step 2: Analyze Friction and Value

2

Perceived Cost

BPY=Pc(VF>CF)

BPY=Pc[Cl+Cr](VF[Ap+Ex]>CF[Rc+Ef])

Perceived Value

WHY DO PEOPLE SAY “YES”?

11

The Original: Seat selection

The original variation is good from a navigational point of view, but does nothing to emphasize the value of “Advanced” or “Preferred” seat selection.

12

Variation A

This variation focused on bringing Value to the top of the page through bullets, while maintaining clear navigation.

13

Variation B

The goal of Variation B was to bring navigation further above the fold, while still featuring the bullet points.

14

Variation C

Answered the question, “What if they really don’t know the difference between Preferred and Advanced seat selection?” by promoting description in text form above navigation.

15

Find a stakeholder whose interest I will work in, and sell them on the concept

Step 3: Find a stakeholder

3

16

Stakeholder – Revenue Management

The group that manages pricing, loads and network and also responsible for Seat Selection revenue. Loved the idea!

17

Testing

Step 4: Set up and run the test

4 Control

Variable A Variable B Variable C

18

Step 5: Report to stakeholders

5 Report on findings

We ran the test for one week to 95% significance, with a follow-up test for three weeks, that showed that variation C was generating 48.78% more revenue than the original.

Control Variable C

19

We have a winner…

So we celebrated!

Variable C

20

One small hitch… The numbers in the months following the experiment did not reflect the

experiment results

My Mistake: By communicating the results the way I did, I led the stakeholders to believe that we would see those numbers once the test went live.

• Revenue Management did not understand how this could happen

• My boss was disappointed

• I was confused

21

So, what happened?

• Visitors on Aircanada.com are often repeat visitors

• There are many points

in the process where visitors can purchase seat selection (see the email below)

• We cannibalized on

other channels

22

Analytics data

Preferred Seat Revenue/flight order

0

1

2

3

4

5

6

7

Jul 1

, 2011

Jul 6

, 2011

Jul 1

1, 2

011

Jul 1

6, 2

011

Jul 2

1, 2

011

Jul 2

6, 2

011

Jul 3

1, 2

011

Aug

5, 2

011

Aug

10,

201

1

Aug

15,

201

1

Aug

20,

201

1

Aug

25,

201

1

Aug

30,

201

1

Advanced Seat Selection

Revenue/Order

00.5

11.5

22.5

33.5

4

Jul 1

, 201

1

Jul 6

, 201

1

Jul 1

1, 2

011

Jul 1

6, 2

011

Jul 2

1, 2

011

Jul 2

6, 2

011

Jul 3

1, 2

011

Aug 5

, 201

1

Aug 1

0, 2

011

Aug 1

5, 2

011

Aug 2

0, 2

011

Aug 2

5, 2

011

Aug 3

0, 2

011

Even though analytics data showed a change in the amount of revenue for Preferred Seat selection, Revenue Management was used to looking at its own data, which is not slice and trend-able like analytics data. Using conservative numbers the actual impact of the test was an approximate increase of 5% on Preferred Seat selection revenue.

23

The irony…

…is that this is the most successful test I have run to date, in terms of profitability.

24

Key takeaways

Remember the big picture

Manage expectations

Predict scenarios, not results

Keep testing

Remember…

The way in which hypotheses and results are presented is pivotal to building support and making optimization a long-term strategy.

25

Thank you!

Questions?