Getting Strategic with Digital Intro to Analytics

Research & Product Management: How I Learned to Stop Worrying and

Love the Customer

Who am !?• ME = James Klassen• TALKING ABOUT = Research and putting the

customer 1st!• If you have a question, please just yell!

• Background in market research, online marketing, SEO, social media, analytics… blah blah blah

• Digital Product Manager @ AMA Insurance• In addition to product management, also lead a

small, elite team of marketing hackers

Questions

• Who regularly uses analytics/research to learn about their customers? Online specifically?

• Who works in digital/ directly with people in digital?

I have a case study!

First, some results

Which one converts better?

Which one converts better?

50%increase

Which one converts better?

Which one converts better?

75%decrease

Some context…

What are we trying to learn?• Big question: who are the people using our

website?– What do they want, how do they want it, what do

they already know?– How can we organize our site to help them find

what they’re looking for?– Because it’s insurance, are they looking for the

right things?• We want to help educate about insurance

policies/coverage/etc.– What functionality should we add/delete?– How can we design our site better?

Our path• Market research: understand broad strokes• Online survey 1: User intent

– We had no idea, clean slate• Traditional market research gave us direction, but we wanted to

validate online: were those people already talking to us?– Tried to keep in mind that the state of our current site was

going to influence responses• Online survey 2: Service preference

– Offline vs. online – many services not available online– How do people want to interact with us?– What type of insurance are they looking for?

• Online survey 3: Current customer?– Already insured? Already a member?

Our path

• So, we decided to change our main-page templates to reflect what we learned

• For /insurance/ and central products:1. Make it easy to accomplish high-priority

tasks2. Make it more obvious that you’re on an

insurance-related page3. (/insurance/) make it easier to access

second-level self-service

Thus…

And…

Successes are easy… so about that grey box...

• Initially:– Made the change without testing… OBVIOUSLY

THE NEW DESIGN IS BETTER– Saw big increase in bounce rate, decrease in

conversion rate– This was basically a before/after test

• Ok, we need to learn more:– Run it A/B style in a more controlled environment

(maybe external factors were impacting the results… this is what I was hoping for)

2nd test: A/B

2nd test: A/B

70%decreaseinquotes

2nd test: A/B

194%increaseinbouncerate

The horrible grey box

• Still really hate the grey box

• There must be something wrong inside of the 2-box setup– Let’s try different headings

3rd test: A/B 3 treatments

3rd test: A/B 3 treatments

58%increaseinquotes

133%increaseinquotes

The horrible, terrible grey box• Simply removing 1 of the boxes, led to 133%

increase in quotes…• Unfortunately, it looks like 2 boxes are just a

losing combination

• SIGH

• So we switched back to the original grey box…

The horrible, terrible, ugly grey box

• BUT I still think we can make it better!

• So let’s try making the single box better!– More visible text– More visible box colour– Remove overall page header

4th test: A/B 4 treatments

4th test: A/B 4 treatments

INCONCLUSIVE!!!

So what’s the outcome ofJames vs. the grey box?

So where are we now with the grey box?

• Slowly coming to terms with it

• We’re still learning and looking for ways to improve– “best practice” doesn’t work

• The fact is: it performs well!

So where are we now with the grey box?

• What we should all learn:– This stuff doesn’t always go according to plan– Users don’t always want what you want them to

want– Testing isn’t magical, it’s actually more work– Check your bias – even THIS IS OBVIOUSLY

better doesn’t always work– The same approach in 2 different contexts might

not work… and you might not realize there are different contexts!

• Can’t set it and forget it... Once you start down the dark path...

Some common mistakes• Not using the appropriate research tool for the

question• Making bias or stats-related errors in A/B tests

– Chasing 100% confidence– Not determining the parameters of your experiments– Not accounting for environmental factors like

seasonality• Putting too much weight into results• Testing too much, too big, too small• Testing things you can’t change

Research tips• If you don’t have research/ stats experience, get help

– Ask your internal business intelligence and market research team(s)

– There are loads of great resources online• Test a lot, but know you can test too much

– Easy to test ”too big” or “too small”• Get to know your analytics platform• Research isn’t just a/b testing

– Don’t forget about surveys, focus groups, analytics, internal interviews (retail, product experts)

• Think about using Agile– Puts the customer at the centre and that’s really what

we’re talking about here

Questions?

Thanks!James Klassen

Digital Product ManagerAMA Insurance

AMAInsurance.caca.linkedin.com/in/jrklassen

@forsoothed