Accessible, Actionable, Auditable(extended mix)

The AAA standard of web analyticsAlban Gérôme, MeasureCamp Berlin, Oct 8th 2016

Accessible

• Let’s track everything from day one!

• The insights will stick out like a sore thumb like on those vendor webinars and events

• Gimme access to all reports and schedule them for email delivery too, daily, weekly, monthly and quarterly

Ignore the waterfall model

• Delivering complex implementations from day one will always fail

• You will have to start from scratch with something simple and iterate

• A working complex implementation always started small and basic

The sun always shines on TV

• The vendor will often use test data rather than real live data to sell their products

• In their test data the insights stick out like a sore thumb

• In real-life the insights will require more digging and are often less obvious and a matter of interpretation

I print them to decorate my office

• Grant your stakeholders full access to your analytics tools

• One day 2 of them will compare what they believe should show the same data and it won’t

• They will not know who is right, blame the tool and stop trusting the data coming out of it

On a need to know basis

• Restrict access to data

• If this is not in someone’s remit they have no need to access the data or scheduled reports

• Now 2 people comparing what should be identical data should belong to the same team

Data visualisation and storytelling

• Accessible also means that the data should be easy to understand

• Data visualisation helps a lot with reducing a large data set into a simple story about what happened

• Read books by Stephen Few and Edward Tufte

Fully decentralised

– Pro: Easy to implement

– Con: Multiple versions of the truth often due to inadequate training

– Solution: Restrict access, start small and progressively give access to more reports as your implementation iterates but give your stakeholders time to learn and understand

Fully centralised

– Pro: higher data quality and consistency as the data is extracted by a tool expert who also understands how the data was collected, sanitised and processed

– Con: adequately trained prople are hard to find and expensive which leads to a backlog of requests and slow delivery

– Solution: delegate and automate reporting

Hub and spoke modelHybrid of the decentralised and centralised models

• Peripheral spokes:– do their own reporting– monitor their web traffic– help the central spoke prioritise their requests with guidance on what has business value– canaries in the coalmine

• Central hub:– more time to analyse and enrich web data with other sources– do data visualisations– find good stories– train the people working in the spokes– evangelise web analytics

Accessible

• Start with a basic implementation• Iterate fast enough but not too fast so that

your stakeholders can learn at their pace• If you or your stakeholders cannot find

immediate and obvious insight that’s normal• Restrict data access on a need to know basis• Run your web analytics using the hub and

spoke model

Actionable insight

• The C-suite or your top management has decided that the organisation must be data-driven

• They hire or appoint a manager who must make recommendations but often has no web analytics experience so they hired you

• You make recommendations that might not get actioned, why?

Signal vs Noise

• The more complex your implementation the more data you will collect but most of it will have no value

• Some information may be very interesting but with little potential impact on the ROI or the Customer Satisfaction

PrioritisationPut a money figure on each of your insights, that’s the only thing the upper management understands

If it is:

• Quick, cheap and easy to implement

• Has high potential impact on the ROI or customer satisfaction or both

Go ahead with these first

Test all changes first

• For day to day traffic run A/B tests

• For campaigns use bandit algorithms

• MVT are resource-intensive, you could use that time to do more A/B or BA tests

• Thanks Guillaume at Moonpig for the tip

PDCA• Plan– Lists which assumptions you want to test, the expected

outcome, the confidence interval• Do– Configure your tests, create your controller and challenger

groups• Check– Measure the performance of your test

• Act– Implement the winner of your test if your assumption was

correct or try another assumption

Other common excuses I heard and things I wish I had answered back

• The web data is fine and dandy but there are many other data sources out there– First let’s implement recommendations based on web

data. With some quick wins there the team could grow and web data might no longer be enough

• You should spend more time balancing your learning between web analytics and the business– You learn web analytics a bit and I’ll learn the business a

bit. Deal?

Here today…• Non-managerial staff can be very specialised, hard to replace

and fail to adapt to a managerial role

• Many of us are geeks and we may be perceived as having poor social skills and poor communication skills, low emotional intelligence. All this is considered as must have skills for a manager though

• Preparing the succession for the C-suite is very important. Many started as business consultants and people tend to hire people similar to them who will speak the same language

… Gone tomorrow

• After a couple of years the business consultant will rotate to another role in the same company or elsewhere until they get a shot at the C-suite

• If they are a little too good in a role promoting them could present a risk that they might not be as good in a new role

A perverse scenario?• A business consultant hired to lead a web analytics team

will often see their role as a little but a stepping stone

• Paradoxically showing a true passion for web analytics and a desire to turn the team into a data-driven profit centre might stall their careers

• A web analyst could not be more proud to go from being a one-man-band to having a whole team and rock this joint with actionable insights

Why are heads of IT an exception?

• Typically the head of IT never was a business consultant but a lead developer or network engineer or held another technical role

• They take pride in developing and maintaining an infrastructure that supports the whole business

• They make sure that their team is staffed with people that sometimes can be hard to find and expensive too

Actionable insight

• Actionable insight once executed could result in a few big wins and prove the potential of web analytics as a profit centre

• Get more resources, build a bigger team, get a team lead, different specialities in your team

• Reporting time comes at the expense of finding actionable insights

• Recruiting is easier for you because other web analysts want to do real web analytics instead of reporting and traffic monitoring

Auditable• Your manager or stakeholders may start with a belief or a

conviction

• They will ask for a lot of data

• Pick what confirms their idea, discard the rest

• This is not what data-driven means

• This is being data-justified

Question, Data, Conclusions

• Always start with a business question

• Then extract the data that will help you answer the business question or run tests

• Finally come to a conclusion

Reproducible research

• Many conclusions will be a matter of interpretation• Document– how the data was obtained and where– how the data was sanitised, which outliers you

removed, how and why– How the data was processed to reach your conclusions

• Someone else should be able to reproduce your entire analysis and come to the same conclusions

It’s all about trust

• Web analytics is a black box for the business and therefore difficult to trust the data and any recommendations that come out of it

• When the data confirms the stakeholders’ understanding of the market, they will trust that data

• When the data suggests something unexpected they will easily challenge you, the code, the tool or all 3

Reporting outages• If reporting comes at the expense of finding actionable insights,

monitoring traffic is an even worst time consuming task

• Set up alerts, focus only on the top line metrics for monitoring, let the peripheral hub teams (hub & spoke model) do the monitoring of their own remit. They will spot and report issues like canaries in the coalmine, i.e. sooner than you

• When you deliver results with web analytics, a few occasional outages are more easily forgiven. If all you do is reporting spotting an issue 3 weeks later when you do the monthly report could get you in trouble

Look into PhantomJS

• Chromeless open-source browser that lets you execute Javascript and capture the tracking requests and more

• Checkout my deck on SlideShare about PhantomJS

• Watch the Adobe Summit 2016 presentation by Jan Exner about automating the analytics QA

Speak to your testers

• Selenium is a pretty common testing tool

• Ask your developers and try to get your QA scripts and their Selenium test scripts integrated. Get peace of mind when a release happens

Train the web developers too

• Training your users of the web analytics tool is essential but your developers need training too or get your vendor to train them

• Create proof of concept pages showing basic implementation scenarios and more complex ones

Conclusions

• The 3 dimensions, Accessible, Actionable and Auditable are similar to the triangle of fire– Heat– Fuel– Oxygen

• Remove one and you will stop the fire• Remove one and you will not be a data-driven

organisation but probably a data-justified one

Conclusions (ctnd)

If your web analytics function is focussed on reporting:1. You will be perceived as a cost-centre2. Not enough budget, your manager tasked

with cutting costs by 10%3. Salary and recruitment freeze4. Always told you lack team leadership skills so

5 years’+ experience counts for nothing

Conclusions (ctnd)

The web analytics function is supposed to be a profit-centre:1. Constantly improve the ROI and online

customer satisfaction2. Get more budget for the following year3. Offer competitive salaries and hire away the

web analysts from data-justified companies4. Leadership opportunities in your team which

you can use to break into management roles

Herzlich vielen dank!

http://www.albangerome.com@albangeromeAll pictures taken by myself at Christiania, Copenhagen, Denmark

Credits, links etc• The AAA paradigm comes from The Lean Startup by Eric Ries. I thought it was very appropriate to how web analytics should be and I adapted it

• Complex implementations always fail, this idea comes from John Gall, a systems theory expert from the 70s. His ideas also apply to web analytics and he calls this subject Systemantics

• The Hub and spoke model comes from Cult of Analytics by Steve Jackson

• Bandit Algorithms for seasonal campaigns rather than A/B testing, shouts to Guillaume Lombard @SeoGuillaume

• Visualisation: Books by Stephen Few and Edward Tufte

• Confirmation Bias and Belief Perseverance:– https://en.wikipedia.org/wiki/Confirmation_bias– https://en.wikipedia.org/wiki/Belief_perseverance

• PhantomJS:– My deck from MeasureCamp London 7: http://www.slideshare.net/AlbanGrme/using-phantom-js-to-qa-your-analytics-implementation– Jan Exner’s talk at the EMEA Adobe Summit 2016: http://summit.adobe.com/emea/sessions/summit-online/online2016/#/video/624

• One article that will get you mad once you replace « sales person » with « web analyst »: Think twice before promoting your best sales person by Andris Zoltners et al, published in the Harvard Business Review on July 6, 2012: https://hbr.org/2012/07/think-twice-before-promoting-your-best/

• Didn’t use but I also recommend a back issue of The Harvard Business Review OnPoint - Spring 2011: How to Get Results When You Are Not In Charge. This is a collection of recent articles of the HBR magazine. OnPoint is quarterly as opposed to the regular HBR magazine.

You’re talking to me?

• During your next job interviews, try asking the head of analytics these:

– What’s the last web analytics book you have read?– Name me one thought leader in web analytics– What’s your definition of web analytics?

• If they can’t answer just tell the recruiting consultant you are not interested in the role. Starve this madness right at the source