using data to inform product decisions - cape town, 26 march '15
TRANSCRIPT
Using data to inform product decisions
Why we do need data to inform product development
ProductTank Cape Town - 26 March ‘15
What do we want?
Assumption: “Sitting on a tractor all day isn’t the best use of my time”
Assumption: My users want to spend less time on the tractor so that they can spend more time on other tasks
Question: Is there a market need for driverless tractors?
What do we want?
Hypothesis: We believe this is true if the users of our MVP spend 20% more time on the farm
Approaches: One Single Metric, prototypes, MVP, observations, market research, diary studies
How should it work?
Question: How can we encourage people to discover and configure multiple cars?
Assumption: People will be encouraged to explore multiple cars if they see nice images of cars similar to the one they have just configured
How should it work?Hypothesis: We believe that adding images will drive car discovery. We know this is true if there’s a 30% increase in the average number of cars configured per person by end of May ’15
Approaches: A/B and MVT, behavioural plan & KPIs, prototypes and usability testing Approaches: A/B and MVT, behavioural plan & KPIs, prototypes and usability testing
“Blank slate”
One Single Metric: Percentage of users per variant who configure another car
Design and sample size: Minimum of 200 conversions per page to reach “statistical significance”
BUT: I can’t learn everything through this experiment!
How is it working?
Question: Is our product / feature meeting the hypothesis?
Assumption: We believe that this feature will be used by 50% of our first time car buyers in the UK within the first month after release
Question: What is our strongest market or user segment?
How is it working?Hypothesis: We know that our assumption is correct if we see a 20% increase (on the current benchmark) in the number of UK first time car buyers purchasing a car through our site
Approaches: Usage tracking, user testing, product retrospectives and refine or reject hypothesis
Identify opportunities for product improvement or reasons for discontinuation
What can qual data tell us?Qualitative data can help us: !
Understand the why behind quantitative data !
Get insight into what people think and feel !
Learn about a product idea or prototype !
!
Data drivenA/B or multi-variate test continuously !
Focus on the “One Metric That Matters” !
Build hypothesis around key KPI !
Optimise your product based on data !
Are we making a noticeable difference?
BUT... What data cannot tellIs it a good product idea? !
Metrics do not always offer you the full picture !
Data is one of the factors that feed into a decision !
We typically do not own all product decisions
Data informedData is one of the factors to consider !
Focus on the questions that you want answered !
You cannot replace intuition or creative ideas with data !
Assess impact on relevant areas
5 things to be mindful ofFocus on asking the right questions !
Data can’t replace intuition !
Be clear on hypothesis, sample size and timings !
Build and launch with data in mind !
Listen to the data and act accordingly!
Related linkshttp://svpg.com/assessing-product-opportunities/ !http://www.romanpichler.com/blog/goal-oriented-agile-product-roadmap/
http://vimeo.com/14999991
http://www.realityisagame.com/archives/390/wooga-follows-zynga-in-metrics-driven-game-design/
http://marcabraham.wordpress.com/2013/05/03/book-review-lean-analytics/
http://www.kaushik.net/avinash/web-analytics-101-definitions-goals-metrics-kpis-dimensions-targets/
http://marcabraham.wordpress.com/2013/09/09/some-considerations-regarding-data-driven-design/
http://insideintercom.io/the-problem-with-data-driven-decisions/
http://www.webdesignerdepot.com/2013/05/the-perils-of-ab-testing/
http://andrewchen.co/2008/09/08/how-to-measure-if-users-love-your-product-using-cohorts-and-revisit-rates/
http://codeascraft.com/2012/06/21/building-websites-with-science/
Related links!
https://marcabraham.wordpress.com/2015/03/05/what-is-guerrilla-testing/
http://www.slideshare.net/LilyDart/guerrilla-testing-for-content
https://marcabraham.wordpress.com/2015/02/14/learning-more-about-running-ab-tests/
https://marcabraham.wordpress.com/2015/02/04/book-review-thinking-with-data/
https://marcabraham.wordpress.com/2015/01/26/book-review-designing-for-behavior-change/
http://www.simplypsychology.org/qualitative-quantitative.html
http://data.heapanalytics.com/dont-stop-your-ab-tests-part-way-through/
https://marcabraham.wordpress.com/2014/12/22/book-review-web-metrics/
!
!
!