"is data science inhibiting product evolution?", steven lemm, chief data officer at...

Is data science inhibiting product development?

Steven LemmChief Data Officer @ Outfittery GmbH

steven@outfittery.de | Data Natives 20162

WHAT’S A PRODUCTDefinitiona product is something that is created and then made available to external or internal customers (eg. a software application)

steven@outfittery.de | Data Natives 20162

WHAT’S A PRODUCTDefinitiona product is something that is created and then made available to external or internal customers (eg. a software application)

Example e-commerce platformFrontend products: registration/authentication, check-out, recommender engine, article search, …Backend products: inventory control, order engine, logistic processes, payment service, payment control, …

steven@outfittery.de | Data Natives 20162

WHAT’S A PRODUCTDefinitiona product is something that is created and then made available to external or internal customers (eg. a software application)

Regis-tration

article search

article details

check-out

payment control

Example e-commerce platformFrontend products: registration/authentication, check-out, recommender engine, article search, …Backend products: inventory control, order engine, logistic processes, payment service, payment control, …

For the rest of the talk consider a simplified platform

steven@outfittery.de | Data Natives 2016

Definitionmodification of products with new or different characteristics that offer new or additional benefits to the customer.

PRODUCT DEVELOPMENT

3

steven@outfittery.de | Data Natives 2016

Definitionmodification of products with new or different characteristics that offer new or additional benefits to the customer.

PRODUCT DEVELOPMENT

3

Regis-tration

article search

article details

check-out

payment control

How to measure additional benefits?

steven@outfittery.de | Data Natives 2016

Definitionmodification of products with new or different characteristics that offer new or additional benefits to the customer.

KPI: search rank

KPI: revenue

PRODUCT DEVELOPMENT

3

Regis-tration

article search

article details

check-out

payment control

How to measure additional benefits?

steven@outfittery.de | Data Natives 2016

Definitionmodification of products with new or different characteristics that offer new or additional benefits to the customer.

Regis-tration

article searchVer. A article

detailscheck-

outpayment control

KPI: revenue

article searchVer. B

KPI: search rank

PRODUCT DEVELOPMENT

3

How to measure additional benefits? A/B testing!

steven@outfittery.de | Data Natives 2016

Definitionmodification of products with new or different characteristics that offer new or additional benefits to the customer.

Regis-tration

article searchVer. A article

detailscheck-

outpayment control

KPI: revenue

article searchVer. B

KPI: search rank

PRODUCT DEVELOPMENT

3

How to measure additional benefits? A/B testing!

steven@outfittery.de | Data Natives 2016

Example smart payment control

Regis-tration

article search

article details

check-out

payment control

WHAT CHANGES WITH DATA SCIENCE?

4

ext. solvency score

payment control

steven@outfittery.de | Data Natives 2016

Example smart payment control

Regis-tration

article search

article details

check-out

payment control

WHAT CHANGES WITH DATA SCIENCE?

4

ext. solvency score+ cust. engagement

steven@outfittery.de | Data Natives 2016

Example smart payment control

Regis-tration

article search

article details

check-out

payment control

WHAT CHANGES WITH DATA SCIENCE?

4

ext. solvency score+ cust. engagement

Conversion

PnL

steven@outfittery.de | Data Natives 2016

Example smart payment control

Regis-tration

article searchVer. A article

detailscheck-

outpayment control

Revenue

article searchVer. B

search rank

WHAT CHANGES WITH DATA SCIENCE?

4

ext. solvency score+ cust. engagement

steven@outfittery.de | Data Natives 2016

Example smart payment control

Regis-tration

article searchVer. A article

detailscheck-

outpayment control

Revenue

article searchVer. B

search rank

WHAT CHANGES WITH DATA SCIENCE?

4

ext. solvency score+ cust. engagement

steven@outfittery.de | Data Natives 2016

Conversion

Example smart payment control

Regis-tration

article searchVer. A article

detailscheck-

outpayment control

Revenue

article searchVer. B

search rank

WHAT CHANGES WITH DATA SCIENCE?

4

ext. solvency score+ cust. engagement

steven@outfittery.de | Data Natives 2016

Regis-tration

article details

check-out

payment control

article searchVer. B

HOW TO PROTECT AS PAYMENT?

5

ext. solvency score+ cust. engagement

Conversion

steven@outfittery.de | Data Natives 2016

Hint 1: create a thorough monitoring of the data inputs, e.g.

Regis-tration

article details

check-out

payment control

article searchVer. B

HOW TO PROTECT AS PAYMENT?

5

ext. solvency score+ cust. engagement

Conversion

steven@outfittery.de | Data Natives 2016

Hint 1: create a thorough monitoring of the data inputs, e.g.

Regis-tration

article details

check-out

payment control

article searchVer. B

HOW TO PROTECT AS PAYMENT?

5

ext. solvency score+ cust. engagement

Hint 2: make payment control dependency on article search explicit

Conversion

steven@outfittery.de | Data Natives 2016

Hint 1: create a thorough monitoring of the data inputs, e.g.

Regis-tration

article details

check-out

payment control

article searchVer. B

HOW TO PROTECT AS PAYMENT?

5

ext. solvency score+ cust. engagement

Hint 3: document the dependencies and educate product managersHint 2: make payment control dependency on article search explicit

Conversion

steven@outfittery.de | Data Natives 2016

Regis-tration

article searchVer. A article

detailscheck-

outpayment control

Revenue

article searchVer. B

search rank

WHAT TO DO AS SEARCH?

6

ext. solvency score+ cust. engagement

steven@outfittery.de | Data Natives 2016

Regis-tration

article searchVer. A article

detailscheck-

outpayment control

Revenue

article searchVer. B

search rank

WHAT TO DO AS SEARCH?

6

ext. solvency score+ cust. engagement

Step 0: connect with product manager of payment control first

steven@outfittery.de | Data Natives 2016

Regis-tration

article searchVer. A article

detailscheck-

outpayment control

Revenue

article searchVer. B

search rank

Step 1: make payment control independent on article search and run A/B test

WHAT TO DO AS SEARCH?

6

ext. solvency score

payment control

Step 0: connect with product manager of payment control first

steven@outfittery.de | Data Natives 2016

Regis-tration

article details

check-out

payment control

article searchVer. B

Step 1: make payment control independent on article search and run A/B test

WHAT TO DO AS SEARCH?

6

ext. solvency score

payment control

Step 2: deploy article search and collect training data for new payment control logic

Step 0: connect with product manager of payment control first

steven@outfittery.de | Data Natives 2016

Regis-tration

article details

check-out

payment control

article searchVer. B

Step 1: make payment control independent on article search and run A/B test

WHAT TO DO AS SEARCH?

6

ext. solvency score+ cust. engagement

Step 2: deploy article search and collect training data for new payment control logicStep 3: train and deploy an updated payment control logic

Step 0: connect with product manager of payment control first

steven@outfittery.de | Data Natives 2016

Regis-tration

article details

check-out

payment control

article searchVer. B

Step 1: make payment control independent on article search and run A/B test

WHAT TO DO AS SEARCH?

6

ext. solvency score+ cust. engagement

Step 2: deploy article search and collect training data for new payment control logicStep 3: train and deploy an updated payment control logic

Step 0: connect with product manager of payment control first

THIS MAKES PRODUCT DEVELOPMENT VERY EXPENSIVE

steven@outfittery.de | Data Natives 2016

… Excecutives:• do not underestimate the costs of data science - there is no free lunch

•prioritize features according to , where total costs cover development, testing, data acqusition, data science

•bring together product management and data science as close as possible•centralise all A/B testing & personalisation efforts in a data department

… Product manager• map out all and document all data dependencies

… Data scientist• build a thorough monitoring & alerting systems for all inputs to your logics• prefer explicit over implicit data dependencies• create sparse solutions, i.e. limit the dependencies don’t take everyone’s data• educate product management

RECOMMENDATIONS TO …

7

Thank you