how to eliminate ideas as soon as possible
TRANSCRIPT
![Page 1: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/1.jpg)
HypothesisTesting:HowtoEliminateIdeasasSoonasPossible
RomanZykovRetailRocket
Boston,RecSys2016
![Page 2: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/2.jpg)
Context• Intro
• OfflinevsOnlinetesting
• Makeofflinetestingshorter
• Artificialdiversitymetric
• Onlinetests
![Page 3: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/3.jpg)
RetailRocket• Personalisedreal-timerecommendations
• E-commerceonly
• Multiplechannels(site,email,…)
• Foundedin2012
• Offices:Amsterdam,Barcelona,Milan,Moscow
• 1000+retailpartners
• 100+milliondailyevents
![Page 4: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/4.jpg)
Whytestingisimportant?
• Highlycompetitivemarket• It’snothardtocreateownrecommendation• Constantchangesintheproductandalgorithms• Fastandreliabledecisions
![Page 5: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/5.jpg)
OfflinevsOnlinetesting
Offlinetestingforecastsonlinetestingresults• Relativelyfast,testingofminorchangesrequireshours• Fewresources:data,computationalresources,code,1dev• Hardtoforecastonlinemetricsinsomecases• Influenceofanalgorithmonusers'behaviourisignored• Badvaluesofofflinemetricspreventonlineimplementation
Onlinetest-finaldecisionpoint• Requiresmuchtime.Atleasttwocyclesofdecisionmaking• Requiresmanyresources:design,onsiteproduction,etc
![Page 6: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/6.jpg)
Testingfacts
• Nineoutoftenideasdonotimproveanything• Mostideashaveminorimpact:
o addnewdata:extractedfromtext,images,etco adjustparametersofalgorithm
![Page 7: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/7.jpg)
Offlinetesting
![Page 8: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/8.jpg)
OfflinepredictsOnline
Majorchangesornewalgorithm• Alwayscheckbyonlineexperiment• Findappropriateofflinemetricafter• Trydifferentdefinitionsofusers’sessions• Trydifferenteventssequences
Minorchanges•Useofflinetestsifyouhaveprovedofflinemetric
![Page 9: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/9.jpg)
MakeofflinetestingshorterRetailRocket
Whatwedid• FunctionalprogrammingonScala/Spark.Fourlanguages(Python,Java,Pig,Hive)hadbeenpreviouslyused.
• ResearchinScala/SparkNotebookswithaddedRintegrationforgraphics
• Offlineevaluationframeworkforallofourtaskswithmetricscalculations.ThemostcomplicatedprojectamongothersinRetailRocket
Whatwegot• Ittakeshourstoproveordisapproveanysimpleideawhereaspreviouslyitcouldhavetakendays
• Researchislimitedbythepowerofourclusterandthenumberofdatascientists
![Page 10: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/10.jpg)
Scala/SparknotebookwithR
![Page 11: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/11.jpg)
Offlineframework
• ScalaonSpark• Dealswithexistingweblogs• Implicitfeedback• Majormetrics:
o Recall,Diversity,RecallwithNN,EmptyRecs• Minormetrics:
o Serendipity,Novelty,Coverage• Differenttypesofeventssequences• Differentdefinitionsofusers’sessions• Personalised/Non-personalisedrecommendations• AdjustableTOPofviewablerecommendations• Testpanelofsitesfromdifferentdomains
![Page 12: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/12.jpg)
Offlineeventssequences
view1view2view3cart1 cart2view4view5 view6purchase1
View2View View2Cart View2Purchase Cart2Purchase Cart2Cart
view1->view2
view2->view3
view3->view4
view4->view5
view5->view6
view1->cart1
view2->cart1
view3->cart1
view4->cart1
view5->cart2
view6->cart2
view1->purchase1
view2->purchase1
view3->purchase1
view4->purchase1
view5->purchase1
view6->purchase1
cart1->purchase1
cart2->purchase1
cart1->cart2
*Events:productview,addtocart,purchase,mainpageview,search,catalogpage,…
![Page 13: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/13.jpg)
Offlinemetricexamples
view1view2view3cart1 cart2view4view5 view6purchase1
WhatCustomersBuyAfterViewingThisItem• View2Cart• View2Purchase• …
CustomersWhoBoughtThisItemAlsoBought• Cart2Cart• Cart2Purchase• View2Cart• …
![Page 14: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/14.jpg)
Case:Artificialdiversification
![Page 15: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/15.jpg)
ArtificialdiversificationOriginal
After
Problem:It’snotimpossibletouseRecallforevaluating
![Page 16: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/16.jpg)
RecallwithNearestNeighbours(NN)
Top4recs
0.8 0.7 0.5 0.5
0.8 0.7 0.5 0.5
0.6 0.5 0.4
0.9 0.8 0.3 0.5
Contentbasedsimilarity(Nearestneighbours)
Realitem
0.5
Indirecthit
1.0
Directhit
Nohit
0.0
Metric=Averageoverallsessions
![Page 17: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/17.jpg)
OnlineA/Btesting
![Page 18: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/18.jpg)
AA/BBtests
Agroup
Agroup
Bgroup
Bgroup
Controlgroup
Testgroup
![Page 19: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/19.jpg)
AA/BBtests
A
A
B
B
A
A
B
B
IdealDirty
![Page 20: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/20.jpg)
Bayesianapproach• Conversionrates
o Betadistributionwithnormalpriors• AverageOrderValues
o Normaldistribution(afterlog)withnormalpriors• Priorsfromhistoricaldatabeforeexperiment
Anythingmaybedonewithposteriors.
E.g.:Thereisa95%chancethatAhasan1%liftoverB
![Page 21: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/21.jpg)
Conclusion• Offlinetestingcanpredictonlineresults
• OneprogramminglanguageforR&Dreducesthetesttime
• TheScalalanguageisagoodalternativeforMLtasks
• Differenteventsequencesforofflinemetrics
• RecallwithNearestNeighbours(NN)metric
![Page 22: How to eliminate ideas as soon as possible](https://reader034.vdocument.in/reader034/viewer/2022042517/586f8fa11a28ab54768b7699/html5/thumbnails/22.jpg)
Thankyou!
https://github.com/RetailRocket/SparkMultiTool