yuri m. brovman, data scientist, ebay
TRANSCRIPT
InnovationsinRecommenderSystemsforaSemi-structuredMarketplace
March24,2017
YuriM.BrovmanMerchandisingTeam,eBay,NewYorkCity,USA
1
2
Acknowledgements
MerchandisingTeamNatraj Srinivasan
PaulWangBenKleinJin Chung
RyanSnyderSteveNeola
DanielGalronMichalWiejaMikeFirer
PavelStepanovMarcusGallagher
Giri Iyengar
3
eBay MarketplaceChallenges:• 1billionactiveitems• 150millionactiveusers• Limitedstructureddatacoverage• Volatileinventory• Coldstartproblem
4
ML/DeepLearning@eBayNYCMachineLearned
Ranking
Optimizing Similar Item Recommendations in aSemi-structured Marketplace to Maximize Conversion
Yuri M. BrovmaneBay Inc., New York City, USA
Marie JacobeBay Inc., New York City, USA
Natraj SrinivasaneBay Inc., New York City, [email protected]
Stephen NeolaeBay Inc., New York City, USA
Daniel GalroneBay Inc., New York City, USA
Ryan SnydereBay Inc., New York City, USA
ABSTRACTThis paper tackles the problem of recommendations in eBay’slarge semi-structured marketplace. eBay’s variable inven-tory and lack of structured information about listings makestraditional collaborative filtering algorithms di�cult to use.We discuss how to overcome these data limitations to pro-duce high quality recommendations in real time with a com-bination of a customized scalable architecture as well as awidely applicable machine learned ranking model. A point-wise ranking approach is utilized to reduce the ranking prob-lem to a binary classification problem optimized on past userpurchase behavior. We present details of a sampling strategyand feature engineering that have been critical to achieve alift in both purchase through rate (PTR) and revenue.
Keywordse-commerce; recommender systems; machine learning; learn-ing to rank
1. INTRODUCTIONRecommender systems in e-commerce have been exten-
sively studied over the last few decades. Recommendationsdrive a considerable portion of site revenue, and ensure thatusers stay engaged with content for as long as possible. Un-like general marketplaces such as Amazon and Walmart,which o↵er warehouse products in a documented catalog,the eBay marketplace o↵ers more diverse listings ranginganywhere from a new iPhone (a specific product with struc-tured data attributes) to o↵hand antique items with noknown characteristics. With over 800 million active listingsat any given time as well as over 150 million active buyers,this semi-structured marketplace exhibits a heavy-tailed dis-tribution of items – a large number of listings are unique,unidentifiable items that are popular among niche buyers. Inaddition, multiple item conditions and selling formats makeserving relevant recommendations significantly harder.
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).
RecSys ’16 September 15-19, 2016, Boston , MA, USA
c� 2016 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-4035-9/16/09.
DOI: http://dx.doi.org/10.1145/2959100.2959166
Figure 1: Item page showing similar item recom-mendations above the fold.
In this paper, we highlight several unique challenges inproviding high quality item recommendations to users inreal time that are specific to the scale and variety of thisdata. There is limited structured data coverage of itemswhich makes it di�cult to utilize specific item attributes.Additionally, many items tend to be short-lived – they sur-face on the site for one week and are never listed again.Traditional collaborative filtering algorithms [5, 10, 8] arenot e↵ective in this environment due to this volatility of in-ventory and limitation of structured data coverage.While our recommendations are driven through a large
number of channels, including desktop, mobile and emails,we discuss the problem setting of our most viewed place-ment – the eBay item page, shown in Figure 1. This pageshows the details of a seed item, with five recommendationsshown above the fold. The recommendations are generatedbased on both similarity between the seed item and the rec-ommended item, and likelihood of purchase1. We use a twostage system where we first retrieve a subset of relevant itemsto the seed item, and then rank the subset of possible recom-mendations maximizing the chance of conversion. A com-parable multi-stage approach where multiple sources wereaggregated together to produce improved search results has
1Other placements on this site target more diverse recom-mendations. Past analysis showed that most users look forthe same/similar product to the seed in this placement.
199
ImageBasedDeepLearning
TextBasedDeepLearning
• Similaritemrecommendations
• Pointwisemachinelearnedrankingmodel
• PublishedinACMRecSys 2016
• Similaritemrecommendationsusingimages
• DeeplearningmodelbasedonGoogLeNetCNNusingGPUFigure 1: Items from eBay that have the words “Chinese” and “Vase” in their title. This exemplifies that textual data is not
always informative enough to find stylistically similar items.
Figure 2: Comparison of the recommendations before the proposed algorithm and after. The first column is the seed item.The second, third, and fourth columns show the previous recommendations. The fifth, sixth, and seventh columns show thenew recommendations. The first three rows are books, and the last two are movies.
2
• Complementaryitemrecommendationsusingtitleandaspectstext
• NoveldeeplearningarchitecturetrainedoneBaydatausingTheano
5
ML/DeepLearning@eBayNYCMachineLearned
Ranking
Optimizing Similar Item Recommendations in aSemi-structured Marketplace to Maximize Conversion
Yuri M. BrovmaneBay Inc., New York City, USA
Marie JacobeBay Inc., New York City, USA
Natraj SrinivasaneBay Inc., New York City, [email protected]
Stephen NeolaeBay Inc., New York City, USA
Daniel GalroneBay Inc., New York City, USA
Ryan SnydereBay Inc., New York City, USA
ABSTRACTThis paper tackles the problem of recommendations in eBay’slarge semi-structured marketplace. eBay’s variable inven-tory and lack of structured information about listings makestraditional collaborative filtering algorithms di�cult to use.We discuss how to overcome these data limitations to pro-duce high quality recommendations in real time with a com-bination of a customized scalable architecture as well as awidely applicable machine learned ranking model. A point-wise ranking approach is utilized to reduce the ranking prob-lem to a binary classification problem optimized on past userpurchase behavior. We present details of a sampling strategyand feature engineering that have been critical to achieve alift in both purchase through rate (PTR) and revenue.
Keywordse-commerce; recommender systems; machine learning; learn-ing to rank
1. INTRODUCTIONRecommender systems in e-commerce have been exten-
sively studied over the last few decades. Recommendationsdrive a considerable portion of site revenue, and ensure thatusers stay engaged with content for as long as possible. Un-like general marketplaces such as Amazon and Walmart,which o↵er warehouse products in a documented catalog,the eBay marketplace o↵ers more diverse listings ranginganywhere from a new iPhone (a specific product with struc-tured data attributes) to o↵hand antique items with noknown characteristics. With over 800 million active listingsat any given time as well as over 150 million active buyers,this semi-structured marketplace exhibits a heavy-tailed dis-tribution of items – a large number of listings are unique,unidentifiable items that are popular among niche buyers. Inaddition, multiple item conditions and selling formats makeserving relevant recommendations significantly harder.
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).
RecSys ’16 September 15-19, 2016, Boston , MA, USA
c� 2016 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-4035-9/16/09.
DOI: http://dx.doi.org/10.1145/2959100.2959166
Figure 1: Item page showing similar item recom-mendations above the fold.
In this paper, we highlight several unique challenges inproviding high quality item recommendations to users inreal time that are specific to the scale and variety of thisdata. There is limited structured data coverage of itemswhich makes it di�cult to utilize specific item attributes.Additionally, many items tend to be short-lived – they sur-face on the site for one week and are never listed again.Traditional collaborative filtering algorithms [5, 10, 8] arenot e↵ective in this environment due to this volatility of in-ventory and limitation of structured data coverage.While our recommendations are driven through a large
number of channels, including desktop, mobile and emails,we discuss the problem setting of our most viewed place-ment – the eBay item page, shown in Figure 1. This pageshows the details of a seed item, with five recommendationsshown above the fold. The recommendations are generatedbased on both similarity between the seed item and the rec-ommended item, and likelihood of purchase1. We use a twostage system where we first retrieve a subset of relevant itemsto the seed item, and then rank the subset of possible recom-mendations maximizing the chance of conversion. A com-parable multi-stage approach where multiple sources wereaggregated together to produce improved search results has
1Other placements on this site target more diverse recom-mendations. Past analysis showed that most users look forthe same/similar product to the seed in this placement.
199
ImageBasedDeepLearning
TextBasedDeepLearning
• Similaritemrecommendations
• Pointwisemachinelearnedrankingmodel
• PublishedinACMRecSys 2016
• Similaritemrecommendationsusingimages
• DeeplearningmodelbasedonGoogLeNetCNNusingGPUFigure 1: Items from eBay that have the words “Chinese” and “Vase” in their title. This exemplifies that textual data is not
always informative enough to find stylistically similar items.
Figure 2: Comparison of the recommendations before the proposed algorithm and after. The first column is the seed item.The second, third, and fourth columns show the previous recommendations. The fifth, sixth, and seventh columns show thenew recommendations. The first three rows are books, and the last two are movies.
2
• Complementaryitemrecommendationsusingtitleandaspectstext
• NoveldeeplearningarchitecturetrainedoneBaydatausingTheano
6
SimilarAlgorithm(SIM)oneBay• Poweringseveralprominentplacementsacrossdesktopandmobile• Serving1billionimpressionsdaily• Responsetime=200ms end-to-end
ItemPage
Goal:Findmostsimilaritemsand
maximizeconversion
7
Architecture
MerchandisingBackEnd
itemtitles productscoviews
Offline
Offline
itemimagesOfflineGPU
8
MachineLearnedRankinginSearch• Learningtorankforinformationretrievaltoreducerankingproblemto
classificationproblem• Trainingon𝒙𝒊 = {query,URL}pairsasinput• Binaryormulti-classrelevancelabelare𝒚𝒊 collectedwithcrowdsourcing
pair
• Pointwiserankingapproachusingbinaryclassifiertorankrecommendationbytheprobabilityofpurchase
• Trainingon𝒙𝒊 = {seeditem,recommendeditem}pairsloggedfromimplicituserdata• Classlabelare𝒚𝒊 = {0=non-clicked,1=purchased}• Trainingdatasetsize≈350Kpositivetrainingpairs
pair1 pair2 pair3 pair4
Sampleimpressionfromitempage
9clicked non-clicked non-clicked purchased
MachineLearnedRanking
10
FeatureEngineering: Price• Howtocompareseedpricetorecommendationprice?• FittheratioofrecommendationpricetoseedpricewithCauchydistribution• FeaturescoredefinedbynormalizedCauchydistributionwithparametersdefinedby
pastpurchaseevents
Pricedistributionofratiofrompastpurchasedata
PDF = 1
πγ 1+ x − x0
γ
"
#$
%
&'
2(
)**
+
,--
γ =HWHMx0 =median
• Consideredseveralclasscombinationsforbinaryclassifier• Calculatedclassseparability ofeachfeatureusingtheKLDivergence• Usingclass0[non-clicked]andclass1[purchased]producedhighestKLDivergence
inpricefeature
SamplingStrategy
KLDivergence=0.15
KLDivergenceNegative Class PositiveClass PriceFeaturenon-clicked clicked 0.01
non-purchased purchased 0.13clicked notpurchased purchased 0.07
non-clicked purchased 0.15
11
KLDivergence=0.01
ClassificationResults
12
Metricsfromvalidationdataset:
AUC
LogisticRegression GradientBoostingClassifierAccuracy 0.698 0.734
class0 class1 class0 class1Precision 0.70 0.70 0.75 0.72Recall 0.70 0.70 0.70 0.77LogLoss 0.58 0.53AUC 0.766 0.81
13
RankingMetricNDCG@k• EvaluateperformancewithrankingmetricDiscountedCumulativeGain(DCG):
DCG =2li −1
log2(ri )+1i=1
n
∑
li = relevanceri = rank
• NormalizedDCGtruncatedatpositionk(NDCG@k)• Relevancefunctiondefinedtobe{non-clicked=0,clicked=0,purchased=1}
Rankingperformanceimprovedwithclassifiermodel
14
15
A/BTestResults
CTR(%) PTR(%) Revenue (%)+3.0 +6.6 +6.0
• Siteandcategorysegmentedlogisticregressionimplementedinproduction• A/Btestshowedpositiveresultsoverexistingbaselinerankingmodelinkeyoperational
metricsclickthroughrate(CTR),purchasethroughrate(PTR),andrevenue
LaunchedMLRmodeltofulltrafficworldwidein2016!
%Lift
Conclusion• Presentedscalablearchitecture withamachinelearnedranking
model usedintheSimilaritemsalgorithmtoproducehighqualityrecommendations
• Therearemanyinterestingengineering andmachinelearning/deeplearningchallengesthatarebeingsolved@ eBayNYC!
16
Thankyou!Wearehiring!