f actorization m achine : model, optimization and applications yang liu email: [email protected]...
TRANSCRIPT
![Page 1: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/1.jpg)
1
FACTORIZATION MACHINE:MODEL, OPTIMIZATION AND APPLICATIONS
Yang LIUEmail: [email protected]: Prof. Andrew Yao
Prof. Shengyu Zhang
![Page 2: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/2.jpg)
2
OUTLINE
Factorization machine (FM) A generic predictor Auto feature interaction
Learning algorithm Stochastic gradient descent (SGD) …
Applications Recommendation systems Regression and classification …
![Page 3: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/3.jpg)
3
DOUBAN MOVIE
![Page 4: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/4.jpg)
4
PREDICTION TASK
e.g. Alice rates Titanic 5 at time 13
??
![Page 5: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/5.jpg)
5
PREDICTION TASK
Format: for regression, for classification
Training set:
Testing set: ,
Objective: to predict
![Page 6: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/6.jpg)
6
LINEAR MODEL – FEATURE ENGINEERING
Linear SVM
Logistic Regression
�̂� (𝑥 )= 1
1+𝑤0 exp (−𝑤𝑇𝑥 )
![Page 7: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/7.jpg)
7
FACTORIZATION MODEL
Model parameters , where
is the inner dimension
Linear:
FM:
Interaction between variables
![Page 8: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/8.jpg)
8
W
𝑤𝑖 , 𝑗= ⟨𝑣 𝑖 ,𝑣 𝑗 ⟩INTERACTION MATRIX
![Page 9: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/9.jpg)
9
W
𝑤𝑖 , 𝑗= ⟨𝑣 𝑖 ,𝑣 𝑗 ⟩INTERACTION MATRIX
![Page 10: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/10.jpg)
10
W?
𝑤𝑖 , 𝑗= ⟨𝑣 𝑖 ,𝑣 𝑗 ⟩INTERACTION MATRIX
![Page 11: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/11.jpg)
11
VVT
k
W
𝑤𝑖 , 𝑗= ⟨𝑣 𝑖 ,𝑣 𝑗 ⟩INTERACTION MATRIX
=
![Page 12: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/12.jpg)
12
=
𝑤𝑖 , 𝑗= ⟨𝑣 𝑖 ,𝑣 𝑗 ⟩INTERACTION MATRIX
VVT
Wk
![Page 13: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/13.jpg)
13
=
𝑤𝑖 , 𝑗= ⟨𝑣 𝑖 ,𝑣 𝑗 ⟩INTERACTION MATRIX
VVT
W
![Page 14: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/14.jpg)
14
=
𝑤𝑖 , 𝑗= ⟨𝑣 𝑖 ,𝑣 𝑗 ⟩INTERACTION MATRIX
VVT
W¿ 𝒗𝑨
𝑻 𝒗𝑻𝑰
![Page 15: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/15.jpg)
15
=
𝑤𝑖 , 𝑗= ⟨𝑣 𝑖 ,𝑣 𝑗 ⟩INTERACTION MATRIX
VVT
W𝑤𝑖𝑗
𝑣 𝑖T
𝑣 𝑗
Factorization
![Page 16: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/16.jpg)
16
=
𝑤𝑖 , 𝑗= ⟨𝑣 𝑖 ,𝑣 𝑗 ⟩INTERACTION MATRIX
VVT
W𝑤𝑖𝑗
𝑣 𝑖T
𝑣 𝑗
FactorizationMachine
![Page 17: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/17.jpg)
17
FM: PROPERTIES
Expressiveness:
Feature dependency: and are dependent
Linear computation complexity:
![Page 18: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/18.jpg)
18
OPTIMIZATION TARGET
Min ERROR Min ERROR + Regularization
Loss function
![Page 19: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/19.jpg)
19
STOCHASTIC GRADIENT DESCENT (SGD)
For item , update by:
: initial value of : learning rate : regularization
Pros Easy to implement Fast convergence on big training data
Cons Parameter tuning Sequential method
![Page 20: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/20.jpg)
20
APPLICATIONS
EMI Music Hackathon 2012 Song recommendation
Given: Historical ratings User demographics
# features: 51K # items in training: 188K
?
![Page 21: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/21.jpg)
21
RESULTS FOR EMI MUSIC
FM: Root Mean Square Error (RMSE) 13.27626 Target value [0,100] The best (SVD++) is 13.24598
Details Regression Converges in 100 iterations Time for each iteration: < 1 s
Win 7, Intel Core 2 Duo CPU 2.53GHz, 6G RAM
![Page 22: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/22.jpg)
22
OTHER APPLICATIONS
Ads CTR prediction (KDD Cup 2012) Features
User_info, Ad_info, Query_info, Position, etc. # features: 7.2M # items in training: 160M Classification Performance:
AUC: 0.80178, the best (SVM) is 0.80893
![Page 23: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/23.jpg)
23
OTHER APPLICATIONS
HiCloud App Recommendation Features
App_info, Smartphone model, installed apps, etc. # features: 9.5M # items in training: 16M Classification Performance:
Top 5: 8%, Top 10: 18%, Top 20: 32%; AUC: 0.78
![Page 24: F ACTORIZATION M ACHINE : MODEL, OPTIMIZATION AND APPLICATIONS Yang LIU Email: yliu@cse.cuhk.edu.hk Supervisors: Prof. Andrew Yao Prof. Shengyu Zhang 1](https://reader030.vdocument.in/reader030/viewer/2022032606/56649eaa5503460f94baed29/html5/thumbnails/24.jpg)
24
SUMMARY
FM: a general predictor Works under sparsity Linear computation complexity Estimates interactions automatically Works with any real valued feature vector
THANKS!