predicting loyal customers - galit shmueli 9... · team 9: yuchi shih, wendy huang, zoe cheng,...

Team 9: Yuchi Shih, Wendy Huang, Zoe Cheng, Jessy Yang

Predicting Loyal Customers to Increase Merchants’ Income

Business Goal

Goal

Stakeholder

Reduce the promotion cost & increase the rate of the repeated customers

Problem One-time buyers attracted by coupons and discounts (Low ROI)

Sellers on TMall

Data Mining GoalPredicting whether this client will be the loyal customer to this seller.

Outcome variable: binary outcome(0,1)

Data• Source: Tmall.com (A huge E-commerce company in China)• Data: 1476911 obs. based on actual user activity on the platform• Population: customers who bought on 1111• Loyal customer: customers bought again in the next 6 month

Preprocessing

0 Click1 Add-to-chart2 Purchase3 Add-to-favorite

Data(Data cleaning)

Data• Source: Tmall.com (Top B2C E-commerce platform in China)• Data: 1476911 obs. (those customers all bought on 1111)• Definition of Loyalty: Customers bought again at the same seller in the next 6 month after 1111

• Preprocessing:1. Missing value: replace missing value by MICE (tree)2. Derived variable:

a. separate action_type variable into action_0, action_1….b. sum up action_0, action_1…. according to user, seller ID,

time_stamp 3. Variable selection: age, gender, action_0~3

Data visualization

Click

Add to cart

Purchase

Add to favorite

Visualization (before November)

Methods & Performance Evaluation● Data mining methods: Regression, Random forest, XGBoost

a. K-NN and Hierarchical clustering are not suitable in this case because of the huge dataset.

● Performance measure: lift chart ● How to map our business goal?

a. Use the K-means clustering method to cluster the training dataset.b. Build different model for each group by decision tree and regression.c. Predict the outcome and evaluate the performance by confusion matrix.d. The results allow us to predict the potential repeated buyers in each group.

Data

ALLOriginal Undersampling

# Records loyal (%) # Records loyal (%)

Training (50%) 738,455 10.47% 738,455 50%

Validation(30%) 443,073 10.46% 443,073 10.46%

Testing (20%) 295,383 10.53% 295,383 10.53%

Result (data=all) Accuracy of top 10%

Best Model: Random Forest

Naive Rule 10 %Benchmark: Naive Rule

Model Logistic Random forest Xgboost

Lift Chart

Accuracy of top 10% 23.69% 25.78% 20.43%

Result(data=0511-1025)

Model Logistic Random forest Xgboost

Lift Chart

Accuracy of top 10% 33.96% 48.86% 20.56%

Accuracy of top 10%

Best Model Random Forest

Naive Rule 10 %Benchmark: Naive Rule

Result(data=1026-1111) Accuracy of top 10%

Best Model Random Forest

Naive Rule 10 %Benchmark: Naive Rule

Model Logistic Random forest Xgboost

Lift Chart

Accuracy of top 10% 20.4% 24.61% 18.32%

Recommendations

For Implementation For Model Risk For Business Policy

1. Models might require updates

2. When encountering missing value, we have to understand what happened so that we can properly deal with it.

1. Low accuracy of top 10% 2. The total error rate would

be higher if undersampling. 3. The dataset exists some

missing value such age and gender.

4. Since the samples increase enormously and rapidly in the last two weeks, the bias might affect the outcome.

1. Collect more data about characteristics of the sellers.

2. Increase the customer’s willingness to add their product into favorite.

3. Lurkers are potentially loyal customers, which the sellers should not ignore these people.

Implementation/Production ConsiderationsTime difference: This dataset was collected from 2 years ago.

A certain period of data:

● The dataset only contains half year of the business data. Therefore the model might require updates as time goes, in case features changes.

● For instance, if there is a new brand not included in this dataset, it might not be able to predict the results.

Lack of information: Not included merchants’ or discounts’ data.