data mining lectures lecture 17: web log mining padhraic smyth, uc irvine ics 278: data mining...

Data Mining Lectures Lecture 17: Web Log Mining Padhraic Smyth, UC Irvine

ICS 278: Data Mining

Lecture 17: Web Log Mining

Padhraic SmythDepartment of Information and Computer Science

University of California, Irvine


Outline

• Basic concepts in Web log data analysis

• Predictive modeling of Web navigation behavior– Markov modeling methods

• Analyzing search engine data

• Ecommerce aspects of Web log mining


Introduction

• Useful to study human digital behavior, e.g. search engine data can be used for– Exploration e.g. # of queries per session?– Modeling e.g. any time of day dependence?– Prediction e.g. which pages are relevant?

• Applications– Understand social implications of Web usage– Design of better tools for information access– E-commerce applications


How our Web navigation is recorded…

• Web logs– Record activity between client browser and a specific Web server– Easily available– Can be augmented with cookies (provide notion of “state”)

• Search engine records– Text in queries, which responses were clicked on, etc

• Client-side browsing records– Produced for research purposes as part of a study– Automatically recorded by client-side software– Harder to obtain, but much more accurate than server-side logs

• Other sources– Web site registration, purchases, email, etc– ISP recording of Web browsing


Web Server Log Files

• Server Transfer Log: – transactions between a browser and server are logged– IP address, the time of the request– Method of the request (GET, HEAD, POST…)– Status code, a response from the server– Size in byte of the transaction

• Referrer Log: – where the request originated

• Agent Log: – browser software making the request (spider)

• Error Log: – request resulted in errors (404)


W3C Extended Log File FormatField Date Description

Date date The date that the activity occurredTime time The time that the activity occurredClient IP address c-ip The IP address of the client that accessed your server

User Name cs-usernameThe name of the autheticated user who access your server, anonymous users are represented by -

Servis Name s-sitename The Internet service and instance number that was accessed by a clientServer Name s-computername The name of the server on which the log entry was generatedServer IP Address s-ip The IP address of the server that accessed your serverServer Port s-port The port number the client is connected toMethod cs-method The action the client was trying to performURI Stem cs-uri-stem The resource accessedURI Query cs-uri-query The query, if any, the client was trying to performProtocol Status sc-status The status of the action, in HTTP or FTP termsWin32 Status sc-win32-status The status of the action, in terms used by Microsoft WindowsBytes Sent sc-bytes The number of bytes sent by the serverBytes Received cs-bytes The number of bytes received by the serverTime Taken time-taken The duration of time, in milliseconds, that the action consumedProtocol Version cs-version The protocol (HTTP, FTP) version used by the clientHost cs-host Display the content of the host header

User Agent cs(User Agent) The browser used on the clientCookie cs(Cookie) The content of the cookie sent or received, if any

Referrer cs(Referrer)The previous site visited by the user. This site provided a link to the current site

cs = client-to-server actions

s = server actionsc = client actions

sc = server-to-client actions

http://www.w3.org/TR/WD-logfile


Example of Web Log entries

Apache web log:205.188.209.10 - - [29/Mar/2002:03:58:06 -0800] "GET

/~sophal/whole5.gif HTTP/1.0" 200 9609 "http://www.csua.berkeley.edu/~sophal/whole.html" "Mozilla/4.0 (compatible; MSIE 5.0; AOL 6.0; Windows 98; DigExt)"

216.35.116.26 - - [29/Mar/2002:03:59:40 -0800] "GET /~alexlam/resume.html HTTP/1.0" 200 2674 "-" "Mozilla/5.0 (Slurp/cat; [email protected]; http://www.inktomi.com/slurp.html)“

202.155.20.142 - - [29/Mar/2002:03:00:14 -0800] "GET /~tahir/indextop.html HTTP/1.1" 200 3510 "http://www.csua.berkeley.edu/~tahir/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)“

202.155.20.142 - - [29/Mar/2002:03:00:14 -0800] "GET /~tahir/animate.js HTTP/1.1" 200 14261 "http://www.csua.berkeley.edu/~tahir/indextop.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)“


Routine Server Log Analysis

• Most and least visited web pages• Entry and exit pages• Referrals from other sites or search engines• What are the searched keywords• How many clicks/page views a page received• Error reports, like broken links


Visualization of Web Log Data over Time

http://www.swig.org/papers/Tcl96/hits.gif


Server Log Analysis


Descriptive Summary Statistics

• Histograms, scatter plots, time-series plots– Very important!– Helps to understand the big picture– Provides “marginal” context for any model-building

• models aggregate behavior, not individuals

– Challenging for Web log data

• Examples– Session lengths (e.g., power laws)– Click rates as a function of time, content


100

101

102

10-6

10-5

10-4

10-3

10-2

10-1

100

Session Length L

Em

piric

al F

requ

ency

of L

L = number of page requests in a single sessionfrom visitors to www.ics.uci.eduover 1 week in November 2002(robots removed)

http://www.ics.uci.edu/


100

101

102

103

10-6

10-5

10-4

10-3

10-2

10-1

100

Pro

ba

bili

ty o

f L

Session Length L

Best fit of simple power law model

Log P(L) = -a Log L + b

or P(L) = b L-a


100

101

102

10-6

10-5

10-4

10-3

10-2

10-1

100

Pro

ba

bili

ty o

f L

Session Length L

POISSON

GEOMETRIC

INVERSE GAUSSIAN

POWER-LAW


Web data measurement issues

• Important to understand how data is collected

• Web data is collected automatically via software logging tools– Advantage:

• No manual supervision required

– Disadvantage:• Data can be skewed (e.g. due to the presence of robot traffic)

• Important to identify robots (also known as crawlers, spiders)


A time-series plot of ICS Website data

Number of page requests per hour as a function of time from page requests in the www.ics.uci.edu Web server logs during the first week of April 2002.

http://www.ics.uci.edu/


Robot / human identification

• Robot requests identified by classifying page requests using a variety of heuristics– e.g. some robots self-identify themselves in the server logs

(robots.txt)– Robots explore the entire website in breadth first fashion– Humans access web-pages in depth first fashion

• Tan and Kumar (2002) discuss more techniques


Page requests, caching, and proxy servers

• In theory, requester browser requests a page from a Web server and the request is processed

• In practice, there are– Other users– Browser caching– Dynamic addressing in local network– Proxy Server caching



A graphical summary of how page requests from an individual user can be masked at various stages between the user’s local computer and the Web server.



• Web server logs are therefore not so ideal in terms of a complete and faithful representation of individual page views

• There are heuristics to try to infer the true actions of the user: -– Path completion (Cooley et al. 1999)

• e.g. If known B -> F and not C -> F, then session ABCF can be interpreted as ABCBF

• Anderson et al. 2001 for more heuristics

• In general case, hard to know what user viewed


Identifying individual users from Web server logs

• Useful to associate specific page requests to specific individual users

• IP address most frequently used

• Disadvantages– One IP address can belong to several users– Dynamic allocation of IP address

• Better to use cookies– Information in the cookie can be accessed by the Web server

to identify an individual user over time– Actions by the same user during different sessions can be

linked together


Identifying individual users from Web server logs

• Commercial websites use cookies extensively

• 97% of users have cookies enabled permanently on their browsers (source: Amazon.com, 2003)

• However …– There are privacy issues – need implicit user cooperation– Cookies can be deleted / disabled

• Another option is to enforce user registration– High reliability– Can discourage potential visitors


Sessionizing

• Time oriented (robust)– E.g., by gaps between requests

• not more than 25 minutes between successive requests

• Navigation oriented (good for short sessions and when timestamps unreliable)– Referrer is previous page in session, or– Referrer is undefined but request within 10 secs, or – Link from previous to current page in web site


Client-side data

• Advantages of collecting data at the client side:– Direct recording of page requests (eliminates ‘masking’ due to

caching)– Recording of all browser-related actions by a user (including

visits to multiple websites)– More-reliable identification of individual users (e.g. by login ID

for multiple users on a single computer)

• Preferred mode of data collection for studies of navigation behavior on the Web

• Companies like comScore and Nielsen use client-side software to track home computer users


Client-side data

• Statistics like ‘Time per session’ and ‘Page-view duration’ are more reliable in client-side data

• Some limitations– Still some statistics like ‘Page-view duration’ cannot be totally

reliable e.g. user might go to fetch coffee– Need explicit user cooperation– Typically recorded on home computers – may not reflect a

complete picture of Web browsing behavior

• Web surfing data can be collected at intermediate points like ISPs, proxy servers– Can be used to create user profile and target advertise


Early studies from 1995 to 1997

• Earliest studies on client-side data are Catledge and Pitkow (1995) and Tauscher and Greenberg (1997)

• In both studies, data was collected by logging Web browser commands

• Population consisted of faculty, staff and students

• Both studies found – clicking on the hypertext anchors as the most common action– using ‘back button’ was the second common action


Early studies from 1995 to 1997

• high probability of page revisitation (~0.58-0.61)• Lower bound because the page requests prior to the start of the

studies are not accounted for• Humans are creatures of habit?• Content of the pages changed over time?

• strong recency (page that is revisited is usually the page that was visited in the recent past) effect

• Correlates with the ‘back button’ usage

• Similar repetitive actions are found in telephone number dialing etc


The Cockburn and McKenzie study from 2002

• Previous studies are relatively old

• Web has changed dramatically in the past few years

• Cockburn and McKenzie (2002) provides a more up-to-date analysis– Analyzed the daily history.dat files produced by the Netscape browser

for 17 users for about 4 months– Population studied consisted of faculty, staff and graduate students

• Study found revisitation rates higher than past 94 and 95 studies (~0.81)– Time-window is three times that of past studies



• Revisitation rate less biased than the previous studies?

• Human behavior changed from an exploratory mode to a utilitarian mode?– The more pages user visits, the more are the requests for new

pages– The most frequently requested page for each user can account

for a relatively large fraction of his/her page requests

• Useful to see the scatter plot of the distinct number of pages requested per user versus the total pages requested



The number of distinct pages visited versus page vocabulary size of each of the 17 users in the Cockburn and McKenzie (2002) study (log-log plot)



Bar chart of the ratio of the number of page requests for the most frequent page divided by the total number of page requests, for 17 users in the Cockburn McKenzie (2002) study


Outline






Markov models for page prediction

• General approach is to use a finite-state Markov chain– Each state can be a specific Web page or a category of Web

pages– If only interested in the order of visits (and not in time), each

new request can be modeled as a transition of states

• Issues– Self-transition– Time-independence



• For simplicity, consider order-dependent, time-independent finite-state Markov chain with M states

• Let s be a sequence of observed states of length L. e.g. s = ABBCAABBCCBBAA with three states A, B and C. st is state at position t (1<=t<=L). In general,

• first-order Markov assumption

• This provides a simple generative model to produce sequential data

L

ttt sssPsPsP

2111 ),...,|()()(

L

ttt ssPsPsP

211 )|()()(

)|(),...,|( 111 tttt ssPsssP



• If we denote Tij = P(st = j|st-1 = i), we can define a M x M transition matrix

• Properties– Strong first-order assumption– Simple way to capture sequential dependence

• If each page is a state and if W pages, O(W2), W can be of the order 105 to 106 for a CS dept. of a university

• To alleviate, we can cluster W pages into M clusters, each assigned a state in the Markov model

• Clustering can be done manually, based on directory structure on the Web server, or automatic clustering using clustering techniques



• Tij = P(st = j|st-1 = i) represents the probability that an individual user’s next request will be from category j, given they were in category i

• We can add E, an end-state to the model• E.g. for three categories with end state: -

• E denotes the end of a sequence, and start of a new sequence

1000

)3|()3|3()3|2()3|1(

)2|()2|3()2|2()2|1(

)1|()1|3()1|2()1|1(

EPPPP

EPPPP

EPPPP

T



• First-order Markov model assumes that the next state is based only on the current state

• Limitations– Doesn’t consider ‘long-term memory’

• We can try to capture more memory with kth-order Markov chain

• Limitations– Inordinate amount of training data O(Mk+1)

),..,|(),..,|( 111 kttttt sssPsssP


Parameter estimation for Markov model transitions

• Smoothed parameter estimates of transition probabilities are

• If nij = 0 for some transition (i, j) then instead of having a parameter estimate of 0 (ML), we will have allowing prior knowledge to be incorporated

• If nij > 0, we get a smooth combination of the data-driven information (nij) and the prior

i

ijij

ij n

qnT

)/( iij nq


Parameter estimation for Markov models

• One simple way to set prior parameter is– Consider alpha as the effective sample size– Partition the states into two sets, set 1 containing all states

directly linked to state i and the remaining in set 2– Assign uniform probability r/K to all states in set 2 (all set 2

states are equally likely)– The remaining (1-r) can be either uniformly assigned among

set 1 elements or weighted by some measure– Prior probabilities in and out of E can be set based on our prior

knowledge of how likely we think a user is to exit the site from a particular state


Predicting page requests with Markov models

• Deshpande and Karypis (2001) propose schemes to prune kth-order Markov state space– Provide systematic but modest improvements

• Another way is to use empirical smoothing techniques that combine different models from order 1 to order k (Chen and Goodman 1996)


Mixtures of Markov Chains

• Cadez et al. (2003) and Sen and Hansen (2003) replace the first-order Markov chain:

with a mixture of first-order Markov chains

where c is a discrete-value hidden variable taking K values k P(c = k) = 1

and

P(st | st-1, c = k) is the transition matrix for the kth mixture component

• One interpretation of this is user behavior consists of K different navigation behaviors described by the K Markov chains

)(),|(),...,|(1

111 kcPkcssPsssPK

ktttt

)|(),...,|( 111 tttt ssPsssP


Modeling Web Page Requests with Markov chain mixtures

• MSNBC Web logs– 2 million individuals per day– different session lengths per individual– difficult visualization and clustering problem

• WebCanvas– uses mixtures of Markov chains to cluster individuals based on

their observed sequences– software tool: EM mixture modeling + visualization


128.195.36.195, -, 3/22/00, 10:35:11, W3SVC, SRVR1, 128.200.39.181, 781, 363, 875, 200, 0, GET, /top.html, -, 128.195.36.195, -, 3/22/00, 10:35:16, W3SVC, SRVR1, 128.200.39.181, 5288, 524, 414, 200, 0, POST, /spt/main.html, -, 128.195.36.195, -, 3/22/00, 10:35:17, W3SVC, SRVR1, 128.200.39.181, 30, 280, 111, 404, 3, GET, /spt/images/bk1.jpg, -, 128.195.36.101, -, 3/22/00, 16:18:50, W3SVC, SRVR1, 128.200.39.181, 60, 425, 72, 304, 0, GET, /top.html, -, 128.195.36.101, -, 3/22/00, 16:18:58, W3SVC, SRVR1, 128.200.39.181, 8322, 527, 414, 200, 0, POST, /spt/main.html, -, 128.195.36.101, -, 3/22/00, 16:18:59, W3SVC, SRVR1, 128.200.39.181, 0, 280, 111, 404, 3, GET, /spt/images/bk1.jpg, -, 128.200.39.17, -, 3/22/00, 20:54:37, W3SVC, SRVR1, 128.200.39.181, 140, 199, 875, 200, 0, GET, /top.html, -, 128.200.39.17, -, 3/22/00, 20:54:55, W3SVC, SRVR1, 128.200.39.181, 17766, 365, 414, 200, 0, POST, /spt/main.html, -, 128.200.39.17, -, 3/22/00, 20:54:55, W3SVC, SRVR1, 128.200.39.181, 0, 258, 111, 404, 3, GET, /spt/images/bk1.jpg, -, 128.200.39.17, -, 3/22/00, 20:55:07, W3SVC, SRVR1, 128.200.39.181, 0, 258, 111, 404, 3, GET, /spt/images/bk1.jpg, -, 128.200.39.17, -, 3/22/00, 20:55:36, W3SVC, SRVR1, 128.200.39.181, 1061, 382, 414, 200, 0, POST, /spt/main.html, -, 128.200.39.17, -, 3/22/00, 20:55:36, W3SVC, SRVR1, 128.200.39.181, 0, 258, 111, 404, 3, GET, /spt/images/bk1.jpg, -, 128.200.39.17, -, 3/22/00, 20:55:39, W3SVC, SRVR1, 128.200.39.181, 0, 258, 111, 404, 3, GET, /spt/images/bk1.jpg, -, 128.200.39.17, -, 3/22/00, 20:56:03, W3SVC, SRVR1, 128.200.39.181, 1081, 382, 414, 200, 0, POST, /spt/main.html, -, 128.200.39.17, -, 3/22/00, 20:56:04, W3SVC, SRVR1, 128.200.39.181, 0, 258, 111, 404, 3, GET, /spt/images/bk1.jpg, -, 128.200.39.17, -, 3/22/00, 20:56:33, W3SVC, SRVR1, 128.200.39.181, 0, 262, 72, 304, 0, GET, /top.html, -, 128.200.39.17, -, 3/22/00, 20:56:52, W3SVC, SRVR1, 128.200.39.181, 19598, 382, 414, 200, 0, POST, /spt/main.html, -,

…

5115

11111151511151

77777777

111333

3333131113332232

…

User 5

User 4

User 3

User 2

User 1

From Web logs to sequences


Clusters of Finite State Machines

B

E

D

A

B

E

D

A

B

E

D

A

Cluster 1 Cluster 2

Cluster 3


Learning Problem

• Assumptions– data is being generated by K different groups– Each group is described by a stochastic finite state machine (SFSM)

• aka, a Markov model with an end-state

• Given– A set of sequences from different users of different lengths

• Learn– A “mixture” of K different stochastic finite state machines

• Solution– EM is very easy: fractional counts of transitions– efficient and accurate, scales as O(KN)


Experimental Methodology

• Model Training:– fit 2 types of models

• mixtures of histograms• mixtures of finite state machines

– Train on a full day’s worth of MSNBC Web data

• Model Evaluation:– “one-step-ahead” prediction on unseen test data

• Test sequences from a different day of Web logs

– negative log probability = predictive entropy


20 40 60 80 100 120 140 160 180 2002

2.2

2.4

2.6

2.8

3

3.2

3.4

3.6

3.8

4N

eg

ativ

e lo

g-l

ike

liho

od

[bits

/toke

n]

Number of mixture components [K]

Predictive Entropy Out-of-Sample

Mixtures of Multinomials

Mixtures of SFSMs


0 5 10 15 20

0

5

10

log

coun

t(R) Cluster 1: Category 13

0 5 10 15 20

0

5

10Cluster 1: Category 14

0 10 20 30 40-2

0

2

4 Cluster 1: Category 8

0 5 10 15 20

0

5

10

log

coun


0 5 10

0

5


0 10 20 30 40-2

0

2


0 5 10 15 20

0

5

10

log

coun


0 5 10

0

5


0 5 10 15 20-2

0

2


0 10 20 30 40

0

5

10

log

coun


0 5 10

0

5


0 5 10

0

5


0 5 10 15 20

0

5

10

R = Run Length

log

coun


0 1 2 3 4 5

0

5

10

R = Run Length

Cluster 5: Category 12

0 1 2 3 4

0

5

10

R = Run Length

Cluster 5: Category 6

RUN LENGTH DISTRIBUTIONS WITHIN MARKOV CLUSTERS


Timing Results

0 20 40 60 80 100 120 140 160 180 200-500

0

500

1000

1500

2000

2500

Tim

e [s

ec]

Number of mixture components [K]

N = 70,000

N=110,000

N=150,000


WebCanvas

• Software tool for Web log visualization– uses Markov mixtures to cluster data for display– in use by msnbc.com administrators at Microsoft– also being applied to non-Web data

• Model-based visualization– random sample of actual sequences– interactive tiled windows displayed for visualization– more effective than

• planar graphs• traffic-flow movie in Microsoft Site Server v3.0


WebCanvas: Cadez, Heckerman, et al, 2003


Insights from WebCanvas

• From msnbc.com site adminstrators….– significant heterogeneity of behavior– relatively focused activity of many users

• typically only 1 or 2 categories of pages

– many individuals not entering via main page– detected problems with the weather page– missing transitions (e.g., tech <=> business)


Extensions

• Adding time-dependence– adding time-between clicks, time of day effects

• Uncategorized Web pages– coupling page content with sequence models

• Modeling “switching” behaviors– allowing users to switch between models

• Individualized weights (hierarchical Bayes)

• Update: WebCanvas tool will be part of 2004 SQLServer release


Prediction with Markov mixtures

P(st+1 | s[1,t] ) =



P(st+1 | s[1,t] ) = P(st+1 , k | s[1,t] ) = P(st+1 | k , s[1,t] ) P(k | s[1,t] )



P(st+1 | s[1,t] ) = P(st+1 , k | s[1,t] ) = P(st+1 | k , s[1,t] ) P(k | s[1,t] )

= P(st+1 | k , st ) P(k | s[1,t] )

Prediction of kth component

Membership, basedon sequence history

=> Predictions are a convex combination of K different component transition matrices,with weights based on sequence history


Related Work

• Mixtures of Markov chains– special case: Poulsen (1990)– general case: Ridgeway (1997), Smyth (1997)

• Clustering of Web page sequences– non-probabilistic approaches (Fu et al, 1999)

• Markov models for prediction– Anderson et al (IJCAI, 2001):

• mixtures of Markov outperform other sequential models for page-request prediction


Predicting page requests with Markov models

• K can be chosen by evaluating the out-of-sample predictive performance based on– Accuracy of prediction– Log probability score– Entropy

• Other variations:– Sen and Hansen 2003– Position-dependent Markov models (Anderson et al. 2001,

2002)– Zukerman et al. 1999


Modeling Clickrate Data

• Data– 200k Alexa users, client-side, over 24 hours– ignore URLs requested– goal is to build a time-series model that characterizes user

click rates


0 5 10 15 200

50

100

150

200

250

300

HOURS

NU

MB

ER

OF

CL

ICK

S


5 5.5 6 6.5 7-60

-40

-20

0

20

40

60

80

100

120

HOURS

NU

MB

ER

OF

CLI

CK

S


0 5 10 15 200

50

100

150

200

250

300

HOURS

NU

MB

ER

OF

CLI

CK

S


Markov-Poisson Model

• Doubly stochastic process– Locally constant Poisson rate– indexed by M Markov states

• Fit a model with M = 3 states• absence of a Web session • Web session with slow click rate: 1 minute rate• Web session with rapid click rate: 10 second rate

– Used hierarchical Bayes on individuals


Outline






Analysis of Search Engine Query Logs

# of Sample Query Source SE Time Period

Lau & Horvitz 4690 of 1 Million Excite Sep 1997

Silverstein et al 1 Billion AltaVista 6 weeks in Aug & Sep 1998

Spink et al (series of studies)1Million for each time period

Excite Sep 1997Dec 1999May 2001

Xie & O’Hallaron 110,000 Vivisimo 35 days Jan & Feb 2001

1.9 Million Excite 8 hrs in a day, Dec 1999


Main Results

• Average number of terms in a query is ranging from a low of 2.2 to a high of 2.6

• The most common number of terms in a query is 2

• The majority of users don’t refine their query – The number of users who viewed only a single page increase

29% (1997) to 51% (2001) (Excite)– 85% of users viewed only first page of search results (AltaVista)

• 45% (2001) of queries are about Commerce, Travel, Economy, People (was 20% in 1997)– The queries about adult or entertainment decreased from 20%

(1997) to around 7% (2001)


Xie and O Halloran Study (2002)

• All four studies produced a generally consistent set of findings about user behavior in a search engine context– most users view relatively few pages per query– most users don’t use advanced search features

- Query Length Distributions (bar)

- Poisson Model(dots & lines)


Power-law Characteristics of Common Queries

• Frequency f(r) of Queries with Rank r– 110000 queries from Vivisimo– 1.9 Million queries from Excite

• There are strong regularities in terms of patterns of behavior in how we search the Web

Power-Law in log-log space


Outline






The next few slides are from Ronny Kohavi, director of data mining and personalization at Amazon.com. His full set of slides are available online – see the PPT slides and related papers on ecommerce and

data mining online at http://robotics.stanford.edu/~ronnyk/ronnyk-

bib.html


ECommerce

• Page request Web logs combined with– Purchase (market-basket) information– User address information (if they make a purchase)– Demographics information (can be purchased)– Emails to/from the customer

• Main focus here is to increase revenue– Data mining widely used an online commerce companies like

Amazon

• This is a very rich source of problems for data mining– What products should we advertise to this person?– Can we do dynamic pricing?– If a person buys X should we also suggest Y?– Who are our best customers?– etc


Combining Data Sources

• Comprehensive collection of US consumer and telephone data available via the internet

– Multi-sourced database– Demographic, socioeconomic, and lifestyle information. – Information on most U.S. households– Contributors’ files refreshed a minimum of 3-12 times per year.

– Data sources include: County Real Estate Property Records, U.S. Telephone Directories, Public Information, Motor Vehicle Registrations, Census Directories, Credit Grantors, Public Records and Consumer Data, Driver’s Licenses, Voter Registrations, Product Registration Questionnaires, Catalogers, Magazines, Specialty Retailers, Packaged Goods Manufacturers,

Accounts Receivable Files, Warranty Cards

• Much of this data can be accessed in real-time once a customer self-identifies


Map of World Wide Revenue

UK – 98.8%

US – 0.6%

Australia – 0.1%

NOTE: About 50% of the non-UK orders are wedding list purchases

Low Medium High

Although Debenhams online site only ships in the UK, we see some revenue from the rest of the world.


Results from Blue-Martini

People who have a Travel and Entertainment credit card are 48% more likely to be online shoppers (27% for people with premium credit card)

People whose home was built after 1990 are 45% more likely to be online shoppers

Households with income over $100K are 31% more likely to be online shoppers

People under the age of 45 are 17% morelikely to be online shoppers

Online Consumer Demographics


A higher household income means you are more likely to be an online shopper

Demographics - Income


Demographics – Credit Cards

• The more credit cards, the more likely you are to be an online shopper


Example: Web Traffic

Weekends

Sept-11 Note significant drop in human traffic, not bot

traffic

Registration at Search Engine sites

Internal Perfor-

mance bot


Product Affinities at MEC

• Minimum support for the associations is 80 customers• Confidence: 37% of people who purchased Orbit Sleeping Pad also purchased Orbit Stuff Sack• Lift: People who purchased Orbit Sleeping Pad were 222 times more likely to purchase the Orbit Stuff Sack compared to the general

population

Product Association Lift Confidence

Orbit Sleeping Pad Cygnet

Sleeping Bag Aladdin 2Backpack

Primus Stove

OrbitStuff Sack

WebsiteRecommended Products

222 37%

Bambini Tights Children’s

Bambini CrewneckSweater Children’s

195 52%

Yeti Crew NeckPullover Children’s

Beneficial T’sOrganic LongSleeve T-Shirt Kids’

Silk CrewWomen’s

SilkLong JohnsWomen’s

304 73%

Micro Check Vee Sweater

VolantPants

Composite Jacket

CascadeEntrant Overmitts

Polartec300 DoubleMitts

51 48%

VolantPants

WindstopperAlpine Hat

Tremblant 575Vest Women’s

http://www.mec.ca/Products/product_detail.jsp?PRODUCT%3C%3Eprd_id=15665&bmUID=1037661934664





http://www.mec.ca/Products/product_detail.jsp?PRODUCT%3C%3Eprd_id=197075&FOLDER%3C%3Efolder_id=500461&bmUID=1037666806555








Customer Locations Relative to Retail Stores

Map of Canada with store locations.

Black dots show store locations.

Heavy purchasing areas away from retail stores can suggest new retail store locations No stores in several hot areas:

MEC is building a store in Montreal right now.


Building The Customer Signature

• Building a customer signature is a significant effort, but well worth the effort

• A signature summarizes customer or visitor behavior across hundreds of attributes, many which are specific to the site

• Once a signature is built, it can be used to answer many questions.

• The mining algorithms will pick the most important attributes for each question

• Example attributes computed:– Total Visits and Sales– Revenue by Product Family– Revenue by Month– Customer State and Country– Recency, Frequency, Monetary– Latitude/Longitude from the Customer’s Postal Code

data mining lectures lecture 17: web log mining padhraic smyth, uc irvine ics 278: data mining...

Documents

data mining lectures

specific web server

web browsing slide

irvine slide

transaction referrer

request originated agent

request spider error

visited web pages entry