Prof. Panos Ipeirotis

Search and the New Economy

Session 2

Web Analytics

Frequency of Access• Frequency of access decreases by distance^2

(Result from traditional library science)

• Result carries over to physical stores

• Result carries over to information environments

Question: How do we understand how users/customers access what we offer?

Data-driven Decisions:Testing, testing, testing, testing

• Common scenario: – Boss, designer, employee “knows what works best” (for him/her)– Boss, designer, employee wants to do site design

• Common error: Think that we know what customers want

• 80% of the time we are wrong about what a customer wants or expects from site experience (e.g., consider reasons to visit Amazon.com)

• The only truth: Constant experimentation and testing improves customer “relevancy” and improves conversion

The objectives of this session

• Web analytics for own website– What customers look at– Where they come from– How to engage them

• Web analytics for monitoring competitors– How customers behave in general– Why they go to competitors– What they do there

Outline of today’s classHow customers behave in our own website• Micro / In-site / Quantitative (What)

– Eyetracking– Clickstreams, log analysis

• Meso / In-site / Qualitative (Why)– Surveys– Lab-usability tests– In-situ tests

How customers behave in other websites• Macro / Across-site

– Panel data (ComScore, Alexa)– ISP measurements (HitWise)– Search engine data (Google Trends, Microsoft adCenter)

Eyetracking monitors

Eyetracking studies

Eye Tracking Studies

• Golden Triangle– Top left corner– Not only in search engines

• Quick scan– For candidate

• Longer scan– For relevance

Analysis of Washington Post

Eyetracking (again)

•FDIC distrusts us * No Bank Quality * Will Lose Value •Not ready to event an insurance? Tax group of our manager discussion free of funds. •Get $25 to close an E*Trade Bank Money Market Plus Advice! Tax a gear cool and ATM access!

Eyetracking: The F-pattern

• Users don't read text thoroughly

• The first two paragraphs must state the most important information.

• Start subheads, paragraphs, and bullet points with information-carrying words

• Merge “foreign content” with page information

Clickstream / Log Analysis

• Eyetracking studies are limited to a lab

• Often we need to analyze how users behave when visiting our site

Sources of click data

• Web Server Logs Pros: You own the data, Capture search engine visits Cons: Difficult to customize, Misses cached requests

• Web Beacons (1x1 pixel images) Pros: Easy to add Cons: Bad reputation, often blocked

• Javascript Tags Pros: Capture real visitors, Customizable Cons: ~5% of users have JavaScript off

Assignment 2: Use Google Analytics (Javascript-based) to capture user behavior on your website

Foundational Metrics• Visitors / Unique visitors

– Pay attention on definition of “unique” (cookie? date? IP?)

• Time on site– Tricky! Should consider the goal of

the site• Page views

– Good for content/brand sites– Unclear for other sites– Increasingly outdated

(blogs, Gmail, Flash, dynamic content)• Bounce rate

– Reveals real visitors– % of single page visits– (or % of <5 second visits)

Segment, segment, segment!

Goal and Lead Metrics• “Unique Visitors” tends to be THE

metric to follow, BUT instead:

• Set up goals and measure conversion rate and goal value (SettingsEditGoal)

• Segment by:– Referring sites– Search engines + Keywords– AdWords campaigns

• Analyze for leads!– “Wikipedia referrals are more engaged and have low bounce rate”– Use Microsoft AdCenter Labs to analyze demographics (will get back to this)

Content Metrics• Top content

– Why users are coming– What they are looking for

• Top landing (entry) pages– First impression!– Polish and direct users to goals

• Click density analysis– Use CrazyEgg.com

• Funnel analysis– In multi-page processes, where users abandon? – Mortgage application at Agency.com move personal information form later– Abandoned purchases at Lane Bryant offer free shipping

Click Density Analysis

Click OverlayClick Heatmap

Where users click, and which users click in each link

Your own experience?

• Questions?

• Anything that you would like to add?

• Lessons from practical experience?

Redesign and Experimentation

• After detecting problems or opportunities:1. Make a hypothesis2. Redesign3. Test for performance(Common error: Skipping step 1)

Two common approaches for testing• A/B testing• Multivariate testing

A/B testing

Version A Version B

Run versions A and B and see which improves

the target performance indicator

Image on the left“add to shopping cart” bottom right

Image on the right“add to shopping cart” top left

Multivariate TestingModularize page and test variations for each module

(see Google Website Optimizer, Offermatica, Optimost, SiteSpect, Kefta, …)

Headline

Image Text

Call to action

Multivariate Testing

3 different headlines

3 different images

…

Examine Results

Example: Dale & Thomas• Popcorn company

• Variables:– Main layout– Order area headline (6)– Order area image (6)– Order area button – Popcorn flavors image (4)– “Free shipping” – “Sign-up for mailings”→ 1.9 million variations possible

+13% in sales, within a month

OK, we optimized our own site

• Is +13% good?• …or lagging behind the competitors?

• What types of customers go to our competitors?• Why?• …

Outline of today’s classHow customers behave in our own website• Micro / In-site / Quantitative (What)

– Eyetracking– Clickstreams, log analysis

• Meso / In-site / Qualitative (Why)– Surveys– Lab-usability tests– In-situ tests

How customers behave in other websites• Macro / Across-site / Competitive Intelligence

– Panel data (ComScore, Alexa)– ISP measurements (HitWise)– Search engine data (Google Trends, Microsoft adCenter)

Need for Competitive Data

• Understand how competitors perform

• How competitors get visitors

• Where customers go after visiting competitor’s site

• Demographics

ISP-Based Data (HitWise)

• Benefits– Big sample size (~25M users)– Captures all types of traffic– Good for relatively small sites as well (~100K visitors)

• Concerns– (Relative) lack of depth of analysis– Lack of purchase / payment data (no https logging)

Anonymous data, bought from multiple Internet Service Providers

HitWise: Upstream and Downstream

HitWise: Industry Statistics

HitWise: Keyword Statistics

Panel-Based Data (Alexa)

• Benefits– Free– Large sample size (>20M)

• Concerns– Percentage reporting (not absolute values)– Self-selection bias due to targeting (webmasters?)

• Useful mainly for comparing similar sites

Users install toolbar and agree to have their traffic anonymously

monitored

Panel-Based Data (ComScore)

• Benefits– Detailed demographics for users– Provides conversion rates and purchases– 100% of traffic

• Concerns– Sample size (relatively) small, 100K-2.5M– Sample selection bias due to incentives– Mainly home usage, no work-based (but also avoid

double counting?)– Not good for sites with less than 1M visitors

Recruits users who agree to have their traffic monitored, in exchange

for payment and benefits

What service would you use? Why?