Download - Anatomy of Relevance - From Data to Action: Presented by Saïd Radhouani, Yellow Pages Canada
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
Who we are
• Leading media and marketing solutions company in Canada
• Champion the neighbourhood economy by connecting businesses and consumers
7.3 million Unduplicated unique visitors to our network of websites
6.5 million Downloads of our mobile apps
26% Internet users reached in Canada each month
300 000 Local and national advertisers
Who I am
• Entrepreneur
• Ph.D. in Search and Knowledge Management • IT Director – Content, Search and Relevance
• Noble cause: leverage data to help build smarter organizations
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
User Objective Understand the user intent to maximize their satisfaction Data • Who the user is (individual or group) • Where they come from (direct, SEs, …) • Their context (Location, TOD/DOW/Season, Device, …) • Their interactions history
KPIs • Coverage: %tracked user, %tracked user interactions • Freshness: Daily updates of User profiles & interactions
User
Content Objective Have accurate and “fresh” content to satisfy your audience needs Data • Enriched Directory Content (Merchants, Products, Deals, etc.) • Rich Editorial Content (Neighborhoods, Smart Tips, etc.) • Third-party content (social networks, etc. ) • User-Generated Content
KPIs • Coverage: %Canadian businesses, %Neighborhoods • Completeness: e.g., for each merchant, have all required info • Freshness: Frequency of validation & updates Content User
Knowledge Objective Help to bridge the gap between user queries and content data Data • Linguistic (synonyms, acronyms, etc.) • Rich and “fresh” geo data (location names, polygons, etc.) • Multilingual
KPIs • Coverage: %searched entities (keywords, location names, etc.) • Completeness: e.g., for each location: centroid, polygon,
synonyms, relationships, etc. • Freshness: Frequency of validation & updates
Content Knowledge User
Search Objective Action-driven, personalized and contextualized relevant search experience KPIs Offline • Recall & Precision Online • #returned results • Mean Reciprocal Rank (MRR) – the multiplicative inverse of
the rank of the first correct answer. Perfect SE should have a MRR of 1
Content Knowledge
Search
User
Presentation Objective Allow users to access quickly to actionable content through optimal & dynamic search results presentation e.g., “Movie Theater” vs. “Restaurant” KPIs [that really matter] • Page Views • Unique Visitors • Visit Duration • Bounce Rate • Return Visits Frequency • (Useful) Interactions per Visits Content
Knowledge
Presenta4on Search
User
Optimization
UserSatisfaction improveRelevance() { While (alive) { Act; Measure; Learn; Grow; } }
Content Knowledge
Op4miza4on
Presenta4on Search
User
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
Data-Driven Actions
• Implicit User Feedback (analytics & logs)
• Stable Test Collection
• Side-By-Side Comparison
• A/B Testing
• Explicit User Feedback (Survey)
• Find gaps in data and improve them
• Improve processes
• Improve presentation
• Improve algorithms
• Confirm hypothesis
Measure Act
KPIs
From Data To Action
Examples: • Missing geo-location • Missing businesses • Seasonal trends (adapt search algo)
Manual or automa8c ac8ons
Test Collection KPIs: Recall & Precision • Predefined test collection built manually
• Variety of use cases that cover most users needs
• Evaluate new updates before going to production
Side-By-Side Comparison
• Comparison of two (versions of the) Search Engines
• Pre-defined set of real searches selected from logs
• Pre-classified (keywords, business names, language, geo location, etc.)
• “Blind” comparison
• Significant number of testers per search
A/B Testing
Principle • Experiment one or many ingredients at a
time with real users • Some non-trivial amount of traffic is
always being tested
KPIs • Page Views • Unique Visitors • Visit Duration • Bounce Rate • Return Visits Frequency • Useful Interactions per Visits
Simulation • Run new versions on a significant
search logs sample
• Estimate the impact prior deployment to prod and avoid bad surprises
Relevance In Action
e.g., Bounce Rate – Why the user left quickly? • Who is the user? • Where they come from (direct vs. SE) • Their devices • Which content was provided? • Which page? • etc.
Valuable signals, NOT only for marketing!
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion