yahoo making the web searchable
TRANSCRIPT
![Page 1: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/1.jpg)
Making the Web Searchable
Peter Mika
Researcher, Data Architect
Yahoo! Research
![Page 2: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/2.jpg)
- 2 -
Yahoo! Research (research.yahoo.com)
![Page 3: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/3.jpg)
- 3 -
Yahoo! Research Barcelona
• Established January, 2006
• Led by Ricardo Baeza-Yates
• Research areas
– Web Mining
• content, structure, usage
– Distributed Web retrieval
– Multimedia retrieval
– NLP and Semantics
![Page 4: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/4.jpg)
- 4 -
Yahoo! by numbers (April, 2007)
• There are approximately 500 million users of Yahoo! branded services, meaning we reach 50 percent – or 1 out of every 2 users – online, the largest audience on the Internet (Yahoo! Internal Data).
• Yahoo! is the most visited site online with nearly 4 billion visits and an average of 30 visits per user per month in the U.S. and leads all competitors in audience reach, frequency and engagement (comScore Media Metrix, US, Feb. 2007).
• Yahoo! accounts for the largest share of time Americans spend on the Internet with 12 percent (comScore Media Metrix, US, Feb. 2007) and approximately 8 percent of the world’s online time (comScore WorldMetrix, Feb. 2007).
• Yahoo! is the #1 home page with 85 million average daily visitors on Yahoo! homepages around the world, an increase of nearly 5 million visitors in a month (comScore WorldMetrix, Feb. 2007).
• Yahoo!’s social media properties (Flickr, delicious, Answers, 360, Video, MyBlogLog, Jumpcut and Bix) have 115 million unique visitors worldwide (comScore WorldMetrix, Feb. 2007).
• Yahoo! Answers is the largest collection of human knowledge on the Web with more than 90 million unique users and 250 million answers worldwide (Yahoo! Internal Data).
• There are more than 450 million photos in Flickr in total and 1 million photos are uploaded daily. 80 percent of the photos are public (Yahoo! Internal Data).
• Yahoo! Mail is the #1 Web mail provider in the world with 243 million users (comScore WorldMetrix, Feb. 2007) and nearly 80 million users in the U.S. (comScore Media Metrix, US, Feb. 2007)
• Interoperability between Yahoo! Messenger and Windows Live Messenger has formed the largest IM community approaching 350 million user accounts (Yahoo! Internal Data).
• Yahoo! Messenger is the most popular in time spent with an average of 50 minutes per user, per day (comScore WorldMetrix, Feb. 2007).
Nearly 1 in 10 Internet users is a member of a Yahoo! Groups (Yahoo! Internal Data).• Yahoo! is one of only 26 companies to be on both the Fortune 500 list and the Fortune’s “Best Place to Work”
List (2006).
![Page 5: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/5.jpg)
- 5 -
Agenda
• The Annotated Web
• SearchMonkey – Demo
– Technology• DataRSS format
• Query language
• Lessons learned
• Toward Semantic Search
• BOSS– Build your Own Search Service
• Y!OS 1.0– Yahoo! Open Strategy
![Page 6: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/6.jpg)
The Annotated Web
![Page 7: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/7.jpg)
- 7 -
Previously in search
• Horizontal search– Yahoo…– Keyword-based indexing– Minimal natural language processing– Limited experiments with ontologies (query expansion)
• Vertical search– e.g. shopping.com, Kelkoo– Faceted search, browsing– Fixed ontology
• Combinations– Google Base, Google Co-op
• Web-scale, but fixed ontologies• Proprietary technology
• Can we do better with the Semantic Web?– Address the long tail of queries (88% of queries) – Use standard technology
• Not a new question. But the answer may be new.
![Page 8: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/8.jpg)
- 8 -
Which Semantic Web?
• Two visions
– Data Web
• Bringing the content of databases to the Web (linkeddata.org)
• Rich data, heavyweight semantics
• Deep Web
– Annotated Web
• Annotating the content of Web resources (documents, mm)
• Simple data, lightweight semantics
• Shallow Web
• This presentation is about the Annotated Web.
![Page 9: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/9.jpg)
- 9 -
Brief history of the Annotated Web
• 1995: HTML meta tags• 1996: Simple HTML Ontology Extensions (SHOE)• 1998: RDF/XML
– RDF/XML in HTML– RDF linked from HTML
• 2003: Web 2.0– Tagging– Microformats– Metadata in Wikipedia– Machine tags in Flickr
• 2005: eRDF • 2008: RDFa
![Page 10: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/10.jpg)
- 21 -
RDF-based annotation #1: eRDF
• eRDF – Ian Davis (Talis)
– Embedding RDF in HTML• Straightforward mapping to RDF triples (XSLT available)
• HTML4 compatible
– More complex than microformats• Use any RDF/OWL vocabulary
• Reuse of semantic-bearing HTML elements is limited
– More limited than RDF• No blank nodes
• No data types
• No statements about subjects other than the current document
– Limited usage
![Page 11: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/11.jpg)
- 22 -
RDF-based annotation #2: RDFa
• RDFa– World Wide Web Consortium (W3C) last call document
– Similar intent as eRDF, but full RDF support
• Requires XHTML
– Big question: user complexity ( data quality)
<p typeof="contact:Info" about="http://example.org/staff/jo">
<span property="contact:fn">Jo Smith</span>.
<span property="contact:title">Web hacker</span> at
<a rel="contact:org" href="http://example.org"> Example.org </a>. You can contact me <a rel="contact:email" href="mailto:[email protected]">
via email </a>.
</p> ...
![Page 12: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/12.jpg)
SearchMonkey
![Page 13: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/13.jpg)
- 24 -
• Creating an ecosystem of publishers, developers and end-users – Motivating and helping publishers to implement semantic
annotation
– Providing tools for developers to create compelling applications
– Focusing on end-user experience
• Rich abstracts as a first application
• Addressing the long tail of query and content production
• Standard Semantic Web technology– dataRSS = Atom + RDFa
– Industry standard vocabularies
• http://developer.yahoo.com/searchmonkey/
SearchMonkey
![Page 14: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/14.jpg)
- 25 -
Before After
an open platform for using structured data to build more useful and relevant search results
What is SearchMonkey?
![Page 15: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/15.jpg)
- 26 -
image
deep links
name/value pairs or
abstract
Enhanced Result
![Page 16: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/16.jpg)
- 27 -YAHOO! CONFIDENTIAL | 27
Infobar
![Page 17: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/17.jpg)
- 28 -
Acme.com’sdatabase
Index
RDF/Microformat Markup
site owners/publishers share structured data with Yahoo!. 1
consumers customize their search experience with Enhanced Results or Infobars
3
site owners & third-party developers build SearchMonkey apps.2
DataRSS feed
Web Services
Page Extraction
Acme.com’s Web Pages
SearchMonkey
![Page 18: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/18.jpg)
- 29 -
Developer tool
![Page 19: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/19.jpg)
- 30 -
Developer tool
![Page 20: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/20.jpg)
- 31 -
Developer tool
![Page 21: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/21.jpg)
- 32 -
Developer tool
![Page 22: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/22.jpg)
- 33 -
Developer tool
![Page 23: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/23.jpg)
- 34 -
Gallery
![Page 24: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/24.jpg)
- 35 -
Example apps
– hCard plus feed data
• Creative Commons by Ben Adida
– CC in RDFa
![Page 25: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/25.jpg)
- 36 -
Example apps. II.
• Other me by Dan Brickley
– Google Social Graph API wrapped using a Web Service
![Page 26: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/26.jpg)
- 37 -
DataRSS
<?profile http://search.yahoo.com/searchmonkey-profile ?><feed xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2005/Atom ../latest/xsd/datarss.xsd“><id>http://www.linkedin.com/datarss/</id><author>
<name>Peter Mika ([email protected])</name></author><title>Example data feed for social</title><updated>2007-11-14T04:05:06+07:00</updated><entry>
<!-- title field of entry is not used for anything --><title>Peter Mika</title><!--URL of the webpage extracted from --><id>http://www.linkedin.com/ppl/webprofile?id=5054019</id><updated>2007-11-14T04:05:06+07:00</updated><content type="application/xml">
<y:adjunct version="1.0" name="social-simple" xmlns:y="http://search.yahoo.com/datarss/"> <y:item rel="dc:subject"> <y:type typeof="foaf:Person"> <y:meta property="foaf:name">John Doe</y:meta> <y:meta property="foaf:gender">male</y:meta> <y:item rel="foaf:homepage" resource="http://www.joeisageek.com"/> <y:item rel="foaf:mbox" resource="mailto:[email protected]"/> <y:item rel="foaf:weblog" resource="http://johnblog.example.org"/> <y:item rel="foaf:knows"> <y:type typeof="foaf:Person"> <y:meta property="foaf:name">Jane Doe</y:meta> <y:meta property="foaf:gender">female</y:meta> <y:item rel="foaf:mbox" resource="mailto:[email protected]"/> </y:type> </y:item> </y:type> </y:item></y:adjunct>
</entry>
</feed>
Atom 1.0
XML + RDFa
![Page 27: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/27.jpg)
- 38 -
The data part
<adjunct version="1.0" id=“com.yahoo.page.rdfa" xmlns="http://search.yahoo.com/datarss/“ updated=“2007-11-14T04:05:06+07:00”> <item rel="dc:subject"> <type typeof="foaf:Person"> <meta property="foaf:name">John Doe</meta> <meta property="foaf:gender">male</meta> <item rel="foaf:homepage" resource="http://www.joeisageek.com"/> <item rel="foaf:mbox" resource="mailto:[email protected]"/> <item rel="foaf:weblog" resource="http://johnblog.example.org"/> <item rel="foaf:knows"> <type typeof="foaf:Person"> <meta property="foaf:name">Jane Doe</meta> <meta property="foaf:gender">female</meta> <item rel="foaf:mbox" resource="mailto:[email protected]"/> </type> </item> </type> </item></adjunct>
![Page 28: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/28.jpg)
- 39 -
DataRSS
• An Atom extension for structured data
• Why a new format?
– A feed format is required by publishers
• Exclusive content (e.g. partnerships, paid inclusion)
• No changes necessary to the web page
• No standard named graph format for the Semantic Web
– Needed to capture meta-metadata such as source and timestamp of information
– Not really a new format
• An Atom extension
• Use any RDFa parser to get the triples out
• cf. Google Base feeds
![Page 29: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/29.jpg)
- 40 -
What happened since the launch?
• It’s starting to work!
– Click rates improve Publishers are willing to invest More structured data More applications More users Click rates improve
• Increasing excitement all around
– Standardization of RDFa is bringing new energy
– Good market for companies that help publishers to ‘semantify’ or support developers in extracting structured from web pages
• OpenCalais, Dapper, AdaptiveBlue, Intel MashMaker, Zemanta…
• There have been some lessons learned…
![Page 30: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/30.jpg)
- 41 -
Data quality
• Publishers/developers want the quick and dirty answer, not the long and clean one
• Resource or literal?– <meta property=“vcard:url”>http://www.example.org</meta>
– <meta property=“vcard:tel”>0034691792522</meta>
• Webpage or resource?– Should we allow a resource have the same URI as an existing
webpage?
– This is the default in eRDF/RDFa!• <div class=“foaf-name”>Peter Mika</div>
• Types vs. datatypes– <meta property=“use:email”>[email protected]</meta>
• Extensibility– rdfs:movies
Complexity of the formalism = Data quality down
![Page 31: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/31.jpg)
- 42 -
Vocabularies
• Coverage is small
– Books, movies, stuff people care about…
• Competing proposals
– Versions floating around
• Not maintained
– I cannot maintain your vocabulary for you
• Vocabularies for microformats
– A must
• The role of the W3C
– Ontologies as member submission….
• Vocabularies not designed for the annotated Web
Distributed ontology development = Mess
![Page 32: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/32.jpg)
- 43 -
eRDF
• Difficult for complex pages and dangerous in non-expert hands
– Serious limitations
• No datatypes
• No subjects other than identifiers within the current page
• Reuse of the id attribute
<div id=“blue_panel”>
<span class=“foaf-name”>Peter Mika</span>
<div id=“white_clickable_thingy_on_the right”>
<span class=“foaf-mbox”>[email protected]</span>
….
![Page 33: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/33.jpg)
- 44 -
RDFa
• A huge improvement– E.g. no repurposing of HTML attributes
• Still, not everything is intuitive to the uninitiated:
<div about=“#id”><span property=“foaf:name“>Peter Mika</span><span rel=“foaf:img“ typeof=“foaf:Image”>
<span property=“dc:format”>jpg</span>…
</span</div>
<div about=“#id”><span property=“foaf:name“>Peter Mika</span><span rel=“foaf:img“ resource=“http://www.example.org/photo.jpg”>
<span typeof=“foaf:Image”> <span property=“dc:format”>jpg</span></span
</span</div>
![Page 34: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/34.jpg)
Semantic Search
![Page 35: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/35.jpg)
- 46 -
Semantics and IR
• Hard searches that cannot be solved with purely syntactical approach– Ontologies in IR shown to work in limited domains
– In Web IR most attempts (e.g. query expansion) have failed
• What is new? The scale and breadth.– Growth in annotations, all domains (Web 2.0)
– Data Web vs. Deep Web
![Page 36: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/36.jpg)
- 47 -
Hard searches
• Ambiguous searches– Paris Hilton
• Multimedia search– Images of Paris Hilton
• Imprecise or overly precise searches – Publications by Jim Hendler– Find images of strong and adventurous people (Lenat)
• Searches for descriptions– Search for yourself without using your name– Product search (ads!)
• Searches that require aggregation– Size of the Eiffer tower (Lenat)– Public opinion on Britney Spears
• Queries that require a deeper understanding of the query, the content and/or the world at large
– Note: some of these are so hard that users don’t even try them any more
![Page 37: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/37.jpg)
- 48 -
Example…
![Page 38: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/38.jpg)
- 49 -
Application: query intent
Paris Hilton is a person!
![Page 39: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/39.jpg)
- 50 -
Application: query intent #2
Hugo is a person!
![Page 40: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/40.jpg)
- 51 -
Time to experiment!
• Future work in IR– Ranking documents– Ranking resources/triples
• Ranking resources in the presence of documents
– New interfaces• Query interface• Result presentation
• Metrics– Precision/recall– CTR– ROI
![Page 41: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/41.jpg)
- 52 -
Time to experiment! #2
• Future work in Semantic Web– (Semi-)automated ways of metadata creation (NLP!)– Entity resolution– Scale– Data quality
• We allow providing metadata for other people’s sites!
– Reasoning• To the extent that it’s useful
• Constraints– Keyword-based search can not suffer
• Preference for very conservative solutions
– Metadata in the context of documents• Assumption is still that the user will want to see/visit the source• (what to do with linked data?)
![Page 42: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/42.jpg)
- 53 -
Going even further…
• NLP, Information Extraction
• New types of application– Aggregators
– Stateful apps
– Intent-driven apps
– Mobile apps
– …
• Beyond search– Analysis, design, diagnosis etc. on top of aggregated data
• Personalization– Building rich user profiles
– Re-ranking results based on interests
• Monetization– No more “buy virgins on eBay”
![Page 43: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/43.jpg)
- 54 -
BOSS: Build your Own Search Service
• Unlimited queries per day
• Ability to re-order results and blend-in addition content
• No restrictions on presentation
• No branding or attribution
• Access to multiple verticals (web search, image, news)
• Ability to monetize
• 40+ supported language and region pairs
• http://developer.yahoo.com/search/boss/
![Page 44: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/44.jpg)
- 55 -
Y!OS 1.0
• Yahoo! Open Strategy
– Yahoo! Social Platform
• Profiles, Connections, Updates, Contacts and Status
– Yahoo! Query Language (YQL)
• Access (other) web services using a SQL-like language
– Yahoo! Application Platform (YAP)
• Developer hosted execution of applications with access to Yahoo's Social APIs and YQL;
• Support for OpenSocial's JavaScript API; and
• Support for server-side YML tags.
• Future: run applications on Y! sites
• OpenID, OAuth
![Page 45: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/45.jpg)
- 56 -
Contact
• Peter Mika
– Come to Barcelona and stop by
• SearchMonkey
– developer.yahoo.com/searchmonkey/
– mailing lists
– forums
• http://suggestions.yahoo.com/searchmonkey
• http://suggestions.yahoo.com/datarss
– Semantic Web FAQ
• http://devel.yahoo.com/searchmonkey/smguide/faq.html
![Page 46: Yahoo Making The Web Searchable](https://reader033.vdocument.in/reader033/viewer/2022052821/554920d2b4c9059f4c8b7e5a/html5/thumbnails/46.jpg)
- 57 -
the monkey is out!