1

Web Search EnvironmentsWeb Crawling Metadata using RDF and Dublin Core

Dave Becketthttp://purl.org/net/dajobe/

Slides:http://ilrt.org/people/cmdjb/talks/tnc2002/

2

Introduction• Overview of SGs and Web Crawling• Why WSE, what’s new? Novel

results• Future work (or stuff we didn’t do)

and conclusions

3

Overview• Digital Library community• In UK, subject-specific gateways (SGs)• Want to improve: scope (more),

timeliness (fresh), cost (less)• Stay professional – the Quality word• Compete with web search engines – the

Google Test

4

Human Cataloguing of the Web• Pros: High quality, domain

knowledge selection, subject-specialised, cataloguing done to well-known and developed standards

• Cons: Expensive, slow, descriptions need to be reviewed regularly to keep them relevant

5

Software running web crawls• Pros: vastly comprehensive (Con:

too much), can be very up-to-date• Cons: cannot distinguish “this page

sucks” from “this page rocks”, indiscriminate, subject to spamming, very general (but…)

6

Combining Web Crawling and High Quality DescriptionA solution• Seed the web crawl from high quality

records• Crawl to other (presumably) good quality

pages• Track the provenance of the crawled

pages• Provenance can be used for querying and

result ranking

7

Web Search Environments (WSE) Project• Research by ILRT and later

Resource Discovery Network (RDN)• RDN funds UK SGs (ILRT also had

DutchESS)

8

WSE Technologies• Simple Dublin Core (DC) records

extracted from SGs• OAI protocol used to collect these

records in one place (not required)• Combine Web Crawler• RDF framework to connect the

resource descriptions together

9

Simple DC RecordsReally simple:• Title• Description• Identifier (URI of resource)• Source (URI of record)

10

Information model 1• DC records describe all the

resources• Web crawler reads these and

returns crawled web pages• These generate a new web crawled

resource

11

Information model 2• Link back to original record(s), plus

web page properties• RDF model lets these be connected

via page, record URIs• Giving one large RDF graph of the

total information

12

WSE graph

13

Novel Outcomes?It is obvious that:• Metadata gathering is not new

(Harvest)• Web crawling is not new (Lycos)• Cataloguing is not new (1000s of

years)So what is new?

14

WSE – Areas Not FocusedI digress…• Gathering data together – not crucial,

Combine is a distributed harvester• Full text indexing – not optimised• Web crawling algorithm – the routes

through the web were not selected in a sophisticated way

15

WSE – General Benefits• Connecting separate systems (one

less place needed to go)• RDF graph allows more data

mixing (not fragile)• Leverages existing systems

(Combine, Zebra), standards (RDF, DC)

16

WSE – Novel Searching• “game theory napster” – zero hits• Cross-subject searching in one

system – “gmo”• Can navigate resulting provenance

17

WSE – Gains• Web crawling gains from high quality

human description• SGs gain from increase in relevant

pages• Fresher content than human-catalogued

resource• More focused than a general search

engine

18

WSE as a new tool• For subject experts• Which includes cataloguers• Gives fast, relevant search

(no formal precision, recall analysis)

19

WSE – new areas• Cross-subject searching possible in

subjects not yet catalogued, or that fall between SGs

• Searching emerging topics is possible ahead of additions to catalogue standards

• Helps indicate where new SGs, thesauri are needed

20

WSE - deploying• ILRT WSE• RDN WSE• RDN – investigating for the main

search system

21

WSE for SGsIndividual SGs – enhancing subject-

specific searches:• Deep / full web crawling of high quality

sites• Granularity of cataloguing and cost

It is better for humans to describe entire sites (or large parts) and let the software do the detailed work of individual pages

22

Future• Improve and target the crawling• Use the SG information with result

ranking• Add other relevant data to the

graph such as RSS news• A Semantic Web application

23

Questions?• Thank You• Slides:

http://ilrt.org/people/cmdjb/talks/tnc/2002/• Project:

http://wse.search.ac.uk/

24

References• Combine Web Crawler: http://www.lub.

lu.se/combine/ • Dublin Core: http://dublincore.org/ • ILRT: http://ilrt.org/ • RDF: http://www.w3.org/RDF/ • Semantic Web: http://www.w3.org/2001/

sw/