document clustering and collection selection diego puppin web mining, 2006-2007

Document Clustering and Collection

SelectionDiego Puppin

Web Mining, 2006-2007

Web Search Engines: Parallel Architecture

To improve throughput, latency, res. quality A broker works as a unique interfaceComposed of several computing serversQueries are routed to a subset The broker collects and merges results

d1d1 d2d2 d3d3 d4d4 d5d5 d6d6 d7d7 d8d8t1t1 x xt2t2 x xt3t3 x x xt4t4 x xt5t5 x x x x x x xt6t6 x x xt7t7 x x x xt8t8 x x xt9t9

d1d1 d2d2 d3d3 d4d4 d5d5 d6d6 d7d7 d8d8t1t1 x xt2t2 x xt3t3 x x xt4t4 x xt5t5 x x x x x x xt6t6 x x xt7t7 x x x xt8t8 x x xt9t9

d1d1 d2d2 d3d3 d4d4 d5d5 d6d6 d7d7 d8d8t1t1 x xt2t2 x xt3t3 x x xt4t4 x xt5t5 x x x x x x xt6t6 x x xt7t7 x x x xt8t8 x x xt9t9

Doc-Partitioned Approach

The document base is split among serversEach server indexes and manages queries for its own documentsIt knows all terms of some documentsBetter scalability of indexing/search

Each server is independentDocuments can be easily added/removed

Term-Partitioned Approach

The dictionary is split among serversEach server stores the index for some termsIt knows documents where its terms occurPotential for load reductionPoor load balancing (some work...)

Some considerations

Every time you add/remove a docYou must update MANY servers

With queries:only relevant servers are queried but... servers with hot terms are overloaded

How To Load BalancePut together related terms

This minimizes the number of hit servers

Try to put together group of documents with similar overall frequence

Servers shouldn’t be overloadedQuery logs could be used to predict

Multiple indexesTerm-based index

Query-vector or Bag-of-wordHot text index

Titles, Anchor, Bold etcLink-based index

It can find related pages etcKey-phrase index

For some idioms

terms links hot

query terms

referrer, relatedquery terms

How To Doc-Partition?

1. Random doc assignment + Query broadcast

2. Collection selection for independent collections (meta-search)

3. Smart doc assignment + Collection selection

4. Random assignment + Random selection

1. Random + Broadcast

Used by commercial WSEsNo computing effort for doc clusteringVery high scalability

Low latency on each serverResult collection and merging is the heaviest part

IR Core1idx

IR Core2idx

IR Corekidx

t1,t2,…tq r1,r2,…rr

query results

Broker

Distributed/Replicated Documents

2. Independent collections

The WSE uses data from several sourcesIt routes the query to the most authoritative collection(s)It collects the results according to independent ranking choices (HARD)Example: Biology, News, Law

Coll1 Coll2 Coll.k

t1,t2,…tq r1,r2,…rr

query results

Broker

3. Assignment + Selection

WSE creates document groupsEach server holds one groupThe broker has a knowledge of group placementThe selection strategy routes the query suitably

Doc cluster1idx

Doccluster2idx

Doccluster kidx

t1,t2,…tq r1,r2,…rr

query results

Broker

4. Random + Random

If data (pages, resources...) are replicated, interchangeable, hard to indexData are stored in the server that publishes themWe query a few servers hoping to get something

Doc 1idx

Doc2idx

Dockidx

t1,t2,…tq r1,r2,…rr

query results

Broker

CORI

The Effect of Database Size Distribution on Resource Selection Algorithms, Luo Si and Jamie CallanExtends the concept of TF.IDF to collections

CORINeeds a deep collaboration from the collections:

Data about terms, documents, sizeUnfeasible with independent collections

Statistical sampling, Query-based sampling

Term-based: no links, no anchorsVery large footprint

Querylog-based Collection SelectionUsing Query Logs to Establish Vocabularies in Distributed Information RetrievalMilad Shokouhi; Justin Zobel; S.M.M. Tahaghoghi; Falk ScholerThe collections are sampled using data from a query logBefore: Queries over a dictionary (QBS)

RecallNumber of found documents (not for web)

Precision at X (P@X)Number of relevant documents out of the first X results (X= 5, 10, 20, 100)

Average precisionAverage of Precision for increasing X

MAP (Mean Average Precision)The average over all queries

And now for something completely

different!

Important features

It can use any underlying WSELinks, snippets, anchors...

Good or bad as the WSE it uses!Small footprint

It can be added as another index

terms links hot query-vector

query terms

referrer, relatedquery terms

query mapping

DevelopmentsQuery suggestions

The system finds related queriesResult grouping

Documents are already organized into groups

Query expansionCan find more complex queries still matching

d1d1 d2d2 d3d3 d4d4 d5d5 d6d6 d7d7 d8d8Q1Q1 x xQ2Q2 x xQ3Q3 x x xQ4Q4 x xQ5Q5 x x x x x x xQ6Q6 x x xQ7Q7 x x x xQ8Q8 x x xQ9Q9

d1d1 d2d2 d3d3 d4d4 d5d5 d6d6 d7d7 d8d8Q1Q1 x xQ2Q2 x xQ3Q3 x x xQ4Q4 x xQ5Q5 x x x x x x xQ6Q6 x x xQ7Q7 x x x xQ8Q8 x x xQ9Q9

d1d1 d2d2 d3d3 d4d4 d5d5 d6d6 d7d7 d8d8Q1Q1 x xQ2Q2 x xQ3Q3 x x xQ4Q4 x xQ5Q5 x x x x x x xQ6Q6 x x xQ7Q7 x x x xQ8Q8 x x xQ9Q9

Still missingUsing the QV model as a full IR model:

Is it possible to perform queries over this representation?

New query terms cannot be foundCORI should be used for unseen queries

Better testing with topic shift

Possible seminars/projects

Advanced collection selectionStatistical sampling, Query-based sampling

Partitioning a LARGE collection (1 TB)Load balancing for doc- and term-partitioningTopic shift

Query log analysis