WebPage Summarization Using

Clickthrough Data

JianTao Sun & Yuchang Lu , TsingHua University, China

Dou Shen & Qiang Yang, HK University of Science & Technology

HuaJun Zeng & Zheng Chen , Microsoft Research Asia

SIGIR 2005

2

INTRODUCTION The clickthrough data of Web search engine can be

used to improve Web-page summarization.

The clickthrough data contain many users’ knowledge on Web pages' contents.

The query-word set of a page may cover the multiple topics of the target Web page.

3

INTRODUCTION Problems:

Web pages may have no associated query words since they are not visited by any Web users through a search engine.

the clickthrough data are often very noisy. By using ODP website (http://dmoz.org/), we can buil

d a thematic lexicon to solve these problems. Adapt two text-summarization methods to summa

rize Web pages: The first approach is based on significant-word selection

adapted from Luhn's method . The second method is based on Latent Semantic Analysis

(LSA) .

4

RELATED WORK Amitay and Paris first extracted the text segments c

ontaining a link to a Web page. They then chose the most accurate sentence from the text segments as the snippet of the target page

Delort et al. proposed two enhanced Web-page summarization methods using hyperlinks

These methods directly obtain their knowledge from hyperlinks which may be sparse and noisy

5

RELATED WORK Sun et al. proposed a CubeSVD approach to utilizing

the clickthrough data for personalized Web search .

Liu et al. proposed a technique for categorizing Web-query terms from the clickthrough logs into pre-defined subject taxonomy based on their popular search interests.

Hulth et al. proposed to extract keywords using domain knowledge .

6

Adapted Significant Word (ASW) Method In Luhn's method, each sentence is assigned a signif

icance factor and the sentences with high significance factors are selected to form the summary.

A set of significant words are selected according to word frequency in a document. That is, those words with frequency between high-frequency cutoff and low-frequency cutoff are selected as significant words.

7

Adapted Significant Word (ASW) Method Compute the significance factor :

(1) Set a limit L for the distance at which any two significant words could be considered as being significantly related.

(2) Find out a portion in the sentence that is bracketed by significant words not more than L non-significant words apart.

(3) Count the number of significant words contained in the portion and divide the square of this number by the total number of words within the portion.

The result is the significance factor of each sentence. Modify this method to use both the local contents of

a Web page and query terms collected from the clickthrough data

8

Adapted Significant Word (ASW) Method

After the significance factors for all words are calculated, we rank them and select the top N% as significant words. Then we employ Luhn's algorithm to compute the significance factor of each sentence.

Where tfip and tfi

q denote frequencies of the i-th term in the local text content of a Web page and in the query set respectively.α is a trade-off parameter when combining the two significance measurements.

(1)

9

Adapted Latent Semantic Analysis (ALSA) Method Suppose that there are m distinct terms in a n docum

ent collection. The corpus can be represented by a term-document matrix X Rmxn, whose component xij is the weight of term ti in document dj . The Singular Value Decomposition (SVD) of X is given by:

X = UΣVT (2)

In Equation 2, U and V are the matrices of the left and right singular vectors. Σ is the diagonal matrix of singular values.

10

Adapted Latent Semantic Analysis (ALSA) Method LSA approximates X with a rank-k matrix:

by setting the smallest r - k singular values to zero (r is rank of X). That is, the documents are represented in the k dimensional space spanned by column vectors of Uk

(3)

11

Extraction based summarization algorithm Gong et al. proposed an extraction based summarizatio

n algorithm.

Firstly, a term-sentence matrix is constructed from the original text document.

Next, LSA analysis is conducted on the matrix. In the singular vector space, the i-th sentence is represented by the column vector

of VT .

Each element in measures the importance factor of this sentence on the corresponding latent concept.

12

Extraction based summarization algorithm In the last step, a document summary is produced

incrementally. For the most important concept, the sentence having the largest importance factor is selected into the summary.

Then, the second sentence is selected for the next most important concept. This procedure repeated until a predefined number of sentences are selected.

13

Our LSA-based summarization method Our LSA-based summarization method is a variant of

Gong's method.

We utilize the query-word knowledge by changing the term-sentence matrix:

if a term occurs as query words, its weight is increased according to its frequency in query word collection. In this approach, we expect to extract sentences whose topics are related to the ones reflected by query words.

14

Our LSA-based summarization method According to experiments, the weighting and

normalization schemes lower the summarization performance.

Thus, a term frequency (TF) approach without weighting or normalization is used to represent the sentences in Web pages. Terms in a sentence are augmented by query terms as follows:

(4)

β is a parameter used to tune the weights of query terms.is the frequency of term i in a sentence, while denotes term fre

quency in query set.

15

Summarize Web Pages Not Covered by Clickthrough Data According to statistics, only 23.1% out of the crawled

ODP pages (in English) was browsed and associated with query words. So we build a hierarchical lexicon using the click-through data and apply it to help summarize those pages.

We use TS(c) to represent a set of terms associated with category c. Thus the thematic lexicon is a set of TS, which are organized using the ODP category structure.

16

Summarize Web Pages Not Covered by Clickthrough Data The lexicon is built as follows:

First, TS corresponding to each category is set empty. Next, for each page covered by the clickthrough data, its

query words are added into TS of categories which this page belongs to as well as all its parent categories.

When a query word is added into TS, its frequency is added to its original weight in TS. If a page belongs to more than one category, its query terms will be added into all TS associated with all its categories.

At last, term weight in each TS is multiplied by its Inverse Category Frequency (ICF). The ICF value of a term is the reciprocal of its frequency occurring in different categories of the hierarchical taxonomy.

17

Summarize Web Pages Not Covered by Clickthrough Data For each page to be summarized, we first look up the

lexicon for TS according to the page's category. Then the summarization methods proposed previous

are used. Weights of the terms in TS can be used to select significant words or to update the term-sentence matrix.

If a page to be summarized has multiple categories, the corresponding TS are merged together and weights are averaged.

When a TS does not have sufficient terms, TS corresponding with its parent category is used.

18

EXPERIMENTS Data Set

A set of Web pages of the ODP directory are crawled. After removing those which belong to “\World" and “\Regional" categories, we got 1,125,207 Web pages, 260,763 of which are clicked by Web users using 1,586,472 different queries.

DAT1: It consists of 90 pages selected from the 260,763 browsed pages.

DAT2: We preprocess the 260,763 pages and extract description (\DESCRIPTION" metadata) of each page and keep the pages with a description of over 200 characters and containing at least 10 sentences, from which 10,000 pages are randomly selected.

19

EXPERIMENTS Ideal summary DAT1

Three human evaluators were employed to summarize these pages. Each evaluator was requested to extract the most important sentences for a Web page. There is no constraint on the number of sentences to be extracted.

DAT2 We use the descriptions provided by the page editors as the

ideal summary.

20

EXPERIMENTS ROUGE Evaluation (Recall-Oriented

Understudy for Gisting Evaluation) ROUGE is a software package that measures

summarization quality by counting overlapping units such as the n-gram, word sequences, and word pairs between the candidate summary and the reference summary. ROUGE-N is an n-gram recall measure which is defined as follows:

21

EXPERIMENTS N stands for n-gram, Countmatch(gramn) is the maximu

m number of n-grams co-occurring in the candidate summary and the reference summary, Count (gramn) is the number of n-grams in the candidate summary.

In this paper, we only reported ROUGE-N where N=1.

22

EXPERIMENTSResults on DAT1

23

EXPERIMENTS

24

EXPERIMENTS

Results on DAT1 without queries

25

EXPERIMENTS

26

EXPERIMENTSResults on DAT2

27

Discussions Experiments indicate the clickthrough data are helpful

for generic Web-page summarization and both our proposed methods can leverage this knowledge source well.

When the Web pages are not covered by the click-through data, the lexicon-based approach achieves better results than pure-text-based summarizers.

The thematic lexicon built from clickthrough data can discover the topic terms associated with a specific category and the ICF-based approach can effectively assign weights to terms of this category.

28

CONCLUSIONS We leverage extra knowledge from clickthrough data

to improve Web-page summarization. Two extract-based methods are proposed to produce generic Web-page summaries.

For the pages not covered by the click-through data, we build a thematic lexicon using the clickthrough data in conjunction with an hierarchical Web directory.

The experimental results show that significant improvements are achieved compared with summarizers without using clickthrough logs.

29

FUTURE WORK Automatically determine trade-off parameter.

Study how to leverage other types of knowledge,

such as word clusters and thesaurus, hidden in the clickthrough data.

Plan to evaluate our methods using extrinsic evaluation metrics and much larger data sets.