Shopping for Top Forums: Discovering Online Discussionfor Product Research

Jonathan L. Elsas∗

Language Technologies InstituteCarnegie Mellon University

Pittsburgh, PA 15213jelsas@cs.cmu.edu

Natalie GlanceGoogle, Inc.

Pittsburgh, PA 15213nglance@google.com

ABSTRACTCommunity generated content, or social media, has becomeincreasingly important over the past several years. Socialmedia sites such as blogs, twitter and online discussion boardshave been recognized as valuable sources of market intelli-gence for companies wishing to keep abreast of their cus-tomers’ attitudes expressed online. There has been littlefocus, however, on providing a similar service to potentialcustomers.

In this paper we present a system for aiding consumerswith their product research by providing access to commu-nity generated content. We focus specifically on online fo-rums or message boards, which are particularly useful forproduct research. These web sites often host discussionamong users with first-hand product experiences, expertusers and enthusiasts.

The system presented here is designed to integrate witha shopping search portal, providing access to online forumsthat are likely to have a significant amount of discussionrelating to a user’s expressed interest in product brands andcategories. We describe this system and present experimentsshowing that in the context of a shopping search engine, theproposed system is preferred or equivalent to results from aweb search engine 80% of the time and achieves accuracy atthe top ranked result of 85%.

Categories and Subject DescriptorsH.3.3 [Information Search and Retrieval]: Miscella-neous

General TermsAlgorithms, Experimentation

KeywordsOnline discussion forums, message boards, product search

∗This research was done while at Google.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.1st Workshop on Social Media Analytics (SOMA ’10), July 25, 2010,Washington, DC, USA.Copyright 2010 ACM 978-1-4503-0217-3 ...$5.00.

1. INTRODUCTIONConsumers researching products for the purposes of mak-

ing purchasing decisions frequently visit online shopping por-tals sites. These sites such as Google Product Search1, BingShopping2 or Yahoo! Shopping3 aggregate many types ofcontent for the consumer: editorial and user reviews, buy-ing guides, and price comparison tools. But missing fromthe current product research landscape is the presence oflarge-scale conversational reviews, such as those found ononline forums and discussion boards. In these sites, fre-quently many authors share their first-hand experiences withproducts, as well as troubleshooting tips, advice, or generaldiscussion.

There are an enormous variety of online forums on theweb, generally topically focused and often cultivating activecommunities of enthusiastic contributors. These types of so-cial media outlets, however, can be difficult to discover byindividuals who may not already be familiar with the com-munity. The current tools to access online forum archivesare lacking, and although web search engines index onlineforum data, many distinguishing characteristics of online fo-rums are ignored by traditional ad-hoc information retrievaltechniques. Additionally, to our knowledge, there are nopublicly available tools to help in identifying forums, ratherthan forum threads or posts.

This paper addresses the task of identifying discussion fo-rums rich with product-related discussion. In these forums apotential shopper may find first-hand reviews, product com-parisons or other user experiences. We approach this taskas an information retrieval problem, ranking forums withrespect to product search related information needs. Thissystem is designed to integrate with a shopping portal toprovide users with access to archives of community gener-ated commentary as well as a forum to interact with expertsand enthusiasts when making purchasing decisions.

The main contribution of this work is on a novel forumranking model (Section 4.3), aimed at identifying online fo-rums with a high density of discussions on product-relatedtopics. This ranking model leverages a rich set of documentannotations: document classifications, identification of thestructure within the forum, annotation of product mentions,and categorization of those mentions to a product ontology.The ranking model achieves greater than 85% precision atthe top ranked result and is preferred or equivalent to webresults restricted to online forum pages 80% of the time.

1http://www.google.com/products2http://www.bing.com/shopping3http://shopping.yahoo.com/

2. MOTIVATION AND TASK DESCRIPTIONThere are three main use cases for online shopping: prod-

uct navigation, browsing and product research. A completeshopping experience must support all three to be successfulas a destination site. Shoppers doing research prior to mak-ing a purchase tap into many kinds of online information,in particular they may seek out editorial or user reviews ofspecific products, buying guides for categories of productsor informal conversational product discussion such as thosefound in message boards.

Message boards, or discussion forums, are an especiallygood place to find product comparisons within a category ofitems, to find expert opinions, and to find first-hand prod-uct experiences. But, these outlets are rarely integrated intoonline shopping sites. Some shopping sites address this bycreating their own set of forums, but these are not necessar-ily successful at attracting the critical mass of expertise to beuseful for aiding shopping decisions. In many cases, there al-ready exists incredibly rich message boards with well estab-lished communities and large archives of product-related dis-cussion. These message boards may be run by product man-ufacturers (such as discussions.apple.com), brand enthusi-asts (such as forums.macrumors.com), independent interestgroups (such as dpreview.com or gpsreview.net) or profes-sional reviewing organizations (such as forums.cnet.com).

We propose to tap into these existing rich outlets of prod-uct discussion by pulling online forum results from the webinto the user interaction flow of the shopping site. In or-der to do this, we must address the questions: when do wechoose to show discussion forum results and what exactlydo we show?

2.1 Information needs in shopping portalsThere are may ways shoppers may express their informa-

tion needs in a shopping search portal, for example by typinga search query into a search box or by clicking on productfacet values to restrict the results show. Let’s consider thefirst question about when to trigger forum results when aquery is entered to a search box. What kind of query fallsinto the product research bucket? Searches for particularitems or for one of a product line, like “HP Laserjet 1020”,can be interpreted as product navigation queries. In thiscase the user’s intent to find information about a particularproduct is clear, and the best result is to provide pricinginformation and reviews for products that match the query.Searches for a broad category of product, like “microwaveoven,” may be interpreted as seeking a browsing entry-pointfor that category. In this case, we can argue that the useris best served by being shown, for example, a set of topbrands, best-selling products and buying guides, or othertools to narrow down the product landscape.

The third bucket of product research queries fall betweenthe specific navigational query and broad product categoryqueries. There are queries like “Bose speaker” or “Applelaptop” where the user has some notion of a relevant cate-gory and brand, but has not yet narrowed down to a spe-cific product or product line. While the existing productportal content such as buying guides, product reviews andprice comparisons are still likely to be helpful, consumersmay also be interested in more informal sources of infor-mation. Online forum discussions can serve this purpose,as rich sources of product comparison information, productsupport, trouble-shooting and informal reviews on a range

of items in the same class. Perhaps equally important, pro-viding access to the right forums not only provides contentlikely to be useful, but also provides access to experts andenthusiasts willing to answer questions.

Thus, we frame our task to be: finding top forums rele-vant to this third bucket of queries, characterized roughly asbrand-category pairs. Although we discuss possibly identi-fying these types of queries from the text entered in a searchbox, there are other ways for a user to express their interestin a in a category-brand pair. For example, a user may selectfact values in a browsing interface in order to limit the dis-played items, as in Figure 1. Or, we may be able to identify

!"#$%&'()*'"+,-."/'-

Figure 1: Example facet value selection in a prod-uct browsing interface, with the category-brand pairquery (Laptops, Apple) selected.

the shopper’s intent implicitly through recent interactionswith the system. We leave as an orthogonal task the jobof identifying such expressions of a user’s information eitherfrom the query stream of a search engine or other means. Forthe remainder of the paper, we will assume a user has ex-pressed an information need in the form of a category-brandpair, which we will refer to as the query.

2.2 Addressing Category-Brand QueriesThe second main question is what consitutes the search

result, and especially, what is the correct level of granular-ity for the search result? Online forums are typically orga-nized hierarchically: an online forum site often has severalhigh-level topical forum categories, which are split into finer-grained categories. Each of these contains many threads,collections of user-contributed messages. Should the forumresults be top-level forum site, a lower-level forum, a messagethread, or message? Top-level forums are almost always toobroad and topically diverse to be useful as a result. Sendinga user to a top-level forum generally means the user stillneeds to search within the forum. Returning posts is un-helpful in the opposite extreme: taken out of context, anindividual post is rarely informative. The sweet spot over-

all seems to be returning both the forum, plus a list of themost relevant threads. As with most web search results dis-play, we choose to provide not just the the online forum, butalso contextual information with the result. In this case, thecontextual information includes the top ranking thread titleswith metadata possibly including the number of messages ora date range of posting times.

Figure 2 shows an example organization from an onlineforum. In this example, we can see the hierarchical organi-

!"#$%&'(#)*+&,-./012&

•  !"#$%&'(%#$)"#*'+,-".*'

•  /*01#2'345).6'788-'219:'

•  ;<=>.?@8'A8%B*'"C*#.DE'•  @1A%';F=>G;F=>H?!88-'IJ8%B*:'

•  K.#**.'"&-'H#%,D'3>;>'

•  @1A%'L55'&8.'2#%7J.'*&817J'

!"#$%&3-)#*#4(52&•  !MK'K9D.*$D'

–  !MK'/*B8$$*&-"N8&D'–  H%,D6'H#%BOD'P'Q8-D'–  E'

•  !"#$%&'K1,,8#.'–  !"#$%&'@1A%'(8#1$'–  !"#$%&'K.#**.M%C8.'(8#1$'–  E'

•  H8$H8$'K1,,8#.'–  H8$H8$'!R'(8#1$'–  H8$H8$'R&*'(8#1$'–  E'

Figure 2: Example hierarchical organization of anonline forum, gpsreview.net. Topical forum organi-zation shown at left, and thread listing in a singleforum shown at right.

zation of an online forum on the left, with a thread listingon the right. Given a shopper’s interest in a category-brandpair, such as (GPS, Garmin), we may deem the forum en-titled “Garmin Nuvi Forum” a relevant result. In this case,any individual thread within this forum would be too spe-cific, and the forum site as a whole would be too general.In general, the leaf-node forum may not always be the bestchoice of a result. For example, a higher level in the hier-archy (eg. the “Garmin Support” forum) may be a betterchoice in this case. But, the leaf-node forum tends to pro-vide a good trade-off between the generality of the forumsite and specificity of a message or message thread. Weleave for future work investigation of identifying per-querythe appropriate level of hierarchical organization.

3. RELATED WORKSocial media, including online discussion forums, have

been the focus of much recent research. In the area of in-formation retrieval, the TREC blog track has focused theresearch community on techniques for ranking and opin-ion mining of blogs and blog posts with respect to userqueries [5, 8]. Recently several models for thread retrieval inonline message boards have been proposed [3, 11]. This pre-vious work on retrieval in online forums has focused on themessage thread as the primary unit of retrieval, whereas inthis work we are concerned with ranking forums, or collec-tions of threads. The forum ranking model presented belowbuilds upon this previous work studying blog retrieval andmessage thread retrieval [1, 3].

Online forums have also been the focus of several datamining studies. Wanas et al. [12] developed methods to au-tomatically identifying high quality quality posts in a largediscussion board. Yang et al. [13] apply information extrac-tion and techniques to the task of automatically identifying

online forum structure from web pages, such as segmentingthreads into messages, identifying author names and mes-sage posting dates. Zhang et al. [15] present a study of thesocial dynamics in online forums to identify author exper-tise.

Online forums and blogs have been recognized as fertileground for mining product discussion. Glance et al. [4]present a system for mining online discussion for the pur-poses of monitoring popular opinion about brands or prod-ucts. This system provides facilities for extracting threadingstructure from online discussion boards, opinion mining andaggregation, and social network analysis. The work pre-sented here similarly focuses on finding product discussionin online forums, but for the goal of aiding consumers intheir product research rather than aiding companies moni-tor popular opinion.

4. FORUM RANKING APPROACHOur approach to identifying forums with rich product dis-

cussion is based on two levels of information aggregation:

• From lower-level product mentions to higher level prod-uct brands and categories.

• From lower-level messages to collections of messagesand threads, the forums.

Both of these aggregation steps require rich levels of docu-ment annotation, as well as a model for scoring forums toaggregate from the message level.

The focus of this work is on the forum ranking model (Sec-tion 4.3) but we provide a high-level description of the anno-tations used in the following sections. There are numerousautomatic techniques for document classification, structureextraction, product annotation, and product name normal-ization [9, 10, 13], and the details of those applied here areout of the scope of the current work.

4.1 Product AnnotationIn each document in our collection, all references to prod-

ucts, product lines, and brands are annotated. Each an-notated product mention is mapped to a single entry in aproduct catalog. This catalog contains all known brands,product lines and products, and each entry in the catalogis associated with one node in a product category ontology.This ontology providing a hierarchical organization of prod-ucts, useful for faceted search and browsing in the productsearch portal.

An illustration of the product annotation and mapping tonodes in the product category ontology is shown in Figure3. In this figure, we can see a span of text containing twoproduct mentions, one to a brand (“Switcheasy”) and one toa product line (“Switcheasy Vulcan”). Both of these spansof text are annotated as product mentions and assigned amapping to a node in a product catalog. Note that neitherof these mentions refer to the specific product. Each entryin the product catalog is mapped to a node in the productcategory ontology, in this case the “MP3 Player Cases” leafnode. The resulting annotations in the text correspond tothe category-brand pair (MP3 Player Cases, Switcheasy).

4.2 Forum Structural AnnotationIn addition to the product annotation, we also produce

annotations of the online forum structure in our collection.

!"#$%&'()*'+,#"-(./'#0#,-1(

!"#$%&'()*'*0#,1(

2#&%3+/'(4+5'1(

!"#$%&'()$*+,++++++-.*)$+/"01#&*+,++++++++++-/2+/"01#&+3$$#**'&)#*+,++++++++++++++-/2+/"01#&+40*#*+

/&'5.$%6+7"0$8+9:;<=>?<7?@+

/&'5.$%+?)(#6+=."$0(+

7&0(56+:A)%$B#0*1+ ,+,+

Figure 3: An example annotation and mapping fromproduct mentions in a document text (bottom) tonodes in the product category ontology (top).

See Figure 4 (dashed lines) for example extracted structure.The following fields and attributes are extracted from thesedocuments:

• Message-Level Annotations: post date, author name,body text

• Thread-Level Annotations: parent forum, numberof messages, title text

Each message thread is assigned to a single parent forumcontaining that thread. Online forum sites are typically or-ganized hierarchically, containing many forums within a sin-gle site. We make the assumption that each message threadbelongs to only the immediate parent forum in that site,typically a leaf node in the forum hierarchy. In the exampleshowin in Figure 4 (top), the thread shown belongs to the“iPhone Accessories” forum, and other higher-level forums(eg. “iPhone Forums”) are ignored.

4.3 Forum Ranking ModelWe approach the task of identifying forums with rich prod-

uct discussion as an information retrieval problem, rankingforums with respect to a category-brand query. We takea probabilistic language modeling approach to scoring theonline forums with respect to the query. In our approach,we aggregate information from the lowest-level in the forumhierarchy, the message text, to the level we’re interested inscoring, the forum. We rank forums by their conditionallikelihood given the query, P (f |q). The estimation of thisprobability is shown below, first applying Bayes theorem andmarginalizing over the message threads in the collection t.Letting f be the forum, and q be the user’s query:

Score(f, q) =P (f |q) rank= P (f)

Xt

P (t|f)P (q|t). (1)

Note, we drop the probability of observing the query P (q)as it is doesn’t influence the thread ranking when the queryis fixed. We assume a uniform prior probability of forumrelevance, P (f), and the probability P (t|f) is given by

P (t|f) =

(1

|f |+αfif thread t belongs to forum f

0 otherwise(2)

where |f | is the number of threads in forum f and αf ≥ 0is a discount penalizing forums with few thread.

In keeping with previous research on ranking structureddocuments [7], we model the query likelihood with respectto the thread, P (q|t), as a mixture model. The mixturecomponents in this case, are the thread title language modelθt and the message body language models θm. Marginalizingover the messages in the collection, we have

P (q|t) =λP (q|θt) + (1− λ)Xm

P (m|t)P (q|θm) (3)

and we similarly define P (m|t) as

P (m|t) =

(1

|t|+αtif message m belongs to thread t

0 otherwise(4)

where |t| is the number of messages in thread t and αt ≥ 0 isa discount penalizing threads with few messages. Note thatthe query likelihood given the thread P (q|t), in addition tobeing an additive component of the forum scoring (Equation1) also provides a natural means to identify top relevantthreads from within a forum.

The normalizing terms above (Equations 2 and 4), pro-vide parameters to discount the score for threads with fewmessages and forums with few threads. We can interpretthese probabilities as assuming there are some number of“unseen” threads and messages (αt and αm) in the collec-tion with a zero score. Similar normalization techniqueshave been shown to be effective in blog feed search [1].

The language model probabilities above (P (q|θt) and P (q|θm)in Equation 3) are estimated as multinomials with Bayesiansmoothing using Dirichlet priors [16]. In this setting, we arenot scoring documents with respect to the degree of textualmatch with the query, but rather to favor documents with ahigh density of product category mentions. To this end, wetreat a product mention as equivalent to a text token in thedocument.4

When scoring against the thread title language model, wecalculate these probabilities as:

P (q|θt) =n(q, t) + µtP (q)

|t|+ µt(5)

where n(q, t) is the the the number of times the query qwas mentioned the title, |t| is the number of text tokens inthe title, and µt is a smoothing parameter. When scoringagainst the post text, we have two background languagemodels: the collection (as above) and the entire thread body.To take advantage of these two background models, we applytwo-stage Dirichlet smoothing [14], giving the following:

P (q|θm) =n(q,m) + µm

n(q,T )+µTP (q)|T |+µT

|m|+ µm(6)

where n(q,m) and n(q, T ) is the the the number of times thequery q was mentioned the message body and thread bodyrespectively, |m| and |T | are the number of text tokens inthe message and thread body, and µm and µT are smoothingparameters. In all the above, P (q) is the background proba-bility of observing a mention of the category-brand pair inthe collection, estimated via maximum likelihood.

4This treatment of a product mention as a single text to-ken is identical to the scoring used by the Indri informationretrieval engine (http://lemurproject.org/indri) whenscoring annotated text with the #any operator.

!"#$%

&'#()*%

+(*$",%+-#.$%

/$00"1$%2),3%

4*),'5#%/$67)60%

8)*'9%:-$*"*5(3%

4"*$6#%8)*'9%

Figure 4: An example forum thread page (from forums.macrumors.com), showing forum structural annota-tions (black, dashed lines) and product mention annotations (blue, solid lines).

The ranking model here is derived for a single category-brand pair query. This is the equivalent to a single-termquery in a text search engine, and avoids the difficulty incombining relevance scores across several terms which mayhave different collection-level characteristics or significanceto the underlying information need. This model, can eas-ily be extended to multi-term queries in the same way thatstandard language modeling retrieval models approach theproblem [6]. Because we focus on single-term queries in thiswork, the model is not very sensitive to the language mod-eling estimation details (ie. calculation of P (q|θx)). If themodel is extended to handle multi-term queries, closer at-tention must be paid to these estimation details.

5. DATASET DESCRIPTIONThe dataset used in the following study is a richly anno-

tated document collection, as well as a product catalog andproduct category ontology. The documents used are a sam-ple of webpages from online forums from the first tier of acommercial search engine index, excluding those documentsidentified as pornography and spam. The forum structure isextracted, as described above in Section 4.2. The documenttext is annotated with product mentions and those mentionsare mapped into a product category ontology as describedin Section 4.1. For the purposes of this study, we focus onlyon consumer electronics products.

The final dataset contains over 3.5 million online forums,with almost 400 million messages organized into over 40 mil-lion message threads and contributed by over 45 million au-thors. Almost 40% of the message threads containing atleast one product mention, and there are over 350 milliontotal mentions in the collection, corresponding to 95 millionunique category-brand pairs.

6. EXPERIMENTAL SETUP & RESULTS

We evaluate the system presented here along two dimen-sions. In this section, we refer to the forum ranking pro-duced by the above algorithm as the Top Forums results.First we look at a precision-oriented evaluation of the topresults ranked by this system. Because of the limited screenreal estate in a product search portal, it is likely that onlya small number of online forums can be presented to a user.For this reason, we view precision at the top ranked resultsas the primary measure of performance for this task.

Second, we compare the ranking produced by the systemto a commercial search engine ranking, limiting those re-sults to only pages from online forums. The search enginequeries used are a simple concatenation of the category nameand brand, for example [garmin gps] or [hewlett packard

laptops]. Both evaluations use a set of 96 queries sam-pled to represent both popular and unpopular brands andcategories, restricted to “concrete” categories roughly corre-sponding to leaf nodes in our product category ontology.

The evaluation dataset was collected through a web in-terface, shown in Figure 5. Participants were shown a listof category-brand pairs, and asked to select one from thelist. After doing so, the top ten results from the system de-scribed here were presented, and participants were asked toidentify those results deemed relevant to the query. Partic-ipants were also shown the top ten results from a commer-cial search engine, restricted to pages only from online forumwebsites, and asked to identify which results they would findmore helpful in making purchasing decisions or performingproduct research.

Figure 6 shows example results for several queries. Theseresults show the top 3-4 results for the queries (Headphones,Sennheiser) and (GPS, Garmin), as well as the top-scoring2-3 threads in each forum. Each result provides the forumand thread titles, and some metadata indicating the sizes ofthese threads and forums. The displayed threads are thosethat have the highest contribution to the forum score, theirscore given by Equation 3. These results clearly demonstrate

the model’s ability to identify threads with a high densityof product mentions. In these results, the retrieved threadsare direct product comparisons, reviews, purchasing adviceand other related content.

6.1 Algorithm Parameter TuningThe ranking algorithm above has several parameters that

can be tuned to change the characteristics of the systemoutput. Table 1 shows the parameter values used for thisevaluation. These parameters were identified through man-

Parameter Description Value

αf Small forum discount 200αt Small thread discount 50λ Title weight 0.8µt Thread title smoothing 300µm Message body smoothing 1000µT Thread body smoothing 2500

Table 1: Parameter settings used in the experi-ments..

ual tuning prior to the evaluation. We leave for future workinvestigation of automated methods for tuning parametersin this retrieval model.

6.2 Precision-Oriented EvaluationAs a component to a product search and browsing portal,

where screen real-estate is is shared with browsing controls,product listings, merchant offerings and advertisements, ahigh-precision ranking is crucial. For this reason, we focuson the precision of the ranked list of top forums at the firstfive rank positions. Table 2 shows the precision of the systemaveraged across all 96 queries. From this figure, we can seethat precision at the top rank exceeds 85%, and drops tojust under 80% at rank position five.

Figure 3 shows the precision at the top rank position forthe product categories with the most queries in our dataset.From this figure we can see that there is a range in perfor-mance across categories, with some categories like “hand-helds & pdas” retrieving a relevant result at the top rank100% of the time. Other categories like “computer moni-tors” have much lower precision, only retrieving a relevantforum 63% of the time.

6.3 Side-by-side EvaluationIn addition to the precision oriented evaluation described

above, we also evaluated the Top Forums results againstthose of a web search engine restricted to online forumspages. Participants were asked to identify the most use-ful set of results for product research in the context of an

Rank Cutoff Precision

1 0.85112 0.81913 0.79434 0.79265 0.7894

Table 2: Precision at top five rank cutoffs, averagedacross all 96 queries.

Category Num. PrecisionQueries @1 @2 @3

computer monitors 8 0.625 0.563 0.417home theater systems 9 0.667 0.778 0.741

radios 8 0.750 0.750 0.750memory 8 0.875 0.857 0.810

digital cameras 11 0.909 0.864 0.849bridges & routers 5 1.000 1.000 0.933handhelds & pdas 7 1.000 1.000 1.000

digital cameras 10 1.000 1.000 1.000flat panel televisions 10 1.000 0.950 0.933

Table 3: Precision at top rank per category.

online shopping experience. They were given three choices:Top Forums results, web results restricted to online forumpages, or neither.

Table 4 presents the results of that study, showing thatthe top forums results are as good or better than the webresults 83% of the time.

System # Queries Preferred

Top Forums 46 (48%)Neither 34 (35%)

Web Results 16 (17%)

Table 4: Annotator preference for Top Forums re-sults or filtered Web results.

7. CONCLUSIONS & FUTURE WORKThis paper presents a system for identifying online forums

rich in product discussion relating to category-brand pairs.The algorithms proposed here perform two levels of aggrega-tion to find forums relevant to these higher-level concepts.First, using a product catalog and category ontology, weaggregate specific product mentions to category-brand ref-erences. Second, using structural annotation of the onlineforum, we aggregate relevance scores from online forum mes-sages and threads to forum scores. The system achieves over85% accuracy in identifying the top-ranked online forum re-sult, and is preferred or equivalent to web search results 83%of the time.

Although the system presented here focuses on identify-ing forums relevant to category-brand pairs, the approach ismuch more general. For example, with the same annotationscheme, forums could be scored with respect to category orbrand only. Or, as mentioned above, the equivalent to multi-term queries could be run in the system to identify forumsdiscussing several brands in the same category.

The algorithm presented here identified top forums basedon the density of discussion threads relevant to the query.An alternative approach would be to identify top forumsbased on the density of expert authors with respect to thetopic of the query. A fertile area for exploration is the ap-plication of expert finding models [2] to this task, and thecombing of thread- and expert-models to finding top forums.

Finally, these aggregation techniques could also be appliedto improving general web search, specifically when scoringonline forum content. Currently published models for websearch do not take into account the unique structure of on-

line discussion boards. Leveraging that structure within webretrieval models may yield further improvements.

8. REFERENCES[1] J. Arguello, J. L. Elsas, J. Callan, and J. G.

Carbonell. Document representation and queryexpansion models for blog recommendation. InProceedings of the Second International Conference onWeblogs and Social Media (ICWSM), 2008.

[2] K. Balog, L. Azzopardi, and M. de Rijke. Formalmodels for expert finding in enterprise corpora. InProceedings of the 29th annual international ACMSIGIR conference on Research and development ininformation retrieval, pages 43–50. ACM New York,NY, USA, 2006.

[3] J. L. Elsas and J. G. Carbonell. It pays to be picky:an evaluation of thread retrieval in online forums. InSIGIR ’09: Proceedings of the 32nd internationalACM SIGIR conference on Research and developmentin information retrieval, pages 714–715, New York,NY, USA, 2009. ACM.

[4] N. Glance, M. Hurst, K. Nigam, M. Siegler,R. Stockton, and T. Tomokiyo. Deriving marketingintelligence from online discussion. In KDD ’05:Proceedings of the eleventh ACM SIGKDDinternational conference on Knowledge discovery indata mining, pages 419–428, New York, NY, USA,2005. ACM.

[5] C. Macdonald, I. Ounis, and I. Soboroff. Overview ofthe TREC 2007 blog track. In Proceedings of the 2007Text Retrieval Conference, 2007.

[6] D. Metzler and W. B. Croft. Combining the languagemodel and inference network approaches to retrieval.Inf. Process. Manage., 40(5):735–750, 2004.

[7] P. Ogilvie and J. Callan. Combining documentrepresentations for known-item search. In SIGIR ’03:Proceedings of the 26th annual international ACMSIGIR conference on Research and development ininformaion retrieval, pages 143–150, New York, NY,USA, 2003. ACM.

[8] I. Ounis, C. Macdonald, and I. Soboroff. Overview ofthe TREC 2008 blog track. In Proceedings of the 2008Text Retrieval Conference, 2008.

[9] S. Sarawagi. Information extraction. Found. Trendsdatabases, 1(3):261–377, 2008.

[10] F. Sebastiani. Machine learning in automated textcategorization. ACM Comput. Surv., 34(1):1–47, 2002.

[11] J. Seo, W. B. Croft, and D. A. Smith. Onlinecommunity search using thread structure. In CIKM’09: Proceeding of the 18th ACM conference onInformation and knowledge management, pages1907–1910, New York, NY, USA, 2009. ACM.

[12] N. Wanas, M. El-Saban, H. Ashour, and W. Ammar.Automatic scoring of online discussion posts. InWICOW ’08: Proceeding of the 2nd ACM workshopon Information credibility on the web, pages 19–26,New York, NY, USA, 2008. ACM.

[13] J.-M. Yang, R. Cai, Y. Wang, J. Zhu, L. Zhang, andW.-Y. Ma. Incorporating site-level knowledge toextract structured data from web forums. In WWW’09: Proceedings of the 18th international conference

on World wide web, pages 181–190, New York, NY,USA, 2009. ACM.

[14] C. Zhai and J. Lafferty. A study of smoothingmethods for language models applied to informationretrieval. ACM Trans. Inf. Syst., 22(2):179–214, 2004.

[15] J. Zhang, M. S. Ackerman, and L. Adamic. Expertisenetworks in online communities: structure andalgorithms. In WWW ’07: Proceedings of the 16thinternational conference on World Wide Web, pages221–230, New York, NY, USA, 2007. ACM.

[16] L. Zhao and J. Callan. A generative retrieval modelfor structured documents. In CIKM ’08: Proceeding ofthe 17th ACM conference on Information andknowledge management, pages 1163–1172, New York,NY, USA, 2008. ACM.

!"#$%&'()*'"+,-.$/$01&+-

.(2#$3-4'$5$'$+0$-.$/$01&+-

6&7-8&'932-:$29/#2-

8;/#$'$,-<$=-

:$29/#2-

Figure 5: Evaluation interface.

!"#$%& (Headphones, Sennheiser)!'#()*+,-#./01"223.45#6!"#$%!&'()*+,!-!$$$#.!/0,&,!1112')*+-320(4560(78,5695!!!!!!!:)1!;)<<')=,)(!80+)>,!?!@A9$"B!@A9C"B!@AD.EB!@ADE"!E.!/0,&,!!!!!!;)<<')=,)(!@A"C.B!@A"C.!FFB!@A"C.!G()H=,=0<222!1'*&I,!&')!+=J)()<H)K!E$!/0,&,!!!!!!@A"#"!L,!@A$..!L,!@A$".!E$!/0,&,!7,$8(92#/'#()*+,-#.:/;($*+,-#./(-)/<-3;($/=,-48,$.!D9.D!&'()*+,!-!D$"CM!/0,&,!1112')*+-320(4560(78,56E.%5!!!!!!!NO%..!L,!NO9..!L,!NO"..K!P(!,08)&'=<4!)>,)K!E$!/0,&,!!!!!!:)1!;)<<')=,)(!H*<*>/'0<),!QNO9..B!NO"..B!NOR9..B!NO""B!NO#"S!E$!/0,&,!!!!!!!;)<<')=,)(!FTM?!U')!V)L=)1!QR0&,!06!/=H,!W!H08/*(=,0<!1=&'!FTC!W!@A$..!E$!/0,&,!!'#()*+,-#./>,$/?(2#/@/A$()#!E"$9!&'()*+,!-!#M%9!/0,&,!1112')*+-320(4560(78,56E.5!!!!!!!;PRA?!;)<<')=,)(!@A"C.!15!@A$..!X(=>>,!W!@A$".!N*Y>)!QZ7,&(*>=*S!#!/0,&,!!!!!!;PRA?!;)<<')=,)(!@AD"-E!FF!15!@A"C.!N*Y>)!!9!/0,&,!!!!!!![U\?!;)<<')=,)(!]O9..5]O"..!9!/0,&,!!

!"#$%& (GPS, Garmin)!B($C4-/D"E4/>,$"C!M$#!&'()*+,!-!C#%.!/0,&,!60(78,24/,()L=)12<)&5L=)160(782/'/K6^D!!!!!!E%#.U!U(*_H!;=4<*>!G)(60(8*<H)!L,!M$"U!L,!D$"[U!D!/0,&,!!!!!!D."[!L,2!D""[!L,2!D$.[!-!X*(8=<!:7L=!`0(78!"!/0,&,!!FFFG*,431(H8,$%GH,C@!%DCE!&'()*+,!-!9C"E#!/0,&,!!1112/0=-6*H&0(a2H085!!!!!!!N08/*(=<4!X*(8=<!CC"&!1=&'!M$"&!b!GPF!`*H&0(a!%.!/0,&,!!!!!!!X*(8=<!:7L=!D""[!L,!D$.[!b!GPF!`*H&0(a!EE!/0,&,!B($C4-/-IE4/1,$"C.!#DM!&'()*+,!-!EE"%C!/0,&,!11124/,/*,,=0<2H08560(78,)<560(782*,/K`PVc]dFA^EMD!!!!!!!X/,G*,;=0<!`0(78,!-!;)(L=H)!3(81*()!2(4<!60(!<7L=!D."15D""15D$"1!M!/0,&,!!!!!!X/,G*,;=0<!`0(78,!-!:ceF!%".5%$.!e;!fc]P!"".Q]0&0(HaH>)S!EM!/0,&,!B7?/J#H,CC#-)(K,-.!!D##!&'()*+,!-!D$""!/0,&,!60(78,24/,()L=)12<)&5L=)160(782/'/K6^%.!!!!!!X*(8=<!N0>0(*+0!9..&B!P()40<!9..&B!0(!$.N,g!-!XG;!V)H088)<+*h0<,!"!/0,&,!!!!!D""1B!D""1&B!D$"1&KKKKKKK!-!XG;!V)H088)<+*h0<,!%!/0,&,!

Figure 6: Example Top Forums results from two category-brand queries, (Headphones, Sennheiser) at left and(GPS, Garmin) at right.