evaluating heterogeneous information access (position paper)
DESCRIPTION
Information access is becoming increasingly heterogeneous. We need to better understand the more complex user behaviour within this context so that to properly evaluate search systems dealing with heterogeneous information. In this paper, we review the main challenges associated with evaluating search in this context and propose some avenues to incorporate user aspects into evaluation measures. Full paper at http://www.dcs.gla.ac.uk/~mounia/Papers/mube2013.pdf. To presented at SIGIR workshop MUBE, 2013. PhD work/reflection/conclusions of Ke (Adam) Zhou.TRANSCRIPT
![Page 1: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/1.jpg)
Evalua&ng Heterogeneous Informa&on Access (Posi&on Paper)
Ke Zhou1, Tetsuya Sakai2, Mounia Lalmas3, Zhicheng Dou2 and Joemon M. Jose1
1University of Glasgow 2MicrosoN Research Asia 3Yahoo! Labs Barcelona
SIGIR 2013 MUBE workshop
![Page 2: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/2.jpg)
IR Evalua&on
• System-‐oriented Evalua&on (test collec&on + metrics)
• User-‐oriented Evalua&on (interac&ve user study)
• Current endeavor to incorporate user into system-‐oriented metrics – Time-‐Biased Gain (Smucker, Clarke.) – U-‐measure (Sakai, Dou) – etc.
![Page 3: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/3.jpg)
Increasing Heterogeneous Nature on Search
![Page 4: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/4.jpg)
Increasing Heterogeneous Nature on Search
……
![Page 5: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/5.jpg)
Posi&on
• Compared with tradi&onal homogeneous search, evalua&on in the context of heterogeneous informa&on is more challenging and requires taking into account more complex user behaviors and interac4ons.
![Page 6: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/6.jpg)
Challenges
• Non-‐linear Traversal Browsing • Diverse Search Tasks • Coherence • Diversity • Personaliza&on • etc.
![Page 7: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/7.jpg)
Various Presenta&on Strategies
Non-‐linear Blended Blended
……
Tabbed
![Page 8: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/8.jpg)
User Browsing Pa^ern “E” Browsing Pa?ern on Aggregated Search Page
“F” Browsing Pa?ern on Organic Search Page
h^p://searchengineland.com/eye-‐tracking-‐on-‐universal-‐and-‐personalized-‐search-‐12233
![Page 9: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/9.jpg)
Non-‐linear Traversal Browsing
h^p://searchengineland.com/eye-‐tracking-‐on-‐universal-‐and-‐personalized-‐search-‐12233
![Page 10: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/10.jpg)
Search Tasks: Ver&cal Orienta&on
CIKM’10 (Sushmita et al.), WWW’13 (Zhou et al.)
![Page 11: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/11.jpg)
Search Tasks: Complexity
SIGIR’12 (Arguello et al.)
![Page 12: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/12.jpg)
Coherence
Car
Animal
Car
Car
Car
Car
Car
Car
Car
Car
Car
Car
vs.
CIKM’12 (Arguello et al.)
![Page 13: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/13.jpg)
Coherence
Car
Animal
Animal
Car
Animal
Car
Car
Car
Animal
Car
Animal
Car
vs.
CIKM’12 (Arguello et al.)
![Page 14: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/14.jpg)
Diversity
vs.
Image News+Map+Image SIGIR’12 (Zhou et al.)
![Page 15: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/15.jpg)
Personaliza&on
vs.
SIGIRer Average User
![Page 16: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/16.jpg)
Avenues of Research
• Be^er understanding of users – Click models: WSDM’12 (Chen et al.), SIGIR’13 (Wang et al.)
– Ver&cal orienta&on: CIKM’10 (Sushmita et al.), WWW’13 (Zhou et al.)
– Task complexity: SIGIR’12 (Arguello et al.) – Task coherence: CIKM’12 (Arguello et al.) – Diversity: SIGIR’12 (Zhou et al.) – Personaliza&on – Non-‐linear presenta&on strategies
![Page 17: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/17.jpg)
Avenues of Research
• Be^er incorpora&on of learned user behavior into evalua&on metrics – follow SIGIR’13 (Chuklin et al) and convert obtained aggregated search click models into system-‐oriented evalua&on metrics.
– model addi&onal features into powerful evalua&on framework (e.g. TBG, U-‐measure, AS-‐metric).
![Page 18: Evaluating Heterogeneous Information Access (Position Paper)](https://reader033.vdocument.in/reader033/viewer/2022051819/54c7a0a64a7959e7478b4589/html5/thumbnails/18.jpg)
Thank you! Ques&ons?
Ke Zhou, [email protected]
TREC 2013 FedWeb track: h^ps://sites.google.com/site/trecfedweb/