history by diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · jaspreet singh test...
TRANSCRIPT
![Page 1: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/1.jpg)
History by Diversity
Jaspreet Singh, Avishek Anand & Wolfgang Nejdl (singh,anand,nejdl @l3s.de)
1
![Page 2: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/2.jpg)
Exploring News Archives
Ranking
Relevant to?
Search Intent
![Page 3: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/3.jpg)
Why can’t it be like Google?
• Precise
• Diverse
• Driven by popularity
What is the intent of a user when using google?
![Page 4: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/4.jpg)
Historical Search on News Archives
I want to know the history of ....... Rudolph Giuliani
• Newspaper articles encode history as it happens. • Aspects are diverse across time. • Time windows can be diverse in aspects.
4
Mayoral Campaign
Mayoral Campaign
Mayoral Campaign 9/11
Post politics endeavours
Senate run, Cancer, Allegations of cheating
Num
ber o
f Doc
umen
ts
Mayor
![Page 5: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/5.jpg)
History by Diversity
• Historical Search task: I want documents covering the most important aspects when they were important for a given topic.
• I want documents from the most important aspects and from the most important time windows. (New Retrieval Task)
• History by Diversity - Extending the standard diversity problem to include time
5
![Page 6: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/6.jpg)
How do you evaluate historical search?
• To measure coverage of important time windows and aspects we introduce a new information space:
• Aspect-Time space - encodes which aspects are relevant and when.
SBR@k =
• Adapt standard diversity metrics like intent aware precision, subtopic recall, NDCG, etc to function on this space.
• A document is relevant only if it is relevant to an aspect and is also published in its corresponding time period.
6
![Page 7: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/7.jpg)
Jaspreet Singh
Test Collection for Historical Search
• TREC datasets are short time spans; the topics and subtopics are not suited for historical search.
• We created our own test collection using the New York Times 20 year annotated dataset.
• Manually created topics and subtopics using relevant wikipedia history sections.
• Expert binary relevance judgements for 30 topics.
7
![Page 8: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/8.jpg)
8
![Page 9: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/9.jpg)
Jaspreet Singh
Why not use standard diversification algorithms?
• Diversify just aspects: no guarantee we will get temporal diversity. • Diversify just time: no guarantee we will get aspect diversity.
• Aspect Diversity - Aspect utility is updated using a discounting function.
• Temporal Diversity - considers aspects of a topic as time windows. • Discounting time using exponential decay
9
![Page 10: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/10.jpg)
Jaspreet Singh
The HistDiv Approach
• Aspects are temporal in nature • Time windows are diverse themselves
• Compute utility and discount accordingly
10
Mayoral Campaign
Mayoral Campaign
Mayoral Campaign 9/11
Post politics endeavours
Senate, Cancer, Allegations
Num
ber o
f Doc
umen
ts
Mayor
![Page 11: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/11.jpg)
Jaspreet Singh 11
1993 1996 1997 1998
DMayoral Campaign
Mayorality
T = 2000Util
ity
Only Time Only Aspect HistDiv
T = 2000T = 2000 T = 2000
Mayoral Campaign
Mayorality
Mayoral Campaign
Mayorality
Mayoral Campaign
Mayorality
![Page 12: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/12.jpg)
Jaspreet Singh
The HistDiv Algorithm
• Extend the multi-dimension diversification algorithm • 2 dimensions: Time and Aspects
• Dimension are interdependent
12
Time Decay based discounting Coverage based discounting
![Page 13: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/13.jpg)
Jaspreet Singh 13
1993 1996 1997 1998
D1Mayoral Campaign
Mayorality
Util
ity
1993 1996 1997 1998
Mayoral Campaign
Mayorality
Util
ity
Discounting window width (w) set dynamically using burst detection
![Page 14: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/14.jpg)
Jaspreet Singh 14
D2
Mayoral Campaign
Mayoralty
1993 1996 1997 1998
Mayoral Campaign
Mayoralty
Util
ity
D3
1993 1996 1997 1998
Util
ity
![Page 15: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/15.jpg)
Jaspreet Singh
Experiments
• New York Times Test Collection
• Metrics: Time Aware NDCG, ERR, Subtopic Recall, Precision, MAP & TIA-SBR
• Window size: year & month
• Aspects mined from AIDA & wikiminer
• Tuned for best performance in subtopic recall.
15
![Page 16: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/16.jpg)
Jaspreet Singh
Competitors
• Competitors - Time diversification, Aspect Diversification, Adapted aspect diversification & Multi Dimension Diversification
• We strengthen the following competitors by linearizing aspects and time
• IA-SELECT • PM2
Other competitors: Non temporal IA-SELECT & PM2, MDIV, OnlyTime
Baseline: Language Model with dirichlet smoothing
16
![Page 17: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/17.jpg)
Jaspreet Singh
Results
17
TIA-SBR (Win/Loss)
![Page 18: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/18.jpg)
18
![Page 19: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/19.jpg)
Jaspreet Singh
Conclusion
✓ Historical Query Intents, test collection to evaluate retrieval models for HQIs and a new metric TIA-SBR.
✓ HistDiv Algorithm - special semantics to discount time and aspects
✓ Outperform competitors in most measures.
✓ Aspects and time are interlocked
✓ Robust - temporal references alone are also effective as well as simple NER ✓ Recall at the cost of precision ✓ Good starting point for further exploration of a news archive
19
![Page 20: History by Diversityalexandria-project.eu/wp...alex_ws_jaspreet_singh.pdf · Jaspreet Singh Test Collection for Historical Search • TREC datasets are short time spans; the topics](https://reader034.vdocument.in/reader034/viewer/2022042311/5eda015828db2d5ca2493d1e/html5/thumbnails/20.jpg)
Demo
http://pharos.l3s.uni-hannover.de:7080/ArchiveSearch/starterkit/
20