effects of popularity and quality on the usage of query suggestions during information search can...
TRANSCRIPT
Effects of Popularity and Quality on the Usage of Query Suggestions during Information SearchCan users be induced to take bad query suggestions because they believe many others have used the suggestions in the past?
Diane Kelly, Amber Cushing,
Maureen Dostert, Xi Niu and
Karl Gyllstrom University of North
Carolina
at Chapel Hill
Motivation & Background
• Detrimental impact of social search features• Social influences on behavior– Psychology, sociology, and economics– Recommender systems, business and marketing
• Query suggestions as a social search feature and system usage information
• Users’ abilities to identify ‘good’ suggestions
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Research Questions
1. Are users influenced by usage information associated with recommended queries?
2. Can users distinguish between high and low quality query suggestions?
3. What are users’ perceptions of the usefulness of query suggestions and usage information for open search tasks?
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
• Use of query suggestions as idea tactics– A move to help users generate new approaches or
solutions to information search problems– Potentially useful when user has a limited model
of the problem space, is researching a difficult topic, and/or exhausts own ideas for queries
Motivation & Background
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Research Questions
4. What is the relationship among topic difficulty, users’ willingness to take recommendations and their abilities to distinguish between high and low quality queries?
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Method
• Laboratory experiment with experimental search system
• Twenty-three undergraduate subjects • Four assigned search topics (15 minutes each)• Closed collection of newspaper articles (3GB,
or about 1 million documents)• 8 query suggestions per search topic
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Method
Low Quality Queries
High Quality Queries
20-40 High Usage Information
0-9 Low Usage Information
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Outcome Measures & Analysis
• Suggestion Usage/Selection (Binary)• Post-Search Evaluations (5-point scale, 1=low;
5=high)– Query Quality– Confidence in Rating– Willingness to Recommend to Others– Topic Difficulty
• Exit Interview and Manipulation Check• Analysis: Logistic regression, t-tests, open-coding
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Results: General Usage of Suggestions
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Source Number (%)
Subject Queries 425 (59%)
Suggested Queries 297 (41%)
Total Queries 722
Number of Subject Queries that Matched Query Suggestion
106
Results: Popularity
Were subjects influenced by the usage information associated with recommended queries?
No. Usage information (popularity) was not a significant predictor of selection behavior.
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Results: PopularityWhat about subjects’ post-search
evaluations of query quality? Was there a difference in ratings according to popularity?
No. Subjects’ mean ratings were similar.
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Popularity Mean (SD)
Quality Low 2.87 (1.15)
High 2.93 (1.18)
Confidence Low 2.90 (1.11)
High 2.93 (1.08)
Recommend Low 2.84 (1.19)
High 2.88 (1.23)
Results: Query Quality
Could subjects distinguish between high and low quality query suggestions?
Yes. Query quality was a significant predictor of selection behavior.
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Results: Query QualityQuery Quality Mean (SD)
Quality* Low 2.80 (1.18)
High 3.00 (1.15)
Confidence Low 2.84 (1.09)
High 2.99 (1.09)
Recommend* Low 2.74 (1.20)
High 2.89 (1.20)
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
*p<.05
What about subjects’ post-search evaluations of query quality? Was there a difference in ratings according to actual query quality?
Yes. Subjects rated high quality queries significantly higher than low quality queries.
Results: Topic Difficulty
What was the relationship among topic difficulty and subjects’ willingness to take recommendations?
Yes. Subjects were significantly more likely to use suggestions when the topic was hard.
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Results: Subjects’ PerceptionsWhat were subjects’ perceptions of the usefulness of query
suggestions and usage information for open search tasks?
• the usage information did not influence their selections that much and was not that important for this task
• they ignored the usage information and did their own testing• the query suggestions stimulated thinking outside the box and
helped them narrow their searches• the query suggestions were useful when they ran out of ideas
or were unsure of how to start the search
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Summary of Findings• Subjects integrated the suggestions into their
searching fairly quickly• The usage information did not influence subjects’
selection• Subjects selected more high quality queries than
low quality queries and also rated these higher• Subjects took significantly more suggestions
when searching for difficult topics• Suggestions seemed to function as an idea tactic• Task type mediates the effects of social influence
on selection behaviors (suggestive)
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Implications
• Design– Tailor suggestions to properties of task (type,
difficulty, and stage)• Suggestion algorithms• Presentation methods
– Query performance prediction
• Method– Data collection and evaluation of suggestion use
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA
UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL
Thank You.
Diane Kelly, [email protected]