Download - Detecting Good Abandonment in Mobile Search
![Page 1: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/1.jpg)
Detecting Good Abandonment in Mobile Search
Kyle Williams Julia Kiseleva Aidan C. Crook
Imed Zitouni Ahmed Hassan Awadallah Madian Khabsa
Pennsylvania State UniversityEindhoven University of Technology
Microsoft
WWW’16, Montréal, Québec, Canada
![Page 2: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/2.jpg)
Mobile Search
![Page 3: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/3.jpg)
Mobile Search• More and more popular: 2008 31% 2013 63% • Mobile Search differs from traditional search [Human et. al, 2009]
• On Mobiles users are satisfied by the SERP [Li et. al, 2009]
• Mobiles screen is much smaller
• Mobiles are used on the way
![Page 4: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/4.jpg)
Mobile Search• More and more popular: 2008 31% 2013 63% • Mobile Search differs from traditional search [Human et. al, 2009]
• On Mobiles users are satisfied by the SERP [Li et. al, 2009]
• Mobiles screen is much smaller
• Mobiles are used on the way
Search Engines need to adapt
And to Evaluate!
![Page 5: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/5.jpg)
![Page 6: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/6.jpg)
Knowledge Pane
Image Answer
![Page 7: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/7.jpg)
Knowledge Pane
Image Answer
Image Answer
Organic Results: Snippets
![Page 8: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/8.jpg)
Knowledge Pane
Image Answer
Image Answer
Organic Results: Snippets
Knowledge Pane
![Page 9: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/9.jpg)
Evaluating User Satisfaction
• We need metrics to evaluate user satisfaction
• Good abandonment [Human et. al, 2009]: Mobile: 36% of abandoned queries in were likely good Desktop: 14.3%
• Traditional methods use implicit signals: clicks and dwell time
![Page 10: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/10.jpg)
Evaluating User Satisfaction
• We need metrics to evaluate user satisfaction
• Good abandonment [Human et. al, 2009]: Mobile: 36% of abandoned queries in were likely good Desktop: 14.3%
• Traditional methods use implicit signals: clicks and dwell time
Don’t work
![Page 11: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/11.jpg)
Our Main Research Problem
In the absence of clicks, what is the relationship between a user's gestures and satisfaction and can we use gestures to detect satisfaction and good abandonment?
![Page 12: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/12.jpg)
Research Questions• RQ1: What SERP elements are the sources of good
abandonment in mobile search?
• RQ2: Do a user's gestures provide signals that can be used to detect satisfaction and good abandonment in mobile search?
• RQ3: Which user gestures provide the strongest signals for satisfaction and good abandonment?
![Page 13: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/13.jpg)
Research Questions• RQ1: What SERP elements are the sources of good
abandonment in mobile search?
• RQ2: Do a user's gestures provide signals that can be used to detect satisfaction and good abandonment in mobile search?
• RQ3: Which user gestures provide the strongest signals for satisfaction and good abandonment?
USE
R
STU
DY
![Page 14: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/14.jpg)
Research Questions• RQ1: What SERP elements are the sources of good
abandonment in mobile search?
• RQ2: Do a user's gestures provide signals that can be used to detect satisfaction and good abandonment in mobile search?
• RQ3: Which user gestures provide the strongest signals for satisfaction and good abandonment?
USE
R
STU
DY
CR
OW
DSO
UR
CIN
G
![Page 15: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/15.jpg)
User Study Participants
75%
25%
GENDER
Male Female
55%45%
LANGUAGEEnglish Other
82%
8%2% 8%
Education
Computer ScienceElectrical EngineeringMathematicsOther
• 60 Participants• 25.53 +/- 5.42 years
![Page 16: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/16.jpg)
User Study Design• Video Instructions (same for all participants)• Tasks:
1. A conversion between the imperial and metric systems2. Determining if it was a good time to phone a friend in
another part of the world3. Finding the score from a recent game of the user’s
favorite sports team4. Finding the user's favorite celebrity's hair color5. Finding the CEO of a company that lost most of its
value in the last 10 years
![Page 17: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/17.jpg)
Find out what is the hair color of
your favorite celebrity
![Page 18: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/18.jpg)
Questionnaire• Were you able to complete the task?
o Yes/No
• Where did you find the answer?o Answer Box, Image, SERP, Visited Website
• Which query led you to finding the answer?o First, Second, Third, >= Fourth
• How satisfied are you with your experience in this task?o 5-point Likert scale
• Did you put in a lot of effort to complete the task?o 5-point Likert scale
![Page 19: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/19.jpg)
Questionnaire• Were you able to complete the task?
o Yes/No
• Where did you find the answer?o Answer Box, Image, SERP, Visited Website
• Which query led you to finding the answer?o First, Second, Third, >= Fourth
• How satisfied are you with your experience in this task?o 5-point Likert scale
• Did you put in a lot of effort to complete the task?o 5-point Likert scale
5 Tasks~20 Minutes
![Page 20: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/20.jpg)
User Study Data• Total queries – 607 563• Abandoned queries – 576 461• Potential abandonment tasks – 274
![Page 21: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/21.jpg)
User Study Data• Total queries – 607 563• Abandoned queries – 576 461• Potential abandonment tasks – 274
Binary Labels
![Page 22: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/22.jpg)
Crowdsourcing ProcedureRandom sample of abandoned queries from the search logs of a personal digital assistant during one week in June 2015 (no query suggestion)
![Page 23: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/23.jpg)
Crowdsourcing ProcedureQuery: Peniston
Previous Query: third eroics
![Page 24: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/24.jpg)
Crowdsourcing Data• Total amount of queries – 3,895
• Judgments agreement (3 per one query) – 73%
• After filtering: SAT – 1,565 and DSAT – 1,924
![Page 25: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/25.jpg)
RQ1: Reasons of Good Abandonment
![Page 26: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/26.jpg)
RQ1: Reasons of Good Abandonment
Mean of Satisfaction
![Page 27: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/27.jpg)
Query and Session Features• Session duration• Number of queries in session
Session Features
![Page 28: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/28.jpg)
Query and Session Features• Session duration• Number of queries in session • Index of query within session• Time to next query • Query length (number of words)• Is this query a reformulation• Was this query reformulated
Session Features
Query Features
![Page 29: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/29.jpg)
Query and Session Features• Session duration• Number of queries in session • Index of query within session• Time to next query • Query length (number of words)• Is this query a reformulation• Was this query reformulated• Click count • Number of SAT clicks (> 30 sec) • Number of back-click clicks (< 30 sec)
Session Features
Query Features
Click Features
![Page 30: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/30.jpg)
Baseline 1:Click & Dwell• Session duration• Number of queries in session • Index of query within session• Time to next query • Query length (number of words)• Is this query a reformulation• Was this query reformulated• Click count • Number of SAT clicks (> 30 sec) • Number of back-click clicks (< 30 sec)
Session Features
Query Features
Click Features
Click > 30 sec
No Refomulation
B1: Click, Dwell with no Reform
ulation
![Page 31: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/31.jpg)
Baseline 2: Optimistic • Session duration• Number of queries in session • Index of query within session• Time to next query • Query length (number of words)• Is this query a reformulation• Was this query reformulated• Click count • Number of SAT clicks (> 30 sec) • Number of back-click clicks (< 30 sec)
Session Features
Query Features
Click Features
NOClick
NO Refomulation
B2: Optimistic
![Page 32: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/32.jpg)
Baseline 3: Query-Session Model• Session duration• Number of queries in session • Index of query within session• Time to next query • Query length (number of words)• Is this query a reformulation• Was this query reformulated• Click count • Number of SAT clicks (> 30 sec) • Number of back-click clicks (< 30 sec)
Session Features
Query Features
Click Features
B3: Query-Session Model:
Training Random Forest
![Page 33: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/33.jpg)
Gesture Features (1)• Viewport features swipes-related:
o up swipes and down swipeso changes in swipe direction o swiped distance in pixels and average swiped distanceo swipe distance divided by time spent on the SERP
![Page 34: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/34.jpg)
Gesture Features (1)• Viewport features swipes-related:
o up swipes and down swipeso changes in swipe direction o swiped distance in pixels and average swiped distanceo swipe distance divided by time spent on the SERP
• Time To Focuso Time to focus on Answero Time to Focus on Organic Search Results
![Page 35: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/35.jpg)
3 seconds
6 seconds33% of
ViewPort 66% of
ViewPort
View
Port
H
eigh
t
2 seconds20% of ViewPo
rt
1s 4s 0.4s 5.4s+ + =
GF(2): Attributed Reading Time
![Page 36: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/36.jpg)
400 pixels
300 pixels
AttributedReading Time: 5.4s
Pixel Area: (400 pix x 300
pix)
0.045 ms/pix2=
GF (3): Attributed Reading Time Per Pixel
![Page 37: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/37.jpg)
Models: Detecting Good Abandonment
M1: Gesture Model:Training Random Forest based on gesture features
M2: Gesture Model + Query and Session Features:Training Random Forest based on gesture, query and session features
![Page 38: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/38.jpg)
RQ2: Are gestures useful? (1)
On only abandoned user study data: 148 SAT queries and 313 DSAT queries
![Page 39: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/39.jpg)
RQ2: Are gestures useful? (2)
On crowdsourced data: 1565 SAT queries and 1924 DSAT queries
![Page 40: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/40.jpg)
RQ2: Are gestures useful? (3)
On all user study data: 179 SAT queries and 384 DSAT queries
Gestures Features are useful to detect user satisfaction in general!
![Page 41: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/41.jpg)
Conclusions• RQ1: What SERP elements are the sources of good
abandonment in mobile search?Answer, Images and Snippet
• RQ2: Do a user's gestures provide signals that can be used to detect satisfaction and good abandonment in mobile search?
Yes
• RQ3: Which user gestures provide the strongest signals for satisfaction and good abandonment
Time spent interacting with Answers is positively correlated. Swipe actions and time spent with SERP is negatively correlated
![Page 42: Detecting Good Abandonment in Mobile Search](https://reader035.vdocument.in/reader035/viewer/2022062823/587558291a28ab00528b5b45/html5/thumbnails/42.jpg)
• Answer, Images and Snippet are potentially source of the good abandonment
• User gestures provide useful signals to detect good abandonment
• Time spent interacting with Answers is positively correlated. Swipe actions and time spent with SERP is negatively correlated
Questions?