surprise me if you can: serendipity in health...

Surprise Me If You Can: Serendipity in Health Information

Xi Niu1, Fakhri Abbas

1, Mary Lou Maher

1, Kazjon Grace

2

1College of Computing and Informatics

University of North Carolina at Charlotte

Charlotte, NC 28223, USA

{xniu2, fabbas1, M.Maher}@uncc.edu

2Design Lab

University of Sydney

Sydney, NSW, Australia

[email protected] ABSTRACT

Our natural tendency to be curious is increasingly important

now that we are exposed to vast amounts of information.

We often cope with this overload by focusing on the

familiar: information that matches our expectations. In this

paper we present a framework for interactive serendipitous

information discovery based on a computational model of

surprise. This framework delivers information that users

were not actively looking for, but which will be valuable to

their unexpressed needs. We hypothesize that users will be

surprised when presented with information that violates the

expectations predicted by our model of them. This surprise

model is balanced by a value component which ensures that

the information is relevant to the user. Within this

framework we have implemented two surprise models, one

based on association mining and the other on topic

modeling approaches. We evaluate these two models with

thirty users in the context of online health news

recommendation. Positive user feedback was obtained for

both of the computational models of surprise compared to a

baseline random method. This research contributes to the

understanding of serendipity and how to “engineer”

serendipity that is favored by users.

Author Keywords

serendipity, computational models, surprise, value,

health news, information retrieval systems ACM Classification Keywords

H3.3. Information storage and retrieval: Information search

and retrieval.

INTRODUCTION

Serendipity has been defined as unexpected and valuable

discoveries made by accident [7]. With rapid advances in

search and information retrieval, query-based interaction

has become the norm for interfacing with the overwhelming

volumes of information that suffuses society. Today’s

information delivery systems have been criticized as

“reinforcement of the same, relatively limited set of

information” rather than promoting unexpected exploration

and discovery [13]. Similarly, Facebook has admitted that

its algorithms form ‘echo chambers’ [6]. The notions of

“filter bubbles” [40] and “blind spots” [36] are some of the

prime motivations behind serendipity research – bursting

the bubble and reducing information blind spots through

thoughtful design. However, serendipity is recognized as

being very challenging to actively stimulate because it is by

definition unpredictable for the user [33].

In this paper we propose a conceptual framework for

serendipity, implement it in the context of health news

recommendation using two different computational models

of surprise, and then evaluate it with a user study. Our

contribution is to advance knowledge of how to model the

concept of serendipity both conceptually and

computationally. We also provide a user-centered approach

to evaluating serendipity.

RELATED WORK

This research brings together two threads from the

literature: how serendipity is distinct from diversity and

novelty, and how serendipity can be operationalized in

online information retrieval systems.

Serendipity versus Diversity and Novelty

First coined by Harold Walpole in 1754 [7], the word

“serendipity” is used to describe the process of making

discoveries by accident. It received little attention until the

mid-1900s when it was used as a descriptor of accidental or

unplanned discovery in the scientific context [7]. Although

there is some disagreement as to the precise nature of

serendipity, most accounts agree that the following two

aspects are central: an unexpected encounter and a valuable

discovery. In this paper, we operationalize these two core

aspects – surprise and value – in order to predict

serendipitous experiences.

Serendipity, diversity and novelty are all “beyond-

accuracy” evaluation metrics proposed for recommender

systems in recent years [26]. It is worth distinguishing

between them since there is significant overlap and

potential for confusion. Diversity has been studied in the IR

field since 1998 when Carbonell and Goldstein investigated

the relationship between diversity and retrieval accuracy

[15]. Before that, quantifying diversity was studied in

business research for managing investment risk, as

diversification of stock portfolio reduces risk [31]. In the

past decade it has been argued that ranking retrieved items

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for profit or commercial advantage and that copies bear this notice and the full

citation on the first page. Copyrights for components of this work owned by others

than ACM must be honored. Abstracting with credit is permitted. To copy otherwise,

or republish, to post on servers or to redistribute to lists, requires prior specific

permission and/or a fee. Request permissions from [email protected].

CHI 2018, April 21–26, 2018, Montréal, QC, Canada

© 2018 Association for Computing Machinery.

ACM ISBN 978-1-4503-5620-6/18/04…$15.00

https://doi.org/10.1145/3173574.3173597

CHI 2018 Paper CHI 2018, April 21–26, 2018, Montréal, QC, Canada

Paper 23

mailto:[email protected]

https://doi.org/10.1145/3173574.3173597

by only their retrieval accuracy (or relevance) increases the

risk of producing a limited, overly similar and unsatisfying

set of results [3, 16, 46]. Diversifying the retrieval results

may solve this problem by including different items that

may be tangential to the query but potentially match

interests missing from the query. There is a growing

consensus that user satisfaction and engagement have been

improved with such diversification, even at the cost of some

retrieval accuracy [42, 45, 49]. We believe diversity

increases the chance of serendipity, but not every

diversified result is serendipitous; equally, not all

serendipitous discoveries arise from diverse results.

A related concept is novelty, which is a measure of how

different an item is to a set of other items [20]. This set can

be a whole database (objective novelty) or an estimate of

the user’s knowledge (subjective novelty). In the

recommender system context, novelty means an item that is

unknown to the user [23]. By contrast, surprise (on which

we base our serendipity recommender) is a measure of how

strongly an item violates expectations. While both surprise

and novelty can be objective or subjective, they do not

measure the same quantity as expectation violation is more

specific than simple difference. Surprising items are novel,

but not all novel items are surprising [20]. The proposed

relationship among diversity, novelty, and serendipity is

presented in Figure 1. We will validate this hypothesis later

with our user study.

Figure 1. The proposed relationship between diversity, novelty,

and serendipity

Serendipity in Information Retrieval

Serendipity arises when something is surprising and also

valuable. The natural human information-seeking process is

full of such chance encounters. Miksa [35] described

acquisition of intellectual knowledge as a relatively

unfocused sense of inquiry, and said that information

retrieval is “better conceived as an exploratory and game-

like mechanism rather than a precise response mechanism”.

Gup in 1997 [22] expressed his concern about the 'end of

serendipity' in the digital world, recalling with fondness his

childhood experiences coming across interesting tidbits of

information while flipping pages in the encyclopedia.

Gup’s concerns are echoed by others in more recent studies

in [34, 41, 43]. The sense that the online environment is

increasingly deterministic and predictable promotes a

widespread feeling that serendipity is threatened.

However, researchers are experimenting with new tools that

support serendipity technologically [33]. Earlier attempts

include the LyricTime system [28], a music recommender

system that accommodated serendipitous access by

occasionally adding randomly-picked songs to the user’s

playlist. Campos and Figueiredo [14] developed a software

agent called Max that incorporated knowledge of concept

association to find information that imperfectly matches a

user’s profile. The idea was inspired by De Bono’s “lateral

thinking” [17], which encourages the acceptance of

accidental aspects of thinking over the adoption of a

sequential process. André, Teevan, and Dumais [5]

identified a set of partially relevant, but highly interesting,

results returned by a search engine and concluded that this

set has the highest potential for being serendipitous.

Iaquinta et al. [24] described a content-based recommender

with items represented by text descriptions. A machine

learning method was used to predict the user ratings of

unseen items. Items for which the prediction was uncertain

between positive and negative were considered potentially

serendipitous. Onuma, Tong, and Faloutsos [39] introduced

a metric called “bridging score” for item nodes in a graph.

Nodes connecting disparate subgraphs in a graph received

high bridging scores, and therefore higher serendipity.

Zhang et al. [48] used topic modeling to represent artists as

a distribution of latent user clusters. Specifically, they used

the Latent Dirichlet Allocation (LDA) model to represent

each artist, allowing for similarity calculations and

clustering among artists. Their recommender generates

potentially serendipitous recommendations by promoting

artists the user has not explicitly preferred.

There is a recent trend in the visual analytics research

community to support serendipitous discoveries through

various interactive visualization techniques, examples

include Bohemian Bookshelf [4] and Serendip [44]. These

studies, though their approaches and implementations vary,

all imply that carefully designed retrieval systems may have

a role in increasing serendipitous discoveries. Although

generating positive user feedback, these systems tended to

rely on diversity, imperfect matches, and randomness to

approximate serendipity. Without direct focus on the core

elements of serendipity, the results tend to be highly

variable or “hit or miss” [30]. By contrast we directly

predict surprise, with the aim of inducing it in a systematic

and controllable way.

Surprise, the response to unexpected stimuli, has been

suggested as a critical element in triggering and evaluating

serendipity [2]. The common approach for generating

surprise in the recommender systems research community

is to compare the recommendations with a primitive

baseline system. Murakami, Mori, and Orihara [37] assume

that primitive methods generate expected recommendations

and anything that deviates is unexpected. Following

Murakami’s idea, Adamopoulos and Tuzhilin [1] derived a

set of unexpected recommendations by deducting the items

that are recommended by a primitive prediction from the

items generated by a serendipity recommender. The


Paper 23

usefulness of recommendations was left to user judgement.

A limitation of this comparative approach to serendipity

measurement is its sensitivity to the choice of the baseline

system. Our approaches to modelling surprise are

independent of any baseline as they are based on models of

expectation. We compare two such models: one based on

topic co-occurrence and one on the distribution of themes.

An article that contains topics with a lower likelihood of co-

occurring or has a larger divergence from a typical theme

distribution is rated as more surprising.

CONCEPTUAL FRAMEWORK FOR SERENDIPITY

The proposed conceptual framework for serendipity

consists of two components: Surprise and Value, as shown

in Figure 2. The Surprise component constructs an

expectation model to capture what the user would expect to

see. In the field of artificial intelligence, models of surprise

are based on the observation of unexpected events. For

example, Grace and Maher [19, 21] define surprise as the

violation of a confidently-held expectation and present a

probabilistic model of surprise based on co-occurrence of

features. Building on this work, we propose that expectation

for an information object is based on the expected

likelihood of a user seeing such an information object. A

violation of such expectation would be a surprise. This

differs in its details from the Bayesian surprise proposed by

Itti and Baldi [25] where surprise is defined as the

difference between prior and posterior expectation using

Bayes theorem.

Figure 2. Framework for serendipity

In information science, the value of an information object is

based on the subjective, affective, or emotional aspects of a

user’s assessment of it [11, 18]. Recognition of the affective

aspect of value for information stems from the observation

that value is invariably determined by subjective, user-

specific functions. This makes value fundamentally distinct

from whatever objective “stuff” a retrieved artifact holds.

Specifically, in this study we consider a value construct of

Willingness-to-Pay (WTP) and Experienced Utility (EU),

developed in [29], as a measure for the subjective value of

information. WTP is suggested to reflect the instrumental-

rational value that an information object has in problem-

solving tasks, while EU reflects the aesthetic-emotional

value that an information object has in its own right, as the

user engages with the object. Thus, it can be argued that

WTP and EU represent two parts of the value construct that

includes rational and emotional aspects. Our Value

component reflects both.

STUMBLEON: IMPLEMENTING OUR FRAMEWORK

We have implemented the framework in Figure 2 in the

domain of online health information, resulting in a

recommender prototype, called StumbleOn, an extension of

our previous work [38]. The architecture of StumbleOn is

shown in Figure 3. Health topics usually have strong but

hidden associations that provide a rich information ground

for inducing serendipity. Our framework is domain

independent, this implementation is exemplary.

Figure 3. Architecture of StumbleOn

StumbleOn implements the Surprise component as the

Computational Surprise Module, and the Value component

as the Personalized Value Module. The two Modules work

together, using the output of the Content Analyzer, and they

both keep the user in the loop via feedback. The Content

Analyzer uses a series of text mining and machine learning

techniques to re-represent the information in text

documents into parsable structures. These structures are

used to construct user expectations, as detailed below. Our

extension of previous work [38] is the specification of two

computational approaches for the Computational Surprise

Module as well as the addition of the content analyzer

component for constructing document representations.

Implementation of the Computational Surprise Module and Personalized Value Module

The Computational Surprise Module constructs an

expectation model, a violation of which is deemed a

surprise. Expectation is strongly domain dependent. Since

this work is about information objects (news), we model

expectation as an extension of the established view of text-

based information as a “bag of words”. We propose that

expectation about a news article is based on “a bag of co-

occurring constituents”. Depending on the granularity, the

constituents could be coarse-grained topics or fine-grained

themes. In our first approach, we view each news article as

“a bag of co-occurring topics”, where topics are the labels

assigned to an article by experts. The expectation of seeing

an article in the corpus is modeled as the expected

likelihood of a particular a bag of co-occurred topics (an

article) in the corpus, represented as the multiplication of

the individual likelihood of each topic contained in the

article, i.e. p(t1)*p(t2)…*p(tn). The actual or observed

likelihood for an article given a user selected topic tu,

however, is the joint likelihood of the topic combination

conditioned on the selected topic tu, i.e. p(t1, t2, …,tn | tu).

The difference between the observed likelihood and the

expected likelihood reflects the amount of surprise,

represented as the negative of the log ratio to discount the

very large difference, as in Equation 1:

𝑠1 = −𝑙𝑜𝑔2

𝑝(𝑡1, 𝑡2, … 𝑡𝑛|𝑡𝑢)

𝑝(𝑡1)𝑝(𝑡2) … 𝑝(𝑡𝑛)= −𝑙𝑜𝑔2

𝑝(𝑡1, 𝑡2, … 𝑡𝑛 , 𝑡𝑢)

𝑝(𝑡1)𝑝(𝑡2) … 𝑝(𝑡𝑛)𝑝(𝑡𝑢)(1)


Paper 23

In this model, the amount of surprise is represented by the

divergence of the actual joint distribution (the upper part of

the log fraction) from the expected individual distribution

(the lower part of the log fraction). The larger the ratio, the

more likely that combination of topics is in the corpus. A

smaller ratio indicates a rare topic combination, and

therefore a higher surprise score. Our surprise calculation is

a variation of Mutual Information (MI), an established

metric in the text mining field [12]. MI measures how much

information several random variables share, or how much

the uncertainty (entropy) of one variable is reduced by

knowing the other variables. The random variables are the

topics t1, t2, … tn. We label this approach MI.

In our second approach to modeling surprise (s2) we view

each article as “a bag of co-occurring themes”. Probabilistic

topic modeling [9] is a set of algorithms that discover the

themes that run through them and how these themes are

connected. LDA (Latent Dirichlet Allocation) is a popular

example of such a model [10]. According to LDA, each

article is generated by choosing the latent themes zi

probabilistically and then for each latent theme choosing

the words wi probabilistically, represented as:

𝑑 = ∏ 𝑝(𝑧𝑖)𝑝(𝑤𝑖|𝑧𝑖)𝑘𝑖=1 (2)

where p is the distribution of the latent themes in an

article, zi is the latent theme i, 𝑝 is the distribution of the

words for the latent theme zi, and wi is the word i. In fact,

the generated model is the likelihood of observing such an

article with k latent themes. We apply this model such that

the expected likelihood of observing a typical article given

a user preferred topic will be the likelihood of observing an

“average” article by averaging the likelihood of all articles

in the preferred topic corpus. Each individual article’s

divergence from this expectation is that article’s degree of

surprise. We use the Kullback Leibler (KL) divergence as

the divergence measure since it is a common way to

evaluate the divergence between two probability

distributions. The surprise score is calculated as Equation 3,

where p is the distribution of the latent themes in an

article, q is the distribution of a typical article, and i is the

index of the latent themes. We label this approach as KL.

𝑠2 = KL(p, q) = ∑ pilog2pi

qi

ki=1 (3)

For the Personalized Value Module, we use an interactive

process to gather data about the user. Previous research has

been ambiguous on the meaning of value: some researchers

use “interest” [5], while others use “usefulness” [1]. Having

the WTP-EU construct in mind, we asked participants two

questions: usefulness to represent the WTP (rational) aspect,

and interestingness to represent the EU (emotional) aspect.

Specifically, participants were asked to rate on two 5-point

Likert-scales to indicate how useful and how interesting

they think that article is. The sum of the two ratings will

serve as the value rating.

Health News Corpus

We scraped the health news articles from Medical News

Today (MNT) since its launch in 2003 to the present. MNT

is a leading website to provide quality and updated health

news for average readers in the U.S. The corpus contains

268,850 articles, classified into 135 health topics, such as

diabetes, heart disease, anxiety, women’s/men’s health, by

health professionals working with MNT. Most articles have

multiple topic labels, as summarized in Table 1 below.

After reviewing the 135 topics, we removed topics that are

too broad (e.g., primary care, surgery, medical innovation),

too narrow (e.g., statin), or too research-focused (e.g., stem

cell research). The final list contained 100 topics. Total no. of articles 268,850

Total no. of topic labels 135

Total no. of articles with 1 topic label 86,364

Total no. of articles with 2 topic labels 83,060



Table 1. Summary of the health news corpus

For the MI approach, these topic labels were leveraged as

the topics (ti) as in Equation 1. The value of s1 for each

article was calculated using Python’s Math and Sci-Kit

Learn (sklearn) packages. For the KL approach to

calculating s2, the LDA topic modeling technique was

applied to each sub-corpus of the 100 topics (each sub-

corpus corresponding to each topic) to generate those latent

themes on the 100 sub-corpora. A high efficiency topic

modeling tool, Gensim, as a Python library was used for the

LDA analysis and calculating s2 for each article.

To demonstrate the computational result of the surprise

scores, let us take the topic diabetes for example. Figure 4

presents the cumulative distribution of the Z-scores of s1

and s2 respectively of all the 20,458 articles involving the

topic diabetes. For the s1 distribution in Figure 4a, a Z-score

at the right side (above the average 0) indicates a topic

combination with diabetes potentially surprising, co-

occurring less often than expected by chance. One such

example is the combination of bipolar disorder, depression,

and diabetes. For the KL-divergence distribution in Figure

4b, a Z-score at the right side indicates a larger divergence

from an expected article. This suggests a potentially

surprising latent theme distribution, such as an article on

diabetes also talking about kidney, brain, and sleeping

disorder. Both curves are roughly parallel to a normal

distribution, suggesting most articles are centered around

the average level of surprise. Although surprise is a

desirable feature in any news, this health news corpus is

grounded heavily in conventional knowledge (the grey

area) and occasionally reaches out to the highly surprising

zone (greater than the 90th

percentile).

Table 2 lists some examples of the most and the least

surprising articles based on s1. As we can see in the topic

label column, the most surprising articles (highlighted in

pink) have those rare topic combinations, like diabetes

together with infectious disease and public health; whereas

the least surprising articles (highlighted in grey) have

combinations like diabetes, nutrition, and obesity, which

are more common.


Paper 23

Figure 4. MI (a) and KL-Divergence (b) distributions for all the articles involving the topic diabetes

Example Article Titles Topic Labels s1 Z-score

Diabetes: Mining The 'Wisdom of

Crowds' To Attack Disease

Diabetes,

Infectious Diseases,

Public Health

5.40 2.38

Increased Diabetes Risk In HIV-

Positive Children

Diabetes,

HIV, Pediatrics

5.02 2.26

Dogs Sniff Out Diabetes

Diabetes, Psychology,

Public Health

4.90 2.22

Exploring Diabetes' Link to

Eating Disorders

Diabetes,

Nutrition,

Obesity,

Eating Disorder

-12.54 -3.55

Stored fat fights against the

body's attempts to lose weight

Diabetes,

Obesity,

Eating Disorder

Heart Disease

-12.85 -3.65

New research exposes the health risks of fructose and sugary

drinks

Diabetes, Obesity

Heart Disease

Gout

-14.68 -4.26

*the pink area indicates surprising articles whereas the grey area indicates

non-surprising ones

Table 2. Article examples based on s1

To get a sense of what latent themes generated by the LDA

analysis, Table 3 lists the five highest weighted themes

generated from the diabetes sub-corpus (collection of

articles with the topic diabetes). Since LDA treats each

latent theme as “a bag of words”, the top five words that

have the highest probability in each theme are listed to

represent the semantic meaning of that theme, which could

be, from left to right, patient, diabetes study, insulin &

glucose, treatment, and research experiment.

Theme 1 Theme 2 Theme 3 Theme 4 Theme 5

diabetes diabetes insulin patient cell

patient type mouse metformin beta

people study fat treatment university

risk cell glucose control pancreas

blood researcher protein sugar new

Table 3. Five most highly weighted themes

The corpus centroid distribution over these five themes,

which represents a typical diabetes article, is presented as

the red line in Figure 5. The other five lines in Figure 5 are

the five example articles as detailed in Table 4. In Figure 5,

the articles of “Oklahoma student”, “carb quality”, and

“reduce cavities” are highly divergent from the centroid,

suggesting high levels of surprise; whereas “aging”, “post-

term delivery”, and “Schizophrenia” are closer to the

centroid, implying lower levels of surprise.

Figure 5. Centroid and example articles distributions over the

top five latent themes

Article shorthand Example Article Titles s2 Z-score

Oklahoma student Diabetes Management in Schools

Act Will Help Ensure Oklahoma

Students with Diabetes Are Safe at

School

2.57 6.41

Carb quality Carb quality matters - reduces

diabetes risk

2.17 5.06

Reduce Cavities What reduces cavities, food or

fluoride?

2.09 4.76

Aging Diabetes, arthritis, cancer and

Alzheimer's disease: strategy proposed for preventing diseases of

aging

0.20 -1.75

Post term delivery Post-term delivery raises risk of

complications and illness for

newborns

0.19 -1.77

Schizophrenia Dissecting the Relationship Between

Schizophrenia and An Increased

Risk of Type 2 Diabetes

0.18 -1.83

*the pink area indicates surprising articles whereas the grey area indicates non-surprising ones

Table 4. Article examples based on s2

How StumbleOn Works

StumbleOn recommends health news to users on a session

basis. Each user is initially asked to choose five to ten

topics from the list of 100 topics to indicate their

preferences. Then a recommender session contains five to

ten articles with one article corresponding to one preferred

(a) (b)


Paper 23

topic. Since the study will not evaluate the impact of the

specific placement of articles in a list, the rank of the

articles is randomized for each session. Within each

session, users are able to view and click on article titles,

open a different window to browse the article content, and

provide ratings at the side of the window. They are also

encouraged to save their favorite articles to a shopping cart

for later reading. A screenshot of a session is presented in

Figure 6.

Figure 6. Screenshots of StumbleOn

EVALUATION STUDY

After the implementation of StumbleOn, a user study was

conducted to evaluate which computational approach

produces results that best match what users perceive as

simultaneously surprising and valuable.

We adopted a two-way factorial design to present the

different approaches to participants. One dimension in the

design is the computational approach: MI or KL, for which

the order of presentation was counterbalanced. The other

dimension is the computational score level: high and low.

High means articles with a top 33rd

percentile surprising

score (s1 or s2) within that preferred topic sub-corpus while

low means a bottom 33rd

percentile s1 or s2. In order to

better explain how StumbleOn recommends session-based

articles, let us assume one participant chooses six preferred

topics: anxiety, diabetes, depression, sleep disorder, breast

cancer, and hypertension. Each recommended session with

six articles is shown in Figure 7 below.

In addition to those four groups in Figure 7, to investigate if

the computational approaches beat blind luck via

randomness, a baseline group was created to randomly pick

articles from a preferred topic sub-corpus to recommend to

participants. The study is a within-subject design, meaning

each participant experienced all five groups.

Figure 7. Examples of recommended sessions

Thirty graduate students in the College of Computing and

Informatics were recruited, all of whom were enrolled in

computing programs and none of whom had completed any

formal medical education. After the introduction and

obtainment of consent, an entry questionnaire was

administered to collect demographic information and

preferred health topics they would like to read about. Each

participant then experienced ten recommended sessions

with articles corresponding to his/her preferred topics, each

two consecutive sessions representing one of the five

manipulation groups. Participants were encouraged to click

on whatever articles they would like to read. For each

clicked article, they were required to provide their ratings

on a 5-point Likert-scale on four questions: whether the

article content is novel, surprising, useful, and interesting.

The first question represents the concept of novelty, and

was asked to explore the relationship between novelty and

serendipity. The second question represents the surprise

component while the third and the fourth questions

represent the two aspects of the value component. In

addition to these four Likert-scale questions, participants

also needed to answer yes or no on the fifth question:

whether the article sparks a new area of interest. This last

question is inspired by the idea that serendipity is the result

of a divergent process [14]. This model suggests that it

produces a divergent path toward an unexpected new area

that the user was not previously aware of.

EVALUATION RESULTS

On average, the thirty participants had 11.4 years of seeking

information online and 5.6 years of seeking online health

information. Twenty-seven of them have listed one of the

major search engines (Google, Yahoo, etc) as their primary

source for health information. Very few of them mentioned

specialist medical websites, such as WebMD and

MedlinePlus. Out of the 100 health topics, the participants

have chosen 62 as their preferred topics, reasonable

coverage of what was available. Among the chosen topics,

nutrition, mental health, sleep disorder, depression,

diabetes, and back pain were the most frequently chosen.


Paper 23

User Clicks and Number of Saved Articles

User clicks and saves have been regarded as an implicit

indicator of the potential interest in search engine studies

[27]. These can also serve as indicators of the user’s

potential interest in the recommended articles. We tracked

the number of clicked and saved articles for the five groups

in each session. The result in Figure 8 shows that on

average participants clicked 1.4 articles and saved 0.7 per

session, suggesting they were very selective in what they

clicked or saved. For the number of clicks, a one-way

repeated measure ANOVA test shows there was no

significant difference among the different approach groups

(F (4, 116) = 2.016, p = 0.097), suggesting these groups

seemed equally appealing in attracting user clicks.

However, for the number of saved articles, the difference

was significant (F (4, 116) = 3.129, p = 0.015). Tukey’s

post-hoc test shows that the KL-Low group had

significantly higher number of saves than all the other

groups and the KL-High group had significantly lower

number than all the other groups. On the surface, this

suggests that a higher surprise score (at least as measured

by the KL approach) would attract fewer clicks and saves,

but the picture is more interesting when value is included.

Figure 8. The average and standard deviation of the number

of clicks and saves

Surprise Ratings for each Approach

Participants’ ratings of surprise are summarized in Figure 9.

As the result, the MI-High session received the highest

average rating whereas the MI-Low session the lowest. One

way repeated measure ANOVA test shows there was

significant difference among these groups (F(4, 463) =

2.208, p = 0.067). Tukey’s post-hoc method identifies two

tiers with significant difference. Tier 1 with significantly

higher surprise ratings is the Baseline and MI-High groups

and Tier 2 with significantly lower surprise ratings is the

MI-Low, KL-Low, and KL-High sessions. It is interesting

to note that for the KL approach, the higher level of

computational surprise in fact resulted in lower level of

user-perceived surprise. For the Baseline session (randomly

selected articles), the user-perceived surprise was

unexpectedly high, very close to the MI-High group.

Figure 9. The average and standard deviation of the surprise

ratings

Serendipity Rating for each Approach

Serendipity is proposed to arise from the discovery of

information that is surprising and valuable. In this study,

the concept of value is further decomposed into being

useful and being interesting. We evaluate the serendipity

rating of an article as the aggregate of the three ratings of

surprise, usefulness, and interestingness, as in Equation 4

with the assumption that surprise and value carry the same

weight for serendipity; usefulness and interestingness have

the same weight for value:

serendipity = surprise +1

2 usefulness +

1

2 interestingness (4)

Figure 10. The average and standard deviation of the

serendipity ratings

Using this formula, we calculate the average serendipity

ratings for each approach. Shown in Figure 10, the average

serendipity rating is the highest for the MI-High approach

and the lowest for the Baseline. A one-way repeated

measure ANOVA test shows significant difference among

the five groups (F(4,463) = 3.405, p = 0.009). Tukey’s post-

hoc method reveals three tiers with significant difference.

The high tier is the MI-High and KL-Low groups, the

middle tier is the MI-Low and KL-High groups, and the

Base group is in the low tier. Similar to the surprise ratings

in Figure 8, the serendipity rating was unexpectedly higher

for the KL-Low group than the KL-High group, suggesting

that the computational model did not align well with what

the participants perceived. In a departure from the surprise

ratings, user-perceived serendipity was the lowest for the

Baseline group, suggesting the surprise identified via

randomness was not valued as much as that via

computational approaches. Random surprise did not lead to


Paper 23

serendipitous discovery at the same rate as the approaches

based on computational surprise measures.

Previous literature reports that most users value the

usefulness of health information more than the enjoyment

[47]. However, participants’ comments on their surprise

touched on both aspects of interest, with comments like

“enjoy learning a nice variety of stuff” being more

aesthetic-emotional, whereas “relevant to my father’s

condition” was more instrumental-rational. We find a

strong correlation between the ratings of usefulness and

interestingness (r = 0.73, p < 0.0001, as shown in Table 5),

suggesting that when an article is useful, it is very likely to

be interesting as well. This suggests some users did not

express a clear-cut distinction between the two concepts.

When investigating their correlations with surprise,

however, we find a stronger correlation between being

interesting and being surprising (r = 0.63, p < 0.0001) than

that between being useful and being surprising (r = 0.54, p

< 0.0001). This means that while surprising health

information was quite likely to be both interesting and

useful, it was significantly more likely to be interesting than

it was to be useful.

surprise interest usefulness novelty

surprise

interest 0.63*

usefulness 0.54* 0.73*

novelty 0.72* 0.59* 0.48

spark 0.49 0.59* 0.52* 0.45

* indicates significance at the 0.05 level

Table 5. Pearson correlation matrix (n = 497)

Surprise, Novelty, and Inspiration

Since novelty is a well-studied research topic in the

recommender research community, we consider the

relationship between novelty and surprise in our results. We

claim that a novel item is not necessarily serendipitous, and

but all serendipity is novel. To support this claim, we

examined the correlation between the ratings of surprise

and novelty and identified a strong correlation between the

two concepts (r = 0.72, p < 0.0001, as in Table 5), meaning

being surprising is also highly likely to be novel. If we

combine the participant ratings 4 and 5 as positive, 1 and 2

as negative, and ignore the neutral ratings, the cross

distribution of positives and negatives for the five groups is

listed in Table 6. As we can see, most articles were both

surprising and novel, or neither. We still find several cells

with zeros, indicating that for both the MI and KL

approaches, there was no article that was surprising but not

novel, backing up our claim that surprise is a necessary

component of novelty.

As mentioned before, surprises that spark a new area of

interest are highly desirable. A moderate (Pearson)

correlation was found between surprising and sparking new

interest, but it was not significant (r = 0.49, p = 0.23). The

breakdown distribution by approach indicates that there is a

non-trivial number of articles that were surprising but did

not spark a new area of interest or vice versa.

Table 6. Cross distribution of surprising and novel articles

Base MI-Low MI-

High

KL-Low KL-

High

Sparking Y

N

Y

N

Y

N

Y

N

Y

N

Surprisin

g (Yes) 35 11 27 16 39 10 40 10 33 9

Surprisin

g (No) 2 8 3 10 2 15 5 16 2 22

Table 7. Cross distribution of surprising and sparking articles

Exit Questionnaire: What Participants Felt about Each Approach

Participants were asked to fill out a 10-item questionnaire

shown in Table 8, representing five latent dimensions of the

concept of serendipity, adapted from the instrument

developed by McCay-Peet and Toms [32]. These five

dimensions are: enabled connections (Q1, Q6, and Q10),

introduced the unexpected (Q2 and Q7), presented variety

(Q3, Q4, and Q8), triggered divergences (Q5), and induced

curiosity (Q9). The ratings on these ten questions are

summarized in Figure 11. Since the questionnaire was

administered immediately after experiencing each

computational approach (MI, KL, or Baseline), the ratings

were aggregated across the levels of computational surprise.

As in Figure 10, the ratings across all the approaches were

generally higher for Q1, Q2, Q8, and Q9; but lower for Q4,

Q5, and Q7, implying the system’s strength at presenting

variety and making connections, and meanwhile the

weakness at triggering divergence. The MI approach

usually received higher ratings than KL and Baseline.

Follow-Up Interview: What Participants Say About Surprise and Value

Upon finishing the last recommended session, participants

were asked about whether they encountered surprising

articles, whether they liked those surprising pieces, and

whether they thought the surprising news was useful,

interesting, or both.

Whether the users encountered surprise: All participants

indicated that they came across surprising articles. While

some participants found some articles as a whole were

surprising, others found only a paragraph or two had

surprising content. When asked to give examples of

surprising articles, one participant said “it is a surprise to

me that an article talked about how poor dental hygiene

may lead to breast cancer”. Another participant stated “I

read an article saying how obesity is viewed differently

among Muslims and Christians. That was completely

surprising because I didn’t think religion plays a role.”

Another participant cited a piece of news about a head

transplant surgery as very “shocking” to her. One

Base MI-Low MI-

High KL-Low

KL-

High

Novel Y

N

Y

N

Y

N

Y

N

Y

N

Surprisin

g (Yes) 39 2 41 0 48 0 48 0 36 0

Surprisin

g (No) 3 3 3 8 3 7 2 8 4 13


Paper 23

participant stated that he expected that kids who were

bullied would be more likely to develop depression.

However, to his surprise, an article said that kids who

bullied others were also more likely to develop depression.

Figure 11. The average and standard deviation of the

participant ratings on the exit questionnaire

Q1 I was able to examine a variety of health topics

Q2 I explored many topics that normally I do not examine

Q3 Unexpected titles caught my eye and made me click on

Q4 Unexpected words and phrases in the snippet caught

my eye and made me click on the title

Q5 Unexpected words and phrases in the article contents sparked my thinking

Q6 The system enables me to make connections between

different health topics

Q7 I stumbled upon unexpected health topics

Q8 The system encouraged me to browse and explore

Q9 I found myself pausing to look at things more closely

Q10 I could return to topics that I had explored earlier

Table 8. Ten questions in the exit questionnaire

Whether the users like the surprise: Almost all participants

indicated that they liked at least some of the surprising

articles they encountered. One participant stated, “the

surprising ones are actually some of my favorites.” These

participants voiced positive opinions, saying they liked the

articles because they learned something they had not

previously known. Their comments can be summarized as

either presenting new information on a general level or

presenting specific new ideas”. Some other participants

also gave some negative examples of surprising news. One

participant mentioned that he selected flu as his preferred

topic and several articles on swine flu were completely

surprising and irrelevant to his need, even though he

understood the association between flu and swine flu. One

participant mentioned that smoking was his selected topic

but those articles on school policies on student smoking or

law enforcement were unexpected and not what he was

looking for. One participant mentioned that articles

involving mice eating habits or the way cats respond to

different medications, although surprising, were trivial and

irrelevant to her life. Another participant said she

questioned some of the surprising articles, such as an article

that introduced a newly discovered anti-aging hormone and

a new diet supplement product. She suspected that the

author might work with some pharmacy company that

aimed at promoting the sales of this product. Several

participants stated they were not interested in research-

oriented articles. They favored “easy reads” that were for

“people without any medical background”.

None of the participants mentioned surprise explicitly as a

reason for clicking on an article. They used words and

phrases like “catchy”, “caught my attention”, “curious”,

“seemed interesting”, “looked like what I was looking for”,

or “the headline was informative”. One participant just said

“just want to check it out what it is.” Responses were

similarly diverse when participants were asked why they

saved some of the articles into their cart. Some participants

said “for future in-depth reading” because they did not have

enough time to read it thoroughly in the lab setting of the

study. Others mentioned “wanting to email them to others”.

One participant said that he did not save any articles

because he knew he would be able to access any of these

articles through Google. Whether the users thought surprising results were

interesting or useful: Participants mentioned interestingness

more than usefulness when describing the value of

surprising articles, although they did touch on both. One

participant said an article about marijuana causing vomiting

disorder was interesting to know but was not immediately

useful to his life since he did was not a marijuana user.

Another participant found surprising an article on how to

read below the eyes instead of into the eyes to read people.

She said that the article was an interestingly new kind of

interpersonal skill for her, although she was not likely to

user the technique herself. One mentioned an article about a

research finding on a new mechanism for memory

formation. He said he would have never learned this

without the opportunity to participate in our study. He

favored the chance to gain such esoteric knowledge, even if

it was unlikely to be directly relevant to his daily life.

Another participant mentioned that she “got into” an

analysis saying that cigarette manufacturers had increased

the level of nicotine in cigarettes by 11% over the recent

seven-year period. She enjoyed reading the story and being

aware of that fact, even though she did not smoke. Very

few participants said those surprising documents were

useful. The few useful and surprising documents included

one suggesting the best time to take a nap during work, as

well as one revealing a link between a sleeping aid and

blood sugar level (useful to the participant’s father).

There were some surprising articles that the participants felt

could be useful in the future, but not at this moment. One

example was saying that young children living in big cities

are at increased risk for brain inflammation and

neurodegenerative changes: this participant had relatives

with children and planned to have his own in the future.

One point that is noteworthy is that several participants did

not distinguish well the two concepts of usefulness and

interestingness. Once asked, they started to think about

them separately. One participant said “most of the articles

that were interesting were actually useful to me. Because


Paper 23

they were useful they were interesting”. To these

participants, usefulness made information interesting.

DISCUSSION AND CONCLUSION

We have explored whether our computational measures of

surprise correlate with user-perceived surprise. We

presented high and low levels of surprise to users in our

study of computational serendipity, based on each of our

approaches to modelling surprise: MI and KL. We found

that for the MI approach, as expected, the higher

computational scores attracted more clicks, more saved

articles, and received both higher surprise and higher

serendipity ratings. Our interpretation of these results is that

topic co-occurrence likelihood serves as a good predictor of

the level of surprise in health-related documents. Rarer

topic combinations are more likely to contain surprising

content, such as the co-occurrence example of dental

hygiene and breast cancer given by one participant.

However, for the KL approach, the lower surprise scores

consistently outperformed the higher ones in participants’

ratings of surprise. The essence of the KL approach is to

use the KL-divergence to measure how far the content of an

article is from a typical article in a corpus. When we

manually checked into the content of those “atypical”

articles, we found a number of articles on policy, insurance,

conference announcements, management, etc. Such atypical

content captured peripheral themes in the sub-corpus, which

turned out not to be a good proxy for user surprise. Many

participants showed indifference to these themes. “Typical”

articles closer to the centroid, on the other hand, contained

those more likely to elicit feelings of surprise and value.

The user perceived surprise ratings were moderate for the

Baseline (random) group. A possible explanation is that the

nature of any news story, including health news, has some

surprising element at least to some people. Our dataset is

quite diverse, meaning that randomly picked articles may

possibly surprise users. In addition, participants always felt

surprise about why some articles were recommended given

their selected topic preference, such as articles on swine flu

when they chose flu; news about herbal tea when they

selected diabetes. Their surprise was because of a topic that

did not match of their preference, or disagreement with a

label assignment to an article, but not because of the

content of the article. When considering whether the

surprising article was also valuable, reflected by the

serendipity ratings, the Baseline group dropped to the

bottom of the five groups. This means that the articles

presented based on computational models of expectations

and surprise were much more positively received than those

based on random selection. The power of computational

surprise does not lie in a metric of unexpectedness alone,

but instead in finding valued surprises: serendipity.

Surprising content can be unknown to a user, such as the

example of obesity perspectives in Christians and Muslims.

It could also come from unconventional or bold ideas, such

as the head transplant surgery. Surprising content also

confronts participants’ common sense, such as the article

about bullies and depression. From the user’s perspective

surprise stemmed from novelty, boldness, and contradiction.

From the corpus perspective, surprise arose from rare co-

occurrences, such as obesity and religion, as well as

atypical content, such as head transplants and the

depression of bullies.

Our study asked users whether recommended articles

sparked a new area of interest. Participants were quite

conservative in giving a yes to this question, when

compared to the questions on surprise and value. During the

interview, some participants mentioned that they clicked on

articles out of curiosity or they felt curious to learn more

after reading an article. This feeling of curiosity is desirable

in information-seeking behavior since it plays an essential

role in exploration [8] and the divergent process described

by Campos and Figueiredo [14]. By contrast, some short-

term surprise may be less desirable if it does not lead to

open-ended discovery.

Surprise modeling, as a component of serendipity

prediction, heavily relies on natural language understanding

technologies, which are an active research topic and

ongoing challenge in the text mining field. Both of our

approaches, MI and the KL, are fundamentally similar in

that they reduce textual information to a vector-based

topic/theme distribution. This representation has provided

several benefits such as a more structured representation for

algorithmic processing such as likelihood and similarity

calculations, and text understanding at a high level of

abstraction. These representations work well for co-

occurrence and association based surprise, but not as well

for content-based surprise. For the content-based surprise,

more advanced natural language processing techniques are

needed before deeper content-based expectations can be

modeled. In addition to a coarse topic, such as ti in Equation

1, does not always capture the nuanced content that may

generate the surprise feeling. Finer-grained themes, such as

zi in Equation 2, may lead to peripheral latent themes that

loses the main stream information of a corpus. How to

capture expectations across these different levels of

granularity is a question for future research.

This study presents a framework that models the concept of

serendipity as a combination of surprise and value. The

framework was implemented using two computational

approaches to predicting user surprise, which were then

evaluated in a user study. Our results show that the MI,

approach based on topic co-occurrence outperformed the

KL approach and our random baseline in predicting when

users would rate a document as surprising and serendipitous.

As to the broader impact, This work addresses a core

problem of accuracy-oriented search and recommender

systems. Serendipitous retrieval has the potential to

transform the way digital systems deliver information by

shifting from reinforcing similar information to facilitating

unexpected discoveries. This will provide users with

expanded access to information that is surprising, yet

beneficial. This to a variety of domains that can benefit

from such a model.


Paper 23

REFERENCES

1. Panagiotis Adamopoulos and Alexander Tuzhilin. 2015.

On unexpectedness in recommender systems: Or how to

better expect the unexpected. ACM Transactions on

Intelligent Systems and Technology (TIST) 5, 4

(2015),54.

2. Naresh Kumar Agarwal. 2015. Towards a definition of

serendipity in information behaviour. Information

Research: An International Electronic Journal 20,

3(2015), n3.

3. Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson,

and Samuel Ieong. 2009. Diversifying search results. In

Proceedings of the 2nd, ACM International Conference

on Web Search and Data Mining. ACM, 5–14.

4. Eric Alexander, Joe Kohlmann, Robin Valenza, Michael

Witmore, and Michael Gleicher. 2014. Serendip: Topic

model-driven visual exploration of text corpora. In IEEE

Conference on Visual Analytics Science and Technology

(VAST),173-182.

5. Paul André, Jaime Teevan, and Susan T Dumais. 2009.

From x-rays to silly putty via Uranus: serendipity and its

role in web search. In Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems.

ACM, 2033–2036.

6. Eytan Bakshy, Solomon Messing, and Lada A Adamic.

2015. Exposure to ideologically diverse news and

opinion on Facebook. Science 348, 6239 (2015), 1130–

1132.

7. Elinor Barber and Robert K Merton. 2004. The travels

and adventures of serendipity. A Study in Sociological

Semantics and the Sociology of Science. Princeton

(2004).

8. Daniel E Berlyne. 1966. Curiosity and exploration.

Science 153, 3731 (1966), 25–33.

9. David M Blei. 2012. Probabilistic topic models.

Communications of the ACM 55, 4 (2012), 77–84.

10. David M Blei, Andrew Y Ng, and Michael I Jordan.

2003. Latent dirichlet allocation. Journal of Machine

Learning Research 3, Jan (2003), 993–1022.

11. Pia Borlund. 2003. The concept of relevance in IR.

Journal of the Association for Information Science and

Technology 54, 10 (2003), 913–925.

12. Gerlof Bouma. (2009). Normalized (pointwise) mutual

information in collocation extraction. In Proceedings of

the Conference of the German Society for

Computational Linguistics and Language Technology.

31–40.

13. Jacquelyn Burkell, 2012. Encountering New Music: Are

Recommender Systems a Help or a Hindrance? In

Proceedings of Serendipity, Chance, and the

Opportunistic Discovery of Information Research

(SCORE) Workshop. April 28- May 1, Montreal,

Canada.

14. José Campos and Antonio Dias de Figueiredo. 2002.

Programming for serendipity. In Proceedings of the

AAAI Fall Symposium on Chance Discovery–The

Discovery and Management of Chance Events. AAAI,

48-60.

15. Jaime Carbonell and Jade Goldstein. 1998. The use of

MMR, diversity-based re-ranking for reordering

documents and producing summaries. In Proceedings of

the 21st. Annual International ACM SIGIR Conference

on Research and Development in Information Retrieval.

ACM, 335–336.

16. Charles LA Clarke, Maheedhar Kolla, Gordon

VCormack, Olga Vechtomova, Azin Ashkan, Stefan

Büttcher, and Ian MacKinnon. 2008. Novelty and

diversity in information retrieval evaluation. In

Proceedings of the 31st. Annual International ACM

SIGIR Conference on Research and Development in

Information Retrieval. ACM, 659–666.

17. Edward De Bono. 1990. Lateral thinking for managers.

Penguin Books.

18. Brenda Dervin and Michael Nilan. 1986. Information

needs and uses. Annual Review of Information Science

and Technology 21 (1986), 3–33.

19. Kazjon Grace and Mary Lou Maher. 2016. Surprise-

Triggered Reformulation of Design Goals, In

Proceedings of AAAI. 3726–3732

20. Kazjon Grace, Mary L. Maher, Douglas Fisher, and

Katherine Brady, 2015. A data-intensive approach to

predicting creative designs based on novelty, value and

surprise. International Journal of Design, Creativity and

Innovation 3(3-4):125-147.

21. Kazjon Grace, Mary Lou Maher, David Wilson, and

Nadia Najjar. 2016. Personalised Specific Curiosity for

Computational Design Systems, In Proceedings of

Design Computing and Cognition. Springer, 593-610.

22. Ted Gup. 1998. Technology and the end of serendipity.

The Education Digest 63, 7 (1998), 48

23. Jonathan L Herlocker, Joseph A Konstan, Loren G

Terveen, and John T Riedl. 2004. Evaluating

collaborative filtering recommender systems. ACM

Transactions on Information Systems (TOIS) 22, 1

(2004), 5–53.

24. Leo Iaquinta, Marco De Gemmis, Pasquale Lops,

Giovanni Semeraro, Michele Filannino, and Piero

Molino. 2008. Introducing serendipity in a content-based

recommender system. In Proceeding of the 8th. IEEE

International Conference on Hybrid Intelligent Systems

(HIS). IEEE, 168–173.

25. Laurent Itti, and Pierre F. Baldi., 2006. Bayesian

surprise attracts human attention. In Advances in neural

information processing systems, 547-554.

26. Marius Kaminskas and Derek Bridge. 2016. Diversity,

Serendipity, Novelty, and Coverage: A Survey and

Empirical Analysis of Beyond-Accuracy Objectives in


Paper 23

Recommender Systems. ACM Transactions on

Interactive Intelligent Systems (TiiS) 7, 1 (2016), 2.

27. Diane Kelly and Jaime Teevan. 2003. Implicit feedback

for inferring user preference: a bibliography. In ACM

SIGIR Forum, Vol. 37(2). ACM, 18–28.

28. Shoshana Loeb. 1992. Architecting personalized

delivery of multimedia information. Communications of

the ACM 35, 12 (1992), 39–47.

29. Irene Lopatovska and Hartmut B Mokros. 2008.

Willingness to pay and experienced utility as measures

of affective value of information objects: Users’

accounts. Information Processing and Management 44,

1 (2008), 92–104.

30. S Makri, EG Toms, L McCay-Peet, and A Blandford.

2011. Encouraging serendipity in interactive systems: an

introduction. In Proceedings of the 13th. IFIP TC13

International Conference on Human-Computer

Interaction, Springer-Verlag, Lisbon, 728-729.

31. Harry Markowitz. 1952. Portfolio selection. The Journal

of Finance 7, 1 (1952), 77–91.

32. Lori McCay-Peet and Elaine Toms. 2011. Measuring the

dimensions of serendipity in digital environments.

Information Research: An International Electronic

Journal 16, 3 (2011), n3.

33. Lori McCay-Peet, Elaine G Toms, and Anabel Quan-

Haase. 2016. SEADE Workshop Proposal-The

Serendipity Factor: Evaluating the Affordances of

Digital Environments. In Proceedings of the 2016 ACM

on Conference on Human Information Interaction and

Retrieval. ACM, 341–343.

34. William Mckeen. 2006. The endangered joy of

serendipity. The modern world makes it harder to

discover what you didn’t know you were looking for. St

Petersbuirg Times, March 26 (2006). Retrieved July 02,

2016 from

http://www.sptimes.com/2006/03/26/news_pf/Perspectiv

e/The_endangered_joy_of.html

35. Francis L Miksa. 1992. Library and information science:

two paradigms. Conceptions of Library and Information

Science: Historical, Empirical and Theoretical

Perspectives. Londres, Los Angeles: Taylor Graham

(1992), 229–252.

36. Javed Mostafa, Snehasis Mukhopadhyay, and Mathew

Palakal. 2003. Simulation studies of different

dimensions of users’ interests and their impact on user

modeling and information filtering. Information

Retrieval 6, 2 (2003),199–223.

37. Tomoko Murakami, Koichiro Mori, and Ryohei Orihara.

2007. Metrics for evaluating the serendipity of

recommendation lists. In Annual Conference of the

Japanese Society for Artificial Intelligence. Springer,

40–46.

38. Xi Niu and Fakhri Abbas. 2017. A Framework for

Computational Serendipity. In Adjunct Publication of

the 25th. Conference on User Modeling, Adaptation and

Personalization. ACM, 360–363.

39. Kensuke Onuma, Hanghang Tong, and Christos

Faloutsos. 2009. TANGENT: a novel, ’Surprise me’,

recommendation algorithm. In Proceedings of the 15th.

ACM SIGKDD International Conference on Knowledge

Discovery and Data Mining. ACM, 657–666.

40. Eli Pariser. 2011.The filter bubble: what the internet is

hiding from you. Londong: Viking (2011).

41. Victoria L Rubin, Jacquelyn Burkell, and Anabel Quan-

Haase. 2011. Facets of serendipity in everyday chance

encounters: a grounded theory approach to blog analysis.

Information Research 16, 3 (2011).

42. Yue Shi, Xiaoxue Zhao, Jun Wang, Martha Larson, and

Alan Hanjalic. 2012. Adaptive diversification of

recommendation results via latent factor portfolio. In

Proceedings of the 35th. International ACM SIGIR

Conference on Research and Development in

Information Retrieval. ACM, 175–184.

43. Jennifer Thom-Santelli. 2007. Mobile social software:

Facilitating serendipity or encouraging homogeneity?

IEEE Pervasive Computing 6, 3 (2007), 46–51.

44. Alice Thudt, Uta Hinrichs, and Sheelagh Carpendale.

2012. The ohemian bookshelf: supporting serendipitous

book discoveries through information visualization.

In Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems,1461-1470.

45. Saúl Vargas and Pablo Castells. 2014. Improving sales

diversity by recommending users to items. In

Proceedings of the 8th. ACM Conference on

Recommender Systems. ACM, 145–152.

46. Jun Wang and Jianhan Zhu. 2009. Portfolio theory of

information retrieval. In Proceedings of the 32nd.

International ACM SIGIR Conference on Research and

Development in Information Retrieval. ACM, 115–122.

47. Martin Wiesner and Daniel Pfeifer. 2014. Health

recommender systems: concepts, requirements, technical

basics and challenges. International Journal of

Environmental Eesearch and Public Health 11, 3

(2014), 2580–2607.

48. Yuan Cao Zhang, Diarmuid Ó Séaghdha, Daniele

Quercia, and Tamas Jambor. 2012. Auralist: introducing

serendipity into music recommendation. In Proceedings

of the 5th. ACM International Conference on Web

Search and Data Mining. ACM, 13–22.

49. Cai-Nicolas Ziegler, Sean M McNee, Joseph A Konstan,

and Georg Lausen. 2005. Improving recommendation

lists through topic diversification. In Proceedings of

the14th. International Conference on World Wide Web.

ACM, 22–32.


Paper 23

surprise me if you can: serendipity in health...

Documents