the business value of recommendations: a privacy-preserving …people.stern.nyu.edu/padamopo/the...

The Business Value of Recommendations

Thirty Sixth International Conference on Information Systems, Fort Worth 2015 1

The Business Value of Recommendations: A Privacy-Preserving Econometric Analysis

Completed Research Paper

Panagiotis Adamopoulos Department of Information, Operations,

and Management Sciences Leonard N. Stern School of Business

New York University [email protected]

Alexander Tuzhilin Department of Information, Operations,

and Management Sciences Leonard N. Stern School of Business

New York University [email protected]

Abstract

We study the effectiveness of different types of mobile recommendations and their impact on consumers’ utility and product demand, using a privacy-preserving econometric analysis. We find that an increase by 10% in the number of recommendations raises demand by about 6.7%. Our findings highlight the importance of “in-the-moment” marketing. In particular, trending recommendations that provide “in-the-moment” content have a much stronger effect compared to traditional recommendations. This effect is stable across various levels of popularity, whereas traditional recommendations contribute to the rich-get-richer phenomenon. We also examine various moderating effects; for instance, we find that recommendations based on just the novelty of the alternative do not have a significant effect but novel alternatives accrue greater benefits from recommendations when the quality of the item is also taken into consideration. We validate the robustness of our findings using instruments based on a deep learning model as well as different treatment-effects estimators.

Keywords: Recommender Systems; Mobile Recommendations; Deep Learning; Text-mining; Econometrics; Structural Modeling; Demand Estimation; Instrumental Variables

Introduction

Mobile devices have become a major platform for information as consumers spend an increasing amount of time with mobile devices (Nielsen 2014) and use them more often to search for products (Wells Fargo 2014). Mobile devices have also been driving the fast growth in e-commerce sales whereas, at the same time, the contribution of traditional shopping channels has been declining (eMarketer 2013). These trends are mainly attributed to smartphone users and are expected to continue for several years. It is projected that by 2019 the number of mobile shoppers will reach 213.7 million (from 125 million in 2013) with 87% of smartphone users shopping online using their mobile device (compared to 75% in 2013) (eMarketer 2015), while the number of search queries will almost double and the amount of sales will triple (UBS 2015).

At the same time, recommender system (RS) techniques can further increase the usability of mobile devices providing more focused content and effectively limiting the negative effects of information overload, which can be more acute due to various technical characteristics and idiosyncrasies of mobile devices (e.g., smaller screen size, distinct human-computer interaction, impact of the external environment, different behavioral characteristics of mobile users, etc.) (Ghose et al. 2012a; Ricci 2010). In addition, mobile RSes have the potential to be even more effective than desktop RSes as they can also take advantage of the higher involvement of consumers with mobile devices and the differences in consumption processes. Besides, what is probably even more important and revolutionary nowadays is that mobile devices also enable the use of rich contextual information and, consequently, the generation of novel types of recommendations that can further enhance the usability of mobile devices through better adapting the corresponding content to the

mailto:[email protected]

mailto:[email protected]



users’ needs in each context and further reducing the burden associated with information gathering. Hence, apart from the differences between mobile and desktop RSes, it is important to also measure and better understand any differences in effectiveness among the various recommendation types and algorithms. Similarly, it is also important to understand the differences in the effectiveness of recommendations across the various items and contextual settings and thus to examine the moderating effect of various item attributes and contextual factors.

However, despite the increasing prevalence and importance of mobile devices, smartphone applications and m-commerce, there has been relatively little academic research regarding the economic impact of RSes, especially in the context of mobile recommendations. This is mainly due to the inherent difficulty of measuring the economic impact of RSes, the limited availability of appropriate datasets, and the increasingly important privacy concerns that RSes and location-based services raise (Krumm 2009; Riboni et al. 2009). Therefore, even though the impact of recommendations on economic demand and especially their corresponding effects when using a mobile platform is an emerging and promising field of research, our understanding of how recommendations may affect the demand levels for individual products as well as the underlying consumption processes is rather limited yet.

In this study, we measure the effectiveness of mobile recommendations as the increase in demand for the recommended candidate items.1 In particular, we employ an advanced econometric technique in order to estimate the impact of mobile recommendations on consumers’ utility and real-world demand, using a privacy-preserving observational study with actual data corresponding to all the users of a popular real-world mobile application. To the best of our knowledge, this is the most extensive study conducted so far regarding the economic effectiveness of recommendations. The main contributions of this study are the following. First, we measure the effectiveness and economic impact of various types of real-world mobile recommendations in a privacy-preserving approach, based on a structural method following discrete-choice models of product demand that have a long history in econometrics (e.g., (McFadden 1980)). Second, we facilitate the estimation of causal effects in the presence of endogeneity (a common issue in RSes, targeted advertising, etc.) by extending the family of the popular BLP-style instruments of item “isolation” and differentiation (Berry et al. 1995) to the latent space through using deep learning techniques that, instead of commonly treating the individual words of user-generated texts as unique symbols without meaning, reflect semantic and syntactic similarities and differences among words and phrases. Third, we discover interesting and significant findings that extend the current theories regarding the impact of RSes, reconcile contradictory prior findings in the related literature, and have significant business implications. Our results highlight the importance of “in-the-moment” marketing and recommendations on the mobile world as well as the differences in effectiveness of various types of recommendations across various contexts. For instance, we find that the effect of traditional recommendations is stronger for alternatives that are more popular, whereas the effect of “in-the-moment” recommendations is more stable across various levels of popularity, while recommendations simply based on the novelty of each alternative are not effective, unless they also take into consideration the quality of the candidate item.

Literature Review and Research Questions Our work is related to several streams of research, including mobile and location-based recommender systems, recommendation explanations, and effects of recommendations on sales distribution. Due to space limitations, in the next paragraphs, we focus only on the most relevant issues and the corresponding works. For a rigorous review of the related work in RSes, including the effects on consumers’ perceptions and intentions (e.g., (Adomavicius et al. 2013; Greer and Murtaza 2003; Kumar and Benbasat 2006; Vijayasarathy and Jones 2001)), please see, for instance, (Adomavicius and Tuzhilin 2005; Li and Karahanna 2015; Xiao and Benbasat 2007; Yoo and Gretzel 2011).

Studying the effects of recommender systems on aggregate demand and markets, Fleder and Hosanagar (2009) show analytically that RSes can lead to a reduction in aggregate sales diversity, creating a rich-get-richer effect for popular products and preventing what may otherwise be better consumer-product matches. However, Brynjolfsson et al. (2011) provide empirical evidence that RSes are associated with an increase in niche products, reflecting lower search costs in addition to the increased product availability and corroborating the findings of Pathak et al. (2010) regarding the heterogenizing effects of RSes. Then,

1 In the recommender systems literature, there are multiple definitions of business value and effectiveness, such as whether a system helps users make “better”, faster, or more accurate decisions (e.g., (Adamopoulos and Tuzhilin 2014a; Tintarev and Masthoff 2012)).



Oestreicher-Singer and Sundararajan (2012a) measure how much the demand is influenced by the hyperlinked network of recommendations on Amazon.com and provide evidence that product sales became increasingly more homogenized. Finally, Hosanagar et al. (2013) study whether RSes are fragmenting the online population and find that users widen their interests, which in turn creates commonality with others. In this study, we focus on demand levels for individual products rather than effects on aggregate demand.

Focusing on demand levels for individual products, Oestreicher-Singer and Sundararajan (2012b) study how the explicit visibility of related-product networks can influence the demand for products in such networks and find that complementary products have significant influence on each other’s demand. They also find that newer and more popular products benefit more from the attention they garner from their network position in such related-product networks. In contrast, Chen et al. (2004) find that such network-based recommendations are more effective for less-popular books. Similarly, Pathak et al. (2010), examining a desktop recommender for item-to-item networks of books and focusing on 156 top-selling books on Amazon.com, find that the impact of the strength of recommendations on sales rank is moderated by the recency effect. Even though the majority of studies has focused on item-to-item networks of hyperlinked products, there are also empirical studies examining other types of recommendations. For instance, Jannach and Hegelich (2009), probably in the most closely related work to this study, present a case study evaluating the effectiveness of item recommendations for mobile apps (either paid or free apps) in different navigational situations. However, Jannach and Hegelich (2009) focus only on the most frequent users and only compare different RS algorithms with manually-curated recommendation lists, while consumers’ willingness to pay and economic demand is out of the scope of that paper. Focusing on related but different user behaviors, Swaminathan (2003), based on a controlled experiment, finds that category risk moderates the impact of RSes on decision quality and that product complexity moderates the role of recommendation agents on amount of search. Finally, conducting a lab experiment with cameras and music CDs, Aggarwal and Vaidyanathan (2005) conclude that RSes, and especially knowledge-based systems, are perceived as more effective for search goods than experience goods. In this study, we focus on the accurate measurement of the impact of real-world mobile recommendations on sales and the utility of consumers regarding a specific product category, based on actual data corresponding to all the users of a popular real-world mobile application. We particularly focus on the impact of trending mobile recommendations that provide “in-the-moment” content, relatively to traditional types of recommendations that are based on historical trends and data. In addition, even though we examine actual economic activity (purchases), our data set includes observations regarding both consumers’ goal-directed shopping behavior in a mobile setting and exploratory navigation behavior. More specifically, we examine:

RQ1: What is the relative economic effectiveness of mobile recommendations that provide “in-the-moment” content on real-world demand for individual items, compared to various other types of traditional recommendations that are based on historical trends and data?

A main contribution of this paper is that we also examine specific research questions regarding various moderating effects. In particular, our research questions are:

RQ2a: Whether the popularity of an alternative moderates the effectiveness of recommendations.

RQ2b: Whether the price of an alternative moderates the effectiveness of recommendations.

RQ2c: Whether the quality of an alternative moderates the effectiveness of recommendations.

RQ2d: Whether the novelty of an alternative moderates the effectiveness of recommendations.

RQ2e: Whether marketing promotions moderate the effectiveness of recommendations.

RQ2f: Whether the context moderates the effectiveness of recommendations.

Another different but still related stream of research examines the effects of various recommendation explanations, rather than types of recommendations or RS algorithms. Regarding the “effectiveness” of recommendation explanations, prior research has examined the effects of various types of recommendation explanations on different user behaviors, including trusting beliefs of users, and technology acceptance. For instance, Herlocker et al. (2000) show that including a graph of the distribution of ratings from similar users to explain a recommendation increases the acceptance of a collaborative filtering RSes for expert users, although this type of explanations was found to be too complex for ordinary users. Similarly, Bilgic and Mooney (2005) illustrate that content-based explanations or transaction-based explanations (i.e., explanations that present ratings previously provided by the user and caused an item to be recommended) are significantly more effective at enabling accurate assessments of items. Finally, in (Tintarev and Masthoff 2008; Tintarev and Masthoff 2012), personalized recommendations were not found



to be significantly more effective1 (i.e., difference between the estimated rating and after consumption user rating) than non-personalized content-based or standard (baseline) recommendations without explanations, while in several experiments personalized explanations hindered effectiveness.1 Nevertheless, the effects of recommendations explanations regarding different user behaviors are not always positively correlated (Tintarev and Masthoff 2011). For instance, it has been found that transparency does not necessarily aid trust (Cramer et al. 2008) and efficiency (i.e., time used to select an item from the recommendation list) does not affect user satisfaction (Gedikli et al. 2014). In this study, we focus on the accurate measurement of the impact of mobile recommendations on consumers’ utility and real-world demand, rather than other user behaviors (e.g., trust beliefs, technology acceptance). We should note again that the underlying definition of effectiveness of recommendations in this study is different from the definitions of effectiveness or (overestimate- and underestimate-oriented) persuasiveness in some of the prior studies (e.g., (Gedikli et al. 2014; Tintarev and Masthoff 2012)), as we measure the effectiveness in terms of demand effects for the candidate items.

Finally, the presented analysis examines a mobile recommender system in the application domain of city guides and restaurant recommendations. Other mobile RSes for restaurants include, among others, (Lee et al. 2006; Park et al. 2007; Sklar et al. 2012; Yap et al. 2007). Such RSes are also related to tourists guides (e.g., (Abowd et al. 1997; Cena et al. 2006; Höpken et al. 2010)), route recommenders (e.g., (Ge et al. 2010; Yuan et al. 2011; Zheng et al. 2010)), etc. 2 Focusing on mobile RSes, various studies have examined significant privacy issues (e.g., (Knijnenburg and Kobsa 2013; Scipioni 2011; Wilson et al. 2013; Xie et al. 2014)). For instance, Scipioni (2011) discusses privacy issues for location-based RSes focusing on generating recommendations without the need of collecting user information in a centralized way. In this study, we precisely estimate the effect of location-based recommendations on the demand for restaurants in a privacy-friendly manner employing structural econometric techniques.

Mobile Recommendations and Data

Our data set is from a mobile RS that identifies and recommends interesting events and places in real-time. In aggregate, our data set includes 11,684 venues and the corresponding visits of several million active users from December 2014 until March 2015; the maximum number of total visits to a single venue in our data set is 121,618 from 74,321 unique users. In particular, our data set includes all the restaurants in the mobile urban guide app for the 15 most popular cities (in terms of population) in the United States. Table 1 shows the specific cities that are included in our data set and the corresponding number of venues in each city.

Table 1: US Cities in Our Data Set Table 2: Definitions of Some of the Main Variables

City State Venues Variable Definition

Austin TX 687 Visitor share Market share of venue

Chicago IL 1,261 Price category Price category of venue

Columbus OH 365 Trending rec. Trending recommendation

Dallas TX 526 Traditional rec. Historical recommendation

Houston TX 727 Quality rec. Rec. based on reviews and ratings

Indianapolis IN 476 Event rec. Rec. based on upcoming events

Jacksonville FL 271 Expert rec. Recommendation based on expertise

Los Angeles CA 721 Novel rec. Recommendation based on novelty

New York NY 3,287 Weeks open Number of weeks since opening of venue

Philadelphia PA 467 Rating Overall reviewer rating

Phoenix AZ 448 Rating signals Number of visitor reviews

San Antonio TX 538 Alcohol Whether the venue serves alcohol

2 For a review of the literature in mobile recommender systems, please see (Liu et al. 2013; Ricci 2010).



San Diego CA 599 Outdoor seating Whether the venue offers outdoor seating

San Francisco CA 1,049 Holidays Number of holidays in given week

San Jose CA 276 Precipitation Precipitation in millimeters

The dependent variable (DV) of our analysis corresponds to the total number of visits to a particular venue. The independent variables (IVs) of interest include whether a venue is recommended as a “trending” venue (i.e., ‘trending recommendations’). This type of ‘trending recommendations’ takes into consideration the latest trends for the specific alternatives by explicitly discounting past historical trends and data while mainly leveraging available information that captures current trends such as relative differences in attributes (e.g., number of user lists, photos, etc.) during the last time periods, in order interesting alternatives to be recommended to the users and not only the most popular ones. This type of recommendations are generated using big data techniques because of the large number of observations and, from a business perspective, they capture trends related to “in-the-moment” marketing. In addition, such recommendations also have the potential to alleviate various common problems of traditional recommendations based on historical trends and data (e.g., cold start, concentration bias, rich-get-richer effect, etc.). The IVs also include traditional recommendations based on past historical trends and data (i.e., ‘traditional recommendations’). In particular, ‘traditional recommendations’ are based on whether a venue is recommended because of the total number of submitted positive user-generated reviews, positive ratings from the users, etc. (i.e., ‘quality recommendations’), as well as whether a business is recommended because it is endorsed through reviews by famous brands and experts (i.e., ‘expert recommendations’). They also include recommendations for novel venues, as alternatives that opened recently are also recommended to the users (i.e., ‘novel recommendations’). In addition, the mobile application also recommends to the users venues that have scheduled upcoming events for customers (i.e., ‘event recommendations’). Apart from the distinction of recommendations between ‘trending’ and ‘traditional’ recommendations, all the recommendations are seemingly similar as they are presented to the users in the same way, even though they are generated as described above. In addition, all the recommendations are generated before the realization of the demand for each alternative. Besides, all the recommendations lists are explicitly diversified by the RS algorithms (Ziegler et al. 2005) enhancing the exogeneity of recommendations (see Figure 2 for correlation and potential endogeneity), while the ‘trending’ recommendations also exhibit higher levels of temporal diversity (Lathia et al. 2010). For each of the types of recommendations and all the alternatives, our data set includes the relative number of times a venue was recommended to the users as well as statistics (e.g., average) of the ranking of the venue in the generated recommendation lists. Figure 1 shows a prototype of the recommendations screen for recommendations based on historical trends. Moreover, the independent variables of interest also include how many brands and experts have endorsed each venue and whether there are (and how many) scheduled upcoming events in this specific venue. This information is available for all businesses and not only the recommended venues.

Additionally, even though the employed estimation method does not require the researcher to observe all the relevant product characteristics,3 our data set includes a large number of contextual variables and item attributes. Besides, the opinions and preferences of users are also included in our data set as aggregate ratings; even though individual ratings are recorded, they were not made available for this study in order to preserve the privacy of the users. In particular, our data set includes the exact location of the venue, the number of user-generated reviews for the venue, the number of users who have liked this venue, the number of photos uploaded by users for this venue, the average numerical rating of the venue and the corresponding number of ratings submitted by users, the price tier of the venue (i.e., from 1 to 4 with 1 corresponding to least pricey), whether the venue is part of a chain, the categories that have been applied to this venue (e.g., American restaurant, Vegetarian) by users, whether a marketing promotion is taking place, whether the venue offers breakfast, brunch, lunch, dinner, alcohol, delivery, or take-outs, whether the venue takes reservations, whether it accepts credit cards, whether there is live music, DJ, TVs, Wi-Fi, outdoor seating, and parking availability, when the venue opened and was first introduced in the platform, as well as a description of the venue and the user-generated reviews.

We further supplement our data set with additional contextual variables. In particular, for each one of the venues in our data set we also include climate data (e.g., temperature and precipitation) from World-Clim

3 The demand equation is given an explicit structural interpretation as representing unobserved demand factors.



and Weather.com, geospatial data (e.g., elevation level) from the Consultative Group for International Agricultural Research (CGIAR) Consortium for Spatial Information (CSI), U.S. rental prices (e.g., median rental price at this location) from the U.S. Census Bureau, 2009-2013 5-Year American Community Survey, and Zillow Group, as well as travel expenditures per state from the U.S. Census Bureau. Besides, we also include calendar data with information about bank holidays, etc. Table 2 describes the variables of main interest and Figure 2 shows the corresponding correlation matrix.

Figure 1. Outline of recommendation screen. Figure 2. Correlation matrix of variables of main interest.

Finally, since the impact of the mobile recommendations on the generated demand for venues may be observed with lag (e.g., users visiting a recommended alternatives the days following the recommendation), we have aggregated the demand (i.e., number of visits) at a weekly level in order to more accurately estimate any effects. This also allows us to control for users visiting specific venues (e.g., Bar restaurants) after midnight as well as any unobserved word-of-mouth (WOM) effects as users might visit a venue after a recommendation to another user with whom they are connected in real-world.

One of the main advantages of the presented study is the use of actual data corresponding to all the available venues and all the users of a popular real-world application and not only to a specific sub-population (e.g., most frequent users, users opting in either online or offline surveys, etc.). In addition, our data includes observations regarding both consumers’ goal-directed behavior in a mobile setting and exploratory navigation behavior. Finally, another important advantage for methodological reasons is that we can easily quantify the quality of all alternatives and their marketing promotions.

Empirical Method and Models

In this section, we discuss the econometric structural model we apply to estimate the utility of consumers regarding the different alternatives and the corresponding effect of various types of recommendations. In a nutshell, each consumer selects the alternative that gives her/him the highest utility while the utility of consumers depends on the alternative characteristics, specific contextual factors, whether the particular alternative is recommended by the application, as well as individual taste parameters. The alternative (market) shares then are derived as the aggregate outcome of individual consumer decisions and the utility parameters can be inferred based on the functional form assumptions.

In particular, there are 𝑅 markets (i.e., cities) with 𝑁𝑟 alternatives (i.e., venues) in market 𝑟 . For each alternative 𝑗 in market 𝑟 and time period 𝑡 , observed characteristics are denoted by vector 𝑧𝑗𝑟𝑡 ∈ ℝ𝐾𝑧 ,



contextual factors by vector 𝑤𝑗𝑟𝑡 ∈ ℝ𝐾𝑤, and recommendation types by vector 𝜌𝑗𝑟𝑡 ∈ [0,1]𝐾𝜌; for simplicity

𝑧𝑗 , 𝑤𝑗 , and 𝜌𝑗 respectively. The elements of 𝑧𝑗 , 𝑤𝑗 , and 𝜌𝑗 combined include attributes 𝑥𝑗 (e.g., quality,

frequency of type of recommendation, temperature) that affect the demand levels 𝑞𝑗𝑟𝑡 (i.e., number of

visitors); for simplicity 𝑞𝑗. The unobserved characteristics (e.g., perception of status) of alternative 𝑗 are

denoted by 𝜉𝑗 . The utility of user 𝑖 for alternative 𝑗 depends on the characteristics of the alternative and the

user: 𝑈(𝑥𝑗 , 𝜉𝑗 , 𝑝𝑗 , 𝑣𝑖), where 𝑝𝑗 denotes price, and 𝑣𝑖 user-specific unobserved parameters. Then, the utility

of user 𝑖 for alternative 𝑗 is given by:

𝑢𝑖𝑗 = 𝛽𝑖𝑥𝑗 − 𝛼𝑝𝑗 + 𝜉𝑗 + 𝜖𝑖𝑗 , ( 1 )

where 𝛽𝑖 and 𝜖𝑖𝑗 are the user-specific parameters; we can also model parameter 𝛼 as 𝛼𝑖 . Decomposing

user 𝑖’s taste parameter for characteristic 𝑘 as:

𝛽𝑖𝑘 = 𝛽𝑘 + 𝜎𝑘𝜁𝑖𝑘 , ( 2 )

where 𝛽𝑘 is the mean level of taste parameter and the mean-zero 𝜁𝑖𝑘 has i.i.d. distribution (e.g., standard normal), and then combining Eqn. (1) with Eqn. (2), we can denote the mean utility level for alternative j as:4

𝛿𝑗 ≡ 𝛽𝑥𝑗 − 𝛼𝑝𝑗 + 𝜉𝑗 . ( 3 )

This transformation will allow us to estimate the model of individual behavior even when only aggregate data is observed. Following the standard assumption of consumer rationality for utility maximization (i.e., the consumer chooses the alternative that maximizes utility surplus) and assuming that 𝜖𝑖𝑗 follows an

extreme value distribution and 𝛽 = 𝛽 , the probability that a user 𝑖 chooses alternative 𝑗 is (McFadden 1980):

Pr(choice𝑗𝑖) =

𝑒𝛿𝑗

1 + ∑ 𝑒𝛿𝑘𝑘

, ( 4 )

∀ 𝑘 in the same market 𝑟 and 𝑘 ≠ 𝑗.

In addition to the competing venues 𝑗 = 1, … , 𝑁, we also model the existence of an outside option, 𝑗 = 0. This outside option corresponds to alternatives that might not be present in our data set or the option of a user not visiting any venue at all in time period 𝑡. Consumers may choose to select the outside option instead of the 𝑁 “inside” alternatives. The outside option is depicted in the denominator of Eqn. (4) with one, as the mean utility value of the outside option is normalized to zero.

The market share 𝑠𝑗 of each alternative is calculated as 𝑠𝑗 = 𝑞𝑗/𝑀 , where 𝑀 is the total market size. The

market size for each city (i.e., market) is set to the maximum number of unique active users that has been ever observed for each city. Alternatively, the market size could be assumed to be the population of each city or the number of households; the results remain qualitatively the same.

Inverting the market share equation, taking the logarithm in Eqn. (4), and combining with Eqn. (3), the market share of alternative 𝑗 is:

𝑙𝑛(𝑠𝑗) − 𝑙𝑛(𝑠0) = 𝛿𝑗 ≡ 𝛽𝑥𝑗 − 𝛼𝑝𝑗 + 𝜉𝑗 . ( 5 )

Additionally, if we assume that user tastes are correlated across alternatives and group the alternatives into 𝐺 exhaustive and mutually exclusive sets, 𝑔 = 1, … , 𝐺, the market share of alternative 𝑗 is (Cardell 1997):

𝑙𝑛(𝑠𝑗) − 𝑙𝑛(𝑠0) = 𝛿𝑗 ≡ 𝛽𝑥𝑗 − 𝛼𝑝𝑗 + 𝜎𝑙𝑛(�̅�𝑗/𝑔) + 𝜉𝑗 , ( 6 )

where �̅�𝑗/𝑔 is the market share of alternative 𝑗 as a fraction of the total group (nest) share and 𝑗 is in group

𝑔. As the parameter 𝜎 approaches one, the within group correlation of utility levels goes to one, and as 𝜎 approaches zero, the within group correlation goes to zero. In the empirical section of our study, we estimate the proposed model both with and without assuming that user tastes are correlated across alternatives.

4 The utility of alternative 𝑗 for user 𝑖 is 𝑢𝑖𝑗 = 𝛽𝑥𝑗 + 𝜉𝑗 − 𝛼𝑝𝑗 + 𝑣𝑖𝑗, with mean-zero heteroskedastic error 𝑣𝑖𝑗 = (∑ 𝑥𝑗𝑘𝜎𝑘𝜁𝑖𝑘𝑘 ) + 𝜖𝑖𝑗

capturing the effects of random taste parameters.



Then, relaxing the standard assumption that the parameters are constant across panels, we let the cross-sectional specific coefficient vector 𝛽𝑟 to be the outcome of a random process with mean vector β and covariance matrix 𝛴. This specification corresponds to a random coefficient model:

𝑙𝑛(𝑠𝑗) − 𝑙𝑛(𝑠0) = 𝛿𝑗 ≡ 𝛽𝑟𝑥𝑗 − 𝛼𝑟𝑝𝑗 + 𝜎𝑟𝑙𝑛(�̅�𝑗/𝑔) + 𝜉𝑗 , ( 7 )

or, equivalently,

𝑙𝑛(𝑠𝑗𝑟𝑡) − 𝑙𝑛(𝑠0𝑟𝑡) = 𝛿𝑗𝑟𝑡 ≡ 𝛽𝑟𝑧𝑧𝑗𝑟𝑡 + 𝛽𝑟

𝑤𝑤𝑗𝑟𝑡 + 𝛽𝑟𝜌

𝜌𝑗𝑟𝑡 − 𝛼𝑟𝑝𝑗𝑟𝑡 + 𝜎𝑟𝑙𝑛(�̅�𝑗𝑟𝑡/𝑔) + 𝜉𝑗𝑟𝑡 . ( 8 )

In other words, using demand-estimation approaches from economics, we estimate the weights that consumers (implicitly) assign to each individual venue characteristics, recommendations and contextual factors as well as the sensitivity of consumers to changes in these factors and characteristics. This is done by inverting the function defining market shares to uncover the utility levels of the alternatives and relating these utility levels to venue characteristics, recommendations and contextual factors. Then, based on these estimates, we derive the utility gain that each type of recommendations generates. The employed methodology estimates any causal effects in a privacy-preserving manner, as it does not require individual consumer data but only aggregate data and statistics, even though it is a model of individual behavior.

We can treat Eqn. (8) as an estimation equation and use econometric techniques, including standard instrumental variables techniques, to estimate the unknown parameters. In our empirical study, we also use traditional instruments, such as rental prices and variables that are used in the algorithms to generate the recommendations but do not affect the utility of the alternatives for consumers in the current time period given the observed confounders (e.g., lagged variables corresponding to number of lists, photos, etc. in previous time periods) (Sweeting 2013), as well as a novel instrument derived from a metric of alternative differentiation and isolation based on a deep-learning model of the user-generated reviews. The motivation for the latter instrument is twofold: first, it is similar to the BLP-style instruments (Berry 1994; Berry et al. 1995), which measure the isolation in product space as products (or alternatives) that are more isolated are related to higher margins; second, it is related to whether an alternative is recommended by the mobile application as alternatives that are more isolated have higher likelihood of being included in a recommendation list because of the diversification process of the employed recommendation algorithms. In order to implement this instrument, we use the efficient and state-of-the-art method of (Le and Mikolov 2014; Mikolov et al. 2013b). Instead of treating the individual words of user-generated texts as unique symbols (tokens) without meaning, as in common bag-of-words and bag-of-n-grams models, the employed method reflects semantic and syntactic similarities and differences among words and phrases by representing them as dense vectors, usually referred to as “neural embeddings”. The common paradigm for deriving such representations is based on the distributional hypothesis of Harris (1954) that words in similar contexts have similar meanings. In essence, the continuous space representations are learned in an unsupervised fashion by trying to maximize the dot-product between the vectors of frequently occurring word-context pairs and minimize it for random word-context pairs (Levy and Goldberg 2014). The neural network based word vectors are usually trained using stochastic gradient descent, where the gradient is obtained via back-propagation (Mikolov et al. 2013a; Rumelhart et al. 1988). After the training converges, words with similar meaning are mapped to a similar position in the latent space. We use a pre-trained open-source model that contains 300-dimensional vectors for 3 million words and phrases trained on part of Google News dataset (about 100 billion words) and described in (Mikolov et al. 2013b), in order to estimate the distance of the various alternatives in the latent space. We should note that this method is general and applicable to texts of any length (e.g., sentences, paragraphs, documents) and does not require task-specific tuning nor does it rely on the parse trees (Le and Mikolov 2014).

Apart from benefits discussed in the previous section (e.g., privacy, real data and users, popular real-world application, quantifiable quality, etc.), this method allows also for unobserved product characteristics, including also determinants that are difficult to measure (e.g., consumers’ perceptions about status). In particular, we can consistently obtain the causal effects of interest even if other covariates are unobserved or endogenous and we have no outside instruments for them. In addition, using discrete choice model with random coefficients we allow for heterogeneity in the parameters. Besides, the resulting model can make predictions not only for the existing alternatives (i.e., venues) under different conditions and contextual factors but also for new alternatives that might not be included in our data set. Finally, our approach also allows for an “outside option” as a consumer might not select any alternative.



Outline of Main Results

In order to discover the causal impact of different types of mobile recommendations, we estimate different specifications of the structural econometric model we presented in the previous section. First, we estimate different specifications based on the logit model of demand presented in Eqn. (5) and the nested model in Eqn. (6). Then, we allow for cross-sectional heterogeneity and estimate a random-coefficients model as in Eqn. (7-8). Due to space limitations, we only present the results corresponding to the latter (nested logit) specifications (see Tables 3 and 4); the results are qualitatively the same across the different specifications. Apart from estimating the main variables of interest, we also examine a number of interaction effects (see Tables 5 - 10). In order to test the robustness of our findings and control for endogeneity, we also employ different treatment-effects estimators (see Table 11) and instrumental variable techniques (see Table 12). Even though we are mainly interested in identifying the relative differences in effectiveness of the various types of recommendations, in order to control for endogeneity, apart from the aforementioned techniques and alternative estimators, we also use city-specific and category-specific effects and time trends. Finally, we also conduct various falsification tests in order to verify the validity of our findings (see Tables 13-14).

Table 3 shows the estimated coefficient and the corresponding impact of the various types of mobile recommendations, based on the nested logit model with multi-level fixed effects and a time trend in order to control for correlation of user tastes across alternatives as well as different potentially unobserved effects. Models 1 and 2 identify the average effect of recommendations, in general. Only the effects of the variables of interest and the effects that were found to be statistically significant across all the econometric specifications are reported; all the models were estimated including the extensive set of variables (see Table 2 and Fig. 2). In Model 1 the variable of interest is binary and Model 2 provides a more precise measurement using the relative frequency of recommendation of the specific venue. Model 3 separates the effect of two major types of recommendations (i.e., ‘trending’ alternatives and ‘traditional’ alternatives) in which we are interested (RQ1). Then, Model 4 also controls for the ranking of each alternative in the generated recommendation lists; this ranking is available for all the alternatives in our data set. Finally, Model 5 further separates the effect of ‘traditional’ recommendations into distinct types of recommendations and recommendation explanations (i.e., ‘quality recommendations’, ‘event recommendations’, ‘expert recommendations’, and ‘novel recommendations’).

Table 3: Coefficient Estimates of Nested Logit Model with 2 level F.E. and Time-trend

Model 1 Model 2 Model 3 Model 4 Model 5

Recommendation 0.834 ***

Recommendation frequency 2.078 ***

Trending recommendation 1.962 *** 0.376 * 1.967 ***

Traditional recommendation 0.883 *** 0.167 **

Quality recommendation 1.046 ***

Event recommendation 1.600

Expert recommendation 1.750 ***

Novel recommendation 0.011

Ranking -0.006 ***

Price category -0.004 -0.000 -0.001 -0.005 -0.002

Rating 0.313 *** 0.324 *** 0.327 *** 0.312 *** 0.323 ***

Rating signals 0.002 *** 0.002 *** 0.002 *** 0.002 *** 0.002 ***

Weeks open -0.001 *** -0.001 *** -0.001 *** -0.001 *** -0.001 ***

Breakfast 0.071 ** 0.077 ** 0.077 ** 0.080 ** 0.080 **

Alcohol 0.191 *** 0.196 *** 0.199 *** 0.193 *** 0.196 ***



Delivery -0.048 * -0.048 * -0.047 * -0.047 * -0.047 *

Outdoor seating 0.040 * 0.046 * 0.048 ** 0.043 * 0.050 **

Wi-Fi 0.058 ** 0.073 *** 0.077 *** 0.073 *** 0.081 ***

With-in group share 0.249 *** 0.264 *** 0.264 *** 0.262 *** 0.265 ***

Holidays 0.094 *** 0.091 *** 0.092 *** 0.096 *** 0.094 ***

Precipitation -0.004 *** -0.004 *** -0.004 *** -0.003 *** -0.004 ***

Time trend -0.043 *** -0.042 *** -0.041 *** -0.041 *** -0.041 ***

Adjusted R-squared 0.343 0.338 0.339 0.341 0.339

* p<0.05, ** p<0.01, *** p<0.001

As Models 1 – 5 show, recommendations have a positive impact on demand. This impact is significant at level p < 0.001 in most of the cases. In particular, computing alternative-level derivatives of the demand function (elasticities), an increase by 1% in the number of times a venue is recommended raises the demand by 0.6694% for already recommended alternatives and by 0.0626% on average for all alternatives in general (RQ1). Hence, there are positive effects on both individual demand and aggregate demand in the market. These effects are both statistically and economically significant; comparing these numbers with typical click-through-rates, the mobile recommendation effect is orders of magnitude larger. Moreover, trending recommendations that provide “in-the-moment” content to the users have a much stronger effect, in the particular mobile setting, compared to traditional recommendations based on historical trends and data. This significant difference in effects persists also after controlling for both the overall ranking in the recommendation list and the with-in ranking in the specific types of recommendations. Further inspecting these differences, we decompose the traditional recommendations into distinct types. In details, recommendations regarding high-quality venues based on the number of positive ratings and reviews, recommendations of venues based on upcoming events, recommendations based on reviews of brands and experts, and recommendations of novel venues based on whether an alternative opened very recently. Based on the results, we can see that the “in-the-moment” recommendations have the strongest effect on demand. Additionally, recommendations based on the quality of venues and experts’ reviews outperform other types of recommendations. On the contrary, recommendations based on the number of upcoming events or simply the novelty of the alternative were not found to have a significant effect. This result regarding the novelty of recommendations is in accordance with the findings of Adamopoulos and Tuzhilin (2011); Adamopoulos and Tuzhilin (2014c) that novelty and unexpectedness should be considered vis-à-vis the overall quality and utility of each alternative when generating recommendations, rather than recommending the most novel or unexpected items.

Furthermore, we then also allow for parameter heterogeneity in the effect of the various types of recommendations. In particular, we assume that the panel-specific vector 𝛽𝑟 to be the outcome of a random process with mean vector 𝛽 and covariance matrix 𝛴 as in (Greene 2003; Swamy 1970). Table 4 reports the results based on the random coefficient nested model for the subset of panels where significant variation is observed. The presented results extend the results presented in Table 3 and control for venue attributes, climate attributes, geospatial attributes, and calendar attributes as before. These findings corroborate our previous results. Additionally, based on the panel-specific coefficient estimates, it was found that the effect of “in-the-moment” recommendations is consistently greater than the effects of other types of recommendations for all the panels in this estimation sample. Regarding the other types of mobile recommendations, in some cases it was found that the impact of quality recommendations is higher than event recommendations, while in other panels the opposite result was observed.

Table 4: Coefficient Estimates of Random Coefficients Nested Logit Model

Model 1 Model 2 Model 3 Model 4 Model 5

Recommendation 0.980 ***

Recommendation frequency 1.878 ***



Trending recommendation 4.153 *** 2.083 *** 4.041 **

Traditional recommendation 0.162 *** 0.785 *

Quality recommendation -0.182

Event recommendation 0.918

Expert recommendation 1.931 ***

Novel recommendation 0.280

Ranking -0.009 *

Moderating (interaction) effects on effectiveness of recommendations

Moreover, we delve further into the differences in effectiveness of the various types of mobile recommendations and we examine the moderating effect of various item attributes and contextual factors. Tables 5 - 10 present the moderating effects of the various attributes and contextual factors (RQ2). In particular, Table 5 examines the moderating interaction of price and recommendations (RQ2b); Table 6 investigates the interaction effect of marketing promotions and recommendations (RQ2e); Table 7 presents the interaction effect with the quality of alternative (RQ2c); Table 8 with the number of user-submitted ratings (RQ2a); Table 9 presents the interaction effect between recommendations and novelty of an alternative (RQ2d); and Table 10 the interaction effect with public holidays (RQ2f). All the presented results extend the results presented in Table 3 and control for venue attributes, climate attributes, geospatial attributes, and calendar attributes as before, as well as for multi-level fixed effects. All the controls and effects are included in all Tables 5 – 10, even though they are not depicted in the results, due to space restrictions. Base levels are also estimated as usual, even though they are not reported as well.

Based on the results presented in Tables 5 and 6, we do not find significant moderating effects of price category and promotions on the effectiveness of recommendations. Further decomposing the ‘traditional’ recommendations into different types, as before, we do not find any statistically significant differences among the various types of recommendations. The effect of marketing promotions in the presence of recommendations is a topic of interest that should be thoroughly examined in future research.

Table 5: Interaction Effects - Price

Model 1 Model 2 Model 3

Recommendation=1 × Price category -0.078

Recommendation frequency × Price category -0.080

Trending recommendation × Price category 0.162

Traditional recommendation × Price category -0.212

Table 6: Interaction Effects - Promotion


Recommendation=1 × Promotions -0.077

Recommendation frequency × Promotions -0.512

Trending recommendation × Promotions 0.644

Traditional recommendation × Promotions -0.285

Based on the results presented in Table 7, the rating of an alternative has a negative and significant moderating effect, illustrating that high-quality items are benefiting less from recommendations compared to other alternatives (RQ2c). In other words, in the specific mobile setting, the recommendations seem to



be perceived by the users as providing information similar to this of numerical ratings, indicating that endorsements and recommendations can potentially substitute for a lower rating in this setting. This moderating effect is stronger for trending recommendations, event recommendations, and recommendations of novel alternatives.

Table 7: Interaction Effects - Rating


Recommendation=1 × Rating -0.128 *

Recommendation frequency × Rating -1.109 ***

Trending recommendation × Rating -3.004 ***

Traditional recommendation × Rating -0.349 *

We further investigate this finding examining the moderating effect of the number of ratings on the effectiveness of the various types of recommendations. Table 8 illustrates that alternatives with fewer rating signals are benefiting more from recommendations (RQ2a). This result is consistent with search cost arguments in the information systems literature (e.g., (Brynjolfsson et al. 2011)). In addition, it further supports the observation that recommendations might potentially substitute for numerical ratings, as they seem to be perceived by the users as providing similar information.

We also investigate the effect of the different types of recommendations across different levels of venue popularity using the number of visits as metric of popularity and employing the technique of quantile regression. Figure 3 graphically illustrates the differences between the main types of recommendations in the mobile application. Recommendations have a stronger positive effect on average for more popular alternatives. This empirical finding is consistent with the observation of Oestreicher-Singer and Sundararajan (2012b) that in recommendation networks in electronic markets more popular products use more efficiently the attention they garner from their network position. An interesting and unexpected observation though is that the effect of traditional recommendations is much stronger for more popular alternatives whereas the effect of “in-the-moment” recommendations is more stable across alternatives. This finding can alleviate the concentration bias and popularity reinforcement effects of recommender systems (Adamopoulos 2013; Adamopoulos and Tuzhilin 2014b), which have already been observed in various settings (e.g., (Fleder and Hosanagar 2009; Oestreicher-Singer and Sundararajan 2012a)). Besides, based on the detailed results (not presented here due to space limitations), we can also see that the strong and increasing moderating effect of popularity on the effect of recommendations on demand is mainly through recommendations based on quality and experts’ reviews whereas this is not observed for recommendations based on upcoming events and the novelty of alternatives which are characterized of much higher variance; it is worth noting that the effect of novel recommendations is stronger for less popular alternatives.

Table 8: Interaction Effects – Rating Signals


Recommendation=1 × Rating Signals -0.001 ***

Recommendation frequency × Rating Signals -0.003 ***

Trending recommendation × Rating Signals -0.004 ***

Traditional recommendation × Rating Signals -0.001 ***



Figure 3. Recommendations effects across popularity levels.

Additionally, based on the results presented in Table 9, novel alternatives gain in general greater benefits from recommendations (RQ2d). In combination with the findings presented in Tables 3 and 4, this result illustrates that novel alternatives accrue greater benefits from recommendations when those recommendations are not solely based on the characteristic of novelty but also take into consideration attributes such as the quality of the item. Hence, this finding reconciles different contradictory findings in prior literature in recommender systems. For instance, (Ekstrand et al. 2014; Matt et al. 2014) maintain that novelty has a significant negative effect on consumers’ satisfaction and perceived enjoyment whereas Vargas and Castells (2011) argue that novelty is a key quality of recommendations in real scenarios.

Table 9: Interaction Effects - Novelty


Recommendation=1 × Weeks open -0.002 ***

Recommendation frequency × Weeks open -0.007 ***

Trending recommendation × Weeks open -0.004 ***

Traditional recommendation × Weeks open -0.004 ***

Finally, Table 10 shows that the effect of recommendations is stronger during holidays. Based on the detailed results, this moderating effect is stronger for ‘trending’, ‘quality’ and ‘expert’ recommendations, highlighting the differences in user behavior across various contexts (RQ2f). This finding contributes to the emerging literature on mobile marketing and contextual attributes (e.g., effect of crowdness on mobile offers (Andrews et al. 2014), geographic mobility and responsiveness to mobile ads (Ghose and Han 2011)).

Table 10: Interaction Effects - Holidays


Recommendation=1 × Holidays 0.345 ***

Recommendation frequency × Holidays 2.499 ***

Trending recommendation × Holidays 2.846 **

Traditional recommendation × Holidays 1.284 ***



Robustness Checks

Moreover, Table 11 shows the estimates of the Average Treatment Effect (ATE) via regression adjustment (RA) and inverse-probability-weighted regression adjustment (IPWRA) for a random time period and considering any venue with non-zero number of recommendations as “treated” (Joffe et al. 2004; Rubin 2006; Rubin and Thomas 2000; Tsiatis 2007). The RA estimator first fits separate models for the treated and control groups and then computes the average of the predicted outcome for each subject and treatment group based on which the ATE is estimated (Wooldridge 2002). The IPWRA estimator first fits a model to predict treatment status and based on this computes the inverse-probability weights (Robins 1999), which are then used to fit weighted regression models for each treatment group (Wooldridge 2002; Wooldridge 2007). Assuming that the treatment is (conditional mean) independent of the potential outcome after conditioning on all the covariates, we derive consistent estimates of the effects. In particular, the IPWRA estimator has the “double-robust” property; only one of the two models must be correctly specified for the estimator to be consistent (Tsiatis 2007; Wooldridge 2002). Both models include controls for venue attributes, climate attributes, geospatial attributes, and calendar attributes, even though not depicted in the following table due to space restrictions. Based on the results, despite the binary formulation of the variables of interest under these alternative specifications, both methods corroborate our finding that different types of recommendation can result in different effects and that “in-the-moment” recommendations outperform traditional types of recommendations; the differences between the various effects are statistically significant as well.

Table 11: Treatment-effects estimation

Regression Adjustment Inverse-Probability-Weighted

Regression Adjustment

ATE Trending recommendation 1.040 *** 0.917 ***

ATE Traditional recommendation 0.779 *** 0.840 ***

Finally, even though we are interested in the relative differences of effectiveness across the various types of mobile recommendations and we leverage the exogenous variation in the recommendation mechanisms while we also control for numerous confounders, including quality and marketing promotions, we also assume as robustness check that different variables are endogenous. Table 12 presents the effects of the different types of recommendations after accounting for potential endogeneity in prices, with-in group share, and recommendations. Model 1 uses as instruments rental prices and Hausman instruments (e.g., the average price of other venues in the same market and same rating category), as well as the novel metric of alternative differentiation and isolation based on the employed deep-learning model of the user-generated reviews (see section ‘Empirical Method and Model’). Then, Model 2, in addition to the metric based on the deep learning model, uses the number of other venues in the same category, as well as lagged values of the variables of interests and the attributes used in generating the recommendations (e.g., number of lists, photos, etc.). We have tested the instruments’ validity using the Hansem-Sargan test (Hansen 1982) and the Stock-Yogo critical values (Stock and Yogo 2005). In addition, both models include controls for venue, climate, geospatial, and calendar attributes, even though not depicted in the following table due to space restrictions. These results further corroborate our previous findings highlighting the importance of “in-the-moment” recommendations; the differences between the main recommendation effects are statistically significant.

Table 12: Coefficient Estimates of Nested Logit Model with 2 level F.E. Time-trend and IVs

Model 1 Model 2

Trending recommendation 1.229 * 1.802 **

Quality recommendation 0.609 * 1.536 ***

Event recommendation 0.780 1.202

Expert recommendation 0.839 1.417 ***

Novel recommendation -2.872 -1.733



Falsification Tests

One might think that it is plausible that the previous set of models are simply picking up spurious effects as a result of pure coincidence, a general increase in the corresponding metrics, or other unobserved factors. To assess the possibility that the aforementioned findings are a statistical artifact and the identified positive significant effects were captured by chance or because of other confounding factors, we run a falsification test (“placebo” study) using the same models as above (in order to maintain consistency) but randomly indicating which alternatives were recommended and when, respectively. The results of the falsification tests are shown in Tables 13 and 14. We see that, under these checks, the corresponding effects are not statistically significant, indicating that our previous findings are not a statistical artifact of our specifications, but we indeed discovered the actual effects. It is interesting to notice that the RA and IPWRA models resulted in the same estimates because of the random “treatment” and thanks to the “double-robust” property of the IPWRA estimator.

Table 13: Treatment-effects estimation (Pseudo Recommendations)

Regression Adjustment Inverse-Probability-Weighted

Regression Adjustment

ATE Trending recommendation 0.005 0.005

ATE Traditional recommendation 0.023 0.023

Table 14: Coefficient estimates (Pseudo Recommendations)

Model 1 Model 2

Trending recommendation 0.042 0.047

Quality recommendation 0.007 0.003

Event recommendation 0.071 0.087

Expert recommendation -0.021 -0.022

Novel recommendation 0.008 0.002

Discussion, Theoretical Integration, and Managerial Implications

Apart from quantifying the impact of various types of mobile recommendations and identifying various moderating effects on this impact, we theoretically integrate our research questions and the corresponding findings into the current literature on RSes by extending a conceptual model of the effects of RS use, RS characteristics, and other factors on consumer decision making that was first articulated by Xiao and Benbasat (2007). Figure 4 depicts the updated conceptual model incorporating our research questions and findings regarding the decision outcome of consumers; RQ1 is related to RS principle of function, RQ2a-d to the recommended product, and RQ2f to the particular context of use.

The discussed impact of RSes on consumers’ decision-making and choices can be supported through the lenses of various IS theories. A possible utilitarian explanation of the aforementioned finding that recommendations have a positive and significant effect on the demand levels for the recommended products is that consumers might perceive the recommendations as endorsements of the candidate items from the RS or as another reputation dimension (in addition to the item rating and consumer reviews). Apart from arguments based on utility theory, we can also find support for our findings in the theories of human information processing. For instance, Shafir et al. (1993b) maintain that people evaluate alternatives by comparing them separately on distinct dimensions and that relationships among alternatives may be perceived to be more compelling reasons or arguments for choice than deriving overall values for each alternative and choosing the alternative with the best value. Because such differences (e.g., whether an alternative is recommended, ranking in recommendation list) can be perceived with little effort, relationships among alternatives may be used to make choices even in situations with simple alternatives and even if they do not provide good justifications (Bettman et al. 1998). Another possible explanation is that the recommended alternatives maximize the ease of justifying consumers’ decisions. This explanation



Figure 4. Conceptual model of effects of recommender systems.

is becoming even more important in the case of collective decisions (e.g., restaurant selection) and group recommenders; Shafir et al. (1993a) have demonstrated that decision makers often construct reasons in order to justify a decision to themselves (i.e., increase their confidence in the decision) and/or justify (explain) their decision to others. Alternatively, another possible explanation is that the recommended alternatives are simply becoming more salient and hence are more frequently selected by the consumers either deliberately (e.g., lower search costs (Wilde 1980)) or inadvertently since they capture their attention. Finally, the significant difference in effectiveness among the various types of recommendations and, especially, the effect of “in-the-moment” recommendations can be explained based on the IS success model (DeLone and McLean 1992) and the component of timeliness of information which affects user satisfaction through the construct of information quality. Similarly, the differences of events recommendations and recommendations from “experts” from the other types of examined recommendations can be explained based on the relevance of information.

The findings of this study, apart from a theoretical contribution, also have important managerial implications. Based on our results we find that mobile recommendations have a positive effect on both individual demand levels for the recommended alternatives and aggregate demand across all products. Disentangling the effects of various types of recommendations, we further find that recommendations that provide “in-the-moment” content to the users have a much stronger effect compared to traditional recommendations based on historical trends and data. The importance and economic significance of these findings is further amplified by the prediction that in the next years the number of mobile shoppers will reach 213.7 million with 87% of smartphone users shopping online using their mobile device. Apart from highlighting the economic impact of various types of recommendations and their effects on consumers’ decision-making, we also illustrate how managers and practitioners can leverage observational data to evaluate different recommender system algorithms and estimate their effects using a privacy-preserving approach. Another actionable finding of significant importance for businesses and managers is that novel alternatives accrue greater benefits from recommendations when those recommendations are not solely based on the characteristic of novelty but also take into consideration the quality of the item. Nevertheless, the remaining moderating effects we examined in this study are also important for businesses and recommender system practitioners as they can guide the development of future RS algorithms and highlight the importance of various product design decisions. Finally, another important finding with significant managerial implications is that the effect of traditional recommendations is much stronger for more popular products, contributing to a rich-get-richer effect, whereas the effect of “in-the-moment” recommendations is more stable across alternatives, allowing businesses to effectively leverage the benefits of long-tail products.



Future research, apart from estimating the causal impact of RSes across various settings, can also apply the proposed techniques in order to generate contextual mobile recommendations lists based on consumers’ surplus, rather than the predicted numerical ratings for the various alternatives or the probability of acceptance of each recommendation, as proposed by Ghose et al. (2012b) in the context of web-based travel search engines. Future research should also further examine the economic effectiveness of mobile recommendations in different application domains investigating a wider range of recommended items. Besides, future research should also compare the effectiveness of mobile recommendations vis-à-vis traditional recommendations in other channels (e.g., web). In addition, the effect of marketing promotions in the presence of recommendations is another topic of significant academic and managerial interest that should be thoroughly examined in future research. Finally, future research should also disentangle the effect of recommendations on repeated visits of existing customers from the corresponding effects for new users and alternatives.

One of the limitations of this study is that we focus only on the effect of mobile recommendations on the demand for the candidate items (e.g., restaurants). Nevertheless, the explicit user satisfaction should also be considered and precisely measured. However, explicit ratings from individual users were not available for privacy reasons. In addition, the empirical analysis is conducted at a weekly aggregate level rather than a more granular unit of analysis. Despite these limitations, our contribution may be widely relevant to managers while also seeding a number of new directions in future research. Our hope is that these limitations are viewed not as a liability but as a path toward future research that extends our research question while strengthening the relevant theory and empirical evidence.

Conclusions

In this study, we measure the economic effectiveness of recommendations as the increase in demand for the recommended candidate items. In particular, we employ advanced econometric techniques in order to estimate the impact of mobile recommendations on consumers’ utility and real-world demand, using a privacy-preserving observational study based on a model of individual behavior where only aggregate data is observed. Based on our analysis, we find that on average an increase by 10% in the number of times a venue is recommended raises the demand by up to 6.7%. Examining the impact of different types of recommendations, our findings highlight the importance of “in-the-moment” marketing in the mobile world. In particular, trending recommendations that provide “in-the-moment” content to the users have a much stronger effect in the mobile channel, compared to traditional recommendations based on historical trends and data. This effect of “in-the-moment” recommendations is stable across various levels of popularity, whereas the effect of traditional recommendations is stronger for alternatives that are more popular. Furthermore, recommendations based on the quality of venues or experts’ reviews outperform other types of recommendations. On the contrary, recommendations based on the number of upcoming events or simply the novelty of the alternative were not found to have a significant effect. However, compared to other alternatives, novel alternatives accrue greater benefits from recommendations when these recommendations are not solely based on the characteristic on novelty but also take into consideration other attributes such as the quality of the item. We validate the robustness of our findings using multiple econometric specifications as well as instrumental variable methods with instruments based on a deep learning model. The employed deep learning techniques and the corresponding instruments, instead of commonly treating the individual words of user-generated texts as unique symbols without meaning, reflect semantic and syntactic similarities and differences among words and phrases.

In particular, our main contributions lie in i) the accurate measurement of the effectiveness and economic impact of various types of real-world mobile recommendations in a privacy-preserving approach based on a structural econometric method following discrete-choice models of product demand, ii) the introduction of a new econometric instrument that extends a popular family of instruments to the latent space and facilitates the usage of instrumental variable techniques for causal inference in the presence of endogeneity in the field of RSes, as well as iii) the discovery of interesting and significant findings that extend the current theories regarding the impact of recommender systems and reconcile contradictory prior findings in the related literature. To our knowledge, this study is the most extensive study conducted regarding the economic effectiveness of mobile recommendations. Our findings also have important managerial and business implications as the number of mobile shoppers is projected to reach 213.7 million during the next years with 87% of smartphone users shopping online using their mobile device.



References

Abowd, G.D., Atkeson, C.G., Hong, J., Long, S., Kooper, R., and Pinkerton, M. 1997. "Cyberguide: A Mobile Context-Aware Tour Guide," Wireless networks (3:5), pp. 421-433.

Adamopoulos, P. 2013. "Beyond Rating Prediction Accuracy: On New Perspectives in Recommender Systems," Proceedings of the seventh ACM conference on Recommender Systems (RecSys'13), Hong Kong, China: ACM.

Adamopoulos, P., and Tuzhilin, A. 2011. "On Unexpectedness in Recommender Systems: Or How to Expect the Unexpected," Workshop on Novelty and Diversity in Recommender Systems (DiveRS 2011), at the 5th ACM International Conference on Recommender Systems (RecSys'11), P. Castells, J. Wang, R. Lara and D. Zhang (eds.), Chicago, Illinois, USA: ACM, pp. 11-18.

Adamopoulos, P., and Tuzhilin, A. 2014a. "Estimating the Value of Multi-Dimensional Data Sets in Context-Based Recommender Systems," Proceedings of the 8th ACM Conference on Recommender systems: ACM.

Adamopoulos, P., and Tuzhilin, A. 2014b. "On over-Specialization and Concentration Bias of Recommendations: Probabilistic Neighborhood Selection in Collaborative Filtering Systems," Proceedings of the 8th ACM Conference on Recommender systems: ACM, pp. 153-160.

Adamopoulos, P., and Tuzhilin, A. 2014c. "On Unexpectedness in Recommender Systems: Or How to Better Expect the Unexpected," ACM Transactions on Intelligent Systems and Technology (TIST) (5:4), p. 54.

Adomavicius, G., Bockstedt, J.C., Curley, S.P., and Zhang, J. 2013. "Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects," Information Systems Research (24:4), pp. 956-975.

Adomavicius, G., and Tuzhilin, A. 2005. "Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions," Ieee Transactions on Knowledge and Data Engineering (17:6), Jun, pp. 734-749.

Aggarwal, P., and Vaidyanathan, R. 2005. "Perceived Effectiveness of Recommendation Agent Routines: Search Vs. Experience Goods," International Journal of Internet Marketing and Advertising (2:1), pp. 38-55.

Al-Natour, S., Benbasat, I., and Cenfetelli, R.T. 2006. "The Role of Design Characteristics in Shaping Perceptions of Similarity: The Case of Online Shopping Assistants," Journal of the Association for Information Systems (7:12), p. 34.

Andrews, M., Luo, X., Fang, Z., and Ghose, A. 2014. "Mobile Ad Effectiveness: Hyper-Contextual Targeting with Crowdedness,").

Berry, S. 1994. "Estimating Discrete-Choice Models of Product Differentiation," The RAND Journal of Economics (25:2), Sum, pp. pp. 242-262.

Berry, S., Levinsohn, J., and Pakes, A. 1995. "Automobile Prices in Market Equilibrium," Econometrica (63:4), Jul, pp. 841-890.

Bettman, J.R., Luce, M.F., and Payne, J.W. 1998. "Constructive Consumer Choice Processes," Journal of consumer research (25:3), pp. 187-217.

Bilgic, M., and Mooney, R.J. 2005. "Explaining Recommendations: Satisfaction Vs. Promotion," Beyond Personalization Workshop, IUI.

Brynjolfsson, E., Hu, Y.J., and Simester, D. 2011. "Goodbye Pareto Principle, Hello Long Tail: The Effect of Search Costs on the Concentration of Product Sales," Management Science (57:8), Aug, pp. 1373-1386.

Cardell, N.S. 1997. "Variance Components Structures for the Extreme-Value and Logistic Distributions with Application to Models of Heterogeneity," Econometric Theory (13:2), pp. 185-213.

Cena, F., Console, L., Gena, C., Goy, A., Levi, G., Modeo, S., and Torre, I. 2006. "Integrating Heterogeneous Adaptation Techniques to Build a Flexible and Usable Mobile Tourist Guide," AI Communications (19:4), pp. 369-384.

Chen, P.-Y., Wu, S.-y., and Yoon, J. 2004. "The Impact of Online Recommendations and Consumer Feedback on Sales," ICIS 2004 Proceedings), p. 58.

Cramer, H., Evers, V., Ramlal, S., Van Someren, M., Rutledge, L., Stash, N., Aroyo, L., and Wielinga, B. 2008. "The Effects of Transparency on Trust in and Acceptance of a Content-Based Art Recommender," User Modeling and User-Adapted Interaction (18:5), pp. 455-496.

DeLone, W.H., and McLean, E.R. 1992. "Information Systems Success: The Quest for the Dependent Variable," Information systems research (3:1), pp. 60-95.



Ekstrand, M.D., Harper, F.M., Willemsen, M.C., and Konstan, J.A. 2014. "User Perception of Differences in Recommender Algorithms," Proceedings of the 8th ACM Conference on Recommender systems: ACM, pp. 161-168.

eMarketer. 2013. "Smartphones, Tablets Drive Faster Growth in Ecommerce Sales," http://totalaccess.emarketer.com/view/Article/Smartphones-Tablets-Drive-Faster-Growth-Ecommerce-Sales/1009835.

eMarketer. 2015. "Holiday Shopping Preview," http://totalaccess.emarketer.com/view/Report/Holiday-Shopping-Preview/2001535.

Fleder, D., and Hosanagar, K. 2009. "Blockbuster Culture's Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity," Management Science (55:5), May, pp. 697-712.

Ge, Y., Xiong, H., Tuzhilin, A., Xiao, K., Gruteser, M., and Pazzani, M. 2010. "An Energy-Efficient Mobile Recommender System," Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining: ACM, pp. 899-908.

Gedikli, F., Jannach, D., and Ge, M. 2014. "How Should I Explain? A Comparison of Different Explanation Types for Recommender Systems," International Journal of Human-Computer Studies (72:4), pp. 367-382.

Ghose, A., Goldfarb, A., and Han, S.P. 2012a. "How Is the Mobile Internet Different? Search Costs and Local Activities," Information Systems Research (24:3), pp. 613-631.

Ghose, A., and Han, S.P. 2011. "An Empirical Analysis of User Content Generation and Usage Behavior on the Mobile Internet," Management Science (57:9), Sep, pp. 1671-1691.

Ghose, A., Ipeirotis, P.G., and Li, B. 2012b. "Designing Ranking Systems for Hotels on Travel Search Engines by Mining User-Generated and Crowdsourced Content," Marketing Science (31:3), May-Jun, pp. 493-520.

Greene, W.H. 2003. Econometric Analysis. Prentice hall. Greer, T.H., and Murtaza, M.B. 2003. "Web Personalization: The Impact of Perceived Innovation

Characteristics on the Intention to Use Personalization," Journal of Computer Information Systems (43:3), pp. 50-55.

Hansen, L.P. 1982. "Large Sample Properties of Generalized Method of Moments Estimators," Econometrica: Journal of the Econometric Society), pp. 1029-1054.

Harris, Z.S. 1954. "Distributional Structure," Word). Herlocker, J.L., Konstan, J.A., and Riedl, J. 2000. "Explaining Collaborative Filtering Recommendations,"

Proceedings of the 2000 ACM conference on Computer supported cooperative work: ACM, pp. 241-250.

Höpken, W., Fuchs, M., Zanker, M., and Beer, T. 2010. "Context-Based Adaptation of Mobile Applications in Tourism," Information Technology & Tourism (12:2), pp. 175-195.

Hosanagar, K., Fleder, D., Lee, D., and Buja, A. 2013. "Will the Global Village Fracture into Tribes? Recommender Systems and Their Effects on Consumer Fragmentation," Management Science).

Jannach, D., and Hegelich, K. 2009. "A Case Study on the Effectiveness of Recommendations in the Mobile Internet," Proceedings of the third ACM conference on Recommender systems: ACM, pp. 205-208.

Joffe, M.M., Ten Have, T.R., Feldman, H.I., and Kimmel, S.E. 2004. "Model Selection, Confounder Control, and Marginal Structural Models," The American Statistician (58:4).

Knijnenburg, B.P., and Kobsa, A. 2013. "Making Decisions About Privacy: Information Disclosure in Context-Aware Recommender Systems," ACM Transactions on Interactive Intelligent Systems (TiiS) (3:3), p. 20.

Krumm, J. 2009. "A Survey of Computational Location Privacy," Personal and Ubiquitous Computing (13:6), pp. 391-399.

Kumar, N., and Benbasat, I. 2006. "Research Note: The Influence of Recommendations and Consumer Reviews on Evaluations of Websites," Information Systems Research (17:4), pp. 425-439.

Lathia, N., Hailes, S., Capra, L., and Amatriain, X. 2010. "Temporal Diversity in Recommender Systems," in: Proceeding of the 33rd international ACM SIGIR conference on Research and development in information retrieval. Geneva, Switzerland: ACM, pp. 210-217.

Le, Q.V., and Mikolov, T. 2014. "Distributed Representations of Sentences and Documents," arXiv preprint arXiv:1405.4053).

Lee, B.-H., Kim, H.-N., Jung, J.-G., and Jo, G.-S. 2006. "Location-Based Service with Context Data for a Restaurant Recommendation," Database and Expert Systems Applications: Springer, pp. 430-438.

http://totalaccess.emarketer.com/view/Article/Smartphones-Tablets-Drive-Faster-Growth-Ecommerce-Sales/1009835

http://totalaccess.emarketer.com/view/Article/Smartphones-Tablets-Drive-Faster-Growth-Ecommerce-Sales/1009835

http://totalaccess.emarketer.com/view/Report/Holiday-Shopping-Preview/2001535

http://totalaccess.emarketer.com/view/Report/Holiday-Shopping-Preview/2001535



Levy, O., and Goldberg, Y. 2014. "Dependencybased Word Embeddings," Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pp. 302-308.

Li, S.S., and Karahanna, E. 2015. "Online Recommendation Systems in a B2c E-Commerce Context: A Review and Future Directions," Journal of the Association for Information Systems (16:2), p. 2.

Liu, Q., Ma, H., Chen, E., and Xiong, H. 2013. "A Survey of Context-Aware Mobile Recommendations," International Journal of Information Technology & Decision Making (12:01), pp. 139-172.

Matt, C., Benlian, A., Hess, T., and Weiß, C. 2014. "Escaping from the Filter Bubble? The Effects of Novelty and Serendipity on Users’ Evaluations of Online Recommendations,").

McFadden, D. 1980. "Econometric Models for Probabilistic Choice among Products," Journal of Business), pp. S13-S29.

Mikolov, T., Chen, K., Corrado, G., and Dean, J. 2013a. "Efficient Estimation of Word Representations in Vector Space," arXiv preprint arXiv:1301.3781).

Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. 2013b. "Distributed Representations of Words and Phrases and Their Compositionality," Advances in Neural Information Processing Systems, pp. 3111-3119.

Nielsen. 2014. "Total Audience Report: Q3 2014," http://totalaccess.emarketer.com/view/Chart/Average-Daily-Time-Spent-with-Media-Among-US-Consumers-Q3-2012-Q3-2013-Q3-2014-hrsmins/164966.

Oestreicher-Singer, G., and Sundararajan, A. 2012a. "Recommendation Networks and the Long Tail of Electronic Commerce," Mis Quarterly (36:1), Mar, pp. 65-83.

Oestreicher-Singer, G., and Sundararajan, A. 2012b. "The Visible Hand? Demand Effects of Recommendation Networks in Electronic Markets," Management Science).

Park, M.-H., Hong, J.-H., and Cho, S.-B. 2007. "Location-Based Recommendation System Using Bayesian User’s Preference Model in Mobile Devices," in Ubiquitous Intelligence and Computing. Springer, pp. 1130-1139.

Pathak, B., Garfinkel, R., Gopal, R.D., Venkatesan, R., and Yin, F. 2010. "Empirical Analysis of the Impact of Recommender Systems on Sales," Journal of Management Information Systems (27:2), Fal, pp. 159-188.

Riboni, D., Pareschi, L., and Bettini, C. 2009. "Privacy in Georeferenced Context-Aware Services: A Survey," in Privacy in Location-Based Applications. Springer, pp. 151-172.

Ricci, F. 2010. "Mobile Recommender Systems," Information Technology & Tourism (12:3), pp. 205-231. Robins, J.M. 1999. "Association, Causation, and Marginal Structural Models," Synthese (121:1), pp. 151-179. Rubin, D.B. 2006. Matched Sampling for Causal Effects. Cambridge University Press. Rubin, D.B., and Thomas, N. 2000. "Combining Propensity Score Matching with Additional Adjustments

for Prognostic Covariates," Journal of the American Statistical Association (95:450), pp. 573-585. Rumelhart, D.E., Hinton, G.E., and Williams, R.J. 1988. "Learning Representations by Back-Propagating

Errors," Cognitive modeling (5). Scipioni, M.P. 2011. "Towards Privacy-Aware Location-Based Recommender Systems.'," IFIP Summer

School 2011). Shafir, E., Simonson, I., and Tversky, A. 1993a. "Reason-Based Choice," Cognition (49:1), pp. 11-36. Shafir, E.B., Osherson, D.N., and Smith, E.E. 1993b. "The Advantage Model: A Comparative Theory of

Evaluation and Choice under Risk," Organizational Behavior and Human Decision Processes (55:3), pp. 325-378.

Sklar, M., Shaw, B., and Hogue, A. 2012. "Recommending Interesting Events in Real-Time with Foursquare Check-Ins," Proceedings of the sixth ACM conference on Recommender systems: ACM, pp. 311-312.

Stock, J.H., and Yogo, M. 2005. "Testing for Weak Instruments in Linear Iv Regression," Identification and inference for econometric models: Essays in honor of Thomas Rothenberg).

Swaminathan, V. 2003. "The Impact of Recommendation Agents on Consumer Evaluation and Choice: The Moderating Role of Category Risk, Product Complexity, and Consumer Knowledge," Journal of Consumer Psychology (13:1), pp. 93-101.

Swamy, P.A. 1970. "Efficient Inference in a Random Coefficient Regression Model," Econometrica: Journal of the Econometric Society), pp. 311-323.

Sweeting, A. 2013. "Dynamic Product Positioning in Differentiated Product Markets: The Effect of Fees for Musical Performance Rights on the Commercial Radio Industry," Econometrica (81:5), pp. 1763-1803.

http://totalaccess.emarketer.com/view/Chart/Average-Daily-Time-Spent-with-Media-Among-US-Consumers-Q3-2012-Q3-2013-Q3-2014-hrsmins/164966





Tintarev, N., and Masthoff, J. 2008. "The Effectiveness of Personalized Movie Explanations: An Experiment Using Commercial Meta-Data," Adaptive Hypermedia and Adaptive Web-Based Systems: Springer, pp. 204-213.

Tintarev, N., and Masthoff, J. 2011. "Designing and Evaluating Explanations for Recommender Systems," in Recommender Systems Handbook. Springer, pp. 479-510.

Tintarev, N., and Masthoff, J. 2012. "Evaluating the Effectiveness of Explanations for Recommender Systems," User Modeling and User-Adapted Interaction (22:4-5), pp. 399-439.

Tsiatis, A. 2007. Semiparametric Theory and Missing Data. Springer Science & Business Media. UBS. 2015. "Us Internet & Interactive Entertainment: Can Internet Stocks Rise Despite Headwinds?,"

http://totalaccess.emarketer.com/view/Chart/US-Mcommerce-Sales-by-Device-2013-2019-millions-change-of-total/165119.

Vargas, S., and Castells, P. 2011. "Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems," in: Proceedings of the fifth ACM conference on Recommender systems. ACM, pp. 109-116.

Vijayasarathy, L.R., and Jones, J.M. 2001. "Do Internet Shopping Aids Make a Difference? An Empirical Investigation," Electronic Markets (11:1), pp. 75-83.

Wang, W., and Benbasat, I. 2007. "Recommendation Agents for Electronic Commerce: Effects of Explanation Facilities on Trusting Beliefs," Journal of Management Information Systems (23:4), pp. 217-246.

Wells Fargo. 2014. "Data, Data, Data: 2015 Internet Advertising Themes," http://totalaccess.emarketer.com/view/Chart/Digital-Ad-Spending-Worldwide-by-Format-2013-2018-billions/164449.

Wilde, L.L. 1980. "The Economics of Consumer Information Acquisition," The Journal of Business (53:3), pp. S143-S158.

Wilson, S., Cranshaw, J., Sadeh, N., Acquisti, A., Cranor, L.F., Springfield, J., Jeong, S.Y., and Balasubramanian, A. 2013. "Privacy Manipulation and Acclimation in a Location Sharing Application," Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing: ACM, pp. 549-558.

Wooldridge, J.M. 2002. Econometric Analysis of Cross Section and Panel Data. Mit Press. Wooldridge, J.M. 2007. "Inverse Probability Weighted Estimation for General Missing Data Problems,"

Journal of Econometrics (141:2), pp. 1281-1301. Xiao, B., and Benbasat, I. 2007. "E-Commerce Product Recommendation Agents: Use, Characteristics, and

Impact," Mis Quarterly (31:1), Mar, pp. 137-209. Xie, J., Knijnenburg, B.P., and Jin, H. 2014. "Location Sharing Privacy Preference: Analysis and

Personalized Recommendation," Proceedings of the 19th international conference on Intelligent User Interfaces: ACM, pp. 189-198.

Yap, G.-E., Tan, A.-H., and Pang, H.-H. 2007. "Discovering and Exploiting Causal Dependencies for Robust Mobile Context-Aware Recommenders," Knowledge and Data Engineering, IEEE Transactions on (19:7), pp. 977-992.

Yoo, K.-H., and Gretzel, U. 2011. "Creating More Credible and Persuasive Recommender Systems: The Influence of Source Characteristics on Recommender System Evaluations," in Recommender Systems Handbook. Springer, pp. 455-477.

Yuan, J., Zheng, Y., Xie, X., and Sun, G. 2011. "Driving with Knowledge from the Physical World," Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining: ACM, pp. 316-324.

Zheng, V.W., Zheng, Y., Xie, X., and Yang, Q. 2010. "Collaborative Location and Activity Recommendations with Gps History Data," Proceedings of the 19th international conference on World wide web: ACM, pp. 1029-1038.

Ziegler, C.-N., McNee, S.M., Konstan, J.A., and Lausen, G. 2005. "Improving Recommendation Lists through Topic Diversification," in: Proceedings of the 14th international conference on World Wide Web. Chiba, Japan: ACM, pp. 22-32.

http://totalaccess.emarketer.com/view/Chart/US-Mcommerce-Sales-by-Device-2013-2019-millions-change-of-total/165119

http://totalaccess.emarketer.com/view/Chart/US-Mcommerce-Sales-by-Device-2013-2019-millions-change-of-total/165119

http://totalaccess.emarketer.com/view/Chart/Digital-Ad-Spending-Worldwide-by-Format-2013-2018-billions/164449

http://totalaccess.emarketer.com/view/Chart/Digital-Ad-Spending-Worldwide-by-Format-2013-2018-billions/164449