how does variety of previous ads inﬂuence consumer’s ad ... · mobile in-app advertising is now...

How Does Variety of Previous Ads InfluenceConsumer’s Ad Response?

Omid Rafieian∗

University of Washington

Hema Yoganarasimhan∗

University of Washington

Abstract

Mobile in-app advertising is now a major source of revenue for many app develop-ers. In this paper, we focus on a unique aspect of in-app advertising – sequential adplacement. In this form of advertising, users are exposed to a sequence of potentiallydifferent ads within a session. This gives rise to a series of questions related to theeffects of ad sequences. In particular, we are interested in the effects of variety ofprevious ads on user’s clicking behavior on the next ad. We use data from the leadingin-app ad-network from an Asian country to examine this question. A unique featureof our data is the use of probabilistic auction for ad placement that has created greatvariation in the sequence of ads users are exposed to within the session. Using thisfeature, we develop an identification strategy that allows us to obtain intent-to-treatestimates. We find that when exposed to a higher variety of previous ads, users aremore likely to click on the next ad. We then explore the sources for the effects ofvariety and identify the sequential organization of exposures as a major source. Thismotivates us to develop a measure of sequential variety that capture variety of objectswhen presented in a sequence. Finally, we show the heterogeneity in the effects ofvariety across user’s past history.

Keywords: mobile advertising, randomized experiment, ad sequencing, variety, dynamic selection,intent-to-treat∗We are grateful to an anonymous firm for providing the data and to the UW-Foster High Performance Computing

Lab for providing us with computing resources. We thank Ryan Dew for detailed comments that have improved thepaper. We also thank the participants of the 2018 UW/UBC Conference, 2018 ISMS Marketing Science Conference,2018 SICS Conference, for their feedback. Please address all correspondence to: [email protected], [email protected].

1

1 IntroductionThe smartphone industry has seen an unprecedented growth in the last few years, with over twobillion smartphone users worldwide (eMarketer, 2017a). In 2016, Internet usage via mobile devicesexceeded that via desktop for the first time worldwide (StatCounter, 2016). An average US adult nowspends 3.3 hours per day on her phone and over 80% of this time is spent on mobile apps (eMarketer,2017b). As a result of the rapid growth of user engagement with apps, mobile in-app advertisinghas become an important media channel for advertisers. In-app advertising now generates over$72 billion worldwide and is one of the primary reasons why most apps are free (AppAnnie, 2017;Hollander, 2017).

Despite this growth, in-app advertising has remained an understudied area in marketing literature.This is partly due to the (mistaken) belief that in-app ads are similar as desktop display ads. Whilethese two forms of advertising share many common features, there are some unique aspects ofmobile in-app advertising because the way users interact with their mobile apps is fundamentallydifferent from browsing a website. This in turn has led to some fundamental differences in how adsare served in mobile environments.

First, sessions tend to be longer and more coherent in mobile apps. Second, mobile screensizes are smaller and users are closer to the screen. As a result, ads are harder to ignore in mobilesettings. Therefore, it can be irritating for the user to see the same ad within the app she is using forminutes at a stretch. Because of these reasons, most in-app ad slots are dynamic, i.e., the ad shownwithin a session is replaced after a fixed time period (e.g., 30 seconds or one minute). So the usermay be exposed to multiple ads within the same session. This is in contrast to desktop display ads,which tend to be static, i.e., the ad is the same during a browsing session. Dynamic ad-slots arealso popular among publishers (and ad-networks) because showing multiple ads during long mobilesessions (instead of one) allows them to monetize their app better. Figure 1 shows an example of astatic and dynamic ad slot

The use of dynamic ad slots by publishers gives rise to a series of questions related to the effectsof ad sequences. In particular, we are interested in quantifying the causal effect of sequence of priorads in the session on user’s ad-response (clicking behavior) – does the variety of the set of ads seenearlier affect a user’s responsiveness to the current ad? Variety effects can stem from the potentialdifferences in how users process information when she has seen a low variety of ads previously vs. ahigh variety of ads. One the one hand, past behavioral research finds that consumers’ categorizationability increases when exposed to higher variety of information, which in turn, can increases theirengagement with content (Redden, 2008). As such, when exposed to a higher variety of previousads, users are more likely to actively engage with the next ad, since they can differentiate it more

(a) Static Ad Slot (b) Dynamic Ad Slot

Figure 1: Examples of a static ad slot (on desktop, on CBS website) and a dynamic ad slot (on amobile device on CBS news apps). Both screen shots were taken at the same time and show theexact same content.

easily. On the other hand, the literature on information overload argues that excessive informationsupply makes users ignore new information, which in turn, depresses their engagement and decisionmaking ability (Jacoby, 1977, 1984). Thus, we do not have a clear prediction on the direction andsource of variety effects in the case of mobile ads. In this paper, we seek to address this gap byproviding answers to the following key questions:

• How does variety of previous ads influence users’ clicking behavior on the next ad?

• What is the source for the effects of variety on users’ clicking behavior? Can we identify whichaspects of variety contribute more to the outcome?

• Is there any heterogeneity in the effects of variety across user-level attributes?

To answer these questions, we use the impression-level data from the leading mobile advertisingad-network from a large Asian country. Given the observational nature of our data, we face twomain challenges in quantifying the causal effect of the variety of previous ads (or other sequencerelated features) on ad response – 1) cross-sectional selection and 2) dynamic selection. The formerstems from the lack of full randomization in users’ assignment to different levels of variety, whereasthe latter relates to the compliance issue – the observed data only include the cases where the usershave complied to receive variety treatment.

Cross-sectional selection issue is similar to the main challenge in most observational studiesmeasuring the effects of advertising, i.e., targeting of ads. Given that advertisers can target users bytheir location and time of the day, the set of ads competing for one user-session combination may

2

not be the same as that for another user-session combination. Thus, sessions where more advertisersare targeting will naturally have higher variety of ads competing and also higher likelihood of clicks.This can bias the estimates of variety.

To address this issue, we exploit two notable features of our setting. First, the ad-networkemploys a quasi-proportional auction mechanism (Mirrokni et al., 2010b) wherein each ad has achance of winning the auction proportional to the expected revenue they can potentially generate(bid × quality score). As such, this auction uses a probabilistic allocation rule which is in contrastwith the common practice of using a deterministic mechanism such as second-price auction. Suchprobabilistic allocation mechanism creates a great degree of randomization in ads within a session.We argue that if the auction is the same across all units (in the set of advertisers bidding), theassignment to sequence of ads, and particularly variety, is exogenous. Hence, if we identify suchauction-invariant strata in our data, we can view our problem as a stratified randomized experiment.Here the second feature of our setting comes into play: the ad-network only allows limited targeting.This helps us identify auction-invariant strata with enough observations, mitigating the issuesrelated to the statistical power. Thus, controlling for the variation in the auction leaves us with theexogenous variation in variety only caused by the randomized nature of the auction.

While controlling for the variation in the auction helps us address the cross-sectional selectionproblem, dynamic selection of users is still an issue in determining the causal effects of variety.That is, some users may leave the session before assigned to a certain level of variety. In particular,if the assignment to variety affects users’ decision to leave the session, the estimates on the observedsample would not reveal the effects of variety on users who are intended to receive the treatment.This is equivalent to the issue of non-random non-compliance in randomized controlled trials. Ourchallenge, however, is that we only observe users who have received the full treatment (compliers)and the sequence for those who have left the session (non-compliers) is missing.

We address this issue by recovering the intended variety assignment for users who have leftsessions (non-compliers). Again, we make use of our auction-invariant strata to construct suchintended variety assignment. Since auctions are identical within the same stratum, we can fill in thenext ads for users who have left the session, using the actual ad placement in that stratum in the fulldata. This gives us the sequence of ads the user would have been assigned to if she had not left thesessions. We can then use this intended sample to obtain intent-to-treat estimates for the effects ofvariety.

We estimate the effects of variety on both observed and intended samples. We find that varietyof previous ads will lead to higher probability of click on the next ad. The results are robust acrossdifferent exposure numbers and measures of variety. We also find that repeated exposures of an

3

ad within the session increase the likelihood of clicking on that ad. Comparing our estimates onthe observed sample with those on the intended sample, we find evidence for the existence ofnon-random non-compliance: users who are assigned to a lower variety are more likely to leave thesession. Thus, if we assume random compliance, we will underestimate the intent-to-treat estimatesof variety effects.

Next, we explore the source for the effects of variety. Using different measures of variety, wecan decompose the variety into smaller pieces that relate to a specific aspect of variety. First, wecompare distinctiveness of previous ads with consecutive changing of the ads. While distinctivenessof previous ads plays an important role, we find that a larger fraction of the variety effects isattributed to consecutive changing of ads. Second, we find that diversity in consecutive pairs of priorexposures (e.g., second and third exposures) is more important than the diversity in non-consecutivepairs (e.g., first and third exposures). Both findings identify the sequential organization of theexposures as a major source driving the effects of variety. This motivates us to propose a newmeasure for sequential variety that capture variety of objects when presented in a sequence.

Finally, we find that the effect of variety is heterogeneous across users and user histories.Specifically, we find that users are more likely to respond to variety when they are new to theplatform. Similarly, the effects of variety tend to be stronger on users with fewer past interactionswith the current ad. However, users who already expressed some interest in ads (by clicking at leastonce before) are more likely to be responsive to variety-based manipulations. On the other hand,variety sometimes has negative effects on those who have not clicked at all.

In sum, our paper contributes to the literature in several ways. Methodologically, we proposean identification strategy for observational studies that focus on advertising effectiveness. Theonly requirement for our strategy to be useful is probabilistic allocation mechanism. In particular,as in this paper, our identification strategy can help with questions related to temporal effects ofadvertising where dynamic selection is an issue. Substantively, we introduce variety of previous adsas an important factors that affect user’s click behavior. To the best of our knowledge, this is thefirst work to recognize the effects of prior ad variety on current ad response. We further explore thesources for variety effects and identify the sequential organization of exposure within the session asa major source. To capture the sequential aspect of variety, we develop a new measure for sequentialvariety which is a generalization of entropy measure introduced by Simpson (1949a) to cases whereobjects are presented in a sequence.

2 Related LiteratureOur paper relates and contributes to different streams of literature in marketing and economics.

Broadly, our paper relates to the literature on measuring the causal effects of advertising. Selec-

4

tion problems induced by the ad allocation mechanism make it hard to obtain unbiased estimatesfor the effects of advertising (Blake et al., 2015; Gordon et al., 2017). Besides, statistical power isusually very low requiring a data-rich environment for identification of the effects (Lewis and Rao,2015; Johnson et al., 2016c). A series of works have conducted large-scale randomized experimentsto answer various research questions while mitigating the above challenges (see Lewis et al. (2015)for an excellent review of this stream of research). To address these challenges in observational data,however, researchers have employed novel identification strategies such as difference-in-differencestrategy to examine the effects of privacy regulations on advertising effectiveness (Goldfarb andTucker, 2011), border strategy to quantify the effects of TV advertising (Shapiro, 2016), regres-sion discontinuity design (RDD) to determine the position effects in sponsored search advertising(Narayanan and Kalyanam, 2015), and within-advertiser exogenous variation in positions due torandomization in Bing’s search auctions to study the interplay between position and prominence(Jeziorski and Moorthy, 2017). Our paper contributes to this literature in two ways. First, we pro-pose an identification strategy for settings where ads are placed through a probabilistic mechanism.Second, we bring behavioral literature on variety and establish the causal link between varietyof previous ads shown to the user and her propensity to click on the next ad. To the best of ourknowledge, this is the first paper to study the effects of ad variety on consumers’ ad response.

More specifically, our paper relates to a somewhat narrower area of research on temporal effectsof advertising. This stream of research includes the works on spillover and carryover effects inonline advertising (Rutz and Bucklin, 2011; Sahni, 2016; Johnson et al., 2016a; Sahni, 2015b)and attribution problem (Li and Kannan, 2014; Li et al., 2016). Our work closely relates to Sahni(2015b), as both focus on temporal features of ad exposures, the general idea of temporal sequencingof ads and its impacts on user’s information processing. Our paper adds to this stream of literature intwo ways. First, we focus on variety of ads as main construct and offer it as an important aspect ofad sequencing. Second, we develop a new approach for observational data to recover intent-to-treatestimates for temporal effects of advertising when compliance is influenced by the intervention.Conceptually, our approach is close to Johnson et al. (2017) whose predicted ghost ad approach isto transform the intended sample into that of compliers to estimate average treatment effects onthe treated. However, we do the opposite by constructing the intended sample in order to recoverintent-to-treat estimates.

Finally, our paper relates to the marketing literature on the concept of variety. We can broadlycategorize prior research on variety into two streams based on the role of variety in the analysis.In the first stream of work, variety serves as the main outcome of interest and consumers canactively choose different levels of variety. This includes studies on consumer’s demand for variety

5

(Kim et al., 2002; Datta et al., 2017) and variety-seeking behavior (McAlister, 1982; Ratner et al.,1999). The second stream, however, views variety as a factor influencing the outcome, such asthe link between variety of assortments and store choice (Hoch et al., 1999), variety of episodesand consumer’s engagement (Redden, 2008), and the dispersion of word-of-mouth and TV ratings(Godes and Mayzlin, 2004). Related to the second stream of research, we bring the notion ofvariety to the advertising context and examine whether variety of ads seen earlier affect user’s adresponse. To our knowledge, the only study related to variety in this context is Schumann et al.(1990) who show the variation of ad content over a repeated advertising schedule will increaseuser’s responsiveness to that ad. While they only focus on variation in the content for one ad in alab setting, our work extends it to the variety of potentially competing ads in a large-scale in-appadvertising market. Further, our paper adds to the marketing literature on variety by providing aframework for decomposing the effects of variety and developing a new measure of sequentialvariety that can be used for cases where objects are presented in a sequence.

3 Setting and DataOur data come from the leading mobile in-app advertising network of a large Asian country. Wehave data on all the impressions and corresponding clicks (if any) in the platform, for all theparticipating apps for a one month period from 30 September 2015 to 30 October 2015. The totaldata we see in this one month interval is quite large. Overall, we observe a total of 1,594,831,699impressions and 14,373,293 clicks in this time-frame, implying a 0.90% CTR.

The setting and data source are the same as that used in Rafieian and Yoganarasimhan (2018).So some of the descriptions in §3.1 and §3.2.1 are similar to those in the earlier paper. However, thesampling procedure and the data used for the analysis are very different here.

3.1 Setting

3.1.1 Main Players

We now describe the setting in greater detail, starting with a description of the four main players inthis market. First, we have users, who use apps and generate impressions. They see the ads shownwithin the apps that they use and may choose to click on the ads. Second, we have publishers/app-developers, who sell the impressions on their app through the ad-network. They earn 70% of thecost of each click in their app (paid by the advertiser), and the remaining 30% is the ad-network’scommission. Third, we have advertisers – firms that show ads through the ad-network. They designbanner ads, specify their bid (as the WTP for a click), and can include a maximum daily budget ifthey choose to. Finally, we have the ad-network or platform, which functions as the matchmakerbetween advertisers and publishers. It runs a real-time auction for each impression generated by the

6

participating apps and shows the winning ad during the impression.

3.1.2 Dynamic Ad Slot

The ad-network has two possible choices for designing the ad-slot: 1) static ad-slot and 2) dynamicad-slot. In the former, the ad shown in a session remains constant, i.e., ads are not replaced within asession. In the latter, ads within a session are sequentially replaced after a fixed time period. So theuser may be exposed to multiple ads within the same session.

In general, display ads shown in desktop/browsers tend to be static whereas mobile in-app adstend to be dynamic.1 This difference stems from a few contextual distinctions between mobile appusage and desktop-based browsing. First, the amount of time a user spends on a mobile app islonger and more coherent than what she spends browsing a website on a desktop. So it is possible toshow multiple ads within the same session. In contrast, users spend only a short time on a websitebefore scrolling/clicking, which makes it unnecessary to change the ad on a given website duringthe browsing session. Second, mobile screens are generally small and users tend to be closer to thescreen. As a result, ads are harder to ignore in mobile apps. So it can be irritating for the user to seethe same ad within the app she is using for minutes at a stretch. In contrast, the user only stays onthe website for a short time and is unlikely to be bothered by seeing the same ad for a short timeon a large screen. Finally, publishers and ad-networks view eye-balls as a valuable resource. Ifthey do not replace ads in mobile app sessions, they may be simply wasting potential monetizationopportunities.

Our ad-network employs dynamic ad-slots, where each impression lasts one minute. When auser starts using an app, the ad-network runs an auction to determine the winning ad and servesthis ad for one minute. If a user continues using the app beyond one minute, then the ad-networktreats this as a new impression and runs another auction to determine the next ad to show the user.This feature helps us formally define a session, which is the main unit of analysis in this paper.We define a session as a set of consecutive impressions within an app for a specific user, such thatthe gap between two consecutive impressions is less than 10 minutes. Figure 2 shows a two-hoursnapshot of a user’s exposure to ads in order to help illustrate the main idea behind the definitionof sessions. Impressions are visualized by their starting and ending points, depicted using red [ ].The grey areas show the time between the session, i.e., the length of each grey area exceeds 10minutes. Overall, we see that this user has initiated five separate sessions of varying lengths. Forexample, the third session in this picture lasts for 20 minutes and contains 10 impressions, whereasthe fourth one lasts only for two minutes (and contains two back-to-back impressions). Note that

1Recently, some ad-networks like Google Display have started allowing publishers to refresh ads on their web pageafter a fixed period of time or after a specific action taken by the user ().

7

[ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ]

20 40 60 80 100 120Time (Minutes)

Figure 2: Definition of the Session

not all impressions within a session are back to back, i.e., there can be some gap in time (< 10

minutes) between consecutive impressions within a session. For example, in the third session, thereis a 5 minute space between two consecutive impressions.

3.1.3 Auction Mechanism

To allocate ads to impressions, the ad-network uses a quasi-proportional auction mechanism(Mirrokni et al., 2010a). The main distinction between a quasi-proportional auction and othercommonly used auction mechanisms (e.g., Second Price or Vickrey) is the use of a probabilisticwinning rule: for each ad a ∈ A, the auction assigns πa as the probability of winning as follows:

πa =bama∑j∈A bjmj

(1)

where ba and ma are advertiser a’s bid and quality score respectively. The quality score reflectsthe probability of click for each ad. This score can be calculated at the impression-level. Inour data, however, the extent of customization in the quality scores is quite low: The platformsimply aggregates all total past impressions and clicks for an ad, and uses the ad-specific eCTR(expected Click Through Rate) as the quality score (ma). Intuitively, an ad’s probability of winningis proportional to the expected revenue generated from it. Thus, unlike deterministic auctions, thead that can generate the highest expected revenue for the platform is not guaranteed to win.

The combination of dynamic ad-slots and the probabilistic nature of the auction generates a greatdegree of randomization in the sequence of ads within a session. Further, users cannot self-selectinto ad exposures. As such, with proper controls, we can treat our setting as a quasi-experiment, inwhich users are randomly assigned to different sequences of ads.

8

3.2 Data

3.2.1 Variables

For each impression in the data, we have access to the following information:

• Time and date: The time-stamp of the impression.

• AAID: Android Advertising ID is a user re-settable, unique, device ID that is provided by theAndroid operating system.2 It is accessible to advertisers and ad networks for tracking andtargeting purposes. We use it as the user-identifier in our main analysis.

• App ID: A unique identifier for apps that advertise through the platform.

• Ad ID: This is an identifier for ads that are shown to the users.

• Location: The exact location of a user based on latitude and longitude. The ad-network usesthis information to determine the province/state where the user is currently situated.

• Connectivity type: It refers to the user’s type of connectivity (e.g., WiFi or cellular data)

• Smartphone brand: The brand of user’s smartphone (e.g., Samsung, Huawei, etc.)

• MSP: The user’s mobile-phone service provider.

• ISP: The user’s Internet service provider.

• Click indicator: This variable indicates whether or not the user has clicked on the ad.

3.2.2 Sampling Procedure

We are interested in understanding how the sequence of ads shown previously affects a user’sresponse to future ads. Thus, by definition, user-level history plays a central role in our analysis.To fully exploit and control for user history, we need a user’s complete behavioral history in thead-network. So our goal is to identify new users during our sample period and follow them till theend of our sampling period and construct their user history for each impression. This leads to achallenge: the starting point of our data is not the starting point of the ad-network. As a result, someusers may have seen ads before 30 September 2015. Below, we present a step-by-step discussion ofhow we clean and sample the data. Note that users are identified by a unique AAID, i.e., a AAIDrefers to a unique user (as mentioned in §3.2.1).

2Apple’s app store is not available in the country where our data are sourced from. Hence, all smartphones use theAndroid operating system.

9

• Before sampling, we clean the data by excluding two types of users/AAIDs. First, somesmartphone models generate the same AAID for multiple users, which makes it impossiblefor us to uniquely identify users for these models. We therefore exclude all AAIDs from suchsmartphone models. Second, we exclude users who have been tagged as fraudulent by thead-network (typically those with unreasonably high CTR).3

• Next, to identify new users, we split the data into two parts. The first part of the data consistsof impressions from 30 September 2015 to 21 October 2015. The second part consists ofimpressions from 22 October 2015 to 30 October 2015. For every user in the second part of thedata, we examine if that user was seen at least once in the first part of the data. If not, then weconclude that the user is new to the ad-network (joined on or after 22 October 2015) and includeall impressions by this user in the second part of the data in our sample. Instead, if the user wasseen before, then we drop her from our analysis because we cannot be sure that this user is new.

• Within the set of users we identify as new, we exclusively focus on their impressions in the mostpopular app. This is a messaging app, is widely used in the country, and generates over 30%of the total traffic observed in the ad-network. Just focusing on one app allows us to hold thecontext of the app constant and derive the causal effect of the past sequence of ads.4

While we focus on the top app for our analysis, we keep the data for other apps to track userswho use multiple apps. As such, some features we will use are generated using all the data.

Overall, we observe 7,066,483 impressions in the top app generated by 96,076 users. This corre-sponds to 1,291,477 sessions in this app. The average CTR is 0.022, which is relatively high. Asmentioned above, we also track users when they use other apps to construct their full impressionhistory. We find that over 40% of the users use other apps. Including those impressions will give usa data set of 8,549,641 impressions over 1,453,039 sessions, with an average CTR of 0.020.

3.3 Summary Statistics

We now describe our data and present some summary statistics for the key variables of interest.

3.3.1 Categorical Variables

As described in §3.2.1, each impression in our data is associated with a set of categorical variables:province, connectivity type, MSP, ISP and smartphone brand. The statistics we provide for eachof these variables include number of categories and the share of top three categories within each

3Since most publishers monetize their apps through ads served in their apps, some of them click on all ads shown intheir apps to generate more revenue, i.e., engage in click fraud. The ad-network uses proprietary algorithms to identifysuch fraudulent users.

4In principle, we can include more apps in the analysis as long as we have sufficient number of impressions within eachapp so that we can stratify the data at the app level.

10

variable. Table 1 reports these statistics on the main categorical variables in our data. All thesenumbers are shown for the final sample we use for analysis.

Variable Number of Share of top categories Number ofcategories 1st 2nd 3rd impressions

Province 31 24.55% 9.52% 7.42% 7,066,483Smartphone Brand 7 44.72% 38.12% 6.65% 6,416,152Connectivity Type 2 50.23% 49.47% 7,066,483ISP 8 70.89% 15.86% 4.98% 3,003,829MSP 3 50.14% 44.04% 5.81% 6,890,873

Table 1: Summary statistics of the categorical variables. The number of non-missing observationsfor each variable are shown. While the location/province information is always available, the othervariables are missing for some impressions. The shares shown are computed after excluding themissing observations for each variable.

3.3.2 Session Length

Sessions form the main unit of analysis in our study and therefore we now present some summarystatistics on the length of a session. Session length is defined as the number of impressions (orexposures) that were served during the session. We see significant variation in the length of thesessions in our data. This is because we focus on the messaging app, which people mainly use tochat with their friends, and such chat sessions can vary quite a bit in length. Figure 3 shows theempirical CDF of the session length of the 1,453,039 sessions in our main data. About half of thesessions end after the first two impressions/exposures. Further, a vast majority of sessions last forless than 10 impressions. However, we see some very long sessions during which the user sees over60 ad exposures. This implies that the distribution of session length has a long tail.

3.3.3 Click-through Rate Across Different Exposure Numbers

We focus on click as the main outcome of interest. Since we are interested in session-levelinterventions, a good starting point would be the comparison of user’s clicking behavior in differentexposure numbers within the session. We show the likelihood of a user’s clicking on an impressionat different exposure numbers in Figure 4. This figure illustrates the average CTR at each exposurenumber, from 1 to 30. It reveals a downward trend in CTRs over exposure number, suggesting thatusers who have continued for long periods of time are less likely to click.

3.3.4 Share of Ads

As mentioned earlier, the platform runs a probabilistic auction that determines the winner foreach impression using a probabilistic rule. As such, all ads participating in the auction for animpression have a chance to win. However, the propensity to win the auction for an impression

11

0.00

0.25

0.50

0.75

1.00

0 20 40 60Session Length

Fra

ctio

n of

Ses

sion

s

Figure 3: Empirical CDF of session length (truncated at 60).

●

●

●

●

●

● ●

● ●

● ●●

●● ● ●

● ●●

● ● ● ●

●●

●●

● ●●

0.00

0.01

0.02

0.03

0 10 20 30Exposure Number

Ave

rage

Clic

k−th

roug

h R

ate

Figure 4: Click-through Rate Across Different Exposure Numbers

varies significantly across ads due to three main factors – bid, budget, and targeting. First, adsthat have a higher bid have a higher probability of winning the auction as shown in Equation (1).However, if an ad has a high bid but a fixed budget, then it would win impressions earlier in theday, but may run out of the budget soon as a result of getting clicks. Thus, they will be out ofcompetition for later impressions and may only get a few impressions overall. In some cases, the ad

12

may have a high bid and also set a high budget, but target more narrowly. Then, in spite of the highbid and budget, it will still get a smaller share of the impressions in the ad-network because it onlycompetes for (or targets) a small set of impressions. Thus, the bid, budget, and targeting decisionsof the ad together determine the share of total impressions awarded to it.

We see a total of 327 distinct ads in our data. We first calculate the shares of a given ad as thefraction of impressions showing that specific ad in the total set of impressions in our data. We thensort the ads by their share and present the CDF of the shares in Figure 5. Note that most ads onlyget a tiny fraction of impressions due to the reasons outlined above. However, a few ads accountfor the vast majority of impressions. Specifically, the top 50 ads account for over 90% of the totalimpressions.

0.00

0.25

0.50

0.75

1.00

0 100 200 300Top Ads

Fra

ctio

n of

Impr

essi

ons

Figure 5: Cumulative fraction of impressions associated with ads.

4 Variety of Previous AdsOur main goal is to see how the variety of previous ads affects user’s clicking behavior on the nextad. As such, variety is the main independent variable of interest in this study. In this section, wefirst provide some background on the concept of variety in marketing in §4.1. We then present aformal definition of variety in our context in §4.2. In §4.3, we show some summary statistics onvariety in our data. Finally, we offer model-free evidence for the link between variety and user’sclicking behavior.

13

A A B A C C B D E

Previous Exposures Current Exposure

1st 2nd 3rd 4th 5th 6th 7th 8th 9th

Figure 6: An example of a session with where the user is at the ninth exposure. The numbersrepresent exposure number t and each rectangle represents the ad shown in that exposure. The lettercoding refers to ad IDs, i.e., each letter represents one unique ad. For instance, the user is shown thesame ad (coded in letter A) during the first, second, and fourth exposures.

4.1 Theoretical Background

In advertising context, effects of variety can stem from potential differences in user’s informationprocessing when being exposed to high vs. low variety of ads. That is, the user processes informationabout the current ad differently, having seen a high (vs. low) variety of previous ads. Suchdifferences, however, can be in either direction depending on the context. One the one hand, pastbehavioral research consumers’ categorization ability increases when exposed to higher variety ofobjects, which in turn, can focus their attention differentiating aspects of the current object (Redden,2008). As such, when exposed to a higher variety of previous ads, users are more likely to activelyengage with the next ad, since they can differentiate it more easily. On the other hand, the literatureon information overload argues that excessive information supply makes users less receptive of newinformation, which in turn reduces their engagement and responsiveness to subsequent interventions(Jacoby, 1977, 1984).

4.2 Definition of Variety

Measuring variety is often a hard and challenging task. Depending on the context, researchers havedeveloped various metrics capturing the breadth of variety, diversity, concentration, etc. (Harrisonand Klein, 2007). For any exposure, our measure of variety is defined over the sequence of prior adsshown in a session. Figure 6 presents a pictorial depiction of the sequence of ads, wherein the userhas seen eight exposure prior to the current exposure. Our goal is to define measures that capturevariety of previous ads. We describe metrics that we use and their purpose below:

• Breadth of variety: This metric counts the number of distinct ads shown within the session sofar. As such, it captures the breadth of variety. For exposure number t in session i, we can definebreadth of variety as follows:

Varietyi,t = |{Ai,1, . . . , Ai,t−1}|, (2)

14

where Ai,t denotes the ad shown in exposure number t in session i. For example, the user hasseen four distinct ads (A, B, C, D) prior to the current exposure in Figure 6. For simplicity,we put our primary focus on this measure of variety. Henceforth, we mean breadth of varietywhenever we use the word variety in a plain way.

• Consecutive changes: Since ads are shown sequentially, users are not able to see the full sequenceat the same time. However, the sequential organization of exposures can potentially influenceusers’ perception of variety. This is in line with prior research on variety of assortments thatshow the organization and structure of the assortment play an important role in user’s perceptionsof variety (Hoch et al., 1999; Kahn and Wansink, 2004). One measure of variety that reflect thesequential organization of the session is the total number of consecutive exposures that showdifferent ads. That is, the number of times the user has noticed a change in ads in consecutiveexposures. The formal definition of sequential variety is shown below:

Changei,t =t−1∑j=2

1(Ai,j 6= Ai,j−1), (3)

In Figure 6, we see 5 consecutive changes: consecutive changing of ads does not happen inthe second and sixth exposure. It is easy to show that Changei,t ≥ Varietyi,t − 1, as the firstexposure a distinct ad would be a change of the last exposure, except for the first ad.

• Gini–Simpson index: To capture diversity of the sequence, we can use different diversity(concentration) metrics. The most commonly used metric in marketing and economics literatureis Herfindahl index which is the equivalent of Simpson’s entropy measure (Simpson, 1949b).This measure simply calculates the sum of squared shares of ads shown in the sequence, whichintuitively means the probability that two random exposures from the sequence show the samead. However, sum of squared shares is not the unbiased estimator of this probability for smallsamples. For exposure number t, let Ii,a,t denote the number of prior exposures showing ad a.The probability that two random exposures from the past show the same ad can then be writtenas follows:

Simpsoni,t =∑a

Ii,a,t(Ii,a,t − 1)

t(t− 1)(4)

The higher the value of this metric, the more concentrated and less diverse the sequence of priorads is. To obtain a measure of diversity, we can use the inverse of this metric or the probabilityof the complementary event, i.e., two random exposures from the sequence show different ads.

15

0.00

0.25

0.50

0.75

1.00

1 2Variety of Previous Ads

Fra

ctio

n of

Impr

essi

ons

Exposure Number = 3

0.00

0.25

0.50

0.75

1.00

1 2 3Variety of Previous Ads

Fra

ctio

n of

Impr

essi

ons

Exposure Number = 4

0.00

0.25

0.50

0.75

1.00

1 2 3 4Variety of Previous Ads

Fra

ctio

n of

Impr

essi

ons

Exposure Number = 5

0.00

0.25

0.50

0.75

1.00

1 2 3 4 5Variety of Previous Ads

Fra

ctio

n of

Impr

essi

ons

Exposure Number = 6

0.00

0.25

0.50

0.75

1.00

1 2 3 4 5 6Variety of Previous Ads

Fra

ctio

n of

Impr

essi

ons

Exposure Number = 7

0.00

0.25

0.50

0.75

1.00

1 2 3 4 5 6 7Variety of Previous Ads

Fra

ctio

n of

Impr

essi

ons

Exposure Number = 8

0.00

0.25

0.50

0.75

1.00

1 2 3 4 5 6 7 8Variety of Previous Ads

Fra

ctio

n of

Impr

essi

ons

Exposure Number = 9

0.00

0.25

0.50

0.75

1.00

1 2 3 4 5 6 7 8 9Variety of Previous Ads

Fra

ctio

n of

Impr

essi

ons

Exposure Number = 10

Figure 7: Empirical CDF of variety for different exposure numbers. Note that by definition varietyis less than exposure number.

We use the latter which is called Gini-Simpson index and can be written as follows:

GiniSimpsoni,t = 1− Simpsoni,t = 1−∑a

Ii,a,t(Ii,a,t − 1)

t(t− 1)(5)

In Figure 6, this measure is approximately equal to 0.82, which is the probability that tworandom exposures show the same ad. Obviously, it takes its highest value when all exposuresshow different ads, i.e., GiniSimpsoni,t = 1. With the same breadth of variety as in Figure 6, thesequence could have been more diverse if the fourth exposure was ad D instead of ad A, with allads having the same share and Gini-Simpson index of 0.93.

4.3 Variation in Variety

A necessary condition to identify the effects of variety is to have sufficient variation in the varietymetric. To visualize the extent of this variation, we plot the empirical CDF of the variety of previousads for different exposure numbers in Figure 7. We observe substantial variation in the variety ofprevious ads that users have been exposed to earlier in the session at all exposure numbers. Theexistence of this variation satisfies a necessary condition for estimating the effects of variety.

16

●

●●

●

●

●

●

●

● ● ●

●

●

●

0.000

0.025

0.050

0.075

2 4 6Variety

Ave

rage

CT

R

Group ● ●Ad 1 Ad 2

Figure 8: Average CTR of top two ads at exposure number 8 when exposed to different levels ofvariety

This variation can stem from two possible sources. First, the proportional auction mechanismnaturally induces variation in variety (if there are enough ads competing in a given session) and thisvariation is random conditional on the set of competing ads and helps with identification. Second,this variation could stem from variation in the set of ads competing for any given session. That is,some sessions could have more ads competing for them (which naturally leads to higher varietyin proportional auctions) compared to others. This type of variation is non-random since the adscompeting in a given session are self-selected. Thus, we have to control for this selection in ourmodeling exercise. We discuss the latter issue in greater detail §5.1.

4.4 Model-free Evidence

To see whether our data show any patterns indicating a link between variety of previous ads andclicking on the next ad, we focus on impressions of top two ads in our data shown in exposurenumber 8. We then plot each ad’s average CTR against variety of previous ads. The patterns areshown in Figure 8, suggesting a general increasing trend in average CTR by variety. While thisfigure gives us some hints about the link between variety and clicking behavior, we must be cautious

17

in causal interpretation of this result. There may be potential selection issues constituting suchpatterns. For example, users with higher propensity to click may be targeted by more advertisers,thereby shown a higher variety of ads. Further, showing a low variety of ads could make usersleave the session before getting treated. In the next section, we discuss selection issues and ouridentification strategy in detail.

5 Selection Issues and Identification StrategyBroadly speaking, we are interested in quantifying the causal effect of the previous sequence of adson a user’s clicking behavior, with a particular focus on the variety of previous ads. We face twotypes of selection issues in this task: (a) cross-sectional selection, which stems from differences inthe auction across impressions, and (b) dynamic selection, which is caused by users longitudinaldifferences in drop-off rates across users in different variety conditions. We discuss these twoproblems and our identification strategy to address them in §5.1 and §5.2.

5.1 Cross-Sectional Selection

A necessary condition to identify treatment effects is unconfoundedness or Strong Ignorability

of Treatment Assignment (Rosenbaum and Rubin, 1983). That is, users’ assignment to differentlevels of treatment is random given his/her observed pre-treatment covariates. In a fully randomizedexperiment, this assumption holds by definition. However, in most observational studies, assignmentto treatments is conditioned on user-level observables leading to some selection bias. We denotethis as cross-sectional selection, i.e., selection due to users from different cross-sections having adifferent propensity of getting a treatment.

5.1.1 Evidence for Cross-Sectional Selection

In our case, cross-sectional selection stems from the differences in the auction across impressions.In principle, if the same set of advertisers participate in the auction for every impression and theirbids remain constant, all impressions at a given exposure number would have the same propensity ofbe assigned to a certain level of variety. In that case, the treatment assignment would be independentof potential outcomes. However, in our setting, this assumption can fail due to three reasons. First,the number of advertisers competing for an impression varies over time due to the entry of newadvertisers, exit of some older advertisers, budget exhaustion, and etc. Second, while targeting islimited in our platform, it nevertheless happens – some advertisers target certain impressions (basedon location, time of day, etc.). As such, the set of advertisers competing for a given impression isselected. Third, advertisers can change their bid over time, which in turn would affect the propensityof ads allocated (and consequently the propensity of variety). Thus, impressions drawn from two

18

0.00

0.25

0.50

0.75

1.00


Fra

ctio

n of

Impr

essi

ons

(a) Large State, Evening

0.00

0.25

0.50

0.75

1.00


Fra

ctio

n of

Impr

essi

ons

(b) Small State, Morning

Figure 9: Empirical CDF of variety for two slices of the data at the fifth exposure. Both figures are onOctober 30. The first one is in the evening (from 6 pm to midnight) and has 9737 impressions. Thesecond one is in the morning (from 6 am to noon) and contains 1002 impressions. A Kolmogorov-Smirnov test rejects the null hypothesis of identical distributions.

different time periods or targeting areas may have different propensities of being assigned to a givenvariety. If advertisers decision to compete for an impression and their bid is correlated with theexpected CTR of that impression, then the unconfoundedness assumption fails.

We illustrate this problem using an example of two different slices of our data. The first sliceconstitutes impressions from a large province/state in the evening, and the second slice constitutesimpressions from a smaller state in the morning of the same day. Obviously, impressions in thefirst slice are attractive to advertisers because they come from users in a big affluent state at atime of the day when users have leisure time. In contrast, the second slice consists of impressionsfrom a smaller state during work hours and is less attractive to advertisers. As a result, we see aconsiderable difference in number of ads competing in these two slices. There are 73 ads shown inimpressions shown in the first slice, but only 25 ads are shown in the second slice. This results in asignificant difference in the distribution of variety of previous ads, as shown in Figure 9. Users aremore likely to be assigned to a higher variety condition in the larger state in the evening comparedto the smaller state in the evening. Moreover, we find that the users first slice, who attract moreadvertisers, have a higher CTR. Indeed, the first slice has a CTR of 0.023, while the second has aCTR of 0.015. These two observations together suggest that there is selection based on observables,and that we need to adjust for propensities in our inference (Imai and Van Dyk, 2004).

19

5.1.2 Solution to Cross-Sectional Selection: Identifying Auction-Invariant Strata

Our identification strategy is to transform our problem into a stratified randomized experimentwherein the impressions within the strata can be treated as fully randomized experiments (Imbensand Rubin, 2015). To do so, we need to identify the strata within which the underlying auction isidentical. This brings us to our first claim that is proposed below:

Claim 1. Controlling for the auction, the variation in the sequence of ads is exogenous.

The main reason behind this claim is that the auction is the only data generating process forthe ad variable. In light of Claim 1, our goal is to find auction-invariant strata within which theunderlying auction does not vary. To do so, we first need to pin down the reasons accounting forauction variation. The source for auction variation is advertisers’ presence which relates to twobroad categories of advertiser’s active decisions as outlined below:

1. Targeting: Advertisers can target their ads based on app category, province, connectivity type,time of the day, MSP, ISP, and smartphone brand. For each of these targeting variables, theycan select a set of categories wherein they want to show their ad. It means that their ad willnot be shown to the excluded categories. For example, if an advertiser selects Huawei andLG as the set of smartphone brand categories she wants to target, her ad will not be shown toany iPhone users, as iPhone is excluded from her targeting set. Differences in ads’ targetingstrategies will therefore create variation in auctions. However, auctions for two impressionswith the same targeting variables cannot be different due to such differences in ads’ targetingstrategies. This is because the group of impressions with the same targeting variables aretargeting-invariant. We call such groups targeting area and they play a crucial role in findingauction-invariant strata.

2. Entry, exit, and budget: Besides targeting decisions, some time-dependent factors can alsocreate variation in auctions, including advertisers’ entry/exit decision or budget exhaustion.In fact, ads are sometimes not present in the auction because 1) they have not entered yet, 2)they have exited, or 3) their budget has been exhausted and they have not refilled it. Figure 10illustrates this point by showing an ad’s availability (vs. unavailability) during the time ofstudy. Given such events, two auctions in distant points of time will not be identical. Thus,we need to find time period during which such events do not happen.

In light of the variation in auction caused by advertisers’ targeting decision and time-dependentfactors mentioned above, we present our second claim:

20

Timeline of the study

Available Before entry No budget After exit

Entry Exit

Figure 10: Availability of an ad in the timeline of the study due to entry, exit, and budget exhaustion

Claim 2. If no time-dependent factor changes in the time between two impressions in the sametargeting area, their auctions are identical.

Now, our goal is to find time intervals during which no time-dependent factor changes. Inprinciple, if Ea is the set of time-stamps for changes in ad a’s availability,

⋃aEa would give us

time-stamps for all changes in auction over time, whereby we can construct such time intervals.We start with hourly splits of the time variable as an approximation of these time intervals. Thisis motivated by the platform’s decision on updating these changes in auctions on an hourly basis,which makes within hour changes unlikely. However, we employ other approaches to approximatethe true

⋃aEa to show robustness of our results.

We can now use Claim 2 to construct auction-invariant strata. A targeting area g in hour h isa stratum within which the auction does not vary. Thus, given Claim 1 and 2, we can treat ourproblem as a stratified randomized experiment controlling for auction-invariant strata. It is worthnoting that our strategy to address cross-sectional selection problem can be applied in any casewhere there is some degree of randomization in the allocation mechanism.

5.2 Dynamic Selection

The second selection problem we have to address has to do with user drop-out or availability ofusers to receive the full treatment. Since the treatment is given through time, users are not generallyat full compliance. That is, some users may leave the session before assigned to the treatment. Forexample, if assignment to a higher variety of previous ads increases the likelihood of clicking andthen leaving, then the sample of users at a given exposure number (t) will not be homogeneousacross treatments: the group that is assigned to a higher variety of previous ads is less likely tohave survived until exposure number t. As a consequence, the estimated effects of variety on userswho have survived to receive the treatment would not reveal the effects of variety on users who areintended to receive the treatment.

Figure 11 depicts the problem of compliance in our case. In all these cases, the user is supposed

21

Treatment Arm

Variety = 4

Treatment Arm

Variety = 1

Treated

Not

Treated

A B C D X

A B C D X

A B C D X

A A A A X

A A A A X

A A A A X

Figure 11: Compliance and non-compliance in receiving the treatment

to see ad X in the fifth exposure. For simplicity, we only consider two different treatment arms –high (Varietyi,t = 4) and low variety (Varietyi,t = 1). If all these impressions come from a singleauction-invariant stratum, the assignment to each treatment arm would be random, allowing usto identify the treatment effects. However, as shown in Figure 11, treatment arms have differentcompliance rates: one-third of users in the higher variety condition complied to receive the treatment,whereas two-thirds in the lower variety condition did so. Now, if only one impression in each armhas been clicked, mean comparison of the outcomes on the survived sample would indicate a higherCTR when exposed to a higher variety, while there is no difference in the outcomes for groups thatare intended to receive each treatment. In our setting, this is a major challenge for our analysis aswe do not fully observe the sequences that ended before receiving the treatment. Thus, we need tofind a way to address this selection issue.

5.2.1 Evidence for Dynamic Selection

At first glance, it may seem reasonable to assume that usage decision in a messaging app is notinfluenced by the sequence of ads shown to users. This assumption can resolve the issue ofcompliance and dynamic selection, whereby we can only use the treated sample to draw validinference. However, we can test it empirically to see whether users’ leaving decision is affected byads shown to them. A simple examination of this assumption would be comparing users’ leavingdecision following a click. Figure 12 shows the leave probabilities for two groups of users at

22

different exposure numbers in a session – (a) users who clicked at that exposure, and (b) users whodid not click at that exposure. On average, users who clicked are more likely to leave than thosewho did not. While the differences in the leave rates of these two groups is not substantial, thisprevents us from assuming the exclusion restriction mentioned above.

●

●

●

●

●

●●

●

●

●

●

●

● ●

0.00

0.05

0.10

0.15

0.20

0.25


Ave

rage

Lea

ve R

ate

Group ● ●Not Clicked Clicked

Figure 12: Leave rate

In particular, if variety of the sequence of previous ads affects user’s clicking behavior in anyways, the compliance would be endogenous since users in different treatment arms might havedifferent survival probabilities. Later, when we present our results, we revisit the assumption ofexogenous app usage with more elaborate models.

5.2.2 Solution to Dynamic Selection: Intent to Treat Estimates

While users’ intended assignment to different treatment arms is random in a particular auction-invariant stratum, their compliance could be influenced by the treatment arm they are assignedto. Thus, for any exposure number t, estimates on the sample of sessions that have complied toreceive the treatment may be misleading when deciding to assign users to different treatment armsin advance.

Ideally, one would report estimates on both survived and intended samples as both can be of

23

high importance for managers depending on their objective. However, the challenge is that wedo not observe the full treatment for users who have left before getting treated. It is just partiallyobserved till the point they have left the session. Our goal is to construct what sequence users wouldhave been exposed to if they had not left the session. To do so, we first recover the time-stamps ofnext exposures if the user had survived to see exposure number 10.5 Given the auction-invariantstratum each session belongs to, we fill the next ads using the full data to see what ad has beenshown around that point in this stratum. Since auctions are independently run, it gives the sequencethe user would have been assigned to if she had not left the session. Filling the outcomes for missingimpressions as zero, we can construct the full intended data and use to obtain ITT estimates for ourtreatment.

An alternative approach to account for dynamic selection is Inverse Probability Weighting (IPW)which is widely used in longitudinal studies in biostatistics literature (Hernan et al., 2006; Hoganet al., 2004). In this approach, we first estimate the survival probability of the user till that exposurefor the non-missing sample (treated) for each exposure number t. We then weight each impressionin the sample by the inverse of its survival probability, and estimate the models of interest on theweighted sample (Wooldridge, 2007). The intuition behind IPW can be explained using this simpleexample – at any given exposure number t, if we see a user whose survival probability is 0.1 at thisexposure, then IPW gives a weight of ten to this impression to represent nine other impressions bysimilar users who did not survive till this exposure. However, this approach requires sequential

missingness at random assumption (Rotnitzky et al., 1998). That is, given the past history, survivaltill exposure number t does not depend on current and future outcomes. While this assumptionseems reasonable in medical studies where we want to measure an objective outcome (e.g., bloodpressure) when assigned to different treatment arms, it is likely violated in our setting as both leaveand click are active decisions of a user.

Overall, it is worth noting that intent-to-treat (ITT) approach is conceptually the opposite ofinverse probability weighting (IPW). Observed impressions with low survival probability get asmaller weight in the former as it represents more impressions that are intended to receive thistreatment arm but left, whereas in the latter, such impressions have higher weights as we assumethat the missing impressions would have had the same outcome if survived. In advertising context,this assumption is not realistic because missing impressions do not generate any outcome. That iswhy we use intent-to-treat as our main approach in this paper.

5We do not go beyond exposure number 10 as the vast majority of sessions end in less than 10 exposures (see Figure 3).

24

6 Model SpecificationWe now present the model specification. We take care of different modeling pieces presented below:

• Auction-invariant strata: As discussed in §5.1, our solution to cross-sectional problem isstratification. That is, we control for auction-invariant strata, since not controlling for thevariation in auctions likely induces selection bias. Each stratum is characterized by a targetingarea g in a time interval during which no change happens in the auction. In our setting, westart with hourly intervals, but we will later use other time intervals to show the robustness ofour results. Controlling for such auction-invariant strata will allow us to treat our problem as astratified randomized experiment.

• Ad-specific features: Our experimental comparison is made on the same focal ad at time twhen exposed to different variety of ads in the previous sequence of exposures. As such, it isimportant to control for the ad-specific information at exposure number t, especially because theallocation of ads is through the same auction that decides on the assignments to variety. Thus,failing to control for such ad-specific effects would give rise to confounding factors.

We capture two types of information about the ad Ai,t shown during an exposure – 1) a fixedeffect of the ad that captures inherent differences in the effectiveness of ads, i.e., some ads maybe better made, come from more reputable firms etc., and therefore have a naturally higherlikelihood of getting clicks, and 2) a time-varying component which relates to the state of Ai,t

within the session. In this time-varying component, we control for the number of previousexposures of Ai,t within the session (which is not captured by the ad fixed effects) because theseexposures clearly affects user’s decision to click on Ai,t. We denote this variable as Ii,t anddefine it as follows:

Ii,t =t−1∑j=1

1(Ai,j = Ai,t) (6)

Figure 13 illuminates what comparisons we wish to make by including these two ad components.In Figure 13a, we want to compare the first two cases where the focal ad is the same. Comparisonof the first and third case is not of our interest, because ad shares are different (see Figure 5).Including ad fixed effects allows us to control for the ad that is currently being shown. Thecase in Figure 13b is more subtle where we do not want to include the third case in the samebucket with the first two cases, because the number of previous exposures of the focal ad isdifferent in this case. This goal is entailed by controlling for number of previous exposures ofthe focal ad (Ii,t). It is important to notice that while assignment to the sequence of previous adsis independent of Ai,t, it is not independent of Ii,t. For example, larger ads are more likely to

25

A B C D X

A B A A X

A B A A Y

×

A X C D X

A X A A X

A B A A X

×

(a) Controlling for ad fixed effects

A B C D X

A B A A X

A B A A Y

×

A X C D X

A X A A X

A B A A X

×

(b) Controlling for previous exposures of the focal ad

Figure 13: Controls for ad-specific features and their experimental equivalents

have prior exposures thereby being assigned to a lower variety. Thus, failing to control for thisvariable would result in omitted variable bias.

• Exposure number: An important point of note is that the exposure number or t is an importantfactor that influences click and leave outcomes (recall the decreasing patterns shown in Figure4) as well as the independent variables (e.g., variety and other session-level features). Thus, ourmodeling exercise needs to correctly account for exposure number.

We adopt the approach of specifying and estimating a separate model for each exposure number,instead of controlling for the exposure number fixed effects within one global equation. Thereare two reasons for this. First, the nature and interpretation of variety and other session-level temporal features is different across exposure numbers (t). For instance, a variety oftwo at the 3rd exposure is naturally different from the same variety at the the 10th exposure.Thus, interpreting the results can be challenging if we pool the data. Second, the impact anddistributions of variety at each exposure number are likely to be different. So if we want tounderstand the impact of variety at each exposure number separately, we have to interact all thesession-level features (including variety) with the exposure number, which is tedious and messy.Therefore, we estimate a separate model for each exposure number t.

• Treatment variable: Our goal is to estimate the effects of variety of previous ads on user’sclicking behavior on the current ad. While different measures of variety are defined in §4.2,we simply refer to variety of previous ads in session i at exposure number t by Vi,t. It is worthnoting that we can use any measure of variety for Vi,t.

Incorporating all four pieces listed above, we present the main specification of our model as follows:

Yi,t = βtVi,t + γtIi,t + ηa,t + ζg,t + εi,t, (7)

26

where ηa,t and ζg,t are respectively ad and auction-invariant strata fixed effects for exposure numbert, and εi,t is the error term. Our main goal is to estimate βt for any exposure number. We run thisspecification for both the survived sample and intended sample to estimate both treatment effect onthe survived and intent to treat estimates.

7 ResultsIn this section, we first present the main results from the model shown in Equation (7) and thencharacterize the heterogeneity in the effects of variety across user-level observables.

7.1 Effects of Variety on the Survived Sample

We start with estimating the effects of variety on clicks using the survived sample. For eachexposure number t, we use one measure of variety as Vi,t in Equation (7) at time to estimate themodel. Overall, we estimate 7× 4 = 28 models the results of which are shown in Table 2. Withno exception, all estimates for the coefficient of variety indicate a positive association betweenvariety of previous ads and clicking on the next ad. Further, the estimates are close in magnitude,except for exposure number 5 which has the highest coefficient of variety. To have a sense of howeconomically significant the estimates are, we present the average CTR for each exposure number inTable 2. Comparing our estimates with average CTRs, we can simply calculate that one incrementin variety amounts to 5-10% of the average CTR.

Besides variety of previous ads, we find that the number of previous exposures of the currentad also leads to higher likelihood of click. That is, the more an ad has been shown within thesession, the more likely the user is to click on that ad. It reinforces the prior findings of the literatureregarding the positive effects of repeated exposure (Sahni, 2015a; Johnson et al., 2016b). It alsogives rise to an inter-temporal trade-off in ad placement. On the one hand, repeated exposures ofthe same ad increases the user’s likelihood of clicking on it. On the other hand, showing the samead repeatedly reduces variety. Thus, it suggests that employing a mix of both would be optimal inplacement of ads.

7.1.1 How Reasonable is the Assumption of Exogenous App Usage?

The estimates in Table 2 are obtained using the sample of users who have survived to receive thefull treatment. However, as shown in §3.3.2, not all users comply to receive the full treatment. Ifcompliance is influenced by prior assignment to variety, the estimated effects of variety on userswho have survived to receive the treatment would not reveal the effects of variety on users who areintended to receive the treatment. In this section, we test the assumption of exogenous app usage,i.e., the user’s decision to leave the session is not affected by their assignments within the session,

27

Dependent variable: Click (Yi,t)

(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t 0.00151∗∗ 0.00242∗∗∗ 0.00152∗∗∗ 0.00118∗ 0.00157∗∗∗ 0.00140∗∗ 0.00156∗∗∗

(2.99) (4.82) (3.34) (2.51) (3.43) (2.95) (3.34)Previous 0.00138∗∗ 0.00264∗∗∗ 0.00154∗∗∗ 0.00158∗∗∗ 0.00095∗ 0.00128∗∗ 0.00150∗∗∗

Exposure (Ii,t) (3.17) (5.82) (3.68) (3.68) (2.28) (2.99) (3.61)R2 0.229 0.258 0.296 0.316 0.337 0.354 0.375Adjusted R2 0.023 0.020 0.035 0.027 0.024 0.015 0.013

B. Consecutive ChangesChangei,t 0.00133∗∗ 0.00239∗∗∗ 0.00163∗∗∗ 0.00150∗∗∗ 0.00170∗∗∗ 0.00209∗∗∗ 0.00166∗∗∗

(2.68) (4.93) (3.74) (3.41) (4.01) (4.82) (3.94)Previous 0.00133∗∗ 0.00259∗∗∗ 0.00154∗∗∗ 0.00164∗∗∗ 0.00096∗ 0.00144∗∗∗ 0.00153∗∗∗


C. Gini-Simpson Index for DiversityGiniSimpsoni,t 0.00302∗∗ 0.00794∗∗∗ 0.00874∗∗∗ 0.00645∗ 0.01158∗∗∗ 0.01421∗∗∗ 0.01724∗∗∗

(2.62) (4.34) (3.84) (2.18) (3.31) (3.37) (3.63)Previous 0.00136∗∗ 0.00264∗∗∗ 0.00166∗∗∗ 0.00159∗∗∗ 0.00102∗ 0.00143∗∗ 0.00166∗∗∗


Ad FE X X X X X X XStrata FE X X X X X X XAvg. CTR 0.0234 0.0260 0.0227 0.0233 0.0215 0.0221 0.0203No. of Obs. 365,166 285,373 234,109 194,485 165,558 142,878 124,864

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table 2: OLS Estimates of the Effects of Variety on Clicks Using the Survived Sample UsingDifferent Measures of Variety

particularly variety. To do so, we define leave variable indicating whether the user has left after thatexposure. We then substitute click with leave as the outcome variable in (7) and estimate the modelssimilar to Table 2. For brevity, the results are presented in Table A1 in the Appendix. We find thatat some exposure numbers, there is a statistically significant link between variety and user’s leavedecision, calling for the rejection of the assumption of exogenous app usage. Thus, we need toconstruct the full sample with the intended treatment assignment.

28

7.2 Effects of Variety on the Intended Sample

While the results of Table 2 suggest that variety of previous ads will lead to a higher CTR on thenext ad, it is not clear whether the intention to treat users with higher variety of previous ads wouldalso increase the likelihood of click. In fact, if the number of people who click in each intendedgroup is the same (i.e., no significant difference), but people tend to leave more when assigned to ahigher variety, our estimates on the survived sample may indicate a positive effect of variety, sincethe survived sample is smaller for this group. However, this can mislead managers who want todecide on ad placement strategies. To address this dynamic selection issue, we first construct thesample involving the intended treatments for all the sessions, including those who have left beforereceiving the full treatment. Our approach to construct such sample is presented in §5.2.2. We thenuse the specification in Equation (7) and obtain intent-to-treat estimates on the intended sample.The results are shown in Table 3. Again, we find positive effects of variety and previous exposuresof the focal ad. However, as expected, intent-to-treat estimates are smaller in magnitude.

7.2.1 Discussion on the Difference Between the Estimates on the Survived vs. IntendedSample

As discussed earlier, if compliance is influenced by the treatment assignment, the estimates obtainedon the survived sample would not reveal those on the intended sample. However, if compliance iscompletely at random, we can treat the estimates on the survived sample as average treatment effect

on the treated (ATT) and directly obtain intent-to-treat (ITT) estimates. We can then compare suchITT estimates with those presented in Table 3 to see how biased our ITT estimates will be under theassumption of random compliance.

If compliance is completely at random, we can write the following equation for the ITT estimatesat exposure number t (βITT

t ):

βITTt = πco

t βITT,cot + (1− πco

t )βITT,nct , (8)

where βITT,cot and βITT,nc

t are ITT estimates on the sample of compliers vs. non-compliers respectively,and πco

t denotes the share of compliers for exposure number t. The ITT estimates on the sampleof compliers are actually what we estimated in Table 2. On the other hand, since the outcome iszero for any non-treated user regardless of their intended treatment assignment, we have βITT,nc

t = 0.Therefore, we can obtain ITT estimates using the results in Table 2 as follows:

βITTt = πco

t βITT,cot =

Nt

N1

βsurvivedt , (9)

29


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t 0.00066∗∗∗ 0.00078∗∗∗ 0.00045∗∗∗ 0.00024∗∗ 0.00028∗∗∗ 0.00028∗∗∗ 0.00024∗∗∗

(3.67) (5.59) (4.28) (2.71) (3.74) (4.20) (4.21)Previous 0.00042∗∗ 0.00073∗∗∗ 0.00038∗∗∗ 0.00032∗∗∗ 0.00015∗ 0.00021∗∗∗ 0.00021∗∗∗


B. Consecutive ChangesChangei,t 0.00056∗∗ 0.00079∗∗∗ 0.00043∗∗∗ 0.00027∗∗ 0.00032∗∗∗ 0.00037∗∗∗ 0.00024∗∗∗

(3.18) (5.83) (4.29) (3.19) (4.58) (5.94) (4.57)Previous 0.00040∗∗ 0.00072∗∗∗ 0.00037∗∗∗ 0.00033∗∗∗ 0.00016∗ 0.00023∗∗∗ 0.00021∗∗∗


C. Gini-Simpson Index for DiversityGiniSimpsoni,t 0.00133∗∗ 0.00261∗∗∗ 0.00245∗∗∗ 0.00125∗ 0.00226∗∗∗ 0.00285∗∗∗ 0.00283∗∗∗

(3.23) (5.09) (4.64) (2.18) (3.87) (4.66) (4.68)Previous 0.00041∗∗ 0.00073∗∗∗ 0.00041∗∗∗ 0.00032∗∗∗ 0.00017∗ 0.00024∗∗∗ 0.00024∗∗∗


Ad FE X X X X X X XStrata FE X X X X X X XNo. of Obs. 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table 3: OLS Estimates of the Effects of Variety on Clicks Using the Intended Sample UsingDifferent Measures of Variety (Intent-to-Treat)

where Nt and N1 are the number of sessions available at exposure number t and 1 respectively, andβsurvivedt is simply the estimates on the survived sample. As such, βITT

t gives us the ITT estimatesunder the random compliance assumption. We compare these estimates with those obtained usingthe intended sample (Table 3) to see how random compliance assumption would bias our ITTestimates. As shown in Figure 14, the actual ITT estimates are higher than those under the randomcompliance assumption. This is in line with the results presented in Table A1 that show both varietyand previous exposures of the focal ad have negative effects on user’s decision to leave, at someexposure numbers. This lower compliance rate for higher variety conditions explains why the

30

ITT estimates under random compliance is biased downward. Since compliance is not completelyrandom, we use our intended sample for the rest of models we will run throughout this paper.

●

●

●

●

● ●

●

●

●

●

●

●

● ●

0e+00

2e−04

4e−04

6e−04

8e−04


Coe

ffici

ent o

f Var

iety

Group ● ●Actual ITT Estimates ITT Estimates Under Random Compliance Assumption

Figure 14: Comparison of ITT estimates using the intended sample vs. ITT estimates under randomcompliance assumption

7.3 What is the Source for the Effects of Variety?

Our findings in §7.1 and §7.2 establish the main effects of variety of previous ads on the likelihoodof clicking on the next ad. We now want to explore the sources for the effects of variety. Sincevariety has multiple definitions, we can exploit the differences between the three measures we havedefined for variety to see which aspects of variety most contribute to the clicks.

7.3.1 Variety-Increasing vs. Variety-Constant Changes

As shown in Table 11, both breadth of variety (Varietyi,t) and total number of consecutive changes(Changei,t) have positive effects on clicks. Obviously, the first time we show a distinct ad in thesequence, we need to change the last ad (if any). Hence, the total number of such consecutivechanges equals the breadth of variety minus one, because we do not count the first exposure numberin total number of changes. As such, we can decompose the total number of consecutive changes

31

into two parts – variety-increasing changes and variety-constant changes. The former is the totalnumber of consecutive changes that increase the variety of the sequence (i.e., the second ad inthe pair has not been shown before within the sequence), whereas the latter is the total number ofconsecutive changes that do not increase the variety (i.e., the second in the pair has been shownbefore within the sequence). Let VICi,t and VCCi,t denote the total number of variety-increasing

changes and variety-constant changes respectively. We can write:

Changei,t = VICi,t + VCCi,t = (Varietyi,t − 1) + VCCi,t (10)

We can now include both VICi,t and VCCi,t in Equation (7) as Vi,t to see which one contributesmore to the outcome – the actual variety/distinctiveness or consecutive changing of previous ads.As shown in Table 4, we find support for both sources. The point estimates for variety-increasingchanges are higher than those of variety-constant changes, indicating that the distinctiveness ofprevious ads is a source for the effects. However, the estimates of variety-constant changes arealso significant for some exposure numbers, suggesting that a large fraction of the effects of varietystems just from the consecutive changing of ads. Further, the estimates of variety-constant changesbecome significant from exposure number eight onward. The fact that the gap between the estimatesof variety-increasing and variety-constant changes shrink in later exposure numbers might be due todiminishing returns to variety. To summarize, while we find the positive effects of distinctiveness ofprevious ads in all exposures, a large fraction of the effects of variety is attributed to consecutivechanging of ads, especially in later exposures.

7.3.2 Consecutive vs. Non-Consecutive Diversity

One of our measures for variety is the Gini-Simpson index for diversity. This measure calculatesthe probability that two random exposures from a sequence show different ads. Now we want todistinguish between the diversity caused by consecutive exposures as opposed to non-consecutiveexposures. Recall the definition of Gini-Simpson index:

GiniSimpsoni,t =

∑s<s′<t 1(Ai,s′ 6= Ai,s)

(t−1)(t−2)2

, (11)

where the numerator is the total number of times that two random exposures show different ads. Wecan now decompose the numerator into two parts:∑

s<s′<t

1(Ai,s′ 6= Ai,s) =∑

s+1=s′<t

1(Ai,s′ 6= Ai,s) +∑

s+1<s′<t

1(Ai,s′ 6= Ai,s), (12)

32


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

Variety-Increasing 0.00067∗∗∗ 0.00091∗∗∗ 0.00053∗∗∗ 0.00031∗∗ 0.00038∗∗∗ 0.00041∗∗∗ 0.00032∗∗∗

Changes (3.61) (6.12) (4.68) (3.23) (4.60) (5.61) (4.97)Variety-Constant 0.00009 0.00051∗ 0.00026 0.00021 0.00026∗∗ 0.00033∗∗∗ 0.00017∗∗

Changes (0.29) (2.54) (1.89) (1.92) (2.95) (4.33) (2.74)Previous 0.00042∗∗ 0.00075∗∗∗ 0.00040∗∗∗ 0.00034∗∗∗ 0.00017∗ 0.00024∗∗∗ 0.00023∗∗∗

Exposure (Ii,t) (2.75) (5.89) (4.07) (4.05) (2.50) (3.92) (4.29)

Ad FE X X X X X X XStrata FE X X X X X X XNo. of Obs. 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426R2 0.099 0.088 0.087 0.077 0.071 0.063 0.061Adjusted R2 0.028 0.024 0.029 0.024 0.022 0.017 0.019

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table 4: OLS Estimates of the Effects of Variety-Increasing vs. Variety-Constant Changes Usingthe Intended Sample

where the first term it the total number of consecutive exposures that show different ads, whereas thesecond term is the total number of non-consecutive exposures that show different ads. Interestingly,the former is our Changei,t variable. Dividing both sides of (12) by (t−1)(t−2)

2, we can decompose

Gini-Simpson index into two parts – consecutive diversity and non-consecutive diversity denoted byCDi,t and NCDi,t respectively. We can write:

CDi,t =

∑s+1=s′<t 1(Ai,s′ 6= Ai,s)

(t−1)(t−2)2

NCDi,t =

∑s+1<s′<t 1(Ai,s′ 6= Ai,s)

(t−1)(t−2)2

GiniSimpsoni,t = CDi,t + NCDi,t

We include both consecutive and non-consecutive diversity in the main specification and estimatethe model. It helps us disentangle whether the effects come from consecutive diversity as comparedto non-consecutive diversity. As shown in Table 5, we find that consecutive diversity plays a moreimportant role in driving users’ clicks. However, for exposure number 7 and 10, we find that thecoefficients for non-consecutive diversity are also significant. Overall, our results suggest thatthe sequential organization of exposures is critical since there is a difference between consecutiveand non-consecutive diversity. This is similar to the findings of Hoch et al. (1999) and Kahn

33

and Wansink (2004) who find that the organization and structure of the assortment affect users’perceptions of variety.


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

Consecutive 0.00160∗∗ 0.00459∗∗∗ 0.00354∗∗∗ 0.00422∗∗ 0.00621∗∗∗ 0.00984∗∗∗ 0.00662∗∗

Diversity (3.00) (5.43) (3.30) (3.04) (3.79) (4.99) (3.02)Non-Consecutive 0.00079 0.00063 0.00173∗ -0.00023 0.00068 0.00054 0.00175∗

Diversity (1.00) (0.75) (2.12) (-0.27) (0.80) (0.62) (2.06)Previous 0.00042∗∗ 0.00073∗∗∗ 0.00041∗∗∗ 0.00032∗∗∗ 0.00017∗ 0.00024∗∗∗ 0.00024∗∗∗

Exposure (Ii,t) (2.68) (5.74) (4.18) (3.82) (2.44) (3.84) (4.39)


Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table 5: OLS Estimates of the Effects of Consecutive vs. Non-Consecutive Diversity Using theIntended Sample

7.3.3 Developing a New Measure for Sequential Variety

As highlighted in §7.3.1 and §7.3.2, the sequential organization of the exposures plays an importantrole in driving the effects of variety. It motivates us to introduce a new measure for sequentialvariety, i.e., objects that are presented in sequential manner. In light of our findings in Table 5, weknow that the effects of diversity are mostly driven by the diversity in exposures that are temporallycloser. Hence, we incorporate the temporal distance of exposures in our definition of sequentialvariety. Building on the Gini-Simpson measure of diversity, we can define the general form of ourmeasure of sequential variety as follows:

SeqVarietyi,t =∑

s<s′<t ω(s, s′)1(Ai,s′ 6= Ai,s)

(t−1)(t−2)2

, (13)

where ω(s, s′) is a function that assigns weights to pairs of exposures. Since we show that consecu-tive diversity is more important, ω(s, s′) is generally decreasing in the temporal distance between sand s′. That is, we assign higher weight to diversity caused by exposures that are temporally closer.The choice of function ω, however, depends on the research purpose. We use two functional formsω(s, s′) = 1

2s′−s−1 and ω(s, s′) = 1

s′−s−1 for our measure of sequential variety and estimate our

34

model. As shown in Table A2, both result in significant coefficients for the measure of sequentialvariety.

7.4 Heterogeneity Across User History

While we exclusively focus on new users, their paths can evolve very differently based on thesequence of ads they are assigned (both within and across sessions) as well as their own usage, clickdecisions. Thus, at any given exposure number within a strata, we see significant heterogeneity inuser history. For example, some users may have seen more impressions in the past than others. Sincethe effects of variety and other session-level temporal features are in-the-moment, we expect theseeffects to be heterogeneous across historical user features. For instance, users with longer historymay react differently to manipulation strategies compared to those who are new. This motivates usto explore heterogeneity in the effects of variety across different types of users.

We take a simple descriptive approach to show heterogeneity in the effects of variety. Thisinvolves splitting our data into two parts based on users’ historical features (e.g., users’ past activity).We then obtain estimates for the breadth of variety (Vi,t) on both splits of the data for all exposurenumbers. Since historical features do not affect ad allocation, such estimates are consistent. Belowwe present three dimensions of user history we are interested in and determine the splits of the dataaccordingly:

• Length of history: Users in our sample vary by the number of impressions they have seen inthe past. In general, we expect this variable to be a major source of heterogeneity, as it shapesuser’s behavior towards ads. We call a user tenure if he/she has seen over 50 impressions. Wesplit the data into two parts each containing impressions shown to tenured vs. non-tenuredusers. The effects on the sample of tenured and non-tenured users are shown in Figure 15aand 15b respectively. While the effects are significant both economically and statistically fornon-tenured users, tenured users seem to be unaffected by variety interventions. This may bebecause tenured users are less susceptible to just-in-time manipulations such as variety.

• Clicking history: Users’ clicking record is usually a good indicator of their interest in ads. Wecall a user responsive if he/she has at least clicked on one ad in his/her record. We estimate ourmodel on the sample of responsive vs. non-responsive users and present the results in Figure 15cand 15d respectively. The patterns reveal a stark contrast: while variety of previous ads leads toa higher CTR on the sample of responsive users, the effects flip on the non-responsive sample.This can stem from the fundamental difference between these two groups: the responsive usersare more interested in ads, thereby variety increases their engagement with ads. However,non-responsive users who have not shown any interest in ads seem not to enjoy higher varietyof ads.

35

● ● ●●

● ● ●0.000

0.001

0.002

0.003


Coe

ffici

ent o

f Var

iety

(a) Tenured

●

●

●● ● ●

●

0.000

0.001

0.002

0.003


Coe

ffici

ent o

f Var

iety

(b) Non-tenured

●●

●

● ● ● ●

−0.001

0.000

0.001

0.002


Coe

ffici

ent o

f Var

iety

(c) Responsive

● ●

●● ●

●●

−0.001

0.000

0.001

0.002


Coe

ffici

ent o

f Var

iety

(d) Non-responsive

●

●

●

●

●● ●0.0000

0.0005

0.0010

0.0015


Coe

ffici

ent o

f Var

iety

(e) High Repetition

●

●

● ● ● ●●

0.0000

0.0005

0.0010

0.0015


Coe

ffici

ent o

f Var

iety

(f) Low Repetition

Figure 15: Heterogeneity in the effects of variety of previous ads

36

• Past interaction with the focal ad: Another historical feature of an impression that can bean important source of heterogeneity is the number of previous exposures of the focal adthat the user has seen prior to the session. Clearly, the number of previous exposures of thecurrent ad affects user’s information processing of the ad. Thus, the effects of variety are likelyheterogeneous conditional on the previous exposures of the current ad. We split the data into twoparts based on the previous exposures of the current ad. We call the condition high repetition ifthe user has seen over 5 exposures prior to the session and split the data based on this condition.As shown in Figure 15e and 15f, we find that the effects of variety are stronger if the user hasseen fewer exposures of the current ad prior to the session. However, variety of prior ads is lesseffective if the current ad has been shown over 5 times before.

7.5 Robustness Checks

In this section, we present some robustness check to examine whether our findings hold underdifferent specifications. We outline different checks below:

• Temporal Spacing: One confounding factor for temporal features can be temporal spacing.Sahni (2015a) finds that temporal spacing between the current ad and the last time the ad wasshown has a positive effect on the clicking outcome. One would argue that omitting this variablewould result in biased estimates for variety, as higher spacing might be correlated with highervariety. To control for this potential confounding factor, we create the variable spacei,t, which isthe number of impressions the user has seen in session i, since she last saw ad Ai,t. Obviously,spacei,t is not defined if this is the first time showing Ai,t within the session. We run twodifferent models to address this issue. First, we add the variable spacei,t to the specificationpresented in Equation (7) and estimate the model. However, this might reduce our power aswe drop many data points. This brings us to our second strategy where we define the dummyvariable for first exposure of an ad. The result of these models are presented in Table A3 andA4 in Appendix. By and large, the coefficient for variety is positive and significant, implyingpositive effects of variety on click. Further, we find positive effects for temporal spacing in mostexposures, which confirms the findings of Sahni (2015a).

• Other Measures of Variety: Besides measures that we used for variety, Shannon entropy isanother metric for dispersion and diversity which is widely used in information theory literature(Shannon, 1948).6 Below is a formal definition of this metric:

Shannoni,t = −∑a

pi,a,t log2 pi,a,t, (14)

6For a marketing use and discussion of this metric, please see Godes and Mayzlin (2004).

37

where pi,a,t is the share of ad a in the session i prior to exposure number t, i.e., pi,a,t =Ii,a,tt−1 .

This measure captures the amount of information in the past sequence.7 We use this measure ofvariety as Vi,t in (7) and estimate the model on the intended sample. As presented in Table A5,our estimates for the effects of variety are robust to Shannon measure of variety.

• Different Stratification: In our main model, we used strata fixed effects to control for cross-sectional selection. Each stratum is a combination of a targeting area g and hour h. Clearly,such strata can be very narrow containing only a few corresponding observations. While it helpsus avoid the cross-sectional selection bias, it may reduce our power of identifying the effects.On the other hand, one would argue that the current stratification may not fully eliminate theselection bias, as the time intervals can be wide and we pool all the ads and control for ad effectsseparately. We conduct a series of different specifications with more or less fixed effects. Theresults are shown in the appendix. Table A6, A7, and A8 present the estimates when usingless fixed effects. By and large, the substantive results remain the same, though the effectsare underestimated compared to the main results. Such discrepancy suggests that advertiserstake into account the interactions between targeting variables when targeting their ads, therebyinducing bias in the models that do not capture such interactions.

Further, we estimate models with even narrower strata. As discussed in §5.1, to control fortime-dependent factors that change the auction, we used hourly intervals. Here we use half-an-hour intervals to see whether the results are robust to tighter time intervals. As shown inTable A9, the results are almost the same as our main model. Finally, we incorporate ad fixedeffects in our stratification and estimate the model. That is, for any auction-invariant stratumand ad, we define a separate group and control for such group fixed effects. The results, aspresented in Table A10, show the same pattern as our main model. However, for later exposuresthe coefficients lose significance which is likely due to lack of power.

• Top Ad: In our main model, we pool all ads together and control for their fixed and time-varyingeffects. An alternative approach is to run models for each ad separately to show the effects oneach individual ad and explore the heterogeneity in the effects across ads. However, we do nothave enough power for each ad to identify the effects. Here we estimate the model on the top adfor which we have enough observations. As show in Table A11, we see the same pattern as ourmain model.

• Controlling for User’s Panel Length: In our main analysis, we take session as the unit of analysis.However, we may observe multiple sessions for only one users. As a result, our main resultsmay be driven by some users for whom we see many sessions. Further, one would argue that

7More practically, it translates into number of bits required to store the sequence.

38

user’s behavior may evolve over time with respect to session-level temporal features, which inturn, violates Stable Unit Treatment Value Assumption (SUTVA) (Cox, 1958; Rubin, 1980). Toaddress this issue, we only focus on user’s first assignment to an experimental condition. Thatis, for any exposure number t, we focus on the first time users get to see that specific exposurenumber and estimate our model. This changes our unit of analysis from sessions to users. Theonly problem with that approach is the significant loss in the power of our analysis, as we haveto drop many sessions. We estimate our model using this approach and present the results inTable A12. The estimates of the effects of variety show the same patterns as our main results.

8 Conclusion and Future DirectionsMobile in-app advertising is now a major source of revenue for many app developers. Given somespecific features of this medium of advertising, sequential ad placement is a common practice amongpublishers who serve ads. That is, users are exposed to a sequence of potentially different ads withina session. This motivates a series of questions related to the sequencing of ads. We particularly focuson variety as an important feature of ad sequences and study how variety of previous ads is linkedto user’s clicking behavior on the next ad. To answer this question, we use data from the leadingin-app ad-network of an Asian country to examine this question. A unique feature of our data isthe use of probabilistic auction for ad placement that creates a great degree of randomness in thesequence of ads users are exposed to within the session. We develop an identification strategy thatallows us to exploit the exogenous variation in users’ assignment to variety and obtain intent-to-treatestimates. We find that variety of previous ads leads to higher likelihood of click on the next ad. Wethen examine the source for such effects using different measures of variety and identify sequentialorganization of exposures as a major source. We develop a general measure for sequential varietythat captures perceptions of variety when objects are presented in a sequence. Finally, we use adescriptive approach to document heterogeneity in variety effects across user’s past history.

In sum, our paper contributes to two broad streams of literature on advertising and variety. First,our paper adds to the methodological literature on quantifying causal effects of advertising. Wepropose an identification strategy that can be applied to cases where ads are allocated through aprobabilistic mechanism. Specifically, our approach is useful for identification of temporal effectsof advertising using observational data, where users may drop out before fully assigned to thetreatment. Second, our paper adds to the marketing literature on variety by establishing the role ofvariety in advertising context. We also provide a framework for decomposing the effects of varietyand develop a measure of sequential variety that is a generalization of entropy measure introducedby Simpson (1949a) to cases where objects are presented dynamically. Our findings provide somemanagerial insights for both advertisers and publisher in mobile in-app advertising industry. For

39

advertisers, our findings wouldOur findings have managerial implications for both advertisers and publishers. One implication

for advertisers is to participate in auctions with many competitors to be assigned to higher varietyconditions. Further, our results on heterogeneity in variety effects would guide advertisers’ targetingdecisions. For publishers, our results highlight the importance of questions related to mechanismdesign – whether they must use deterministic vs. probabilistic auctions and how they should designthe sequence of ads.

Nevertheless, there are certain aspects of the problem that our paper overlooks but can serve asavenues of future research. First, we only focus on click as the main outcome of interest. However,click is not the ultimate outcome that advertisers care about. Examining whether this increment inclicks will lead to higher conversion would be an interesting area for future research. Second, whileour results establish the causal effects of variety, it is not clear what the optimal policy would be.Increasing variety may come with the cost of showing irrelevant ads giving rise to an inter-temporaltrade-off in sequencing of ads. Future research can explore how the publisher can optimally designthe sequence of ads. Third, we show that assignment to variety affects user’s decision to stay inthe app. Questions related to the effects of advertising interventions on app usage are particularlyimportant yet understudied in the literature. Thus, such question can open fruitful avenues for futureresearch.

40

ReferencesAppAnnie. In-App Advertising Spend to Triple, Reach $201 Billion by 2021, 2017. URL https://www.appannie.com/en/insights/market-data/app-advertising-spend-2021/.

T. Blake, C. Nosko, and S. Tadelis. Consumer heterogeneity and paid search effectiveness: A large-scalefield experiment. Econometrica, 83(1):155–174, 2015.

D. R. Cox. Planning of experiments. 1958.H. Datta, G. Knox, and B. J. Bronnenberg. Changing their tune: How consumers’ adoption of online

streaming affects music consumption and discovery. Marketing Science, 37(1):5–21, 2017.eMarketer. Worldwide Internet and Mobile Users: eMarketer’s Estimates for 2016–2021 , 2017a. URLhttp://totalaccess.emarketer.com/reports/viewer.aspx?r=2002038.

eMarketer. US Time Spent with Media: eMarketer’s Updated Estimates and Forecast for 2014–2019 , 2017b.URL http://totalaccess.emarketer.com/reports/viewer.aspx?r=2002021.

D. Godes and D. Mayzlin. Using online conversations to study word-of-mouth communication. Marketingscience, 23(4):545–560, 2004.

A. Goldfarb and C. E. Tucker. Privacy regulation and online advertising. Management science, 57(1):57–71,2011.

B. R. Gordon, F. Zettelmeyer, N. Bhargava, and D. Chapsky. A comparison of approaches to advertisingmeasurement: Evidence from big field experiments at facebook. 2017.

D. A. Harrison and K. J. Klein. What’s the difference? diversity constructs as separation, variety, or disparityin organizations. Academy of management review, 32(4):1199–1228, 2007.

M. A. Hernan, E. Lanoy, D. Costagliola, and J. M. Robins. Comparison of dynamic treatment regimes viainverse probability weighting. Basic & clinical pharmacology & toxicology, 98(3):237–242, 2006.

S. J. Hoch, E. T. Bradlow, and B. Wansink. The variety of an assortment. Marketing Science, 18(4):527–546,1999.

J. W. Hogan, J. Roy, and C. Korkontzelou. Handling drop-out in longitudinal studies. Statistics in medicine,23(9):1455–1497, 2004.

R. Hollander. Apps Are Becoming Extremely Reliant on In-App Ads, 2017. URL http://www.businessinsider.com/apps-are-becoming-extremely-reliant-on-in-app-ads-2017-8.

K. Imai and D. A. Van Dyk. Causal inference with general treatment regimes: Generalizing the propensityscore. Journal of the American Statistical Association, 99(467):854–866, 2004.

G. W. Imbens and D. B. Rubin. Causal inference in statistics, social, and biomedical sciences. CambridgeUniversity Press, 2015.

J. Jacoby. Information load and decision quality: Some contested issues. Journal of Marketing Research,pages 569–573, 1977.

J. Jacoby. Perspectives on information overload. Journal of consumer research, 10(4):432–435, 1984.P. Jeziorski and S. Moorthy. Advertiser prominence effects in search advertising. Management Science, 2017.G. A. Johnson, R. A. Lewis, and E. I. Nubbemeyer. The online display ad effectiveness funnel & carryover:

A meta-study of predicted ghost ad experiments. 2016a.G. A. Johnson, R. A. Lewis, and D. Reiley. Location, location, location: repetition and proximity increase

advertising effectiveness. 2016b.G. A. Johnson, R. A. Lewis, and D. H. Reiley. When less is more: Data and power in advertising experiments.

Marketing Science, 36(1):43–53, 2016c.G. A. Johnson, R. A. Lewis, and E. I. Nubbemeyer. Ghost ads: Improving the economics of measuring online

ad effectiveness. Journal of Marketing Research, 54(6):867–884, 2017.

41

https://www.appannie.com/en/insights/market-data/app-advertising-spend-2021/

https://www.appannie.com/en/insights/market-data/app-advertising-spend-2021/

http://totalaccess.emarketer.com/reports/viewer.aspx?r=2002038

http://totalaccess.emarketer.com/reports/viewer.aspx?r=2002021

http://www.businessinsider.com/apps-are-becoming-extremely-reliant-on-in-app-ads-2017-8

http://www.businessinsider.com/apps-are-becoming-extremely-reliant-on-in-app-ads-2017-8

B. E. Kahn and B. Wansink. The influence of assortment structure on perceived variety and consumptionquantities. Journal of consumer research, 30(4):519–533, 2004.

J. Kim, G. M. Allenby, and P. E. Rossi. Modeling consumer demand for variety. Marketing Science, 21(3):229–250, 2002.

R. Lewis, J. M. Rao, and D. H. Reiley. Measuring the effects of advertising: The digital frontier. In EconomicAnalysis of the Digital Economy, pages 191–218. University of Chicago Press, 2015.

R. A. Lewis and J. M. Rao. The unfavorable economics of measuring the returns to advertising. The QuarterlyJournal of Economics, 130(4):1941–1973, 2015.

H. Li and P. Kannan. Attributing conversions in a multichannel online marketing environment: An empiricalmodel and a field experiment. Journal of Marketing Research, 51(1):40–56, 2014.

H. Li, P. Kannan, S. Viswanathan, and A. Pani. Attribution strategies and return on keyword investment inpaid search advertising. Marketing Science, 35(6):831–848, 2016.

L. McAlister. A dynamic attribute satiation model of variety-seeking behavior. Journal of Consumer Research,9(2):141–150, 1982.

V. Mirrokni, S. Muthukrishnan, and U. Nadav. Quasi-Proportional Mechanisms: Prior-Free RevenueMaximization. In Latin American Symposium on Theoretical Informatics, pages 565–576. Springer, 2010a.

V. Mirrokni, S. Muthukrishnan, and U. Nadav. Quasi-proportional mechanisms: Prior-free revenue maxi-mization. In Latin American Symposium on Theoretical Informatics, pages 565–576. Springer, 2010b.

S. Narayanan and K. Kalyanam. Position effects in search advertising and their moderators: A regressiondiscontinuity approach. Marketing Science, 34(3):388–407, 2015.

O. Rafieian and H. Yoganarasimhan. Targeting and privacy in mobile advertising. Available at SSRN 3163806,2018.

R. K. Ratner, B. E. Kahn, and D. Kahneman. Choosing less-preferred experiences for the sake of variety.Journal of consumer research, 26(1):1–15, 1999.

J. P. Redden. Reducing satiation: The role of categorization level. Journal of Consumer Research, 34(5):624–634, 2008.

P. R. Rosenbaum and D. B. Rubin. The central role of the propensity score in observational studies for causaleffects. Biometrika, 70(1):41–55, 1983.

A. Rotnitzky, J. M. Robins, and D. O. Scharfstein. Semiparametric regression for repeated outcomes withnonignorable nonresponse. Journal of the american statistical association, 93(444):1321–1339, 1998.

D. B. Rubin. Randomization analysis of experimental data: The fisher randomization test comment. Journalof the American Statistical Association, 75(371):591–593, 1980.

O. J. Rutz and R. E. Bucklin. From generic to branded: A model of spillover in paid search advertising.Journal of Marketing Research, 48(1):87–102, 2011.

N. S. Sahni. Effect of temporal spacing between advertising exposures: evidence from online field experiments.Quantitative Marketing and Economics, 13(3):203–247, 2015a.

N. S. Sahni. Effect of temporal spacing between advertising exposures: Evidence from online field experi-ments. Quantitative Marketing and Economics, 13(3):203–247, 2015b.

N. S. Sahni. Advertising spillovers: Evidence from online field experiments and implications for returns onadvertising. Journal of Marketing Research, 53(4):459–478, 2016.

D. W. Schumann, R. E. Petty, and D. Scott Clemons. Predicting the effectiveness of different strategies ofadvertising variation: A test of the repetition-variation hypotheses. Journal of Consumer Research, 17(2):192–202, 1990.

C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423,July 1948. ISSN 0005-8580. doi: 10.1002/j.1538-7305.1948.tb01338.x.

42

B. Shapiro. Advertising in health insurance markets. Browser Download This Paper, 2016.E. H. Simpson. Measurement of diversity. Nature, 1949a.E. H. Simpson. Measurement of Diversity. Nature, 1949b.StatCounter. Mobile and tablet internet usage exceeds desktop for first

time worldwide , 2016. URL http://gs.statcounter.com/press/mobile-and-tablet-internet-usage-exceeds-desktop-for-first-time-worldwide.

J. M. Wooldridge. Inverse probability weighted estimation for general missing data problems. Journal ofEconometrics, 141(2):1281–1301, 2007.

http://gs.statcounter.com/press/mobile-and-tablet-internet-usage-exceeds-desktop-for-first-time-worldwide

http://gs.statcounter.com/press/mobile-and-tablet-internet-usage-exceeds-desktop-for-first-time-worldwide

AppendicesA Effects of Variety on App Usage

Dependent variable: Leave (Yi,t)

(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t -0.00715∗∗∗ -0.00254∗ -0.00268∗ -0.00171 -0.00124 -0.00039 -0.00005

(-5.20) (-2.11) (-2.33) (-1.55) (-1.15) (-0.36) (-0.05)Previous -0.00284∗ -0.00323∗∗ -0.00084 -0.00223∗ -0.00115 -0.00383∗∗∗ 0.00030Exposure (Ii,t) (-2.39) (-2.96) (-0.79) (-2.21) (-1.17) (-3.98) (0.32)R2 0.222 0.250 0.279 0.305 0.329 0.348 0.382Adjusted R2 0.014 0.010 0.013 0.012 0.012 0.006 0.023

B. Consecutive ChangesChangei,t -0.00402∗∗ -0.00126 -0.00144 -0.00062 0.00136 0.00027 -0.00044

(-2.97) (-1.08) (-1.32) (-0.60) (1.36) (0.28) (-0.46)Previous -0.00217 -0.00292∗∗ -0.00052 -0.00196 -0.00053 -0.00367∗∗∗ 0.00020Exposure (Ii,t) (-1.83) (-2.69) (-0.50) (-1.95) (-0.54) (-3.84) (0.22)R2 0.222 0.250 0.279 0.305 0.329 0.348 0.382Adjusted R2 0.014 0.010 0.013 0.012 0.012 0.006 0.023

C. Gini-Simpson Index for DiversityGiniSimpsoni,t -0.01529∗∗∗ -0.00809 -0.01489∗∗ -0.00457 0.00322 -0.01270 -0.01093

(-4.86) (-1.84) (-2.59) (-0.66) (0.39) (-1.34) (-1.03)Previous -0.00283∗ -0.00322∗∗ -0.00102 -0.00203∗ -0.00071 -0.00418∗∗∗ -0.00003Exposure (Ii,t) (-2.37) (-2.92) (-0.96) (-1.98) (-0.71) (-4.25) (-0.03)R2 0.222 0.250 0.279 0.305 0.329 0.348 0.382Adjusted R2 0.014 0.010 0.013 0.012 0.012 0.006 0.023

Ad FE X X X X X X XStrata FE X X X X X X XNo. of Obs. 365,166 285,373 234,109 194,485 165,558 142,878 124,864

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A1: OLS Estimates of the Effects of Variety on Leave in the Survived Sample Using DifferentMeasures of Variety

i

B Results for the Model Using Our Measure of Sequential Variety


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Sequential Variety with ω(s, s′) = 1s′−s−1

SeqVarietyi,t 0.00160∗∗∗ 0.00388∗∗∗ 0.00376∗∗∗ 0.00254∗∗ 0.00456∗∗∗ 0.00619∗∗∗ 0.00626∗∗∗

(3.33) (5.71) (4.81) (2.72) (4.42) (5.30) (5.05)Previous 0.00042∗∗ 0.00075∗∗∗ 0.00041∗∗∗ 0.00033∗∗∗ 0.00018∗ 0.00025∗∗∗ 0.00024∗∗∗


B. Sequential Variety with ω(s, s′) = 12s′−s−1

SeqVarietyi,t 0.00320∗∗∗ 0.00787∗∗∗ 0.00771∗∗∗ 0.00537∗∗ 0.00974∗∗∗ 0.01328∗∗∗ 0.01383∗∗∗

(3.33) (5.72) (4.80) (2.75) (4.42) (5.21) (5.01)Previous 0.00042∗∗ 0.00075∗∗∗ 0.00041∗∗∗ 0.00033∗∗∗ 0.00017∗ 0.00024∗∗∗ 0.00023∗∗∗


Ad FE X X X X X X XStrata FE X X X X X X XNo. of Obs. 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A2: OLS estimates of the effects of variety on clicks with measures of sequential varietyusing the intended sample

ii

C Results for Robustness Checks


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

Varietyi,t 0.00062 0.00168∗∗∗ 0.00085∗∗∗ 0.00046∗∗ 0.00066∗∗∗ 0.00052∗∗∗ 0.00047∗∗∗

(1.23) (5.53) (4.24) (2.96) (5.38) (4.88) (5.28)Previous 0.00056 0.00170∗∗∗ 0.00040∗ 0.00045∗∗ 0.00043∗∗∗ 0.00023∗ 0.00025∗∗

Exposure (Ii,t) (1.06) (5.39) (1.98) (2.95) (3.65) (2.28) (3.13)Spacei,t 0.00115∗∗∗ 0.00080∗∗∗ 0.00004 0.00020∗ 0.00030∗∗∗ 0.00006 -0.00006

(4.79) (4.98) (0.42) (2.54) (4.80) (1.14) (-1.55)

Ad FE X X X X X X XStrata FE X X X X X X XNo. of Obs. 449,667 519,690 573,096 616,887 652,311 681,160 711,391R2 0.157 0.129 0.120 0.104 0.093 0.076 0.075Adjusted R2 0.039 0.033 0.037 0.032 0.028 0.017 0.022

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A3: OLS estimates of the effects of variety after controlling for temporal spacing using theintended sample with at least one prior exposure of the focal ad

iii


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

Varietyi,t 0.00062∗∗ 0.00070∗∗∗ 0.00037∗∗ 0.00018 0.00024∗∗ 0.00023∗∗ 0.00022∗∗∗

(3.01) (4.45) (3.26) (1.90) (2.98) (3.21) (3.70)Previous 0.00090∗∗ 0.00098∗∗∗ 0.00028 0.00031∗ 0.00028∗∗ 0.00014 0.00013Exposure (Ii,t) (2.69) (4.25) (1.80) (2.49) (2.86) (1.61) (1.92)Spacei,t 0.00101∗∗∗ 0.00080∗∗∗ 0.00014 0.00021∗∗ 0.00035∗∗∗ 0.00008 -0.00004

(4.59) (5.44) (1.44) (2.77) (6.05) (1.72) (-1.11)First Exposure 0.00240∗∗ 0.00191∗∗ -0.00002 0.00033 0.00111∗∗ -0.00022 -0.00049Dummy (3.27) (3.24) (-0.05) (0.83) (3.27) (-0.69) (-1.76)


Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A4: OLS estimates of the effects of variety after controlling for temporal spacing using theintended sample


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

Shannoni,t 0.00123∗∗∗ 0.00184∗∗∗ 0.00135∗∗∗ 0.00073∗ 0.00108∗∗∗ 0.00123∗∗∗ 0.00116∗∗∗

(3.51) (5.47) (4.59) (2.56) (4.08) (4.72) (4.73)Previous 0.00042∗∗ 0.00074∗∗∗ 0.00040∗∗∗ 0.00033∗∗∗ 0.00017∗ 0.00024∗∗∗ 0.00023∗∗∗

Exposure (Ii,t) (2.72) (5.80) (4.11) (3.91) (2.47) (3.79) (4.35)


Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A5: OLS estimates of the effects of variety as measure by Shannon entropy using the intendedsample

iv


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t 0.00051∗∗ 0.00069∗∗∗ 0.00039∗∗∗ 0.00023∗∗ 0.00023∗∗∗ 0.00024∗∗∗ 0.00025∗∗∗

(3.29) (5.83) (4.56) (3.19) (3.91) (4.72) (5.71)Previous 0.00014 0.00045∗∗∗ 0.00010 0.00010 0.00006 0.00010∗ 0.00009∗



(2.78) (6.30) (4.54) (4.16) (3.53) (5.37) (5.57)Previous 0.00012 0.00044∗∗∗ 0.00009 0.00011 0.00005 0.00010∗ 0.00009∗


C. Gini-Simpson Index for DiversityGiniSimpsoni,t 0.00098∗∗ 0.00222∗∗∗ 0.00178∗∗∗ 0.00112∗ 0.00135∗∗ 0.00215∗∗∗ 0.00201∗∗∗

(2.80) (5.26) (4.20) (2.53) (3.11) (4.93) (4.79)Previous 0.00013 0.00045∗∗∗ 0.00011 0.00009 0.00006 0.00012∗ 0.00010∗


Ad FE X X X X X X XProvince FE X X X X X X XConnectivity Type FE X X X X X X XMSP FE X X X X X X XSmartphone Brand FE X X X X X X XHour FE X X X X X X XNo. of Obs. 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A6: OLS estimates of the effects of variety on clicks with separate targeting variable fixedeffects using the intended sample

v


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t 0.00054∗∗∗ 0.00072∗∗∗ 0.00041∗∗∗ 0.00028∗∗∗ 0.00024∗∗∗ 0.00028∗∗∗ 0.00026∗∗∗

(3.41) (5.99) (4.61) (3.75) (3.85) (5.15) (5.59)Previous 0.00025 0.00057∗∗∗ 0.00023∗∗ 0.00021∗∗ 0.00011∗ 0.00017∗∗∗ 0.00015∗∗∗



(2.99) (6.65) (4.67) (4.55) (3.99) (6.27) (5.70)Previous 0.00023 0.00056∗∗∗ 0.00022∗∗ 0.00021∗∗ 0.00010 0.00017∗∗∗ 0.00014∗∗∗


C. Gini-Simpson Index for DiversityGiniSimpsoni,t 0.00107∗∗ 0.00245∗∗∗ 0.00199∗∗∗ 0.00154∗∗∗ 0.00153∗∗∗ 0.00276∗∗∗ 0.00213∗∗∗

(2.98) (5.65) (4.54) (3.33) (3.35) (5.95) (4.73)Previous 0.00024 0.00057∗∗∗ 0.00024∗∗ 0.00021∗∗ 0.00011∗ 0.00019∗∗∗ 0.00015∗∗∗


Ad FE X X X X X X XProvince-Hour FE X X X X X X XNo. of Obs. 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A7: OLS estimates of the effects of variety on clicks with province-hour fixed effects usingthe intended sample

vi


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t 0.00052∗∗∗ 0.00070∗∗∗ 0.00039∗∗∗ 0.00024∗∗ 0.00024∗∗∗ 0.00025∗∗∗ 0.00025∗∗∗

(3.33) (5.91) (4.46) (3.27) (3.97) (4.82) (5.75)Previous 0.00015 0.00046∗∗∗ 0.00012 0.00012 0.00007 0.00011∗ 0.00009∗



(2.71) (6.36) (4.53) (4.16) (3.45) (5.46) (5.76)Previous 0.00012 0.00045∗∗∗ 0.00011 0.00013∗ 0.00005 0.00011∗ 0.00008∗


C. Gini-Simpson Index for DiversityGiniSimpsoni,t 0.00099∗∗ 0.00225∗∗∗ 0.00173∗∗∗ 0.00117∗∗ 0.00134∗∗ 0.00218∗∗∗ 0.00211∗∗∗

(2.81) (5.31) (4.08) (2.64) (3.06) (4.96) (4.97)Previous 0.00014 0.00046∗∗∗ 0.00012 0.00012 0.00006 0.00012∗∗ 0.00010∗


Ad FE X X X X X X XTargeting Area FE X X X X X X XHour FE X X X X X X XNo. of Obs. 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A8: OLS estimates of the effects of variety on clicks with targeting area and hour fixed effectsusing the intended sample

vii


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t 0.00083∗∗∗ 0.00080∗∗∗ 0.00042∗∗∗ 0.00029∗∗ 0.00032∗∗∗ 0.00029∗∗∗ 0.00021∗∗

(4.24) (5.18) (3.64) (2.89) (3.82) (3.88) (3.21)Previous 0.00054∗∗ 0.00078∗∗∗ 0.00047∗∗∗ 0.00034∗∗∗ 0.00016∗ 0.00024∗∗∗ 0.00018∗∗


B. Consecutive ChangesChangei,t 0.00074∗∗∗ 0.00079∗∗∗ 0.00043∗∗∗ 0.00034∗∗∗ 0.00031∗∗∗ 0.00039∗∗∗ 0.00023∗∗∗

(3.86) (5.22) (3.79) (3.58) (3.82) (5.50) (3.74)Previous 0.00052∗∗ 0.00076∗∗∗ 0.00047∗∗∗ 0.00035∗∗∗ 0.00015 0.00026∗∗∗ 0.00018∗∗


C. Gini-Simpson Index for DiversityGiniSimpsoni,t 0.00176∗∗∗ 0.00272∗∗∗ 0.00248∗∗∗ 0.00164∗ 0.00255∗∗∗ 0.00281∗∗∗ 0.00218∗∗

(3.93) (4.78) (4.18) (2.53) (3.82) (4.00) (3.11)Previous 0.00054∗∗ 0.00078∗∗∗ 0.00051∗∗∗ 0.00034∗∗∗ 0.00018∗ 0.00026∗∗∗ 0.00019∗∗


Ad FE X X X X X X XStrata ( 12 hour) FE X X X X X X XNo. of Obs. 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A9: OLS estimates of the effects of variety on clicks with strata constructed by half an hourtime intervals using the intended sample

viii


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t 0.00088∗∗∗ 0.00117∗∗∗ 0.00034∗ 0.00028∗ 0.00047∗∗∗ 0.00021 0.00005

(3.61) (5.81) (2.17) (2.01) (3.81) (1.83) (0.47)Previous 0.00053∗∗ 0.00063∗∗∗ 0.00010 0.00028∗ 0.00016 0.00010 0.00028∗∗


B. Consecutive ChangesChangei,t 0.00077∗∗ 0.00100∗∗∗ 0.00039∗∗ 0.00030∗ 0.00030∗∗ 0.00036∗∗∗ 0.00007

(3.27) (5.28) (2.68) (2.35) (2.77) (3.69) (0.86)Previous 0.00049∗ 0.00056∗∗ 0.00011 0.00029∗ 0.00011 0.00017 0.00029∗∗∗


C. Gini-Simpson Index for DiversityGiniSimpsoni,t 0.00182∗∗∗ 0.00339∗∗∗ 0.00184∗ 0.00095 0.00254∗∗ 0.00094 0.00162

(3.31) (4.70) (2.37) (1.09) (2.76) (0.95) (1.62)Previous 0.00053∗∗ 0.00060∗∗∗ 0.00013 0.00025 0.00016 0.00008 0.00034∗∗∗


Ad-Strata FE X X X X X X XNo. of Obs. 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426 1,054,426

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A10: OLS estimates of the effects of variety on clicks with ad-strata fixed effects using theintended sample

ix


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t 0.00188∗∗ 0.00235∗∗∗ 0.00189∗∗∗ 0.00068 0.00135∗∗∗ 0.00067∗ 0.00069∗

(3.04) (4.33) (4.35) (1.73) (3.98) (2.07) (2.42)Previous 0.00118∗ 0.00193∗∗∗ 0.00098∗∗ 0.00108∗∗∗ 0.00091∗∗∗ 0.00031 0.00079∗∗∗


B. Consecutive ChangesChangei,t 0.00131∗ 0.00158∗∗∗ 0.00156∗∗∗ 0.00059 0.00060∗ 0.00076∗∗ 0.00052∗

(2.32) (3.40) (4.39) (1.89) (2.28) (3.11) (2.46)Previous 0.00093∗ 0.00158∗∗∗ 0.00086∗∗ 0.00107∗∗∗ 0.00064∗∗ 0.00043 0.00079∗∗∗


C. Gini-Simpson Index for DiversityGiniSimpsoni,t 0.00409∗∗ 0.00564∗∗ 0.00912∗∗∗ 0.00409 0.00858∗∗∗ 0.00537 0.01056∗∗∗

(3.00) (3.01) (4.47) (1.76) (3.50) (1.96) (3.83)Previous 0.00120∗∗ 0.00170∗∗∗ 0.00118∗∗∗ 0.00118∗∗∗ 0.00105∗∗∗ 0.00043 0.00120∗∗∗


Strata FE X X X X X X XNo. of Obs. 188,892 189,232 187,562 188,462 187,077 187,895 187,188

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A11: OLS estimates of the effects of variety on clicks for the top ad using the intended sample

x


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

A. Breadth of VarietyVarietyi,t 0.01371∗∗∗ 0.01700∗∗∗ 0.00846∗∗∗ 0.00888∗∗∗ 0.00890∗∗∗ 0.00807∗∗∗ 0.00699∗∗∗

(4.40) (5.95) (3.56) (3.85) (4.24) (3.96) (3.63)Previous 0.00471 0.00411 0.00030 -0.00144 -0.00019 0.00303 0.00262Exposure (Ii,t) (1.68) (1.54) (0.13) (-0.65) (-0.09) (1.53) (1.41)R2 0.565 0.596 0.599 0.614 0.633 0.644 0.644Adjusted R2 0.035 0.059 0.031 0.028 0.036 0.026 -0.007

B. Consecutive ChangesChangei,t 0.00843∗∗ 0.01242∗∗∗ 0.00308 0.00631∗∗ 0.00917∗∗∗ 0.00914∗∗∗ 0.00765∗∗∗

(2.67) (4.34) (1.31) (2.80) (4.47) (4.61) (4.08)Previous 0.00343 0.00281 -0.00113 -0.00222 -0.00045 0.00306 0.00268Exposure (Ii,t) (1.23) (1.05) (-0.50) (-1.01) (-0.23) (1.56) (1.45)R2 0.564 0.596 0.599 0.614 0.633 0.644 0.644Adjusted R2 0.034 0.059 0.030 0.027 0.036 0.027 -0.007

C. Gini-Simpson Index for DiversityGiniSimpsoni,t 0.02760∗∗∗ 0.05400∗∗∗ 0.04075∗∗∗ 0.04251∗∗ 0.05747∗∗∗ 0.05679∗∗ 0.06266∗∗

(3.82) (5.07) (3.34) (2.83) (3.49) (2.99) (3.06)Previous 0.00447 0.00396 0.00054 -0.00168 -0.00027 0.00282 0.00270Exposure (Ii,t) (1.59) (1.47) (0.23) (-0.75) (-0.13) (1.39) (1.42)R2 0.565 0.596 0.599 0.614 0.633 0.643 0.644Adjusted R2 0.035 0.059 0.031 0.027 0.036 0.026 -0.008

Ad FE X X X X X X XStrata FE X X X X X X XNo. of Obs. 57,363 50,684 45,882 41,332 37,773 34,717 32,241

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A12: OLS estimates of the effects of variety on clicks for the first time users are exposed tothe experimental condition using the survived sample

xi


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

Variety (Vi,t) 0.0062∗∗ 0.0116∗∗∗ 0.0070∗∗∗ 0.0061∗∗∗ 0.0057∗∗∗ 0.0070∗∗∗ 0.0059∗∗∗

(2.73) (5.86) (4.25) (3.92) (3.91) (5.10) (4.65)

Variety-constant (Qi,t) -0.0002 0.0042 -0.0004 0.0031 -0.0000 0.0040∗∗ 0.0040∗∗

Changes (-0.07) (1.55) (-0.20) (1.78) (-0.02) (2.95) (3.24)

Previous Exposure (Ii,t) 0.0001 0.0033 -0.0003 0.0004 0.0006 0.0024∗ 0.0020(0.05) (1.91) (-0.22) (0.26) (0.48) (2.16) (1.87)

Ad FE X X X X X X X

Province-Hour-Day FE X X X X X X X

N 58981 54271 50844 47273 44060 41077 38528R2 0.156 0.183 0.172 0.177 0.183 0.191 0.184Adjusted R2 0.057 0.080 0.060 0.059 0.058 0.059 0.042

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A13: Linear regressions estimates of click model with for the first time users are exposed tothe experimental condition controlling for province-hour-day fixed effects

xii


(t = 4) (t = 5) (t = 6) (t = 7) (t = 8) (t = 9) (t = 10)

Variety (Vi,t) 0.0105∗∗ 0.0149∗∗∗ 0.0083∗∗ 0.0072∗ 0.0070∗ 0.0085∗∗ 0.0060∗

(2.58) (4.13) (2.69) (2.49) (2.49) (3.23) (2.39)

Variety-constant (Qi,t) -0.0013 -0.0007 -0.0047 0.0008 0.0025 0.0051 0.0037Changes (-0.20) (-0.14) (-1.23) (0.23) (0.81) (1.85) (1.46)

Previous Exposure (Ii,t) 0.0017 0.0017 0.0006 -0.0008 -0.0019 0.0025 0.0023(0.49) (0.53) (0.20) (-0.31) (-0.79) (1.11) (1.06)

Ad FE X X X X X X X

Strata FE X X X X X X X

N 58981 54271 50844 47273 44060 41077 38528R2 0.573 0.596 0.599 0.612 0.629 0.641 0.635Adjusted R2 0.070 0.092 0.078 0.083 0.092 0.096 0.055

Note: ∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001

Table A14: Linear regressions estimates of click model with for the first time users are exposed tothe experimental condition controlling for auction-invariant strata fixed effects

xiii

how does variety of previous ads inﬂuence consumer’s ad ... · mobile in-app advertising is now...

Documents