The Influence of Design Dimensions on Stated Choices – An Example from Environmental Valuation Using a Design of Designs Approach
By
Jürgen Meyerhoff, Malte Oehlmann, Priska Weller
033 /2013
WORKING PAPER ON MANAGEMENT
IN ENVIRONMENTAL PLANNING
Working Paper on Management in Environmental Planning 33/2013 Arbeitspapiere zum Management in der Umweltplanung 33/2013
Authors Jürgen Meyerhoff * Technische Universität Berlin Institute for Landscape and Environmental Planning Straße des 17. Juni 145 D-10623 Berlin [email protected] Malte Oehlmann Technische Universität Berlin Institute for Landscape and Environmental Planning Straße des 17. Juni 145 D-10623 Berlin [email protected] Priska Weller Johann Heinrich von Thünen Institute Institute of Forest Economics Leuschnerstraße 91 D-21031 Hamburg [email protected] * corresponding author
ABSTRACT This paper investigates the influence of task complexity on dropout rates and model results in stated choice experiments in the context of environmental valuation. We systematically vary the number of choice sets, the number of available alternatives, the number of attributes and their levels as well as the level range presented to each respondent. Largely, we follow a design of designs approach originally introduced in the context of transportation using 16 different split samples. First, we relate choice task complexity to participants dropout behav-ior. We find that the probability to drop out of the survey is influenced by socio-demographic characteristics and increases with the number of choice sets as well as by the number of alternatives. Second, we investigate the impact of the design dimensions on stated choices by estimating a multinomial logit model and a heteroskedastic logit model. Results show that with the exception of the number of choice sets all design dimensions influence the error term variance. Finally, we compare willingness to pay measures from both models finding that the absolute willingness to pay estimates differ between both models. However, the point estimates are each time included in the confidence interval of the other model for the same attribute. Keywords: choice complexity, design of designs, stated choice experiment, error variance
- 1 -
Working Paper on Management in Environmental Planning 33/2013
1. Introduction
Discrete Choice Experiments (CE) are widely used to elicit consumer preferences in different
fields of application such as marketing, health economics, transportation and environmental
valuation. CEs usually consist of a series of hypothetical scenarios or choice sets which are
composed of two or more alternatives. The characteristics or attributes of the alternatives are
varied in a systematic way by means of their levels. For each choice set, respondents are
asked to choose their preferred alternative (Louviere et al. 2000). The number of choice sets,
alternatives, and attributes as well as the number of levels and their range can be seen as
the design dimensionality of the CE to be specified by the researcher. Across different stud-
ies, the dimensionality of the choice experiment can vary significantly. For instance, in the
majority of CE studies respondents are offered 4 to 10 choice sets and 2 to 4 alternatives.
However, there are occasions in which participants are asked to assess up to 26 or even
more choice questions (Czajkowski et al. 2012) and choose among 12 or more alternatives
(Chung et al. 2011). Similar observations can be made for the other design dimensions.
Assuming neoclassic economic theory, which suggests that individuals are omnipotent,
fully rational decision makers who have stable preferences and utility functions, the design
dimensionality should not influence choice outcomes. As a consequence, the complexity of
the choice experiment, the ability of the individual to make complex decisions and the effect
of the choice context on decision strategies are often not considered in statistical models
(Swait and Adamowicz 2001). However, many authors have highlighted that the limited abil-
ity of individuals to process complex information needs to be taken into account. For in-
stance, Heiner (1983) argued that choice complexity can influence choice consistency and
that the more complex the choice task is, the higher is the gap between individual’s cognitive
ability and cognitive demand. This result leads to a trade-off to be made by the researcher.
On the one hand, one might be tempted to increase task complexity to, for instance, increase
the number of observations or to gather as much information as possible by increasing the
number of attributes. However, this comes at the cost of higher cognitive burden and the
possibility of significantly distorted model estimates (DeShazo and Fermo 2002). As a result,
there is so far no agreement in the literature on the optimal task complexity in CEs.
Task complexity, in general, can be seen as a part of the unobserved factors influencing
choice outcomes. The suit of unobserved candidate influences can be classified as follows:
omitted variables, measurement error in the observed attributes and alternatives, true task
complexity that imposes variation in cognitive difficulty, and uncertainty attributable to many
sources such as stimulus ambiguity, beliefs about future states and peer impacts (Hensher
2006). In our study, we focus on the issue of true task complexity expressed through the var-
iation of five design dimensions. Our research is largely motivated by a series of studies in-
vestigating the influence of the number of choice sets, the number of alternatives in each
choice set, the number of attributes, the number of attribute levels, and level range on choice
experiment outcomes in the context of transportation (Hensher 2004, Caussade et al. 2005,
Hensher 2006, Rose et al. 2009). In these studies, 16 different treatments were used which
were generated by a design of designs (DoD) approach originally introduced by Hensher
(2004). The attributes used considered different travel times and cost. Our study is based on
- 2 -
Working Paper on Management in Environmental Planning 33/2013
a nation-wide online survey in Germany carried out in the context of land use changes in
December 2012. We incorporated aspects of land consumption, the share of forest and dif-
ferent biodiversity attributes.
Similar to Hensher (2004) and Caussade et al. (2005), we used 16 different split samples
following a design master plan. To our knowledge, this is the first study to apply a DoD ap-
proach in environmental valuation systematically varying five design dimensions. We analyze
the influence of the five design dimensions in terms of two aspects: First, we investigate the
relationship between task complexity and participants dropout behavior by using descriptive
statistics and by specifying a binary logit model. Second, we estimate a joint multinomial logit
model (MNL) for all samples and a heteroskedastic logit model (HL) with the scale parameter
specified as a function of the design dimensions. This allows us to study the impact of the
five design dimensions on the error term variance which is inversely related to the scale pa-
rameter. Willingness to pay (WTP) estimates are subsequently obtained from both models
and compared to each other. The remainder of this paper is structured as follows: Section 2
outlines previous literature and the hypotheses to be tested in our study. Section 3 presents
the modeling approach before giving details on the study design and implementation in Sec-
tion 4. The main results are presented in Section 5 (dropout analysis) and Section 6 (estima-
tion results). Finally, main conclusions and further research will be discussed.
2. Literature Review and Hypotheses to be Tested
The influence of task complexity of stated choice experiments has been investigated in sev-
eral studies. Mostly, complexity issues have been analyzed in the context of health econom-
ics (Ryan and Wordsworth 2000, Ratcliffe and Longworth 2002, Bech et al. 2011), marketing
research (Dellaert et al. 1999, Dellaert et al. 2012) and transportation (Hensher et al. 2001,
Hensher 2004, Caussade et al. 2005, Hensher 2006, Rose et al. 2009). The research has
largely focused on dimensionality influences on the error variance or scale parameter
through effects of fatigue and learning (Dellaert et al. 1999, DeShazo and Fermo 2002,
Arentze et al. 2003, Caussade et al. 2005, Rolfe and Bennett 2009, Chung et al. 2011, Czaj-
kowski et al. 2012) and on WTP estimates (Ryan and Wordsworth 2000, Hensher 2004,
Hensher 2006, Bech et al. 2011, McNair et al. 2011). Additionally, effects on attribute weights
(Arentze et al. 2003), response rate (Hensher et al. 2001, Bech et al. 2011), decision time
(Dellaert et al. 2012) or perceived choice certainty (Rose et al. 2009, Brouwer et al. 2010,
Bech et al. 2011) have been analyzed.
Subsequently, we review the literature in order to develop hypotheses of possible influ-
ences on the relationship between dropout rates and error variance. With respect to the im-
pacts on the error variance and in the light of studies published in recent years, we largely
follow the hypotheses which were put forward by Caussade et al. (2005) in order to compare
their results with our findings in the context of environmental valuation.
- 3 -
Working Paper on Management in Environmental Planning 33/2013
2.1. Hypothesis on the Relation between Task Complexity and Dropout
Rates
To our knowledge, this is the first study to investigate the impact of task complexity on partic-
ipants dropout behavior in discrete choice experiments. What we have found were two stud-
ies looking at the relationship between the number of choice sets and the response rate.
Hensher et al. (2001) found little differences in response rates for mail surveys with varying
numbers of treatments, while Bech et al. (2011) found no significant influence in an online
survey. However, it has to be noted that response and dropout rates need to be distin-
guished from each other. Whereas the former gives the ratio between those respondents
who completed the questionnaire and those contacted, the later shows the percentage of
those participants who prematurely abandoned the questionnaire (Vicente and Reis 2010).
Supported by studies carried out in other disciplines which concluded that the survey length
has significant influence on dropout rates (see for example Vicente and Reis 2010), we hy-
pothesize that the number of choice sets is positively related to dropout rates since the sur-
vey length increases with the number of choice questions. For the other design dimensions,
we also rely on insights from studies in other areas of research. Among others, Galesic
(2006) showed that the lower the experienced burden is, the lower is the risk of dropping out.
Based on this, we expect the dropout rate to grow with the number of alternatives and the
number of attributes. For the number of levels and their range, it may be argued that the
number of comparisons to be made increases with the number of levels and that compari-
sons might be easier to assess for attribute levels which have a narrow range (Caussade et
al. 2005). As a result, we also hypothesize a positive relationship between these design di-
mensions and dropout rates.
2.2. Hypothesis on the Relationship between Task Complexity
and Error Variance
2.2.1. Number of Choice Sets
The design dimension which has probably been analyzed most is the number of choice sets.
However, there is no consensus in the literature with respect to impacts on the error vari-
ance. On the one hand, Hensher et al. (2001) found little evidence for fatigue effects for even
32 choice sets. Brouwer et al. (2010) and Czajkowski et al. (2012) have also supported this
inference. On the other hand, Bradley and Daly (1994), who were probably the first to inves-
tigate fatigue effects in CEs, found the error variance to grow with the number of choice sets.
Caussade et al. (2005) and Chung et al. (2011) observed a U-shaped relation with the error
variance decreasing up to a threshold (9/10 sets in Caussade et al. 2005, 6 sets in Chung et
al. 2011) and increasing after this. A similar pattern was found by Bech et al. (2011). These
authors argue that the error variance initially decreases due to learning effects while subse-
quently fatigue effects cause the error variance to increase. We follow this argumentation
and expect a U-shaped pattern.
- 4 -
Working Paper on Management in Environmental Planning 33/2013
2.2.2. Number of Alternatives
The evidence for the impact of the number of alternatives tends to be mixed. Arentze et al.
(2003) found no effects on the error variance distinguishing between a design of 2 and 3 al-
ternatives. As opposed to this, Caussade et al. (2005) as well as Chung et al. (2011) found a
U-shaped pattern with the lowest error variance for 5 and 4 alternatives, respectively. A simi-
lar pattern emerged earlier in DeShazo and Fermo (2002), who argued that the initial de-
crease of the error variance results from a better match of preferences while the increase at
the later stage is caused by a more complex choice.
2.2.3. Number of Attributes
With respect to the number of attributes, there is clear evidence that an increase in attributes
results in an increase in the error variance. Caussade el al. (2005) found the number of at-
tributes to have a strong detrimental effect on the ability to choose contributing to higher error
variance. Similar inferences were drawn by DeShazo and Fermo (2002) and Arentze et al.
(2003). As the information load to be processed by the respondent grows with the number of
attributes, we also expect a positive relationship between the error variance and the number
of alternatives.
2.2.4. Number of Attribute Levels and Level Range
Based on the findings from Dellaert et al. (1999) and Caussade et al. (2005) and following
the same argumentation presented above regarding the influences on dropout rates, we ex-
pect a positive relationship between the number of attribute levels and the error variance and
an increase in the error variance with a wider level range.
Hypothesis 1:
There exists a U‐shaped relationship between the number of choice sets and the error variance.
Hypothesis 2:
There exists a U‐shaped relationship between the number of alternatives and the error variance.
Hypothesis 3:
There exists a positive relationship between the number of attributes and the error variance.
- 5 -
Working Paper on Management in Environmental Planning 33/2013
3. Econometric approach
Random utility theory assumes that the modeler does not possess complete information con-
cerning the individual decision maker (subscript n) and thus considers individual preferences
to be the sum of a systematic (Vin) and random (εin) components as
in in in inU V (x ) (1)
where Uin is the true but unobservable utility associated with alternative i out of a set of avail-
able alternatives, Vin is the measurable or deterministic part which is itself a function of the
attributes (xinß), ß is a vector of coefficients reflecting the desirability of the attributes, and εin
is a random term with zero mean. The error term εin represents attributes and characteristics
unknown to the researcher, measurement error and/or taste heterogeneity among respond-
ents. Selection of one alternative over another implies that the utility (Uin) of that alternative is
greater than the utility of the other alternative:
i i j jP(i) Pr ob(V V ) j C, j i (2)
Assuming that the error components are distributed independently and identically (IID) fol-
lowing a type 1 extreme value distribution, we get the multinomial logit (MNL) model where
the probability of choosing alternative i chosen by individual n takes the form:
exp( )
exp( )in
injn
j C
VP
V
(3)
where μ is a scale parameter which is commonly normalized to 1 in practical applications for
any one data set as it can not be identified separately from the vector of parameters. The
scale parameter is inversely proportional to the error variance 2 :
26
(4)
The assumption of a constant error variance across individuals has been questioned and
alternatively a heteroskedastic logit model (HCL; e.g., DeShazo and Fermo 2002, Hole
2006). Here, the scale parameter is no longer a constant term as it allows for unequal vari-
Hypothesis 4:
There exists a positive relationship between the number of attribute levels and the error variance.
Hypothesis 5:
There exists a positive relationship between width of the level range and the error variance.
- 6 -
Working Paper on Management in Environmental Planning 33/2013
ances across survey characteristics such as different treatments, e.g., the design dimen-
sions, or individual characteristics. It can be described by
exp( )
exp( )n in
inn jn
j C
VP
V
(5)
where n is a function of survey characteristics, i.e., individuals assigned to specific treat-
ments, or individual characteristics that influence the scale parameter and accordingly the
error variance. The parameterization of n can be done as exp( )nZ with Zn a vector of the
individual characteristics and as a vector of parameters indicating the influence of those
characteristics on the error variance. If turns out to be zero, then the heteroskedastic logit
collapses to a conditional logit. For estimating the heteroskedastic logit model we use the
STATA program clogithet provided by Hole (2006).
4. Study Design and Implementation
4.1. Study Design
Our study design largely follows the design master plan introduced by Hensher (2004). Tak-
ing into account that several studies published in recent years have shown that respondents
can cope with a fairly large number of choice situations (Czajkowski et al. 2012), we slightly
adapted the design master plan of Hensher (2004) by using 6, 12, 18 and 24 choice situa-
tions rather than 6, 9, 12, and 15 choice sets. As distinct from Hensher (2004), we also in-
creased the number of attributes from using 4 to 7 insteads of 3 to 6. Other dimensions were
kept equal to those of Hensher (2004). In order to obtain 16 treatments, we generated 16
different generic designs using Ngene software. Unlike Hensher (2004) and Caussade et al.
(2005), who employed a D-efficient experimental design, we made use of the C-error as a
design criterion in each treatment since it allows to minimize the variance of the WTP esti-
mates (Scarpa and Rose 2008). Our adaptation of the design master plan can be seen in
Table 1.
- 7 -
Working Paper on Management in Environmental Planning 33/2013
Table 1: Our adaptation of the Design master plan
Design Sets Alternatives Attributes Levels Range
1 24 4 5 3 Base
2 18 4 5 4 +20%
3 24 3 6 2 +20%
4 12 3 6 4 Base
5 6 3 4 3 +20%
6 24 3 4 4 -20%
7 6 4 7 2 -20%
8 12 5 4 4 +20%
9 24 5 4 4 Base
10 6 5 7 3 +20%
11 6 4 6 4 -20%
12 12 5 5 2 -20%
13 18 4 7 2 Base
14 18 3 4 3 -20%
15 12 3 5 2 Base
16 18 5 6 3 -20%
The attributes in our study deal with land use changes. Three different groups of attributes
are distinguished: First, the attributes “share of forest” and “land consumption” were included
in all treatments. The number of levels as well as the level range were varied due to the de-
sign master plan (see Table 1). The different level values were expressed in percentage
changes compared to the current state. Second, different biodiversity attributes were used
which are based on an indicator using stocks of bird populations, which was developed as
part of an indicator system for sustainable development in Germany (BMU 2010). As the
indicator can be split up to bird populations in different parts of the landscape, e.g. “birds in
the whole landscape” equals “birds in agricultural landscapes” plus “birds in other land-
scapes”, we could vary the number of attributes across treatments following Hensher (2004),
who used different types of travel time and cost. This allowed us to aggregate and disaggre-
gate the biodiversity attribute as a combination of already existing attributes. Doing so, one
can systematically account for the influence of the number of attributes on model outcomes
(Hensher 2004). Figure 1 illustrates the split-up of the biodiversity attribute across different
designs. The level values were expressed as indicator values. Respondents were informed
- 8 -
Working Paper on Management in Environmental Planning 33/2013
that an indicator of 100 or more means that the landscape type is a good habitat for a variety
of species.
Third, “contribution to a landscape fund” was utilized as the payment vehicle. This attrib-
ute was presented in all designs with its number of levels and level range being constant
over all treatments. We were well aware that the payment vehicle used in our study might
cause problems concerning the issue of incentive compatibility. However, we decided not to
use taxes due to several experiences with respect to protest respondents in Germany. Table
2 summarizes the attributes used in our study as well as their levels for a base design with 2
levels and a base level range. The attribute levels for designs with 3 and 4 levels as well as a
narrow and a wide level range are available on request.
Figure 1: Split-up of the biodiversity attribute
- 9 -
Working Paper on Management in Environmental Planning 33/2013
Table 2: Attributes and their levels for the base design
Attribute Description Level
Share of forest Percentage changes in the share of forest -25; +25
Land consumption Percentage changes in the land consumption -50; 50
Bio_whole Biodiversity in the whole landscape including all land-
scape types
70; 100
Bio_agrar Agricultural landscape biodiversity 65; 100
Bio_forest Forest landscape biodiversity 80; 100
Bio_urban Urban area biodiversity 60; 100
Bio_other1 Biodiversity in other landscape types: Forests, urban
areas, mountains, waters
75; 100
Bio_other2 Biodiversity in other landscape types: Urban areas,
mountains, waters
60; 100
Bio_other3 Biodiversity in other landscape types: Mountains, wa-
ters
75; 100
Cost Contribution to a landscape fund in € per year 10; 25; 50;
80; 110; 150
As one can see from Table 1, the number of alternatives varies from 3 to 5 including the sta-
tus quo alternative, which was defined as the current situation with the cost attribute being 0.
As we asked respondents to assess land use changes for around 15 km of her or his resi-
dence without knowing the shape of the landscape for each respondent, the attribute levels
for the status quo alternative were chosen to be “like today”. This is another aspect which
distinguishes our study from Caussade et al. 2005. Further differences are summarized in
Table 3.
- 10 -
Working Paper on Management in Environmental Planning 33/2013
Table 3: Differences across complexity studies
Characteristic Hensher (2004), Caussade et al. (2005) Present Study
Application Transportation Environmental valuation
Number of choice sets 6, 9, 12, 15 6, 12, 18, 24
Number of attributes 3 – 6 4 – 7
Survey mode Computer aided personal interview (CAPI)
Online survey
Experimental design D-efficient C-efficient
Status quo Current route Current situation
Region Sydney / Santiago German-wide
Aim of assessment Route changes Land use changes around 15 km of resi-dence
Variation of attribute numbers
Based on different types travel time and cost
Based on Biodiversity in different landscapes types
Payment vehicle Travel cost Contribution to a land-scape fund
4.2. Survey Implementation
A nation-wide online survey was conducted between December 7th and December 21st
2012 using an online panel from a survey company. When participants entered the survey,
they were randomly assigned to 1 of the 16 designs. The questionnaire started by asking
respondents socio-demographic questions concerning date of birth, gender and education.
We did this at that stage of the questionnaire in order to be able to control for socio-
demographic characteristics within the dropout analysis. Then, participants were asked sev-
eral warm-up questions in order for respondents to get introduced to the topic including the
assessment of the current state of the landscape characteristics around 15 km of the re-
spondent’s residence. Before presenting the first choice set, participants were given an in-
struction page with information on the choice experiment as well as the attribute descriptions,
which varied across treatments with 4, 5, 6 and 7 attributes. The choice sets were subse-
quently presented in a randomized order with the number of choice sets depending on the
design respondents were assigned to. For each choice set, respondents were asked to as-
sess land use changes for an area around 15 km of their residence by choosing their pre-
ferred alternative. After finishing the CE, several follow-up questions were asked including
perceived choice certainty and the attendance to attributes. At each stage of the question-
naire, respondents could only abandon the survey by closing their internet browser. If so, the
exact position of dropout was recorded. In total, 2133 interviews were collected with 1673
(78.43%) respondents having completed the whole questionnaire and 460 partial completed
- 11 -
Working Paper on Management in Environmental Planning 33/2013
surveys (dropouts). The average interview length was measured to be 23 minutes and the
response rate was 29.49%.
5. Dropout Analysis
5.1. The Influence of the Design Dimensionality on Dropout Rates
As a first empirical illustration, Table 4 depicts the dropout rates while progressing through
the questionnaire. Around 8.20% of the participants dropped out before starting to answer
the CE. The highest dropout rate was observed to be within the choice experiment (11.02%),
while only 2.35% quit the survey after finishing the CE. The relatively high dropout rate in the
first part of the questionnaire has also been observed in studies carried out in other areas of
research. For instance, Hoerger (2010) found 10% of the participants to drop out of the sur-
vey instantaneously. This behavior may be explained by a low interest in the topic. Among
others, Galesic (2006) found that the lower respondent’s overall interest, the higher the drop-
out rate.
Table 4: Position of dropout
Dropout position Frequency %
Before CE 175 8.20
Within CE 235 11.02
After CE 50 2.34
Completed 1673 78.43
Total 2133 100.00
Next, Table 5 presents the 16 treatments with their corresponding design dimensions as
well as the number of interviews and the number of dropouts per design. We excluded those
respondents who quit the survey before starting to answer the choice experiment since all
questions were identical across designs up to this stage. The split samples which we ob-
served to have the highest dropout rates are designs 1, 3 and 9 each having 24 choice sets
with all other design dimensions varying across designs (3 to 5 alternatives, 4 to 6 attributes,
etc). With respect to the split samples with the lowest dropout rates, we found design 5, 11,
15 all to have 6 choice sets. Again, all other dimensions varied between, for instance, 4 to 6
attributes and 2 to 4 attribute levels. Based on this, there is already some indication that
dropout rates might especially be influenced by the number of choice sets.
- 12 -
Working Paper on Management in Environmental Planning 33/2013
Table 5: Design-dependent Dropouts while answering choice sets
Design Sets Alternatives Attributes Levels Range Interviews Completed
Droppers In %
1 24 4 5 3 Base 83 21 20.19
2 18 4 5 4 +20% 81 10 10.99
3 24 3 6 2 +20% 123 33 21.15
4 12 3 6 4 Base 80 12 13.04
5 6 3 4 3 +20% 102 8 7.27
6 24 3 4 4 -20% 81 16 16.49
7 6 4 7 2 -20% 219 29 11.69
8 12 5 4 4 +20% 80 13 13.98
9 24 5 4 4 Base 81 21 20.59
10 6 5 7 3 +20% 150 29 16.20
11 6 4 6 4 -20% 89 10 10.10
12 12 5 5 2 -20% 79 12 13.19
13 18 4 7 2 Base 112 24 17.65
14 18 3 4 3 -20% 84 18 17.65
15 12 3 5 2 Base 146 9 5.81
16 18 5 6 3 -20% 83 20 19.42
Total 1673 285 14.56
In order to analyse the relationship between the five design dimensions and dropout rates
in detail, we specified a binary logit model with the dependent variable being 0 if a participant
completed the survey and 1 if the respondent dropped out after starting to answer the choice
experiment. The results are presented in Table 6. As expected, the number of choice sets
has a highly significant positive impact on the probability to drop out. The same pattern was
observed for the number of alternatives and the number of attributes, although for the later
the coefficient became only significant at the 10% level. For the attribute levels and the level
range we found no significant influence. So we could not reject our null hypothesis of no ef-
fect. Since we also expected socio-demographic characteristics to be possible sources of
explanation of the probability to drop out, we included age, gender and education as further
independent variables. As shown in Table 6, age has a highly significant effect on the proba-
bility to drop out. The same applies to the dummy variable for being female.
- 13 -
Working Paper on Management in Environmental Planning 33/2013
Table 6: Model estimates for the influence between design dimensions and dropout
rates
Dimension Coefficient Standard Error Z-score
Age 0.019 0.005 3.71
Dummy gender female 0.617 0.135 4.56
Dummy degree after 10 years of school-ing
0.011 0.256 0.04
Dummy degree high-school degree (13 years of schooling)
-0.413 0.279 -1.48
Dummy university degree -0.145 0.256 -0.57
Dummy no degree 0.728 1.253 0.58
Number of choice sets 0.055 0.011 4.92
Number of alternatives 0.191 0.089 2.15
Number of attributes 0.141 0.080 1.77
Dummy narrow range 0.190 0.171 1.11
Dummy wide range 0.180 0.170 1.06
Number of levels 0.012 0.103 0.12
Constant -5.133 0.818 -6.28
Log-likelihood null -810.178
Log-likelihood model -772.701
Pseudo R2 0.046
Based on 1,955 observations.
5.2. Dropouts within the Choice Experiment
To finish our analysis on dropout rates, Table 7 shows the position of dropout within the
choice experiment. We observe the highest number of participants (126) to drop out within
the first six choice sets. This figure corresponds to 53.6% of the dropouts within the choice
experiment and a dropout rate of 6.54%. At the other extreme, only 14 respondents or 6% of
the dropouts abandoned the survey between choice set 19 and set 24. The total dropout rate
at this stage of the CE is therefore only 3.24%.
- 14 -
Working Paper on Management in Environmental Planning 33/2013
Table 7: Dropout rates within the choice experiment
Position dropout
Number of presented choice sets Total Number of respondents
Dropout rate
6 Sets 12 Sets 18 Sets 24 Sets
Set 1 to 6 59 25.1
20 8.5
21 8.9
26 11.1
126 53.6
1952 6.54%
Set 7 to 12 0 16 6.8
20 8.5
16 6.8
52 22.1
1272 4.09%
Set 13 to 18 0 0 18 7.66
25 10.6
43 18.3
835 5.15%
Set 19 to 24 0 0 0 14 5.96
14 6.0
432 3.24%
59 25.1
36 15.32
59 25.1
81 34.5
235 100.0
6. Estimation Results
We started our modelling by estimating a MNL model in which the data from all split samples
were pooled. Furthermore, we specified a HL model with the scale parameter as a function of
the design dimension. The results of the MNL and HL model are shown in Table 8. In both
models, all parameters are significant at a 1% level of significance and have the expected
sign. On average, respondents want a larger share of forests while at the same time prefer-
ring reduced land consumption in their surroundings. The biodiversity attributes all have posi-
tive signs indicating that respondents want to increase the levels of biodiversity as measured
on the underlying scale. The model also contains an alternative specific constant (ASCsq) for
the current situation. It is positive and statistically significant suggesting that on average re-
spondents have a propensity to choose the current situation instead of one of the hypothet-
ical alternatives describing future land use changes. Table 8 also contains the log-likelihoods
for the MNL and HL. To compare both models to each other, we conducted a log-likelihood
ratio test with the test statistic being 108.948. Since the critical Chi-squared value is 21.666
(1% level with 9 degrees of freedom), we can reject the null hypothesis that the HL model is
no better than the MNL model.
- 15 -
Working Paper on Management in Environmental Planning 33/2013
Table 8: Estimation results for the MNL and the HL model
Coefficients MNL model t-ratio HL model t-ratio
ASC_sq 1.374 17.18 2.514 5.51 Share of forest 0.017 35.59 0.030 5.59 Land consumption -0.009 38.45 -0.016 5.62 Bio_whole 0.011 12.85 0.022 5.37 Bio_agrar 0.005 8.40 0.009 4.78 Bio_forest 0.005 6.70 0.008 4.37 Bio_urban 0.003 3.42 0.004 2.09 Bio_other1 0.008 10.67 0.015 5.03 Bio_other2 0.003 4.76 0.006 3.62 Bio_other3 0.003 3.51 0.008 3.25 Cost -0.006 26.54 -0.007 6.13 Log-likelihood null -31278.157 -31278.157 Log-likelihood model -27616.061 -27507.133 Pseudo R2 0.117 0.121
Based on 90,831 observations.
Table 9 reports the estimates of the design dimension variables. For the number of choice
sets, we attempted to show a U-shaped relationship by estimating a linear and a quadratic
effect. However, neither the coefficient for the number of choice sets, nor the coefficient for
the squared number of choice sets became statistically significant. As a result, we cannot
reject the null hypothesis of no relationship between the number of choice sets and the error
variance. This finding contradicts the results from Caussade et al. (2005), but is in agreement
with other studies such as Czajkowski et al. (2012) or Hess et al. (2012). With respect to the
number of alternatives, we can reject the null hypotheses by observing a U-shaped pattern
with dummy variables specified for 4 and 5 alternatives. With respect to a base design with 3
alternatives, designs with 5 and 5 alternatives have a higher scale parameter (lower error
variance). However, the drop in the error variance is less pronounced for designs with 5 al-
ternatives. This suggests exactly the same pattern as observed by Caussade et al. (2005).
The same is true for the number of attributes, which we found to have a highly significant
negative impact on the scale parameter (positive on the error variance) allowing us to reject
our null hypothesis. For the number of levels, we could not find any statistically significant
linear effect. Nevertheless, when we specified dummy variables for 3 and 4 levels, coeffi-
cients became significant suggesting a non-linear relationship between the number of attrib-
ute levels and the error variance with respect to a design with 2 levels. Positive impacts on
the error variance were higher for 3 levels than for designs containing 4 levels with the coef-
ficient only significant at a 10% level. This contradicts the observation of a linear relationship
found by Caussade et al. (2005), but allows us to reject our null hypothesis. Similar inference
was drawn for the level range as we observe the error variance to decrease for designs with
a narrow range and to increase for designs with a wide range compared to the base. This
result is in line with Caussade et al. (2005).
- 16 -
Working Paper on Management in Environmental Planning 33/2013
Table 9: Coefficient estimates for the parameterization
Dimension Coefficient t-value
Number of choice situations -0.007 0.52 Squared number of choice situations 0.001 0.59 Dummy 4 alternatives 0.154 3.88 Dummy 5 alternatives 0.072 2.07 Number of attributes -0.956 4.59 Dummy 3 levels -0.314 7.35 Dummy 4 levels -0.079 1.72 Dummy narrow range 0.202 5.85 Dummy wide range -0.128 3.68
Based on 90,831 observations
Turning now to the willingness to pay (WTP) estimates, we calculate the marginal WTP as
the ratio between the attribute coefficients and the cost at constant utility levels. Table 10
reports the WTP values for the MNL as well as the HL model; the confidence intervals were
calculated using the Delta method. Note that the marginal WTP estimates refer to environ-
mental changes in the 15 km surrounding of respondent’s place of residence. Starting with
the WTP estimates based on the MNL model, respondents are, on average, willing to pay
2.73 € per year for a one percent increase in the share of forests, but would experience a
disutility of 1.39 € per year for a one percent growth in land consumption. The biodiversity
attributes, which were aggregated and disaggregated across designs, show values as ex-
pected. Respondents are willing to pay more for the aggregated attribute “whole landscape
biodiversity” (Bio_whole), which is 1.84 € per year for an improvement by one point of the
indicator (see Section 4.1), than for attributes at a lower aggregation level such a forest land-
scape biodiversity (0.78 € per year for one point improvement) or agricultural landscape bio-
diversity (0.77 € per year for an improvement by one point). Moreover, the WTP for
bio_other1, which includes biodiversity in forests, urban areas, mountains and waters, is
higher than bio_other2 considering biodiversity in mountains, urban areas and waters.
The absolute WTP estimates differ between the MNL and HL model but the point esti-
mates are each time included in the interval of the other model for the same attribute. Thus,
taking into account the impact of the design dimension on the error variance does not result
in statistically significant different WTP estimates.
- 17 -
Working Paper on Management in Environmental Planning 33/2013
Table 10: Marginal willingness to pay estimates for attributes in Euro per year
Attribute MNL model 95% - CI HL model 95% - CI
Share of forest 2.73 2.44 / 3.01 2.96 2.65 / 3.27 Land consumption -1.39 -1.51 / -1.28 -1.50 -1.63 / -1.37 Bio_whole 1.84 1.51 / 2.18 2.09 1.71 / 2.47 Bio_agrar 0.77 0.57 / 0.96 0.78 0.55 / 1.00 Bio_forest 0.78 0.54 / 1.03 0.79 0.50 / 1.08 Bio_urban 0.44 0.19 / 0.70 0.38 0.08 / 0.67 Bio_other1 1.29 1.02 / 1.56 1.34 1.04 / 1.64 Bio_other2 0.52 0.30 / 0.74 0.63 0.36 / 0.90 Bio_other3 0.49 0.21 / 0.77 0.63 0.30 0.96
7. Discussion and Conclusion
In this study, we analyzed the impact of the number of choice sets, the number of alternative
in each choice set, number of attributes as well as the number of levels and their range on
the error variance and on dropout rates. To our knowledge, it is not only the first study em-
ploying the design of design approach introduced by Hensher (2004) in environmental valua-
tion but also, this is the first attempt to investigate the relationship between task complexity in
discrete choice experiments and participant’s dropouts.
With respect to the dropout rates, we found that the probability to abandon the survey sig-
nificantly increases with the number of choice sets and the number of alternatives presented
on a choice set. All other design dimensions did not significantly influence the probability to
quit the survey. Additionally, older respondents as well as women are more likely to drop out.
Among the dropouts that happened while answering the choice tasks, it is noteworthy that
the majority of the respondents abandoned the choice experiment within the first 6 choice
questions. One reason for this could be that people did not like the choice format and thus
decided not to proceed. During focus groups, we have repeatedly discovered that some re-
spondents do not like to make comparisons among bundles of attributes, but would prefer to
rate each attribute separately. Another reason might be that people realized that choosing
among the alternatives on a choice sets also includes payments to the landscape fund and
that quitting the survey at this stage was motivated by protest votes. However, we do not
conclusively know the reasons why respondents abandon the questionnaire. We also re-
frained from sending those people another set of questions in order to find out why they act-
ed as they did. The survey company expected very low response rates for such a kind of
debriefing interview. Although this study is not sufficient to draw general conclusions, the
results suggests that if dropout rates in an online survey are of crucial importance, less
choice sets and fewer alternatives should be presented. These findings might depend very
much on the survey mode. In a CAPI interview, as used by Czajkowski et al. (2012), the re-
spondents might be more motivated or might feel more obliged to go through a series of 26
- 18 -
Working Paper on Management in Environmental Planning 33/2013
choice tasks as presented in their survey. Due to missing information on the dropout rates in
other studies, we are not able to compare our results to other survey formats.
Turning to the results with respect to influences of the design dimensions on the error var-
iance, we mainly find the same results as presented by Caussade et al. (2005). The main
difference is that we cannot find any impact of the number of choice tasks on the error vari-
ance. In all model specifications that we tried, there was no significant association. There-
fore, we cannot reject the null hypothesis that the error variance is not associated with the
number of choice sets. This is in line with the findings of Czajkowski et al. (2012) and Hess et
al. (2012), who also could not find any systematic influence of the number of choice tasks on
the scale parameter. For the other dimensions, we can each time reject our null hypothesis
of no relationship between the design dimensionality and the error variance.
However, there are some reasons to interpret these findings with a degree of caution.
First, our payment vehicle, a contribution to a fund, might not be as incentive compatible as
the costs respondents faced in the survey done by Caussade et al. (2005). Second, we did
not adress taste heterogeneity. Taste heterogeneity is likely to be present among the re-
spondents of our study when it comes to land use options. As the survey was conducted
nation-wide and respondents live in very different landscapes, it is likely that participants pre-
fer different changes. Similarly, the present situation in the 15 km surrounding of residence of
the respondents is very likely to be different and thus might strongly affect the propensity to
choose the status quo option. Within the survey, we asked participants to report on the cur-
rent situation with respect to the choice attributes but have not included these responses into
the models.
Finally, we estimated the willingness to pay for all attributes considered in our study using
the MNL as well as the HL model. We found that the amount participants are willing to pay is
always higher for the biodiversity aggregated attribute than for biodiversity attributes at a
lower level of aggregation. The willingness to pay estimates from both models are not signifi-
cantly different from each other since the point estimate of one model is included in the con-
fidence interval of the other.
Further research could start by applying a more flexible model in terms of taste heteroge-
neity. Moreover, we will include an analysis to which extent the design dimension influence
the number of times the status quo alternative is chosen. The results presented so far indi-
cate that the status quo option is more likely to be chosen when the choice design is more
complex (Boxall et al. 2009, Zhang and Adamowicz 2011). Additionally, effect of the choice
task on choices and WTP estimates will be analysed using an entropy based measure of
complexity This could provide further insights on whether the design dimensions have a
stronger impact on the results of stated choice experiments or whether researchers do not
have to care too much about this issue, at least within the ranges investigated here.
- 19 -
Working Paper on Management in Environmental Planning 33/2013
References
Arentze, T., A., Borgers, H., Timmermans and R., DelMistro, 2003. Transport Stated Choice Responses: Effects of Task Complexity, Presentation Format and Literacy. Transpor-tation Research Part E, 39, 229–244.
Bech, M., T., Kjaer and J. Lauridsen, 2011. Does the number of choice sets matter? Re-sults from a web survey applying a discrete choice experiment. Health Economics, 28(3), 273–286.
Boxall, P., W.L., Adamowicz A., Moon, 2009. Complexity in choice experiments: choice of the status quo alternative and implications for welfare measurement. Aust J Agr Resour Eco-nomics 53, 503-519.
Bradley, M. and A., Daly, 1994. Use of the logit scaling approach to test for rank-order and fatigue effects in stated preference data. Transportation 21, 167–184.
Brouwer, R., T. Dekker, J. Rolfe and J., Windle, 2010. Choice Certainty and Consistency in Repeated Choice Experiments. Journal of Environmental and Resource Economics, 46(1), 93–109.
Bundesministerium für Umwelt, Naturschutz und Reaktorsicherheit (BMU), 2010. Der In-dikatorenbericht 2010 zur Nationalen Strategie zur biologischen Vielfalt. Berlin.
Caussade, S., J., Ortúzar, L.I., Rizzi and D.A., Hensher, 2005. Assessing the influence of design dimensions on stated choice experiment estimates. Transportation Research Part B, 39, 621–640.
Czajkowski, M., M., Giergiczny and W.H., Greene, 2012. Learning and Fatigue Effects Revisited. The Impact of Accounting for Unobservable Preference and Scale Heterogeneity on Perceived Ordering Effects in Multiple Choice Task Discrete Choice Experiments. Work-ing Paper 8, Faculty of Economics, University of Warsaw.
Chung, C., T., Boyer and S. Han, 2011. How many choice sets and alternatives are opti-mal? Consistency in choice experiments. Agribusiness, 27(1), 114-125.
Dellaert, B., J., Brazell and J.J., Louviere, 1999. The effect of attribute variation on con-sumer choice consistency. Marketing Letters, 10, 139-147.
Dellaert, B.G.C., B., Donkers and A., van Soest, 2012. Complexity Effects in Choice Ex-periment–Based Models. Journal of Marketing Research 49(3), 424-434.
DeShazo, J.R. and G., Fermo, 2002. Designing choice sets for stated preference meth-ods: the effects of complexity on choice consistency. Journal of Environmental Economics and Management 44, 123-143.
Galesic, M., 2006. Dropouts on the Web: Effects of Interest and Burden Experienced Dur-ing an Online Survey. Journal of Official Statistics, 22(2) 313–328.
Heiner, R., 1983. The origin of predictable behaviour. The American Economic Review, 73, 560–595.
Hensher, D.A., P.A., Stopher and J.J., Louviere, 2001. An exploratory analysis of the ef-fect of numbers of choice sets in designed choice experiments: an airline choice application. Journal of Air Transport Management, 7(6), 373–379.
Hensher, D.A., 2004. Identifying the Influence of Stated Choice Design Dimensionlity on Willingness to Pay for Travel Time Savings. Journal of Transport Economics and Policy 38(3), 425–446.
- 20 -
Working Paper on Management in Environmental Planning 33/2013
Hensher, D.A., 2006. Revealing Differences in Willingness to Pay due to the Dimension-ality of Stated Choice Designs: An Initial Assessment. Journal of Environmental and Re-source Economics 34(1), 7-44.
Hess, S., D.A., Hensher and A., Daly, 2012. Not bored yet – Revisiting respondent fatigue in stated choice experiments. Transportation Research Part A, 626–644.
Hoerger, M., 2010. Participant dropout as a function of survey length in internet-mediated university studies: implications for study design and voluntary participation in psychological research. Cyberpsychology, Behavior and Social Networking, 13(6), 697-700.
Hole, A.R., 2006. Small-sample properties of tests for heteroscedasticity in the conditional logit model. Economics Bulletin 3, 1-14.
Louviere, J.J., D.A., Hensher and J.D., Swait, 2000. Stated Choice Methods. Analysis and Application. Cambgridge.
McNair, B.J., J. Bennett and D.A., Henser, 2011) A comparison of responses to single and repeated discrete choice questions. Resource and Energy Economics 33(3), 554-571.
Ratcliffe, J. L., Longworth, 2002. Investigating the structural reliability of a discrete choice experiment within health technology assessment. International Journal of Technology As-sessment in Health Care, 18(1), 139–144.
Rolfe, J., and J. Bennett, 2009. The impact of offering two versus three alternatives in choice modelling experiments. Ecological Economics 68, 1140-1148.
Rose, J.M., D.A., Hensher, S., Caussade, J., Ortúzar and J.-C., Jou, 2009. Identifying dif-ferences in willingness to pay due to dimensionality in stated choice experiments: a cross country analysis. Journal of Transport Geography 17, 21-29.
Ryan, M. and S., Wordsworth, 2000. Sensitivity of Willingness to Pay Estimates to the Level of Attributes in Discrete Choice Experiments. Scottish Journal of Political Economy 47(5), 504–524.
Scarpa, R. and J.M., Rose, 2008. Design efficiency for non-market valuation with choice modelling: how to measure it, what to report and why. The Australian Journal of Agricultural and Resource Economics 52, 253-282.
Swait, J.D. and W.L Adamowicz, 2001. Choice environment, market complexity and con-sumer behaviour: a theoretical and empirical approach for incorporating decision complexity into models of consumer choice. Organizational Behaviour and Human Decision Processes, 86, 141–167.
Vicente, P. and E. Reis, 2010. Using Questionnaire Design to Fight Nonresponse Bias in Web Surveys. Social Science Computer Review, 28(2), 251-267.
Zhang, J. and W.L., Adamowicz, 2011. Unraveling the Choice Format Effect: A Context-Dependent Random Utility Model, Land Economics 87(4), 730-743.