This document describes a study that used a choice experiment and contingent grouping format to value environmental attributes of Mount Jaizkibel in Spain. Follow-up questions were used to capture respondents' uncertainty and difficulty. On average, respondents reported a medium-high level of certainty (5.75 out of 7) and tasks were moderately easy (3.33 out of 7). Those reporting different certainty levels across tasks showed lower certainty and higher variability. Tasks were perceived as more difficult for early screens. Response times averaged slightly over 27 seconds.

1. Allowing for Uncertainty in Choice Experiments
Dolores Garcia ∗
Departament d'Economia Aplicada
Universitat de les Illes Balears
E-07122 Palma de Mallorca, Spain
e-mail: dolores.garcia@uib.es
David Hoyos
Instituto de Economía Pública
Edificio Central, 4ª. Planta
Avda. Lehendakari Aguirre, 83
48015 Bilbao
e-mail: david.hoyos@ehu.es
Pere Riera
Departament d’Economia Aplicada
Universitat Autònoma de Barcelona
Edifici B, Campus UAB
08193 Bellaterra, Spain
e-mail: pere.riera@uab.es
January, 2008
Abstract
In recent years choice modeling methods have gained popularity among the group of
valuation techniques used to elicit the population’s preferences on non-market goods.
Despite the large number of available applications, to our knowledge available studies
have not explicitly addressed the problem of WTP uncertainty. This paper is based on
the results of an exercise that combines a choice experiment and a contingent grouping
format. Follow-up questions were distributed in two sub-samples in order to capture
either the degree of certainty or the level of difficulty of the choice set, as perceived by
respondents. Also, the time spent to fulfill the tasks was accounted for, for each of the
choice sets faced by respondents. The main results are reported, comparing choice and
contingent grouping elicitation formats. Also, we investigate whether marginal values
are affected by reported uncertainty. The exercise was applied in the Basque Country
and Navarra in order to value a set of key environmental attributes representative of the
natural area of Jaizkibel.
JEL Classification: Q51
∗
Corresponding author

2. 1. Introduction
The valuation of environmental goods has received a great deal of attention in
the last decades. Among the methods used for this purpose, stated preference (SP)
techniques have become very popular, and numerous methodological advances have
taken place.
One of the topics that has received attention is how uncertainty might affect
estimated values. Thus, uncertainty might be present in the definition of the provided
good, the effects of a particular environmental policy, or even in the person’s
knowledge about her true willingness to pay for the environmental change (Li and
Mattsson, 1995). Several studies have dealt with this topic in the CV literature (Berrens
et al., 2002; Burton et al., 2003; Alberini et al., 2003; Vossler and Poe, 2005). The usual
way of incorporating uncertainty into respondents’ answers is to allow them to state the
degree of certainty with which they answered. Depending on their answers, then
responses are included or not in the estimation process that leads to the calculation of
marginal values.
This approach has been considered in this paper. A choice modeling framework
has been developed to value a set of environmental attributes characteristics of mount
Jaizkibel, a natural area on the Cantabrian Sea, near the French border (Bateman et al,
2002). In particular, both a choice experiment and a contingent grouping exercise are
developed. Choice experiments have been widely applied for the valuation of
environmental amenities, and are probably the most popular among the group of choice
modeling methods. A contingent grouping approach was also used (Brey et al, 2005).
Under the latter, individuals are asked to group the alternatives in sub-sets of those
which are better that the status quo or business-as-usual alternatives, and those which
are worse.

3. Follow-up questions were included to obtain information about the degree of
certainty as perceived by respondents, or alternatively, about the difficulty when
responding to the choice set scenario. We analyze the results of such questions, and
discuss the extent to which every single respondent discriminates in their answers, how
reported answers vary with the particular elicitation format used, and how marginal
values are modified when uncertainty is accounted for.
The structure of the paper is as follows. Next section briefly describes the site
object of the valuation exercise, and the survey design. In section 3 the main results of
the debriefing questions are presented, with three subsections: that devoted to the
certainty question, another one dedicated to the difficulty question, and a final
subsection in which how times of responses vary under different scenarios are given.
Section 4 first provides the results of a main effects conditional logit model, then
discuss the new estimation outcomes when the result of the certainty question is
interacted with the cost attribute and included as explanatory variable. The final section
includes the main conclusions of the paper.
2. Description of the exercise
2.1. Description of the site and attributes
The application was developed in order to value the relevant environmental
characteristics of mount Jaizkibel, a 2.400 hectares natural site that contains 15 zones
declared of high ecological interest by the European Union, situated in the municipality
of Pasaia (Basque Country). The environmental characteristics of this site might be
affected if the project to build a new port in the outside of the bay of Pasaia, under the
hills of mount Jaizkibel, is undertaken. According to a recent study, the construction of
the new seaport would provoke some critical impacts (Pozueta 2004), including cliffs
destruction, loss of vegetable cover and loss of seabed and local beaches, among other.

4. Detailed information about the environmental characteristics of Jaizkibel can be found
in Hoyos et al. (2008)
In order to identify the key environmental attributes of mount Jaizkibel and the
business-as-usual levels of provision, interviews with experts and focus groups were
undertaken. It was concluded that the most important ones were landscape, flora,
avifauna and seabed. In particular, landscape was defined as the percentage surface
from which today’s landscape could be seen in the future; flora was measured by the
future level of protection of today’s population of armeria euskadiensis, an endemism
of basque seacoast; avifauna, described in terms of the future level of protection of
today’s population of lesser and peregrine falcon; and seabed, measured by the future
level of protection of today’s extension of red algae. Pictures were used to illustrate the
different levels of all four attributes (see figure 1). To decide upon the levels of the cost
attribute, focus groups together with a pilot survey including open-ended contingent
valuation questions were used. The payment consisted of an annual contribution to a
Foundation exclusively dedicated to protecting mount Jaizkibel, that all Basque citizens
would be required to make. Table 1 summarizes the levels of all the attributes used.
Table 1. Attributes and levels considered
Attribute Level
Landscape 40%* 60% 80% 100%
Flora 50%* 70% 85% 100%
Fauna 25%* 50% 75% 100%
Seabed 50%* 70% 85% 100%
Annual payment 5€ 10 € 15 € 20 € 30 € 50 € 100 €
*BAU alternative levels

5. Combining all these attributes and levels, near two thousand different
combinations were obtained (44x71). As it is usually done when the universe of
alternatives is very large, statistical design methods were used to simplify the choice
sets construction (Louviere, Henser, and Swait, 2000). A main effects fractional
factorial design with second order interactions reduced the number of alternatives to 96
pairs of protection alternatives. The profiles were grouped in 24 blocks of four sets
containing two alternative protection programmes (programs A and B), plus the
business as usual option. Also, the “don’t know” option was included in order to avoid
the “yea saying” bias (Arrow et al. 1993). Then, each individual was shown 4 screens.
Two of them asked for the most preferred alternative (choice), and the remaining two
required their grouping as explained in the introductory section. The order in which
choice and grouping were combined was randomly picked.
2.2. Questionnaire and data collection
A questionnaire was developed to simulate a market in which individuals would
be willing to choose among varying levels of protection of the attributes. The
questionnaire was finally structured in three parts. The first part described the main
attributes and, the current situation of mount Jaizkibel and the potential future damages
to its environmental attributes. The second part (preference elicitation part) contained
the choice experiment and contingent grouping questions. An example of a card
including a choice set and the screen the interviewee saw is shown in figure A.1 in the
annex.
The last section collected some debriefing and socioeconomic questions,
including those aimed at investigating about the degree of certainty in responses and the
perceived level of difficulty when choosing or grouping alternatives. Thus, after each

6. set of alternatives had been shown, a follow-up question was added. A split sample
approach was used. Half the sample was asked to state, in a scale from 1 to 7, how sure
they felt when making their choice or groups. The other half was asked to state how
difficult the choice or grouping had been, again in a scale from 1 to 7.
Interviews were conducted face-to-face at people’s homes, using laptop
computers. Respondents could read the screens and listen to a recorded voice. Answers
could be typed in by themselves or dictated to the interviewer, at respondents’
discretion. This way of administering the questionnaire allowed for the possibility of
collecting information about the time individuals took to make their choices or grouping
decisions. The relevant population considered was the population from the Basque
Autonomous Community and Navarra in Spain as well as some French cities next to the
Spanish border, accounting for 2.5 million people being at least 18 years old. The pilot
was conducted in October 2006, while the final survey was undertaken between
November and December, 2006. A stratified random sample of 636 individuals was
selected from the relevant population. The strata used included age, gender and size of
the town of residence, following official statistical information (EUSTAT). In each
location, the questionnaires were distributed using random survey routes.
3. Descriptive statistics
The sample accurately represented the population in terms of age, gender and
income levels. Even though the questionnaire included a number of questions asking for
the respondents’ view on the importance of the included attributes, their involvement in
recreational activities, and many other, here we mostly focus on the results of the
follow-up questions dealing with uncertainty and difficulty issues. For other outcomes,
see Annex II of Hoyos et al. (2007).

7. 3.1. Certainty in choices and grouping tasks
The follow-up question asked the respondent how “sure” she felt with the
choice/grouping just provided, in a scale of 1 to 7, 7 representing the highest level of
security. The means for the four follow-up questions resulted in values comprised
between 5.5 and 6, representing a medium-high level of confidence. The overall mean
was 5.75 (Table 2), and responses did not vary between the choice and the grouping
elicitation formats.
Dummy variables were created in order to capture whether the respondent had
felt sure about the response (when answered 5, 6 or 7) or unsure (when answered 1,2, 3
or 4).
Table 2. Degree of certainty . Screens 1 to 4, and average.
Mean (Std.Dev.)
(values 1 to 7) % of unsure responses
1st screen 5.78 (1.57) 16.98%
2nd screen 5.63 (1.55) 22.28%
3rd screen 5.79 (1.44) 15.66%
4th screen 5.84 (1.44) 14.45%
Average 5.76 (1.50) 18.30%
When analyzing individual responses, it was found that a 54.61% of respondents
did not change the value of certainty with the cards, that is, they picked the same value
for all four choice sets. The remaining 45.39% did provide a different value depending
on the particular card just seen. Mean values when considering those who do and who
do not discriminate in their certainty answers are different. In average, those individuals

8. who do modify the degree of certainty in response show a smaller mean and a larger
standard deviation (table 3).
Table 3. Degree of certainty according to individuals’ responses
Mean (Std.Dev.)
Respondent answered… (values 1 to 7) % of unsure responses
Different scores 5.14 (1.62) 31’88%
Same scores 6.26 (1.22) 6,02%
When working on the experimental design, dominated profiles and profiles’ sets
in which attributes’ levels did not vary tried to be eliminated or reduced (Huber and
Zwerina, 1996). However, a small number of choice sets remained in which the levels
of some attribute were the same. We describe these sets as easy. Although they
represent a small number, it results that the mean degree of certainty in responses is
higher (6), compared to difficult sets (5.75).
Table 4. Degree of certainty depending on the difficulty of the choice set
Mean (Std.Dev.)
Profiles’ set (values 1 to 7) % of unsure responses
Easy 6 (1.14) 17’64%
Difficult 5.75 (1.51) 18,32%
3.2. Difficulty of choices and grouping tasks
As shown in the annex, some respondents faced follow-up questions after the
choice and grouping screens, asking for the degree of difficulty of the tasks of choosing
or grouping the alternatives, and the answer again followed a closed format, in a scale

9. going from 1 to 7 (1 meaning the task had been very easy, 7 meaning it had been very
difficult). The main results are reported here.
Table 5 summarizes the mean scores from the first to the fourth task. There was
little variation in average means, results did not change either when grouping and choice
elicitation formats were compared. The average score resulted in 3.33, and so tasks
could be considered as moderately easy.
Answers were also recoded in a binary way: when the stated degree of difficulty
was 1,2, 3 or 4, the task was considered not difficult, and difficult for scores above 4.
The percentage of difficult was above 37% for the first choice set, and diminished as the
subsequent screens were shown, resulting in 31.68% for the last choice set.
Table 5. Degree of difficulty. Screens 1 to 4, and average.
Mean (Std.Dev.)
(values 1 to 7) % of difficult responses
1st screen 3.41 (2.13) 37.12%
2nd screen 3.32 (2.08) 32.67%
3rd screen 3.39 (2.13) 32.17%
4th screen 3.27 (2.13) 31.68%
Average 3.33 (2.12) 33.17%
With respect to the certainty question, the percentage of individuals who did not
keep the same answer for all four choice sets was larger, achieving a 62.69%. Only the
remaining 37.31% answered the same in all instances. The reported average degree of
difficulty was
Table 6. Degree of difficulty according to individuals’ responses
Mean (Std.Dev.)

10. Respondent gave… (values 1 to 7) % of difficult responses
Different scores 3.39 (1.93) 34.50%
Same scores 3.19 (2.42) 32.30%
Finally, it would be expected that easy choice sets, as described in the previous
subsection, would result easier for respondents. As table 7 shows, this was the case for
choice tasks; however, for contingent grouping tasks the average reported score resulted
higher for presumably easier tasks.
Table 7. Degree of difficulty depending on the difficulty of the choice set
Mean
Profiles’ set (values 1 to 7)
Choice experiment Contingent grouping
Easy 3 3.53
Difficult 3.4 3.27
3.3. Times of response
As explained in the previous section, we used a computer-aided questionnaire in which
responses were automatically stored in a dataset, and in which the time spent in
choosing from or grouping the alternatives was stored.
The average amount of time used to fulfill the task of choosing or grouping the
alternatives was slightly above 27 seconds. Grouping the alternatives took an average of
29’8 seconds, while for choices 24’8 seconds were required. The histogram in figure 1
represents the percentage of responses for different time intervals.

11. Figure 1. Histogram for time intervals
45
40
35
30
25
20
15
10
5
0
<15 15-30 30-45 45-60 60-120 >2
seconds seconds seconds seconds seconds minuts
% of cases
In almost 40 per cent of the cases respondents answered in less than 15 seconds,
and this proportion rises to more than 70 per cent for those under the 30 second
threshold. More than 90 per cent of cases individuals invested less than 1 minute in
fulfilling the task of choosing or grouping.
Also, the time of response clearly diminished as choice sets were presented. For
the first choice of profiles shown, individuals took an average of 45 seconds to answer,
while they invested about one third of such time for the last choice set (17 seconds). The
corresponding standard deviations diminish as well. Table 8 summarizes some of the
commented results.
Table 8. Time of response for choice/grouping screens 1 to 4, and average
Mean (Std.Dev.)
(figures in seconds)
1st screen 45.03 (51)
2nd screen 25.91 (32)
3rd screen 20.38 (21)
4th screen 17.09 (16)

12. Grouping
29.75 (39)
Average 27.40 (34.27) Choices
24.84 (31)
Combining the fact that respondents were progressively faster when providing
their responses, and that they invested some more time in the grouping of alternatives,
the times of response have been compared. Thus, average times of response have been
found for grouping and choosing tasks, differentiating the cases in which individuals
first faced the grouping screens and then the choice ones, or the other way around. The
following table summarizes the results.
Table 9. Time of response (seconds) depending on task and sequence of of tasks
Mean (Standard deviation)
Choice 1st, Grouping 2nd Grouping 1st, Choice 2nd
Choice 32.26 (38) 16.79 (15)
Grouping 20.58 (22) 39.22 (49)
As expected, in all instances the first two tasks take more time than the second
ones. The reduction of time is greater when the grouping screens appear first, with
average times going from almost 40 seconds (for grouping) to around 17 (for choice).
The shortest times correspond to the case in which choices appear in the second place
(approximately 17 seconds), and the longest to the instance in which grouping screens
go first (39.22 seconds).
Finally, times of response were compared taking into account whether the
profiles’ set was easy or not. There was a small difference of 1.31 seconds in favor of

13. difficult sets, which took less time. This somehow unexpected result is driven by
differences in times for choice tasks: grouping tasks were performed quicker when the
set of alternatives were “easy”, but with choice tasks it happened the other way around.
4. Estimation results considering uncertainty
In order to carry out the estimation of the values of the attributes at stake, the
database was depurated. In this section, only choice responses are considered. An 8 per
cent of interviewees declared not having understood the objective of the program to
preserve mount Jaizkibel, and were not shown the choice sets. Similarly, protests to the
valuation questions took place for 171 individuals, representing the 26.8% of the total
sampled population. After all the aforementioned rejections, 415 observations out of the
original 637 remained valid for estimation purposes. Finally, since a split sample
procedure was used to include follow-up questions, the number of valid observations in
which certainty questions have been included diminishes to 304.
As explained above, the attributes included in the choice sets were the
percentage values of landscape preservation, flora, avifauna and seabed. All of them
were treated as continuous variables. A conditional logit model was specified, including
the attributes in the terms explained above, and considering all main effects. The model
including a constant for the business-as-usual alternative was also considered, but the
restricted model was accepted according to the test of the likelihood ratio test.
Table 10. Conditional logit model (model 1)
Attribute Coefficient (t-ratios)
Landscape 0.0246 (5.786)
Flora 0.0160 (3.074)
Avifauna 0.131 (3.622)

14. Seabed 0.0070 (1.813)
Payment -0.0206 (-5.705)
Log-likelihood function: -243.6517
Observations: 304
The results show that all variables resulted significant (for seabed, the
coefficient is significant at a 10% significance level), and the expected signs were
found.
A second model specification including information on the reported certainty of
response was included, for the exact same observations. The reported scored was
recoded into a binary variable capturing whether the choice had been made with
certainty or not, as explained above. Then, this variable was interacted with the cost
payment. The objective was to see whether uncertainty could help explain choices, in
the first place, and see how the resulting marginal values would be affected. The results
of the estimation appear in table 11.
Table 11. Conditional logit model (model 2) considering uncertainty
Attribute Coefficient (t-ratios)
Landscape 0.0246 (5.796)
Flora 0.0161 (3.087)
Avifauna 0.0137 (3.774)
Seabed 0.0072 (1.873)
Payment -0.0185 (-5.034)
(Payment)*(choice is uncertain) -0.0191 (-1.858)
Log-likelihood function: -241.6445
Observations: 304

15. Again all attributes’ coefficients result significant, and so results the interaction
including certainty of the choice made. The values corresponding to WTP for each
attribute, or implicit prices, could be calculated out of the coefficients in table 11. They
are included in the following table.
Table 12. Annual WTP for Jaizkibel attributes (in € of 2007/person-year)
Model 1 Model 2
Attribute Marginal WTP Marginal WTP
Certain Uncertain
Landscape 1.194 1.330 0.654
Flora 0.777 0.870 0.428
Avifauna 0.636 0.741 0.364
Seabed 0.340 0.389 0.191
The values included in the table correspond to 1% marginal changes in the
attribute’s preservation. The values corresponding to model 1 would be the resulting
ones when all choices are considered, and the information on the declared level of
certainty in the response is ignored. When using the information on whether the choice
was a sure one or not, willingness to pay is affected by certainty. Thus, when choices
are certain, willingness to pay is larger for all four attributes. In other words, deleting
the “uncertain” responses from the dataset from which the models are estimated would
lead to greater marginal WTPs.
5. Conclusions
This paper constitutes an attempt to include information on the degree of
certainty of individuals, when making their choices, into the statistical analysis of data
resulting from a choice modeling exercise. The incipient results suggest that values
might result affected by feeling of confidence in the responses, and this poses the
question of whether choices should be included when respondent do not feel sure about

16. them. Variation in resulting values might be of importance when calculating welfare
changes, especially at a more aggregate level.
Other interesting issues have to do with the interpretation of uncertainty and how
it might relate to other aspects of the exercise, including the level of difficulty of the
task at stake. The results show that respondents felt rather sure about their responses,
and that they found the tasks easy enough not to be considered difficult (in average). On
the other hand, the percentages of instances in which the tasks were fulfilled with
uncertainty were considerably lower than the proportion of tasks considered difficult.
While both types of responses could be related, the issue needs further investigation.
The results were obtained from different sub-samples, too, so comparisons are not
straightforward. Likewise, the relationship between the times of response and the degree
of uncertainty in the response should be analyzed.
The analysis here introduced had the drawback, however, that the final model
specifications were made on a relatively low number of observations, for the sake of
comparison, given that split samples were used for choice and contingent grouping
formats, and also for certainty and difficulty questions. An immediate object of further
research consists in extending the analysis by undertaking the joint estimation of choice
and grouping answers.
Also, more accurate tests on the significance of the differences in resulting
scores should be used to compare results dealing with uncertainty and difficulty. Also,
the exploitation of the information provided by recording the times spent to responding
might prove extremely useful to help understand the cognitive burden of alternative
choice modeling approaches such as contingent grouping and choice experiments.

17. Acknowledgements
The authors acknowledge the financial support from the Department of Environment of
the Basque Government, and from CICYT Project No. SEJ2004-00143/ECON and
CEDEX project from the Spanish government.
Bibiliography
Alberini, A., K.Boyle and M.Welsh (2003) “Analysis of Contingent Valuation
Data with Multiple Bids and Response Options Allowing Respondents to Express
Uncertainty”, Journal of Environmental Economics and Management, vol. 45, pp. 40-
62
Arrow K. et al. (1993). “Report of NOAA Panel on Contingent Valuation”.
Bateman, I. J., Carson, R. T., Day, B., Hanemann, M., Hanley, N., Hett, T.,
Jones-Lee, M., Loomes, G., Mourato, S., Özdemiroglu, E., Pearce, D. W., Sugden, R.,
& Swanson, J. (2002). “Economic valuation with stated preference techniques: a
manual” Cheltenham, UK: Edward Elgar.
Berrens, R.P. et al. (2002) “Further Investigation of Voluntary Contribution in
Contingent Valuation: Fair Share, Time of Contribution, and Respondent Uncertainty”,
Journal of Environmental Economics and Management, vol. 44, pp. 144-68.
Brey R., O. Bergland O. and P. Riera (2005) “A contingent Grouping Approach
for Stated Preferences”, Working Paper #22/2005, Norwegian University of Life
Sciences.
Burton, A.C. et al. (2003) “An Experimental Investigation of Explanations for
Inconsistencies in Responses to Second Offers in Double Referenda”, Journal of
Environmental Economics and Management, vol. 46, pp. 472-89.
Hausman J. and D. McFadden (1984) “Specification tests for the multinomial
logit model”, Econometrica, vol 52, pp. 1219-1240.
Hoyos, D., P.Riera, M.C. Gallastegui, J. Fernández-Macho and D. Garcia (2008)
“Choice modelling applied to assessing environmental impacts of transport
infrastructures: the case of Pasaia’s new seaport, Spain”. Manuscript
Hoyos, D., P.Riera, M.C. Gallastegui, J. Fernández-Macho and D. Garcia (2007)
“Informe final: Valoración Económica del entorno natural del monte Jaizkibel”. Final
report for the Environmental Council of the Basque Government.

18. Huber, J., and K. Zwerina (1996). The importance in utility balance in efficient
choice designs. Journal of Marketing Research, 33, 307-317.
Louviere J., D. A. Henser, and J. Swait (2000) “Stated choice methods: analysis
and applications”. Cambridge University Press, Cambridge.
Li, C. and L. Mattsson (1995) “Discrete Choice under Preference Uncertainty:
An Improved Structural Model for Contingent Valuation”, Journal of Environmental
Economics and Management, vol. 28, pp. 256-69.
Pozueta J. (2004). “Estudio comparado de las alternativas de desarrollo del
Puerto de Pasajes en relación con su grado de impacto medio-ambiental y
sostenibilidad.” Departamento para el Desarrollo Sostenible. Diputación Foral de
Gipuzkoa.
Vossler, C.A. and G.L. Poe (2005) “Analysis of Contingent Valuation Data with
Multiple Bids and Response Options Allowing Respondents to Express Uncertainty: A
Comment”, Journal of Environmental Economics and Management, vol. 49, pp. 197-
200.

19. Annex
Figure A.1. Example of choice set in the valuation exercise

20. Figure A.2. Screen showing the questions about the degree of certainty in the response

21. Figure A.3. Screen showing the question about the degree of difficulty in the response