Web-based tool for expert elicitation of

the variogramPhuong Truong, Gerard Heuvelink, John Paul Gosling

Wageningen University (NL)University of Leeds (UK)

UncertWeb Workshop, Muenster 10 Sep, 2012

• Introduction of Statistical Expert Elicitation• Expert Elicitation in UncertWeb Project• Expert Elicitation for the Variogram• Introduction of the Web-based tool • Introduction of the Case study• Demo of the Web-based tool

Outline

Statistical expert elicitation

• Process of reliably extracting and formulating a person’s knowledge and beliefs about uncertain quantities into probability distributions.

• Combination of mathematical probability theory, cognitive psychology and decision theory

• Experts – who have qualified knowledge about the uncertain quantities.

Expert elicitation procedure involves six steps

1. Background and preparation2. Identify and recruit expert(s)3. Motivating and training the expert(s), e.g. to avoid

overconfidence and anchoring4. Structuring and decomposition (ensure that expert

agrees with how the problem is structured)5. The elicitation itself6. Assess the adequacy of the elicitation

• Eliciting beliefs about the quantities of interest• Fitting of an appropriate distribution (parametric or

non parametric fitting)• Feedback of implications of fitted distribution –

preferably graphical• Revision of judgements until satisfied• Pooling multiple experts’ opinions - two main ways:

“behavioural” and “mathematical”

Within the elicitation itself

• Sheffield ELicitation Framework (SHELF): http://www.tonyohagan.co.uk/shelf/.

• MATCH Uncertainty elicitation tool: http://optics.eee.nottingham.ac.uk/match/uncertainty.php.

• Software: – Elicitator– ElicitN– EXCALIBUR

Some elicitation tools

• To assess uncertainties in inputs and parameters of the model web where data and observations do not provide enough information by themselves.

• To provide web-based tools for quantifying experts beliefs about continuous, categorical and spatially continuously distributed variables - all univariate.

• See Deliverable 3.1 of WP3 for more information http://www.uncertweb.org/documents/deliverables

Expert Elicitation in UncertWeb Project

The ElicitatorQuantifying multiple experts' beliefs about continuous & categorical variables

http://elicitator.uncertweb.org

• Demand for estimation of the variogram when few or no observations available.

• Demand for a priori variogram in Bayesian geostatistics and spatial sampling optimization.

• Expert knowledge – an important source of information about spatial variability of environmental variables.

• Elicitation protocol and tool to facilitate eliciting the variogram from expert knowledge not available.

Expert elicitation for the variogram

Elicitation protocol for the variogram:

Round 1• Elicitation of the marginal probability distribution at

random location using bisection method. • Elicit the finite bounds min and max:

– P(<min) = P(>max) = 0

• Elicit the three quartiles1, 2 and 3:

– P(<1) = 0.25 – P(<2) = 0.5– P(<3) = 0.75

(Source: Oakley, 2010)

Elicitation protocol for the variogram:

Round 2

Web-based tool

http://www.variogramelicitation.org/

Case study: surface temperature in the Netherlands

• Maximum daily surface temperature on April 1st, 2020.

• Spatially continuously distributed variable at fixed time slot.

• Access the URL: http://www.variogramelicitation.org

• Everyone is given username and password to login

• Carefully read the information provided in the websites

• Answer the questions in given orders

Let’s try the live demo

Thank you for your attention!

The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under

grant agreement n° [248488].

• http://surveys.ifgi.de/

• Part B.2

Evaluation