distinguishing between analytical precision and assessment … · 2017. 2. 1. · accuracy and...
TRANSCRIPT
-
Distinguishing between analytical
precision and assessment accuracy
in relation to materials
characterisation
Steven Pearce| Principal environmental scientist Perth
-
Presentation Title
Presentation overview
• Heterogeneity, glossing over the elephant in the data
room
• Paradigm of the lab as the provider of accurate data
• Statistics and data smoothing (professional lies)
• Materials characterisation, adopting a rational approach
Materials characterisation
-
Presentation Title
Materials characterisation
• Geochemical and geophysical properties
• Defining specific characteristics that are taken to be
representative of the bulk properties of the material
• Classifying materials into groupings based on
characterisation process
• Examples:
• Landfill waste classification
• Contaminated sites assessment
• Mine waste classification
Materials characterisation
-
Presentation Title
General problems of carrying out
material characterisation studies
• Remote location of sites
• Time pressure (everyone has a schedule), and ultimately
time is money
• Need an answer quickly, not recommendations for more
testing
• Size of site, volume of material to sample
• Heterogeneity of materials that require sampling
End result: Defining bulk properties using “snapshots”
Materials characterisation
-
Presentation Title
Bulk properties
Particle size distribution
Point source effects
Geological controls
Weathering
Geochemical controls
Fractionation
Mine Closure Materials characterisation
-
Presentation Title
Characterisation
Concentration
Distribution
Volume
Mine Closure Materials characterisation
-
Presentation Title
Heterogeneity is the measure of the degree of
compositional variability of a material. This can be divided
into
• inter sample (macro scale): Concentration may occur
within a particular material (for example in a vein of
primary mineralisation), at a particular location, or at a
particular depth (point source contamination)
• intra sample (micro scale): e.g. various mineral phases
may be present and unequally distributed (for example
isolated macro pyrite crystals)
Materials characterisation
Defining bulk properties, a scalar problem
-
Presentation Title
Intra sample variability shown from
multiple XRF results from a single sample
(approx 50g dry weight)
*50g sample split into 3 parts: bulk, 2mm (coarse)
Composite
lab result
within 10%
of mean of
XRF
results
-
Presentation Title
• Typically, for analysis the lab will extract a small 1-10g
sub sample of the 1000g parent sample on which to
complete analysis
• selection of the sub sample may take place after sieving
or crushing of the parent sample.
• Therefore, if intra sample heterogeneity is significant
then clearly it will be unlikely that a single 1-10g sub
sample will be representative (chemically or
mineralogicaly) of the sample as a whole.
• Bias introduced at early stage
Mine Closure
Intra sample variability (analysis bias)
Materials characterisation
-
Presentation Title
• Contamination may be concentrated within a particular
material, at a particular location, or at a particular depth
• If random sampling is being employed then it is clear that
inter sample heterogeneity will have a potentially
significant impact on the ability of the sampling
programme to characterise the distribution of
contamination on site
Mine Closure
Inter sample variability
Materials characterisation
-
Presentation Title
Inter sample variability (90 sample data
base)
Copper results:
Note wide inter
sample variability
probably not
captured the full
sample population
distribution range
Very large
distribution
“tail”
Materials characterisation
-
Presentation Title
Very different terms although commonly mixed up.
• Precision is the repeatability of a testing method
• Accuracy is a reflection of how well the testing characterises
the sample composition
Testing a 10g sub sample of material in a laboratory may
therefore yield high precision result if a duplicate test is
completed on the same 10g sub sample
however if a separate 10g sample of the material is tested
very likely that intra sample variability will be introduced
that will result in increased error (i.e. reduce accuracy).
Mine Closure
Accuracy and precision
Materials characterisation
-
Presentation Title
• No analytical technique is 100% precise and so random and systematic
errors will affect the final result
• Generally, the error introduced by modern analytical instrumentation as
analytical bias is likely to be relatively low.
• A study carried out in Europe (CLAIRE technical bulletin 7) indicates that for
a particular case study sampling was by far the greatest cause of uncertainty
rather than analysis.
• Precision was estimated at 83% of the concentration value for the
sampling method, but was much lower at 7.5% for analytical method.
• The overall random component of uncertainty was estimated as being
83.6%, that is to say, the value of any concentration for an individual
location was reproduced to within ± 83.6% of the quoted value (at 95%
confidence).
• Given that analytical precision was only 7.5%, then clearly the majority of
the overall variability was related to sampling rather than analytical
factors.
Mine Closure
Accuracy and precision
Materials characterisation
-
Presentation Title Mine Closure
Lithological characterisation, vertical
sampling intervals
-20
0
20
40
60
80
100
0 25 50 75 100 125 150 175 200 225 250
Depth (m)
Est NAPP
NA
PP
(k
g H
2S
O4/t
on
ne
Materials characterisation
-
Presentation Title Mine Closure
Elevated heavy metals not related to
sulfur…
-20
0
20
40
60
80
100
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
0 25 50 75 100 125 150 175 200 225 250
NA
PP
(kg
H2SO
4/t
on
ne
)
Depth (m)
Mercury
Est NAPP
Metal x
Materials characterisation
-
Presentation Title Mine Closure
Lithological characterisation, vertical
sampling intervals
-50
0
50
100
150
200
250
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
0 25 50 75 100 125 150 175 200 225 250
Depth (m)
Mercury
NAG pH 7
Est NAPP
Metal x
Materials characterisation
-
Presentation Title
Traditional “collect and analyse”
Environmental Sampling: Aiming for high
precision on limited sample numbers
1000 m3 500g 5g
Typical volume of
material represented by
one site sample
Typical weight of
sample collected
Typical weight of
laboratory sub-
sample that is
analysed
Intra/inter
sample
variability
Intra
sample
variability
Materials characterisation
-
Presentation Title
Heterogeneity testing (used in mining industry
for grade control)
• Coarse fragment ores (stockpile, mill feed etc)
• 50-100 individual fragments picked one by one
• Each assayed to extinction
• Consecutive results graphed
• Often can see that removal of top few results will drop
mean grade by orders of magnitude
• Caused by heterogenity, small pockets of high grade
material in general low grade background
• Applies to contaminated sites, similar distributions
Materials characterisation
-
Presentation Title
Heterogeneity testing profile
0
50
100
150
200
250
0 10 20 30 40 50 60 70 80
mg/
kg
HT group
Inclusion of top 5
results considerable
increases mean grade
Materials characterisation
-
Presentation Title
Worldwide sampling and testing standards (e.g. ASTM, USEPA,
UK EA, DEC Australia) based on the premise that collecting
limited samples from the field and using laboratory to analyse to
high precision provides the most “accurate” data
Based on assumption that
• Only laboratory derived data is acceptable
• That analytical precision is the cause of most sampling error
• That statistics can “fill” the data gaps left by low sampling density.
This assumption is flawed however as multiple studies have
shown that sample variability (heterogeneity) has the greatest
impact on accuracy, and that statistics do a poor job of
interpolation (i.e. Data gap filling).
The solution is increase to sampling density
Mine Closure
Current paradigm “The lab as the provider of
accurate information”
Materials characterisation
-
Presentation Title
Paradigm reflected in guidance docs
NEPC 1999: 4.7 FIELD TESTING
• “A variety of field testing devices may be used as a
limited contribution to the screening of samples on
contaminated sites”.
• “The role in providing real-time data needs to be
augmented by chemical analysis of soil. Their use as the
sole source of analytical data in the assessment of
potentially contaminated sites is inappropriate as they
may give falsely high or low results”.
Materials characterisation
-
Presentation Title
Mine Closure
Embedded assumptions
Precision - measures the reproducibility of measurements under a given set of conditions. The
precision of the data is assessed by calculating the Relative Per cent Difference (RPD) between
duplicate sample pairs.
200(%)
do
do
CC
CCRPD
Where Co = Analyte concentration of the original sample
Cd = Analyte concentration of the duplicate sample
The Environmental Consultant will adopt nominal acceptance criteria of 30% RPD for field duplicates
and splits for inorganics, and nominal acceptance criteria of 50% RPD for field duplicates and splits
for organics, however it is noted that this will not always be achieved, particularly in heterogeneous
soil or fill materials, or at low analyte concentrations.
Question: If analytical techniques are precise why such
high acceptable RPDs? And why therefore is field
testing “unacceptable”?
There is inconsistent logic in this approach which is
embedded in the industry
Materials characterisation
-
Presentation Title
• A given site (or location) will have a given sample
population distribution that cant be known (without
testing every gram of material)
• As we can never know the ‘true’ sample population,
taking a few soil samples from the site is therefore is
akin to ‘random’ sampling as nothing prior is known
about the population distribution
• Generally the more samples that are analysed from a
given site (or location) the greater the confidence in the
overall assessment. The direct relationship between
increasing levels of confidence with sample numbers
comes down to simple statistics.
Mine Closure
Statistical considerations
Materials characterisation
-
Presentation Title
• Guidance on the number of samples to be taken on a site for the
purposes of contaminated sites assessment based on assumptions
that do not generally apply to most sites.
• The key assumptions include that the occurrence of contamination is
described by normal distribution, and that hotspots present are of
uniform size, shape and vertical profile.
• Common for the 95th percentile value to be quoted by assessors
and requested by regulators as a representative concentration for a
contaminant upon which to base decisions (to portray an illusion of
statistical certainty).
• However, in reality the probability of being able to define anything
close to the true 95th percentile representative concentration for a
site is very unlikely when sampling at densities similar to that
recommended by published guidance
Mine Closure
Statistical considerations
Materials characterisation
-
Presentation Title
• Statistics commonly cited as a method to account for variability and
to allow for interpolation to fill data gaps (e.g. US95, outlier tests, non
parametric analysis etc)
• However data created in this way is prone to large error and can
introduce bias into interpretation of data sets.
• Many broad assumptions are made, most commonly analysis
techniques assume a normal distributed data set. Problem is most
data sets are not normally distributed, and in the majority of
instances the data set is to small to define the true mean, median,
and minimum/maximum values.
• Increasing sampling frequency is the only way to accurately fill “data
gaps” and therefore to reduce the error in calculation of descriptive
statistics (mean, minimum etc)
Mine Closure
Statistics (the magic data gap filler)
Materials characterisation
-
Presentation Title
Generated concentration profile, however
apparent inter sample variability is in fact likely
to be intra sample heterogeneity
Materials characterisation
-
Presentation Title
Increased sampling density
using on site screening
Intra sample
variability assessment
completed on sub
samples (as little as
1g material required)
10 or more
samples
analysed 1000m3
Materials characterisation
-
Presentation Title
2D contour plots produced from XRF
data to show distribution of metals
-
Presentation Title
Inter sample variability (90
sample data base)
Copper results:
Note wide inter
sample variability
probably not
captured the full
sample population
distribution range
Very large
distribution
“tail”
-
Presentation Title
Intra sample variability shown from XRF
results from a single sample location
Intra sample variability
0
200
400
600
800
1000
1200
1400
Co
pp
er
[mg
/kg
]
Bulk fraction
Fine fraction
Coarse fraction
Lab
Note: Up to
800 mg/kg
variance
*50g sample split into 3 parts: bulk, 2mm (coarse)
Composite
lab result
within 10%
of mean of
XRF
results
-
Presentation Title
Characterising the distribution of
contamination, the case for more samples
rather than higher precision
Sample population range: Likely to be larger than defined
Intra sample variability >10% of inter sample variability,
difficult to differentiate between the two in some samples
and therefore determine what is the cause of spatial
variation in concentration
Laboratory results rely on compositing, unclear at what
scale is this acceptable given the level of intra sample
variability (>100%)
If we had relied on limited lab samples alone the
distribution would be even more poorly characterised as a
result of smaller data set, and compositing, preventing
understanding of intra sample variability Materials characterisation
-
Presentation Title
Conclusions
Limitations of standard methodology of taking less
samples but aiming for higher precision (lab ICP)
• Logistical and cost implications of taking physical samples
and sending to laboratory
• Poor understanding of intra sample variability
• Very easy to assume variability between samples is a
function of lateral or vertical distribution, could be simply be
a function of intra sample variability
• Low probability of defining the sample population range and
true mean, but a high chance of thinking you have
• Therefore: Not an ideal method for defining areas or
volumes of material as contaminated
Materials characterisation
-
Presentation Title
www.ghd.com