uncertainty-based measurement quality control
Post on 03-Feb-2016
11 Views
Preview:
DESCRIPTION
TRANSCRIPT
GENERAL PAPER
Uncertainty-based measurement quality control
Hening Huang
Received: 8 August 2013 / Accepted: 24 December 2013
� Springer-Verlag Berlin Heidelberg 2014
Abstract According to a simple acceptance decision rule
for measurement quality control, a measured value will be
accepted if the expanded uncertainty of the measurements
is not greater than a preset maximum permissible uncer-
tainty. Otherwise, the measured value will be rejected. The
expanded uncertainty may be calculated as the z-based
uncertainty (the half-width of the z-interval) when the
measurement population standard deviation r is known or
the sample size is large (30 or greater), or by a sample-
based uncertainty estimator when r is unknown and the
sample size is small. The decision made based on the
z-based uncertainty will be deterministic and may be
assumed to be correct. However, the decision made based
on a sample-based uncertainty estimator will be uncertain.
This paper develops the mathematical formulations for
computing the probability of acceptance for two sample-
based uncertainty estimators: the t-based uncertainty (the
half-width of the t-interval) and an unbiased uncertainty
estimator. The risk of incorrect decision-making, in terms
of the false acceptance probability and false rejection
probability, is derived from the probability of acceptance.
The theoretical analyses indicate that the t-based uncer-
tainty may result in significantly high false rejection
probability when the sample size is very small (especially
for samples of size 2). For some applications, the unbiased
uncertainty estimator may be superior to the t-based
uncertainty for measurement quality control. Several
examples from acoustic Doppler current profiler stream-
flow measurements are presented to demonstrate the
performance of the t-based uncertainty and the unbiased
uncertainty estimator.
Keywords Measurement uncertainty �Maximum permissible uncertainty � Uncertainty estimator �Samples of size 2
Introduction
The problem of interest is that an unknown constant
quantity is measured n times by a measurement system to
give the observations of x1,…, xn. Assume the observations
only have random errors and follow a normal distribution;
the mean, denoted as l, of the observation population is the
true value of the quantity, and the sample mean �X is taken
as an estimate of l. That is, the sample mean is taken as the
measured value. The error of the measured value, though
may be unknown, is e ¼ �X � l. The standard uncertainty
and expanded uncertainty associated with the measurement
may be estimated based on the principles described in the
Guide to the expression of uncertainty in measurement
(GUM) [1]. The uncertainty, defined as the half-width of an
interval having a stated level of confidence (see p. 2 in [1])
(i.e., the coverage probability), is an indicator of the quality
of the measurement. The uncertainty may be attributed to
instrument noise and/or environmental factors (e.g., tur-
bulence). This paper deals with the statistical quality
control of measurements based on the uncertainty associ-
ated with the measurement, regardless of whatever
instrument or system is used in the measurement.
Having measured a quantity, one might decide to (a)
accept the measured value or (b) reject the measured
value, based on an acceptance criterion (i.e., a decision
rule) for measurement quality control. According to a
H. Huang (&)
Teledyne RD Instruments, 14020 Stowe Drive, Poway,
CA 92064, USA
e-mail: hhuang@teledyne.com
123
Accred Qual Assur
DOI 10.1007/s00769-013-1032-5
simple acceptance decision rule, a measured value will be
accepted if the expanded uncertainty U associated with
the measurement at a specified coverage probability 1-a(usually 95 %) is not greater than a preset maximum
permissible uncertainty (MPU)
U�MPU ð1Þ
When the population standard deviation r is known or the
sample size is large (30 or greater), the expanded uncer-
tainty U is often calculated as the z-based uncertainty (i.e.,
the half-width of the z-interval). The decision made based
on the z-based uncertainty will be deterministic and may be
assumed to be correct.
However, in complicated measurements, such as
acoustic Doppler current profiler (ADCP) streamflow
measurement, the population standard deviation is often
unknown and the sample size is often small; the z-based
uncertainty is not available. In this situation, the expanded
uncertainty U may be estimated by a sample-based
uncertainty estimator. However, the decision made based
on a sample-based uncertainty estimator will be uncertain;
it may be correct or incorrect. That is, there is a risk in the
decision-making. The risk may be measured by the false
acceptance probability or the false rejection probability,
which depends on the sample-based uncertainty estimator
used.
Two sample-based uncertainty estimators are available
in the literature. One is the well-known t-based uncertainty
(i.e., the half-width of the t-interval) and the other is an
unbiased uncertainty estimator (or called the Craig model
[2, 3]). Although the concept of false acceptance or false
rejection has been well known in statistics and has been
used in conformity assessment (e.g., [4, 5, 6]), to the
author’s knowledge, the formulations of the false accep-
tance probability and the false rejection probability
associated with the t-based uncertainty and the unbiased
uncertainty estimator in measurement quality control have
not been found in the literature.
The objective of this paper is to develop the mathe-
matical formulations for computing the probability of
acceptance associated with the t-based uncertainty and that
associated with the unbiased uncertainty estimator. The
false acceptance probability and the false rejection proba-
bility can be derived from the probability of acceptance.
Several examples from ADCP streamflow measurements
are presented to examine the performance of the t-based
uncertainty and the unbiased uncertainty estimator.
When r is known or sample size is large
When the population standard deviation r is known or the
sample size is large (30 or greater), the expanded
uncertainty U is calculated as the z-based uncertainty
(denoted as Uz). Note that, in practice, it is widely accepted
that Uz may be calculated with a sample of size equal or
greater than 30. Accordingly, the uncertainty-based
acceptance criterion is
Uz ¼ za=2
rffiffiffi
np �MPU ð2Þ
where za/2 is the z-score. Equation (2) applies for n = 1, 2,
3, …, including the single observation.
Maximum permissible uncertainty is an uncertainty-
based tolerance limit for the measurement in consideration.
Note that MPU is a constant and independent from the
sample size. Since the uncertainty Uz decreases with
increasing sample size, Eq. (2) can always be met with a
larger sample.
As long as Eq. (2) is met, the measurement error will not
be greater than MPU with a probability of 1-a or greater.
That is,
Pðjej �MPUÞ� 1� a ð3Þ
Let ±MPU denote (-MPU, ?MPU), which is a coverage
interval for the measurement error e. The coverage
probability of ±MPU depends on the ratio between MPU
and Uz. The ratio is defined as the measurement quality
index (MQI)
MQI ¼ MPU
Uz
ð4Þ
Note that MQI defined by Eq. (4) is different from the
measurement capability index Cm defined in conformity
assessment (e.g., [4, 5]) in which the numerator is maxi-
mum permissible error (MPE). Cm measures the
measurement quality (Uz) relative to a measurement
instrument error limit (MPE); it is an indicator of the
instrument’s measurement capability. MQI measures the
measurement quality (Uz) relative to a permissible mea-
surement uncertainty limit (MPU); it is an indicator of the
quality of measurement results. The higher the MQI, the
better the measurement quality. This can be seen by
examining the coverage probability of ±MPU as a function
of MQI. When MQI = 1, the coverage probability of
±MPU is 1-a, the same as the coverage probability of the
z-interval. When MQI [ 1, the coverage probability of
±MPU will be greater than 1-a. For example, when
MQI = 1.3143 and za/2 = 1.96 (thus 1-a = 95 %), the
coverage probability of ±MPU is 99 %. On the other hand,
if MQI \ 1, the coverage probability of ±MPU will be
smaller than 1-a; the measurement result does not meet
the desired precision requirement (MPU).
It is important to note that, according to Eq. (2),
whether a sample (i.e., a measured value) is acceptable
solely depends on the population standard deviation r and
Accred Qual Assur
123
the sample size n. That is, if Eq. (2) is met, the whole
population of observations will satisfy Eq. (3) and any
sample drawn from the population is acceptable. This can
be readily understood by considering a simple case that
water temperature is measured by a temperature sensor
and MPU is set at 0.01 �C. Assume there are no random
error sources in the water temperature measurement other
than the sensor’s noise. If the temperature sensor is cal-
ibrated with the expanded uncertainty Uz = za/2rB 0.01 �C at n = 1, any reading from the sensor’s single
measurement is acceptable. However, if the sensor is
calibrated with the expanded uncertainty Uz = za/2r B
0.02 �C at n = 1, four observations (n = 4) must be
made to meet MPU = 0.01 �C.
When r is unknown and sample size is small
When the population standard deviation r is unknown and
the sample size is small (less than 30), the expanded
uncertainty U may be estimated by a sample-based
uncertainty estimator. Two sample-based uncertainty esti-
mators are considered in this paper: the t-based uncertainty
and an unbiased uncertainty estimator.
The t-based uncertainty, denoted as Ut, is written as
Ut ¼ ta=2 s=ffiffiffi
np
, where s is the sample standard deviation
and ta/2 is the t-score. When Ut is used, the uncertainty-
based acceptance criterion becomes
Ut ¼ ta=2
sffiffiffi
np �MPU ð5Þ
It should be pointed out that the t-based uncertainty is not
an unbiased estimate of the z-based uncertainty. It exhibits
significantly high positive bias error and precision error
when the sample size is very small [2, 3, 7].
The unbiased uncertainty estimator is written as
za=2 s= ðc4
ffiffiffi
npÞ, where c4 is the bias correction factor for
s (e.g., [8]). The author proposed the unbiased uncertainty
estimator for estimating the uncertainty of ADCP stream-
flow measurements in 2006 [9]. Two years later, the author
discovered through Internet search that the unbiased
uncertainty estimator is exactly the same as the first-order
approximation of the Craig’s approach to the probable
error of a mean [10]. Detailed discussion on the unbiased
uncertainty estimator can be found in [2, 3]. The unbiased
uncertainty estimator is denoted as Uz/c4 and is called the
z/c4-based uncertainty hereafter. In addition, Jenkins [7]
developed an empirical, unbiased uncertainty estimator
that is nearly the same as Uz/c4.
When Uz/c4 is used for measurement quality control, the
uncertainty-based acceptance criterion becomes
Uz=c4 ¼ za=2
s
c4
ffiffiffi
np �MPU ð6Þ
Note that Eqs. (5) or (6) applies for n = 2, 3, …. They do
not apply for the single observation.
According to Eq. (5), a sample with its s [ MPU �ffiffiffi
np
=ta=2 will be rejected. According to Eq. (6), a sample
with its s [ MPU � c4
ffiffiffi
np
=za=2 will be rejected. Since
ta/2 [ za/2/c4, the samples rejected based on the t-based
uncertainty will always be more than the samples rejected
based on the z/c4-based uncertainty.
To visualize how the samples are accepted or rejected
according to Eqs. (5) or (6), we employed the Monte Carlo
simulation to generate 5000 pairs of error e and sample
standard deviation s. The Monte Carlo simulation involved
randomly drawing samples of size n from a normally dis-
tributed population with l = 50 (any unit) and r = 5
(same unit as l). Two sample sizes, n = 2 and n = 4, were
considered because our discussions focused on very small
samples. The simulation was implemented using an Excel
spread sheet to generate 2 9 5000 random numbers for
n = 2 and 4 9 5000 random numbers for n = 4. The error
e and the sample standard deviation s were calculated for
each sample.
Figure 1a, b shows the scatter plots of |e| and s (nor-
malized by r = 5) from the Monte Carlo simulation for
n = 2 and n = 4, respectively. The MPU is assumed to
be 1.3143Uz at the 95 % coverage probability (za/2 =
1.96). Thus, the normalized MPU is 1.822 for n = 2 and
1.288 for n = 4. The indications of the samples rejected
(or accepted) according to Eqs. (5) and (6) are shown in
the figures. Note that the measurement quality index
MQI = 1.3143 in this situation so that, due to MQI [ 1,
all of the samples from the Monte Carlo simulation are
acceptable according to Eq. (2). However, as can be seen
in Fig. 1, many samples are rejected according to Eqs. (5)
or (6), which, apparently, are false rejections. Therefore,
there is an uncertainty in the quality control when using
Eqs. (5) or (6). In order to examine the uncertainty, we
define the probability that the sample satisfies either Eqs.
(5) or (6) as the probability of acceptance, Pa. The
probability of acceptance Pa associated with the t-based
uncertainty is
Pa ¼ Pta=2ffiffiffi
np s�MPU
� �
ð7Þ
Substituting MPU ¼ MQI � za=2r=ffiffiffi
np
into Eq. (7) yields
Pa ¼ Ps
r�MQI �
za=2
ta=2
Þ ¼ Pðffiffiffiffiffiffiffiffiffiffiffi
n� 1p s
r�MQI �
ffiffiffiffiffiffiffiffiffiffiffi
n� 1p za=2
ta=2
� �
ð8Þ
Denote a random variable x ¼ffiffiffiffiffiffiffiffiffiffiffi
n� 1p
s=r that follows the
chi distribution. Thus, the probability of acceptance Pa is
the same as the cumulative probability function of x. It is
calculated as
Accred Qual Assur
123
Pa ¼cðk=2; x2=2Þ
Cðk=2Þ ð9Þ
where k = n-1 is the degree of freedom and c (.) is the
lower incomplete gamma function.The probability of
acceptance Pa associated with the z/c4-based uncertainty is
Pa ¼ Pza=2
c4
ffiffiffi
np s � MPU
� �
ð10Þ
Substituting MPU ¼ MQI � za=2r=ffiffiffi
np
into Eq. (10) yields
Pa ¼ Ps
r�MQI � c4
� �
¼ Pffiffiffiffiffiffiffiffiffiffiffi
n� 1p s
r�MQI �
ffiffiffiffiffiffiffiffiffiffiffi
n� 1p
c4
� �
ð11Þ
When MQI C 1, all of the samples drawn from the mea-
surement population are supposed to be accepted according
to Eq. (2), and the probability of acceptance Pa should be
100 %. However, when using Eqs. (5) or (6), the proba-
bility of acceptance Pa will be less than 100 %. In this
situation, 1-Pa is the false rejection probability or the
probability of the Type I error. On the other hand, when
MQI \ 1, all of the samples are supposed to be rejected
according to Eq. (2), and the probability of acceptance Pa
should be zero. However, when using Eqs. (5) or (6), the
probability of acceptance Pa will not be zero. In this situ-
ation, Pa becomes the false acceptance probability or the
probability of the Type II error.
We first examine a case in which MQI [ 1. Figure 2
shows the probabilities of acceptance associated with the
t-based uncertainty and the z/c4-based uncertainty for
MQI = 1.3143 with the nominal coverage probability
1-a = 95 %.
In this case (MQI = 1.3143), the probability of accep-
tance should be 100 %. However, it can be seen from
Fig. 2 that the probability of acceptance associated with the
t-based uncertainty is very small for small samples, only
16.1 % at n = 2. It increases with increasing sample size
and the increase is rapid. The probability of acceptance is
76.8 % at n = 10 and approaches 100 % when n [ 60. On
the other hand, the probability of acceptance associated
with the z/c4-based uncertainty is 70.6 % at n = 2 and
90.1 % at n = 10; it approaches 100 % when n [ 60. Note
that in this case, 1-Pa is the false rejection probability. For
the samples of size 2, the t-based uncertainty would result
in 84 false rejections out of 100 samples; the z/c4-based
uncertainty would result in 30 false rejections out of 100
samples. The results indicate that the z/c4-based uncertainty
is conservative, whereas the t-based uncertainty is overly
conservative and even misleading when the sample size is
very small.
Next, we examine a case in which MQI \ 1. Figure 3
shows the probabilities of acceptance associated with the
t-based uncertainty and the z/c4-based uncertainty for
0
1
2
3
4
5N
orm
aliz
ed e
rror
||/
Normalized sample standard deviation s/
Rejected according to Eq. (6)
Rejected according to Eq. (5)
(a)
0
1
2
3
4
5
0 1 2 3 4
0 1 2 3 4
Nor
mal
ized
err
or |
|/
Normalized sample standard deviation s/
(b)
Rejected according to Eq. (5)
Rejected according to Eq. (6)
Fig. 1 Samples rejected (or accepted) according to Eqs. (5) and (6)
with MPU = 1.3143Uz at the 95 % coverage probability: (a) n = 2;
(b) n = 4. The dashed line is |e| = MPU, normalized by r
0
10
20
30
40
50
60
70
80
90
100
0 20 40 60 80 100
Pro
babi
lity
of a
ccep
tanc
e P a
(%)
Sample size n
t-based uncertainty
z/c4-based uncertainty
Fig. 2 Probability of acceptance Pa for MQI = 1.3143. In this case,
1-Pa is the false rejection probability
Accred Qual Assur
123
MQI = 0.5 with the nominal coverage probability
1-a = 95 %.
In this case (MQI = 0.5), the probability of acceptance
should be zero. However, it can be seen from Fig. 3 that Pa is
not zero for small samples. It is 6.2 % at n = 2 and less than
1 % when n [ 7 for the t-based uncertainty. It is 31 % at
n = 2, 7.3 % at n = 10, and less than 1 % when n [ 10 for
the z/c4-based uncertainty. Note that in this case, Pa becomes
the false acceptance probability, and 1-Pa is the rejection
probability. For the samples of size 2, the t-based uncertainty
would result in 6 false acceptances out of 100 samples; the
z/c4-based uncertainty would result in 31 false acceptances
out of 100 samples. The results are expected because the
quality control based on the t-based uncertainty will be
always more conservative (overly conservative for very
small samples) than that based on the z/c4-based uncertainty.
Finally, we examine the case in which MQI = 1. Fig-
ure 4 shows the probabilities of acceptance associated with
the t-based uncertainty and the z/c4-based uncertainty for
MQI = 1 with the nominal coverage probability
1-a = 95 %.
In this case (MQI = 1), the probability of acceptance
should be 100 %. However, it can be seen from Fig. 4 that
the probability of acceptance associated with the t-based
uncertainty is only 12.3 % at n = 2. It increases with
increasing sample size, but the increase is slow. The
probability of acceptance is only 33.8 % at n = 10 and
45.1 % at n = 100. It approaches 50 % when the sample
size n goes to infinity. On the other hand, the probability of
acceptance associated with the z/c4-based uncertainty is
57.5 % at n = 2; it approaches 50 % as the sample size
increases. This is expected because the z/c4-based uncer-
tainty is an unbiased estimate of the z-based uncertainty.
When the sample size is large, the estimated uncertainty
will approach the z-based uncertainty and fluctuate about it.
That is, half of the total samples will have an estimated
uncertainty, that is, slightly greater than the z-based
uncertainty and the other half slightly smaller than the z-
based uncertainty. As a result of the fluctuation, the prob-
ability of acceptance becomes 50 %. Note again that, in
this case, 1-Pa is the false rejection probability. For the
samples of size 2, the t-based uncertainty would result in
88 false rejections out of 100 samples; the z/c4-based
uncertainty would result in 43 false rejections out of 100
samples.
However, if MQI is just slightly less than unity, the
discussions on the MQI = 1 case should be reversed. Pa
becomes the false acceptance probability, and 1-Pa
becomes the rejection probability. Thus, in the situation
that MQI is near unity, the false rejection probability is
significantly higher than the false acceptance probability
when using the t-based uncertainty for very small samples.
It becomes less significantly high for large samples. On the
other hand, the false rejection probability and the false
acceptance probability are about the same (i.e., about
50/50 %) when using the z/c4-based uncertainty regardless
of the sample size (except for very small samples).
We further examine the performance of the two sample-
based uncertainty estimators with fixed sample sizes.
Figure 5a, b, and c shows Pa as a function of MQI at
1-a = 95 % for n = 2, 4, and 32, respectively. Note again
that 1-Pa is the false rejection probability for MQI C 1,
and Pa is the false acceptance probability for MQI \ 1. It
can be seen from Fig. 5a that, for sample size 2, using the t-
based uncertainty will result in much higher false rejection
probabilities than the z/c4-based uncertainty. The false
rejection probability associated with the t-based uncer-
tainty is very high, as high as 64.4 %, even at MQI = 3.
The situation improves for sample size 4 (Fig. 5b). When
0
10
20
30
40
50
60
70
80
90
100
0 20 40 60 80 100
Pro
babi
lity
of a
ccep
tanc
e P a
(%)
Sample size n
t-based uncertainty
z/c4-based uncertainty
Fig. 4 Probability of acceptance Pa for MQI = 1. In this case, 1-Pa
is the false rejection probability
0
10
20
30
40
50
0 10 20 30 40 50
Pro
babi
lity
of a
ccep
tanc
e P a
(%)
Sample size n
t-based uncertainty
z/c4-based uncertainty
Fig. 3 Probability of acceptance Pa for MQI = 0.5. In this case, Pa
becomes the false acceptance probability and 1-Pa is the rejection
probability
Accred Qual Assur
123
the sample size is large, say, n = 32 (Fig. 5c), the risk in
making an incorrect decision associated with the t-based
uncertainty and the z/c4-based uncertainty gets closer.
In addition, it should be pointed out that the coverage
probability of ±MPU defined by Eq. (3) is the ‘‘true’’
confidence level an operator may have for a measured
value. Note that the coverage probability of ±MPU is
independent from a sample-based uncertainty estimator.
That is, whatever sample-based uncertainty estimator, with
any nominal coverage probability, is used, the coverage
probability of ±MPU is the same. Although the t-based
uncertainty has a greater coverage probability than the z/c4-
based uncertainty (assume the coverage probabilities are
estimated by the same random-interval procedure), it may
not warrant a higher level of confidence for making correct
decisions in measurement quality control. It should also be
pointed out that, in the z/c4-based uncertainty approach, the
reduction of false rejection probability (due to reduction of
coverage probability) is achieved on account of increasing
the risk of a false acceptance.
In practice, the preference of the t-based uncertainty or
the z/c4-based uncertainty may depend on a specific prob-
lem or application. It may be resolved based on an analysis
of expected losses resulting from false decisions (false
rejection or false acceptance). In ADCP streamflow mea-
surement quality control, there is no specific ‘‘loss’’
associated with a less accurate discharge data that are
falsely accepted. Moreover, our experiences indicate that,
in most situations, a false acceptance is unlikely because
the measurement quality index MQI associated with ADCP
streamflow measurements is often greater than unity. On
the other hand, if a measured discharge that is supposedly
accepted was falsely rejected, more repeated measurements
would be required, resulting in unnecessarily additional
costs of labor, time, and energy. Therefore, the z/c4-based
uncertainty is preferred in the quality control of ADCP
streamflow measurement.
Examples
This section presents several examples of quality evalua-
tion and control for ADCP streamflow measurements. First,
we analyze a large data set. Then, we analyze a number of
small data sets.
A large data set
A large data set for the ADCP streamflow measurements in
the Mississippi River on January 30, 1992, was available
from Gordon [11]. The data set contains 30 discharge
observations under a steady flow condition. The average
discharge of the 30 observations, 14240 m3/s, is assumed
to be the true discharge, and the standard deviation with a
bias correction, 223 m3/s, is assumed to be the population
standard deviation.
It should be pointed out that this large data set was from
an exceptional experiment. The ADCP streamflow mea-
surements under a steady flow condition usually involve
four observations [12, 13]. This large data set offers an
opportunity to evaluate the performance of the t-based
0
10
20
30
40
50
60
70
80
90
100P
roba
bilit
y of
acc
epta
nce
Pa
(%)
Measurement quality index (MQI )
t-based uncertainty
z/c4-based uncertainty
(a)
0
10
20
30
40
50
60
70
80
90
100
Pro
babi
lityo
f acc
epta
nce
P a(%
)
Measurement quality index (MQI )
t-based uncertainty
z/c4-based uncertainty
(b)
0
10
20
30
40
50
60
70
80
90
100
0 0.5 1 1.5 2 2.5 3
0 0.5 1 1.5 2 2.5 3
0 0.5 1 1.5 2 2.5 3
Pro
babi
lityo
f acc
epta
nce
P a(%
)
Measurement quality index (MQI)
(c)
t-based uncertainty
z/c4-based uncertainty
Fig. 5 Probability of acceptance Pa as a function of MQI at
1-a = 95 %: (a) n = 1, (b) n = 4, and (c) n = 32
Accred Qual Assur
123
uncertainty and the z/c4-based uncertainty for small sam-
ples as compared to the z-based uncertainty that is
available. We consider two and four observations (n = 2
and 4) only. We group the data in sequence into samples of
size 2 and 4 (n = 2 and n = 4), obtaining 29 and 27
samples, respectively. Note that the way that the data is
grouped does not produce independent samples. However,
the correlation between the samples is not a concern for the
measurement quality evaluation discussed here.
Since the population standard deviation is known, the
z-based uncertainty for each sample size can be calculated
and Eq. (2) can be used to evaluate the measurement
quality. The relative MPU for ADCP streamflow mea-
surement is 4.3 % [14]. The z-based uncertainty (relative to
the assumed true discharge) at the 95 % coverage proba-
bility is calculated as 3.06 % for n = 1, 2.17 % for n = 2
and 1.53 % for n = 4, all of which are smaller than the
relative MPU 4.3 %. Thus, any measured discharge (i.e.,
any sample, regardless of the number of observations)
meets Eq. (2) and is acceptable.
The measurement quality index is calculated as
MQI = 1.4052, 1.9816, and 2.8105, for n = 1, 2, and 4,
respectively. Accordingly, the coverage probabilities of the
relative ±MPU (4.3 %) are 99.4116, 99.9887, and
99.999998 %. Thus, as expected, the quality of the mea-
surement results get higher as the number of observations
increases.
We then pretend that the population standard deviation
is unknown and use Eqs. (5) and (6) to evaluate the mea-
surement quality. The t-based uncertainty and the z/c4-
based uncertainty (also relative to the assumed true dis-
charge) at the nominal coverage probability 95 % for
n = 2 and n = 4 are calculated. Figure 6a and b shows the
results. The relative z-based uncertainty at the coverage
probability 1-a = 95 % and the relative MPU 4.3 % are
also shown in the figures.
It can be seen from Fig. 6a that, for n = 2, there are 21
samples whose relative t-based uncertainties are greater
than the relative MPU 4.3 %. Therefore, using the t-based
uncertainty for the measurement quality control will result
in 21 false rejections. On the other hand, there are three
samples whose relative z/c4-based uncertainties are greater
than the relative MPU 4.3 %. Therefore, using the
z/c4-based uncertainty will result in 3 false rejections. The
results indicate that the z/c4-based uncertainty is conser-
vative, whereas the t-based uncertainty is overly
0
5
10
15
20
25
30
35
40R
elat
ive
unce
rtai
nty
(%)
Sample (n=2)
t-based uncertaintyz/c4-based uncertaintyz-based uncertainty: 2.17 %MPU: 4.3 %
(a)
0
1
2
3
4
5
6
7
8
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
Rel
ativ
e un
cert
aint
y (%
)
Sample (n=4)
t-based uncertaintyz/c4-based uncertaintyz-based uncertainty: 1.53 %MPU: 4.3 %
(b)
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
Fig. 6 Measurement quality evaluation based on uncertainty analysis
for Mississippi River ADCP discharge measurements: (a) n = 2 and
(b) n = 4
Table 1 Data sets of ADCP streamflow measurements at 7 sites
Site
#
River site Observations
(discharge in m3/s)
#1 #2 #3 #4
1a Yangtze River at Yichang,
China (300 kHz ADCP)
11234 11582 11485 11476
1b Yangtze River at Yichang,
China (150 kHz ADCP)
11092 10980 11367 11319
2a Aksu River, Antalya, Turkey
(600 kHz ADCP)
38.88 40.30 38.23 39.78
2b Aksu River, Antalya, Turkey
(2 MHz ADCP)
37.96 38.78 38.85 39.05
3 Manavgat River, Antalya,
Turkey (600 kHz ADCP)
228.7 225.9 233.1 228.4
4 Kopru River, Antalya, Turkey
(600 kHz ADCP)
96.60 96.53 93.90 91.79
5 Irrigation Canal at Imperial
Irrigation District, California
(2 MHz ADCP)
3.99 4.02 4.05 4.14
6 Irrigation Canal at Angoori
Barrage, India (2 MHz
ADCP)
22.33 22.07 22.3 22.19
7 River Elbe near Hamburg,
Germany (1200 kHz ADCP)
4588 4576 4631 4517
The author involved in the data collection at Sites 1, 5, and 6. The
data sets for Sites 2, 3, and 4 are from Michel [15]; the data sets for
Site 7 are from Gordon [16]
Accred Qual Assur
123
conservative and even misleading because it results in so
many false rejections. This is consistent with the theoreti-
cal analysis for the probability of acceptance.
It can be seen from Fig. 6b that, for n = 4, all of the
relative t-based or z/c4-based uncertainties are smaller than
the relative MPU 4.3 %, yielding 100 % acceptance. The
results indicate that both of the t-based uncertainty and the
z/c4-based uncertainty become less conservative for n = 4
than n = 2. This is also consistent with the theoretical
analysis for the probability of acceptance.
Small data sets
Table 1 shows 9 data sets (each contains four observations)
of the ADCP streamflow measurements at 7 sites. Since the
population standard deviation is unknown at these sites,
Eq. (2) cannot be used for the measurement quality eval-
uation and control. Instead, Eqs. (5) and (6) are used.
Table 2 shows the sample mean �X and standard deviation
s, the z/c4-based uncertainty Uz/c4, and the t-based uncer-
tainty Ut for the first two observations (n = 2) and those for
the total four observations (n = 4). The uncertainties are
shown in percentage, relative to the sample means.
It can be seen from Table 2 that, for four observations
(n = 4), the z/c4-based uncertainties and the t-based
uncertainties are all smaller than the relative MPU 4.3 %;
therefore, all samples are acceptable. However, for two
observations (n = 2), one out of the 9 z/c4-based uncer-
tainties and 7 out of the 9 t-based uncertainties are greater
than the relative MPU 4.3 %. These are apparently the
false rejections.
The above examples suggest that the z/c4-based uncer-
tainty is in general conservative for measurement quality
control. It is appropriate even if the sample size is only two.
The t-based uncertainty is overly conservative and may be
misleading for very small samples (e.g., many false
rejections for the two observations).
The above examples also suggest that two observations
may be enough to meet the relative MPU 4.3 % criterion
for the ADCP streamflow measurements, and it is unnec-
essary to make four observations. Considering the fact that
hydrologists around the world may conduct hundreds of
thousands of ADCP streamflow measurements each year, it
will lead to significant savings in labor and time, and in gas
consumption (many measurements are conducted using gas
engine boats) if the number of observations is reduced to
two.
Conclusions
The uncertainty-based measurement quality control cannot
be deterministic unless the z-based uncertainty is available.
The quality control will be uncertain when a sample-based
uncertainty estimator is used. The false rejection proba-
bility and the false acceptance probability depend on the
uncertainty estimator used. A good estimator should pro-
vide a balance between the false rejection and false
acceptance, i.e., the balance between making the Type I
and Type II errors.
Both of the theoretical analyses and examples suggest
that the measurement quality control based on the t-based
uncertainty is overly conservative and may be misleading
when the sample size is very small. This is because the
t-based uncertainty is not an unbiased estimate of the
z-based uncertainty and exhibits significantly high positive
bias error and precision error when the sample size is very
small. Therefore, the t-based uncertainty is inappropriate
for measurement quality control for small samples. On the
other hand, the measurement quality control based on the
z/c4-based uncertainty is, in general, conservative. For some
applications, the z/c4-based uncertainty may be superior to
the t-based uncertainty. The reason why the z/c4-based
uncertainty is appropriate, even for very small samples, is
Table 2 Sample mean and standard deviation, and estimated uncertainties (at the nominal coverage probability 95 %) for the data sets shown in
Table 1
Site # Two observations (n = 2) Four observations (n = 4)
�X (m3/s) s (m3/s) Uz/c4 (%) Ut (%) �X (m3/s) s (m3/s) Uz/c4 (%) Ut (%)
1a 11408 246 3.75 19.38 11444 148 1.38 2.06
1b 11036 79.5 1.25 6.47 11189 184.2 1.75 2.62
2a 39.6 1.00 4.40 22.78 39.3 0.92 2.50 3.73
2b 38.4 0.58 2.62 13.53 38.7 0.48 1.32 1.98
3 227.3 1.92 1.47 7.60 229.0 2.97 1.38 2.07
4 96.6 0.05 0.09 0.48 94.7 2.31 2.60 3.89
5 4.01 0.021 0.92 4.76 4.05 0.065 1.70 2.55
6 22.2 0.18 1.44 7.44 22.2 0.12 0.57 0.85
7 4582 8.49 0.32 1.66 4578 47.02 1.09 1.63
Accred Qual Assur
123
that the z/c4-based uncertainty is an unbiased estimate of
the z-based uncertainty.
Very small samples, even the samples of size 2, are still
useful when the z/c4-based uncertainty is employed for
measurement quality control. In contrast, the t-based
uncertainty basically rules out the usefulness of very small
samples, especially the samples of size 2. The use of the
z/c4-based uncertainty for measurement quality control
makes it possible to make reasonable, less incorrect deci-
sions based on a small number of observations. This may
lead to a potential reduction in the number of observations
required for measurement quality assurance and control,
which may have a great significance for time and labor
consuming or costly measurements or experiments.
Acknowledgments The author would like to thank the anonymous
reviewers for their valuable comments that helped improve the paper.
The author also would like to thank Mark Kuster of Pantex Metrology
for making the copy of the Jenkins paper available to the author and
for stimulating discussions.
References
1. Joint Committee for Guides in Metrology (JCGM) (2008) Eval-
uation of measurement data—Guide to the expression of
uncertainty in measurement (GUM 1995 with minor corrections).
Sevres, France. http://www.bipm.org/utils/common/documents/
jcgm/JCGM_100_2008_E.pdf
2. Huang H (2010) A paradox in measurement uncertainty analysis.
Presented at the 2010 Measurement Science Conference in Pas-
adena, CA, Mar 25–26. In: Global Measurement: Economy and
Technology 1970–2010 Proceedings (DVD)
3. Huang H (2012) Comparison of uncertainty calculation models.
Cal Lab Int J Metrol 19(1):24–29
4. Joint Committee for Guides in Metrology (JCGM) (2012) Evalu-
ation of measurement data—The role of measurement uncertainty
in conformity assessment. Sevres, France. http://www.bipm.org/
utils/common/documents/jcgm/JCGM_106_2012_E.pdf
5. Pendrill LR (2007) Optimized measurement uncertainty and
decision-making in conformity assessment. NCSLI Meas
2(2):76–86
6. Pendrill LR (2008) Operating ‘cost’ characteristics in sampling
by variable and attribute. Accred Qual Assur 13:619–631
7. Jenkins JD (2007) The Student’s t-distribution uncovered. Mea-
surement Science Conference Proceedings, Long Beach
8. Wadsworth HM Jr (1989) Summarization and interpretation of
data. In: Wadsworth H M Jr (ed) Handbook of statistical methods
for Engineers and Scientists, 2.1–2.21. McGraw-Hill, New York
9. Huang H (2006) Analysis of random error in ADCP discharge
measurement I: model for predicting random uncertainty.
J Hydraul Eng 37(5):619–624 (in Chinese)
10. Craig CC (1927) Remarks on the probable error of a mean. Am
Math Mon 34(9):472–476
11. Gordon RL (1992) Mississippi river discharge. RD Instruments
Report, San Diego, California
12. Mueller DS, Wagner CR (2002) Measuring discharge with
acoustic Doppler current profilers from a moving boat. USA.
Geological Survey Techniques and Methods, 3A-22, 72
13. Oberg KA, Morlock SE, Caldwell WS (2005) Quality-assurance
plan for discharge measurements using acoustic Doppler current
profilers. Scientific investigations rep. 2005-5183, U.S. Geolog-
ical Survey (USGS), Reston, VA
14. Huang H (2008) Estimating precision of moving boat ADCP dis-
charge measurement. In: Proceedings (DVD) of 14th Australian
Hydrographers Association Conference: Water Initiatives 2008
and Beyond, Australia Hydrographers Association, Northcote
15. Michel L (2011) Ekoton, Turkey Visit Report. Teledyne RD
Instruments Europe, La Gaude, France
16. Gordon RL (1989) Acoustic measurement of river discharge.
J Hydraul Eng ASCE 115(7):925–936
Accred Qual Assur
123
top related