a unified approach for assessing agreement
DESCRIPTION
A Unified Approach for Assessing Agreement. Lawrence Lin, Baxter Healthcare A. S. Hedayat, University of Illinois at Chicago Wenting Wu, Mayo Clinic. Outline. Introduction Existing approaches A unified approach Simulation studies Examples. Introduction. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/1.jpg)
A Unified Approach for Assessing Agreement
Lawrence Lin, Baxter Healthcare A. S. Hedayat, University of Illinois at
Chicago Wenting Wu, Mayo Clinic
![Page 2: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/2.jpg)
Outline
IntroductionExisting approachesA unified approachSimulation studiesExamples
![Page 3: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/3.jpg)
Introduction Different situations for agreement
Two raters, each with single readingMore than two raters, each with single readingMore than two raters, each with multiple readings• Agreement within a rater• Agreement among raters based on means• Agreement among raters based on individual
readings
![Page 4: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/4.jpg)
Existing Approaches (1)
Agreement between two raters, each with single reading
Categorical data: • Kappa and weighted kappa
Continuous data: • Concordance Correlation Coefficient (CCC)• Intraclass Correlation Coefficient (ICC)
![Page 5: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/5.jpg)
Existing Approaches (2)
Agreement among more than two raters, each with single reading
Lin (1989): no inferenceBarnhart, Haber and Song (2001, 2002): GEEKing and Chinchilli (2001, 2001): U-statisticsCarrasco and Jover (2003): variance components
![Page 6: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/6.jpg)
Existing Approaches (3)
Agreement among more than two raters, each with multiple readings
Barnhart (2005)• Intra-rater/ inter-rater (based on
means) /total (based on individual observations) agreement
• GEE method to model the first and second moments
![Page 7: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/7.jpg)
Unified Approach
Agreement among k (k≥2) raters, with each rater measures each of the n subjects multiple (m) times.Separate intra-rater agreement and inter-rater agreementMeasure relative agreement, precision, accuracy, and absolute agreement, Total Deviation Index (TDI) and Coverage Probability (CP)
![Page 8: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/8.jpg)
Unified Approach - summary
Using GEE method to estimate all agreement indices and their inferencesAll agreement indices are expressed as functions of variance componentsData: continuous/binary/ordinaryMost current popular methods become special cases of this approach
![Page 9: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/9.jpg)
Unified Approach - model
Set up
subject effect subject by rater effect error effect
rater effect
ijlijjiijl ey ),0(: 2
i
),0(: 2eijle
k
jj
1
0
),0(: 2ij
ni ,...,2,1
kj ,...,2,1
ml ,...2,1
)1()(1
1 1'
2'
2
kkk
j
k
jjjj
mj ,...,2,1
![Page 10: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/10.jpg)
Unified Approach - targets
Intra-rater agreement: overall, are k raters consistent with themselves?
Inter-rater agreement: Inter-rater agreement (agreement based on mean): overall, are k raters agree with each other based on the average of m readings?Total agreement (agreement based on individual reading): overall, are k raters agree with each other based on individual of the m readings?
![Page 11: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/11.jpg)
Unified Approach – agreement(intra)
: for over all k raters, how well is each rater in reproducing his readings?
]...,|)1()([
])1()([1
,,2,11
2.
1
2.
,
tindependenyyymyyE
myyE
ijmijij
m
lijijl
m
lijijl
Intrac
222
22
e
Intrac,
![Page 12: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/12.jpg)
Unified Approach – precision(intra) and MSD
: for any rater j, the proportion of the variance that is attributable to the subjects (same as )Examine the absolute agreement independent of the total data range:
Intra
Intrac,
22'
2 2)( eijlijl yyEMSD
![Page 13: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/13.jpg)
Unified Approach – TDI(intra) : for each rater j, % of observations are within unit of their replicated readings from the same rater.
is the cumulative normal distribution is the absolute value
||)2
11(1
)(
Intra
21 2)2
11( e
)( Intra )100*()( Intra
|.|
![Page 14: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/14.jpg)
Unified Approach – CP(intra)
: for each rater j, of observations are within unit of their replicated readings from the same rater
)( Intra %100*)( Intra
)]2/(1[21 2)( eIntra
![Page 15: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/15.jpg)
Unified Approach – agreement(inter)
: for over all k raters, how well are raters in reproducing each others based on the average of the multiple readings?
],,...,,|)1()([
)]1()([
1
..2.11
2...
1
2...
,
tindependenyyykyyE
kyyE
ikii
k
jiij
k
jiij
Interc
2222
2
/
me
Interc,
![Page 16: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/16.jpg)
Unified Approach – precision(inter) : for any two raters, the proportion of the variance that is attributable to the subjects based on the average of the m readings
)var()var(
),cov(
'..
'..
ijij
ijijInter
yy
yy
me /222
2
Inter
![Page 17: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/17.jpg)
Unified Approach – accuracy(inter)
: how close are the means of different raters:
2222
222
, /
/
m
m
e
eIntera
InteraInterInterc ,,
Intera,
![Page 18: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/18.jpg)
Unified Approach – TDI(inter)
: for overall k raters, % of the average readings are within unit of the replicated averaged readings from the other rater.
)100*(
||)2
11(1
)(
Inter
me /222)2
11( 2221
)( Inter
)( Inter
![Page 19: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/19.jpg)
Unified Approach – CP(inter)
: for each rater j, of averaged readings are within unit of replicated averaged readings from the other rater
)( Inter %100*)( Inter
)]/222/(1[21 222)( meIntra
![Page 20: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/20.jpg)
Unified Approach – agreement(total)
: for over all k raters, how well are raters in reproducing each others based on the individual readings?
],,...,,|)1()([
])1()([
1
211
2.
1
2.
,
tindependenyyykyyE
kyyE
ikllili
k
jliijl
k
jliijl
totalc
2222
2
e
Totalc,
![Page 21: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/21.jpg)
Unified Approach – precision(total)
: for any two raters, the proportion of the variance that is attributable to the subjects based on the individual readings
)var()var(
),cov(
''
''
lijijl
lijijltotal
yy
yy
222
2
e
total
![Page 22: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/22.jpg)
Unified Approach – accuracy(total)
: how close are the means of different raters (accuracy)
2222
222
,
e
etotala
totala,
totalatotalTotalc ,,
![Page 23: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/23.jpg)
Unified Approach – TDI(total)
: for overall k raters, % of the readings are within unit of the replicated readings from the other rater.
)(Total)100*(
||)2
11(1
)(
Total
2221 222)2
11( e
)(Total
![Page 24: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/24.jpg)
Unified Approach – CP(total)
: for each rater j, of readings are within unit of replicated readings from the other rater
)(Total %100*)(Total
)]222/(1[21 222)( eTotal
![Page 25: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/25.jpg)
Unified Approach
is the inverse cumulative normal distribution is a central Chi-squre distribution with df=1
)2
11(1 Q
Statistics INTRA INTER TOTAL M=1
Agreement
Precision
Accuracy NA
MSD
TDIπ
CPδ
222
22
e
me /2222
2
2222
2
e
222
2
e
222
22
e
me /222
2
222
2
e
22
2
e
m
m
e
e
/
/2222
222
2222
222
e
e
222
22
e
e
22 e me /222 222 222 222 e 22 22 e
IntraMSDQ InterMSDQ TotalMSDQ MSDQ
)1,(2
2
IntraMSD
)1,(2
2
InterMSD
)1,(2
2
TotalMSD
)1,(2
2
MSD
)1,(2
2
MSD
![Page 26: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/26.jpg)
Estimation and Inference
Estimate all means, variance components,
and their variances and covariances by GEE methodEstimate all indices using above estimatesEstimate variances of all indices using above estimates and delta method
222221 ,,,,,,,, ek
![Page 27: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/27.jpg)
Estimation and Inference (2)
: the covariance of two replications,
and ,with coming from rater and
coming from rater
'j
''lljjjl
)1(
2
2
1
1 1' 1 1'''
2
kkm
k
j
k
jj
m
l
m
llljj
'l l'l
![Page 28: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/28.jpg)
Estimation and Inference (3)
: the variance from each combination
of (i, j), i.e., each cell. Thus is the average of all cells’ variances.
nk
n
i
k
jij
e
1 1
2
2
2ij
2e
![Page 29: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/29.jpg)
Estimation and Inference (4)
: the variance of replication of rater : the covariance of two replications, and
, both of them coming from rater .
kmA
k
j
m
ljl
2
1 1
2
kmB
k
j
m
l
m
lljll
2
1
1
1 1'
'2
mBA e /222
2jl
'jll
l j
jl 'l
![Page 30: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/30.jpg)
Estimation and Inference (5)
Using GEE method to estimate all indices through estimating the means and all variance components: 2222
21 ,,,,,...,, ek
n
iiiii YYHF
1
1' 0)(
![Page 31: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/31.jpg)
Estimation and Inference (6)
E
D
C
B
A
A
A
YY k
j
i
..
..1
222
2
2
222
1
/
/
.
.
)(
m
m
YYE
e
e
e
k
j
ii
![Page 32: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/32.jpg)
Estimation and Inference (7)myyyA ijmijijj /)..( 21
)1(/)(1
1 1'
2'..
kkyyBk
j
k
jjijij
)1(/]))([(2 21
1 1' 1 1'''.''.
kkmyyyyCk
j
k
jj
m
l
m
lljlijjlijl
k
j
m
lijijl kmyyD
1 1
2. /])1/()([
kmyyyykmyyEk
j
m
l
m
lljlijljlijl
k
j
m
ljlijl
2
1
1
1 1''.'.
2
1 1
2. /]))([(2/)(
![Page 33: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/33.jpg)
Estimation and Inference (8)
is the working variance-covariance structure of , “working” means assume following normal distribution is the derivative matrix of expectation of with respective to all the parameters
iH
iYYiYY
),,,,,...,(/ 22221 ekiiF
iF
iYY
![Page 34: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/34.jpg)
Estimation and Inference (9)GEE method provides:
estimates of all meansestimates of all variance componentsestimates of variances for all variance componentsEstimates of covariances between any two variance components
![Page 35: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/35.jpg)
Estimation and Inference (10)
Delta method is used to estimate the variances for all indices
2222
22,
22222,
, )(
)var()],cov(2)var()[var()1()var(
e
eIntracIntracIntrac
2222
2222,,
)(
)],cov(),[cov()1(2
e
eeIntracIntrac
![Page 36: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/36.jpg)
Estimation and Inference (12)
2222
22222,
22,
, )(
)var(/)var()[var()()var()1()var(
e
eIntercIntercInterc
m
22222
222222
)/(
]/),cov(2/),cov(2),cov(2
m
mm
e
ee
22222
222222,,
)/(
]/),cov(),cov(),[cov()1(2
m
m
e
eIntercInterc
![Page 37: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/37.jpg)
Estimation and Inference (13)
2222
2222,
22,
, )(
)var()var()[var()()var()1()var(
e
eTotalcTotalcTotalc
22222
222222
)(
)],cov(2),cov(2),cov(2
e
ee
22222
222222,,
)(
)],cov(),cov(),[cov()1(2
e
eTotalcTotalc
![Page 38: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/38.jpg)
Estimation and Inference (14)
2222
22222222
)/(
]/),cov(2/)var()[var()()var()1()var(
m
mm
e
eeInterInterInter
2222
2222
)/(
]/),cov(),[cov()1(2
m
m
e
eInterInter
2222
2222222
)(
)],cov(2)var()[var()()var()1()var(
e
eeTotalTotalTotal
2222
2222
)(
)],cov(),[cov()1(2
e
eTotalTotal
![Page 39: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/39.jpg)
Estimation and Inference (15)
22222
2222222,
, )/(
/),cov(2/)var()var()[var()1()var(
m
mm
e
eeInteraIntera
22222
222222,,
)/(
)],cov(/),cov(),[cov()1(2
m
m
e
eInteraIntera
22222
22,
2222
)/(
)var(]/),cov(2),cov(2
m
m
e
Interae
![Page 40: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/40.jpg)
Estimation and Inference (16)
22222
222222,
, )(
),cov(2)var()var()[var()1()var(
e
eeTotalaTotala
22222
22,
2222
)(
)var()],cov(2),cov(2
e
Totalae
22222
222222,,
)(
)],cov(),cov(),[cov()1(2
e
eTotalaTotala
![Page 41: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/41.jpg)
Estimation and Inference (17)
)var(4)var( 22eIntra
]),cov(),cov(
),cov()var(
)var()[var(4)var(2222
222
2222
mmmeee
Inter
)],cov(),cov(),cov()var()var()[var(4)var( 2222222222eeeTotal
![Page 42: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/42.jpg)
Estimation and Inference (18)Transformations for variances
Z-transformation: CCC-indices and precision indices
Logit-transformation: accuracy and CP indices
Log-transformation: TDI indices
)1
1ln(
2
1
c
cz
)1
ln(a
a
)ln( 2e
![Page 43: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/43.jpg)
Simulation Studythree types of data: binary/ordinary/normalthree cases for each type of data
k=2, m=1 / k=4, m=1 / k=2, m=3
for each case: 1000 random samples with sample size n=20for binary and ordinary data: inferences obtained through transformation vs. no-transformationFor normal data: transformation
![Page 44: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/44.jpg)
Simulation Study (2)Conclusions:
Algorithm works well for three types of data, both in estimates and in inferencesFor binary and ordinary data: no need for transformationFor normal data, Carrasco’s method is superior than us, but for categorical data, our is superior. For ordinal data, both Carrasco’s method and ours are similar.
![Page 45: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/45.jpg)
Example One
Sigma method vs. HemoCue method in measuring the DCHLb level in patients’ serum299 samples: each sample collected twice by each method Range: 50-2000 mg/dL
![Page 46: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/46.jpg)
Example One – HemoCue method
HemoCue method first readings vs. second readings
![Page 47: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/47.jpg)
Example One – Sigma method
Sigma method first readings vs. second readings
![Page 48: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/48.jpg)
Example One – HemoCue vs. Sigma
HemoCue’s averages vs. Sigma’s averages
![Page 49: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/49.jpg)
Example One – analysis result (1)
Statistics Estimates 95% CI* Allowance
ccc_inter 0.9866 0.9818 0.9775
ccc_total 0.9859 0.9809
precision_intra 0.9986 0.9982 0.9943
precision_inter 0.9866 0.9818
precision_total 0.9860 0.9809
accuracy_inter 0.9999 0.9974
accuracy_total 0.9999 0.9974
![Page 50: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/50.jpg)
Example One – analysis result (2)
*: for all CCC, precision, accuracy and CP indices, the 95% lower limits are
reported. For all TDI indices, the 95% upper limit are reported.
Statistics Estimates 95% CI* Allowance
TDIintra(0.9) 41.0903 47.2713 75
TDIinter(0.9) 127.273 149.799 150
TDItotal(0.9) 130.548 152.678
CPintra(75) 0.9973 0.9942 0.9
CPinter(150) 0.9475 0.9170 0.9
CPintra(150) 0.9412 0.9102
![Page 51: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/51.jpg)
Example Two
Hemagglutinin Inhibition (HAI) assay for antibody to Influenza A (H3N2) in rabbit serum samples from two labs64 rabbit serum samples: measured twice by each labAntibody level: negative/positive/highly positive
![Page 52: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/52.jpg)
Example Two – Lab one
Second Reading
First Reading
Negative
Positive Highly positive
Negative
6 1 0
Positive 0 49 0
Highly positive
0 0 8
![Page 53: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/53.jpg)
Example Two – Lab two
Second Reading
First Reading
Negative Positive Highly positive
Negative 2 0 0
Positive 0 22 2
Highly positive
0 5 33
![Page 54: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/54.jpg)
Example Two: Lab one vs. lab two
Lab Two First Reading
Lab OneFirst Reading
Negative Positive Highly positiv
e
Negative 2 5 0
Positive 0 19 30
Highly positive
0 0 8
![Page 55: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/55.jpg)
Example Two: lab one vs. lab two
Lab Two Second Reading
Lab OneSecond Reading
Negative Positive Highly positive
Negative 2 4 0
Positive 0 23 27
Highly positive
0 0 8
![Page 56: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/56.jpg)
Example TwoStatistics Estimates 95% CI* Allowance
ccc_inter 0.37225 0.22039 0.4375
ccc_total 0.35776 0.20970
precision_intra
0.88361 0.79692 0.75
precision_inter
0.56795 0.4359
precision_total
0.53489 0.39999
accuracy_inter
0.65543 0.51586
accuracy_total
0.66885 0.53561
![Page 57: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/57.jpg)
Conclusions (1)
When data are continuous and m goes to ∞:
agreement indices are the same as that proposed by Barnhart (2005), both in estimates and inferencesimprovements• Precision indices, accuracy indices TDIs
and CP• Variance components
![Page 58: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/58.jpg)
Conclusions (2)When m=1:
agreement index degenerates into OCCC as proposed by King (2002), Carrasco (2003) for continuous data Improvements:
• For categorical data:– King’s method: approximates to kappa and weighted
kappa, our estimates (without transformation) are exactly the same as kappa and weighted kappa, both in estimate and in inference.
– Our estimates superior to Carrasco’s estimates when precision and accuracy are high
• Covariates adjustment become available
![Page 59: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/59.jpg)
Conclusions (3)
When data are continuous, k=2 and m=1:
agreement index degenerates to the original CCC by Lin (1989)
When data are binary, k=2 and m=1:
agreement index degenerates into kappa, both in estimate and inference
![Page 60: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/60.jpg)
Conclusions (4)When data are ordinary, k=2 and m=1:
agreement index degenerates into weighted kappa with below weight set, both in estimate and in inference.
kjik
jiwij ,...,2,1,,
)1(
)(1
2
2
![Page 61: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/61.jpg)
Conclusions (5)
Unified approach Relative agreement indices: CCC with precision and accuracy – data rangeAbsolute agreement: Total deviation indices and Coverage Probability – normal assumptionLink function need more workRequire balanced data
![Page 62: A Unified Approach for Assessing Agreement](https://reader036.vdocument.in/reader036/viewer/2022062518/568147f1550346895db529e6/html5/thumbnails/62.jpg)
ReferencesBarkto, John J (1966): The intraclass correlation coefficient as a measure of reliability. Pshchological Reports 19, 3-11.Barnhart, H. X. and Williamson, J. M. (2001). Modeling concordance correlation via GEE to evaluate reproducibility. Biometrics 57, 931-940.Barnhart, H. X. Song, Jingli and Haber, Michael J. (2005): Assessing intra, inter and total agreement with replicated readings. Statistics in Medicine 19: 255-270.Carrasco, J. L. and Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics 59, 849-858.Fleiss, J., Cohen, J. and Everitt, B (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin 72, 323-327.King, Tonya S. and Chinchilli, Vernon M. (2001): A generalized concordance correlation coefficient for continuous and categorical data. Statistics in Medicine 20: 2131-2147.Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics 45, 255-268.Lin, L. I., Hedayat, A. S., Sinha, B., and Yang, M. (2002). Statistical methods in assessing agreement: models, issues & tools. Journal of American Statistical Association 97(457), 257-270.Wu, Wenting. A unified approach for assessing agreement. Ph.D. thesis, UIC, 2006