Moon Seok Park, MD

Seoul National University Bundang Hospital

Testing reliability and validity in medical research

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Reliability

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

• 1 년 차 때 , 교수님이 “ 내일까지 X-ray 1000 장 재 봐서 결론 내 !!” 고 오더를 내리셔서 .

• 처음 재보는 각도 , 밤새 측정을 했다 . 힘들어서 인턴도 시켰다 . 제대로 했는지도 잘 모르겠다 .

• 그런데 , 결과는 의미 있게 나왔다 . OK!!

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

• 두 개의 다른 방법으로 측정을 했을 때 , 신뢰도를 알아

보려면 paired t-test 로 하면 안 되는가 ?

• Paired t-test 는 어떨 때 쓰는 방법일까 ?

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Reliability

• Extent to which scale items measure the same construct, with freedom of random error

• 신뢰도• 측정 시 마다 측정치가 비슷한가 ?• Test-retest reliability, Inter-rater reliability,

Intra-rater reliability, Alternative form reliability, Internal consistency.

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Test-retest reliability

• 주로 Psychometric analysis : 인터뷰 , 설문지… .

• 일정한 시간 간격을 두고 , 같은 검사를 시행 .• Cohen’s kappa, weighted kappa, Pearson’s

correlation, Intraclass correlation coefficient(ICC).

• Cf) Intra-rater(observer reliability) : 방사선 검사 계측… .

• Memory contamination

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Inter-rater reliability

• 전문가에 의한 인터뷰 , scoring, 신체 계측 , 방사선 계측 .

• 여러 명이 한 객체를 계측하여 , 비슷한가 비교 .• Cf) Agreement : 혼용되어 사용되지만 , 특히 다른

기구를 이용한 측정 , 예를 들어 MRI 와 CT 의 비교 등…

• 방사선 계측 등에서는 intra- and inter-observer(rater) reliability 를 set 로 .

• Cohen’s kappa, weighted kappa, Pearson’s correlation, Intraclass correlation coefficient(ICC)

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Internal consistency

• 이전의 reliability 와는 조금 다른 의미 . Psychometric analysis ( 설문지 , 인터뷰 ) 등에 주로 국한 되어 사용 .

• Homogeneity • 가령 10 개의 문항이 있다고 하면 , 각각의

문항이 서로 비슷 .• Item to item, Item to total, Cronbach’s

alpha• Too high internal consistency = Item

redundancy.• Cf) Uni-dimensionality, Item response

theory, Rasch analysis(INFIT statistics)

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Question: which is reliable?

1 2

3 4

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

What are the main measures of reliability?

• What if the data are dichotomous or polychotomous?– Kappa coefficient

• What if the data are quantitative (interval or ratio scale?– Intraclass Correlation Coefficient (ICC)

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

ICC

• Intraclass correlation coefficient

• Reliability test for quantitative data

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Models of ICC

• One-way random effect model– Raters: a random effect

• Two-way random effect model– Raters: a random effect– Subjects: a random effect

• Two-way mixed effect model– Raters: a fixed effect– Subjects: a random effect

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Types of ICC

• Absolute agreement– Measures if raters assign the same

absolute score

• Consistency– Measures if raters’ scores are highly

correlated even if they are not identical in absolute terms

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Measures of ICC

• Single measures– Individual ratings constitute the unit of

analysis

• Average measures– The mean of all ratings is the unit of

analysis

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

ICC

• Affected by true subject variability as well as measurement error

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Example

• Measurement error– Data 1 = Data 2

• Subject variability– Data 1 < Data 2

Data 1 Data 2

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

ICCs for sample data 1 and 2

models Sample data 1 Sample data 2

1 way random-0.059 (-

0.308~0.407)0.922 (0.799~0.978)

2 way random0.217

(0.007~0.614)0.924 (0.237~0.986)

2 way mixed0.217

(0.007~0.614)0.924 (0.237~0.986)

ICC values were calculated with the assumption of absolute agreement and single measurementData are presented as ICC (95% confidence interval)

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

• Propose 6 ICC types:

ICC(1,1) ICC(2,1) ICC(3,1) ICC(1,k) ICC(2,k) ICC(3,k)

Shrout and Fleiss, 1979

Expected Reliability of a Single Rater’s Rating

Expected Reliability of the Mean of a set ofk Raters

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

k (no.of observers), n (no.of targets)

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

between-target mean square (BMS); within-target mean square(WMS); BMS represents true subject variability, and WMS represents measurement error

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Shrout and Fleiss, 1979

• Important issue in the choice of an appropriate index– Whether the ANOVA design should be

one way or two way– Whether raters are considered fixed

or random effects– Whether the unit of analysis is a

single rater or the mean of several raters

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Pitfalls and important issues in

testing reliability using ICC in

orthopaedic research

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Literature review

• Pubmed database

• Orthopaedic articles that used ICC

• Of the 92 articles identified, 58 (63%) did not clarify the ICC model used.

• The model, types, and measures used were clearly declared in only 5 (5%)

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

ICC of physical examinations

• 30 patients with CP• Interobserver reliability of physical

examinations using ICC– Popliteal angle– Thomas test– Staheli test

Same dimension !! (joint angle)

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Reliability of physical examinations evaluated by various statistical methods

Popliteal angle Thomas test Staheli testMean (°) 47.6 4.7 2.5SD (°) 15.2 5.9 8.8Range (°) 8~80 0~20 -17~28ICC2 way randomconsistency/average 0.881 (0.794~0.936) 0.742 (0.552~0.860) 0.463

(0.067~0.708)consistency/single 0.713 (0.562~0.829) 0.490 (0.291~0.672) 0.224

(0.023~0.447)absolute/average 0.880 (0.792~0.935) 0.742 (0.553~0.860) 0.464

(0.070~0.708)absolute/single 0.710 (0.560~0.826) 0.490 (0.292~0.671) 0.224

(0.024~0.447)2 way mixedconsistency/average 0.881 (0.794~0.936) 0.742 (0.552~0.860) 0.463

(0.067~0.708)consistency/single 0.713 (0.562~0.829) 0.490 (0.291~0.672) 0.224

(0.023~0.447)absolute/average 0.880 (0.792~0.935) 0.742 (0.553~0.860) 0.464

(0.070~0.708)absolute/single 0.710 (0.560~0.826) 0.490 (0.292~0.671) 0.224

(0.024~0.447)1 way random average 0.880 (0.792~0.935) 0.742 (0.553~0.860) 0.464

(0.072~0.708) single 0.709 (0.559~0.826) 0.489 (0.292~0.671) 0.224

(0.025~0.447)SEM (SDx√(1-reliability)

0.112~0.175 0.590~0.830 2.02~2.43

MAD 9.4 3.6 6.1CV (SD/mean) 0.32 1.16 2.76ICC, intraclass correlation coefficient; SEM, standard error of measurement; MAD, mean absolute difference; CV, coefficient of variation

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Simulated data

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Conclusion

• ICC value could represent the opposite tendency to true measurement error (mean absolute difference) even when measuring similar dimension

• ICC could be variable depending on the model used.

• ICC value was affected by measurement error, subject variability, and slopes.

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

결론적으로 이렇게 해야 ..

• ICC values were large when measurement errors were small, subject variability large, and slopes parallel.

• Clinical context need to be considered when interpreting ICC.

• ICC setting should be declared.

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Validity

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Validity

• Extent to which instruments is really measuring what it purpose to measures.

• 보통 internal validity 라고 이야기 한다 .

• Cf) external validity = generalisability

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Validity

• Face validity

• Content validity

• Criterion(concurrent, predictive) validity

• Construct(convergent, discriminant) validity

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Face validity

• 안면 타당도 ( 액면 타당도 )

• Content validity 와 혼동될 수 있지만 , 좀 더 추상적임 .

• 예를 들어 영어 시험의 문항에 수학 문제가 있으면 , face validity 에 문제가 있는 것 .

• 대게 저자들이 screening 하는 정도로 표현 .

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Content validity

• 내용 타당도

• Face validity 와 비슷하지만 , 좀 더 systematic 하게 분석 .

• 일정 수의 panel 이 모여서 content validity를 scoring 하여 , 점수화 하고 , 평균 점수가 미달이면 기각 .

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Criterion validity

• Concurrent validity : gold standard 와 얼마나 비슷한가 ?

• 방사선 지표를 측정한다 . Gold standard 로 생각하는 CT 측정치와 비교 .

• Cf) convergent validity.

• Predictive validity

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Construct validity

• 구인 타당도• Convergent validity : 비슷한 지표 (gold

standard 는 아님 ) 와 상관관계가 있는가 ?• TEPS 라는 영어시험을 만들었다 . 타당도를

보려고 , TOFLE 과 상관관계를 보았다 . (영어실력의 gold standard 는 ?)

• 사람이 측정한 방법과 컴퓨터가 측정한 방법에 상관 관계가 있는가 ?

• Pearson correlation.

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Construct validity

• Discriminant validity : 전혀 다른 것을 측정하는 지표와 상관 관계가 있는가 ?

• 인성검사와 지능검사의 상관관계

• Cf) Known group validity : 확실히 다른 집단에서 다른 점수가 나오는가 ?

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Others

• Precision• Responsiveness• Sensitivity• Specificity• Sensitivity analysis• Item response

theory• Rasch analysis

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Introduction

• Increased femoral anteversion and coxa

valga are common deformities associated

with intoeing gait and unstable hips in CP,

which need surgical correction.

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Introduction

• Physical examination and neck shaft angle

measured on hip radiographs are primary

tools evaluating femoral anteversion and

coxa valga.

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Introduction

• Physical examinations measuring femoral

anteversion include

– Trochanteric prominence angle test (TPAT)

– Hip internal rotation (IR)

– Hip external rotation (ER)

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Introduction

• CT measurement is accurate, but

expensive and involves radiation

exposure.

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Purpose of Study

• To assess the validity and reliability of physical exams measuring femoral anteversion and neck shaft angle on hip X-ray– Concurrent validity

– Intra- and interobserver reliability

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Reliable and valid Not reliable but valid

Reliable but not valid Not reliable and not valid

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Materials and Methods

• Prospective study approved by IRB

• 36 consecutive patients with CP– Mean age 11.0 years (SD 1.3)

– M : F = 26 : 10

– GMFCS I / II / III / IV / V 5 / 11 / 11 / 7 / 2

• Exclusion– Previous Op, trauma, infection, etc.

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Hip Internal Rotation

• Prone position

• Angle between vertical line & long axis of the leg– legs are rotated

outward maximally

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Hip External Rotation

• Prone position

• Angle between vertical line & long axis of the leg– leg is rotated inward

maximally

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Trochanteric Prominence Angle Test

• Prone position

• Palpate G. trochanter

• External rotate limb until G. T. reaches most lateral

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

NSA on x-ray

• AP hip X-ray with hips 20°-30° internally rotated

• Angle : a line through midpoint of shaft & line through head and neck center

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Femoral anteversion on 2D CT

• Standard method

• Radiation hazard

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

NSA on 3D MRP Image

Standard method for concurrent validity of NSA on X-ray

MRP: multiplanar reformatted

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Valdity

• Physical exam measuring femoral AV– Correlation with femoral anteversion

measured on 2D CT

• NSA measured on X ray– Correlation with NSA measured on 3D MPR CT

image

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Reliability

• Interobserver reliability of physical exam using three orthopaedic surgeons on a single day

• Intra- and interobserver reliability of NSA on X-ray– Repeated measurements with an interval of 3

wks

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Statistics

• Validity– Pearson’s correlation coefficients

• Reliability– Intraclass correlation coefficients (ICCs)– 2 way random effects, single measurement &

absolute agreement• Multiple regression test

– To predict the accurate femoral anteversion (CT) from physical exam

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Results

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Femoval AV on CT= 0.92 x TPAT - 3.2 (R2=0.829)

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Conclusions

• TPAT and NSA on X ray showed clinically

relevant validity and reliability compared

with CT measurement.

• CT evaluating proximal femoral geometry

could be replaced by physical exam and

X-ray in patients with CP, avoiding

unnecessary radiation exposure.

SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL

Thank you !