joshua naranjo - wmich.edu · joshua naranjo questionnaire validity. joshua naranjo questionnaire...
TRANSCRIPT
Questionnaire validity
Joshua Naranjo
Stat 5630Western Michigan University
Joshua Naranjo Questionnaire validity
Questionnaire theory
Recall: Study context
research aims
design
measurement/questionnaire (satisfaction, ease of use,religiosity)
analysis
Statistical concepts in questionnaire theory
reliability and validity
intraclass correlation coefficient
Cronbach’s alpha
factor analysis
Joshua Naranjo Questionnaire validity
Questionnaire theory
Reliability and validity of measurements
Ex: ”Association between intelligence, communication skills, andeffective leadership”
Validity – does an instrument measure what it claims to measure?Reliability – does it work consistently?
Joshua Naranjo Questionnaire validity
Questionnaire theory
Types of validity:
Construct validity – is it valid in theory?
Face validity – does it measure what it aims to measure?
Convergent validity – does the survey compare well withanother measure of the same thing?
Discriminant validity – can it tell different groups apart?
(Ex: intelligence, communication skills, leadership)
Joshua Naranjo Questionnaire validity
Questionnaire theory
Types of Reliability:
Internal reliability – do respondents give consistent answersover similar questions?
Test-retest reliability – do respondents give consistent answersover time?
Inter-rater reliability – do raters give consistent answers forthe same item?
Joshua Naranjo Questionnaire validity
Intraclass correlation coefficient
Ex: “How much agreement is there between physicians reading CTscans for disease progression?”
CT Scan Physician 1 Physician 21 7 52 6 9: : :: : :n 2 2
Q: Pearson correlation?
A: Which student is X and which is Y ?
Joshua Naranjo Questionnaire validity
Intraclass correlation coefficient
Ex: “How much agreement is there between physicians reading CTscans for disease progression?”
CT Scan Physician 1 Physician 21 7 52 6 9: : :: : :n 2 2
Q: Pearson correlation?A: Which student is X and which is Y ?
Joshua Naranjo Questionnaire validity
Intraclass correlation coefficient
ICC measures agreement between exchangeable variables (i.e. noinherent ordering).Used for test-retest reliability, and inter-rater reliability.
Let
x =1
2n
∑(xi1 + xi2)
s2 =1
2n
{∑(xi1 − x)2 +
∑(xi2 − x)2
}Then
ICC =1
ns2
∑(xi1 − x)(xi2 − x)
Joshua Naranjo Questionnaire validity
Intraclass correlation coefficient
For groups of three,
x =1
2n
∑(xi1 + xi2 + xi3)
s2 =1
2n
{∑(xi1 − x)2 +
∑(xi2 − x)2 +
∑(xi3 − x)2
}Then
ICC =1
3ns2
∑{(xi1 − x)(xi2 − x) + (xi1 − x)(xi3 − x)
+(xi2 − x)(xi3 − x)}
Joshua Naranjo Questionnaire validity
Cronbach alpha
Internal Consistency: Hearing aid survey
Suppose we want to measure how well the hearing aid “listens inconversation”
Q: Do the questions “measure the same thing”?
Joshua Naranjo Questionnaire validity
Cronbach’s alpha:
Suppose that T = x1 + · · ·+ xk is the composite score for a factoror construct. Then
α =k
k − 1
(1−
∑var(xi )
var(T )
)
Joshua Naranjo Questionnaire validity
Cronbach alpha
α = 1010−1
(1− 19.022
40.6933
)= 0.59172
Joshua Naranjo Questionnaire validity
It may be easier to understand as follows:
α =k
k − 1
(1−
∑var(xi )
var(T )
)=
k
k − 1
(var(T )−
∑var(xi )
var(T )
)
=k
k − 1
(∑i 6=j cov(xi , xj)
var(T )
)So α is an average covariance between the xi s.
Joshua Naranjo Questionnaire validity
Cronbach alpha
But this is still just a computing formula. In CronbachPsychometrika, 1951, Cronbach proposes the statistic as a
“....mean of all split-half coefficients resulting from differentsplittings of a test... therefore an estimate of the correlationbetween two random samples of items...”
Joshua Naranjo Questionnaire validity
Examples
Published example:
Joshua Naranjo Questionnaire validity
Factor Analysis
Joshua Naranjo Questionnaire validity
Factor Analysis
Joshua Naranjo Questionnaire validity
Factor Analysis
Joshua Naranjo Questionnaire validity
Factor Analysis
Let Yi be the response to the ith question, i = 1, . . . , 15. The ideabehind factor analysis is that the 15 responses actually depend ononly two or three common underlying unobservable factors, i.e.
Yi = µi + ai f1 + bi f2 + ε
History of factor analysis
Charles Spearman (1904) two-factor theory of intelligence:general and specific
Thurstone 7-factor: numerical, verbal comprehension, wordfluency, perceptual speed, memory, inductive reasoning,spatial ability
Sternberg 3-factor: analytical (problem-solving), creative (newideas), and practical (everyday logic)
Joshua Naranjo Questionnaire validity
Factor Analysis
Back to article, let
Y1
Y2
:Y15
=
µ1µ2:µ15
+
l11 l12l21 l22: :
l15,1 l15,2
( f1f2
)+
ε1ε2:ε15
or
Y = µ+ LF + ε
where µ and L are constants, F and ε have mean 0
V (F) =
(1 00 1
), V (ε) = Ψ =
ψ1 0 . . . 00 ψ2 . . . 0...
......
0 0 . . . ψ15
Joshua Naranjo Questionnaire validity
so thatV (Y) = LL′ + Ψ
Estimation:
S =15∑i=1
λivivi′ ∼=
2∑i=1
λivivi′ =
(√λ1v1
√λ2v2
)( √λ1v1′
√λ2v2
′
)= L̂L̂′
where λ1 > λ2 > · · · > λ15. The estimated coefficients of theunderlying factors are derived from the eigenvalues andeigenvectors of S.
Joshua Naranjo Questionnaire validity
Factor Analysis: Rotation
Let A be any 2× 2 orthogonal matrix, i.e. AA′ = I. Then
Y = µ+ LF + ε = µ+ LAA′F + ε = µ+ L∗F∗ + ε
so that there is an ambiguity to the factor analysis model. But thiscan be exploited, might as well choose A′F∗ = (f ∗1 f ∗2 )′ so that thefactors are more interpretable. In practice, this means we want torotate so that each of the 15 questions will have high coefficient inonly one factor.
Joshua Naranjo Questionnaire validity
Joshua Naranjo Questionnaire validity
Factor Analysis: Explained variance
Recall that S =∑15
i=1 λivivi′ ∼=
∑2i=1 λivivi
′
trace(S) = s21 + s22 + · · · s215= λ1 + λ2 + · · ·+ λ15
The proportion of total variation explained by the ith factor is
λiλ1 + λ2 + · · ·+ λ15
Joshua Naranjo Questionnaire validity
Joshua Naranjo Questionnaire validity