inter-rater reliability march 1, 2013 emily phillips galloway william johnston
Post on 16-Dec-2015
212 Views
Preview:
TRANSCRIPT
Inter-Rater Reliability
March 1, 2013
Emily Phillips GallowayWilliam Johnston
www.gse.harvard.edu
Accessing Workshop Materials
Go to:
isites.harvard.edu/research_technologies
– Click on Workshops tab (on the left) and then the Inter-Rater Reliability folder (near the bottom)
– Save all of the files to the desktop (right click and ‘Save Link As’)
www.gse.harvard.edu
AgendaI. Introducing IRRII. What is Kappa?
I. ‘By-hand’ Example
III. Limitations and Complications of KappaIV. Working Through a Complex Example
I. Data setupII. Estimation III. Interpretation
V. Reporting Results
www.gse.harvard.edu
What is Inter-Rater Reliability?
IRR can be defined as the degree of agreement among raters. Numerous statistics can be calculated to provide a score of how much consensus exists between raters.
Why does it matter in educational research?:
In IRR we trust The quality of a coding scheme and the ability to replicate results is
connected with the overall ‘believability’ of the results. To publish results, we must demonstrate that our coding scheme is reliable.
www.gse.harvard.edu
What is Inter-Rater Reliability?
IRR can be defined as the degree of agreement among raters. Numerous statistics can be calculated to provide a score of how much consensus exists between raters.
Why does it matter in language & literacy research? Language data can be challenging to code and can fall prey to
subjectivity given that interlocutors will not always say or write all that may be inferred by scorers.
www.gse.harvard.edu
What is Inter-Rater Reliability?
IRR can be defined as the degree of agreement among raters. Numerous statistics can be calculated to provide a score of how much consensus exists between raters.
Why does it matter in language & literacy research? Language data can be challenging to code and can fall prey to
subjectivity given that interlocutors will not always say or write all that may be inferred by scorers.
Our Task: To design coding schemes that are not subjective.
www.gse.harvard.edu
IRR: A Beginning and an End
How does formative IIR/inter-rater agreement tell us during the beginning / design phase of a study?
If your coding scheme is being developed, calculating IRR can tell you if your codes are functioning in the same way across raters.
If you are using an existing coding scheme, calculating IRR can tell you if your raters may need additional training.
www.gse.harvard.edu
IRR: A BeginningHow does formative IIR/inter-rater agreement tell us during the beginning / design phase of a study?
If your coding scheme is being developed, calculating IRR on 15%-20% of your data can tell you if your codes are functioning in the same way across raters.
If there are numerous disagreements, this signals a need to revise your coding
scheme (and recode the data), to locate clear examples to help raters to understand codes better, or (when all else fails) to
abandon codes that do not function well.
Disagreements are OK at this
point!
www.gse.harvard.edu
IRR: A Beginning How does formative IIR/inter-rater agreement tell us during the beginning / design phase of a study?
If are using an existing coding scheme, calculating IRR on 15%-20% of your data can tell you if your codes are functioning in the same way across raters.
If there are numerous disagreements, this signals a need to retrain your raters (and recode the data)
or to evaluate if the coding scheme you are working with may need to be revised for your
data.
Disagreements are OK at this
point!
www.gse.harvard.edu
IRR (Agreement) by Hand!
While kappa statistics give us insight into user disagreements on the aggregate, a ‘confusion matrix’ can help us to identify specific codes on which raters disagree.
(Bakeman & Gottman, 1991)
www.gse.harvard.edu
Let’s try itScenario: Marie and Janet are coding students’ definitions for the presence of nominalized words. For each definition, 0=no nominalizations and 1=nominalizations. They are at the beginning of scoring the data with a new coding scheme that Janet has developed.
How does it seem to be working?
www.gse.harvard.edu
Are Janet & Marie still friends?
www.gse.harvard.edu
IRR: The Middle? How does IRR tell us during the middle of a study?
In the middle of a study, especially if we are coding data over a long period, we may again conduct IRR analysis (a ‘reliability check’) to be sure we are still coding the data reliably.
involves selecting 20% of the data at random to assess for agreement.
www.gse.harvard.edu
IRR: An End Why does summative IRR matter during the analysis phase of a study?
At the end of a study, we calculate IRR to demonstrate to the academic community that our coding scheme functioned reliably.
Generally, if we have been diligent in developing our coding scheme and training our raters there are few surprises.
www.gse.harvard.edu
What is Kappa?
Cohen’s Kappa (Cohen, 1960) is a numeric summation of agreement that accounts for agreements simply by chance.
po = Proportion of agreement that is actually observedpc = Proportion of agreement by chance
See pages 63-64 in the Bakeman & Gottman (1991) for an excellent example of how po, pc , and Cohen’s Kappa are calculated.
www.gse.harvard.edu
Limitations of KappaWhat if there are more than two possible
ratings and the size of the discrepancy between raters matters?
Use weighted KappaWhat if there are more than two raters?
Use Fleiss’ KappaWhat if different participants have different numbers of raters?
Use Krippendorff’s alpha
www.gse.harvard.edu
A Detailed ExampleThe sample: 37 studentsThe data structure: 8 key variables• 2 different work explanation tasks
• “bicycle” and “debate”• 2 coders
• coder1 and coder2• 2 rating subscales
• Super ordinate scale (0-5 points)• Syntax scale (0-6 points)
Which statistic should we be
using?
Weighted Kappa!(p. 66 in B & G)
www.gse.harvard.edu
Estimating IRR in StataStart with a simple cross tabulation:
coder1_bic |ycle_super | coder2_bicyclel_superordinate ordinate | 0 2 3 4 5 | Total-----------+-------------------------------------------------------+---------- 0 | 13 0 0 1 0 | 14 2 | 0 5 0 0 0 | 5 3 | 0 0 3 0 0 | 3 4 | 0 0 0 5 0 | 5 5 | 0 0 0 0 10 | 10 -----------+-------------------------------------------------------+---------- Total | 13 5 3 6 10 | 37
What do you notice?• Very strong agreement! Why might this be? • Neither reviewer ranked anybody a “1”. Why might this be?
• This has implications for how we do this in Stata…
www.gse.harvard.edu
Estimating IRR in Stata• Because of the nature of the ratings, we
have to make some changes to the data in order for things to run. • Our weight matrix implies there are 6 possible
ratings but only 5 are used by the raters we must use the “absolute” option in Stata
• BUT… in order for this to work we need to change the scale from 0,1,2…5 to 1,2,3…6.
replace c1_bicycle_so = c1_bicycle_so+1replace c2_bicycle_so = c2_bicycle_so+1 tab c1_bicycle_so c2_bicycle_so
www.gse.harvard.edu
Estimating IRR in StataThe command:
kap c1_bicycle_so c2_bicycle_so, wgt(s_o_wgt) absolute
The output:
ExpectedAgreement Agreement Kappa Std. Err. Z Prob>Z----------------------------------------------------------------- 97.84% 53.81% 0.9532 0.1284 7.43 0.0000
www.gse.harvard.edu
Apply Your Knowledge
For each of the three remaining subscales repeat the steps:
• Inspect a cross tabulation to get an idea of the data distributions
• Create a weighting matrix • “w” or “w2” are built-in weight matrices that you
can use (w is linear and w2 is quadratic)
• Estimate the Kappa and interpret the results!
www.gse.harvard.edu
Estimating IRR with web resources
If you do not have access to Stata, there are some great web resources you can use as well:
http://www.stattools.net/CohenKappa_Exp.php
http://www.agreestat.com/
www.gse.harvard.edu
Reporting Results“inter-rater reliability, calculated based on double coding of 20% of the tasks, was very high (Agreement = 98%; Cohen’s Kappa = .96).” (Kieffer & Lesaux, 2010)
“To calculate inter-rater reliability for the coding scheme developed in the first phase, a research coordinator and a graduate research assistant randomly selected 20 writing samples for each of four tasks from different years, cohorts, and writing ability levels: narratives by pen, the sentence integrity task by pen, essays by keyboard, and the sentence integrity task by keyboard. One rater served as the anchor for computing percent agreement between coders. The inter-rater reliability was generally very good for each coded category. Except for two categories, initial percent agreement ranged from 84.6% to 100%. For the only two categories of low interrater reliability, subordinate and adverbial clauses, additional training and reliability checks improved inter-rater reliability for these to acceptable levels of over 0.80.” (Berninger, Nagy, & Scott, 2011)
top related