Chapter 5Chapter 5

Understanding, Calculating, Understanding, Calculating, and Evaluating Reliability and Evaluating Reliability

and Objectivityand Objectivity

Methods for Evaluating Reliability Methods for Evaluating Reliability and Their Calculationsand Their Calculations

TestTest–Retest–Retest

• Most straightforward way to determine Most straightforward way to determine reliabilityreliability

• Must have:Must have:– No major changes in the construct being No major changes in the construct being

measuredmeasured– Sufficient recovery time between Sufficient recovery time between

measurementsmeasurements

Evaluating TestEvaluating Test–Retest Reliability–Retest Reliability

• Reliability coefficient: Reliability coefficient: – A ratio that shows the relationship between A ratio that shows the relationship between

two measurements, indicating the consistency two measurements, indicating the consistency (or reliability) between them. (or reliability) between them.

• Intraclass correlation:Intraclass correlation:– A statistical technique used to compute the A statistical technique used to compute the

reliability coefficient to assess the relationship reliability coefficient to assess the relationship between measures of the same class as in a between measures of the same class as in a testtest–retest study.–retest study.

Calculating the Intraclass CorrelationCalculating the Intraclass Correlation

R=[(SSa/R=[(SSa/nn––1) 1) – – (SSw/((SSw/(n*n*(k(k––1)))]/ (SSa/1)))]/ (SSa/nn––1)1)SSa=(ΣTSSa=(ΣT22//kk) ) – – ((ΣX) ((ΣX) 22//nknk))SSw= ΣXSSw= ΣX2 2 – – (ΣT(ΣT22/k)/k)

Where R is the intraclass reliabilityWhere R is the intraclass reliabilityΣ represents the sumΣ represents the sumN N = the number of test subjects= the number of test subjectskk = the number of trials for each person (usually = the number of trials for each person (usually

two)two)ΣTΣT2 2 = the sum of all the squared total scores for = the sum of all the squared total scores for

each personeach personΣXΣX22 = the sum of all the scores of everyone tested = the sum of all the scores of everyone tested

An Alternative for An Alternative for Calculating the Intraclass Correlation Calculating the Intraclass Correlation

• Try this website: Try this website:

http://department.obg.cuhk.edu.hk

and go the Statistics Tool Box link.and go the Statistics Tool Box link.

Results of a Calculation Results of a Calculation of Intraclass Reliabilityof Intraclass Reliability

Evaluating Reliability with Evaluating Reliability with a Single Test Administrationa Single Test Administration

• Split-half reliability:Split-half reliability:– Compare one half Compare one half

of a test with the of a test with the other halfother half

– Spearman-Brown Spearman-Brown Prophecy FormulaProphecy Formula

• Internal Internal consistency consistency reliability:reliability:– Average all possible Average all possible

split-half estimatessplit-half estimates– Cronbach’s alphaCronbach’s alpha

Evaluating the Reliability Evaluating the Reliability of Criterion-Referenced Measurementsof Criterion-Referenced Measurements

• Calculate percentage of agreement between the Calculate percentage of agreement between the test and the retest.test and the retest.

Percentage of Agreement= [(Cboth + NCboth) / (Cboth + Percentage of Agreement= [(Cboth + NCboth) / (Cboth + NCboth + C/NC + NC/C)] NCboth + C/NC + NC/C)] ** 100 100

Where Cboth = people scored as competent in both Trials 1 Where Cboth = people scored as competent in both Trials 1 and 2and 2

NCboth = people scored as not competent in both trialsNCboth = people scored as not competent in both trials NC/C = people scored as not competent in Trial 1 but NC/C = people scored as not competent in Trial 1 but

competent in Trial 2competent in Trial 2 C/NC = people scored as competent in Trial 1 but not C/NC = people scored as competent in Trial 1 but not

competent in Trial 2competent in Trial 2

Example Diagram for Evaluating the Example Diagram for Evaluating the Reliability of a Criterion-Referenced Reliability of a Criterion-Referenced

MeasurementMeasurement

Standard Error of MeasurementStandard Error of Measurement

• Defined: Defined: – An estimation of the error inherent in any An estimation of the error inherent in any

individual’s test score.individual’s test score.

SEM = SD * SEM = SD * √1–r√1–rrcrc

where SD = the standard deviation for the testwhere SD = the standard deviation for the test

rrrc rc = the reliability coefficient for the test= the reliability coefficient for the test

Increasing ReliabilityIncreasing Reliability

• Repeat a measurement several Repeat a measurement several times—times—– To improve both validity and reliabilityTo improve both validity and reliability– To discover and minimize errorsTo discover and minimize errors– To average out the errorsTo average out the errors

Methods for Evaluating ObjectivityMethods for Evaluating Objectivity

Calculating ObjectivityCalculating Objectivity

• Objectivity can be considered a special Objectivity can be considered a special case of reliability.case of reliability.– Inter-rater reliabilityInter-rater reliability

• Most techniques used to evaluate reliability Most techniques used to evaluate reliability can be used to evaluate objectivity.can be used to evaluate objectivity.

Calculating Objectivity Calculating Objectivity of Different Types of Measuresof Different Types of Measures

• For continuous measures:For continuous measures:– Intraclass correlationIntraclass correlation

• For discrete measures:For discrete measures:– Calculate the percent agreement Calculate the percent agreement

between test administratorsbetween test administrators

Validity, Reliability, and ObjectivityValidity, Reliability, and Objectivity

• It is possible to have high reliability or It is possible to have high reliability or objectivity without high validity.objectivity without high validity.

• Good reliability or objectivity will always Good reliability or objectivity will always be present with a valid measurement.be present with a valid measurement.

• Good reliability and objectivity do not Good reliability and objectivity do not establish good validity; they simply establish good validity; they simply suggest that a measurement suggest that a measurement maymay be be valid.valid.

Your ViewpointYour Viewpoint

• Can you think of any times in your life Can you think of any times in your life when you have had to evaluate the when you have had to evaluate the reliability or objectivity of something or reliability or objectivity of something or someone?someone?

• What did you do with the results of this What did you do with the results of this evaluation? Did it cause you to make any evaluation? Did it cause you to make any changes in your daily routines or change changes in your daily routines or change your mind about a decision?your mind about a decision?