iop 301-t test validity. iop 301-t what is validity? in simple english, the validity of a test...

IOP 301-TIOP 301-T

Test ValidityTest Validity

IOP 301-TIOP 301-T

What is validity?What is validity?

In simple English, the validity of a test concerns In simple English, the validity of a test concerns

what the test measures andwhat the test measures and how well it measureshow well it measures

It is the accuracy of the measure in reflecting It is the accuracy of the measure in reflecting the concept it is supposed to measure.the concept it is supposed to measure.

IOP 301-TIOP 301-T

Types of validityTypes of validity

Content-descriptionContent-description Criterion-descriptionCriterion-description Construct-identificationConstruct-identification

IOP 301-TIOP 301-T

Faceva lid ity

Contentva lid ity

Concurren tva lid ity

P red ictiveva lid ity

C rite rion-re la tedva lid ity

Corre la tion

Factoria lva lid ity

Convergentva lid ity

D iscrim inantva lid ity

Constructva lid ity

VALIDITY

IOP 301-TIOP 301-T

Content validityContent validity

non-statistical in naturenon-statistical in nature involves determining whether the involves determining whether the

sample used for the measure is sample used for the measure is representative of the aspect to representative of the aspect to be measuredbe measured

IOP 301-TIOP 301-T


It adopts a It adopts a subjectivesubjective approach approach whereby we have recourse, for e.g, whereby we have recourse, for e.g, to to expert opinionexpert opinion for the evaluation for the evaluation of items during the test of items during the test construction phase.construction phase.

IOP 301-TIOP 301-T


Relevant for evaluatingRelevant for evaluating

AchievementAchievement EducationalEducational OccupationalOccupational

measures.measures.

IOP 301-TIOP 301-T


Basic requirement forBasic requirement for Criterion-referencedCriterion-referenced Job sampleJob sample

measures, which are essential formeasures, which are essential for Employee selectionEmployee selection Employee classificationEmployee classification

IOP 301-TIOP 301-T


Measures are interpreted in terms ofMeasures are interpreted in terms of

Mastery of knowledgeMastery of knowledge SkillsSkills

for a specific job.for a specific job.

IOP 301-TIOP 301-T


Not appropriateNot appropriate for for

AptitudeAptitude PersonalityPersonality

measures since validation has measures since validation has to be made through criterion-to be made through criterion-prediction procedures.prediction procedures.

IOP 301-TIOP 301-T

Face validityFace validity

It is not validity in psychometric It is not validity in psychometric terms !terms !

It just refers to what the test It just refers to what the test appearsappears to measure and not to to measure and not to what it measures in fact.what it measures in fact.

IOP 301-TIOP 301-T

Face validityFace validity

It is not useless since the aim It is not useless since the aim may be achieved by using may be achieved by using appropriate phrasing appropriate phrasing only !only !

In a sense, it ensures relevance In a sense, it ensures relevance to the context by employing to the context by employing correct expressions.correct expressions.

IOP 301-TIOP 301-T

Criterion-related validityCriterion-related validity

DefinitionDefinition

A A criterion variablecriterion variable is one with is one with (or against) which psychological (or against) which psychological measures are compared or measures are compared or evaluated.evaluated.

A criterion must be A criterion must be reliablereliable ! !

IOP 301-TIOP 301-T


It is a It is a quantitative quantitative procedure which procedure which involves calculating the involves calculating the correlation correlation coefficientcoefficient between one or more between one or more predictor variables and a criterion predictor variables and a criterion variable.variable.

IOP 301-TIOP 301-T


The validity of the measure is also The validity of the measure is also determined by its determined by its ability to predictability to predict performance on the criterion.performance on the criterion.

IOP 301-TIOP 301-T


Concurrent validityConcurrent validity Accuracy to identify the current Accuracy to identify the current

status regarding skills and status regarding skills and characteristicscharacteristics

Predictive validityPredictive validity Accuracy to forecast future Accuracy to forecast future

behaviour. It implicitly contains the behaviour. It implicitly contains the concept of decision-making concept of decision-making

IOP 301-TIOP 301-T


Warning !Warning !

Sometimes a factor may affect Sometimes a factor may affect the criterion such that it is no the criterion such that it is no longer a valid measure.longer a valid measure.

This is known as This is known as criterion criterion contaminationcontamination..

IOP 301-TIOP 301-T


Warning !Warning ! For example, the rater mightFor example, the rater might Be too lenientBe too lenient Commit Commit halo errorhalo error, i. e, rely on , i. e, rely on

impressionsimpressions

IOP 301-TIOP 301-T


Warning !Warning ! Therefore, we must make sure Therefore, we must make sure

that the criterion is free from that the criterion is free from biasbias (prejudice). (prejudice).

Bias definitely influences the Bias definitely influences the correlation coefficient.correlation coefficient.

IOP 301-TIOP 301-T

Common criterion measuresCommon criterion measures

Academic achievementAcademic achievement Performance in specialised trainingPerformance in specialised training Job performanceJob performance Contrasted groupsContrasted groups Psychiatric diagnosesPsychiatric diagnoses

IOP 301-TIOP 301-T


Academic achievementAcademic achievement Used for validation of intelligence, Used for validation of intelligence,

multiple aptitude and personality multiple aptitude and personality measures.measures.

Indices includeIndices include School, college or university gradesSchool, college or university grades Achievement test scoresAchievement test scores Special awardsSpecial awards

IOP 301-TIOP 301-T


Performance in specialised trainingPerformance in specialised training

Used for specific aptitude measuresUsed for specific aptitude measures

Indices include training outcomes for Indices include training outcomes for Technical coursesTechnical courses Academic coursesAcademic courses

IOP 301-TIOP 301-T


Job performanceJob performanceUsed for validating intelligence, special Used for validating intelligence, special

aptitude and personality measuresaptitude and personality measures

Indices include jobs in industry, business, Indices include jobs in industry, business, armed services, governmentarmed services, government

Tests describe duties performed and the Tests describe duties performed and the ways that they are measuredways that they are measured

IOP 301-TIOP 301-T


Contrasted groupsContrasted groups

Sometimes used for validating personality Sometimes used for validating personality measuresmeasures

Relevant when distinguishing the nature Relevant when distinguishing the nature of occupations (e.g, social and non-of occupations (e.g, social and non-social – Public Relations Officer and social – Public Relations Officer and Clerk)Clerk)

IOP 301-TIOP 301-T


Psychiatric diagnosesPsychiatric diagnoses

Used mainly for validating personality Used mainly for validating personality measuresmeasures

Based onBased on Prolonged observationProlonged observation Case historyCase history

IOP 301-TIOP 301-T


Academic Academic achievementachievement

Performance Performance in specialised in specialised

trainingtraining

Job Job performanceperformance

Contrasted Contrasted groupsgroups

Psychiatric Psychiatric diagnosesdiagnoses

IntelligenceIntelligence

AptitudeAptitude

PersonalityPersonality

IOP 301-TIOP 301-T

Ratings Ratings

Suitable for almost any type of Suitable for almost any type of measuremeasure

Very subjective in natureVery subjective in nature Often the only source availableOften the only source available

IOP 301-TIOP 301-T

Ratings Ratings These are given by teachers, lecturers, These are given by teachers, lecturers,

instructors, supervisors, officers, etc,…instructors, supervisors, officers, etc,…

Raters may be trained to avoid common errors Raters may be trained to avoid common errors likelike

Halo errorHalo error AmbiguityAmbiguity Error of central tendencyError of central tendency LeniencyLeniency

IOP 301-TIOP 301-T


We can always validate a new measure We can always validate a new measure by correlating it with another valid test by correlating it with another valid test (obviously, reliable as well !)(obviously, reliable as well !)

IOP 301-TIOP 301-T


Some modern and popular Some modern and popular criterion-criterion-prediction proceduresprediction procedures which are now which are now widely used arewidely used are

Validity generalisationValidity generalisation Meta-analysisMeta-analysis Cross-validationCross-validation

IOP 301-TIOP 301-T


Validity generalisationValidity generalisation

Schmidt, Hunter et al. showed that the Schmidt, Hunter et al. showed that the validity of tests measuring verbal, validity of tests measuring verbal, numeric and reasoning aptitudes can numeric and reasoning aptitudes can be generalised widely across be generalised widely across occupations (these require common occupations (these require common cognitive skills).cognitive skills).

IOP 301-TIOP 301-T


Meta-analysisMeta-analysis Method of reviewing research Method of reviewing research

literatureliterature Statistical integration and analysis of Statistical integration and analysis of

previous and current findings on a previous and current findings on a topic. topic.

Validation by correlationValidation by correlation

IOP 301-TIOP 301-T


Cross-validationCross-validation Refinement of initial measureRefinement of initial measure Application to another Application to another representative representative

normativenormative sample sample Recalculation of validity coefficientsRecalculation of validity coefficients Lowering of coefficient expected after Lowering of coefficient expected after

minimisation of minimisation of chance differenceschance differences and and sampling errorssampling errors (spuriousness) (spuriousness)

IOP 301-TIOP 301-T

Construct-identification validityConstruct-identification validity

Construct validity is the seConstruct validity is the sensitivity nsitivity of the instrument to pick up of the instrument to pick up minor variations in the concept minor variations in the concept being measured.being measured.

Can an instrument (questionnaire) to Can an instrument (questionnaire) to measure anxiety pick up different levels of measure anxiety pick up different levels of anxiety or just its presence or absence?anxiety or just its presence or absence?

IOP 301-TIOP 301-T

Construct-identification validityConstruct-identification validity Any data throwing light on the nature of the

trait and the conditions affecting its development and manifestations represent appropriate evidence for this validation.

Example

I have designed a program to lower girls’ Math phobia. The girls who complete my program should have lower scores on the Math Phobia Measure compared to their scores before the program and compared to the scores of girls who have not completed the program.

IOP 301-TIOP 301-T

Construct validity methodsConstruct validity methods

Correlational validityCorrelational validity Factor analysisFactor analysis Convergent and discriminant validityConvergent and discriminant validity

IOP 301-TIOP 301-T


Correlational validityCorrelational validity

This involves correlating a new meThis involves correlating a new measure with similar previous measurasure with similar previous measures of the same name.es of the same name.

Warning !Warning !

High correlation may indicate dupliHigh correlation may indicate duplication of measures.cation of measures.

IOP 301-TIOP 301-T

Construct validity methodsConstruct validity methodsFactor analysis (FA)Factor analysis (FA) It is a multivariate statistical It is a multivariate statistical

technique which is used to group technique which is used to group multiple variables into a few multiple variables into a few factors.factors.

In doing FA you hope to find In doing FA you hope to find clusters of variables that can be clusters of variables that can be identified as new hypothetical identified as new hypothetical factors.factors.

IOP 301-TIOP 301-T


Convergent and discriminant validityConvergent and discriminant validity

The idea is that a test should The idea is that a test should correlate highly with other similar correlate highly with other similar teststests and the test should and the test should correlate correlate poorly with tests that are very poorly with tests that are very dissimilardissimilar..

IOP 301-TIOP 301-T


Convergent and discriminant validityConvergent and discriminant validity

ExampleExample

A newly developed test of motor A newly developed test of motor coordination should coordination should correlate highlycorrelate highly with other tests of motor coordination.with other tests of motor coordination.

It should also have It should also have low correlationlow correlation with tests that measure attitudes.with tests that measure attitudes.

IOP 301-TIOP 301-T

Indices and interpretation of validityIndices and interpretation of validity

Validity coefficientValidity coefficient

- Magnitude of coefficient- Magnitude of coefficient

- Factors affecting validity- Factors affecting validity Coefficient of determinationCoefficient of determination Standard error of estimationStandard error of estimation Regression analysis (prediction)Regression analysis (prediction)

IOP 301-TIOP 301-T



DefinitionDefinition

It is a correlation coefficient between It is a correlation coefficient between the criterion and the predictor(s) the criterion and the predictor(s) variables.variables.

IOP 301-TIOP 301-T



Differential validityDifferential validity refers to refers to differences in the magnitude of differences in the magnitude of the correlation coefficients for the correlation coefficients for different groups of test-takers.different groups of test-takers.

IOP 301-TIOP 301-T


Magnitude of validity coefficientMagnitude of validity coefficient

Treated in the same way as the Treated in the same way as the Pearson correlation coefficient !Pearson correlation coefficient !

IOP 301-TIOP 301-T


Factors affecting validityFactors affecting validity

Nature of the groupNature of the group Sample heterogeneitySample heterogeneity Criterion-predictor relationshipCriterion-predictor relationship Validity-reliability proportionalityValidity-reliability proportionality Criterion contaminationCriterion contamination Moderator variablesModerator variables

IOP 301-TIOP 301-T



Nature of the groupNature of the group Consistency of the validity Consistency of the validity

coefficient for subgroups which coefficient for subgroups which differ in any characteristic (e. differ in any characteristic (e. g. age, gender, educational g. age, gender, educational level, etc, …) level, etc, …)

IOP 301-TIOP 301-T



Sample heterogeneitySample heterogeneity

A wider range of scores results A wider range of scores results in a higher validity coefficient in a higher validity coefficient (range restriction phenomenon)(range restriction phenomenon)

IOP 301-TIOP 301-T



Criterion-predictor relationshipCriterion-predictor relationship There must be a There must be a linearlinear

relationship between predictor relationship between predictor and criterion. Otherwise, the and criterion. Otherwise, the Pearson correlation coefficient Pearson correlation coefficient would be of no use!would be of no use!

IOP 301-TIOP 301-T



Validity-reliability proportionalityValidity-reliability proportionality Reliability has a limiting Reliability has a limiting

influence on validity – we influence on validity – we simply cannot validate an simply cannot validate an unreliable measure!unreliable measure!

IOP 301-TIOP 301-T



Criterion contaminationCriterion contamination Get rid of bias by measuring Get rid of bias by measuring

contaminated influences. contaminated influences. Then correct this influence Then correct this influence statistically by use of statistically by use of partial partial correlationcorrelation..

IOP 301-TIOP 301-T



Moderator variablesModerator variables Variables like age, gender, Variables like age, gender,

personality characteristics may personality characteristics may help to predict performance for help to predict performance for particular variables only – keep particular variables only – keep them in mind!them in mind!

IOP 301-TIOP 301-T


Coefficient of determinationCoefficient of determination Indicates the proportion of Indicates the proportion of

variance in the criterion variable variance in the criterion variable explained by the predictor.explained by the predictor.

E.g. If r = 0.9, rE.g. If r = 0.9, r2 2 = 0.81. 81% of = 0.81. 81% of the changes in the criterion is the changes in the criterion is accounted for by the predictor.accounted for by the predictor.

IOP 301-TIOP 301-T

Standard error of estimation (SE)Standard error of estimation (SE)

Treated just like the standard deviation.Treated just like the standard deviation. (True and predicted values for the (True and predicted values for the

criterion should differ by at most 1.96SE criterion should differ by at most 1.96SE at a 95% confidence level.)at a 95% confidence level.)


21 xyyest rsSE

IOP 301-TIOP 301-T


Regression analysisRegression analysis Mainly used to predict values of Mainly used to predict values of

the criterion variable. the criterion variable.

If If rr is high, prediction is more is high, prediction is more accurate.accurate.

Predicted values are obtained Predicted values are obtained from the from the line of best fitline of best fit..

IOP 301-TIOP 301-T


Regression analysisRegression analysis

Linear regressionLinear regression It involves one criterion It involves one criterion

variable but may involve one variable but may involve one ((simple regressionsimple regression) or more ) or more than one predictor variable than one predictor variable ((multiple regressionmultiple regression).).

IOP 301-TIOP 301-T


Regression analysisRegression analysis

IOP 301-TIOP 301-T


Simple linear regressionSimple linear regression

Multiple linear regressionMultiple linear regression

bXaY ˆ

nn XbXbXbbY ...ˆ22110

IOP 301-TIOP 301-T

Reliability and ValidityReliability and Validity A valid test is always reliable (in order for a test tA valid test is always reliable (in order for a test t

o be valid, it needs to be reliable in the first placo be valid, it needs to be reliable in the first place)e)

A reliable test is not always valid.A reliable test is not always valid. Validity is more important than reliability.Validity is more important than reliability. To be useful, a measuring instrument (test, To be useful, a measuring instrument (test,

scale) must be both reasonably reliable scale) must be both reasonably reliable andand valid.valid.

Aim for validity first, and then try make the test mAim for validity first, and then try make the test more reliable little by little, rather than the other waore reliable little by little, rather than the other way around.y around.

IOP 301-TIOP 301-T

Reliability and ValidityReliability and Validity

IFIF THENTHEN

UnreliableUnreliable Reliable, but not Reliable, but not

validvalid Unreliable and Unreliable and

invalidinvalid Reliable and validReliable and valid

Test validity is undermined.Test validity is undermined. Test is not useful.Test is not useful.

Test is definitely NOT useful!Test is definitely NOT useful!

Test can be used with good Test can be used with good results.results.

IOP 301-TIOP 301-T

Optimising reliability and ValidityOptimising reliability and Validity The more questions the better (the number of

test items)

Ask questions several times in slightly different ways (homogeneity)

Get as many people as you can in your program (sample size n)

Get different kinds of people in your program (sample heterogeneity)

Linear relationship between the test and the criterion (Pearson correlation coefficient)

IOP 301-TIOP 301-T

Selecting and creating measuresSelecting and creating measures Define the construct(s) that you want to measure

clearly

Identify existing measures, particularly those with established reliability and validity

Determine whether those measures will work for your purpose and identify any areas where you may need to create a new measure or add new questions

Create additional questions/measures

Identify criteria that your measure should correlate with or predict, and develop procedures for assessing those criteria

iop 301-t test validity. iop 301-t what is validity? in simple english, the validity of a test...

Documents

t slide

t test validity slide

t content validity measures

t face validity

criterion variable

criterion contamination

t content validity relevant

test measures