technical adequacy of tests

Technical Adequacy of Tests

Dr. Julie Esparza BrownSPED 512: Diagnostic Assessment

Two Essential Concepts

• Reliability: Test consistency• Validity: Test measures what it says it does

Psychometric Properties

•To create an instrument with sound psychometric properties means that it is reliable.

Reliability•Reliability is whether a test or measurement tool measures something consistently.

1. Reliable - Hits same part of the target each time, consistent but not valid because goal is center.

2. Valid – evenly distributed around goal, but not reliable because shots are off the mark and inconsistent.

3. Not reliable – shots not tightly clustered, not consistent and not valid because pattern is not around true center.

4. Reliable – darts are close together and valid because darts clustered around where they were aimed.

Reliability

Type of Reliability

When You Use It

How You Do It Ex. Of What You Can Say When You’re Done

Test-retest reliability

When you want to know whether a test is reliable over time.

Correlate the scores from a test given in Time 1 with the same test given in Time 2

The Bonzo test of identity formation for adolescents Is reliable over time

Parallel form of reliability

When you want to know if several different forms of a test are reliable or equivalent

Correlate the scores from one form of the test with the scores from a second form of the same test of the same content (but not the exact same test)

Reliability

Type of Reliability

When You Use It

How You Do It Ex. Of What You Can Say When You’re Done

Internal consistency reliability

When you want to know if the items on a test assess one, and only one, dimension.

Correlate each individual item score with the total score.

All of the items on the SMART Test of Creativity assess the same construct.

Interrater reliability

When you want to know whether there is consistency in the rating of some outcome.

Examine the percent of agreement between raters.

The interrater reliability for the best-dressed football player judging was .91, indicating a high degree of agreement between judges.

Interpreting Reliability Coefficients

• We want two things:o For reliability coefficients to be positive

(or direct) and not to be negative (or indirect)

oReliability coefficients that are as large as possible (between .00 and +1.00).

oReliability is a function of how much error contributes to the observed score. The lower the error, the higher the reliability.

To Increase Test Reliability…

• Ensure instructions are standardized across all settings when the test is administered.

• Increase the number of items because the larger the sample the more likely it is representative and reliable; this is especially true for achievement tests.

• Delete unclear items.• Minimize the effects of external events.

Validity• The property of an assessment tool that indicates the tool does what it says it does.• A valid test measure what it is supposed to.

Validity

Type of Validity

When You Use It

How You Do It Ex. Of What You Can Say When You’re Done

Content validity

When you want to know whether a sample of items truly reflects an entire universe of items in a certain topic.

Ask Mr. or Ms. Expert to make a judgment that the test items reflect the universe of items in the topic being measured.

My weekly quiz in my class fairly assesses the chapter’s content.

Criterion validity(concur-rent or predictive)

When you want to know if test scores are systematically related to other criteria that indicate the test taker is competent in a certain area

Correlate the scores from the test with some other measure that is already valid and assesses the same set of abilities

The EATS tests (of culinary skills) has been shown to be correlated with being a fine chef 2 years after culinary school (ex. Of predictive validity)

Validity

Type of Validity

When You Use It

How You Do It

Ex. Of What You Can Say When You’re Done

Construct validity

When you want to know if a test measures some underlying psychological construct

Correlate the set of test scores with some theorized outcome that reflects the construct for which the test is being designed.

It’s true – men who participate in body contact and physically dangerous sports score higher on the TEST(osterone) test of aggression.

Quantifying Validity• The maximum level of validity

is equal to the square root of the reliability coefficient.• Example, if the reliability

coefficient of a test is .87, the validity coefficient can be no larger than .93 (the square root of .87).

Reliability and Validity• You can have a test that is

reliable but not valid.• But, you can’t have a valid test

without it first being reliable.• If a test does what it is supposed

to, then it has to do it consistently to work!

Test Scores• A raw score or obtained score on a

test is the number of points obtained by an aminee.

• A true score is that part of an examinee’s observed score uninfluenced by random events.

• The error of measurement or error score is the difference between an obtained score and its theoretical true score counterpart.

Error Score•The error score is that part of the obtained score which is unsystematic, random, and due to chance.

Standard Error of Measurement

• The standard deviation of errors of measurement that is associated with the test scores for a specified group of test takers.”

• It is a measure of the variability of the errors of measurement.

• It is used to help us predict true scores based upon knowledge of obtained score.

• SEM is the standard deviation of errors of measurement associated with test scores for a specific group.

Score Bands• Score bands are sometimes called

confidence intervals or confidence bands because they allow us to make probabilistic statements of confidence about an unknown value.

• Score bands have lower and upper limits on the score scale and provide an estimate that is a range or band of possible test scores.

Score Bands• An example of a score band or

confidence interval is “I am 95 percent confident that the examinee’s obtained score will be between 46 and 54 (given a true score of 50 and an SEM of two).

• 68 percent confidence intervals are the most commonly used.

Confidence Intervals• In a normal distribution:

o The area between one SDs below and one SDs above the mean is 68 % of the total area under the curve

o The area between two (actually 1.96) SDs below and two SDs above the mean is 95%

o The area between 2.58 SDs below and 2.58 SDs above the mean is 99%

Confidence Intervals• If we add and subtract one SEM from a person’s test score, we will have an estimate of the true score.

Standard Scores• Test developers calculate the statistical

average based on the performance of students tested in the norming process of test development.

• That score is assigned a value.• Different performance levels are

calculated based on the differences among student scores from the statistical average and are expressed as standard deviations.

Standard Scores

• These standard deviations are used to determine at what scores fall within the above average, average, and below average ranges.

• Standard scores and standard deviations are different for different tests. Many of the commonly used tests, such as the Wechsler Intelligence Scales, have an average score of 100 and a standard deviation of 15.

Standard Scores• Standardized test scores enable us to

compare a student's performance on different types of tests.

• Although all test scores should be considered estimates, some are more precise than others.

• Standard scores and percentiles, for example, define a student's performance with more precision than do t-scores, z-scores, or stanines.

Standard Deviation• Standard deviation measures how

widely spread data points are. • If data values are all equal to one

another, then the standard deviation is zero.

• Under a normal distribution, ± one standard deviation encompasses 68% of the measurements and ± two standard deviations encompasses 96% of the measurements.

Standard Deviation• If a high proportion of data points

lie near the mean (average) value, then the standard deviation is small.o An experiment that yields data with a low

standard deviation is said have high precision.

• If a high proportion of data points lie far from the mean value, then the standard deviation is large. o An experiment that yields data with a high

standard deviation is said to have low precision.

Test Scores• Observed score – what someone

actually gets on a test.• True score – the true, 100% accurate

reflection of what the someone actually knows.

• An observed score is usually close to the true score but they are rarely the same.

• The difference between the two is in the amount of error that is introduced.

Observed Score• If someone has 89 on a test, but their

true score is 80, that means the 9 points in the difference (the error score) are due to error or the reason why individual test scores vary from being 100% true.

• What may be the source of the error:o Room is too warmo Person didn’t have time to studyo Person has a fevero ????

Observed Score

•We need to reduce the errors as much as possible.• The less error, the more reliable the score.

Reporting Test Scores• Score or confidence bands are the best

way to report test scores.• The score band provides reasonable limits

for estimating true score; it is an adequate approximation when the test reliability is reasonably high and the obtained score is not an extreme deviate from the mean of the reference group.

• You can say, “It is fairly likely your daughter’s true ability lies between 110 and 120.”

Test Selection• In selecting a published test:

oRead test manual to determine if it has reported the reliability, SEM and norms (including confidence bands)

o The above information is reported for a reference group similar to your examinee

oBe sure manual explains clearly how the information was gathered and how the confidence bands in the manual were calculated

Reliable and Value• If the tools you use to collect data are not reliable nor valid, then the results will be inconclusive.