introduction to item analysis objectives: to begin to understand how to identify items that should...

Introduction to Item AnalysisIntroduction to Item Analysis

Objectives:Objectives:To begin to understand how to To begin to understand how to

identify items that should be identify items that should be improved or eliminated.improved or eliminated.

Item AnalysisItem Analysis

No item is perfect.No item is perfect.

An item might be ambiguous, too An item might be ambiguous, too simple, too difficult, or non-simple, too difficult, or non-discriminating.discriminating.

Non-discriminating means the item can Non-discriminating means the item can not be used to measure individual not be used to measure individual differences on the trait that is differences on the trait that is measured by the test.measured by the test.

Achievement TestsAchievement Tests

Item analysis can help diagnose Item analysis can help diagnose student errors.student errors.

It can help improve the quality of It can help improve the quality of tests.tests.

It can lead to instructional It can lead to instructional improvements.improvements.

Achievement TestsAchievement Tests

Item analysis can identify Item analysis can identify problems with the answer key on a problems with the answer key on a teacher-made test, or problems teacher-made test, or problems with the machine scoring on a with the machine scoring on a standardized test.standardized test.

It can help isolate items where the It can help isolate items where the students “guessed” a lot.students “guessed” a lot.

The Basic Indexes of Item AnalysisThe Basic Indexes of Item Analysis

Difficulty – What percentage of Difficulty – What percentage of respondents got the item “right” or respondents got the item “right” or indicated that they possess the trait indicated that they possess the trait being measured?being measured?

Discrimination – The extent to which Discrimination – The extent to which the item differentiates between the item differentiates between persons with high and low scores on persons with high and low scores on the test.the test.

The Basic Indexes of Item AnalysisThe Basic Indexes of Item Analysis

Difficulty – Measured by a simple Difficulty – Measured by a simple percentage.percentage.

Discrimination – Measured by the Discrimination – Measured by the difference between high and low difference between high and low scoring groups on the proportion scoring groups on the proportion answering the “right” answer.answering the “right” answer.

DiscriminationDiscriminationItems that are poor discriminators Items that are poor discriminators

should be eliminated or modified.should be eliminated or modified.

Balance content validity with Balance content validity with construct validity.construct validity.

High discrimination tends to High discrimination tends to increase reliability.increase reliability.

Discrimination and DifficultyDiscrimination and Difficulty

Discrimination and difficulty are Discrimination and difficulty are related.related.

With very difficult items it is harder to With very difficult items it is harder to show high discrimination.show high discrimination.

Balance purpose of assessment with Balance purpose of assessment with the range of difficulty of items.the range of difficulty of items.

Generally .2 - .8 difficulty is desired.Generally .2 - .8 difficulty is desired.

DiscriminationDiscrimination

For educational achievment tests, For educational achievment tests, you want to look at the discrimination you want to look at the discrimination for the “distractors” or wrong options for the “distractors” or wrong options on a multiple choice item.on a multiple choice item.

Ideally, you want them to be selected Ideally, you want them to be selected mostly by the low scoring mostly by the low scoring respondents.respondents.


For Educational tests –For Educational tests –Form a 2 x 2 matrix that crosses Form a 2 x 2 matrix that crosses

“Right” vs. “Wrong” on the item by “Right” vs. “Wrong” on the item by “High” vs. “Low” on the total score of “High” vs. “Low” on the total score of the test.the test.

High and Low can be determined by a High and Low can be determined by a median split, or by quartiles, taking median split, or by quartiles, taking the highest and lowest quartile.the highest and lowest quartile.


For Psychological tests –For Psychological tests –Form a 2 x 2 matrix that crosses Form a 2 x 2 matrix that crosses

“High” vs. “Low” on the item by “High” vs. “Low” on the item by “High” vs. “Low” on the total score of “High” vs. “Low” on the total score of the test.the test.

High and Low can be determined by a High and Low can be determined by a median split, or by quartiles, taking median split, or by quartiles, taking the highest and lowest quartile.the highest and lowest quartile.

An Example from the PRIAn Example from the PRI

““I am able to ask for emotional I am able to ask for emotional support.”support.”

Part of the Social Part of the Social Resourcefulness FactorResourcefulness Factor

Part of the Assistance in Part of the Assistance in Relationships subscale.Relationships subscale.


Statistics

344 344 344 344

0 0 0 0

3.34 3.8723 3.8228 3.7902

.060 .03057 .02431 .02364

4.00 4.0000 3.8571 3.8171

4 4.00 3.93 3.73a

1.116 .56694 .45085 .43850

1.245 .321 .203 .192

-.347 -.430 -.350 -.182

.131 .131 .131 .131

-.721 .541 .805 .645

.262 .262 .262 .262

4 3.00 2.93 2.71

1 2.00 2.00 2.21

5 5.00 4.93 4.91

1149 1332.08 1315.04 1303.82

Valid

Missing

N

Mean

Std. Error of Mean

Median

Mode

Std. Deviation

Variance

Skewness

Std. Error of Skewness

Kurtosis

Std. Error of Kurtosis

Range

Minimum

Maximum

Sum

q15 AST SOC PRI

Multiple modes exist. The smallest value is showna.


q15

20 5.8 5.8 5.8

67 19.5 19.5 25.3

81 23.5 23.5 48.8

128 37.2 37.2 86.0

48 14.0 14.0 100.0

344 100.0 100.0

1

2

3

4

5

Total

ValidFrequency Percent Valid Percent

CumulativePercent

Correlations

.747 .567 .378

.000 .000 .000

344 344 344

Pearson Correlation

Sig. (2-tailed)

N

q15AST SOC PRI


Crosstab

138 30 168

82.1% 17.9% 100.0%

83.1% 16.9% 48.8%

40.1% 8.7% 48.8%

28 148 176

15.9% 84.1% 100.0%

16.9% 83.1% 51.2%

8.1% 43.0% 51.2%

166 178 344

48.3% 51.7% 100.0%

100.0% 100.0% 100.0%

48.3% 51.7% 100.0%

Count

% within q15

% within AST

% of Total

Count

% within q15

% within AST

% of Total

Count

% within q15

% within AST

% of Total

LO

HI

q15

Total

LO HI

AST

Total


Crosstab

111 57 168

66.1% 33.9% 100.0%

69.8% 30.8% 48.8%

32.3% 16.6% 48.8%

48 128 176

27.3% 72.7% 100.0%

30.2% 69.2% 51.2%

14.0% 37.2% 51.2%

159 185 344

46.2% 53.8% 100.0%

100.0% 100.0% 100.0%

46.2% 53.8% 100.0%

Count

% within q15

% within SOC

% of Total

Count

% within q15

% within SOC

% of Total

Count

% within q15

% within SOC

% of Total

LO

HI

q15

Total

LO HI

SOC

Total


Crosstab

94 74 168

56.0% 44.0% 100.0%

59.1% 40.0% 48.8%

27.3% 21.5% 48.8%

65 111 176

36.9% 63.1% 100.0%

40.9% 60.0% 51.2%

18.9% 32.3% 51.2%

159 185 344

46.2% 53.8% 100.0%

100.0% 100.0% 100.0%

46.2% 53.8% 100.0%

Count

% within q15

% within PRI

% of Total

Count

% within q15

% within PRI

% of Total

Count

% within q15

% within PRI

% of Total

LO

HI

q15

Total

LO HI

PRI

Total


Discrimination ClassificationItem Difficulty AST SOC PRI AST SOC PRI15 0.512 0.682 0.454 0.262 83.14% 69.48% 59.59%

AST SOC PRIItem - Total Score Correlation 0.747 0.567 0.378Reliability of Scale Score 0.579 0.822 0.996Reliability without Item 0.528 0.814 0.966

introduction to item analysis objectives: to begin to understand how to identify items that should...

Documents