work sample 3- item analysis

7/24/2019 Work Sample 3- Item Analysis

1/21

Zuki Flores

EIPT 5203: Measurement and Assessment

Unit 3 Project: Item Analysis

Discussion of Split Sample

To calculate item discrimination with data from twenty participants, Kellys 27% rule led me

to choose the top five scorers as my upper group and the bottom five scorers as my lower group.

When taking 27% of 20, we get 5.4, which I chose to round down to five participants per group.

To accomplish this, I sorted the spreadsheet from highest to lowest score. This placed

participants 2, 3, 1, 9, and 11 in the upper group and participants 10, 14, 15, 19, and 20 in the

lower group. It made for an upper group consisting only of people who scored 85% or higher on

the test and a lower group made up of people who scored 70% or below. Of the five participants

in the low group, four scored a 65% and one scored a 70% on the test.

Explanation of Calculations

To calculate the difficulty of each item, I used a spreadsheet program. First I added a

formula that found the sum of all correct answers for the item. Then I entered a formula that

divided the number of people who got the item correct by the number of people who took the

test. For example, 15 of the 20 people correctly responded to item one. The spreadsheet counted

the 15 correct answers and divided this by 20 to get 0.75 as the p-value.

To calculate the D-index, the spreadsheet had to find several values. First, it had to

calculate the p-value for the high group; then it determined a p-value for the low group. Finally,

it subtracted the low groups p-value from the high groups p-value. This is the D-index, or the


2/21

Zuki Flores

discrimination between those test-takers who have the knowledge being tested and those who

dont have the knowledge.

Item Analysis by Question

1. When the center of a roadway has a combination of a solid yellow and a broken yellow

line, you may:

a. Pass if the solid line is on your side.

b. Not pass if the broken line is on your side.

c. Pass if the broken line is on your side. ***

d. Pass on either side.

This item is not a priority for revision. Its difficulty is appropriate, and it has a positive D-index,

which is acceptable for a criterion-referenced test such as this one. Distractors A and D have not

garnered much attention, but the item content is on the easy side, so most people simply know

the answer.

A B C* D

Total 1 3 15 1

Upper 0 0 5 0

Middle 1 2 6 1

Lower 0 1 4 0

Item Difficulty (p value) 0.75

p Value for Upper Group 1.0

p Value for Lower Group 0.8

Item Discrimination (D-index) 0.2


3/21

Zuki Flores

2. When you come to an intersection and try to make a move, you must follow the

_______________________ before you pull out.

a. Left, left, and right rule

b. Left and right rule

c. Right, left, and right ruled. Left, right, and left rule ***

This item does not require considerable revision. It has an average to easy difficulty, and the

people in the lower group are fairly distributed among the answer choices. However, distractor Acould be eliminated, as it is clearly not the correct answer. It is unlikely to be chosen by future

participants with common sense.

A B C D*

Total 0 2 3 15

Upper 0 0 1 4

Middle 0 1 0 9

Lower 0 1 2 2






4/21

Zuki Flores

3. This diamond-shaped sign warns of:

a. A yield sign ahead.

b. A closed road sign ahead.

c. A traffic merging area ahead.

d. A stop sign ahead. ***

This item is extremely easy. Revision is unlikely to change this, though, as the picture/content is

just naturally easy. The distracters are all reasonable, but nothing is going to make people forget

that the red octagon means stop.

A B C D*

Total 0 0 0 20

Upper 0 0 0 5

Middle 0 0 0 10

Lower 0 0 0 5

Item Difficulty (p value) 1

p Value for Upper Group 1

p Value for Lower Group 1

Item Discrimination (D-index) 0


5/21

Zuki Flores

4. This sign warns of:

a. A steep hill ahead. ***

b. A slippery road ahead.

c. A slow-moving vehicle ahead.

d. Low clearance ahead.

Again, the pictorial nature of this question makes it easy content for a multiple-choice test. The

distracters C and D were both non-distracting to test-takers. Of these two, the one that should

most definitely be eliminated is C. Common sense will let people know that a permanently

posted yellow diamond sign cannot possibly warn of the presence of a moving vehicle of any

speed.

A* B C D

Total 18 2 0 0

Upper 5 0 0 0

Middle 10 0 0 0

Lower 3 2 0 0






6/21

Zuki Flores

5. A flashing yellow X over a traffic lane means:

a. A driver may use the lane only to make a left turn. ***

b. A driver must change lanes.

c. A driver is not allowed to use the traffic lane.

d. A driver may use the lane in an emergency.

This is a classic trick question. The people who know the content are being led to choose C

over any other choice. It is not merely a difficult question, though its p-value indicates very high

difficulty. It requires revision as follows.

REV 5. A flashing yellow X over a traffic lane means that a driver:

A. may use the lane only to make a left turn

B. may use the lane only in an emergency

C. may need to change lanes ahead

A* B C D

Total 3 5 8 4

Upper 1 1 3 0

Middle 2 2 4 2

Lower 0 2 1 2






7/21

Zuki Flores

6. What does this sign mean?

a. A narrow bridge or underpass ahead ***

b. A hill ahead

c. A one-way road ahead

d. The road ahead is closed

This is yet another easy content item that is working fairly well, even though the distractors are

not very attractive. The people who missed the question are spread out among the distracters.

A* B C D

Total 17 1 1 1

Upper 5 0 0 0

Middle 9 0 0 1

Lower 3 1 1 0






8/21

Zuki Flores

7. According to Pennsylvania law, ___________________ are required by law to be

buckled up.

a. All front seat passengers and rear seat passengers under the age of 18 ***

b. All rear seat passengers above the age of 18

c. All front seat passengers above the age of 18d. All front seat passengers

This question just may not be well-suited for the multiple choice format. It is very obvious that

B and C are not correct, but it is apparent that some people of all levels of knowledge are tornbetween A and D. If I may, I would suggest a check all that are correct version of this item

with all-or-nothing grading.

REV 7. According to Pennsylvania law, which of these are required to be buckled up?

Check ALL that are correct.

___ drivers under 18

___ drivers over 18

___ front seat passengers under 18

___ front seat passengers over 18___ back seat passengers under 18

___ back seat passengers over 18

A* B C D

Total 10 0 0 10

Upper 3 0 0 2

Middle 5 0 0 5

Lower 2 0 0 3






9/21

Zuki Flores

8. In Pennsylvania, you are considered to be driving under the influence if your blood

alcohol level is __________________, unless you are under 21 years of age.

a. .02% or less

b. Between .01 and .09

c. .06% or lessd. .08% or more ***

This item does not require revision. It is likely that the content being tested here is heavily

emphasized in driver preparation classes and on law enforcement advertisement. It is going to bean easy item no matter what.

A B C D*

Total 0 0 2 18

Upper 0 0 0 5

Middle 0 0 2 8

Lower 0 0 0 5






10/21

Zuki Flores

9. What does this sign mean?

a. A left turn ahead

b. The road merges ahead

c. A series of curves ahead

d. A road entering a curve ahead ***

This is yet another easy item due to picture clues. Even a reasonable distracter like B cant fool

those who are generally scoring low on the assessment.

A B C D*

Total 0 0 0 20

Upper 0 0 0 5

Middle 0 0 0 10

Lower 0 0 0 5






11/21

Zuki Flores

10. A single broken yellow centerline shows the center of a two-way, two-lane road where:

a. Passing is prohibited, except in emergency.

b. Passing is permitted on either side, if safe conditions exist. ***

c. Passing is permitted on one side.

d. No passing is permitted.

This item has a key that is very obviously correct because it is longer, more detailed, and

qualified by the condition. Because of this, the item is not discriminating at all. I would suggest

the following revision.

REV 10. A single broken yellow centerline shows the center of a two-way, two-lane road where:

A. No passing is permitted, except in emergencies

B. Passing is permitted on either side, if safe conditions exist

C. Passing is permitted on one side, if safe conditions exist

A B* C D

Total 1 17 1 1

Upper 0 5 0 0

Middle 1 7 1 1

Lower 0 5 0 0






12/21

Zuki Flores

11. State law requires drivers to use their headlights when they cannot see pedestrians or

vehicles on the highway ___________________ ahead of the vehicle they are operating.

a. 1,000 feet ***

b. 3,000 feet

c. 4,000 feetd. 2,000 feet

This item needs distracter revisions for B and C. These distances must sound far too great to be

plausible. None of the participants in the lower group chose these two distracters. Instead, thequestion could read as follows:

REV 11. State law requires drivers to use their headlights when they cannot see pedestrians or

vehicles on the highway ___________________ ahead of the vehicle they are operating.

A. 100 feet

B. 200 feet

C. 1,000 feet

D. 2,000 feet

A* B C D

Total 14 0 2 4

Upper 4 0 1 0

Middle 8 0 1 1

Lower 2 0 0 3






13/21

Zuki Flores

12. If you must drive below 40 mph on a limited access highway, use ________________ to

warn the drivers behind you.

a. Your emergency flashers ***

b. Your high-beam headlights

c. Your low-beam headlightsd. None of the above

This item is extremely easy, especially as it is written. Nobody will choose distracters B or Cbecause it is obvious that your headlights will do nothing to warn people behindyou.

Furthermore, the word your does not need to be repeated in each of the choices. Instead, the

question stem could read as follows.

REV 12. If you must drive below 40 mph on a limited access highway, use your

________________ to warn other drivers on the road.

A. emergency flashers

B. high-beam headlights

C. low-beam headlights

D. none of the above

A* B C D

Total 20 0 0 0

Upper 5 0 0 0

Middle 10 0 0 0

Lower 5 0 0 0






14/21

Zuki Flores

13. A driver must park a vehicle ________________ away from the curb.

a. No more than 100 inches

b. No more than 36 inches

c. No more than 25 inches

d. No more than 12 inches ***

This item is similar to item 11 in that the number values could be made more reasonable to

prevent the key from sticking out like a sore thumb. As it is written, it is not discriminating at all

between people who know the content and those who dont.

A B C D*

Total 0 0 0 20

Upper 0 0 0 5

Middle 0 0 0 10

Lower 0 0 0 5






15/21

Zuki Flores

14. If a vehicle using high-beams comes toward you, you must ________________ to avoid

being blinded.

a. Look into the side mirrors

b. Look at the oncoming vehicle

c. Look toward the left side of the roadd. Look toward the right side of the road ***

In this question, distracters B and C are not working properly. In fact, distracter C may be giving

away the key. The item may do a better job of discriminating if it read differently.

REV 14. What is the best way to avoid being blinded if a vehicle using high-beams comes

toward you?

A. look into the side mirrors

B. look into the rearview mirror

C. look toward the right side of the road

D. look toward the center line of the road

A B C D*

Total 4 0 0 16

Upper 0 0 0 5

Middle 3 0 0 7

Lower 1 0 0 4






16/21

Zuki Flores

15. When driving near a blind pedestrian who is carrying a white cane or walking with a

guide dog, you must:

a. Proceed with the same speed.

b. Stop, yield the right-of-way, and proceed with caution. ***

c. Blow your horn to alert the pedestrian.d. Accelerate and pass quickly.

This item does a surprisingly good job of discriminating given how poorly the distracters C and

D are working. I would suggest simply eliminating choices C and D, or this item might evenwork better as a True/False question.

REV 15. When driving near a blind pedestrian who is carrying a white cane or walking with a

guide dog, you should proceed with the same speed. True or False

A B* C D

Total 7 10 1 2

Upper 0 5 0 0

Middle 3 4 1 2

Lower 4 1 0 0






17/21

Zuki Flores

16. Zones or spots beside or behind your vehicle where you cannot see other vehicles or

pedestrians are called:

a. Blind spots. ***

b. Zero spots.

c. Invisible spots.d. No spots.

This item is of very easy content. Revision is not necessary even though there is no

discriminating or distracting going on.

A* B C D

Total 20 0 0 0

Upper 5 0 0 0

Middle 10 0 0 0

Lower 5 0 0 0






18/21

Zuki Flores

17. Which of the following statements regarding right-of-way at intersections is NOT true?

a. At a four-way stop, all vehicles must stop.

b. Drivers turning left must yield to oncoming vehicles going straight ahead.

c. A vehicle entering a public highway from a private road must stop and wait until

the main road is clear.d. Drivers already in a traffic circle must yield the right-of-way to drivers entering

the circle. ***

This does not seem like the best item to use in a multiple-choice question. The negative in the

stem, though pointed out by all capital letters, is still not appropriate. It makes choices A and Bstand out as incorrect because those are obviously true. The other two choices are very lengthy

and may be unduly increasing the cognitive load. Either one of those statements could instead be

written as a True/False application scenario.

A B C D*

Total 0 0 4 16

Upper 0 0 0 5

Middle 0 0 2 8

Lower 0 0 2 3






19/21

Zuki Flores

18. Large electronic flashing arrow panels may be used in work zones day and night to:

a. Guide drivers into certain traffic lanes. ***

b. Stop speeding drivers.

c. Divert drivers into work zones.

d. Guide drivers into certain parking zones.

Everything about this question is working well, except distracter C. If the test creator better can

think of nothing better than divert drivers into work zones, then it can simply be a 3-choice

question. Additionally, distracter B could be a little less obvious by saying discourage rather

than stop speeding drivers.

A* B C D

Total 13 2 0 5

Upper 4 1 0 0

Middle 7 0 0 3

Lower 2 1 0 2






20/21

Zuki Flores

19. This sign means:

a. An intersection of U.S. route 22 is ahead. ***

b. A mile marker 22 is ahead.

c. Exit 22 is ahead.

d. Travel 22 miles for next exit.

This question is likely just easy content because people who have ridden in cars have seen signs

such as this one and the actual signs for each of the distracters.

A* B C D

Total 20 0 0 0

Upper 5 0 0 0

Middle 10 0 0 0

Lower 5 0 0 0






21/21

Zuki Flores

20. To avoid rear-end crashes, all drivers must leave plenty of space ahead by following:

a. The four-second rule. ***

b. The three-second rule.

c. The two-second rule.

d. The one second rule.

This is a very tricky question. Even in the upper group, more people chose C over the key. It is

also a very difficult question because it is so specific and the choices are so close together.

People probably ruled out A and D simply because they were the extremes. If the question has to

remain on the test, then I would suggest the distracters be changed. I find this one to be beyondhelp, though.

REV 20. To avoid rear-end crashes, all drivers must leave plenty of space ahead by following the

________________.

A. two-second rule

B. four-second rule

C. six-second rule

A* B C D

Total 3 0 17 0

Upper 2 0 3 0

Middle 1 0 9 0

Lower 0 0 5 0





work sample 3- item analysis

Documents