Estimation of Ability Using

Globally Optimal Scoring Weights

Shin-ichi Mayekawa

Graduate School of Decision Science and Technology

Tokyo Institute of Technology

2OutlineReview of existing methodsGlobally Optimal Weight: a set of

weights that maximizes the Expected Test Information

Intrinsic Category WeightsExamplesConclusions

3BackgroundEstimation of IRT ability on the basis of

simple and weighted summed score X.Conditional distribution of X given

as the distribution of the weighted sum of the Scored Multinomial Distribution.

Posterior Distribution of given X.

h(x) f(x|) h()Posterior Mean(EAP) of given X.Posterior Standard Deiation(PSD)

4Item Score

We must choose w to calculate X.

IRF

5Item Score

We must choose w and v to calculate X.

ICRF

6Conditional distribution of X given Binary items

Conditional distribution of summed score X.Simple sum: Walsh(1955), Lord(1969)Weighted sum: Mayekawa(2003)

Polytomous itemsConditional distribution of summed score X.

Simple sum: Hanson(1994), Thissen et.al.(1995)

With Item weight and Category weight: Mayekawa & Arai(2007)

7ExampleEight Graded Response Model items

3 categories for each item.

8Example (choosing weight)Example: Mayekawa and Arai (2008)

small posterior variance good weight. Large Test Information (TI) good weight

9Test Information Function　Test Information Function is proportional

to the slope of the conditional expectation of X given (TCC), and inversely proportional the squared width of the confidence interval (CI) of given X.

Width of CIInversely proportional to the conditional

standard deviation of X given .

10Confidence interval (CI) of given X 　

11Test Information Functionfor Polytomous Items 　

ICRF

12Maximization of the Test Informationwhen the category weights are known.Category weighted Item Score

and the Item Response Function

13Maximization of the Test Informationwhen the category weights are known.

14Maximization of the Test Informationwhen the category weights are known.Test Information

15Maximization of the Test Informationwhen the category weights are known.First Derivative

16Maximization of the Test Informationwhen the category weights are known.

17Globally Optimal WeightA set of weights that maximize

the Expected Test Informationwith some reference distribution of .

It does NOT depend on .

18Example

NABCT A B1 B2 GO GOINT A AINT Q1 1.0 -2.0 -1.0 7.144 7 8.333 8Q2 1.0 -1.0 0.0 7.102 7 8.333 8Q3 1.0 0.0 1.0 7.166 7 8.333 8Q4 1.0 1.0 2.0 7.316 7 8.333 8Q5 2.0 -2.0 -1.0 17.720 18 16.667 17Q6 2.0 -1.0 0.0 17.619 18 16.667 17Q7 2.0 0.0 1.0 17.773 18 16.667 17Q8 2.0 1.0 2.0 18.160 18 16.667 17

LOx LO GO GOINT A AINT CONST 7.4743 7.2993 7.2928 7.2905 7.2210 7.2564 5.9795

19Maximization of the Test Informationwith respect to the category weights.Absorb the item weight in category

weights.

20Maximization of the Test Informationwith respect to the category weights.Test Information

Linear transformation of the categoryweights does NOT affect the information.

21Maximization of the Test Informationwith respect to the category weights.First Derivative

22Maximization of the Test Informationwith respect to the category weights.Locally Optimal Weight

23Globally Optimal WeightWeights that maximize

the Expected Test Informationwith some reference distribution of .

24Intrinsic category weightA set of weights which maximizes:

Since the category weights can belinearly transformed, we set v0=0, ….. vmax=maximum item score.

25Example of Intrinsic Weights

26Example of Intrinsic Weightsh()=N(-0.5, 1): v0=0, v1=*, v2=2

27Example of Intrinsic Weightsh()=N(0.5, 1): v0=0, v1=*, v2=2

28Example of Intrinsic Weightsh()=N(1, 1 ): v0=0, v1=*, v2=2

29Summary of Intrinsic WeightIt does NOT depend on , but

depends on the reference distributionof : h() as follows.

For the 3 category GRM, we found thatFor those items with high discrimination

parameter, the intrinsic weights tendto become equally spaced: v0=0, v1=1, v2=2

The Globally Optimal Weight isnot identical to the Intrinsic Weights.

30Summary of Intrinsic WeightFor the 3 category GRM, we found that

The mid-category weight v1 increases according to the location of the peak ofICRF. That is:

The more easy the category is,

the higher the weight .

v1 is affected by the relative location ofother two category ICRFs.

31Summary of Intrinsic WeightFor the 3 category GRM, we found that

The mid-category weight v1 decreases according to the location of the reference distribution of h()

If the location of h() is high, the mostdifficult category gets relatively high weight,and vice versa.

When the peak of the 2nd categorymatches the mean of h(), we haveeqaully spaced category weights:

v0=0, v1=1, v2=2

32Globally Optimal w given v

33Test Information

LOx LO GO GOINT CONST 30.5320 30.1109 30.0948 29.5385 24.8868

34Test Information

35Bayesian Estimation of from X

36Bayesian Estimation of from X

37Bayesian Estimation of from X

(1/0.18)^2 = 30.864

38ConclusionsPolytomous item has the Intrinsic

Weight.By maximizing the Expected Test

Information with respect to either Item or Category weights, we can calculate the Globally Optimal Weights which do not depend on .

Use of the Globally Optimal Weights when evaluating the EAP of given X reduces the posterior variance.

39References

40

ご静聴ありがとうございました。

Thank you.

41

42

43