IRT Applications of Kullback-IRT Applications of Kullback-Leibler Divergence and Analysis of Leibler Divergence and Analysis of its Distributionits Distribution

Dmitry BelovLaw School Admission Council

Ronald Armstrong Rutgers University

PlanPlan

1. Detecting answer copying

2. Other applications

3. General framework

4. What is asymptotic distribution of Kullback-Leibler divergence?

5. Summary

Given a set of response vectors, identify pairs of examinees involved in answer copying.

Problem statementProblem statement

Source SubjectCABDAEECDBA…

Response vector of an examineeResponse vector of an examinee

1111101111101110111111111 10110100000010110010ABCDEABCDEABCDEABCDEABCDE ABCDEABCDEABCDEABCDE

operational part (scored) has itemsidentical for all examinees

variable part (unscored) has itemsdifferent for adjacent examinees

General methodGeneral method

Stage 1: Identify potential subject (Kullback-Leibler Divergence).

Stage 2: Given potential subject identify possible source (K-Index or M-Index).

Identify potential subjectIdentify potential subject1111101111101110111111111 10110100000010110010

H()G()

Kullback-Leibler divergence:

One pair found in real dataOne pair found in real data

Operational Variable…1010111111001111101110… 0000000001000000001010100 …5445244253413211313455… 1534134455141423114323454

…1010111111001111101110… 0111111010100111111101101…5445244253413211313455… 1534134455141423114323454

Subject was seated in front and to the left of the Source.

Other pairs found in real dataOther pairs found in real data…101111111010111011… 100110000100000000001000000…531355321223155543… 255254322454242423122254142…531355321223155543… 445254322454242423125254142

…11100110101… 010000000100000000000100000…25312243312… 545322243141243332244534423…25312243312… 54332224312124333224453442

…00911010110100… 000100000000000000001000000…12445111122215… 323531215351541334523214245…12445111122215… 232532215114421252132314245

…11011011110110… 000000000010000001000000100…35122413433333… 121335232115132142254145441…35122413433333… 121332523211513214254145441

Other applications of comparing Other applications of comparing posteriors methodposteriors methodDetecting aberrant responding (operational items

vs. variable items, hard items vs. easy items, unexposed items vs. exposed items, uncompromised items vs. compromised items, analyzing test repeaters)

Checking unidimensionality of a test (items of one type vs. items of another type)

Detecting aberrant speed of responding

General frameworkGeneral framework1111101111101110111111111 10110100000010110010

H()G()

Kullback-Leibler divergence:

What is asymptotic distribution of What is asymptotic distribution of Kullback-Leibler divergence?Kullback-Leibler divergence?

What about posteriors?

Chang, H. H., & Stout, W. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37-52.

Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2004). Bayesian data analysis. Boca Raton: Chapman & Hall.

KL divergence between normal densitiesKL divergence between normal densities

True for any population, allows random

Consider two parallel tests Tg and Th with smooth information functions administered to an examinee with ability

Tests Tg and Th are different

Simulated dataSimulated data

10000 examinees from N(0,1)70 items test (a=1, b~N(0,1), c=0.1)

KSS=0.015

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.0 0.6 1.3 1.9 2.5 3.1 3.8 4.4 5.0 5.7

D(G||H)

CDF Empirical

Asymptotic

Real dataReal data

In LSAT variable part is about 4 times smaller than the operational part.

Operational Variable

+ odd responses from

+ even responses from

Real dataReal data

KSS=0.020

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.0 1.2 2.4 3.6 4.8 5.9

D(G||H)

CDF Empirical

Asymptotic

Alternative distributions of KLAlternative distributions of KL

1. Scaled chi-square with one degree of freedom

2. Scaled noncentral chi-square with one degree of freedom (to check for unidimensionality of a test )

3. Scaled F4. Scaled doubly noncentral F

Symmetric KL divergence between Symmetric KL divergence between normal densitiesnormal densities

SummarySummary

Comparing posteriors has many applications.

For the comparison one can use KL, phi, or (h, phi) divergences.

For normal posteriors in unidimensional IRT the asymptotic distribution of KL is analyzed.

LSAC uses corresponding software to detect aberrant responding.