shojima kojiro the national center for university entrance examinations [email protected]

20
Asymmetric Triangulation Scaling: An asymmetric MDS for extracting inter-item dependency structure from test data SHOJIMA Kojiro The National Center for University Entrance Examinations [email protected]

Upload: reilly

Post on 26-Jan-2016

25 views

Category:

Documents


3 download

DESCRIPTION

Asymmetric Triangulation Scaling: An asymmetric MDS for extracting inter-item dependency structure from test data. SHOJIMA Kojiro The National Center for University Entrance Examinations [email protected]. Purpose of Research. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Asymmetric Triangulation Scaling:An asymmetric MDS for extracting

inter-item dependency structure from test data

SHOJIMA KojiroThe National Center for University Entrance Examinations

[email protected]

Page 2: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Purpose of Research

• Development of method for visualizing inter-item dependency structure– Especially important for analyzing math test data

• Proposal: ATRISCAL– Asymmetric TRIangulation SCALing

– An asymmetric multidimensional scaling

– Conditional correct response rate matrix is object of analysis

Page 3: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Joint correct response rate matrix

• n×n symmetry matrix• The j-th diagonal element P(j,j)=P(j) – Correct response rate of item j

• The ij-th off-diagonal element P(i,j) – Joint correct response rate of items i and j– symmetry P(i,j)=P(j,i)

Item 1 Item 2 ⋯ Item nItem 1 P(1,1) P(1,2) ⋯ P(1,n)Item 2 P(2,1) P(2,2) ⋯ P(2,n)⋮ ⋮ ⋮ ⋱ ⋮

Item n P(n,1) P(n,2) ⋯ P(n,n)

Page 4: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Conditional correct response rate matrix

• n×n asymmetry matrix• The j-th diagonal element P(j|j)=P(j)/P(j)=1.0• The ij-th off-diagonal element P(j|i)=P(i,j)/P(i)– The correct response rate of item j when item i is

answered correctly– P(i|j)≠P(j|i): Usually asymmetric

Item 1 Item 2 ⋯ Item nItem 1 P(1,1) P(1,2) ⋯ P(1,n)Item 2 P(2,1) P(2,2) ⋯ P(2,n)⋮ ⋮ ⋮ ⋱ ⋮

Item n P(n,1) P(n,2) ⋯ P(n,n)

Item 1 Item 2 ⋯ Item nItem 1 P(1,1)/P(1) P(1,2)/P(1) ⋯ P(1,n)/P(1)Item 2 P(2,1)/P(2) P(2,2)/P(2) ⋯ P(2,n)/P(2)⋮ ⋮ ⋮ ⋱ ⋮

Item n P(n,1)/P(n) P(n,2)/P(n) ⋯ P(n,n)/P(n)

Item 1 Item 2 ⋯ Item nItem 1 1 P(2|1) ⋯ P(n|1)Item 2 P(1|2) 1 ⋯ P(n|2)⋮ ⋮ ⋮ ⋱ ⋮

Item n P(1|n) P(2|n) ⋯ 1

Page 5: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Multidimensional scaling (MDS)

X12

X3X9

X7

X13

X10

X8

X11

X2

X4

X6

X5

X14

X1

X15

Q1

QM Q2

O

Asymmetry conditional correct response rate matrixItem 1 Item 2 ⋯ Item n

Item 1 1 P(2|1) ⋯ P(n|1)Item 2 P(1|2) 1 ⋯ P(n|2)⋮ ⋮ ⋮ ⋱ ⋮

Item n P(1|n) P(2|n) ⋯ 1

Page 6: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

𝑂𝑋𝑖𝑗ሬሬሬሬሬሬሬሬԦ⊥𝑋𝑖𝑋𝑗ሬሬሬሬሬሬሬሬԦ

𝑂𝑋𝑖𝑗ሬሬሬሬሬሬሬሬԦ= 𝑘𝑂𝑋𝑖ሬሬሬሬሬሬሬԦ+ሺ1− 𝑘ሻ𝑂𝑋𝑗ሬሬሬሬሬሬሬԦ (0 ≤ 𝑘 ≤ 1)

|𝑂𝑋𝑖ሬሬሬሬሬሬሬԦ| = 𝑃(𝑖) |𝑂𝑋𝑗ሬሬሬሬሬሬሬԦ| = 𝑃(𝑗)

|𝑂𝑋𝑖𝑗ሬሬሬሬሬሬሬሬԦ| = 𝑃(𝑖,𝑗)

Item i Item jItem i 1 P(j|i)Item j P(i|j) 1

X j

Xi

O

Xij

𝑂𝑋𝑖𝑗ሬሬሬሬሬሬሬሬԦ= −𝑋𝑖𝑋𝑗ሬሬሬሬሬሬሬሬԦ∙𝑂𝑋𝑖ሬሬሬሬሬሬሬԦ|𝑋𝑖𝑋𝑗ሬሬሬሬሬሬሬሬԦ| 𝑋𝑖𝑋𝑗ሬሬሬሬሬሬሬሬԦ+ 𝑂𝑋𝑖ሬሬሬሬሬሬሬԦ

Relationship betweenitems i and j

Page 7: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

|𝑂𝑋𝑖ሬሬሬሬሬሬሬԦ| = 𝑃(𝑖) |𝑂𝑋𝑗ሬሬሬሬሬሬሬԦ| = 𝑃(𝑗)

|𝑂𝑋𝑖𝑗ሬሬሬሬሬሬሬሬԦ| = 𝑃(𝑖,𝑗)

Relationship betweenitems i and j

Item i Item jItem i 1 P(j|i)Item j P(i|j) 1

X j

Xi

O

Xij

𝑃ሺ𝑖ȁ�𝑗ሻ= 𝑃(𝑖,𝑗)𝑃(𝑗)

𝑃ሺ𝑗ȁ�𝑖ሻ= 𝑃(𝑖,𝑗)𝑃(𝑖)

|𝑂𝑋𝑖𝑗ሬሬሬሬሬሬሬሬԦ||𝑂𝑋𝑖ሬሬሬሬሬሬሬԦ|

|𝑂𝑋𝑖𝑗ሬሬሬሬሬሬሬሬԦ||𝑂𝑋𝑗ሬሬሬሬሬሬሬԦ|

𝑐𝑜𝑠∠𝑋𝑖𝑂𝑋𝑖𝑗

𝑐𝑜𝑠∠𝑋𝑗𝑂𝑋𝑖𝑗

Page 8: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Asymmetric correct response rate matrix

• The asymmetric matrix lacks information about the correct response rate of each item

• So we add the imaginary n+1-th item whose correct response rate is 1.0

– P(j|n+1)=P(j,n+1)/P(n+1)=P(j)– P(n+1|j)=P(j,n+1)/P(j)=1.0

Item 1 Item 2 ⋯ Item nItem 1 1 P(2|1) ⋯ P(n|1)Item 2 P(1|2) 1 ⋯ P(n|2)⋮ ⋮ ⋮ ⋱ ⋮

Item n P(1|n) P(2|n) ⋯ 1

Item 1 Item 2 ⋯ Item n Item n+1Item 1 1 P(2|1) ⋯ P(n|1) P(n+1|1)Item 2 P(1|2) 1 ⋯ P(n|2) P(n+1|2)⋮ ⋮ ⋮ ⋱ ⋮ ⋮

Item n P(1|n) P(2|n) ⋯ 1 P(n+1|n)Item n+1 P(1|n+1) P(2|n+1) ⋯ P(n|n+1) 1

Item 1 Item 2 ⋯ Item n Item n+1Item 1 1 P(2|1) ⋯ P(n|1) 1Item 2 P(1|2) 1 ⋯ P(n|2) 1⋮ ⋮ ⋮ ⋱ ⋮ ⋮

Item n P(1|n) P(2|n) ⋯ 1 1Item n+1 P(1) P(2) ⋯ P(n) 1

Expanded

Page 9: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Stress function

otherwisen

njiifij 1

,1  

Not

formedWellij 0

1

)1(

)|(1

)(,

nn

jipp

n

jiji  

22

22

||||

)'(||||

||

||)|(

jij

ijij

j

ij

OX

OXji

xxx

xxxx

1

)(,

2

1

)(,

2

*

)|(

)|()|(

)(

)()( n

jiji ijij

n

jiji ij

pjiλδ

jijipλ

T

FF

 

 

X

XX

Page 10: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Xij

δ (delta)Well-formed triangle

• The perpendicular foot from O falls on line segment XiXj

Not well-formed triangle

• The foot from O does NOT fall on line segment XiXj

Xi

X j

O

Xij

XiX j

O

δij=δji=1

δij=δji=0

Page 11: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

λ ( lambda )

Item 1 Item 2 ⋯ Item n Item n+1Item 1 1 P(2|1) ⋯ P(n|1) 1Item 2 P(1|2) 1 ⋯ P(n|2) 1⋮ ⋮ ⋮ ⋱ ⋮ ⋮

Item n P(1|n) P(2|n) ⋯ 1 1Item n+1 P(1) P(2) ⋯ P(n) 1

0.5

0.51

1

otherwisen

njiifij 1

,1  

Page 12: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Spatial indeterminacy and fixed coordinates

• Number of dimensions=3

• Coordinates of item n+1– (xn+1=0, yn+1=0, zn+1=1)

• Coordinate of item k, which has the lowest correct response rate– (xk=0, yk>0, zk)

• Coordinate of item l, which has a moderate P( ・ |k)– (xl>0, yl, zl)

Page 13: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Demonstration of exametrika

13

www.rd.dnc.ac.jp/~shojima/exmk/index.htm

Page 14: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Result of Analysis: Radial Map

• Red dots– Estimated coordinates

• Orange dots – Points of intersections

of extensions of red line segments and the surface of the hemisphere

Page 15: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Relationship betweenimaginary item n+1and item j

• P(j)→1.0• P(k)→0.0

Item j Item n+1Item j 1 1

Item n+1 P(j) 1

Xj

P(j)1

P(k)

Xk

Xn+1

O

Page 16: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Relashinship between items i and j

• P(i)<P(j)• P(i|j)→1.0• P(i|j)→0.0

Item i Item jItem i 1 P(j|i)Item j P(i|j) 1

Xn+1

Xi

O

Xj

Page 17: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Topographic Map

• The coordinates of orange points are projected onto the XY plane

• Voronoi tessellation

• Lift each Voronoi region by the length of the orange line segment

– Separate height with different colors

Page 18: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Mastery Maps

• For each examinee

Page 19: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Demonstration of exametrika

19

www.rd.dnc.ac.jp/~shojima/exmk/index.htm

Page 20: SHOJIMA Kojiro The National Center for University Entrance Examinations shojima@rd.dnc.ac.jp

Thank you for listening.

SHOJIMA KojiroThe National Center for University Entrance Examinations

[email protected]