ingénierie cognitive pour les environnements …€¢ could create item types, but unreliable and...

63
Ing ´ enierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion Ing´ enierie cognitive et environnements d’apprentissage Ing ´ enierie cognitive pour les environnements d’apprentissage M.C. Desmarais Polytechnique Montr´ eal Informatique cognitive, UQAM, 10 juin 2015 Desmarais Ing ´ enierie cognitive 1/60

Upload: truongminh

Post on 17-May-2018

214 views

Category:

Documents


1 download

TRANSCRIPT

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Ingenierie cognitive et environnements d’apprentissage

Ingenierie cognitive pour les environnementsd’apprentissage

M.C. Desmarais

Polytechnique Montreal

Informatique cognitive, UQAM, 10 juin 2015

Desmarais Ingenierie cognitive 1/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Ingenierie cognitive

Structures de connaissances

Arrimage des items aux competences latentes

La representativite d’un modele

Conclusion

Desmarais Ingenierie cognitive 2/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Problem statement

Desmarais Ingenierie cognitive 3/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Diagnostic des connaissances

Desmarais Ingenierie cognitive 4/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Exerciseur

Desmarais Ingenierie cognitive 5/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Exerciseur

Desmarais Ingenierie cognitive 6/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Mapping items to skillsExample 1

Problem VEUPS1aCompute cosine of CAB

9% success rate (N=246)

Problem VEUPS3Compute cosine of CAD

54% success rate (N=675)

Desmarais Ingenierie cognitive 7/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Mapping items to skillsExample 1

Problem VEUPS1aCompute cosine of CAB9% success rate (N=246)

Problem VEUPS3Compute cosine of CAD54% success rate (N=675)

Desmarais Ingenierie cognitive 7/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Mapping items to skillsExample 2

Problem GPCER2aCompute area

44% success rate (N=281)

Problem GPCER2bCompute area

79% success rate (N=841)

Desmarais Ingenierie cognitive 8/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Mapping items to skillsExample 2

Problem GPCER2aCompute area44% success rate (N=281)

Problem GPCER2bCompute area79% success rate (N=841)

Desmarais Ingenierie cognitive 8/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Ingenierie cognitive

Structures de connaissances

Arrimage des items aux competences latentes

La representativite d’un modele

Conclusion

Desmarais Ingenierie cognitive 9/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Modelisation des competences (etude 1)

Objectifs de l’etude 1 :

• Determiner le modele de diagnostic des competences leplus performant

• Comparer une approche basee sur des traits latents aune approche basee sur les caracteristiquesobservables uniquement

Desmarais Ingenierie cognitive 10/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

A Bayesian Network example

BN example from Vomlel (2004)

latent

8>>><

>>>:

item

(

ACMI ACIM ACD

CL

MTCIM ADCDCMI

ACL

SB

X11X10X1

CP

HV1

X3

X9X8

X6X5 X4

X13 X20 X16X14 X12 X7 X18X15X19

X2

X17

MMT4MMT1 MMT2

MAD MSB

MMT3 MC

Desmarais Ingenierie cognitive 11/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Network topologies

latent

8>>><

>>>:

item

(

Desmarais Ingenierie cognitive 12/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Graphical representation of an IRT model

latent

(

item

(X

2

• • •X1

Xn

• IRT: a single node (dimension/skill) to predict theoutcome to items X1, X2, ..., Xn.

• Logistic function determines probability of success:

P (Xi|✓) =1

1 + e�ai(✓�bi)

• Estimation of ability based on:

argmax

✓P (✓|X) = P (✓|X1, X2, ..., Xn) =

nYP (Xi|✓)

Desmarais Ingenierie cognitive 13/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Network topologies

latent

8>>><

>>>:

item

(

Desmarais Ingenierie cognitive 14/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Item to item approach

item

8>>>>>><

>>>>>>: X2

• • •X1

Xn

Xk

• One network for each observable node• Naive Bayes and simple posterior probability•arg max

Xk={0,1}P (Xk|X) =

Y

Xi2XP (Xi|Xk)

• Conditional probabilities replace the logistic function of IRT.They are directly obtained from frequency tables since allnodes are observable.

Desmarais Ingenierie cognitive 15/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

TAN: Tree-Augmented Network

item

8>>>>>><

>>>>>>: X2

X3

X1

Xk

X4

• A Naive Bayesian Network with a tree structure over leafnodes.

• Each leaf node can have at most two parents: Xk andsome other leaf node.

• Follows the usual Bayesian Network semantics:P (X) =

Y

Xi2XP (Xi|Xpa(Xi))

Desmarais Ingenierie cognitive 16/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Performance comparison

• 1 model for single latent trait

• IRT: Item Response Theory X2

• • •X1

Xn

• 3 models for item to item

• NB: Naive Bayes X2

• • •X1

Xn

Xk

• TAN: Tree Augmented Network• BNC: Bayesian Network Classifier, Variant of TAN with K2

algorithm

X2

X3

X1

Xk

X4

Desmarais Ingenierie cognitive 17/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Simulation methodology

Simulation that consists in providing a subset of observednodes and predicting the outcome to all other nodes

• N-folds: 10 to 20 folds with test sample size from 10 to 100• Choice of 4–5 predictors (other items) based on

correlation with target• Measure of:

• AUC (Area Under the ROC Curve)• Accuracy at 0.5 cutoff

Desmarais Ingenierie cognitive 18/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

ROC: Receiver Operator Characteristic Curve

(from Tape, T.G. Interpreting Diagnostic Tests.)(http://gim.unmc.edu/dxtests/roc3.htm)

Desmarais Ingenierie cognitive 19/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Data sets

1 College mathematics: 60 items on algebra and functions,trigonometry, geometry, matrices, and calculus;246 respondents newly registered in engineering

2 Fraction algebra: 20 items on basic fraction algebra rules;171 pupils, 10-12 years old

3 LSAT: 5 items from Law School Admission Test;1000 respondents (higher average: 76%)

4 UNIX: 34 items on UNIX shell commands;48 respondents (wide ranging scores)

Desmarais Ingenierie cognitive 20/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

AUC (Area under ROC Curve) performance

TAN BNC NB IRT AoV significance levelAll TAN-IRT w/o IRT

Coll. math 0.77 0.76 0.75 0.74 *** *** **Frac. algebra 0.90 0.90 0.88 0.85 *** *** **

LSAT 0.59 0.59 0.58 0.57 - - -UNIX 0.96 0.96 0.95 0.91 *** *** -

*** p < 0.001, ** p < 0.01, * p < 0.05 - p > 0.05

N.B. 0.91 ! 0.96 = 44% error reduction

0.85 ! 0.90 = 33% error reduction

0.50 ! 0.54 = 8% error reduction

Desmarais Ingenierie cognitive 21/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Accuracy results

Accuracy at 0.5 cutoff

TAN BNC NB IRT AoV significance levelAll TAN-IRT w/o IRT

Coll. math 0.64 0.64 0.63 0.65 - - -Frac. algebra 0.70 0.70 0.68 0.71 - - -

LSAT 0.83 0.83 0.83 0.83 - - -UNIX 0.93 0.94 0.91 0.86 *** *** ***

*** p < 0.001, ** p < 0.01, * p < 0.05 - p > 0.05

Desmarais Ingenierie cognitive 22/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Discussion

• Item to Item models either outperform or match thesingle skill IRT model

• Large differences between data sets• Small size favours item-to-item Bayesian models

• TAN/BNC slightly better than NB

Desmarais Ingenierie cognitive 23/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Item to Item vs. Latent modelsAdvantages

Advantages of Item to Item models:• Good performance

• Still needs comparison to multidimensional IRT and othermore sophisticated models

• Does have sound cognitive foundations (cf. KnowledgeSpaces of Falmagne and Doignon, 1985)

• No knowledge engineering at the modeling phase• KE postponed to the skills assessment phase

Desmarais Ingenierie cognitive 24/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Item to Item vs. Latent modelsDrawbacks

Drawbacks of Item to Item models:• May perform better, but does not replace knowledge

engineering for didactic purposes• Adding a new item requires learning with old items

• Actually a big drawback• IRT avoids this problem (parameter estimation is not

relative to other items)• Could create item types, but unreliable and falls into

knowledge engineering issues

Desmarais Ingenierie cognitive 25/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Ingenierie cognitive

Structures de connaissances

Arrimage des items aux competences latentes

La representativite d’un modele

Conclusion

Desmarais Ingenierie cognitive 26/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Arrimage des items aux competences latentes

Desmarais Ingenierie cognitive 27/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Problem statement

Desmarais Ingenierie cognitive 28/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Diagnostic des connaissances

Desmarais Ingenierie cognitive 29/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Four Q-matricesVariations on Tatsuoka’s fraction algebra item set

Skills ofQM 1 QM 2 QM 3 QM 4

Item 1 2 31 1 1 02 1 0 13 1 0 14 1 0 05 1 1 06 1 1 07 1 0 18 1 0 19 1 0 0

10 1 0 011 1 1 0

Desmarais Ingenierie cognitive 30/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Four Q-matricesVariations on Tatsuoka’s fraction algebra item set

Skills ofQM 1 QM 2 QM 3 QM 4

Item 1 2 3 1 2 3 4 5 1 2 3 1 2 31 1 1 0 1 1 1 1 0 0 1 0 1 1 02 1 0 1 1 1 1 1 1 0 0 1 1 0 13 1 0 1 0 0 1 0 0 0 0 1 0 1 04 1 0 0 1 1 1 1 0 1 0 0 1 0 05 1 1 0 1 1 1 1 0 0 1 0 1 0 06 1 1 0 1 1 0 0 0 0 1 0 0 0 17 1 0 1 1 0 1 1 1 0 0 1 1 0 18 1 0 1 1 0 1 0 0 0 0 1 0 1 19 1 0 0 1 0 1 1 0 1 0 0 1 0 0

10 1 0 0 1 1 1 1 0 1 0 0 1 0 111 1 1 0 1 1 1 1 0 0 1 0 1 0 0

Desmarais Ingenierie cognitive 31/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Data driven approaches

• Start withtest data:

R =

items

stud

ents

0

BB@

1 1 1

0 0 1

0 1 0

0 0 0

1

CCA

• Define aQ-matrix:

Q =

skills

item

s 0

@1 1 1

0 0 1

1 0 0

1

A

• Assessskills:

S =

skills

stud

ents

0

BB@

1 1 1

0 0 1

0 1 1

0 0 0

1

CCA

What we expect:R = S�QT

R =

items

stud

ents

0

BB@

1 1 1

0 1 00 1 10 0 0

1

CCA

Desmarais Ingenierie cognitive 32/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Detecting perturbationsSynthetic data

Introduce perturbations in Q-matrix and assess suggestedchanges

2 4 6 8 10

01

23

45

67

True Positives (synth.)

Number of perturbations

Aver

age

frequ

ency

● ●

●● ● ● ●

● ● ● ● ●●

TotalChiu (2013)de la Torre (2008)ALS

2 4 6 8 10

False Positives (synth.)

Number of perturbations

Aver

age

frequ

ency

● ●● ●

● ●● ●

● ●

●●

●●

●●

● ● ● ●

● ●● ●

● ●●

●●

Desmarais Ingenierie cognitive 33/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Detecting perturbationsReal data

Introduce perturbations in Q-matrix and assess suggestedchanges

2 4 6 8 10

01

23

45

67

True Positives (real)

Number of perturbations

Aver

age

frequ

ency

●● ●

●● ● ●

● ● ●

TotalChiu (2013)de la Torre (2008)ALS

2 4 6 8 10

False Positives (real)

Number of perturbations

Aver

age

frequ

ency

●●

● ● ●● ● ● ● ●

● ● ●● ● ● ● ● ● ●

●●

● ●●

● ●● ●

Desmarais Ingenierie cognitive 34/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Can we combine methods?

Desmarais Ingenierie cognitive 35/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Three methods

1 MinRSS: minimizes the residual sum of square (RSS)between the real responses and the ideal responses

2 MaxDiff: maximizes the difference in the probabilities of acorrect response to an item between examinees whopossess all the skills required for a correct response to thatitem and examinees who do not

3 ALS: given a Q-matrix, find skills-matrix that minizes sumof square errors, then alternate to find new Q-matrix, andso on.

Desmarais Ingenierie cognitive 36/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Partition tree

• The combination of methods based on a partition treealgorithm

• Factors retained• Number of skills per row• Number of skills per column• Stickyness: persistance of false positives/negatives for a

given Q-matrix

Desmarais Ingenierie cognitive 37/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Partition tree example

node), split, n, deviance, yval

* denotes terminal node

1) root 43213 10583.7900 0.5712633

2) minrss< 0.5 22733 5146.0780 0.3462807

4) alsc< 0.5 13937 2561.8270 0.2427352 *

5) alsc>=0.5 8796 2198.0590 0.5103456 *

3) minrss>=0.5 20480 3009.7720 0.8209961

6) alsc>=1.5 1359 216.9595 0.1994113 *

7) alsc< 1.5 19121 2230.4200 0.8651744

14) alsc< 0.5 3452 720.6475 0.7030707 *

15) alsc>=0.5 15669 1399.0780 0.9008871 *

Desmarais Ingenierie cognitive 38/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Process

Data generation

Test Synthetic

Test Real

2. Permutated QMs(ground truth)

3. Synthetic test outcome data with DINA model

(400 records)

10. Comparison with ground truth

5. Partition trees(3 types)

provides ground truth

labels for learning trees

Perturbations(one per cell)

4. Refinements with three techniques

9. Refinements with partition trees and

the three techniques

7. Refinements with partition trees and

the three techniques

8. Comparison with original QMi

6. Fraction data set

Perturbations(one per cell)

1. QMi

Permutations(1000)

Key principles:• Training of the partition

tree is done over syntheticdata

• This is how the influence offactors such asstickyness and skills perrow and column areassessed

Desmarais Ingenierie cognitive 39/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Partition tree training data

Prediction Skills per Stickyness

Truth (1)M

inR

SS

(2)M

axD

iff

(3)A

LSC

row col (1)M

inR

SS

(2)M

axD

iff

(3)A

LSC

1 1 na 1 0 5 0.00 0.00 0.091 1 1 2 1 7 0.00 0.00 0.091 1 1 2 1 7 0.00 0.00 0.090 0 1 1 3 7 0.04 0.00 0.030 0 0 1 2 7 0.04 0.00 0.030 0 0 1 2 7 0.04 0.00 0.03

Desmarais Ingenierie cognitive 40/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Test data

• Real data from Tatsuoka: 536 respondants• Q-matrices from different authors

• 20 ⇥ 8• 13 ⇥ 5• 15 ⇥ 3

• Common denominator of 11 items

Desmarais Ingenierie cognitive 41/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Four Q-matricesVariations on Tatsuoka’s fraction algebra item set

Skills ofQM 1 QM 2 QM 3 QM 4

Item 1 2 3 1 2 3 4 5 1 2 3 1 2 31 1 1 0 1 1 1 1 0 0 1 0 1 1 02 1 0 1 1 1 1 1 1 0 0 1 1 0 13 1 0 1 0 0 1 0 0 0 0 1 0 1 04 1 0 0 1 1 1 1 0 1 0 0 1 0 05 1 1 0 1 1 1 1 0 0 1 0 1 0 06 1 1 0 1 1 0 0 0 0 1 0 0 0 17 1 0 1 1 0 1 1 1 0 0 1 1 0 18 1 0 1 1 0 1 0 0 0 0 1 0 1 19 1 0 0 1 0 1 1 0 1 0 0 1 0 0

10 1 0 0 1 1 1 1 0 1 0 0 1 0 111 1 1 0 1 1 1 1 0 0 1 0 1 0 0

Desmarais Ingenierie cognitive 42/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Cases

Perturbation Refinement

Value Value Value Outcomebefore after proposed

Perturbed cell(1) 0 1 0 correct (TP)(2) 1 0 1 correct (TP)(3) 0 1 1 wrong (FN)(4) 1 0 0 wrong (FN)

Non Perturbed cell(5) 0 0 0 correct (TN)*(6) 1 1 1 correct (TN)*(7) 0 0 1 wrong (FP)(8) 1 1 0 wrong (FP)

* ignoredDesmarais Ingenierie cognitive 43/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Measures

• Finding back perturbed cells• 1-cell: TP or FN (recall)• 0-cell: TN or FP (precision)

• Harmonic mean (F-score)

F-score = 2⇥ precision⇥ recall

precision + recall

= 2⇥ Acc¬P ⇥AccP

Acc¬P +AccP

Desmarais Ingenierie cognitive 44/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Results for Synthetic DataF-score

QM Technique Partition tree

MinRSS MaxDiff ALSC (1) (2) (3)

F-score

1 0.88 0.51 0.58 0.88 0.90 0.972 0.13 0.35 0.42 0.68 0.69 0.903 0.96 0.34 0.83 0.97 0.97 1.004 0.93 0.52 0.58 0.93 0.94 0.98

X 0.72 0.43 0.60 0.87 0.87 0.96

0.72 ! 0.96 = 86% error reduction

Desmarais Ingenierie cognitive 45/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Results on Real DataF-score

QM Technique Partition tree

MinRSS MaxDiff ALSC (1) (2) (3)

F-score

1 0.42 0.27 0.54 0.42 0.37 0.632 0.50 0.17 0.37 0.73 0.74 0.773 0.38 0.16 0.39 0.64 0.86 0.834 0.48 0.20 0.42 0.48 0.50 0.56

X 0.41 0.23 0.38 0.57 0.62 0.70

0.41 ! 0.70 = 55% error reduction

Desmarais Ingenierie cognitive 46/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Late breaking results!

Boosting brings another:⇡ 10% improvement

Principles of boosting:• compute each record (observation) residual error (fit)• assign weights to records according to residual error• resample with weights as a probability or as factors to

re-estimate model parameters

Desmarais Ingenierie cognitive 47/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Conclusion

• Major improvements obtained• 86% over synthetic data• 55% over real data• better reliability (systematically better than the best

method, while no method is systematically the best)• Some limits

• Single set of 11 questions• Static data

Desmarais Ingenierie cognitive 48/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Boosting: Results on Real DataF-score (unvalidated yet)

QM Technique Boosting

MinRSS MaxDiff ALSC (1) (2) (3)

F-score

1 0.42 0.27 0.54 0.65 0.72 0.982 0.50 0.17 0.37 0.60 0.81 0.883 0.23 0.27 0.18 0.64 0.82 0.984 0.48 0.20 0.42 0.55 0.72 0.99

X 0.41 0.23 0.38 0.61 0.77 0.96

0.41 ! 0.96 = 90% error reduction

Desmarais Ingenierie cognitive 49/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Ingenierie cognitive

Structures de connaissances

Arrimage des items aux competences latentes

La representativite d’un modele

Conclusion

Desmarais Ingenierie cognitive 50/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Comment determiner qu’un modele est representatif desphenomenes derriere les donnees?

Representativite () Meilleure performance ?

Desmarais Ingenierie cognitive 51/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

How do we know a model fits the data?

Standard answer:• Pick the model with the highest predictive performance• Use person or item fit measures (given the ground truth)

Desmarais Ingenierie cognitive 52/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

How do we know a model fits the data?

Standard answer:• Pick the model with the highest predictive performance• Use person or item fit measures (given the ground truth)

Alternative answer:• Use performance signatures• Use parameter signatures (Pardos et al.)

Desmarais Ingenierie cognitive 52/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Parameter signature (Rosenberg-Kima and Pardos)

Key idea: draw a likelihood map of the parameters giventhe data and compare

Desmarais Ingenierie cognitive 53/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Performance signatures

Key idea:find the closest model in the performance spaceAssumptions:

• performance space is stable across conditions of• sample size and characteristic• parameter space

Desmarais Ingenierie cognitive 54/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Performance of models across models’ data

Prediction technique

Perc

ent a

ccur

acy

diffe

renc

e fro

m E

xpec

ted

valu

e

−30

−20

−10

0

10

20

30

Expec

tedPOKS IRT

NMF.con Dina

NMF.add Dino

DINA

Expec

tedPOKS IRT

NMF.con Dina

NMF.add Dino

DINO

Expec

tedPOKS IRT

NMF.con Dina

NMF.add Dino

IRT.Rasch

Expec

tedPOKS IRT

NMF.con Dina

NMF.add Dino

NMF.Add

1 2 3 4 5 6 7

NMF.Con

1 2 3 4 5 6 7

POKS

−30

−20

−10

0

10

20

30

1 2 3 4 5 6 7

Random

Each blockrepresentsa syntheticgenerateddataset

Desmarais Ingenierie cognitive 55/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Performance of models across models’ data

Prediction technique

Perc

ent a

ccur

acy

diffe

renc

e fro

m E

xpec

ted

valu

e

−10

−5

0

5

10

Expec

tedPOKS IRT

NMF.con Dina

NMF.add Dino

ECPE

Expec

tedPOKS IRT

NMF.con Dina

NMF.add Dino

Fraction

Expec

tedPOKS IRT

NMF.con Dina

NMF.add Dino

Fraction1

Expec

tedPOKS IRT

NMF.con Dina

NMF.add Dino

Fraction2.1

Fraction2.2 Fraction2.3

−10

−5

0

5

10

Vomlel Each blockrepresentsa Realdataset

Desmarais Ingenierie cognitive 56/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Real vs. synthetic comparison

Prediction technique

Perc

ent accura

cy d

iffe

rence fro

m E

xpecte

d v

alu

e

−30

−20

−10

0

10

20

30

Expecte

d

POKS IR

T

NMF.con

Dina

NMF.a

ddDino

DINA

Expecte

d

POKS IR

T

NMF.con

Dina

NMF.a

ddDino

DINO

Expecte

d

POKS IR

T

NMF.con

Dina

NMF.a

ddDino

IRT.Rasch

Expecte

d

POKS IR

T

NMF.con

Dina

NMF.a

ddDino

NMF.Add

1 2 3 4 5 6 7

NMF.Con

1 2 3 4 5 6 7

POKS

−30

−20

−10

0

10

20

30

1 2 3 4 5 6 7

Random

Each blockrepresentsa syntheticgenerateddataset

Prediction technique

Perc

ent accura

cy d

iffe

rence fro

m E

xpecte

d v

alu

e

−10

−5

0

5

10

Expecte

d

POKS IR

T

NMF.con

Dina

NMF.a

ddDino

ECPE

Expecte

d

POKS IR

T

NMF.con

Dina

NMF.a

ddDino

Fraction

Expecte

d

POKS IR

T

NMF.con

Dina

NMF.a

ddDino

Fraction1

Expecte

d

POKS IR

T

NMF.con

Dina

NMF.a

ddDino

Fraction2.1

Fraction2.2 Fraction2.3

−10

−5

0

5

10

Vomlel Each blockrepresentsa Realdataset

Desmarais Ingenierie cognitive 57/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Ingenierie cognitive

Structures de connaissances

Arrimage des items aux competences latentes

La representativite d’un modele

Conclusion

Desmarais Ingenierie cognitive 58/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Effervescence des approches numeriques etstatistiques pour l’ingenierie cognitive

• Abondance de donnees educationnelles• Affluence de techniques et d’outils pour la simulation et

l’analyse des donnees• Emergence d’un paradigme numerique et statistique a

l’ingenierie cognitive

Desmarais Ingenierie cognitive 59/60

Ingenierie cognitive Structures de connaissances Arrimage Vrai/Meilleur Conclusion

Questions?

Desmarais Ingenierie cognitive 60/60