bayesian nets in student modeling its- sept 30, 2004

Bayesian Nets in Student ModelingBayesian Nets in Student Modeling

ITS- Sept 30, 2004ITS- Sept 30, 2004

Sources of UncertaintySources of Uncertainty• Incomplete and/or incorrect

knowledge• Slips and/or guesses• Multiple derivations• Invisible inferences• Not showing all work• Help messages• Self-explaining ahead

• Incomplete and/or incorrect knowledge

• Slips and/or guesses• Multiple derivations• Invisible inferences• Not showing all work• Help messages• Self-explaining ahead

Andes student modelAndes student model

• Knowledge tracing• Plan recognition• 1st to use student’s domain

knowledge

• Action prediction• Andes first to support all three

• Knowledge tracing• Plan recognition• 1st to use student’s domain

knowledge

• Action prediction• Andes first to support all three

Goals of AndesGoals of Andes• Students work as much as possible

alone• React to student’s incorrect action,

signal error, explain• React to student’s impasse, provide

procedural help• Assure student understands examples,

prompt self-explaining

• Students work as much as possible alone

• React to student’s incorrect action, signal error, explain

• React to student’s impasse, provide procedural help

• Assure student understands examples, prompt self-explaining

Types of helpTypes of help

• Error help• Procedural help (ask for hints)• Unsolicited help (for non-

physics errors)• Different levels of hints ‘til

“bottom-out hint”

• Error help• Procedural help (ask for hints)• Unsolicited help (for non-

physics errors)• Different levels of hints ‘til

“bottom-out hint”

Usage of student modelUsage of student model

• Plan recognition: recognize and support goals (requires prediction)

• Asses knowledge: help presentation (reminder v. minilesson)

• Assess mastery level: prompt self-explanation or not

• Plan recognition: recognize and support goals (requires prediction)

• Asses knowledge: help presentation (reminder v. minilesson)

• Assess mastery level: prompt self-explanation or not

Self-Explaining CoachSelf-Explaining Coach

• Step correctness (domain)• Rule Browser• E.g.: using force or acceleration

• Step utility (role in solution plan)• Plan Browser• Recognize goals

• Step correctness (domain)• Rule Browser• E.g.: using force or acceleration

• Step utility (role in solution plan)• Plan Browser• Recognize goals

Bayesian networkBayesian network

• Solution graph: map of all solutions with no variables (propositional)

• Solution graph: map of all solutions with no variables (propositional)

Types of nodesTypes of nodes• Domain-general: rules• 2 values indicating mastery

• Task-specific: • facts, goals, rule apps, strategy

nodes• Doable (done already or knows all

needed) • Not-doable

• Domain-general: rules• 2 values indicating mastery

• Task-specific: • facts, goals, rule apps, strategy

nodes• Doable (done already or knows all

needed) • Not-doable

Knowledge evolutionKnowledge evolution

• Dynamic Bayesian network• Analyze each exercise alone• Roll-up: prior probabilities set to

marginal probabilities for previous• Improvements: could model

dependencies & knowledge decay

• Dynamic Bayesian network• Analyze each exercise alone• Roll-up: prior probabilities set to

marginal probabilities for previous• Improvements: could model

dependencies & knowledge decay

Intention or ability?Intention or ability?

• Probability that student can and IS implementing a certain goal

• Decision-theoretic tutor keeps probabilites of “focus of attention”

• Probability that student can and IS implementing a certain goal

• Decision-theoretic tutor keeps probabilites of “focus of attention”

Problem creationProblem creation

• Givens• Goals• Problem-solver applies rules,

generating subgoals until done• Solution graph created

• Givens• Goals• Problem-solver applies rules,

generating subgoals until done• Solution graph created

Andes assessorAndes assessor

• Dynamic belief network for domain-general nodes

• Rules - priors set by test scores• Context-Rules• P(CR=true|R=true)=1• P(CR=true|R=false)=difficulty• One context changes, adjust rest

• Dynamic belief network for domain-general nodes

• Rules - priors set by test scores• Context-Rules• P(CR=true|R=true)=1• P(CR=true|R=false)=difficulty• One context changes, adjust rest

Task-specific nodesTask-specific nodes

• Fact, goal, rule application, strategy

• Context-Rule nodes link task-specific to domain-general rules

• Fact, goal, rule application, strategy

• Context-Rule nodes link task-specific to domain-general rules

Fact & Goal NodesFact & Goal Nodes

• A.k.a. Propositional Nodes• 1 parent for each way to derive• Leaky-OR: T if 1 parent T, also

sometimes true if not• Reasons: guessing, analogy, etc

• A.k.a. Propositional Nodes• 1 parent for each way to derive• Leaky-OR: T if 1 parent T, also

sometimes true if not• Reasons: guessing, analogy, etc

Rule-Application NodesRule-Application Nodes

• Connect CR,Strategy & Prop nodes to new derived Prop nodes

• Doable or not-doable• Parents: 1 CR, pre-condition Props,

sometimes one Strategy node• Noisy-AND: T if ALL parents T, but

sometimes not, 1-alpha

• Connect CR,Strategy & Prop nodes to new derived Prop nodes

• Doable or not-doable• Parents: 1 CR, pre-condition Props,

sometimes one Strategy node• Noisy-AND: T if ALL parents T, but

sometimes not, 1-alpha

Strategy NodesStrategy Nodes

• Used when >1 way to reach a Goal• Paired with a Goal Node• Values are mutually exclusive• No parents in network• Priors=freq. students use this strat.

• Used when >1 way to reach a Goal• Paired with a Goal Node• Values are mutually exclusive• No parents in network• Priors=freq. students use this strat.

Compare FiguresCompare Figures

• Figure 9 before observing A-is-body

• Figure 10 after observing A-is-body

• Figure 9 before observing A-is-body

• Figure 10 after observing A-is-body

HintsHints

• Add a new parent to a Prop node

• Accounts for guessing

• Add a new parent to a Prop node

• Accounts for guessing

SE-CoachSE-Coach• Adds nodes for Read• Link these to Prop nodes• Longer read time, higher prob knows

Prop (p 26)

• Adds nodes for plan selection• Link these to Context-Rules

• Rule Application node prob T if knows CR & all preconditions=Noisy-AND

• Adds nodes for Read• Link these to Prop nodes• Longer read time, higher prob knows

Prop (p 26)

• Adds nodes for plan selection• Link these to Context-Rules

• Rule Application node prob T if knows CR & all preconditions=Noisy-AND

EvaluationEvaluation

• Simulated students, 65% correct for rule mastery

• 95% if no “invis inferences” and has to “show all work”

• Post-test shows significant learning• Voluntary acceptance?• Accuracy of plan recognition

• Simulated students, 65% correct for rule mastery

• 95% if no “invis inferences” and has to “show all work”

• Post-test shows significant learning• Voluntary acceptance?• Accuracy of plan recognition