How much data is enough? – Generating reliable policies w/MDP’s

Joel TetreaultUniversity of PittsburghLRDCJuly 14, 2006

Problem Problems with designing spoken dialogue systems:

How to handle noisy data or miscommunications? Hand-tailoring policies for complex dialogues? What features to use?

Previous work used machine learning to improve the dialogue manager of spoken dialogue systems [Singh et al., ‘02; Walker, ‘00; Henderson et al., ‘05]

However, very little empirical work [Paek et al., ‘05;

Frampton ‘05] on comparing the utility of adding specialized features to construct a better dialogue state

Goal

How does one choose which features best contribute to a better model of dialogue state?

Goal: show the comparative utility of adding three different features to a dialogue state

4 features: concept repetition, frustration, student performance, student moves

All are important to tutoring systems, but also are important to dialogue systems in general

Previous Work

In complex domains, annotation and testing is time-consuming so it is important to properly choose best features beforehand

Developed a methodology for using Reinforcement Learning to determine whether adding complex features to a dialogue state will beneficially alter policies [Tetreault & Litman, EACL ’06]

Extensions: Methodology to determine which features are the best Also show our results generalize over different action

choices (feedback vs. questions)

Outline

Markov Decision Processes (MDP) MDP Instantiation Experimental Method Results

Policies Feature Comparison

Markov Decision Processes

What is the best action an agent should take at any state to maximize reward at the end?

MDP Input: States Actions Reward Function

MDP Output

Policy: optimal action for system to take in each state

Calculated using policy iteration which depends on: Propagating final reward to each state the probabilities of getting from one state to the next

given a certain action Additional output: V-value: the worth of each

state

MDP’s in Spoken Dialogue

MDP

DialogueSystem

Training dataPolicy

User Simulator

HumanUser

MDP works offline

Interactions work online

ITSPOKE Corpus

100 dialogues with ITSPOKE spoken dialogue tutoring system [Litman et al. ’04]

All possible dialogue paths were authored by physics experts

Dialogues informally follow question-answer format

60 turns per dialogue on average Each student session has 5 dialogues

bookended by a pretest and posttest to calculate how much student learned

Corpus Annotations

Manual annotations: Tutor Moves (similar to Dialog Acts)

[Forbes-Riley et al., ’05]

Student Frustration and Certainty [Litman et al. ’04] [Liscombe et al. ’05]

Automated annotations: Correctness (based on student’s response to last question) Concept Repetition (whether a concept is repeated) %Correctness (past performance)

MDP State Features

Features Values

Correctness Correct (C), Incorrect (I)

Certainty Certain (cer), Neutral (neu), Uncertain (unc)

Concept Repetition New Concept (0), Repeated (R)

Frustration Frustrated (F) , Neutral (N)

% Correctness 50-100% (H)igh, 0-49% (L)ow

MDP Action Choices

Action Example Turn

SAQ (Short Answer Question)

“What is the direction of that force relative to your fist?”

CAQ (Complex Answer Question)

“What is the definition of Newton’s Second Law?”

Mix “If it doesn’t hit the center of the pool what do you know about the magnitude of its displacement from the center of the pool when it lands? Can it be zero? Can it be nonzero?

NoQ “So you can compare it to my response…”

MDP Reward Function

Reward Function: use normalized learning gain to do a median split on corpus:

10 students are “high learners” and the other 10 are “low learners”

High learner dialogues had a final state with a reward of +100, low learners had one of -100

)1(

)(

pretest

pretestposttestNLG

Methodology

Construct MDP’s to test the inclusion of new state features to a baseline: Develop baseline state and policy Add a feature to baseline and compare polices A feature is deemed important if adding it results in a

change in policy from a baseline policy given 3 metrics: # of Policy Differences (Diff’s) %Policy Change (%PC) Expected Cumulative Reward (ECR)

For each MDP: verify policies are reliable (V-value convergence)

Hypothetical Policy Change Example

B1 State Policy B1+Certainty

State

1 [C] CAQ [C,Cer]

[C,Neu]

[C,Unc]

2 [I] SAQ [I,Cer]

[I,Neu]

[I,Unc]

+Cert 1

Policy CAQ

CAQ

CAQ

SAQ

SAQ

SAQ

+Cert 2

Policy Mix

CAQ

Mix

Mix

CAQ

Mix

0 Diffs 5 Diffs

Tests

+%Correct

+Concept

+Frustration

B2+

Correctness +Certainty

Baseline 1 Baseline 2

B1+

Baseline

Actions: {SAQ, CAQ, Mix, NoQ} Baseline State: {Correctness}

Baseline network

[C]

FINAL

[I]

SAQ|CAQ|Mix|NoQ

Baseline 1 Policies

Trend: if you only have student correctness as a model of student state, give a hint or other state act to the student, otherwise give a Mix of complex and short answer questions

# State State Size Policy

1 [C] 1308 NoQ

2 [I] 872 Mix

But are our policies reliable?

Best way to test is to run real experiments with human users with new dialogue manager, but that is months of work

Our tact: check if our corpus is large enough to develop reliable policies by seeing if V-values converge as we add more data to corpus

Method: run MDP on subsets of our corpus (incrementally add a student (5 dialogues) to data, and rerun MDP on each subset)

Baseline Convergence Plot

Methodology: Adding more Features Create more complicated baseline by adding

certainty feature (new baseline = B2) Add other 4 features (concept repetition, frustration,

performance, student move) individually to new baseline

Check V-value and policy convergence Analyze policy changes Use Feature Comparison Metrics to determine the

relative utility of the three features

Tests

+%Correct

+Concept

+Frustration

B2+

Correctness +Certainty

Baseline 1 Baseline 2

B1+

Certainty

Previous work (Bhatt et al., ’04) has shown the importance of certainty in ITS

A student who is certain and correct, may require a harder question since he or she is doing well, but one that is correct but showing some doubt is a sign they are becoming confused, give an easier question

B2: Baseline + Certainty Policies

B1 State Policy B1+Certainty

State

+Certainty Policy

1 [C] NoQ [C,Cer]

[C,Neu]

[C,Unc]

Mix

SAQ

Mix

2 [I] Mix [I,Cer]

[I,Neu]

[I,Unc]

Mix

NoQ

Mix

Trend: if neutral, give SAQ or NoQ, else give Mix

Baseline 2 Convergence Plots

Baseline 2 Diff Plots

Diff: For each subset corpus, compare policywith policy generated with full corpus

Tests

+%Correct

+Concept

+Frustration

B2+

Correctness +Certainty

Baseline 1 Baseline 2

B1+

Feature Comparison (3 metrics) # Diff’s

Number of new states whose policies differ from the original

Insensitive to how frequently a state occurs % Policy Change (%P.C.)

Take into account the frequency of each state-action sequence

Occurences State # Total

State Diff of Occurences% PC

Feature Comparison

Expected Cumulative Reward (E.C.R.) One issue with %P.C. is that frequently occurring states

have low V-values and thus may bias the score Use the expected value of being at the start of the

dialogue to compare features ECR = average V-value of all start states

Feature Comparison ResultsState Feature #Diff’s %P.C. E.C.R

Student Move 10 82.2% 43.21

Concept Repetition 10 80.2% 39.52

Frustration 8 66.4% 31.30

Percent Correctness 4 44.3% 28.47

Trend of SMove > Concept Repetition > Frustration > Percent Correctness stays the same over all three metrics

Baseline: Also tested the effects of a binary random feature If enough data, a random feature should not alter policies Average diff of 5.1

How reliable are policies?

Frustration

Concept

Possible data size is small and with increased data we may see more fluctuations

Confidence Bounds

Hypothesis: instead of looking at the V-values and policy differences directly, look at the confidence bounds of each V-value

As data increases, confidence of V-value should shrink to reflect a better model of the world

Additionally, the policies should converge as well

Confidence Bounds

CB’s can also be used to distinguish how much better an additional state feature is over a baseline state space

That is, if the lower bound of a new state space is greater than the upper bound of the baseline state space

Crossover Example

Data

ECR

More complicated Model

Baseline

Confidence Bounds: App #2

Automatic model switching If you know a model, at it’s worst (ie. It’s lower

bound is better than another model’s upper bound) then you can automatically switch to the more complicated model

Good for online RL applications

Confidence Bound Methodology For each data slice, calculate upper and lower

bounds on the V-value Take transition matrix for slice and sample from each row

using direch. statistical formula 1000 times do this b/c real world data is not exactly approximating what

data is like in the real world, but may be close So get 1000 new transition matrices that are all very similar

Run MDP on all 1000 transition matrices to get a range of ECR’s Rows with not a lot of data are very volatile so expect large

range of ECR’s, but as data increases, transition matrices should stabilize such that most of the new matrices produce similar policies and values as the original

Take upper and lower bounds at 2.5% percentile

Experiment

Original action/state setup did not show anything promising State/action space too large for data? Not best MDP instantiation

Looked at a variety of MDP configurations Refined reward metric Adding discourse segmentation

+essay Instantiation with ’03+’05 data

+essay Baseline1

+essay Baseline2

+essay B2+SMove

Feature Comparison ResultsState Feature #Diff’s %P.C. E.C.R

Student Move 5 43.4% 49.17

Concept Repetition 3 25.5% 42.56

Frustration 1 0.03% 32.99

Percent Correctness 3 11.19% 28.50

Reduced state size: Certainty = {Cert+Neutral, Uncert} Trend that SMove and Concept Repetition are the best features B2 ECR = 31.92

Baseline 1

Upper = 23.65 Lower = 0.24

Baseline 2

Upper = 57.16 Lower = 39.62

B2+ Concept Repetition

Upper = 64.30 Lower =49.16

B2+Percent Correctness

Upper =48.42 Lower = 32.86

B2+Student Move

Upper = 61.36 Lower = 39.94

Discussion

Baseline 2 – has crossover effect and policy stability

More complex features (B2 + X) – have crossover effect, but not sure if polices are stable (some stabilize at 17 students)

Indicates that 100 dialogues isn’t enough for even this simple MDP? (but is enough for baseline 2 to feel confident about?)