Learning Individual Behavior in an Educational Game:A Data-Driven Approach

Seong Jae Lee, Yun-En Liu, and Zoran PopovićCerter for Game Science, Computer Science and Engineering

University of Washington{seongjae, yunliu, zoran}@cs.washington.edu

ABSTRACTIn recent years, open-ended interactive educational toolssuch as games have been gained popularity due to their abil-ity to make learning more enjoyable and engaging. Model-ing and predicting individual behavior in such interactiveenvironments is crucial to better understand the learningprocess and improve the tools in the future. A model-basedapproach is a standard way to learn student behavior inhighly-structured systems such as intelligent tutors. How-ever, defining such a model relies on expert domain knowl-edge. The same approach is often extremely difficult in edu-cational games because open-ended nature of these systemscreates an enormous space of actions. To ease this burden,we propose a data-driven approach to learn individual be-havior given a user’s interaction history. This model doesnot heavily rely on expert domain knowledge. We use ourframework to predict player movements in two educationalpuzzle games, demonstrating that our behavior model per-forms significantly better than a baseline on both games.This indicates that our framework can generalize withoutrequiring extensive expert knowledge specific to each do-main. Finally, we show that the learned model can give newinsights into understanding player behavior.

KeywordsEducational Games, Data-driven Learning, Supervised Learn-ing, Logistic Regression, Learning from Demonstration

1. INTRODUCTIONOpen-ended educational environments, especially educationalgames and game-based learning, have been gaining popular-ity as more and more evidence suggests that they can helpenhance student learning while making the process enjoyable[21, 19]. The interactive nature of this media enables us totrack user inputs and provide an immediate personalized re-sponse to make learning more effective and efficient. There-fore, modeling and predicting user behavior lies at the heartof improving engagement and mastery for every learner. For

example, we can detect if a student is struggling with acertain concept by simulating the learned student behav-ior model on a related task; if the student model performspoorly, we can then give more tasks related to that con-cept. The ability to further predict which error the studentis going to make can be used to infer how the learner misun-derstood related concepts and provide a targeted instructionfocusing on that specific misconception.

There has been active research on learning individual behav-ior in both the education and game community. Most of thisresearch focuses on inferring a meaningful latent structure,such as knowledge or skill, often simplifying the behaviorspace into a small number of parameters. Hence the choiceof a model significantly affects the quality of the learnedbehavior, forcing researchers to spend time on experiment-ing with different models and refining various features [6,20]. Moreover, unlike highly-structured systems such as in-telligent tutors, it is difficult to define a behavioral modeldescribing movements in an educational game. This is espe-cially true when the player is given a large number of avail-able moves, resulting in a large scale multi-class predictionproblem. For example, predicting the exact moves someonewill make while solving a puzzle is more challenging thanpredicting whether a student will solve a problem correctly.

Therefore, a purely data-driven approach is both a suit-able and preferable alternative for learning user behavior inhighly open-ended environment, such as educational games.It needs less expert authoring, and it can capture variouslow-level user movements—mistakes and errors, explorationhabits, and adapting a strategy while playing—as they areeven without a specific model describing such moves. Also,we can further analyze learned policies to give more inter-pretable insights into user behavior. For example, we couldanalyze erroneous movements to figure out their misconcep-tions. This knowledge can be used to construct more so-phisticated cognitive models that will give us a deeper andmore accurate understanding of user behavior.

In this paper, we propose a data-driven framework thatlearns individual movements in a sequential decision-makingprocess. It uses a supervised classification method to pre-dict the next movement of a user based on past gameplaydata. For each game state, our framework transforms userplay logs into a high-dimensional feature vector, and learnsa classifier that predicts the next movement based on thisfeature vector. To construct this set of features, we start

from a massive set of default features defined in a domain-independent way over a state-action graph, and then pick asmall set of relevant ones using a univariate feature selectiontechnique. We apply this framework to two very different ed-ucational games DragonBox Adaptive and Refraction, andevaluate the learned behavior by predicting the actions ofheld-out users.

Our contribution is threefold. First, we propose a data-driven individual behavior learning framework, which doesnot rely on heavy domain-dependent expert authoring. Sec-ond, we apply our framework on two different games anddemonstrate our framework improves the prediction qualitysubstantially over a previously proposed algorithm. Finally,combined with a robust feature selection, we show our frame-work learns an efficient yet powerful set of features, whichfurther gives new insight into understanding player behaviorin the game.

2. RELATED WORKLearning user behavior has been actively researched in theeducation and game community. One of the most widelyused models is the Bayesian network model. Knowledgetracing (KT) [9] and its variations [17, 27] are probably themost widely used student models in the field of educationaldata mining. This model estimates the user’s knowledgeas latent variables, or knowledge components (KCs). TheseKCs represent student mastery over each concept and pre-dict whether a student will be able to solve a task. Thereare also other approaches using Bayesian networks, such aspredicting whether a student can solve a problem correctlywithout requesting help in an interactive learning environ-ment [16], predicting a user’s next action and goal in a multi-user dungeon adventure game [5], or predicting a build treein a real-time strategy game [22]. Nevertheless, these ap-proaches usually learn the knowledge of users directly fromcarefully designed network structures, which often need ex-pert authoring to define a task-specific and system-specificnetwork model.

Another widely used approach to learn individual behavioris using factor analysis. Item response theory (IRT) mod-els have been extensively used in psychometrics and edu-cational testing domains [11, 8]. They represent an individ-ual’s score as a function of two latent factors: individual skilland item difficulty. The learned factors can be used to pre-dict user performance on different items. Adapting matrixfactorization techniques from recommender systems is alsopopular. Thai-Nghe et al. predicted personalized studentperformance in algebra problems by modeling user-item in-teractions as inner products of user and item feature vectors[24]. This model can even incorporate temporal behavior byincluding time factors, both in the educational community[25] and in the game community [28]. Nevertheless, such ap-proaches do not easily fit our goal, which requires predictinga fine granularity of action instead of predicting user’s singlevalued performance.

Another line of research on learning a low-level user policy isoptimizing a hidden reward or heuristic function from usertrajectories in a state-action graph. Tastan and Sukthankarbuilt a human-like opponent from experts’ demonstrationsin a first-person shooter game using inverse reinforcement

learning in a Markov decision process context [23]. Jansenet al. learned a personalized chess strategy from individ-ual play records by learning the weights of multiple heuris-tics on finite depth search models [12]. Similarly, Liu etal. learned player moves in an educational puzzle game bylearning the weights of heuristics in a one-depth probabilisticsearch model [15]. However, such methods usually define auser reward as a combination of pre-defined heuristics. Thequality of the learned policy is strongly dependent on theseheuristics, which are often system-specific and need a lotof time to refine. Furthermore, these models assume thatplayers do not change over time. Describing how peoplelearn, which is frequently observed in interactive environ-ments, would be another challenge to trying this approachin our domain.

Unlike the research listed above, our framework focuses ona data-driven approach with less system-specific authoring.Our work is a partial extension of that of Liu et al. [15],which is a mixture of a one-depth heuristic search modeland a data-driven Markov model with no knowledge of in-dividual history other than the current game state. We fo-cus on the latter data-driven approach and build upon thatmodel. Unlike their work, which makes the same predictionfor all players in a state, we build a personalized policy thatconsiders the full history of the user.

3. PROBLEM DEFINITIONOur framework works on a model that consists of a finiteset of states S; a finite set of actions A, representing thedifferent ways a user can interact with the game; and rulesof transitioning between states based on an action. Thispaper considers domains with determinstic transitions (f :S ×A→ S), but this approach could also apply to domainswith stochastic transitions (f : S×A→ Pr(S)). The demon-stration of a user u, or a trajectory τu, is defined as a se-quence of state-action pairs: τu = {(s0u, a0u), · · · , (stuu , atuu )}.We will note As ⊆ A as a set of valid actions on the givenstate, T as a set of trajectories, V and U as the set of usersin the training set and test set, respectively.

Defining such a model is often intuitive in games, becauseactions and transition rules are already defined in game me-chanics. For example, in blackjack, the configurations ofthe visible cards can be states, and available player deci-sions such as hit or stand will be actions. The transitionfunction here will be stochastic, because we do not knowwhich card will appear next. Defining a good state spaceremains an open problem. A good, compact state repre-sentation will capture the information most relevent to userbehavior; this helps greatly reduce the size of the requiredtraining data. For the example above, using the sum of cardscores for each side would be a better state representationthan the individual cards on the table. Note that a staterefers to the observable state of the game, not the state ofthe player. In many games, players base their decisions notjust on what they currently see in front of them, but alsoon past experience. This non-Markovian property of humanplayers motivates our framework, which leverages a user’spast behavior to improve prediction quality.

Our objective is to learn a stochastic policy π : S × T →Pr(As) describing what action a user will take on a certain

Training Data TV(User Trajectories)

User Trajectoryτ

Feature Generationand Selection

Feature Vectorxs ∈ R|Fs|

LogisticRegression

Policyπ(s, τ) ∈ Pr(As)

Features Fs

Classifier cs

Training Stage Prediction Stage

Figure 1: An overview of our framework. For eachstate, we learn a set of features and a classifier in thetraining stage. With these, a trajectory is convertedinto a feature vector, and further into a stochasticpolicy in the prediction stage.

state based on his trajectory. We use a supervised learningapproach, which attempts to predict the next action given adataset of thousands of user trajectories in the training setTV = {τv|v ∈ V }.

4. EVALUATIONWe evaluate the learned policy on every movement of eachuser trajectories in the test set TU = {τu|u ∈ U}. We usetwo evaluation metrics: log-likelihood and accuracy rate.

The log-likelihood is defined as∑u∈U

∑t∈[0,tu]

log(π̃(atu|stu, τ tu)),

while π̃ is the learned policy given a trajectory observed sofar τ tu = {(s0u, a0u), · · · , (st−1u , at−1u )}. Since the log functionis undefined for the case π(a|s, τ) = 0, we smooth the policyby �: π(a|s, τ) = (1 − �)π(a|s, τ) + �/|As|, while � is setto 0.001 in our experiments unless otherwise specified. Thelog-likelihood is always smaller than or equal to zero, with0 indicating perfect prediction.

With our stochastic policy, the accuracy rate is defined as∑u∈U

∑t∈[0,tu]

π̃(atu|stu, τ tu)∑u∈U

∑t∈[0,tu]

1.

We want to note that even though the accuracy rate is in-tuitive and widely used for measuring the performance of aclassifier, using it as a single evaluation metric can be mis-leading, especially when outputs are highly skewed [13]. Forexample, a degenerate constant classifier that predicts onlythe dominant class may produce a high accuracy rate. Nev-ertheless, in all of our experiments, the ordering of perfor-mances in log-likelihoods is preserved with that in accuracyrates.

5. ALGORITHMFigure 1 provides an overview of our data-driven framework.In the training stage, our framework learns a set of features

and a classifier from training data. A simple way to dothis might be to train a global classifier that takes a playertrajectory and predicts an action. However, we suspect thatthe current state is the most important feature; in fact, theset of available actions As differs per state. Thus we traina separate classifier for each state, reducing the amount ofdata in the training set but increasing the relevancy. In theprediction stage, we take a trajectory and convert it into afeature vector using the learned features. Then we convertthe feature vector into a policy from that state, using thelearned classifier. Here, a feature is a function that takes atrajectory and returns a value f : T → R; a set of featuresconverts a trajectory into a multi-dimensional feature vectorF : T → R|F |.

Algorithm 1 Training Stage

Require: a state s, training data TV1: Fs ← a default set of features defined on s2: Ts, ys ← a list of τ t−1v and atv such that s = stv ∀v, t3: Xs ← Fs(Ts)4: Fs ← SelectFeatures(Fs, Xs, ys)5: if Fs = ∅ then6: cs ← LearnMarkovClassifier(ys)7: else8: Xs ← Fs(Ts)9: cs ← LearnClassifier(Xs, ys)

10: end if11: return Fs, cs

Algorithm 1 describes a detailed process in the trainingstage. The SelectFeatures function takes a set of fea-tures, a set of feature vectors, and a set of performed actionsas inputs and filters irrelevant features out. The Learn-MarkovClassifier function takes a set of performed ac-tions and returns a static classifier. The LearnClassifierfunction takes a set of feature vectors and performed ac-tions as inputs and returns a classifier. We will explain eachfunction in detail.

The training stage starts from a default set of features de-fined on a state-action graph. We use three kinds of binaryfeatures:

• whether the user has visited a certain state s: 1[s ∈ τ ],• whether the trajectory contains a certain state-action

pair (s, a): 1[(s, a) ∈ τ ], and• whether the dth recent move is a certain state-action

pair (s, a): 1[(s, a) = (s|τ |−d, a|τ |−d) ∈ τ ].

The maximum number of d is set to 10 in our experiments.The features with sparsely-visited states and transitions arenot counted. These features summarize which states andactions have been visited by the user, both in the full andrecent history. We built a feature package defined on anabstract state-action graph so that it can be used generallyin multiple systems without extra authoring.

The default set of features contains a huge amount of ir-relevant features for the task, making our learning sufferfrom overfitting as well as prolonged training time. To rem-edy this, we apply a feature selection method using train-ing data. For the SelectFeatures function, we use a chi-squared statistic for each feature on Tv and select features

State s Action a Next State s′ = T (s, a) π(a|s, τ)

lv.3.4.1.x+a=b

Subtract a on both sides with merging lv3.4.1.x+0=b-a 0.4Subtract a on both sides lv3.4.1.x+a-a=b-a 0.4Add a on both sides lv3.4.1.x+a+a=b+a 0.2

Table 1: Examples of states, actions, transitions, and policies in DragonBox Adaptive

with p-value lower than 0.001. We used a univariate featurefiltering approach because it is relatively fast compared toother feature selection techniques, such as L1-regularization.Also, we used a chi-squared test because it is one of the mosteffective methods of feature selection for classification [26].

Finally, we learn a classifier with the selected features. If anyfeatures are selected, we learn a supervised learning classifierusing the LearnClassifier. We used a multi-class logisticregressor as a classifier, because it gives a natural proba-bilistic interpretation unlike decision trees or support vectormachines. If no features are selected, we use a predictorbuilt on a Markov model from Liu et al. [15], which learnsthe observed frequency of state-action pairs in training data:

π̃(a|s) =

∑v∈V

∑t∈[0,tv ]

1[(s, a) = (stv, atv)]∑

v∈V

∑t∈[0,tv ]

1[s = stv].

When there are zero samples, i.e., when the denominatoris zero, this equation is undefined and it returns a uniformdistribution instead. For convenience, we will call this pre-dictor the Markov predictor.

We can also interpret the Markov predictor as another lo-gistic regressor with no features but only with an interceptfor each action. With no regularization or features, logisticregression optimizes the log-likelihood on training data witha policy only dependant on the current state, whose optimalsolution is the observed frequency for each class. This isexactly what the Markov predictor does.

In the prediction stage, we take a trajectory and a state asan input, and first check the type of the learned classifier onthe given state. If the classifier is the Markov predictor, itdoes not need a feature vector and returns a policy only de-pendent on the current state. Otherwise, we use the learnedfeatures Fs and classifier cs to convert a user’s trajectory τinto a feature vector x, and then into a state-wise stochasticpolicy π(s, τ).

6. EXPERIMENT AND RESULT6.1 DragonBox Adaptive6.1.1 Game Description

DragonBox Adaptive is an educational math puzzle gamedesigned to teach how to solve algebra equations to childrenranging from kindergarteners to K-12 students [2], which isevolved from the original game DragonBox [1]. Figure 2shows the screenshots of the game. Each game level repre-sents an algebra equation to solve. The panel is divided intotwo sides filled with cards representing numbers and vari-ables. A player can perform algebraic operations by mergingcards in the equation (e.g., ‘-a+a’ → ‘0’), eliminating iden-tities (e.g. ‘0’→ ‘ ’), or using a card in the deck to performaddition, multiplication, or division on the both sides of the

Figure 2: Screenshots of DragonBox Adaptive.(Top) The early stage of the game. It is equiv-alent to an equation a-b=-6+a+x. The card with astarred box on the bottom right is the DragonBox.(Bottom) The game teaches more and more con-cepts, and eventually kids learn to solve complexequations.

equation. To clear a level, one should eliminate all the cardson one side of the panel except the DragonBox card, whichstands for the unknown variable.

Table 1 shows the examples of states, actions, transitions,and policies in the game. Since DragonBox Adaptive is acard puzzle game, the game state and available actions arewell discretized. A state is a pair of level ID and the currentequation (e.g. lv3.x-1=0). Available actions for a specificstate are the available movements, or algebraic operations,that the game mechanics allow the players to perform (e.g.add a on both sides or subtract a on both sides). Performingan action moves a user from one state to another state (e.g.an action adding a on both sides moves a user from lv3.x-a=0 to lv3.x-a+a=0+a), which naturally defines a transitionfunction. Since the transition is deterministic and injective,we use a notation s → s′ for a transition from s to s′ =

125 250 500 1K 2K 4K 8K 12K 16K 20K

-3.5M

-3.0M

-2.5M

-2.0MLog-likelihood

Ours Markov

125 250 500 1K 2K 4K 8K 12K 16K 20K# of Players in Training Set

0.60

0.62

0.64

0.66

0.68

0.70

Accuracy Rate

Ours Markov

Figure 3: Performances of our predictor and that ofthe Markov predictor with different sizes of trainingdata.

f(s, a) (e.g. lv3.x-a=0 → lv3.x-a+a=0+a).

We collected the game logs from the Norway Algebra Chal-lenge [3], a competition solving as many algebra equationsas possible in DragonBox Adaptive. About 36,000 K-12 stu-dents across the country participated in the competition,which was held on January 2014 for a week. One character-istic of this dataset is a low quit rate, which is achieved be-cause students intensively played the game with classmatesto rank up their class. About 65% of the participants playedthe game more than an hour, and more than 200 equationswere solved per student on average. This is important inour task because we can collect a lot of training data evenin higher levels. After cleaning, we collected 24,000 students’logs with about 280,000 states, 540,000 transitions, and 21million moves. 4,000 student’s logs were assigned to testdata and the rest as training data. As the parameters ofour framework were determined with another Algebra Chal-lenge dataset separate from Norway Challenge, there was nolearning from the test data.

6.1.2 Overall PerformanceFigure 3 shows the performances of our framework and thoseof the Markov predictor with different sizes of training data.We use the Markov predictor as a baseline, because it is an-other data-driven policy predictor working on a state-actiongraph structure, and because our model is using the Markovpredictor when it could not find relevant features to thegiven state.

In both metrics, the performance of our predictor increaseswith the size of training data. As it has not convergedyet, we can even expect better performance with additionaltraining data. For the Markov predictor, its performance

0.0 0.2 0.4 0.6 0.8 1.0Accuracy Rate (Markov)

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy Rate (Ours)

0.00 0.05 0.10 0.15 0.20 0.25Accuracy Rate Improvement (Ours - Markov)

0

200

400

# of States

Figure 4: (Top) Scatter plot of accuracy rates of ourpredictor versus that of the Markov predictor foreach state with learned features. We see the perfor-mance of our predictor is strictly better than thatof the Markov predictor in most cases. (Bottom)Histogram of accuracy rate improvement, while onecount is a state with learned features.

notably improves with the size of training data only in log-likelihood metric. We believe this is because the accuracyrate mostly comes from frequently visited states, which thepredictor already reached to the point with no improvement,while a significant portion of the log-likelihood value comesfrom sparsely visited states, which improves as it gathersmore data.

The difference between the two methods is increasing as thesize of training data increases. With 20,000 user trajectoriesas training data, the log-likelihood of our predictor is almost12% lower than that of the Markov predictor and the accu-racy rate improves from 64% from 68%. Since the Markovpredictor gives a static policy, we can say this gain comesfrom considering individual behavior differences. Also, evenwith a relatively small size of training data, the performanceof our method is still higher than that of the Markov predic-tor. We believe our predictor performs strictly better thanthe Markov predictor even with insufficient data, becauseour method switches to the latter when it does not haveenough confidence on selecting features.

6.1.3 Statewise PerformanceIn this subsection, we further analyze the performances oftwo methods in the smaller scope. Our predictor learned1,838 logistic regression classifiers with 20,000 player trajec-tories as training data. Considering there are about 280,000game states in total, our method selected no features formore than 99% of the states and decided to use the Markovpredictor instead. However, more than 60% of the move-

Next State Feature Weight

lv.3.4.1.x+0=b-a

1[(lv.3.5.1.x+a=b→ lv.3.5.1.x+0=b-a) ∈ τu] 0.5831[(lv.3.4.1.x+a=b→ lv.3.4.1.x+0=b-a) ∈ τu] 0.5681[(lv.3.4.1.x+a=b→ lv.3.4.1.x+0=b-a) = (st−3, at−3)] 0.3871[(lv.2.19.x*b+a=c→ lv.2.19.x*b+0=c-a) ∈ τu] 0.181

lv.3.4.1.x+a-a=b-a

1[(lv.3.4.1.x+a+a=b-a→ lv.3.4.1.x+0=b-a) = (st−3, at−3)] 0.4661[(lv.3.4.1.x+a=b→ lv.3.4.1.x+a-a=b-a) = (st−4, at−4)] 0.4571[(lv.3.3.2.x+a+b=c→ lv.3.3.2.x+a+b-b=c-b) = (st−7, at−7)] 0.3891[(lv.2.3.x+1=a→ lv.2.3.x+0=a-1) ∈ τu] 0.119

lv.3.4.1.x+a+a=b+a

1[(lv.3.4.1.x+a+a+a=b+a+a→ lv.3.4.1.x+a+a=b+a) = (st−2, at−2)] 0.2471[(lv.3.4.1.x+a=b→ lv.3.4.1.x+a+a=b+a) = (st−6, at−6)] 0.1921[(lv.1.18.x+a+a=a+b→ lv.1.18.x+a+0=b+0) ∈ τ ] 0.1281[(lv.2.3.x+(-x)/(-x)+x/x=1+a→ lv.2.3.x+(-x)/(-x)+x/x-1=a+0) ∈ τ ] 0.115

Table 2: Selected features with high weights for predicting a transition from state ‘lv.3.4.1.x+a=b’ to eachavailable action. The selected features closely related to the task it is going to predict. Interesting featuresmentioned in the text are highlighted.

Transition Accuracy Recallfrom x+a=b LogReg Markov LogReg Markovx+0=-a+b 88.0 76.7 88.1 76.7x+a-a=-a+b 60.6 19.2 60.3 19.2x+a+a=a+b 11.5 3.7 11.6 3.8

Table 3: Accuracy and recall rate for each actionon state lv.3.4.1.x+a=b. The performance of high-lighted transitions are almost tripled.

ments in training data starts from the states with corre-sponding logistic regression classifiers. It means our frame-work invested its resources on a small set of states, which areso influential that they govern the majority of the predictiontasks.

Figure 4 shows the accuracy rates in both predictors for eachstate that learned a logistic regression classifier. We can seethe performance of our predictor is better than that of theMarkov predictor in the most cases. The other case is ig-norable: the Markov performs better than ours in less than2% of the states with logistic regressors, while the averageperformance drop for those states is 0.03%. This observa-tion further supports our argument that our model performsstrictly better than the Markov model because of our robustfeature selection process.

Table 3 gives the evaluations on a state that showed an ac-curacy rate improvement from 63% to 80%. Since we areevaluating stochastic policies, we use the following defini-tion for accuracy and recall for a pair (s, a):∑

u∈U

∑t∈[0,tu]

1[(s, a) = (stv, atv)] · π̃(a|s, τ tu)∑

u∈U

∑t∈[0,tu]

1[(s, a) = (stv, atv)],

∑u∈U

∑t∈[0,tu]

1[(s, a) = (stv, atv)] · π̃(a|s, τ tu)∑

u∈U

∑t∈[0,tu]

1[s = stv] · π̃tu(a|s, τ tu).

In the table, the starting equation is x+a=b, and a playercan perform addition or subtraction with a card ‘a’ on the

deck. There are three possible transitions because the gamemechanics allow two ways to subtract: one putting ‘-a’ cardnext to ‘a’ card, and another one putting ‘-a’ card over ‘a’ tomerge them to ‘0’. From now on, we will call them ‘normalsubtraction’ and ‘subtraction with merging’. Which move touse among them does not affect to clear the game, but thegame can be cleared more efficiently with the latter move.The third transition is addition making the current equationto the equation x+a+a=b+a. This is not the right way to solvethe level, because the DragonBox is not going to be isolated.We will call it ‘unnecessary addition’.

For the normal subtraction, the predictive power is morethan tripled. We believe our predictor successfully capturedthis habitual movement, which we also believe is not likelyto change once fixed. Indeed, DragonBox Adaptive does notprovide a strong reward on decreasing the number of move-ments, nor suggest a guide to promote the students usingthe subtraction with merging. For the unnecessary addi-tion, the predictive power is also more than tripled. Becauseone must have seen similar problems several times, studentsrarely makes this mistake (3.8% in training data). In spiteof such sparsity, our predictor improved its predictive powerover that of the Markov predictor.

6.1.4 Learned Features and Feature WeightsIn this subsection, we take a look at the learned features andclassifiers. Most of the learned features with high weightsare closely related to the task we are trying to predict wheninspected by experts. Table 2 shows some of the learned fea-tures with high weights in the classifier for each transition inthe previous subsection: subtraction with merging, normalsubtraction, and unnecessary addition. Most of the selectedfeatures with high weights are actually related to the actionthat it is going to predict. For example, the movement inthe first feature in the table x+a=b → x+0=b-a is exactlysame as the movement we are trying to predict. The onlydifference is the level IDs. Note that having a feature setof the future level (lv3.5.1) is possible because our gameprogression forces students to visit previous levels when astudent fails to clear a level.

For another example, the movement in the eleventh featurein the table x+a+a=a+b→ x+a+0=b+0 implies an unnecessary

1000 2000 3000 4000 5000 6000 7000-48K

-47K

-46K

-45K

-44K

-43KLog-likelihood

Ours Markov

1000 2000 3000 4000 5000 6000 7000# of Players in Training Set

0.31

0.32

0.33

0.34

0.35

0.36

0.37

Accuracy Rate

Ours Markov

Figure 5: Performances of two methods on Refrac-tion.

addition was performed, because the game level 1.18 startsfrom x+a=b: to reach the equation x+a+a=a+b, one has toperform the unnecessary action ahead. It is also interest-ing that the game level 1.18 is almost 25 levels away fromthe current level 3.4.1. It would be a natural assumptionthat only recent playdata affect a user’s behavior, but thisprovides an evidence that this is not the case. Consideringthat a player can only proceed a level after correcting pre-viously made mistakes, it also implies the learning curve ofthis concept is relatively shallow.

One more thing to mention is that when a feature is de-cribing more recent part of the playdata, it tends to havea higher weight. In the table, features from level 3 usu-ally receives higher weights compared to the features fromlevel 1 or 2. This is intuitive because a user is more likelyto change his or her behavior as time passes. This impliesthat our model is also capturing the temporal behavior ofindividuals.

6.2 RefractionTo demonstrate that our framework can be used on othersystems without additional authoring, we also run an ex-periment with another educational game, Refraction [4]. Aswe use the same experimental setting and game data as inLiu et al. [15], we omit the details of the game and thestate-action model. The dataset contains 8,000 users’ game-play data, with about 70,000 states, 460,000 transitions, and360,000 moves. 1,000 users’ gameplay data is assigned to atest set, and the rest as a training set. We use the gameplaydata from level 1 to level 8 to predict the movements in level8. There are no changes in our framework, except that thesmoothing parameter � in the log-likelihood metric is set to0.3 to match the performance of the Markov predictor usedin previous work [15].

Figure 5 shows the performance of our predictor and theMarkov predictor. We see our predictor performs betterthan the Markov predictor, although the improvement ismuch smaller compared to DragonBox Adaptive. We be-lieve this is because level 8 is an early level, and we do nothave enough data. Level 8 is the first level without thetutorial, it would be difficult to detect a confident signal de-scribing individual behavior. In other words, students didnot have enough opportunity to show their personality. Wecould not run another experiment on a later level due tolack of players from the high drop-out rate. Moreover, theRefraction dataset (360,000 moves) is much smaller than theDragonBox Adaptive dataset (21 million moves), while thetotal number of transitions is similar in both.

Nevertheless, we successfully showed that our framework canbe used in a different system with no additional expert au-thoring, and showed our predictor still performs better thanthe Markov predictor.

7. CONCLUSION AND FUTURE WORKModeling user behavior in open-ended environments has thepotential to greatly increase undestanding of human learn-ing processes, as well as helping us better adapt to stu-dents. In this paper, we present a data-driven individualpolicy-learning framework that reduces the burden of hand-designing a cognitive model and system-specific features.Our framework automatically selects relevant features froma default feature set defined on a general state-action graphstructure, and learns an individual policy from the trajec-tory of a player. We apply our method to predict playermovements in two educational puzzle games and showedour predictor outperforms a baseline predictor. We alsoshow that the performance improvement comes not onlyfrom frequently observed movements, but also from sparselyobserved erroneous movements. Finally, we see our robustfeature selection makes the predictor more efficient, power-ful, and interpretable by investing its resources on a smallset of influential states and relevant features.

We see numerous opportunities for further improvement ofour framework. First, we can experiment with different clas-sifiers instead of a logistic regressor. Since a logistic regres-sion model is a single layer artificial neural network (ANN),we believe using a multi-layered ANN is a natural exten-sion to improve its predictive power. Using an ensemble ofclassifiers would be another way to boost the performance.Second, we can add more graph navigation features into thedefault feature set. A feature specifying whether a transi-tion is not visited only when it was available to the user isthe first thing to try, because it specifies whether a certainbehavior has been avoided intentionally or if the user sim-ply did not have an opportunity to make such a choice. Avisit indicator of a specific chain of transitions or the timespent on a certain state can also be possible features. Fi-nally, we can try other feature selection techniques. Recur-sive feature elimination or L1-based feature selection mightproduce a better result because univariate approaches, suchas our chi-squared test, do not consider the effect of multiplefeatures working together [10].

Overall, we are also very interested in building applicationsbased on our framework. Integrating the individual behav-

ior predictor into user-specific content generation such as apersonalized hinting system or an adaptive level progressionwould be the first step. Moreover, we believe our frame-work will be also useful in other fields for learning individ-ual behavior, such as spoken dialogue systems [14], roboticlearning from demonstration [7], and recommender systems[18].

8. ACKNOWLEDGMENTSThis work was supported by the University of WashingtonCenter for Game Science, the National Science FoundationGraduate Research Fellowship grant No. DGE-0718124, theBill and Melinda Gates Foundation, and Samsung Schol-arship Foundation. The authors would also like to thankTravis Mandel for helpful discussions.

9. REFERENCES[1] Dragonbox. http://www.dragonboxapp.com/.

WeWantToKnow, Retrieved 2014-02-27.

[2] Dragonbox adaptive. http://centerforgamescience.org/portfolio/dragonbox/. Center for GameScience, Retrieved 2014-02-27.

[3] Norway algebra challenge.http://no.algebrachallenge.org/. AlgebraChallenge, Retrieved 2014-02-27.

[4] Refraction. http://centerforgamescience.org/portfolio/refraction/. Center for Game Science,Retrieved 2014-02-27.

[5] D. W. Albrecht, I. Zukerman, and A. E. Nicholson.Bayesian models for keyhole plan recognition in anadventure game. User modeling and user-adaptedinteraction, 8(1-2):5–47, 1998.

[6] V. Aleven, B. M. McLaren, J. Sewall, and K. R.Koedinger. The cognitive tutor authoring tools (ctat):Preliminary evaluation of efficiency gains. InIntelligent Tutoring Systems, pages 61–70. Springer,2006.

[7] B. D. Argall, S. Chernova, M. Veloso, andB. Browning. A survey of robot learning fromdemonstration. Robotics and autonomous systems,57(5):469–483, 2009.

[8] Y. Bergner, S. Droschler, G. Kortemeyer, S. Rayyan,D. T. Seaton, and D. E. Pritchard. Model-BasedCollaborative Filtering Analysis of Student ResponseData: Machine-Learning Item Response Theory. InEDM, pages 95–102, 2012.

[9] A. T. Corbett and J. R. Anderson. Knowledge tracing:Modeling the acquisition of procedural knowledge.User modeling and user-adapted interaction,4(4):253–278, 1994.

[10] I. Guyon and A. Elisseeff. An introduction to variableand feature selection. The Journal of MachineLearning Research, 3:1157–1182, 2003.

[11] R. K. Hambleton, H. Swaminathan, and H. J. Rogers.Fundamentals of item response theory. Sage, 1991.

[12] A. R. Jansen, D. L. Dowe, and G. E. Farr. Inductiveinference of chess player strategy. In Proceedings of the6th Pacific Rim International Conference on ArtificialIntelligence, PRICAI’00, pages 61–71, Berlin,Heidelberg, 2000. Springer-Verlag.

[13] L. A. Jeni, J. F. Cohn, and F. De La Torre. Facingimbalanced data–recommendations for the use of

performance metrics. In Affective Computing andIntelligent Interaction (ACII), 2013 HumaineAssociation Conference on, pages 245–251. IEEE,2013.

[14] D. J. Litman and S. Pan. Designing and evaluating anadaptive spoken dialogue system. User Modeling andUser-Adapted Interaction, 12(2-3):111–137, 2002.

[15] Y.-E. Liu, T. Mandel, E. Butler, E. Andersen,E. O’Rourke, E. Brunskill, and Z. Popović. Predictingplayer moves in an educational game: A hybridapproach. In EDM, pages 106–113, 2013.

[16] M. Mavrikis. Data-driven modelling of students’interactions in an ILE. In EDM, pages 87–96, 2008.

[17] Z. A. Pardos and N. T. Heffernan. Modelingindividualization in a bayesian networksimplementation of knowledge tracing. In UserModeling, Adaptation, and Personalization, pages255–266. Springer, 2010.

[18] M. J. Pazzani and D. Billsus. Content-basedrecommendation systems. In The adaptive web, pages325–341. Springer, 2007.

[19] M. Prensky. Computer games and learning: Digitalgame-based learning. Handbook of computer gamestudies, 18:97–122, 2005.

[20] S. Ritter, T. K. Harris, T. Nixon, D. Dickison, R. C.Murray, and B. Towle. Reducing the KnowledgeTracing Space. In EDM, pages 151–160, 2009.

[21] D. W. Shaffer. How computer games help childrenlearn. Macmillan, 2006.

[22] G. Synnaeve and P. Bessière. A bayesian model forplan recognition in rts games applied to starcraft. InProceedings of the Seventh Artificial Intelligence andInteractive Digital Entertainment Conference (AIIDE2011), pages 79–84, 2011.

[23] B. Tastan and G. R. Sukthankar. Learning policies forfirst person shooter games using inverse reinforcementlearning. 2011.

[24] N. Thai-Nghe, L. Drumond, A. Krohn-Grimberghe,and L. Schmidt-Thieme. Recommender system forpredicting student performance. Procedia ComputerScience, 1(2):2811–2819, 2010.

[25] N. Thai-Nghe, T. Horváth, and L. Schmidt-Thieme.Factorization Models for Forecasting StudentPerformance. In EDM, pages 11–20, 2011.

[26] Y. Yang and J. O. Pedersen. A comparative study onfeature selection in text categorization. In Proceedingsof the Fourteenth International Conference onMachine Learning, ICML ’97, pages 412–420, SanFrancisco, CA, USA, 1997. Morgan KaufmannPublishers Inc.

[27] M. V. Yudelson, K. R. Koedinger, and G. J. Gordon.Individualized bayesian knowledge tracing models. InArtificial Intelligence in Education, pages 171–180.Springer, 2013.

[28] A. Zook and M. O. Riedl. A temporal data-drivenplayer model for dynamic difficulty adjustment. InAIIDE, 2012.

http://www.dragonboxapp.com/http://centerforgamescience.org/portfolio/dragonbox/http://centerforgamescience.org/portfolio/dragonbox/http://no.algebrachallenge.org/http://centerforgamescience.org/portfolio/refraction/http://centerforgamescience.org/portfolio/refraction/http://educationaldatamining.org/EDM2012/uploads/procs/Full_Papers/edm2012_full_4.pdfhttp://educationaldatamining.org/EDM2012/uploads/procs/Full_Papers/edm2012_full_4.pdfhttp://educationaldatamining.org/EDM2012/uploads/procs/Full_Papers/edm2012_full_4.pdfhttp://www.educationaldatamining.org/EDM2008/uploads/proc/9_Mavrikis_27.pdfhttp://www.educationaldatamining.org/EDM2008/uploads/proc/9_Mavrikis_27.pdfhttp://www.educationaldatamining.org/EDM2009/uploads/proceedings/ritter.pdfhttp://www.educationaldatamining.org/EDM2009/uploads/proceedings/ritter.pdfhttp://educationaldatamining.org/EDM2011/wp-content/uploads/proc/edm2011_paper14_full_Thai-Nghe.pdfhttp://educationaldatamining.org/EDM2011/wp-content/uploads/proc/edm2011_paper14_full_Thai-Nghe.pdf

IntroductionRelated WorkProblem DefinitionEvaluationAlgorithmExperiment and ResultDragonBox AdaptiveGame DescriptionOverall PerformanceStatewise PerformanceLearned Features and Feature Weights

Refraction

Conclusion and Future WorkAcknowledgmentsReferences