Gaming Learning Analytics: using data for improving serious games applicabilityReal Colegio Complutense at Harvard

Baltasar Fernandez-Manjon balta@fdi.ucm.es , @BaltaFMe-UCM Research Group , www.e-ucm.es

Cambridge, 07/04/2016

Cátedra Telefónica-Complutense Educación Digital y Juegos Serios

2

Serious Games: definition and use• Any use of digital games with purposes other

than entertainment (Michael & Chen, 2006)• Serious Games have probed to be educationally

effective in several domains– Medicine, military, business, corporate training

• But still is a low adoption of Serious Games in mainstream education

• Serious Games considered usually as a complementary content– Mainly used for motivational purposes– No actual impact on the final mark

Data in Education: LA and GLA• Learning Analytics Improving education based

on data analysis– Data driven– Evidence-based education

• Gaming Learning Analytics is a specific case when all interaction data is used in serious games for improving the learning process supported by the games– Educational games not as “black boxes”

4

What happens when new technology is used in education?Serious Games….

Educational games in the classroom: game

• The Radix Endeavor– https://www.radixendeavor

.org• MMOG with Science,

Technology, Engeneering and Math (STEM) topics– Many different quests

• Adapted to middle and high school curriculum

• Facilities for teachers– Class configuration– Quest assignation– Tested in actual school

settings

Educational games in the classroom: settings

• Master students in the e-learning class– 13 students, 8

nationalities• No previous info about the

game– If stuck ask for help or

watch the help videos• One hour free play

– Who perform “better” at 30 minutes and at the end

• Publish screen captures in the class Moodle course forum

Educational games in the classroom: outcome• Students were really engaged for the whole hour

– They enjoyed the experience– Consider the game a commercial quality game – No one asked for help

• All of them published the screenshots in the Moodle class forum

• Educational outcome– Only 4 of the students complete more than 2 quests – Most of the students just spend the time exploring the

environment but not doing anything “useful” or learning anything

• Actual data can be contradictory with simple observation

8

Other experiments ..

• Experiment: Gamified experience in the classroomGoals– First to complete the game– Maximum “points” in the gamebut– Initially students were not informed that

Learning Analytics was in place

After some playing time LA data was showed to the students

9

Using data interaction in the classroom

GLA simplify SG deployment in the classroom‣ Teacher knows what is happening in the

class‣ Teacher “keep the control”

Behavior change when students know that LA was in place

• More oriented to success and less exploration or “playing”

Serious games at school (USA)

Takeuchi, L. M., & Vaala, S. (2014). Level up learning: A national survey on teaching with digital games. New York: The Joan Ganz Cooney Center at Sesame Workshop

• 62% teachers have used games to simulate and supplement learning.

• (Harris Poll, 2014 –Pearson-, 1000 teachers K12, USA)

11

How teacher selects SG

Uses of Gaming Learning Analytics in educational games

• Game testing – game analytics– It is the game realiable?– How many students finish the game?– Average time to complete the game?

• Game deployment in the class – tools for teachers– Real-time information for supporting the teacher– “Stealth” student evaluation– Knowing what is happening when the game is

deployed in the class• Different uses for different stakeholders

– Teachers, students, administrators, – Game developpers, researchers

13

Can we generalize and simplify the adoption of Gaming Learning Analytics?RAGE Project….

ProsocialLearn

RAGENo-one-

left-behind 3d-

Tune-In

BEACONING

XX XX

XXXX

XX

XX

XX

XXXX

XX

XX

EU H2020 serious games projects

H2020 RAGE in a nutshell

• RAGE will deliver advanced technology and know-how to support the European Applied Games industry build-up and job creation– February 2015, 4 years, 20 partners, 9M €

• Creating a new serious games ecosystem by making available a set of reusable technology components for developing advance serious games easier, faster and more cost-effectively

• Open source gaming learning analytics framework – provide all the required services (e.g. game tracker,

learning analytics server, visualization of analytics information)

– easy inclusion of gaming learning analytics techniques in the new games

Realising an Applied Gaming Eco-System

RAGE: creating the Gaming Learning Analytics infrastructure

H2020 RAGE project will simplify the process of Serious Games creation with ready to use assets• Game trackers (Unity3D, javascript)• LA server infrastructure and services

• Basic analysis and visualization of LA data• Extensible architecture

• Standards support (e.g. xAPI)• Open code (github)Using interaction data for assessment

Realising an Applied Gaming Eco-System

17

RAGE GLA Open Architecture

Clients AA Applications

Games

Analytics Frontend

AAFrontend

Play

ers

Devs

, Stu

dent

s,

Teac

hers

.Ad

min

s

UsersRolesResourcesPermissionsApplications

Auth

entic

ation

Auth

oriza

tion

JWTJSON WEB

TOKEN

GamesSessionsResults

TopologyAnalytics Backend

GamesSessions

Collector

Results

Application 1

Application N

Kafk

a Q

ueueLRS

We reuse APEREO Open LRS

18

New Reference Models for GLA

Implications of GLA in the design of the game

RAGE, serious games and xAPI• Define interactions model to assess serious

games• Create a xAPI vocabulary to represent the model–Repurpose existing verbs and activities–Create verbs and activities where needed–Create extensions where needed

• Define analysis for the model– Simple metrics processing statements

(serious game agnostic)–More advanced analysis based on serious

game goals

H2020 Beaconing project

• BEACONING stands for ‘Breaking Educational Barriers with Contextualised, Pervasive and Gameful Learning’ • Started in january 2016, 15 partners, 9 countries, 6M

• Global goal is learning ‘anytime anywhere’• Exploitation of technologies for contextual pervasive games and use of

gamification techniques• Problem based approach to learning• Enriching the Gaming Learning Analytics data model with

the contextual, geolocalized and accessibility information• Large pilots in real settings with content providers

– Formal and informal learning across virtual and physical spaces• GLA is a key element in the games and pilots

evaluation• Using RAGE infrastructure and extending it for these

new requirements and applications

21

New models for specific uses: games for people with cognitive dissabilities

22

Cytopathology Challenge

GOALS• Educational application for an Introduction to Cytopathology

course in cooperation with Harvard Med School and MGH• Train cytologists both in Harvard U and in limited-resource

countries CONSTRAINTS• Work in Low-cost Android tablets (50$)• Work in web browsers • No always wifi-connection to play• Limited budget

23

Cytopathology Challenge

2.Send traces

1.Play Cytopathology Challenge

3.Analyze data

4.Recover results

5.Show analysis to evaluate the students

Scientific Goals: from formal pre-post evaluation to GLA?• Off-line learning analytics• Formal evaluation of

games is very complex and expensive– Pre-test– Post-test

• Very few games have been formally probed to be effective

• Similar results with Learning Analytics than with pre-post test?

24

Conclusions

• GLA in Serious Games has a great potential from the application and research perspective

• Still complex to implement GLA in SG– Increases the (already high) cost of the games– New specifications could simplify the task (e.g.

xAPI)• New open frameworks produced by EU projects and

new standards specifications could greatly simplify GLA implementation and adoption

• And this is just the beginning (BYOD, VR, new interactions devices..)– Games are driving the disruption ….

26

Thank you!•¿Questions?• Mail: balta@fdi.ucm.es• Twitter: @BaltaFM

• GScholar:  https://scholar.google.es/citations?user=eNJxjcwAAAAJ&hl=en&oi=ao • ResearchGate:  www.researchgate.net/profile/Baltasar_Fernandez-Manjon• Slideshare: http://www.slideshare.net/BaltasarFernandezManjon