chai: : computer human adapted interaction research group interfaces for learning data...

chai::Computer human adapted interaction research group

Interfaces for Learning Data Visualizations

Judy KayCHAI: Computer Human Adapted Interaction Research Group

School of Information Technologies, University of SydneyPresident of the AIED Society (2009-11), Programme co-Chair ITS2010, General Chair AIED2011

Advisory Board User Modeling Programme co-Chair Pervasive 2012, Chair of the Joint Ubicomp and Pervasive Steering Committee


About me

• Inventing future technology to tackle important problems, notably in learning

• Personalisation• Personal data and its management• Putting people in control• Open Learner Models (OLMs)• Metacognition and OLMs• Interactive surfaces… walls, tables…


Learning analytics as a form of Learner/User Modelling

with interfaces


How to create interfaces for LA?

• User-centred approaches– Stake-holders– Mental models– The problem?

• Core tools and principles• Case studies

– Institution– Class– Individual learner


Interfaces… visualisations


Why visualisations?

Fekete, J. D., Van Wijk, J. J., Stasko, J. T., & North, C. (2008). The value of information visualization. In Information Visualization (pp. 1-18).

Springer Berlin Heidelberg.

“…easy and fast to see that there is no red circle, or to evaluate the relative quantity of red and blue circles. Color is one type of feature that can be processed preattentively, but only for some tasks and within some limits. [eg] if there were more than seven colors …, answering the question could not be done with preattentive processingand would require sequential scanning, a much longer process.


But how to create the right visualisations?

Are there simple rules?Simple principles?

Simple and constant solutions?


Principle: individual data takes on more meaning….

When comparisons are supported:• Others• Temporal• Contextual


Patina: Dynamic Heatmaps for Visualizing Application Usage (CHI2013) Justin Matejka, Tovi Grossman, and George Fitzmaurice

This user’s footprints

Overall population footprints


Extended Case Study

Concrete example of my work to underpin the activities


Defining features

• The problem:– Group work is hard but it is important– Group work in learning context has many

problems that cause great anguish, inefficiency• Target stakeholders:

– Learner as individual– Team leaders (manager, tracker)– Facilitators (tutor, lecturer)


Trac: Tool supporting long term group work

Used by team members, facilitators, teachers, some clients


TRAC• open source tool for supporting software development projects

Wiki page editorTicket ManagerSVN source repository

• Not a learning system but used in a learning context.


Huge amounts of data about the group members and their

interactions


Narcissus

Upton, K., and J. Kay. (2009) Narcissus: interactive activity mirror for small groups. In UMAP09, User Modeling, Adaptation and Personalisation, Springer-Verlag, 54-65

chai::Computer human adapted interaction research group19

Integrated of mirror tool Narcissus

tab


ITS2008Lifelong learning, learner

models and sugmented cognition

Lifelong modelling – mirrors and mining


Header –Group view Display for

one user

Time – activity on that day is shown for each user, on each medium


Wiki contribitions

svn contribitions

ticket contribitions


Click on cell …

…to see details


Explainsscoring


Individual summary

Group average

26

Click on ticket activity for a day

Associated details

Click on ticket label

27

Details of that ticket

ITS2008Lifelong learning, learner models and sugmented cognition

Sequence mining

Managers Developers Loafers Others

Group 1 *1 3 1 1

Group 2 *1 0 1 3

Group 3 0 1 2 **3

Group 4 *1 3 2 0

Group 5 3 *1 0 3

Group 6 *1 1 3 1

Group 7 *1 0 2 4

Group 1 – 1 person had sequences characteristic of managers.

* That person had the manager role

Group 1 – 3 members had developer activity sequences

Group 3 – dysfunctionaland here we might see why

Group 5 – another way to be dysfunctional


Activity 1

• Your Stakeholders?


Activity 1

• Stakeholders?– Learners– Parents, Mentors, Facilitators– Teachers– Supervisors– Institutions– Quality assessors– Researchers


Activity 2

• Problems you would like to tackle?


Activity 2

• Current problems we aim to tackle?– Teacher: Early identification of at-risk individuals– Learner: Decision support

• Am I doing well enough?• Am I doing what is expected of me?

– Institution: Effectiveness of teaching and learning?


Building from SMILI

Bull, S., & Kay, J. (2007). Student Models that Invite the Learner In:

The SMILI:() Open Learner Modelling Framework. International Journal of Artificial Intelligence in Education, 17(2), 89-120.


What is an Open Learner Model?

• Any interface to data that a system keeps about the learner

• Came from AI + personalisation where learner model drives personalisation

• OLM has become a first-class citizen!• Link to Learning Analytics….


SMILI questions

• How does the open learner model fit into the overall interaction? – What problem does it aim to address?

• WHAT is open? • HOW is it presented? • WHO controls access?


The purposes for opening the learner model are:

• Improving accuracy • Promoting learner • Helping learners to plan and/or monitor their • Facilitating collaboration and/or competition • Facilitating navigation of the learning system • Assessment

• Complex of issues of managing personal data:– right of access to data about themselves– Right of control over their learner model – increasing trust


Scrutable user models and personalised systems

Research systems only, so farBut hints of their being ready to emerge in mainstream software


Interfaces to substantial learner models

Core concepts in a whole semester long subject


HCI subject with online lectures

• Exploit data from:– logs of interaction with lecture “slides”– class assessments

• Lightweight ontology for tagging– automatic analysis of online dictionary– augmented with class-specific concepts (as class

glossary) – enabling combination of multiple data sources about

each concept– and inference up/down ontology




SIVLots of green means

learner doing well

Weak aspects visible as red

Overviewvisualisation




SIV

Kay, J and A Lum. "Exploiting readily available web data for scrutable student models.” Proceedings of the conference on Artificial Intelligence in Education 2005.

Little detail


Mental models


Mental models

A set of beliefs that the user holds


Mental models

A set of beliefs that the user holdseg. A whale is a fish

The subject requires rote learningI expect to perform at about the median in this class


Mental models come from:

• Formal education• And so much else

– Experience– Cultural expectations– Context– Emotional state – ….

• Determining what the user– Believes to be true– Trusts– Feels permitted to consider and do– Feeling of competence


Why do mental models matter for interface designers?


Why do mental models matter for interface designers?

They define • what a user can “see” and “hear”• How they interpret informationClashes between user, programmer, expert MMs


Activity

• Mental models• What are key elements for your LA needs?


Pervasive technologies Case study

Lots of embedded interaction devices, ready for interaction

Where things may be headed….


User models in real classrooms

For orchestrationFor in-class monitoring to inform teacher actions

For post-hoc reflection by the teacher

The collaborative task (concept mapping and problem solving)

[6] Novak, J. and A. Cañas, The Theory Underlying Concept Maps and How to Construct and Use Them T.R.I.C. 2006-01, Editor. 2006, Florida Institute for Human and Machine Cognition.

• Concept mapping is: – A tool for externalising knowledge

– Applied in different domains

– Promotes meaningful learning

– Has been used by organisations such asNASA, Navy, and universities around the world.


The big pictureCSCL

Computer Supported Collaborative learning

HCIHuman Computer

Interactions

EDMEducational Data-Mining Interactive tabletops in the

classroom

Interactive Tabletops and Surfaces 2010, 2011Work In Progress , CHI 2012

Int. Conf. in Learning Sciences, ICLS 2012Intelligent Tutoring Systems, ITS 2012

Computer Supported Collaborative Learning CSCL 2011Educational Data Mining 2011

Interactive Tabletops and Surfaces 2012Workshop on Orchestration , ICLS 2012

orchestration


Architecture


Collaid

Our gear

Learner’s physical differentiation


Our enriched interactive tabletop

Kinect sensor

Multi-touch tabletop

R. Martinez, A. Collins, J. Kay, and K. Yacef. Who did what? who said that? Collaid: an environment for capturing traces of collaborative learning at the tabletop. In ACM International Conference on Interactive Tabletops and Surfaces, ITS 2011, pages 172-181, 2011.

Logs:

Differentiated tabletop actions

Snapshots of the artefact


Our gear 2

CmateConcept Mapping at the Tabletop


Learning outcomes for activities

Concentric layout

Significant correlated with higher levels of equity of participation (>0.4).

Concentric

Oriented towards a Learner

“next time I would ask students to use a circular layout”

Teacher:


From Design to Enactment and Reflection

Collaboration and equality

Adherence to the class

script

Learning outcomes for

activities


Classroom activity design


Adherence to the class script

(14 tutorials)

Implications

This was the most important activity from the learning perspective

It forced the teacher to use more time than the 50 minutes

There was not enough time for activity 2 as planned


Adherence to the class script

“This is a very good reminder... maybe the structure for the next tutorials should be changed to give more

time for Activity 2”

Teacher:

Effectiveness of the script

Post-hoc teacher’s reflection

Standardisation if multiple tutors.

Activity re-design


Learning outcomes for activities

* high achieving groups had more than 50% of these crucial propositions

This analysis suggests the low achieving groups took longer to get started.

“It would be more valuable to get this information per each group during the tutorials”.Teacher:


Can tabletops automatically alert teacher which group

may need attention?

Adding real-time learner model What’s the impact of

showing information to the teacher?


The Orchestration Dashboard

To help teachers to control multiple classroom tutorial

sessions


Awareness and Control


The Awareness Dashboard

To help teachers to determine whether groups or individual

learners need attention

Multiplatform!


Group 1

Group 2

Group 3

Class level Dashboard

Martinez Maldonado, R., Kay, J., Yacef, K. and Schwendimann, B. An Interactive Teacher’s Dashboard for Monitoring Groups in a Multi-tabletop Learning Environment. Intelligent Tutoring Systems (2012), 482-492.


Group 1

Group 2

Group 3

• A Best-First tree model trained in another dataset classifies each block 30 seconds of activity

Features:• # of active participants in verbal discussions, • amount of speech, • number of touches • symmetry of activity (Gini coefficient).

• Labels: Collaborative, Non-collaborative, or Average.

• The visualisation shows the accumulation of these.

Martinez R, Wallace J, Kay J, Yacef K Modelling and identifying collaborative situations in a collocated multi-display groupware setting. In: AIED 2011. pp. 196-204 (2011)

Class level: Indicator of detected collaboration.


Class level: Graph of interaction with others’ objects

Group 1

Group 2

Group 3

• The Circles indicate the number of touches

• The Lines represent the number of actions that each learner performed on others’ links and concepts

Alice Bob

Carl


But how to create the right visualisations?

Are there simple rules?Simple principles?

Simple and constant solutions?


State of the art

For learners….

Kahn Academy, what a student sees after the pre-test

Model of learner

Gamification element


State of the art

Skill metersGame elements

Good match to mental models

Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, 46(5), 30-32.


State of the art

Teachers


Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012, May). The student activity meter for awareness and self-reflection. In CHI'12 Extended Abstracts on Human Factors in Computing Systems (pp. 869-884). ACM.

Data about many students in an online learning environment.Current focus is red student


Acknowledgements


Interactive surfaces

Software infrastructure user control, scrutability

Interfaces to user model

Acknowledgements

Data mining

Teacher assessment of usefulness (20 participants, most Computer Science)

“Visualization is much more effective at showing the differences between these datasets than statistics. Although the datasets are synthetic, Anscombe’s Quartet demonstrates that looking at the shape of the data is sometimes better than relying on statistical characterizations alone.

“Spence and Garrison …describe a simple plot called the Hertzsprung Russell … [shows] the temperature of stars on the X axis and their magnitude on the Y axis. … It turns out that no automatic analysis method has been able to find the same summarization,[as graphs at right] due to the noise and artifacts on the data such as the vertical bands.


The eye...the window of the soul,is the principal meansby which the central sense can most completely and abundantly appreciatethe infinite works of nature. Leonardo da Vinci (1452 - 1519)

chai: : computer human adapted interaction research group interfaces for learning data...

Documents

group members

lecturer slide

important group work

long term group work

temporal contextual

learning contexts

learning system

classroom learning