UNIFIED FRAMEWORK FOR ONLINE AND OFFLINE LEARNING ANALYTICS

MAKA ERADZE PHD STUDENT, SCHOOL OF DIGITAL TECHNOLOGIES, RESEARCH FELLOW AT UNIVERSITA NAPOLI FEDERICO II RESEARCH SEMINAR, 11.05.2016

PRESENTER •  Educational background – humanities (teacher of

languages and literatures) - from modern and ancient Greek to interdisciplinary TEL research

•  Experience – 2005 to 2012 – software localizer—>teacher network coordinator (Deer Leap Foundation, MoES of Georgia), eLearning coordinator –(Tbilisi State University, Georgia)

•  Current job – Università di Napoli Federico II, EMMA project, research fellow

•  Research - Unified framework for online and offline learning analytics

MOTIVATION FOR THE RESEARCH Personal •  Personal experience through participation into online fora

(yes, Latin plural for forum - fora J ) •  Personal ethnographic observations on my informal learning

patterns led me to Learning Analytics research •  How? Ask Mart! Professional: •  Interactions and social dimension of learning leading to big

questions like: •  what exactly are we analysing •  What is my research question? •  How are we going to break the concepts down to analyzable

units?

STRUCTURE OF THE PRESENTATION •  Background and statement of the problem •  State of the art •  General methodological framework of the research

•  Literature review

•  Work already done

•  Towards a framework - Mind map and diagram

•  Future research - validation phase •  Applicability

•  Discussion part

BACKGROUND AND STATEMENT OF THE PROBLEM •  The need for this kind of research emerges from the

idea that •  When dealing with data (especially big), we have to

think about our questions first and search for patterns in learning data with specific questions and frameworks in mind. For this, we need to shape our philosophical and methodological standing.

•  We need to think also about the real world interactions and think about the ways to facilitate and connect learning interaction analytics.

•  This research aims at the development of a unified model for offline and online analytics with a purpose of gaining insight into learning interactions.

BACKGROUND AND STATEMENT OF THE PROBLEM

•  It is focused on the development of a methodological framework for the analysis of offline and online interaction patterns and open learning environments.

•  Centered on the idea of the unit of analysis and its appropriateness for learning analytics research and practice, it can be applicable to any kind of interaction analysis (learning analytics). •  How is the issue of the units of analysis presented in different

disciplines (and learning analytics?) •  What are the philosophical concepts and considerations

behind the issue of the unit of analysis? •  How can we make use of xAPI statements and frame this

discourse in a philosophically and methodologically appropriate way?

RESEARCH QUESTIONS •  How can we analyze learning interactions using different

frameworks, theories, philosophies (and pedagogic scenarios)?

•  What is the appropriate unit of analysis for learning interaction analytics in online and offline settings?

GENERAL THEORETICAL AND METHODOLOGICAL FRAMEWORK OF THE RESEARCH Theoretically

•  Constructivist approach •  This research does not use particular theoretical framework. It

rather reviews interdisciplinary approaches and creates a meta language to redescribe the methodological framework for learning analytics research

Methodologically •  Design based research •  Mainly based on the extensive literature review and inquiry

stage •  Based on the literature review artifacts will be created that

redescribe and place the issue of the unit of analysis within the LA research

•  This will serve to create a detailed ontology and validate it

STATE OF THE ART - INTERACTIONS •  In order to actually define how can we make use of interaction

traces and data and locate its analytical framework, it is necessary to define the concept of interaction as it is to educational sciences - TEL.

•  Interaction functions in a different way in virtual learning environments where its must convey, mimic or recreate the traditional communication patterns or build a new type of learning ambience.

•  Underlining the term - learning interaction. I share Wagner’ definition of interaction as follows “reciprocal events that require at least two objects and two actions. Interactions occur when these objects and events mutually influence each other” (Wagner, 1994).

STATE OF THE ART - INTERACTIONS •  According to Anderson (Anderson 2003), interaction online

(or offline) can happen human and inanimate actors.

•  Moore’s theory of Three Types of Interaction •  To Moor learner-instructor interaction is a four-stage support

of the learner:

1.  Designing the content, maintaining motivation, self-direction 2.  Making presentations (student/teacher) 3.  Practicing and applying acquired competences. 4.  Counseling and encouragement of learners according their

levels of progress. The third type of interaction – learner-learner for Moor is a new dimension in distance education

STATE OF THE ART-DEFINITIONS •  Learning analytics is the measurement, collection,

analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.

•  LA as a technology-enhanced learning (TEL) research area that focuses on the development of methods for analyzing and detecting patterns within data collected from educational settings, and leverages those methods to support the learning experience. (Chatti et al 2015)

STATE OF THE ART- LA, ITS CHALLENGES AND FUTURE DIRECTIONS Learning analytics is inherently interdisciplinary

• Operates on the verge of many different fields

Still in its infancy •  It still has to establish itself as field with its

own traditions and theoretical frameworks

•  Two systematic literature reviews found:

•  Nistor et al(2015): The main issue with LA research is the frequent lack of an explicit theoretical framework from educational perspective. The authors call on the educational and psychological theories for significant progress of upcoming LA research

•  Gasevic et al(2015): researchers from educational sciences tend to publish more mature research and show more rigor

•  Fortunately, the field is evolving quickly and one measure of this growth is the increasing breadth of methodologies being applied to learning‐related data sources (Pardo et al, 2014)

•  Big learning analytics t h a t c o m e f r o m heterogeneous sources

•  o p e n l e a r n i n g environments (MOOCs), with data coming from outside of platform.

•  Mobile learning analytics •  Context modeling •  Privacy-aware analytics.

•  Personalized learning analytics. •  Life-long learner modeling •  Learning analytics for open

assessment. •  Embedded Learning analytics •  Learning analyt ics design

patterns. •  Learning analytics evaluation.

Chatti, M. A., Lukarov, V., Thüs, H., Muslim, A., Yousef, A. M. F., Wahid, U., ... & Schroeder, U. (2014). Learning Analytics: Challenges and Future Research Directions.

FACTORS Different factors are involved in LA

Greller and Draschler

From Chatti

LIT.REVIEW -UNIT OF ANALYSIS •  Interdisciplinary approach:

•  When choosing what to analyze, how to analysis, what to take into account, how to operationalize questions, what methodology to use, what discourse to take in in any field, unit of analysis is at the same time a problematic but also an important decision to make.

•  Learning Interaction Analysis needs its own unit of analysis too.

•  influenced by the work of Matusov who has given an extensive account on this issue from different perspectives.

•  Building on his understanding on the unit of analysis we will also bring arguments from: •  Socio-cultural research, education, human-computer

interaction, computer science, sociology, technology-enhanced learning, psychology and other fields.

LIT.REVIEW -UNIT OF ANALYSIS •  This is not a methodological decision only, but also a

political, and of course, philosophical. •  Mostly, two types of approaches are used for the

discussion on the unit of analysis: holistic and reductionist.

•  Holistic approach believes that the unit has to express a whole phenomenon, while reductionist approach breaks the unit of analysis down to its most fine grain unit.

•  Matusov thinks, that the unit of analysis changes with the aim of the research.

•  To Vigotsky it is a “self-contained” “unified smallest whole”.

UNIT OF ANALYSIS •  Matusov takes the socio-cultural approach to the issue

and bases his discussion mainly on psychologist tradition.

•  According to Matusov one must be careful when searching for a general, universal unit of analysis, because it always is shaped by the purpose of the researcher and the subject of study.

•  The construct “unit of analysis” is mainly used for the methodological critique of others’ research

UNIT OF ANALYSIS The discussion about the unit of analysis is also a paradigmatic discussion – exact sciences vs social sciences with

•  “The exact sciences constitute a monologic form of knowledge: the intellect contemplates a thing and expounds upon it. There is only one subject here-cognizing (contemplating) and speaking (expounding). In opposition to the subject there is only a voiceless thing. Any object of knowledge (including man) can be perceived and cognized as a thing. But a subject as such cannot be perceived and studied as a thing, for as a subject it cannot, while remaining a subject, become voiceless, and, consequently, cognition of it can only be dialogic.” (Bakhtin in Matusov)

UNIT OF ANALYSIS •  According to Lefstein, while replacing the reductive units of

analysis with more holistic ones, the issue of manageability arises.

•  Sometimes it can be hard to draw the boundaries, what is the exact phenomenon to be expressed holistically. He argues, that “interactional structures of three or more turns is a methodologically expedient and theoretically sound unit for the analysis of classroom discourse”.

•  To him, unit of analysis is a “discourse move” and dialogical. Though, Lefstein also suggests that aside from the appropriate phenomenon, the unit of analysis has to be expanded to sequence.

UNIT OF ANALYSIS Stahl from his side discusses the issue of unit of analysis in cognition –

•  concepts (Plato), •  mental and material objects (Descartes) (and relationship between them) •  observable physical objects (empiricism), •  mind’s structuring categorization efforts (Kant).

All of the approaches dealt with the inner functions of an individual mind. •  Hegel entered the discussion with a larger unit of analysis – historically, socially and

culturally determined. •  Marx views the main unit of analysis as social movements, class conflicts and

transformations of economic systems. •  Hegel’s philosophy shaped three mainstream schools of thought – Marx (critical

social theory), Heidegger (existential phenomenology) and Wittgenstein (linguistic analysis).

•  To Stahl, these three main directions influence how the CSCL units of analysis are formed: For Heidegger the unit of analysis was the man with unified experience of being-in-the-world.

•  Wingenstein’s shifted unit of analysis from mental meanings to interpersonal communications in the context of getting something done together. (Stahl)

UNIT OF ANALYSIS - LATOUR •  Latour thinks the issue brings us to

•  what phenomenon is worth studying and ‘worth’ for what. According to a sociocultural framework, this question cannot be addressed within purely methodological considerations as it both involves conceptual (paradigmatic) debates within the academic field and is shaped by practical needs and financial (and political and institutional) pressure outside of the field and science in general (Latour, 1987 in Matusov).

•  For Latour and his Action-Network Theory, the main unit of

analysis is embedded in the relationship between the actions.

UNIT OF ANALYSIS

•  Stahl believes that Engestrom is the one taking the unit of analysis to the whole activity system.

•  Engeström in his “Learning By Expanding” writes that the insertion of cultural artifacts into human actions was revolutionary in that the basic unit of analysis now overcame the split between the Cartesian individual and the untouchable societal structure.” After this the understanding of an individual could no longer happen without his or her cultural means; and the understanding of a society was bound to individuals who use and produce artifacts. He believes that the limitation of the first generation was that the unit of analysis remained individually focused. The object-oriented and artifact-mediated collective activity system is the prime unit of analysis in cultural-historical studies of human conduct;

•  But to Stahl’s understanding Engestrom’s theory is not interested in group knowledge building but rather with organizational management of the group, Influenced by Marx, theory tries to see societal issues in the making.

UNIT OF ANALYSIS •  Stahl believes that even in distributed cognition, which

deals with group-cognitive phenomena, mostly socio-technical systems and highly developed artifacts are analyzed.

•  Suthers (2010) with his uptake framework proposes that the event is the core if we want analyze data and understand which interactions lead to learning. The uptake framework suggests that interaction is relational by its nature, so the most important unit of analysis is not isolated acts, but rather relationships (based, again, on Latour 2005) between acts.

UNIT OF ANALYSIS - KAPTELININ In activity-theory

•  activity, in a broad sense, is an interaction of the actor (e.g., a human being) with the world. The interaction, according to activity theory terminology, is described as a process relating the subject (S) and the object (O). A common way to represent activity is “S ⇔ O.

•  According to Kaptelinin, activity is different from other types of interaction by two key aspects: (a) subjects of activities have needs, which should be met through an interaction with the world, and (b) activities and their subjects mutually determine one another; or, more generally, activities are generative forces that transform both subjects and objects. Subjects have needs.

KAPTELININ – ACTIVITY THEORY •  Kaptelinin presents the Activity as •  a “unit of life” of a material subject existing in the

objective world. •  Subjects have their own needs and to survive, have to

engage in activities, that is, interact with the worldy objects fulfill their needs

•  assumptions of the social nature of human mind and inseparability of human mind and activity. (Kaptelinin, Activity theory).

•  Activity is of fundamental importance to psychology because of its special function.

•  It places the Subject into the objective reality and transforms the reality into a form of subjectivity (Leontiev, 1978)

PAAVOLA AND HAIKAINEN •  According to Paavola and Haikaninen, in trialogues, the

interaction through “shared objects” that are in the process of being developed is emphasized.

•  The objects of inquiry can be respresented by the knowledge artefacts, practices, ideas, models, representations, etc. but perceived as something to be collaboratively developed.

•  These objects are important in the interaction and it is an interaction between subjects, other subjects and “objects” but not between the subjects.

•  This means that the things are being developed collaboratively and they do not repeat the existing knowledge

XAPI Event-driven analytics

<Actor (learner)> <verb> <object>, with <result>, in <context>

REVIEW: XAPI

●  The Experience API is a service that allows for statements of experience (typically learning experiences, but could be any experience) to be delivered to and stored securely in a Learning Record Store

●  The Experience API is dependent on Learning Activity Providers to create and track learning.

●  Learning Activity Provider is a software object that communicates with the LRS to record information about the learning experience.

●  Learning activity is a unit of instruction, experience or performance that has to be tracked.

●  A Statement consists of <Actor (learner)> <verb> <object>, with <result>, in <context> to track an aspect of a learning experience.

REVIEW: XAPI ●  It repeats the structure of the sentence in practically any

language and is human readable •  Actor data is a unique information that describes a

specific subject, such as a student or group of students. •  Verb data classifies the type of activity the actor

participated in and often links to a human readable description of the event.

•  Object data will link to an artifact that is typically a byproduct of or related to the activity

XAPI ●  It is much more, than just a simple sentence-like structure. The

context is very important and it makes possible to relate one activity to another* •  Quoting Verbert, Suthers and Rosen, Kevan and Ryan believe that

the issue of variety of educational data taxonomies and also the distributed learning events collection challenges can be solved by xAPI*

●  Some researchers believe that it is aligned with constructivist approach and activity theory

●  xAPI has a lot of potential for The xAPI offers a renewed opportunity to research, develop, and explore theories involving learning beyond academia’s digital environments.

Kevan, J. M., & Ryan, P. R. Experience API: Flexible, Decentralized and Activity-Centric Data Collection.Technology, Knowledge and Learning, 1-7.

WORK ALREADY DONE/PUBLICATIONS •  Eradze, M. Kai, P., Laanpere, M.: (2013) Analysing Learning Flows in

Digital Learning Ecosystems, KMEL 2013, ICWL2013, Taiwan, Springer (LNCS), October 2013

•  The first paper that opens the discussion on the unit of analysis, with detailed background on the interactions, literature review

•  Eradze M., Laanpere M.: (2014) Interrelation Between Pedagogical Design and Learning Interaction Patterns in Different Virtual Learning Environments, HCII 2014, Greece, Springer (LNCS) June 2014

•  Three different platforms studied with CoI framework and coding template (Two MOOC platforms, Moodle and Edufeedr)

•  Eradze, M., Valjataga, T., Laanpere M.: (2014) Observing the use of e-

textbooks in the classroom: towards “Offline” Learning Analytics, FeT workshop, ICWL 2014, Tallinn, Estonia, Springer (LNCS)

•  Observations in the classroom, tried to document offline interactions with an mobile app and xAPI type of statements.

PUBLICATION IN PROGRESS Unit of Analysis in Learning Analytics, Maka Eradze, Mart Laanpere •  The article overviews the state of the art of unit of analysis

issue in educational research, learning analytics and offers a conceptual framework based on the idea of an appropriate unit of analysis for learning interaction analysis.

•  Aiming for - Journal of Computing in Higher Education - Special Issue on Advancing Research on Open Education.

MIND MAP – REDESCRIBING THE UNIT OF ANALYSIS/XAPI

DIAGRAM - REDESCRIBING THE UNIT OF ANALYSIS/XAPI

FUTURE RESEARCH  - VALIDATION PHASE •  Based on the inquiry stage and the artefacts, a detailed

ontology will be developed •  A single semantic structure that can map teacher

intentions – what they are planning to do and what actually is taking place

•  The ontology will be mapped to specific scenarios

•  Together with teachers validate the ontology with participatory approach

APPLICABILITY •  This ontology will be applicable to any context and to any

data •  It will be flexible and can be applied to different

pedagogic scenarios, philosophies, methodologies

•  With this ontology we will be able to grasp the whole structures of interactions and analyse it holistically

I know this is not yet a defense and am not yet on Ithaca and I am no Odysseus (rather Penelope), but this is a journey I am enjoying so:

EXPRESSION OF GRATITUDE •  To all in the School of Digital Technologies for

understanding my absent presence or rather present absence

•  To Mart for being a challenging supervisor and….for scaffolding/challenging/believing in me J

•  To Kai for giving valuable, critical and helpful feedback

DISCUSSION Need feedback J

•  Desperately!

•  For thoughts and feedback coming up later: maka@tlu.ee