learning analytics for communities of inquiry

Learning Analytics for Communities of Inquiry

Vitomir Kovanovic1,3 Dragan Gasevic1,2 Marek Hatala3

[email protected] [email protected] [email protected]

1School of Informatics 2Moray House School of EducationUniversity of Edinburgh University of Edinburgh

Edinburgh, United Kingdom Edinburgh, United Kingdom

3School of Interactive Arts and TechnologySimon Fraser UniversityBurnaby, BC, Canada

March 17, 2015Marist College,

Poughkeepsie, NY, USA

Introduction

Ph.D research Overview

High-level

The goal of my PhD research is to expand theknowldge about inquiry-based digital learningthrough the development of novel learning

analytics models.

Low-level

I’m looking at how use of text analytics, socialnetwork analysis and trace data clustering canbe used to improve the understanding of theCommunity of Inquiry (CoI) model of distance

education.

Vitomir Kovanovic (UoE) Learning Analytics for Communities of Inquiry March 17, 2015, Marist College 1 / 22

Background Inquiry-based online learning

Asynchronous online discussions

• Primary means of social interaction in onlinecommunities,

• Frequently used for various types of educationdelivery (e.g., blended, F2F, distance),

• Well supported by the social-constructivistpedagogies,

• Can serve multiple purposes (e.g., social support,community building, QA, student-instructorinteraction, social learning),

• Widely used, often not appropriately,

• Require substantial effort from instructors.


Background Community of Inquiry (CoI) Framework

Community of Inquiry (CoI) framework

Community of Inquiry is a conceptual framework outlying the important constructsthat define worthwhile educational experience in distance education setting [3].

• Social presence: relationships and socialclimate in a community.

• Cognitive presence: phases of cognitiveengagement and knowledge construction.

• Teaching presence: instructional roleduring social learning.



Community of Inquiry (CoI) framework

Community of Inquiry is a conceptual framework outlying the important constructsthat define worthwhile educational experience in distance education setting [3].

CoI framework is:

• Extensively researched and validated,

• Widely used in distance educationresearch.



Cognitive Presence

Cognitive Presence

“an extent to which the participants in any particular configuration of acommunity of inquiry are able to construct meaning through sustainedcommunication.” [3, p .89]

Four phases of cognitive presence:

1 Triggering event: Some issue, dilemma or problem is identified.

2 Exploration: Students move between private world of reflection and sharedworld of social knowledge construction.

3 Integration: Students filter irrelevant information and synthesize newknowledge.

4 Resolution: Students analyze practical applicability, test differenthypotheses, and start a new learning cycle.



CoI coding instrument

• Based on quantitative content analysis (QCA) [11],• Defines three coding schemes for each of the presences,• Use of whole message as unit of analysis,• Look for particular indicators of different sociocognitive processes,• Additional heuristics,

• Code-up: When a message clearly displays indicators of several phases it iscoded to the latest phase, and

• Code-down: When it is not clear which phase is reflected, code to the earliestphase.

• Requires expertise with coding instrument and domain knowledge.



CoI challenges

• Practical challenges, related to its adoption by instructors,

• Theoretical challenges, related to current understanding of socialinteractions, human agency and technology use.



CoI practical challenges

• Content analysis instrument:• Time consuming, labor intensive manual message coding,• Hard to scale, typically used for small sample studies,• Typically used for research purposes after courses are over.

As a result, content analysis (including CoI) had almost no impact oneducational practice [2].

• Survey instrument:• Removes the need for manual coding,• Administered at the end of the course,• Does not allow any feedback for the duration of the course.

• A need for more proactive use of available data,

• Explanations for observed levels of thee presences,

• Suggestions and guidelines for instructors to direct their instructionalinterventions.



CoI theoretical challenges

Although CoI framework is rooted in social constructivism, there is barely anymention of social networks and social capital in current CoI literature:

• How does student position in social network reflect on the development ofsocial and cognitive presence and ultimately on learning outcomes?

• What types of interactions promote the development of students’ socialcapital within communities of inquiry?

Human agency and self-regulation of learning are also not frequently discussed:

• How does students use available technology to learn in inquiry-based courses?

• How does different technology use affects development of cognitive presence,attainment of learning objectives and course performance?

CoI adoption outside typical small-class DE courses is not widely described:

• What are the challenges of CoI use in larger courses (e.g., MOOCs)?

• Under what conditions can CoI be used in large online courses?


Research questions

Research questions

• How to enable for easier and more scalable quantitative content analysis inaccordance with community of inquiry coding schemes?

• Which social processes, and to what extent, are indicative of the developmentof the social capital in communities of inquiry?

• What is the relationship between students’ social capital and thedevelopment of cognitive presence?

• With respect to the development of cognitive presence, are central positionswithin student social networks beneficial or not?

• What are the main technology use profiles withing communities of inquiry?

• How does different technology-use profile affect the development of cognitivepresence and attainment of learning objectives?


Proposed approach

Proposed approach

Proposed approach

Development of learning analytics for communities of inquiry.

In general,

Build supervised and unsupervised analytical models to address theexisting challenges with Community of Inquiry framework.


Proposed approach

Proposed approach

Proposed approach

Development of learning analytics for communities of inquiry.

More precisely,

1 Develop a text analytics system for automated message coding inaccordance with cognitive presence coding scheme,

2 Develop a predictive model for understanding the relationshipbetween social network capital and social presence,

3 Identify technology-use profiles based on trace data clustering andexamine their relationship with development of cognitive presenceand learning outcomes.


Proposed approach

Text analytics system

• Development of a novel text classifier for CoI message coding,• There have been few attempts to automate CoI coding process [12, 1],• Their accuracy not sufficient for practical adoption,• Based on popular classification techniques and surface text features.

• Warrants development of novel text classification method that is moresuitable for a given problem,

• Provides a more detailed operationalization of cognitive presence codingscheme.


Proposed approach

Text analytics system: typical approach

• Pose a problem as a traditional multi-class supervised classification problem,

• Use of surface-level features (N-grams),

• Use of part-of-speech (POS) tags,

• Use of combinations of N-grams and POS tags (back-off N-grams),

• Use of grammatical dependency triplets,

• Thread position features,

• Standard classification algorithms (SVM, Naive Bayes),

• Cross validation a preferred method of validation accuracy and parametersoptimization,

• In order to be done correctly, requires nesting of cross-validation.

• Does not capture code-up and code-down rules,

• Issues with quoting of messages, and

• Issues with rare classes (i.e., resolution phase).


Proposed approach

Text analytics system: better approach

• Take into the account the class (i.e., phase) of the previous message,

• Take into the account the phase of cognitive development of a given student,and

• Take into the account code-up and code-down rules,• Code-up needs a feature overriding,• Code-down needs an estimate of classification certainty,


Proposed approach

Challenges

• What features of discourse are good indicators that can be used for buildinga classification algorithm?

• How to support and leverage the cyclic nature of cognitive presence?

• How to model feature precedence?

• What features are indicative of the different phases of cognitive presence?

• How to build automated text classification system that has accuracy highenough to warrant its practical use?

• How to automate as much of message coding as possible while preservingsufficiently high classification accuracy?

• How to develop a system that is not a domain dependent?• How to speed up train of the system in new domain?


Proposed approach

Social analytics system

• Development of learning analytics based on social network analysis of studentinteractions,

• Understand the role of social presence on the development of social capital,

• Understand the link between social network position and the development ofcognitive presence.


Proposed approach

Technology-use profiles

• Development of learning analytics based on student trace-data clustering,

• Understand the different ways of educational technology-use,

• Understand the link between different technology-use profiles anddevelopment of cognitive presence,

• Understand the link between different technology-use profiles and success inthe course.


Proposed approach

Challenges

• How to develop a system that is not a domain dependent?• How to speed up train of the system in new domain?

• How to identify technology-use profiles?

• What forms of social presence are indicative of student social capital?

• What trace data to use for profile identification?


Methodology

Methodology

1 Literature review• Educational research• Content analysis in distance

education• Community of Inquiry• Social network analysis• Data mining• Text mining• Online discussion and newsgroup

classification• Clustering• Graph-based data mining• Quantitative research methods

2 Data collection• Selected Topics in Software

Engineering course• UoE E-learning and digital cultures

MOOC

3 Development of classification algorithm• Feature extraction• Algorithm development• Validation

4 Technology-use profiling• Trace data conversion and feature

extraction• Cluster discovery and validation• Examination of relationships with

cognitive presence and learningoutcomes

5 Social network analysis• Social graph extraction• Longitudinal analysis of relationships

between network centrality andlearning outcomes


Current Progress

Current progress

1 Literature review• Educational research• Content analysis in distance

education• Community of Inquiry• Social network analysis• Data mining• Text mining• Online discussion and newsgroup

classification• Clustering• Graph-based data mining• Quantitative research methods

2 Data collection• Selected Topics in Software

Engineering course• UoE E-learning and digital cultures

MOOC

3 Development of classification algorithm• Feature extraction• Algorithm development• Validation

4 Technology-use profiling• Trace data conversion and feature

extraction• Cluster discovery and validation• Examination of relationships with

cognitive presence and learningoutcomes

5 Social network analysis• Social graph extraction• Longitudinal analysis of relationships

between network centrality andlearning outcomes


Current Progress

Publications

• Preliminary classifier implementation [6]

• End of course CoI SNA analysis [7]

• MOOC Research analysis [4]

• MOOC News analysis [10]

• Time-on-task estimation challenges [9]

• Analysis of cognitive presence linguistic properties [5]

• JLA doctoral proposal paper [8]


Expected contributions

Benefits of the proposed research:

• Based on the empirical evidence, expand current state of understanding ofdistance education by investigating learners’ interactions with information(i.e., content), technology, instructors and other learners.

• Make a foundation on which improvements on the current educationalresearch and practice could be developed.

• Provide mode detailed operationalization of the CoI coding instrument by themeans of text analytics,

• Provide basis for easier adoption of CoI model by the researchers andpractitioners,


Thank you

References I

Stephen Corich, Kinshuk Hunt, and Lynn Hunt. “Computerised Content Analysis for Measuring Critical

Thinking within Discussion Forums”. In: Journal of e-Learning and Knowledge Society 2.1 (2012).

Roisin Donnelly and John Gardner. “Content analysis of computer conferencing transcripts”. In:

Interactive Learning Environments 19.4 (2011), pp. 303–315.

D. Randy Garrison, Terry Anderson, and Walter Archer. “Critical Inquiry in a Text-Based Environment:

Computer Conferencing in Higher Education”. In: The Internet and Higher Education 2.2–3 (1999),pp. 87–105.

Dragan Gasevic et al. “Where is Research on Massive Open Online Courses Headed? - A data analysis

of the MOOC Research Initiative”. In: The International Review of Research in Open and DistanceLearning submitted (2014).

Srecko Joksimovic et al. “Psychological characteristics in cognitive presence of communities of inquiry:

A linguistic analysis of online discussions”. In: The Internet and Higher Education 22 (2014), pp. 1–10.

Vitomir Kovanovic et al. “Automated Content Analysis of Online Discussion Transcripts”. In:

Proceedings of the 2014 Learning Analytics and Knowledge (LAK) Conference Workshop on MachineLearning and Learning Analytics. 2014.

References II

Vitomir Kovanovic et al. “What is the source of social capital? The association between social network

position and social presence in communities of inquiry”. In: The first International Workshop onGraph-based Educational Datamining (G-EDM) 2014. 2014.

Vitomir Kovanovic and Dragan Gasevic. “Learning Analytics for Communities of Inquiry”. In: Journal

of Learning Analytics in-press (2014).

Vitomir Kovanovic et al. “Penetrating the Black Box of Time-on-task Estimation”. In: Proceedings of

the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015). Best paperaward nominee. 2015.

Vitomir Kovanovic et al. “What public media reveals about MOOCs?” In: British Journal of

Educational Technology accepted (2014).

Klaus H. Krippendorff. Content Analysis: An Introduction to Its Methodology. 0th ed. SagePublications, 2003.

Tom McKlin et al. “Cognitive presence in web-based learning: A content analysis of students’ online

discussions”. In: IT Forum. Vol. 60. 2002.

learning analytics for communities of inquiry

Data & Analytics