SAMOS Project: A data analysis model based on control charts to monitor online learning processes

IEMAE

Angel A. Juan, Thanasis Daradoumis, Santi Caballé, Fatos XhafaDep. of Computer Science, Multimedia and Telecommunication

Open University of Catalonia (Spain)

{ajuanp, adaradoumis, scaballe, fxhafa}@uoc.edu

July 2nd, 2008Barcelona, Spain

This work is partially supported by the Innovation Vice-rectorate of the Open University of Catalonia under grant IN-PID0702

1. Introduction

� E-learning models can provide high quality educational

offerings at the same time they allow for convenient and

flexible learning environments without space, distance or

time restrictions (Seufert et al., 2002)

� Nowadays, most universities and colleges worldwide are

using some learning management system (LMS) –such

as Moodle, Sakai or WebCT/Blackboard– as part of the

technical resources they make available to their students

and instructors.

� The instructor’s role is moving from one related to a

knowledge transmission agent to another related to a

specialist agent who designs the course, guides, assists

and supervises the student's learning process (Simonson

et al., 2003; Engelbrecht and Harding, 2005)

� Instructors need information systems and tools that help

them monitoring the e-learning process in a similar way

engineers and managers need efficient information

systems and tools that help them to control, in real time,

business or service processes.

2. Need for Monitoring E-learning Processes

� Any type of distance education program

presents higher dropout rates than more

conventional programs (Sweet, 1986) � It is

necessary that instructors provide just-in-time

guidance and assistance to students

� Communication among students need to be

facilitated and promoted by instructors

� Monitoring students’ and groups’ activity and

performance can help to understand students’

interactions and anticipate potential problems:

abandonment, group malfunction, etc.

(Dillenbourg, 1999; Daradoumis et al., 2006)

� Monitoring reports can be used by instructors

to easily track down the learners’ online

behavior and group’s activity and performance

at specific milestones, gather feedback from

the learners and scaffold groups. Some of

these reports can also be employed as

feedback for students

3. Quick Review of Existing Research

� Instructors participating in online learning environments have

very little support by integrated means and tools to monitor and

evaluate students’ activity (Jerman et al., 2001; Zumbach et al.,

2002)

� Rosenkrans (2000) states the necessity of using assessment

tools that monitor students’ progress as a way to empower

instructors’ role in online environments and also as a way to

provide a feedback to students

� Rada (1998) is the first author who proposed the use of

statistical quality control methods and tools to monitoring the

student transactions in online learning environments

� Other authors (Simoff and Maher, 2000; Juan et al., 2008;

Gaudioso et al., 2008) discuss the lack of online data analysis

and analytical processing features of Web-based educational

environments and propose the use of data analysis, data mining

and visualization as an integral part of these environments.

� Recently, an emergent research area called Educational Data

Mining is focused on the application of data mining techniques to

discover information and knowledge from log files data registered

in course management systems (Romero and Ventura, 2007;

Romero et al., 2008)

4. Main Contribution of our approach

� The main goal is to develop, implement and test a

practical information system that allows instructors

in most e-learning environment to efficiently monitor

students’ and groups’ activity and performance in e-

learning courses

� Also, our work can serve as a conceptual

framework that can be used for tracking groups’

and individuals’ activity in any collaborative e-

learning courses that:

a) Span over one or more semesters,

b) Involve a large number of groups and

students that need to carry out a continuous

and intensive collaborative activity, and

c) Need to analyze and evaluate specific

situations at different granularity levels, e.g.:

at the instructor level, the course manager

level and the student level

� Three-layer structure

� Objective: the set of monitoring tools should be independent of the LMS

employed, so that they could be used in combination with any of the most

popular e-learning platforms (Moodle, Sakai, WebCT, etc.)

� The monitoring tools will make use of the server log files and/or academic

database records provided by the LMS to generate the graph-based reports

� Then, an e-mail containing personalized reports will be automatically sent by the

system to each addressee, either instructor or student (“push” strategy)

5. Layer Diagram of SAMOS

1. Students develop different activities (events) in the LMS spaces. Some of them have only

an activity dimension while others have also an academic performance dimension.

2. Events generated by students are registered in log files at the LMS server or in records at

the database server.

3. After processing log files and database records, personalized students and instructors

reports are generated and sent to the e-mail server.

6. Functionality of the Model (1/2)

4. The e-mail server sends the performance reports to each student (personalized feedback)

5. Instructors receive personalized reports regarding both students’ academic activity and

performance. These reports will allow them to easily identify those students who are “at

risk”, i.e.: students with low activity levels and students which are underperforming

6. This way, instructors can offer them “just-in-time” guidance and support

6. Functionality of the Model (2/2)

� We chose to generate periodical (weekly, etc.) monitoring reports with the aim

to show a small set of graphs that were easily and quickly understood by

instructors

� For flexibility and efficiency reasons, we decided that these graphs should

contain only critical information about groups’ and students’ activity &

performance levels

� They should provide instructors with a rough classification for each kind of

entities –groups and students– according to their corresponding activity &

performance levels

� These graphs should also provide information about the historical evolution of

each student activity and performance with respect to the rest of the class

� Having these considerations in mind, we designed the following four charts:

a) Students’ classification according to their activity level,

b) Activity control chart for each student,

c) Students’ classification according to their performance level, and

d) Performance control chart for each student

7. Designing the Charts

� Students’ classification according to their activity level (weekly, Inst.):

• A scatter plot of X = “Number of events generated by student i during this (current)

week” and Y = “Number of events generated by student i during an average week”

• It also includes the vertical lines defined by the first, second and third quartiles of X

8. The SAMOS Monitoring Reports (1/4)

� Activity control chart for each student (weekly, Inst.):

• It is a monitoring chart which shows the weekly evolution of each student academic

activity levels (represented by circular dots connected by line segments)

• The graph also contains quartiles bands

8. The SAMOS Monitoring Reports (2/4)

� Students’ classification based on their performance (at each test, Inst.):

• It is a scatter plot of X = “Score obtained by student i in this (last) test” and Y =

“Average score obtained by student i in the past tests (including the last one)”

8. The SAMOS Monitoring Reports (3/4)

� Performance control chart for each student (at each test, Inst. & Student):

• It is a monitoring chart which shows the evolution of each student academic

performance

• It also includes: (a) the updated average student’s score, and (b) quartiles bands

8. The SAMOS Monitoring Reports (4/4)

� We have discussed the convenience of adapting process

control charts, extensively used in quality engineering, to

the e-learning arena

� We have focused in two major related problems in

distance learning courses: (a) assure that students will

reach a satisfactory performance level in the learning

process, and (b) avoid high dropout rates caused by the

lack of adequate support and guidance

� Monitoring students’ activity and performance is needed

in order to identify non-participating and underperforming

students in “real time”

� The model presented here can be easily adapted and

used in different LMS and in most online courses from

any knowledge area

� It is expected that SAMOS will add value to the

instructors’ role as designers and supervisors of the

learning process and will allow them to offer flexible and

just-in-time guidance and assistance to students

9. Conclusions

� Juan, A.; Daradoumis, T.; Faulin, J.; Xhafa, F. (under review): “A Data Analysis Model

based on Control Charts to Monitor Online Learning Processes”. Int. Journal of

Business Intelligence and Data Mining. ISSN: 1743-8187

� Juan, A.; Daradoumis, A.; Xhafa, F.; Caballe, S.; Faulin, J. (2009): Monitoring and

Assessment in Online Collaborative Environments: Emergent Computational

Technologies for E-Learning Support. IGI Global, Hershey, Pennsylvania, USA.

� Juan, A.; Daradoumis, T.; Faulin, J.; Xhafa, F. (in press): “SAMOS: A Model for

Monitoring Students’ and Groups’ Activity in Collaborative e-Learning“. International Journal of Learning Technology. ISSN: 1477-8386

� Daradoumis, A.; Faulin, J.; Juan, A.; Martinez, F.; Rodriguez, I.; Xhafa, F. (in press): “CRM Applied to Higher Education: Developing an e-Monitoring System to Improve

Relationships in e-Learning Environments”. International Journal of Services

Technology and Management. ISSN: 1460-6720

� Caballe, S.; Juan, A.; Xhafa, F. (2008): “Supporting Effective Monitoring and

Knowledge Building in Online Collaborative Learning Environments”. In Proceedings of

the First World Summit on the Knowledge Society (Springer Lecture Notes in

Computer Science). Athens, Greece, September 24-28.

� Daradoumis, A.; Faulin, J.; Juan, A.; Martinez, F.; Rodriguez, I.; Xhafa, F. (2008):

“Expanding the Customer Relationship Management Scope to the Non-Profit Organizations: an Analysis Focused on the E-University Domain”. In Proceedings of

the IADIS International Conference, e-Commerce 2008. Amsterdam, Netherlands,

July, 25-27.

� Juan, Α.; Daradoumis, Τ.; Faulin, J.; Xhafa, F. (2008): “Developing an Information

System for Monitoring Student’s Activity in Online Collaborative Learning”. In

Proceedings of the 2nd International Conference on Complex, Intelligent and Software

Intensive Systems, pp. 270-275. IEEE Computer Society. ISBN: 0-7695-3109-1. Barcelona, Spain, March 4-7.

10. Related Work

SAMOS Project: A data analysis model based on control charts to monitor online learning processes

IEMAE

Angel A. Juan, Thanasis Daradoumis, Santi Caballé, Fatos XhafaDep. of Computer Science, Multimedia and Telecommunication

Open University of Catalonia (Spain)

{ajuanp, adaradoumis, scaballe, fxhafa}@uoc.edu

July 2nd, 2008Barcelona, Spain

This work is partially supported by the Innovation Vice-rectorate of the Open University of Catalonia under grant IN-PID0702

Thank You!Thank You!