New Approach for E-Learning Evaluation Uranchimeg Tudevdagva1, Wolfram Hardt2, Tsoy Evgeny3 and Mikhail Grif3

1Power Engineering School, Mongolian University of Science and Technology, Ulaanbaatar, Mongolia 2Department of Computer Science, Chemnitz University of Technology, Chemnitz, Germany

3Department of Automated Control System, Novosibirsk State Technical University, Novosibirsk, Russia uranchimeg@must.edu.mn

Abstract – This paper deals with a new approach for evaluation of e-learning programs. Based on methods of general measure theory an evaluation model is developed which can be used for assessment of complex logical target structures in context of e-learning programs.

Basic idea of presented method is the embedding of aims to be reached in an adapted logical structure which contains the key and sub targets of evaluation. For such structures a measure theoretical scoring model is developed. It is described how to given observation data, obtained with an adjusted assessment checklist, a score value for evaluation of an e-learning program can be computed.

The presented model is very versatile. By the measure theoretical background the evaluation method becomes open, comprehensible and traceable.

Keywords - evaluation, evaluation model, target structure, assessment checklist, e-learning, e-learning evaluation

I. INTRODUCTION

E-learning and teaching become in context with realization of lifelong learning more and more important for education and training on almost all areas of our life. Under these conditions the development of adapted quality management systems and evaluation methods for e-learning programs is a consequential task. In accordance with the dynamic development of all branches of information technology especially the call for corresponding quantitative evaluation methods becomes always louder.

Evaluation of e-learning tools is a complex task. So far no generally accepted evaluation method exists. The focus of discussion is mainly directed to methodological aspects. For a corresponding survey we refer to Kirkpatrick [3], Phillips [5], Stufflebeam [6], Khan [4] as well as Ruhe, Zumbo [7], for instance. For the European view on e-learning and the assessment of quality we refer to the report [2].

The quantitative models for the assessment of e-learning quality considered usually are additive models. That means, depending on the considered aspects, which are evaluated or measured based on a pre-given scale, a linear function containing corresponding weight factors is used like, e.g.,

= =1

Here denote , > 0, given weight factors for the obtained measure values , = 1,… , , for the considered aspects. The advantage of this formula is easy to use. The disadvantage is that the choice of proper weight factors can be subjective. Moreover, positive evaluation values can be obtained even in such cases if the targets of certain quality aspects have been failed. A possible logical inner structure of target structures remains out of consideration.

By [7] and [8] a measure theoretical evaluation model has been developed which includes the logical structure of an evaluation target. If the evaluator can describe the logical structure of evaluation target then by our model can be scored how the target of an e-learning course has been reached based on corresponding observation data. Here we describe this approach for a special e-learning course.

II. NEW EVALUATION MODEL

A. Target structure To evaluate how an e-learning program has reached a pre-

given target we have to analyze the logical structure of this target. That begins with identification of key targets. Those are targets which have to be reached. If one of the key targets is not reached the main goal is failed. For instance, if we have six key targets, say 1,… , 6, the corresponding main target C canbe written formal as intersection of key targets. It holds

= 1 ∩ 2 ∩ 3 ∩ 4 ∩ 5 ∩ 6.This can be visualized by a series circuit as it is shown by

Fig. 1.

Figure 1. Key target structure

Key targets can consist of several sub or inner targets in the following sense. A key target is reached if at least one of the assigned sub targets is reached. Hence sub targets describe several ways, directions or alternatives on which a key target can be reached. Key targets are unions of sub targets. Key targets which are defined via sub targets can be illustrated by parallel structures. An example is given by Fig. 2. We will come back to this example in Section III.

This study has been supported by a Grant of Schlumberger Foundation.

B1 B2 B3 B4 B5 B6

978-1-4673-1773-3/12/$31.00 ©2013 IEEE

Figure 2. Sub target structure

According Fig. 2 we have

1 = 1 ,8=1 2 = 2 ,6

=1 3 = 3 ,4=1

4 = 4 ,3=1 5 = 5 ,3

=1 6 = 6 ,9=1

where denotes the j-th sub target of key target , =1,… ,6. Hence, the main target C can be written in more general form as

= = =1=1 =1

where r denotes the number of key targets and is the number of sub targets of key target , = 1,… , . B. Measure theoretical background

Evaluating logical structures as above we consider a measure space which is generated by the involved sub targets. This starts with the construction of measure spaces for the sub targets. After that we will consider the corresponding product measure space. This product space will allow a simultaneous consideration of all included sub targets.

Each sub target in sense of our example above is assigned a measure space consisting of three elements (Ωij, Aij, Qij) defined as follows:

1. Ωij = (0), (1) - the set of elementary targets which

can be reached in sense of sub target . (0) stands for: the sub target is reached (has been reached), (1) for: the sub target is not reached (has not been reached).

2. Aij = { ∅, ij, ̅ij, Ωij} - a σ-algebra over Ωij, the set of all subsets of target space Ωij. That is a formal set structure describing the reachable targets with respects to the considered sub target. It holds = (0) and ̅ ij =(1) .

3. - an evaluation measure on the measureable space (Ωij, Aij ) with = and (Āij) = 1−

to any given , 0≤ ≤ 1.

The values and 1 − are scores for that how the targets ij and ̅ij can be reached (have been reached). Values of ≈ 1 mean the target is reached (has been reached) essentially. Values ≈ 0 mean sub target is failed (has been failed) essentially.

The product space: Describing complex target structures with a logical structure like above we have to consider an adapted product measure space. All sub targets can be described together or simultaneously in measure theoretical sense by a corresponding product space (Ω, A, Q) generated by the measure spaces (Ωij, Aij, Qij), = 1,… , , = 1,… , . The set Ω =× =1× =1 Ω contains the = ∑ =1 – dimensional elementary targets = 11,… , 1 1 ,… , 1,… , ,

11 ∈ Ω . The σ-algebra A = ⦻i=1⦻j=1 Aij includes all possible target structures which can be reduced to the sub targets . Finally, = ⦻i=1⦻j=1 is the corresponding product measure. To any given ∈ Aij with = for = 1,… , , = 1,… , , and =×i=1×j=1 this product measure is defined by ( ) = =1=1 = . (1)=1=1

Let be a given key target defined by disjoint sub targets 1,… , such that

= =1

holds, then can be scored by

( ) = 1 − 1 −=1 . (2)

see formula (4) of [8]. Finally, a main target structure

= = =1=1 =1

A11

A12

A13

A14

A15

A16

A17

A18

A21

A22

A23

A24

A25

A26

A31

A32

A33

A34

A41

A42

A43

A51

A52

A53

A61

A62

A63

A64

A65

A66

A67

A68

A69

B1 B2 B3 B4 B6 B5

generated by r disjoint key targets = ⋃ =1 , = 1,… , , is scored by

( ) = 1 − 1 −=1=1 , (3)

see Formula (4) of [8]. This is our main formula for scoring or evaluation of logical target structures. For more details we refer to [8] or [9], for the measure theoretical background we refer to [1].

C. Data collection and processing According to (3) the score or evaluation of a logical target

structure C can be reduced to the scores of included sub targets . Unfortunately, these scores are not given a-priori and we need corresponding sample based estimation methods for these scores.

There exist several possibilities obtaining corresponding sampling data. We can get samples by adapted checklists or questionnaires, by face to face or phone interviews, via online or web based tests. The result of such a questionnaire is a data vector ⃗ = 11 ,… , 1 1 ,… , 1,… , 1 . Here denotes the result or answer to the question with respect to the sub target

. We assume that (0) ≤ ≤ (1) holds to any given bounds (0) and (0) < (1). Then an estimation value for = ( ) is given by

∗ = − (0) (1) − (0) . (4)

If we have a sample of size n to our data vector ⃗, then according relation (4) we get n estimation values ∗(1)( ),… , ∗( )( ) for ( ). Let ∗( ) be the estimation value for according to (4) for the k-th sample element then it holds

∗( )( ) = 1− 1 − ∗( )=1 .=1

By the method of moments we get then as estimation value ∗( ) for ( ) ∗( ) = 1 1− 1 − ∗( )=1 .=1 (5)=1

This is the statistical version of our main formula (3) and our final evaluation formula for a checklist based evaluation.

If we calculate periodically such scores we obtain time series which describe the quality development of an e-learning tool, for instance. Or, we can use the score values for a

comparative consideration to recognize differences, developments or trends.

Analogously, the scores for the key targets can be estimated. As estimation ∗( )( ) for ( ) based on k-th sample element, = 1,… , , we get

∗( )( ) = 1 − 1 − ∗( )=1

and as estimation for ( ) based on whole sample

∗( ) = 1 1 − 1 − ∗( )=1 . =1

III. EXAMPLE Evaluation is as a rule a very complex task. For a

systematic study a corresponding plan is needed. Such a plan could be look as it is demonstrated in Fig. 3.

Figure 3. Evaluation steps

Table I shows the results of a summative evaluation for a e-learning course for master students of Mongolian University of Science and Technology. The checklist has been adapted to the logical structure described in Section II. At questionnaire attended 24 students. The evaluation range for all sub targets has been the set {0, 1, …, 6}. The value 6 means the associated sub target has been completely reached, value 0 means the sub target has been failed. The columns 1 until 24 contain the 24 sampling records. The last row shows the computed estimation values ∗( )( ) for = 1,… ,24 for each sample record. According Formula (5) we get as estimation value ∗( ) for ( ) the value (see Table 1). ∗( ) = 0.9726.

This is a very high value and means that from side of students the course has been reached its targets almost completely. The last two columns of Table 1 present the estimation values ∗ for the scores of sub targets as well as the estimation values ∗( ) for ( ) for the key targets .

We see the more a key target is resolved into sub targets the more positive or better the corresponding key target has been reached in sense of scoring function. That means one has to be

Key targets

SubTargets

Instruments

Data collection

Data process

Outcomes

very carefully with definition of a logical structure and one should avoid to big parallel structures.

IV. CONCLUSIONS The presented at logical structure of a target oriented

evaluation method is an alternative to today frequently applied linear evaluation models. The advantage of our method is that in contrast to additive evaluation approaches no weighting factors are needed.

The evaluation is carried out based on logical structure of considered target. Hence the method becomes more clear, reasonable and more objective. It is embedded into the general measure theory which forms an established and accepted fundament for evaluation of most diverse structures in most different fields of our life.

The method can be applied, e.g., to a periodical evaluation of e-learning tools recognizing corresponding trends, developments or weak sectors. Based on adapted and open target structures this method could be a methodical base for the development of an open national or international assessment or self-assessment framework.

An essential advantage of our approach is that it becomes more objective by the unity of a transparent logical structure and an adapted traceable processing of observation data based on methods of general measure theory. Thereby a well and

responsibly defined logical structure of target to be reached is a basic assumption for the successful application of our method.

The model is applicable too in other evaluation and scoring context if the corresponding target can be described by an adapted logical structure.

REFERENCES [1] Bauer, H. (2001), “Measure and Integration Theory”, Gruyter - de

Gruyter Studies in Mathematics, Berlin. [2] “E-learning quality, Aspects and criteria for evaluation of e-learning in

higher education”. Hoegskoleverket, Swedish National Agency for Higher Education, Report 2008:11 R. Kirkpatrick, D. L. (1959). “Techniques for evaluating training programs”.Journal of ASTD, 11, 1– 13.

[3] Khan, B.K. (2005), “Managing e-learning: Design, delivery, implementation and evaluation”. Hershey, PA: Information Science Publishing.

[4] Phillips, J. J. (1991), “Handbook of training evaluation and measurement methods (2nd ed.)”. Boston: Butterworth-Heinemann.

[5] Stufflebeam, D.L. (2002), “The CIPP Model for Evaluation”. In: Stufflebeam, D.L., Madaus, G.F. and Kellaghan, T.: Evaluation Models. Viewpoints on Educational and Human Services Evaluation. Second Edition, e-Book, New York, pp. 290-317.

[6] Ruhe, V., Zumbo, B.D. (2009), “Evaluation in distance education and e-learning: the unfolding model”. The Guilford press. New York.

[7] Uranchimeg, T., Hardt, W. (2011), “A new evaluation model for eLearning programs”. Technical Report CSR-11-03, Chemnitz University of Technology, Chemnitz, Germany.

[8] Uranchimeg, T., Hardt, W. (2012), “A measure theoretical evaluation model for e-learning programs”. In Proceedings of the IADIS on e-Society 2012, March 10-13, 2012, Berlin, pp. 44-52.