eindhoven university of technology master personalized e … · vaessen, d. award date: 2009 link...

Eindhoven University of Technology

MASTER

Personalized E-learningshorten the length of an E-learning program

Vaessen, D.

Award date:2009

Link to publication

DisclaimerThis document contains a student thesis (bachelor's or master's), as authored by a student at Eindhoven University of Technology. Studenttheses are made available in the TU/e repository upon obtaining the required degree. The grade received is not published on the documentas presented in the repository. The required complexity or quality of research of student theses may vary by program, and the requiredminimum study period may vary in duration.

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

https://research.tue.nl/en/studentTheses/aff341df-4a3b-45ab-be9b-2c9652acb453

Personalized E-Learning 1

EINDHOVEN UNIVERSITY OF TECHNOLOGY

Department of Mathematics and Computer Science

Personalized E-Learning

by

Dirk Vaessen

Shorten the length of an E-learning program

Supervisors:

Prof. dr. P.M.E. De Bra (Tu/e)

Drs. F.H.A. van Buul CISSP (InfoSecure)

Eindhoven, January 2009


Abstract

This graduation project provides a conceptual model for the design of a personalized E-Learning

module in AHA! (Adaptive Hypermedia Architecture). AHA! is an adaptive web-based system for

creating adaptive websites. Before explaining how to create the personalized E-Learning module, the

differences between AHA! and other technologies are explained and what the advantages of AHA! in

comparison with the other technologies are. There are different types and methods for adaptation.

These are described in this thesis and the most suitable adaptation method for a personalized E-

learning module is further analyzed before the implementation is explained. A fully described

analysis of the time benefits for each adaptation method is given. For the company InfoSecure that

designs E-Learning modules an existing module is adapted in a personalized module with the help of

AHA! and the differences between those two products are explained. The results of the adaptive and

the original module are analyzed. Not only the results, but also how to analyze the results is fully

described in this thesis.


Foreword

After finishing my bachelor on Computer Science and Engineering at the Eindhoven University of

Technology, I decided to continue with the Business Information Systems (BIS) master also at the

department of Mathematics and Computer Science. The emphasis on the relation between computer

sciences and industrial engineering and management sciences in this master appealed to me.

Creating the AHA! application itself connects greatly with my bachelor education. Analyzing what

adaptation to use within the created application, so that it best fits InfoSecure’s business goals is an

aspect where my master education comes into play. Analyzing the results and proposing a suitable

plan to use these results in such a way that InfoSecure can create even better E-learning programs in

the future is also one of the subjects where my master education was very helpful.

I hope that this thesis will be a contribution for InfoSecure and their E-Learning modules and a

contribution in their awareness of new methodologies and techniques that can be used to create E-

learning modules. I also hope that the reader will be challenged to read this report and will

understand the decisions made in the different steps.

Before I wish you good luck with reading this thesis, as a tradition, I like to thank some people who

made it possible to graduate. First of all I like to thank my supervisors, Drs. Frans van Buul and prof.

dr. Paul De Bra for their support, feedback, and suggestions during the project. I also want to thank

the director of InfoSecure, Melle Beverwijk, for giving me the chance to work at his company in

Leusden, which was a great experience for me.

At last I would like to thank all colleagues, both InfoSecure employees and fellow students for their

feedback and their time.

January 2009,

Dirk Vaessen


Contents and Terms

Summarized Contents List

Abstract ................................................................................................................................................... 5

Foreword ................................................................................................................................................. 7

Contents and Terms ................................................................................................................................ 9

1 Introduction ................................................................................................................................... 15

2 E-learning ....................................................................................................................................... 17

3 Technologies .................................................................................................................................. 20

4 Adaptation ..................................................................................................................................... 28

5 Introduction to Awareness Module .............................................................................................. 54

6 Module in AHA! ............................................................................................................................. 64

7 Extracting Test Data ...................................................................................................................... 82

8 Testing Adaptivity .......................................................................................................................... 88

9 Conclusion ..................................................................................................................................... 93

10 References ................................................................................................................................. 94

Appendices ............................................................................................................................................ 95

Extended Contents List

Abstract ................................................................................................................................................... 5

Foreword ................................................................................................................................................. 7

Contents and Terms ................................................................................................................................ 9

Summarized Contents List ................................................................................................................... 9

Extended Contents List ........................................................................................................................ 9

List of Figures and Diagrams.............................................................................................................. 13

Terms and Abbreviations .................................................................................................................. 14

1 Introduction ................................................................................................................................... 15

1.1 General Introduction ............................................................................................................. 15

1.2 Assignment Goals .................................................................................................................. 16

1.2.1 Goals for Adaptive E-Learning Module ......................................................................... 16

1.2.2 Goals for Final Thesis ..................................................................................................... 16

2 E-learning ....................................................................................................................................... 17

2.1 E-learning Programs InfoSecure ............................................................................................ 17

2.1.1 Introduction Program .................................................................................................... 18

2.1.2 Follow-up Program ........................................................................................................ 18

2.1.3 Special Topics Learning Programs ................................................................................. 18

2.1.4 Training for IT Professionals .......................................................................................... 18

2.2 Adaptation InfoSecure Program ............................................................................................ 18

3 Technologies .................................................................................................................................. 20


3.1 AHA!....................................................................................................................................... 20

3.1.1 AHA! Architecture ......................................................................................................... 21

3.2 SCORM ................................................................................................................................... 21

3.2.1 Organization of SCORM ................................................................................................. 22

3.3 SCORM to AHA! ..................................................................................................................... 23

3.4 Improve SCORM Code ........................................................................................................... 24

3.4.1 Sequencing and Navigation ........................................................................................... 25

3.4.2 Selftest Implementation ................................................................................................ 26

4 Adaptation ..................................................................................................................................... 28

4.1 Adaptation Types................................................................................................................... 28

4.1.1 Content Adaptation ....................................................................................................... 28

4.1.2 Link Adaptation ............................................................................................................. 28

4.1.3 Presentation Adaptation ............................................................................................... 29

4.1.4 Information Adaptation ................................................................................................. 29

4.2 Adaptation Rules ................................................................................................................... 29

4.3 Adaptation Methods ............................................................................................................. 30

4.3.1 Adaptation by Pretest Questions .................................................................................. 30

4.3.2 Adaptation regarding to HR ........................................................................................... 30

4.3.2.1 Diplomas and Certificates .......................................................................................... 30

4.3.2.2 Function ..................................................................................................................... 31

4.3.2.3 Department ............................................................................................................... 31

4.4 Analysis .................................................................................................................................. 31

4.4.1 Example Course ............................................................................................................. 31

4.4.2 Pretest Analysis ............................................................................................................. 32

4.4.2.1 Success Percentage ................................................................................................... 32

4.4.2.2 Success Percentage together with Pretest Correlation Percentage ......................... 33

4.4.2.3 Selftest Adaptation .................................................................................................... 37

4.4.3 HR Analysis .................................................................................................................... 37

4.4.3.1 Data Mining ............................................................................................................... 38

4.4.3.2 Breakeven Percentage ............................................................................................... 40

4.5 Scenarios and Time Benefits ................................................................................................. 43

4.5.1 Scenario’s ...................................................................................................................... 43

Scenario 1: Best Case ................................................................................................................. 43

Scenario 2: Best Case 2 .............................................................................................................. 44

Scenario 3: Average Case .......................................................................................................... 46

Scenario 4: Worst Case .............................................................................................................. 49

Scenario 5: Worst Case 2 ........................................................................................................... 51

4.5.2 Time Benefits ................................................................................................................. 51


4.6 Conclusion ............................................................................................................................. 53

5 Introduction to Awareness Module .............................................................................................. 54

5.1 Module Introduction ............................................................................................................. 54

5.2 Module Duration ................................................................................................................... 54

5.3 Time Distribution ................................................................................................................... 55

5.4 Module Construction ............................................................................................................. 55

5.5 Module Adaptation Locations ............................................................................................... 57

5.5.1 Explanation .................................................................................................................... 57

5.5.2 What is Information Security......................................................................................... 57

5.5.3 Status within the Company ........................................................................................... 57

5.5.4 About the Golden Rules ................................................................................................ 57

5.5.5 Selftest ........................................................................................................................... 58

5.5.6 Conclusion ..................................................................................................................... 58

5.5.7 Relevant Links and Contacts .......................................................................................... 58

5.6 Module Adaptation Techniques ............................................................................................ 58

5.6.1 Student Profile ............................................................................................................... 58

5.6.2 Adaptation based on HR Information ........................................................................... 58

5.6.2.1 Accepted Diplomas and Certificates ......................................................................... 59

5.6.3 Adaptation based on Pretest ......................................................................................... 59

5.6.4 Adaptation Results ........................................................................................................ 61

5.6.5 Scenario’s ...................................................................................................................... 62

5.6.6 Time Benefits ................................................................................................................. 63

5.6.7 Conclusion ..................................................................................................................... 63

6 Module in AHA! ............................................................................................................................. 64

6.1 Process of the Adaptive Module ........................................................................................... 64

6.2 Conceptual Structure............................................................................................................. 66

6.2.1 Design Concept Structure .............................................................................................. 66

6.2.2 Creating Concepts ......................................................................................................... 68

6.2.2.1 Menu Structure ......................................................................................................... 69

6.2.2.2 Attributes ................................................................................................................... 69

6.2.2.3 Example .aha file ....................................................................................................... 69

6.2.3 Concept Relationships ................................................................................................... 70

6.2.3.1 Used Attributes.......................................................................................................... 70

6.2.3.2 Adaptation Rules ....................................................................................................... 71

6.2.4 Implementing Concept Structure .................................................................................. 71

6.2.4.1 Hierarchy and Suitability ........................................................................................... 71

6.2.4.2 Access ........................................................................................................................ 74

6.3 Write Pages ........................................................................................................................... 75


6.3.1 Write Standard Pages .................................................................................................... 75

6.3.2 Write Adapting Pages .................................................................................................... 76

6.3.2.1 Pretest Questions ...................................................................................................... 76

6.3.2.2 About the Golden Rules ............................................................................................ 77

6.3.2.3 Selftest ....................................................................................................................... 78

6.4 Look and Feel ......................................................................................................................... 78

6.5 Authoring Tools ..................................................................................................................... 80

6.5.1 Graph Author ................................................................................................................. 80

6.5.2 Concept Editor ............................................................................................................... 81

6.5.3 Form Editor .................................................................................................................... 81

6.5.4 Test Editor ..................................................................................................................... 81

6.6 Other Methods ...................................................................................................................... 81

7 Extracting Test Data ...................................................................................................................... 82

7.1 Analyzing the AHA! Logs ........................................................................................................ 82

7.1.1 Correcting Data .............................................................................................................. 83

7.1.2 Data Group 1 vs. Data Group 2 ...................................................................................... 83

7.1.3 Pretest Ratio .................................................................................................................. 86

7.2 Analyzing the AHA! Profile Logs ............................................................................................ 87

7.3 Testing the Significance ......................................................................................................... 87

7.3.1 F-test Two-Sample for Variances ................................................................................... 87

7.3.2 T-test Two-Sample ......................................................................................................... 87

8 Testing Adaptivity .......................................................................................................................... 88

8.1 Test Group ............................................................................................................................. 88

8.2 Test Group Analysis ............................................................................................................... 88

8.3 Pretest Results ....................................................................................................................... 89

8.3.1.1 Persons that succeeded for selftest first time........................................................... 89

8.3.1.2 Persons that didn’t succeed for selftest first time .................................................... 90

8.3.1.3 Success and Pre-test Question Correctness Percentage ........................................... 90

8.3.1.4 Selftest Adjustments ................................................................................................. 90

8.3.2 Statistical Proof of Results ............................................................................................. 90

8.3.3 Overall Conclusion ......................................................................................................... 91

8.4 HR Results .............................................................................................................................. 92

9 Conclusion ..................................................................................................................................... 93

10 References ................................................................................................................................. 94

Appendices ............................................................................................................................................ 95

Appendix A: Selftest.xhtml ................................................................................................................ 95


List of Figures and Diagrams

Table 1 Terms and Abbreviations .......................................................................................................... 14

Table 2 Fraction current imsmanifest.xml............................................................................................. 24

Table 3 Implement Sequencing and Navigation.................................................................................... 26

Table 4 Selftest implementation ........................................................................................................... 26

Table 5 Pretest maximum percentages ................................................................................................. 36

Table 6 Minimum selftest succeed percentage .................................................................................... 37

Table 7 Pretest scorings percentage ..................................................................................................... 38

Table 8 Pretest scorings percentage with certain diploma ................................................................... 38

Table 9 Pretest scorings percentage with certain certificate ................................................................ 39

Table 10 Pretest scorings percentage with certain function ................................................................ 39

Table 11 Updated diploma scorings percentage ................................................................................... 39

Table 12 Scorings percentage with diploma and certification taken into account ............................... 40

Table 13 Optimized scorings percentages ............................................................................................. 40

Table 14 Average duration pretest adapted course scenario 2 ............................................................ 45


Table 16 HR adaptation part 1 .............................................................................................................. 48

Table 17 HR adaptation part 2 .............................................................................................................. 49



Table 20 Time benefits per scenario ..................................................................................................... 51

Table 21 Individual time benefits .......................................................................................................... 52

Table 22 Duration Introduction Module ............................................................................................... 55

Table 23 Accepted Diplomas ................................................................................................................. 59

Table 24 Accepted Certificates .............................................................................................................. 59

Table 25 Time Benefits Scenario's ......................................................................................................... 63

Table 26 Initializing table for current module ....................................................................................... 65

Table 27 Fraction of question1.xhtml ................................................................................................... 77

Table 28 Example access_John Doe.xml ............................................................................................... 82

Table 29 Table with extra time column ................................................................................................. 83

Table 30 Timings from all test persons ................................................................................................. 84

Table 31 Timings from correct test persons .......................................................................................... 84

Table 32 Average time pretest .............................................................................................................. 86

Table 33 Average time golden rules ...................................................................................................... 86

Table 34 Average time selftest .............................................................................................................. 87

Figure 1 AHA! Architecture (De Bra, et al., 2003) ................................................................................. 21

Figure 2 SCORM Bookshelf (Advanced Distributed Learning (ADL), 2006a) ......................................... 23

Figure 3 Percentages that lead to time profit for subject 3 .................................................................. 35

Figure 4 Time Distribution Introduction Module .................................................................................. 55

Figure 5 Example Basic Module (Information) Security ........................................................................ 56

Figure 6 Example aha file ...................................................................................................................... 70

Figure 7 Frame Structure ....................................................................................................................... 79

Figure 8 Screenshot Module AHA! ........................................................................................................ 80

Diagram 1 E-learning Programs of InfoSecure ...................................................................................... 17

Diagram 2 Non adapted example course .............................................................................................. 32

Diagram 3 Pretest adapted example course ......................................................................................... 32

Diagram 4 Process Pretest Question 3 .................................................................................................. 34

Diagram 5 Scenario 1 pretest adapted course ...................................................................................... 43

Diagram 6 Scenario 1 HR adapted course ............................................................................................. 44


Diagram 7 Scenario 2b possible pretest adapted course ...................................................................... 44

Diagram 8 Scenario 2b possible HR adapted course ............................................................................. 44

Diagram 9 Best case scenario 3 HR adapted ......................................................................................... 47

Diagram 10 Page sequence ................................................................................................................... 56

Diagram 11 Example adaptive page sequence ...................................................................................... 60

Diagram 12 Example 2 adaptive page sequence ................................................................................... 60

Diagram 13 Adaptation Process ............................................................................................................ 62

Diagram 14 Adaptive process generic module ...................................................................................... 65

Terms and Abbreviations

The following terms and abbreviations are used in this thesis.

Table 1 Terms and Abbreviations

Term Description

ADL Advanced Distributed Learning

Developer and implementer of learning technologies across the Department of

Defense (DoD).

AHA! Adaptive Hypermedia Architecture

BIS Business Information Systems

Master program at the Eindhoven University of Technology

HR Human Resources

LMS Learning management system

SCORM Sharable Content Object Reference Model

SQL Common querying language to query results from a relation database and to

update information in this database.

TEL Technology Enhanced Learning

Tu/e Eindhoven University of Technology

XML eXtensible Mark-up Language

Mark-up language commonly used on the Internet to exchange

information that can be irregular.


1 Introduction

In this chapter a general introduction (see chapter 1.1) to the company InfoSecure and their E-

learning modules is given, together with the assignment goals of this final thesis (see chapter 1.2).

1.1 General Introduction

This thesis is the result of the graduation period of Dirk Vaessen, carried out at InfoSecure in Leusden

as a completion of the “Business Information Systems” master at the Eindhoven University of

Technology.

InfoSecure was founded in 1999. Since

then InfoSecure has built up a

worldwide clientele.

In addition to the head office in The

Netherlands, InfoSecure has offices in

Belgium, Germany, Great Britain and

Scandinavia. Additionally they also have

partners in Switzerland, Croatia, China,

Japan and Canada.

The company provides solutions in more than 80 countries and in 22 different languages. The unique

method used has proven itself in practice and is an international best seller.

InfoSecure Group is highly experienced in the field of awareness training programs, knowledge

testing and risk/compliance review.

Within the field of Awareness & Training

InfoSecure has multiple E-learning

modules described in chapter 2.1. The

module that will be adapted is the

Introduction Program, in this program

the users are made aware of the

potential risks that the organization can

face and the measures to be established

in respect to their activities.


1.2 Assignment Goals

Before explaining everything in detail the assignment goals of this final thesis are given. The goals are

given in two subchapters. First the goals of the adaptive module in chapter 1.2.1., afterwards the

goals for this final thesis in general in chapter 1.2.2.

1.2.1 Goals for Adaptive E-Learning Module

The most important goal for the Adaptive E-Learning module is to create a better learning

experience, with the same high standard of the original E-Learning module, in a shorter time period.

Such a module has a few characteristics:

Time benefit

This is most important for the new module because eventually this module hits the market and as

you all know: time is money. If this module saves for instances 10 minutes in comparison with the old

module and has the same high standard, this will be a great improvement. A company with 60.000

employees that work for an average of 50 euro an hour will save half a million euro with the help of

this new adaptive module.

Creating a time benefit without endangering the high standard of the original E-learning, will be done

with the help of adaptation. This adaptation will make sure that the module is personalized.

Personalization

The personalization is done with the help of adaptation. The students that follow the new adaptive

course will only be presented the information that is suitable for them. F.i. information they are

already familiar to will be skipped and only information that is important for the specific student will

be presented. This way the duration of the module is shortened.

Better learning experience

A better learning experience is automatically created for the student, because it will take him less

time to learn the same. No familiar information to the student is presented, but only the information

that is suitable. This creates a much better learning experience for the student.

1.2.2 Goals for Final Thesis

To create such an adaptive module that suits all the aspects described in the previous subchapter is

the main goal. How it is created and which technologies are used is fully described in this thesis.

Another goal is to explain how this adaptive module is build with the help of AHA! Also clearly explain

how to use AHA! on a conceptual level for another adaptive module is another important goal in this

thesis. Analyzing the results of the AHA! adaptive module, and explain how, is also a goal in this

thesis.


2 E-learning

Before explaining the E-learning modules of InfoSecure, E

Nowadays everybody talks about Technology Enhanced Learning (TEL).

technology enhanced learning spread very broad an

nature of this evolving research field. Hence, the definition of TEL must be as broad and general as

possible in order to capture all aspects:

technical innovations (also improving efficiency and cost effectiveness) for learning practices,

regarding individuals and organizations, independent of time, place and pace. The field of TEL

therefore describes the support of any learning activity through technology

elearning or eLearning) is the delivery of a learning, training or education program by electronic

means. E-learning involves the use of a computer or electronic device (e.g. a mobile phone) in some

way to provide training, educatio

Learning, the complete education program is on the computer (or other electronic device).

computer is only used for specific parts of a program it

Because nowadays every employee has a computer at work or at home,

more and more popular. InfoSecure develops E

computer. No other electronic dev

programs of InfoSecure are explained in the next subchapter.

2.1 E-learning Programs InfoSecure

As described in the introduction InfoSecure

Besides e-learning programs InfoSecure

consultancy, risk assessment etc.

InfoSecure has multiple E-learning

subjects described in the next diagram.

Diagram

learning modules of InfoSecure, E-learning itself needs to be exp

talks about Technology Enhanced Learning (TEL). The existing definitions for

technology enhanced learning spread very broad and change continuously due to the dynamic


possible in order to capture all aspects: Technology enhanced learning has the goal to provide socio

ovations (also improving efficiency and cost effectiveness) for learning practices,


therefore describes the support of any learning activity through technology. E-

he delivery of a learning, training or education program by electronic

learning involves the use of a computer or electronic device (e.g. a mobile phone) in some

way to provide training, educational or learning material (Der08). E-Learning is a part of TEL. With E

Learning, the complete education program is on the computer (or other electronic device).

used for specific parts of a program it is still called TEL, but not E

Because nowadays every employee has a computer at work or at home, TEL and e

more and more popular. InfoSecure develops E-learning programs that involve the use of a

computer. No other electronic devices are possible or necessary for following the

of InfoSecure are explained in the next subchapter.

InfoSecure

As described in the introduction InfoSecure plays a big role in the information security sector.

InfoSecure offers lots of solutions for your company, like workshops,

consultancy, risk assessment etc. During my graduation I will only focus on the E

learning programs for different end-users, that implement all kind of

subjects described in the next diagram.

Diagram 1 E-learning Programs of InfoSecure

17

learning itself needs to be explained.

The existing definitions for

d change continuously due to the dynamic


Technology enhanced learning has the goal to provide socio-

ovations (also improving efficiency and cost effectiveness) for learning practices,


-learning (also called

he delivery of a learning, training or education program by electronic

learning involves the use of a computer or electronic device (e.g. a mobile phone) in some

a part of TEL. With E-

Learning, the complete education program is on the computer (or other electronic device). When the

is still called TEL, but not E-Learning.

TEL and e-learning become

learning programs that involve the use of a

ices are possible or necessary for following the programs. The

in the information security sector.

offers lots of solutions for your company, like workshops,

on the E-learning programs.

users, that implement all kind of


2.1.1 Introduction Program

The InfoSecure Awareness concept is based on the Modular Training approach to different target

groups. It starts with an Introduction program for every end-user, as workshops and/or as e-learning.

The solution is used in combination with 6-7 business film clips, learning material and selftests, all

corresponding with Client’s selected Golden Rules/Information Security best practices. In this report

the introduction program made for the Dutch company KPN is used as the subject for adaptation.

KPN selected 9 golden rules, but more about this specific module is in Chapter 5.

Dedicated programs can be built for the target group Management (normally shorter program) and

for example programs for Mobile workers, IT-Professionals and persons handling sensitive

information.

2.1.2 Follow-up Program

In subsequent years a follow-up program can be implemented for every end-user, as workshops

and/or as e-learning. Also here on the Modular Training approach to different target groups is used.

The solution is an extension to the introduction program. Mostly summaries are used from the

Special Topic Learning modules.

Also this program is used in combination with 6-7 business film clips, learning material and selftests,

all corresponding with Client’s selected Golden Rules/Information Security best practices.

2.1.3 Special Topics Learning Programs

Special Topics are developed to learn more in depth about special topic subjects. These programs can

be used to give additional training to those target groups who need more instruction in the subject.

For example mobile working need more instruction about the topics “Loss of Laptops & PDAs” and

“Mobile working”. The target group handling sensitive information can for example be trained more

in “Data Classification”, “Privacy” and “Working with 3rd parties”.

Also this program is used in combination with 3-4 business film clips, learning material and selftests,

all corresponding with Client’s selected Golden Rules/Information Security best practices.

2.1.4 Training for IT Professionals

InfoSecure has developed a training / awareness learning program for information security for IT

Professionals. The training in e-Learning concept content consists of modules on 2 levels of

education; 5 modules for basic training (each 15 minutes) and 6 modules for advanced training (each

30-45 minutes).

Modules are available for Security Essentials, Security Management, Critical Business Applications,

Computer Installations, Networks and Systems Development.

The information in the program is based on ISO standards, other best practices and the in public

domain available ISF Standard of Good Practice v.2005.

2.2 Adaptation InfoSecure Program

As described above InfoSecure has lots of E-learning programs that are all especially made for clients.

In the assignment goals is described that shortening the time duration is one of the main goals of

adaptation. Therefore the program “Training for IT Professionals” is most suitable for adaptation.


This module is by far the most time consuming program and therefore a lot of time profit can be

gained. This is a very technical program, which is very suitable for adaptation.

After an analysis by InfoSecure of his own market group, it was concluded that the clients that

bought this program were not (yet) interested in an adapted version. The clients that bought (or are

going to buy) the introduction program were very enthusiast about an adapted version, and

therefore the switch from “Training for IT Professionals” to “Introduction Program” for commercial

reasons was made.

This program takes far less time than the previous program, but it is expected that still a time benefit

can be gained, albeit a smaller one. The exact module is named “Basic module (information)

Security” for the company KPN. How this module is adapted is described in detail in chapter 5.


3 Technologies

E-Learning becomes more and more popular by the years, but there is still no standard format for E-

learning content. The adaptive E-learning module created in this thesis is done with the help of AHA!

(see chapter 3.1). The E-learning modules of InfoSecure are SCORM compatible. This is a common

and widely used standard for E-learning (see chapter 3.2). How to use SCORM content in AHA! is

explained in chapter 3.3. In chapter 3.4 some improvement points for the current SCORM code of

InfoSecure are given.

3.1 AHA!

AHA! is developed at the Eindhoven University of Technology. After some initial experimental

versions AHA! was released as version 1.0 in 2000. AHA! excelled in the area of simplicity. AHA! has

since evolved into a much more powerful system (version 2.0, 3.0 and soon 4.0), but new versions

maintain that basic simplicity.

• A user model based on concepts: Each time you visit a page in an AHA! application the name

of the page is passed to the adaptation engine which updates the user model. A user model

consists of concepts that have attributes. A typical example of an AHA! action is that visiting

a page may increase a knowledge attribute for (the concept corresponding to) that page. This

knowledge update may propagate to the knowledge attribute of other concepts, perhaps

corresponding to a section or chapter of a textbook. In AHA! a concept can have arbitrarily

many attributes of types Boolean, integer or string.

• Adaptive link hiding or link annotation: The suitability of link destinations (pages) is

determined by an author-defined requirement. This is a (Boolean) expression using arbitrary

user model values. The requirements can express the common prerequisite relationships

between concepts but can be used for any other condition that can be expressed through

such a Boolean expression. When a page is generated, links marked as conditional (using the

link class “conditional”) are displayed differently depending on the suitability of the link

destination. If the expression is true the link is shown in blue (unvisited) or purple (visited),

and when the expression is false the link is shown in black, and not underlined. This results in

hiding the unsuitable or undesired links. The color scheme can also be altered by the end-

user to make all links visible, in different colors.

• Conditional inclusion of fragments: Like for the links to pages the author can also associate a

requirement with fragments in a page. This is done through an <if> tag, with one or two

fragments, enclosed by a <block> tag. If the expression is true the first fragment is shown to

the user, otherwise the second (optional) fragment is shown. This can be used to include

prerequisite explanations, or any other piece of content. Fragments can be external objects,

represented through the <object> tag. Such objects can themselves also be associated with

concepts and accessing them triggers user model updates just like for page accesses.

AHA! is delivered as open source software, implemented entirely in Java, and works with the Java-

based webserver Tomcat and with Java servlets. You need recent versions of Tomcat and of the Java

SDK to make AHA! work. On the browser side you should use recent versions (of for instance Mozilla

Firefox or Microsoft Internet Explorer) to ensure full support of (X)HTML and HTTP.


3.1.1 AHA! Architecture

Figure 1 AHA! Architecture (De Bra, et al., 2003)

The overall architecture of AHA! (see Figure 1) shows that AHA! consists of java servlets that serve

pages from external WWW servers or from the local file system. These servlets interact with the

DM/AM and the UM. A request to a page triggers adaptation rules that perform UM updates. When

UM is updated the requested page is parsed to perform the conditional inclusion of fragments. That

inclusion is based on the new state of UM.

AHA! stores DM/AM and UM (of all users) either as XML files or in a mySQL database. The choice

between these two is made by the Manager who configures AHA!.

AHA! applications mainly consist of a set of concepts, some of which are linked to pages or objects

(or fragments). Concepts can be used to represent topics of the application domain, e.g. subjects to

be studied in a course. In AHA! the author of an application can associate any number of (named)

attributes with a concept. Some attributes have a meaning for the system, like access (a Boolean

attribute that temporarily becomes true when a page is accessed), some have meaning for the

author (and user), like knowledge or interest, and some have meaning for both, like visited

(determining the link color). Since AHA! uses an overlay user model, all attributes of concepts in

DM/AM also appear in UM.

The adaptation rules define how the user model is updated. When the user accesses a page (or an

object included in a page) the rules associated with the access attribute are triggered.

More detailed information about AHA! and how to build an adaptive module in AHA! is given in

chapter 6.

3.2 SCORM

Sharable Content Object Reference Model (SCORM) is a specification of the Advanced Distributed

Learning (ADL) Initiative, which comes out of the Office of the United States Secretary of Defense.

SCORM is a collection of standards and specifications for web-based e-learning. It defines

communications between client side content and a host system called the run-time environment

(commonly a function of a learning management system (LMS)). SCORM also defines how content

may be packaged into a transferable ZIP file.

The essence of SCORM is that any content that conforms to the SCORM specifications will work with

any SCORM conformant LMS. SCORM operates behind the scenes to make things compatible.

Basically SCORM governs two things: packaging content and exchanging data at runtime.


Packaging content determines how a piece of content should be delivered in a physical sense. At the

core of SCORM packaging is a document titled the "imsmanifest.xml". This file contains every piece of

information required by the LMS to import and launch content without human intervention. This

manifest file contains XML that describes the structure of a course both from a learner’s perspective

and from a physical file system perspective. Questions like, "Which document should be launched?"

and "What is the name of this content?" are answered by this document (Advanced Distributed

Learning (ADL), 2006).

Runtime communication, or data exchange, specifies how the content ”talks” to the LMS while the

content is actually playing. This is the part of the equation described as delivery and tracking. There

are two major components to this communication. First, the content has to "find" the LMS. Once

the content has found it, it can then communicate through a series of "get" and "set" calls and an

associated vocabulary. Conceptually, these are things like "request the learner’s name" and "tell the

LMS that the learner scored 95% on this test." Based on the available SCORM vocabulary, many rich

interactive experiences can be communicated to the LMS (Advanced Distributed Learning (ADL),

2006b).

3.2.1 Organization of SCORM

SCORM is a collection, integration and harmonization of specifications and standards that have been

bundled into a collection of “technical books.” Nearly all of the specifications and guidelines are

taken from other organizations. These technical books are presently grouped under three main

topics: the “Run-time Environment (RTE)”, the “Content Aggregation Model (CAM)”, and

“Sequencing and Navigation (SN).”

Of the many organizations working on specifications related to e-learning, there are four in particular

that are key to SCORM. ADL encourages active participation in one or more of these organizations in

support of future specification development.

• Alliance of Remote Instructional Authoring & Distribution Networks for Europe (ARIADNE)

(http://www.ariadne-eu.org/)

• Aviation Industry CBT Committee (AICC) (http://www.aicc.org/)

• Institute of Electrical and Electronics Engineers (IEEE) Learning Technology Standards

Committee (LTSC) (http://ieeeltsc.org/)

• IMS Global Learning Consortium, Inc. (http://www.imsglobal.org/).


Figure 2 SCORM Bookshelf (Advanced Distributed Learning (ADL), 2006a)

The Run-Time Environment specifies how content should behave once it has been launched by the

LMS. The Content Aggregation Model specifies how you should package your content so that it can

be imported into an LMS. This involves creating XML files that an LMS can read and learn everything

it needs to know about your content. SCORM also describes a “Sequencing and Navigation” model

for the dynamic presentation of content based on learner needs. While these various SCORM

books summarized in Figure 2 can stand-alone, there are areas of overlap. For instance, while the

RTE book focuses primarily on communication between content and LMSs, it frequently refers to

the different types of content objects conducting that communication: Sharable Content Objects

(SCOs). More details about SCOs are found in the CAM book. Similarly, while the SN book covers

the details of SCORM sequencing and navigation, the RTE book deals with content delivery and

gives high-level information on how an LMS determines which piece of content to deliver at any

given time.

3.3 SCORM to AHA!

The department of Computer Science at the University of Cordoba developed an Upload SCORM tool

for AHA! (Cristóbal Romero Morales, 2005). With the help of this program, which is still in a beta

phase, the SCORM zip file can be uploaded and the result is a complete new course in AHA!

consisting of the uploaded SCORM data. All you need to enter is the course name, which will already

be the AHA! Web-tree directory where course content is going to be extracted. Unfortunately this

upload program was of no use for the specific SCORM modules of InfoSecure, because this code was

not the exact SCORM code the upload program had expected.

As described in the previous subchapter the SCORM content consists of a ZIP-file, which contains a

descriptor file (imsmanifest.xml) where content organization and resources are described and

referenced. AHA! contents are organized in a root directory (only one in a course), but this directory

only contains the course resources. Usually, this directory already contains course configuration files,

introduction and registration files, but the information about organization and relationship between

different concepts are managed by the AHA! system itself. An AHA! course has a “Concept List”

which links each concept to a resource, and AHA! concepts are organized as a hierarchy. Because of

these similarities between SCORM content and AHA! the upload program creates a “Concept List”

using the imsmanifest.xml file as a source.

The problem with the InfoSecure SCORM code is that it is not really the ideal SCORM code. SCORM is

a collection of standards and specifications for web-based e-learning. Even if the SCORM code applies

to these standards and specifications, the code can still be very confusing. The current SCORM code


consists of a ZIP-file which contains the mandatory imsmanifest file with most of the content

organization and resource information. But this file is not used fully to the abilities of SCORM. A

small fraction of this file is given in the next table.

<manifest>

…

<organizations default="FT02240">

<organization identifier="FT02240">

<title>Computer Installations Basic</title>

<item identifier='ItemFT02240' isvisible='true'>



<title>Introduction</title>

<item identifier='ItemFT02242' isvisible='true' identifierref='SCOFT02242'>


</item>

</item>

…

</item>

</organization>

</organizations>

<resources>

<resource identifier="SCOFT02242" type="webcontent"

href="SCOs/F10_RAL_Wrapper.htm?LaunchURL=FT02242.htm%26WindowDims=height=712,width=1014"

adlcp:scormtype="sco">

<metadata>

<schema>ADL SCORM</schema>

<schemaversion>1.2</schemaversion>

<adlcp:location>FT02242.xml</adlcp:location>

</metadata>

</resource>

…

</resources>

</manifest>

Table 2 Fraction current imsmanifest.xml

Without going in too much detail and complete explanation of this code. It is cleaner SCORM code if

only one unique item with one title exist in the organization, which is linked (via the identifierref) to a

resource. But the problem with creating the concept list with the help of the upload program is the

href of the resource. Instead of a single htm(l) file with proper html (without JavaScript) code, it is

linked to “SCOs/F10_RAL_Wrapper.htm?LaunchURL=FT02242.htm%26WindowDims=height=712,width=1014”

and this will not work in the concept list in AHA!.

What happens in the current code is that the htm file (F10_RAL_Wrapper.htm) has to launch another

htm file (FT02242.htm) with the help of JavaScript. This file launches again with a bunch of JavaScript

code all the subpages and sequence order for this subject. All the subpages are in htm, but instead of

proper htm code it is all javascript with htm code added in. If these pages where changed in proper

html pages and the sequence and navigation of the pages is done with the SCORM standard as will

be described in the next subchapter, the upload program would be able to create the AHA! program

automatically. Because the upload program was of no use with this code, the E-learning modules

were all manually converted to AHA! as described in chapter 6. To create the adaptive E-learning

modules manually, instead of changing the SCORM code and use the upload program has two

reasons. The first reason is a better understanding of AHA!, while the program is completely build

from scratch. The second reason is the upload program itself. This program is still in a beta phase,

and even with the improved SCORM code it is not guaranteed the conversion to AHA! will go

flawlessly.

3.4 Improve SCORM Code

The current code of the InfoSecure modules does not use all the SCORM possibilities fully, which can

be very helpful. As explained in the previous subchapter the actual pages have the extension .htm,


but are nothing more than a bunch of JavaScripts with the html code inside the JavaScript. This

should not be the case to keep a better overview. Some other improvement points are given in the

next subchapters.

3.4.1 Sequencing and Navigation

First the terms sequencing and navigation are explained and afterwards an impression of the desired

implementation is given.

Sequencing

In summary, SCORM Sequencing depends on: a defined structure of learning activities, the Activity

Tree; a defined sequencing strategy, the Sequencing Definition Model; and the application of defined

behavior to external and system triggered events, SCORM Sequencing Behaviors.

By default, if no sequencing and navigation prescription is defined, a learner may choose any content

item at will. Adding specific prescriptions can alter this default behavior. For example, adding a flow

prescription to the items in the content organization will direct the LMS to guide the navigation in

the order defined by the organization tree. More complex adaptive sequencing can be based on the

completion status of certain learning resources or on more complex computation of user preferences

or assessment results(Advanced Distributed Learning (ADL), 2006c).

Past versions of SCORM provided no specific sequencing capabilities, effectively allowing only pure

free play, because it is a difficult and complex subject that required more time to come up with

workable solutions. There are many, and often divergent, requirements in the learning design

community. No approach has been found to solve all possible use cases. However, the approach used

in SCORM, which is based on the IMS SS Specification [5], is flexible enough to allow a wide variety of

learning and instructional design approaches.

Navigation/Presentation

Navigation controls are user interface devices that provide the means for a learner to indicate

the desire to navigate away from the Current Activity in a particular manner. SCORM requires

that an LMS provides, at a minimum, navigation controls that trigger Continue, Previous, and

Choice navigation events, when the processing of those events will result in content identified

for delivery to the learner. In addition, SCORM requires that an LMS not provide navigation

controls that trigger Continue, Previous, and Choice navigation events, when the processing of

those events will result in a Sequencing pseudo-code exception – providing the controls would

enable the learner to trigger navigation events that could disrupt the learner experience. SCORM

does not define how Sequencing and Navigation (SN) provided navigation controls are rendered,

how they are triggered or what navigation events they trigger.

SCORM also provides the means (via <adlnav:presentation>) for a content developer to identify

that the content is providing navigation controls within the content. In these cases, the LMS is

required to honor the request of the content and to not provide any redundant and potentially

confusing user interface controls.

So the manifest in the next table will make sure that navigation menu buttons of the LMS are

disabled (make sure that the necessary navigation controls are implemented with the content),

so no confusing user interface controls appear. And by setting the sequencing control flow to

true a sequencing implementation will automatically evaluate the order in which the activity’s

children should be experienced based on Continue and Previous navigation requests.


<manifest>

…

<organizations default="FT02240">

<organization identifier="FT02240">



<title>Introduction and Explanation</title>

<adlnav:presentation>

<adlnav:navigationInterface>

<adlnav:hideLMSUI>continue</adlnav:hideLMSUI>

<adlnav:hideLMSUI>previous</adlnav:hideLMSUI>

</adlnav:navigationInterface>

</adlnav:presentation>



</item>


<title>Explanation</title>

</item>

<imsss:sequencing>

<imsss:controlMode choice="true" choiceExit="true" flow="true" forwardOnly="false"

useCurrentAttemptObjectiveInfo="true" useCurrentAttemptProgressInfo="true" />

</imsss:sequencing>

</item>

…

</manifest>

Table 3 Implement Sequencing and Navigation

This way the complete navigation and sequencing can be build for the complete module and is

SCORM compatible. Other options within the <imsss:sequencing> are conditioning rules (pre, exit,

and post), rollup-roles, and objectives that certain items need to answer to otherwise they will not

be in the sequencing order. With the help of these options pre-test adaptation can be made possible

in SCORM.

3.4.2 Selftest Implementation

The current selftest is completely implemented with JavaScript and makes no use of the standards of

SCORM. This is not the ideal implementation, because SCORM has perfect standards to implement

such a selftest. First of all, all the questions need to be in the manifest as sub items of the (item)

selftest, this way with the help of sequencing (as explained in the previous subchapter) it can be

made sure all the information is studied before making the selftest. Another option with the help of

randomizationcontrols is to randomly select a number of questions out of a bunch. The manifest for

randomly selecting 2 selftest questions out of the possibly 3 is given:

…

<item identifier="SELFTEST">

<title>Selftest</title>

<item identifier="SELFTEST_QUESTION1" isvisible = "false" identifierref="RESOURCE_QUESTION1">

<title>Question 1</title>

</item>



</item>



</item>

<imsss:sequencing>

<imsss:randomizationControls selectCount="2" selectionTiming="onEachNewAttempt" />

</imsss:sequencing>

</item>

…

Table 4 Selftest implementation


The selftest questions still need to be written with the help of fi. JavaScript, because there is no

standard in SCORM for this, but the variables used in this JavaScript can be used (indirectly) in the

manifest (f.i. to make sure the correct questions are asked and the correct sequence is followed).


4 Adaptation

According to the dictionary (Mer08), adaptation is the act or process of adapting. This dictionary

defines adapting as: to make fit (as for a specific or new use or situation) often by modification.

In this case if a course is adapted, it is completely made fit to a specific user.

In the next subchapter will be described what types of adaptation are possible and which of these

possibilities are suitable for the InfoSecure modules. These adaptation types make use of adaptation

rules as described in chapter 4.2. These adaptation rules base decisions depending on the knowledge

of the student. This knowledge of the student can be determined with the adaptation methods

described in chapter 4.3.

4.1 Adaptation Types

The course can be adapted on different ways. The different adaptation types that make the course fit

to a specific user are explained in the following subchapters.

4.1.1 Content Adaptation

Content adaptation is the action of transforming content to adapt to device capabilities. Content

adaptation is usually related to mobile devices that require special handling because of their limited

computational power, small screen size and constrained keyboard functionality.

Content adaptation could roughly be divided into two fields: Media content adaptation that adapts

media files and browsing content adaptation that adapts websites to mobile devices.

In this case content adaptation is not an option, because all the students use a laptop or desktop

(with a screen resolution of 800-600+) to follow this course, so content adaptation is not necessary.

4.1.2 Link Adaptation

The links in the content can be adapted in such a way, that the presentation style of the links is

associated with the status of these links to pages. This status is determined by a set of rules. It is

typical that some simple rules are used that associate the link presentation according to the

suitability of the link destination. E.g. the presentation can be:

• GOOD: the link points to a suitable page you have not visited before. A standard color for

such link anchors is blue.

• NEUTRAL: the link points to a suitable page you have visited before. A standard color for such

link anchors is purple.

• BAD: the link points to an unsuitable page. Whether or not you visited this page before the

standard color for such link anchors is black.

In addition to link colors the presentation of links may also include icons. The presentation style is

completely in hands of the developer, and the coloring can be completely adjusted to the style of the

course. There is also the possibility to let the user choose his desired colors in a setup menu (AHA08).

In this case link adaptation is an option, but is not of first interest. This is because the sequencing of

the course is already established and changing this will not lead to time benefits, which still is the

first priority.


4.1.3 Presentation Adaptation

In the previous subchapter is described how presentation adaptation is used for the links in the

pages, but presentation adaptation can also be used for the entire course. This way the style of the

complete course is adapted to the style of the user’s preferences. Text fonts, font-sizes, coloring, etc.

In this case presentation adaptation is not an option, because all the courses are adapted to the style

of the company which bought the course, or to the style of InfoSecure. Presentation adaptation will

not lead to great time benefits and the style of the company will be discarded, therefore

presentation adaptation will not be applied.

4.1.4 Information Adaptation

The most important adaptation in this case is the information adaptation, because with this type of

adaptation the most time benefit can be made. Information adaptation is basically skipping or

adjusting information that should be known to the student according to certain adaptation rules (see

chapter 4.2). This way information can be skipped on different levels. The complete course can be

skipped, complete pages can be skipped, or possible only a paragraph or a sentence on a page is

skipped. This way each page can be different for every single student. Instead of skipping, adjusting is

also a possibility. E.g. a 1 minute video instead of a 3 minute video, 2 paragraphs instead of 3

paragraphs etc.

Noteworthy is that skipping the whole page is actually not information adaptation, but link

adaptation, because the link to that specific page is not displayed at all, but the objective is the same:

less information is displayed and more time is saved.

4.2 Adaptation Rules

In the previous subchapters is described that according to adaptation rules certain decisions are

made. The syntax of these adaptation rules are in pseudo-code, because this is program dependant.

The course consist of multiple pages, when the student visits a page, an adaptation rule is triggered.

For instance a student has a certain knowledge level (between 0 and 100) about each page, after

visiting a page a knowledge level can change (possibly depending on answers, or time visited).

For instance the rule on page 1 that makes sure that the knowledge level of page 5 of the student will

change (30% more than the knowledge level of page 2) after reading page 1, looks like this:

page5.knowledge := page2.knowledge + 30

The outcome of these rules can be used to trigger (part of) pages. This will look something like this in

pseudo-code: if (page5.knowledge>40)

then “display this information”;

else “display other information”

end if

Of course answers to certain pretest questions will change the knowledge level of pages (according

to adaptation rules) as well and therefore adaptation is easily applied with these questions.

Adaptation according to the HR information is also possible, because adaptation rules can draw

conclusion out of a student’s HR information. For instance if a student has a CISSP diploma the

following rule will make sure that his knowledge about page 5 is 100%.

If student1.diploma=”CISSP” then page5.knowledge=100


4.3 Adaptation Methods

The above described adaptation rules can base decisions on knowledge of students. This knowledge

can be based on all kinds of information that is available for the student. two adaptation methods

that are most suitable for InfoSecure are described in this chapter. Adaptation by pretest questions,

where a student’s knowledge is based on answers to pretest questions, and adaptation regarding to

HR where a student’s knowledge is based on the information available in the Human Resource

Database of the company.

4.3.1 Adaptation by Pretest Questions

At the start of a course a student is asked a certain number of questions all covering parts of the

course. The parts related to the questions the student answers right, will now be skipped (or

adjusted) during the course. This way a time benefit is gain, if the questions take less time than the

relating parts. The student should only answer the questions of the pretest if he is certain about his

answer, otherwise he should answer the question with the option: “I am not familiar with this

subject and would like to view the information”. This is the standard answer, so if the student

doesn’t change this answer, he is assumed not to know the information. The student is also asked if

he wants to take part of the pretest questions beforehand, if not he gets the normal non-adapted

course, and therefore will not gain time.

The results of the pretest questions will be analyzed to make sure this adaptation method gains time

in comparison with the non adapted course (see chapter 4.4.2).

4.3.2 Adaptation regarding to HR

As described earlier, the adaptation rules use available information of the user. E.g. answers to

questions, pages already visited etc. But another important factor in our course is the information

out of the HR database of the company (e.g. diplomas, certificates, department, function within the

company). How can this information contribute to better adaptation of the course. Adapting the

course according to HR information is more difficult than adaptation on pretest questions, because

there are more aspects that have to be taken into account. All the different HR attributes are

described in the next subchapters. This adaptation method is only applied if it books a time benefit in

comparison with the pretest adapted course. This will be thoroughly analyzed in chapter 4.4.3.

4.3.2.1 Diplomas and Certificates

First of all, people can have the same diploma/certificate, but still have different knowledge on

different subjects. This can be because the students graduated at a different university, but still get

the same diploma/certificate. Another important factor is the year of graduation, because through

the years a lot can change according to the subject material. Even if the year of graduation is taken

into account, students with the same diploma and same certificate still can have different

knowledge, for whatever reason (interest, extra diploma’s, background, current job, etc.). Therefore

no knowledge conclusions can be drawn with a 100% certainty on any HR attribute, including

diplomas and certificates.

In chapter 4.4.3 will be described how this data still can be useful for adaptation, by analyzing the

results of a student’s pretest (and or selftest) in comparison with his HR attributes.

Noteworthy is that if the HR information is not available within the system, this information needs to

be entered once into the company’s database and can then be used for every course the student is

going to follow.

Within the research to adaptation according to diplomas, the year of graduation plays a role. The

university, where the student graduated will not play a role, because with a lot of students, the

variety will be too large. The year of graduation can eventually be divided into groups, e.g. 1975-


1980, 1980-1985. This will depend on the outcome of the analysis. But it is expected that parallels

can be drawn to graduation years that are close to each other.

4.3.2.2 Function

The function of the student can also play in important role for adaptation. A manager or a CEO has

probably more knowledge on certain subjects than a secretary. But the same as for diplomas and

certificates in the beginning no conclusions can be made. Therefore this needs to be investigated as

well.

4.3.2.3 Department

The department the student is working in can be of the same interest as his diplomas, but studying

this results will be different per company. Therefore it is not of first interest, because every company

will have different departments. But in the analysis the department of the student can be analyzed

the same way the other HR information is analyzed. The outcome of the analysis will show if time

benefit can be gained according to adaptation to department. Again this is only possible for large

companies, because than there is a big enough sample to draw conclusions to a student’s

department, before the rest of the students from that department take the course. Of course bigger

companies do have similar departments, but it is not guaranteed that the outcome for every

company will be the same. Possible conclusions can be drawn only after finishing the analysis within

several different companies.

During the analysis in the next subchapter, the student’s department is not taken into account.

4.4 Analysis

As described in the previous chapter there are two methods to adapt the InfoSecure modules. Both

methods need to be analyzed to make sure that a time benefit is booked. In chapter 4.4.2 the pretest

adaptation is analyzed and in chapter 4.4.3 the HR adapted course is analyzed.

To better clarify the analyses in these two chapters first an example course is given in chapter 4.4.1.

4.4.1 Example Course

Every course consists of at least 2 parts. Information pages and a selftest. This is the case if the

course is not yet adapted. If the course is adapted with a pretest it consists of 3 parts. Questions,

information pages, and a selftest. For clarification and simplicity reasons a couple assumptions are

made in this example course.

There is exactly 1 pretest question (QI) that is related to 1 information page (Ii) and the selftest (STi)

consist out of exactly the same number of questions as the pretest and also relates to the same

information pages. In the actual courses, there can be more pretest questions related to more

information pages or even part of information pages. The selftest probably has more questions and is

related to several information pages more than once, but a better clarification can be made with this

example. The selftest will take more time than the pretest, because in the real course it probably

consists out of more (and possibly more time consuming) questions. The student is familiar to the

information while taking his selftest, but still the duration of the selftest will approximately be 1,5

times the duration of the pretest, for twice as many questions. This number can differ per course,

but is roughly in this area.


The non adapted example course will look as follows:

Diagram 2 Non adapted example course

The pretest adapted example course will look as follows (information pages are shown based on the

answers of the pretest questions):

Diagram 3 Pretest adapted example course

The duration of these part is estimated, the real average duration time of (parts of) a course can be

calculated after enough students have finished the course.

4.4.2 Pretest Analysis

During this analysis is checked if the pretest adapted course is making a time benefit in comparison

with the non adapted course. A time benefit can only be made if enough students pass for the

pretest. This percentage is calculated in chapter 4.4.2.1. The correlation between the pre-test

answers and the selftest answers is also very important for calculating the breakeven percentages.

This is explained in chapter 4.4.2.2.

A possible adaptation can be made to the selftest, which will increase the quality of the entire

module (see chapter 4.4.2.3).

4.4.2.1 Success Percentage

The most important percentage for making a time benefit with a pretest adaptive test is the

percentage of students that fail for the pretest and have to study the accompanying information.

This percentage must be lower than its breakeven percentage, otherwise the course makes no time

benefits in comparison with the non adaptive course. The breakeven percentage is calculated as

follows:

��

��

= 1 −

With

: Percentage of students that succeeds for the pretest question i.

�� : Time Profit for question i. The time of the information page of question i minus the time of

pretest question i (Ii -Qi).

�� : Time lost for question i equals Qi.


In the example case pretest question 3 takes 30 seconds, therefore for this question time lost (��) is

30 seconds. Time profit (��) will be 150 seconds. This is the duration of the accompanying

information minus the duration of the pretest question (180-30). The breakeven percentage () for

pretest question 1 is �

�, because

150

30=

1 −16

16

If at least �

� of the students succeeds for pretest question 1, pretest adaptation will make a time

benefit. This is under the assumption that there is a 100% correlation between the pretest and the

selftest. In other words, all the students that succeed for the pretest must succeed for the selftest.

This is probably not the case, so in the next subchapter the success percentage is calculated again,

taking into account the pretest correlation percentage.

4.4.2.2 Success Percentage together with Pretest Correlation Percentage

In the previous subchapter the assumption is made that the correlation between the pretest and

selftest questions is 100%. The actual percentage can be calculated, and together with that the

success percentage explained in the previous subchapter can be calculated more precisely.

When calculating the correlation percentage it is checked if the student really knew the information

he was supposed to know according to his pretest answers. For instance if a majority of the students

answer question Q3 correctly, but fail for question ST3 (which must be quite similar to Q3 of course),

than the pretest question or the selftest question needs to be changed, or at least the correlation

between these questions needs to be altered. There will be a percentage of students that will pass

the pretest question, but will fail for the selftest on the same subject. This correlation percentage

needs to be analyzed for every pretest question, to guarantee a time benefit is gained. This

percentage shouldn’t be too high, otherwise the pretest (and/or selftest) question must be altered or

removed. The lower this percentage the better. The breakeven success percentage depends on the

correlation percentages. When the following formula is smaller than zero, the correct percentage for

both the success percentage, as the correlation percentage can be calculated. The formula will be

explained below:

��1��1 − � + ��2��1 − �� − �� < 0

With:

: Percentage of students that succeeds for pretest question i.

� : Percentage of , that succeeds for selftest question i.

1- : Percentage of students that fails for pretest question i.

1-� : Percentage of , that fails for selftest question i.

� : Percentage of students that succeed for pretest question i and selftest question i.

(1-�� : Percentage that succeeds for pretest question i, but fails for selftest question i.

��1� : Time lost for question i in case the student fails for his pretest question i equals Qi.

��2� : Time lost for question i in case the student succeeds for his pretest i, but fails for his

selftest i. The time of the pretest question i plus the time of (part of) the selftest (Qi +

ST(i)). In case there is no adaptive selftest (which means the complete selftest has to be

done again in case of failure the first time, instead of only the selftest questions that

were answered incorrectly the first time), the time lost will be much higher and

therefore the breakeven percentage will change drastically. Take this in consideration

when calculating these percentages.


In the example course pretest question 3 takes 30 seconds, and the accompanying information

would take 180 seconds, a total time profit (TP3) of 150 seconds is booked if the student passes his

selftest. If the student fails for his selftest, he has to redo the course and (parts of) the selftest again.

The course has an adaptive selftest, so probably not the complete selftest is repeated. This will lead

to a total time lost (TL23) of at least 75 seconds is (30 + 360/8). This process is visualized in the next

diagram:

Diagram 4 Process Pretest Question 3

The students that don’t succeed for the selftest (1-x), will have a duration time of 255 (30+180+45)

seconds for pages related to subject 3. This is 30 seconds (TL13) more than the non-adaptive test,

which has a duration time of 225 (180+45) seconds. The students that succeed for the pre-test and

the selftest (xy) have a duration time of 75 (30+45) seconds, which is a time profit of 150 seconds

(TP3) . The students that succeed for the pre-test, but fail for the selftest ( (1-y)x) have a duration

time of 300 (30+45+180+45) seconds, which is 75 seconds (TL23) more than the non-adaptive test.

�� : Time Profit for question i. The time of the information page of question i minus the

time of pretest question i (Ii -Qi).


Diagram 4 explains the above function, which leads to the following figure for question 3:

Figure 3 Percentages that lead to time profit for subject 3

The above figure gives a clear indication of the possible percentages of students that succeed for the

pretest (success percentage, x), together with the possible percentages of students that fail for the

selftest, but did succeed for the pretest(correlation percentage, y).

The time profit area can be expanded with a few percent, because there are factors that can’t be

exactly calculated, but play a role in the calculation of the percentages in the time profit area:

1. The assumption is made that every student succeeds for his selftest after studying the

information pages. But a small percentage of the students will fail for these selftest (or

pretest) questions even if they would have studied the according information pages at first.

This percentage can be determined by looking at the results from the non adapted course.

The assumption can be made that this percentage isn’t too high, because otherwise the

quality of the course (or the effort of the student) is too low. Therefore the time profit area

can be widened with a few percent.

2. Possibly in the non adaptive course, a student may think he is familiar to the subjects and will

study the information pages only briefly, but will finally fail for the selftest. Because he has to

redo most of the course, he will take more time studying the information pages than in the

first place. While answering pretest questions first, it comes to the student’s attention, that

he’s not so familiar to the subjects as he thought he was, and therefore he will immediately

study the information pages more carefully. This will save quite some time, so therefore the

time profit area can be widened with a few percent.

3. A student also has the option to answer the pretest question with the option “I am not

familiar with this subject and would like to view the information”. This option will cost him

only about 10 seconds. This option is neglected in these calculations, because the


investigation is based on adaptation and if the student admits he is not familiar to the

information, no adaptation is applied. The percentage of students that answer the pretest

question with that option, must be analyzed. Of course the time this group loses in total

must be less than the time benefits of the students that did answer the pretest question.

4. There is also a small group of students that will fail for the specific selftest question, but will

have successfully finished the complete selftest (in case one or more mistakes are permitted

in the selftest). In this case it needs to be considered how this specific group of students is

handled during the calculation of the percentages. Is it more important that the pretest and

selftest questions are perfectly correlated or is time-benefit the most important factor. In

case time benefit is the most important factor, the students in this specific group are handled

as if they passed the specific pretest and the time profit area will expand.

Adaptation can specify this group real easy, and there is an option to handle the selftest

results of this group differently. See chapter 4.4.2.3.

For every question the explained formula should hold, with a little margin in the percentages

possible, because of the above mentioned reasons. The ultimate breakeven points for the success

percentage are given in Table 5, assuming that every student succeeds for his selftest. The ultimate

breakeven points for the correlation percentage are given in Table 6, assuming that every student

succeeds for his pretest.

Pretest question Minimum

percentage (x)

1 21%

2 21%

3 12%

4 21%

5 8%

6 22%

7 22%

8 10%

Table 5 Pretest maximum percentages

Ultimate minimum percentage for pretest question 3 is 12% (rounded of 1/6 according to function

with y=1 (or see top left in time profit area in Figure 3) minus 5% because of above mentioned

reasons). Notice that the number 1/6 is also calculated in chapter 4.4.2.1, because here is also

assumed that every student that succeeds for his pretest also succeeds for his selftest.


Pretest question Minimum

percentage (y)

1 41 %

2 41 %

3 29 %

4 41%

5 22%

6 29%

7 29%

8 25%

Table 6 Minimum selftest succeed percentage

Ultimate minimum percentage for pretest question 3 is 29% (rounded of 0,34 according to function

with x=1 (or see bottom right in time profit area in Figure 3) minus 5% because of above mentioned

reasons).

Try to aim for the highest possible percentage of y, because this means that the pretest questions

and the selftest questions are highly correlated, and the higher y is, the more student can fail for the

pretest with still booking a time profit overall. In other words the lower x can be.

4.4.2.3 Selftest Adaptation

As becomes clear from the analysis, possibly there is a group of students that correctly answers a

pretest question, will therefore not see the according information pages, but answers the according

selftest question incorrectly. Because this student answered all other selftest questions correctly he

passed the complete selftest with success. This is of course strange, because apparently there is

information which he has never seen and is not familiar with. With the help of adaptation and

changing the selftest this problem can be solved. Just change the selftest in such a way that every

answer for which the student hasn’t seen the according information must be correct, otherwise he

will fail for the complete selftest and has to visit the according information pages after.

Possibly in the non adaptive test there will also be students that click through the information

quickly, fail for this specific selftest question, but succeed for the complete selftest. This is also not

the desired result. The problem with the non adaptive module is that this group cannot be specified.

The advantage of the adaptive module is that this group can easily be specified and the selftest can

be adjusted accordingly.

4.4.3 HR Analysis

As described earlier it is impossible to draw knowledge conclusions based on HR attributes with a

100% certainty. Therefore the pretest answers of the student are analyzed. Possibly there are

parallels between the HR attributes and the answers of the students.

The best case scenario is of course that all the students with the same HR information answer the

same questions correctly and the same questions incorrectly, because than the questions are no

longer necessary, because all the information is in the HR database. Unfortunately this is not the

case. In chapter 4.4.3.1 all HR attributes in combination with the answers of the pretest questions

are analyzed. The outcome of the analysis will be that there are different groups of students (with

different HR attributes) that have significantly different answers than the average student. These

outcomes will be compared with a breakeven percentage (see chapter 4.4.3.2) to make sure a time

benefit can be booked. This is the most important outcome. If in total students don’t gain time with

this approach, it will not be applied.


4.4.3.1 Data Mining

Data mining (sometimes called data or knowledge discovery) is the process of analyzing data from

different perspectives and summarizing it into useful information - information that can be used to

increase revenue, cuts costs, or both. Data mining software is one of a number of analytical tools for

analyzing data. It allows users to analyze data from many different dimensions or angles, categorize

it, and summarize the relationships identified. Technically, data mining is the process of finding

correlations or patterns among dozens of fields in large relational databases.

In this case all answers by all students will be analyzed and afterwards data mining will take place to

find correlations between the answers of the students and their HR attributes. This way possibly time

can be saved and therefore costs are cut. In this chapter is described how data mining works for this

example case, without going into too many details, because the data is still hypothetical.

The starting percentages are easy to determine, as described in the next table:

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Number of

students

All student 60% 45% 80% 74% 34% 90% 70% 37% 14000

Table 7 Pretest scorings percentage

The percentages represent the percentage of students, that succeed for the pretest question and

succeed for the complete final selftest. To be clear 100% minus this percentage is the percentage of

these students that fail for the pretest question AND fail for selftest.

This is the basis to compare the rest of the analysis results with. The next tables are:

D:Diploma+year Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Number

of

students

A - 1990 80% 95% 60% 54% 94% 60% 30% 97% 140

A - 1991 85% 85% 30% 72% 92% 50% 80% 64% 150

… 98% 86% 20% 74% 90% 53% 83% 60% …

… 90% 86% 40% 71% 91% 50% 80% 64% …

C – 1975 90% 84% 20% 79% 90% 50% 80% 60%

… … … … … … … …

No diploma 68% 76% 10% 44% 50% 33% 43% 30%

Table 8 Pretest scorings percentage with certain diploma

C:Certificate Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Number of

students

A 95% 86% 50% 74% 90% 53% 83% 60% 120

B 94% 82% 50% 72% 94% 57% 90% 70% 130

… 88% 86% 55% 74% 90% 63% 83% 60% …

… 88% 84% 66% 73% 95% 57% 91% 67% …

E 87% 86% 56% 74% 90% 55% 82% 60%

… … … … … … … … …

No

certificate

68% 76% 10% 44% 50% 33% 43% 30%


Table 9 Pretest scorings percentage with certain certificate

F:Function Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Number of

students

CEO 98% 84% 20% 74% 88% 53% 83% 60% 110

Secretary 44% 42% 23% 66% 90% 44% 44% 33% 140

… 91% 86% 26% 74% 88% 66% 83% 60% …

… 92% 86% 29% 77% 83% 53% 83% 60% …

Manager 98% 86% 33% 74% 90% 66% 83% 60% 70

…. … … … … … … … …

Table 10 Pretest scorings percentage with certain function

The chi-square test (also chi-squared or χ2 test) will investigate if there are groups of students with

certain HR attributes that differ from the basic group of students. This test also makes sure the data

source (number of students) is large enough to draw conclusions on. In this chapter only the basis of

this test will be described.

The test is done by taking one variable at the time. In this case the investigation starts with Table 8,

because a diploma probably has the most influence on the results. The test analyses the results from

students with a certain diploma and compares them to the basic results. For every diploma is

determined if the results can be treated differently. According to the test, students with different

diplomas possibly end up with the same test results and can therefore be treated the same. So after

this part of the test Table 8 will be updated:

Group name D Diploma + year Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8

D1

A1990 – A1995

B1997 – B2004

C1980

D1981-D1999

80% 95% 60% 54% 94% 60% 30% 97%

D2

A1990 – A1995

B1997 – B2004

C1980

C1981

D1981-D1999

85% 94% 70% 60% 89% 70% 50% 80%

… …

D25 No diploma

E2006

68% 76% 10% 44% 50% 33% 43% 30%

Table 11 Updated diploma scorings percentage

Actually this table is not completely correct, because the group names described in Table 11 can very

well be different for each question. So actually 8 tables are necessary, one for each question with

possible different groups. But for clarification reasons in this example one table is given. Notice that

this table has no column named “number of students”, because in this table only data is taken into

account that satisfies the criteria of the chi-square test and therefore is useable. If the student’s

diploma is not in this table, the student will be treated as an average student (see Table 7).

After this analysis the next HR attribute (certification) will be investigated in combination with the

results of Table 11, which will result in a more detailed table:


Group name D Group name C Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8

D1 C1 85% 92% 63% 66% 94% 80% 90% 99%

D1 C2 75% 91% 70% 62% 89% 75% 55% 80%

… …

D25 C9 68% 76% 10% 44% 50% 33% 43% 30%

Table 12 Scorings percentage with diploma and certification taken into account

With group name C containing all certification possibilities. E.g. C1 is certification A or certification B,

and C9 is certification Q. This table is an extension of Table 11 and as you can imagine this table will

expend more by taking into account more HR attributes. Relatively the table will not expend that

much more by taking into account more HR attributes, because the results of the test must make an

significant difference and the first two HR attributes are the most important two and will take credit

for most of the difference.

The test started with the most significant HR attribute diploma, but it could also be possible that the

certification of the student has a bigger impact on the results. Therefore the test also has to be

executed with first certification and afterwards diploma and the other HR attributes. But without

going into too many details, after the complete test is executed a table (this will be a very large table,

in the real program an updated database) will be available:

D C F Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8

D3 C2 F4 100% 100% 100% 100% 100% 100% 100% 100%

D1 C1 F5 94% 95% 97% 95% 94% 94% 95% 94%

D2 C2 F1 98% 84% 80% 74% 88% 53% 83% 60%

D4 C2 F1 80% 95% 60% 54% 94% 60% 30% 97%

D1 C1 - ..% ..% ..% ..% ..% ..% ..% ..%

… … … ..% ..% ..% ..% ..% ..% ..% ..%

D4 C3 F2 30% 33% 30% 27% 24% 30% 30% 26%

D2 C2 F2 18% 18% 9% 18% 5% 9% 9% 7%

- C8 - ..% ..% ..% ..% ..% ..% ..% ..%

D25 C9 F9 ..% ..% ..% ..% ..% ..% ..% ..%

x x x 60% 45% 80% 74% 34% 90% 70% 37%

Table 13 Optimized scorings percentages

With - meaning that this group is not taken into account and x meaning no group. Therefore the last

column represents the percentages equal to Table 7. This is because if the student falls in no

category available in the table on all the different HR attributes (this is the case if the available data

sample, the number of students, is too little or that his HR attributes make no significant difference)

the percentage of the average student is used.

Table 13 is constructed in such a way that the best fitting case scenario is at the top and the worst

fitting case is at the bottom. A student fits in a at least one row, but the best case for this student is

always the highest ranked row in the table.

Noteworthy is that Table 13 also has to be constructed 8 times (once for each question), for the same

reasons as described earlier for Table 11.

4.4.3.2 Breakeven Percentage

If the percentages in Table 13 are higher than the breakeven percentage of that question, the specific

student can skip this question and accompanying information.


This percentage needs to be calculated for every pretest question and is done by the following

formula: ��

��

= �

100 − �

With

In the example case the percentage for pretest question 1 is 47�

��%, because

27

30=

477

19

100 − 477

19

Where the time profit (��) in this case is Q1 = 30, because the breakeven percentage in comparison

with the pretest adaptive test is calculated, and the time lost (��) is ((part of) ST)*1.25 – Q1 = 27,

because in this case the student has to answer the selftest questions immediately and is not familiar

to all the information, this is why the selftest takes approximately 25% more time. The pretest

question is not asked.

Therefore if more than 47�

��% of the students with a certain HR information attribute (diploma,

function, certificate or a combination of those) answer pretest question 1 correctly HR adaptation

will make a clear time benefit for this question and will be applied.

In the example case the percentages are the same for each question, because each pretest question

takes the same time. But again this percentage needs to be rounded off, because there are factors

that can’t be exactly calculated that play a role in the calculation of this percentage:

1. The time the student takes for the selftest is raised with 25%, this is an approximation and

not an exact figure.

2. These percentages are the minimal percentages necessary, assuming that every student will

succeed for the non adapted course after studying the information. This is probably not the

case, so these percentages can be lowered with a few percent.

3. Assumed is that the success percentage and the correlation percentages, explained in

chapter 4.4.2.2 are sufficient, so that the pretest gains time in comparison with the non

adaptive test.

Because of these reasons a correct minimal percentage that should finish the pretest question and

the selftest successfully for the example case is around 40%.

� : Percentage of students the finish pretest question i and the selftest with a good result.

�� : Time Profit for question i equals Qi. In case the breakeven percentages in comparison with

the non adaptive course instead of the pretest adapted course are calculated the time profit

is the time of the pretest question plus the according information pages (Qi + Ii).

�� : Time lost for question i is the time of (part of) the selftest minus the time of the pretest

question (ST(i)-Qi). In case there is no adaptive selftest the percentages will be much lower,

because the complete selftest needs to be done again and therefore the time lost will be

much higher. Take this in consideration when calculating these percentages.


Noteworthy is the fact that this percentage is much higher in case of a non adaptive selftest (around

90% for this example), because then the time lost will be much higher. In the current case, with an

adaptive test, the didactics of the test will change, and this needs attention. With the current

example data possibly 60% percent of the students can immediately make the selftest, fail for it,

have to read the information pages (according to the answers of the selftest), and make the selftest

again. In this case the first selftest acts as a pretest and therefore the actual selftest should consists

of different questions, because otherwise the selftest will be too easy to succeed without having the

required knowledge. There could be made a point for raising the breakeven percentage so that less

students get the same selftest question twice in short period of time, but it is best to make multiple

selftest questions with the same subject. With the help of adaptation this is easily implemented in

the selftest.

Another minus is that in this case 60% of the students possibly have to answer questions which they

don’t know and only afterwards the get the according information. This can be annoying. These

people also get a bad result for their first selftest, which will not motivate them to continue.

Considerations have to be made, when implementing the HR adaptive E-learning module, possibly a

bigger breakeven percentage needs to be established, because this way less students will follow a

course that is not perfectly suitable for them. Again this is a consideration between time benefits and

the didactics of the course.


4.5 Scenarios and Time Benefits

Let’s consider a couple of scenarios together with the time benefits. The time benefits are not

completely correct, because the assumption is made that if the pretest question is answered

correctly, the selftest question about this subject is also answered correctly. In other words, the

percentage explained in chapter 4.4.2.2 is assumed to be 100%. But the percentage of students that

belong to this group needs to be calculated and the time for these students to redo part of the

course and the selftest needs to be added by the total time of the course for these group of students.

But this percentage should be very low, and for these scenario’s it is neglected.

It is assumed that the selftest of the non adapted course will be succeeded by all students the first

time. This is not the case, so the estimated numbers are minimal, because there are also students

that fail for the non adapted course. Another assumption is that in the adaptive test, students after

viewing the necessary information, always succeed for the selftest. This of course needs to be tested

(see chapter 8), but the scenarios and time benefits are sketched to give a proper indication of all the

possibilities and what scenarios to expect during actual testing.

For calculation of the time benefits it is assumed the HR adapted test has a non adaptive selftest, so

(in scenario 2 en 3) the results are a higher than the results with an adaptive selftest would be, but

again these time benefits are just to give an indication.

In chapter 4.5.1 all different kind of scenarios are considered and the time benefits under the above

assumptions are calculated. An overview of these time benefits is given in chapter 4.5.2.

4.5.1 Scenario’s

Five different kind of scenario’s are considered in this case. For all scenario’s the non adaptive course

will take 1710 seconds (see chapter 4.4.1).

Scenario 1: Best Case

Every student that falls into the following categories: D3, C2, and F4 answered question 1 through 8

correctly (see Table 13) and therefore these questions don’t need to be asked or answered by this

group of students. This will save a lot of time, because not only the pretest questions, but also the

accompanying information will be skipped. The time benefit will be as follows:

The pretest adaptive course for this type of student will take 600 seconds, because all the pretest

questions are answered correctly and therefore the accompanying information is not shown, and the

course will look as follows:

Diagram 5 Scenario 1 pretest adapted course

And the HR adaptive course will take 360 seconds, because the pretest questions don’t need to be

asked, and the course will look as follows:


Diagram 6 Scenario 1 HR adapted course

Of course this scenario is not very likely, because a 100% pretest question scoring is very hard to

achieve, other scenarios, like scenario 2 are more likely.

Scenario 2: Best Case 2

Every student that falls into the following categories: D1, C1, and F5 will on average make a solid

time benefit. In Table 13 the percentages are displayed and it is clear that most information is known

by most of these students. Let’s split up this scenario:

Scenario 2a

Most of these students will pass all pretest questions, and therefore the pretest adapted course will

take as much time as scenario1, 600 seconds, as well as the HR adaptive course, 360 seconds. A small

percentage of the students will fail for the selftest and at least 1 pretest question, a possible scenario

is given as scenario 2b.

Scenario 2b

All kind of scenario´s are possible, let´s just assume that this particular student fails for pretest

questions 1 and 6. In this case the pretest adapted course will take him 900 seconds (see Diagram 7).

Diagram 7 Scenario 2b possible pretest adapted course

Noteworthy is that the student thought he know the information, otherwise he would have

answered the pretest questions 1 and 6 with the option “I am not familiar with this subject and

would like to view the information” and would only need 10 instead of 30 seconds for pretest

question 1 and 6 each.

In case of the HR adapted course the student needs 1110 seconds, as described in the next diagram:

Diagram 8 Scenario 2b possible HR adapted course

For this particular student the HR adapted course takes more time than the pretest adapted course,

but overall more students will make time profit and therefore the average student with this HR

attributes will make a clear profit:


Time benefit complete scenario pretest adapted course

For every student the course takes at least 600 seconds (see Diagram 5), but a certain percentage

(see Table 13) will fail for some pretest questions, and therefore has to view the according

information pages. In the case of these students the average duration will be 672 seconds as

described in this table:

pretest

fails / succeeds

%

Duration

information

page (sec)

Total average

duration

per student

(sec)

Total average

duration per

student after

question (sec)

Q1 6 +120 720 607,2

94 0 600

Q2 5 +120 727,2 613,2

95 0 607,2

Q3 3 +180 793,2 618,6

97 0 613,2

Q4 5 +120 738,6 624,6

95 0 618,6

Q5 6 +240 864,6 639

94 0 624,6

Q6 6 +180 819 649,8

94 0 639

Q7 5 +180 829,8 658,8

95 0 649,8

Q8 6 +210 868,8 671,4

94 0 658,8

average total ≈672 sec

Table 14 Average duration pretest adapted course scenario 2

Again, it is assumed that the student will succeed for their selftest after viewing the necessary

information pages first.

Time benefit complete scenario HR adapted course

In case of the HR adaptive test a same calculation can be constructed. This time the course takes at

least 360 seconds (see Diagram 6), but if a student fails for a pretest question, the complete selftest

needs to be done (450 seconds, because the student started with the selftest, and was not familiar

with all the information, it takes him 25% more time than the normal 360 seconds) again. In the case

of this group of students the average duration will be less than the 672 seconds (see Table 14),

because the pretest is passed in more than 90% for all questions (see chapter 4.4.3.2). The exact

calculation can be made on the same way the pretest time is calculated, only with a slide adjustment.

Assumed is that the total time will be 360 seconds plus 450 seconds (time of the complete selftest),

810 seconds. And afterwards, the overall percentage of all students that succeed the selftest the first

time is subtracted from the outcome, resulting in this table:


pretest

fails /

succeeds

%

Duration

information

page (sec)

Total average

duration

per student

(sec)

Total average

duration per

student after

question (sec)

Q1 6% 120 930 817,2

94% 0 810

Q2 5% 120 937,2 823,2

95% 0 817,2

Q3 3% 180 1003,2 828,6

97% 0 823,2

Q4 5% 120 948,6 834,6

95% 0 828,6

Q5 6% 240 1074,6 849,0

94% 0 834,6

Q6 6% 180 1029 859,8

94% 0 849

Q7 5% 180 1039,8 868,8

95% 0 859,8

Q8 6% 210 1078,8 881,4

94% 0 868,8

Percentage

all questions

correct

64,93% -450 431,4

280,1

Percentage

that has to do

selftest

35,07% 0 881,4

309,1

Sum 589,2

Average total ≈590 sec

The average total time for this students is 590 seconds.

Scenario 3: Average Case

Every student that falls into the following categories: D4, C2, and F1 will on average make a time

benefit. To calculate the time benefits for the pretest adapted course, the same table can be

constructed as in scenario 2 only with different percentages:


pretest

fails / succeeds

%

Duration

information

page (sec)

Total average

duration

per student

(sec)

Total average

duration per

student (sec)

Q1 20 +120 720 624

80 0 600

Q2 5 +120 744 630

95 0 624

Q3 40 +180 810 702

60 0 630

Q4 46 +120 822 757,2

54 0 702

Q5 6 +240 997,2 771,6

94 0 757,2

Q6 40 +180 951,6 843,6

60 0 771,6

Q7 70 +180 1023,6 969,6

30 0 843,6

Q8 3 +210 1179,6 975,9

97 0 969,6

Average total ≈ 976 sec


The average duration for this type of students is 976 seconds. In the HR Adapted course it is only

profitable to adapt the questions 2, 5, and 8, because their percentages exceed the breakeven

percentage of 90%. All the other pretest questions are asked. So the HR adapted course will in the

best case look as follows:

Diagram 9 Best case scenario 3 HR adapted

In the best case for this scenario the course will take 510 seconds. The average case will take 636

seconds. This is calculated in two steps, first is calculated what the average duration time of the

pretest questions and accompanying information is, on the same way as above, with temporarily

changing the HR-adapted questions to 100%:


pretest

fails / succeeds

%

Duration

information

page (sec)

Total average

duration

per student

(sec)

Total average

duration per

student (sec)

Q1 20% 120 630 534

80% 0 510

Q2 0% 120 654 534

100% 0 534

Q3 40% 180 714 606

60% 0 534

Q4 46% 120 726 661,2

54% 0 606

Q5 0% 240 901,2 661,2

100% 0 661,2

Q6 40% 180 841,2 733,2

60% 0 661,2

Q7 70% 180 913,2 859,2

30% 0 733,2

Q8 0% 210 1069,2 859,2

100% 0 859,2

Average total ≈ 860 sec

Table 16 HR adaptation part 1

The average time for the questions 1, 3, 4, 6, and 7 with the possibly accompanying information is

859,2 seconds. The added average time for the questions 2, 5, and 8 with HR adaptation will be 86,9

seconds according to the next table.


pretest

fails /

succeeds

%

Duration

information

page (sec)

Total average

duration

per student

(sec)

Total average

duration per

student (sec)

Q1 0% 120 120 0,0

100% 0 0

Q2 5% 120 120 6,0

95% 0 0

Q3 0% 180 186 6,0

100% 0 6

Q4 0% 120 126 6,0

100% 0 6

Q5 6% 240 246 20,4

94% 0 6

Q6 0% 180 200,4 20,4

100% 0 20,4

Q7 0% 180 200,4 20,4

100% 0 20,4

Q8 3% 210 230,4 26,7

97% 0 20,4

Percentage

all questions

correct (PC)

86,62%

+0 +26,7 23,13

1-PC 13,38% +450 +476,7 63,78

Sum 86,91

Average added

total ≈ 86,91 sec

Table 17 HR adaptation part 2

The total average time for a student in this group will therefore be 859,2 + 86,91 ≈ 946 seconds

based on HR adaptation.

Scenario 4: Worst Case

The percentages in Table 13 for a student with the following HR attributes D4, C3, F2 are very low.

Actually the percentages are just high enough to apply the pretest. Compare the row in Table 13 with

Table 6 and it is clear that the time benefit will not be too high for the pretest. HR adaptation will

clearly not make a time profit.

The average pretest time will be 1568 seconds as calculated in Table 18, which still is a minimal

average time benefit of 142 seconds.


pretest

fails / succeeds

%

Duration

information

page (sec)

Total average

duration

per student

(sec)

Total average

duration per

student after

question (sec)

Q1 70 120 720 684

30 0 600

Q2 67 120 804 764,4

33 0 684

Q3 70 180 944,4 890,4

30 0 764,4

Q4 73 120 1010,4 978

27 0 890,4

Q5 76 240 1218 1160,4

24 0 978

Q6 70 180 1340,4 1286,4

30 0 1160,4

Q7 70 180 1466,4 1412,4

30 0 1286,4

Q8 74 210 1622,4 1567,8

26 0 1412,4




Scenario 5: Worst Case 2

The percentages in Table 13 for a student with the following HR attributes D2, C2, F2 are very low.

These percentages are too low to apply the pretest. Compare the row in Table 13 with Table 6 and it

is clear that no time benefit is gained. With these percentages HR adaptation is out of the question.

The average pretest time will be 1810 seconds as calculated in the next table, which is an average

time cost of 100 seconds.

pretest

fails / succeeds

%

Duration

information

page (sec)

Total average

duration

per student

(sec)

Total average

duration per

student after

question (sec)

Q1 82 120 720 698,4

18 0 600

Q2 82 120 818,4 796,8

18 0 698,4

Q3 91 180 976,8 960,6

9 0 796,8

Q4 82 120 1080,6 1059

18 0 960,6

Q5 95 240 1299 1287

5 0 1059

Q6 91 180 1467 1450,8

9 0 1287

Q7 91 180 1630,8 1614,6

9 0 1450,8

Q8 93 210 1824,6 1809,9

7 0 1614,6



4.5.2 Time Benefits

As described in the scenarios above a clear overview of the time benefits is given:

Scenario Non Adaptive Course

(TT)

Pretest Adaptive Course

(TT / TP)

HR Adaptive Course

(TT / TP)

Scenario 1 1710 seconds 600 seconds

1110 seconds

360 seconds

1350 seconds


1038 seconds

590 seconds

1120 seconds


734 seconds

946 seconds

764 seconds


142 seconds

Not applied

-

Scenario 5 1710 seconds 1810 seconds (don’t apply)

-100 seconds

Not applied

-

Table 20 Time benefits per scenario


With

TT = Total time of the course per average student

TP = Total time profit of the course per average student in relation to the non adaptive course

Of course the individual time benefits can be very different, as shown in the next table:

Student Non Adaptive Course

(TT)

Pretest Adaptive Course

(TT / TP)

HR Adaptive Course

(TT / TP)

Best student 1710 seconds 600 seconds

1110 seconds

360 seconds

1350 seconds

Honest

student 1

1710 seconds 1710 seconds

0 seconds

-

-

Honest

student 2


-80 seconds

-

-

Worst

student 1


-240 seconds

-

-

Worst

student 2


-240 seconds

1710 seconds

0 seconds

Worst

student 3


-240 seconds

2160 seconds

-450 seconds

Worst

student 4


-600 seconds

-

-

Worst

student 5

±2650 seconds ±1950 seconds

±700 seconds

-

-

Table 21 Individual time benefits

With

TT = Total time of the course

TP = Total time profit of the course in relation to the non adaptive course

And

Best student is a student who knows all the answers to the pretest questions (or has such a good HR

profile that all the pretest questions are skipped) and succeeds for the selftest.

Honest student 1 is a student that prefers to follow the complete course instead of the adaptive one.

This student selects this option at the beginning of the course, and the little time this will take is

neglected.

Honest student 2 is a student that answers all the pretest question with the option “I am not familiar

with this subject and would like to view the information” has the same results as the non adaptive

course, but needs 10 seconds per pretest question to answer, and therefore spends 80 seconds more

in this case.

Worst student 1 is a student that fails for all the pretest questions.

Worst student 2 is a student that fails for all the pretest questions, but this was expected according

to his HR attributes.

Worst student 3 is a student that is supposed to know all the pretest questions according to his HR

attributes, but fails for the selftest on all parts.


Worst student 4 is a student that knows all the pretest questions, but fails for the selftest on all

parts.

Worst student 5 is a student that thinks he knows all the information, reads the information pages

not thoroughly and therefore fails the selftest in the non adaptive course and has to redo the course.

In case of the pretest adaptive course, the student realizes, because of the pretest questions, he is

not so familiar to the information as he thought he was, and therefore immediately reads the

information pages more thoroughly.

4.6 Conclusion

It is impossible to come up with a list of diplomas/certificates that guarantee knowledge about

certain subjects, therefore first an analysis needs to be done, but at the end a list of diploma’s,

certificates, together with the year of graduation, and possibly function and department will provide

a perfect scheme for what part of the course needs to be done by which type of student. Before this

list is made, good pretest questions need to be made, so adaptation can be applied. Of course it is

still possible an individual student needs more time for an adapted course than for the non adapted

course, but overall most students will gain time benefit and therefore the company will gain a lot of

time and therefore money.


5 Introduction to Awareness Module

As described before, the first module that is subjected to personalization is the module “Basic

module (information) Security”.

Source: http://www.infosecuretools.com/awareness/demo/kpn/#is (only accessible with password

by InfoSecure staff).

Subject: E-learning awareness – English – Basic Module Employees.

5.1 Module Introduction

The module gives an introduction to (information) security within the company, in this case KPN. The

module explains what information security is, and what the objective and importance for the

company is. The module consists of roughly three parts, an explanation of information security, the

golden rules, and a selftest. The module has a duration of approximately 40 minutes including the

selftest, which checks if the student’s knowledge of information security is sufficient.

5.2 Module Duration

The duration of the module is approximately 40 minutes. The duration is distributed over several

chapters (see Table 22 Duration Introduction Module). The time taken per chapter depends on the

student, and his reading speed. For the estimation of the duration a mean reading speed of 200

words per minute is used. Some chapters have movie material and for the total estimation of the

duration the student is expected to watch all the movies completely. The module also contains

exercises. The duration time of these exercises is also estimated for an average student.

Again if a person reads 300 words per minute the complete course will take 30 minutes, but for the

analysis of this module the results of the adaptation will be presented according to an average

student.

Chapter Pages +/- time

Explanation 1 60 sec (reading)

What is Information Security 2 105 sec (reading)

Status within the company 2 116 sec (reading)

About the golden rules 1 60 sec (reading)

Golden rule 1 3 71 sec (reading)

50 sec (movie)

60 sec (exercise)


38 sec (movie)

60 sec (exercise)


53 sec (movie)

30 sec (exercise)


72 sec (movie)

60 sec (exercise)


49 sec (movie)

60 sec (exercise)



Golden rule 7

Golden rule 8

Golden rule 9

Selftest

Conclusion

Relevant links and contacts

Total:

16 chapters

5.3 Time Distribution

The duration of the module can be divided in three parts: reading, movies, and exercises (see

22 Duration Introduction Module

average student spends on each of that parts.

Figure

5.4 Module Construction

The module consists of 16 chapters with a total of 57 pages. The

before completing the course. By using the menu on the left the

pages (see Figure 5 Example Basic Module (Information) Security

Reading

51 sec (movie)

60 sec (exercise)

3 96 sec (reading)

93 sec (movie)

60 sec (exercise)

3 77 sec (reading)

64 sec (movie)

60 sec (exercise)

4 148 sec (reading)

38 sec (movie)

60 sec (exercise)

16 33 sec (reading)

60 sec (exercise)

180 sec (selftest)

1 40 sec (reading)

Relevant links and contacts 1 39 sec (reading)

57 pages

2528 sec

±42 min

Table 22 Duration Introduction Module

can be divided in three parts: reading, movies, and exercises (see

Duration Introduction Module). In the following diagram it becomes visible how much time an

spends on each of that parts.

Figure 4 Time Distribution Introduction Module

The module consists of 16 chapters with a total of 57 pages. The student has to study all these pages,

ing the course. By using the menu on the left the student visits all the necessary

Example Basic Module (Information) Security).

1270 min

50%

508 min

20%

750 min

30%

Reading Movies Exercises

55

can be divided in three parts: reading, movies, and exercises (see Table

n the following diagram it becomes visible how much time an

has to study all these pages,

visits all the necessary


Figure 5 Example Basic Module (Information) Security

The sequence of the pages is completely free of choice. But most students will follow the menu from

top to bottom, by pressing next-button, on the bottom right of the screen. The standard sequence of

the course is described by the following diagram:

Diagram 10 Page sequence


5.5 Module Adaptation Locations

Basically adaptation can be applied throughout the complete course, except for the final selftest at

first, because this test checks if the student has sufficient knowledge about information security. The

selftest is similar for every student. Even if some subjects are not threaded during the course,

because of the adaptation, still questions about these subjects are asked. This is to check if the

adaptation is correctly applied and the student really has sufficient knowledge about these subjects.

If the student fails for his test, the course, or at least a part of the course, needs to be repeated.

Adaptation can be applied throughout the complete course. Every page can be adapted. In the

current module every chapter only consists of a few pages and will take little time (see Table 22

Duration Introduction Module). Adaptation within the pages will therefore gain very little time,

because the information given per page is very minimal. Therefore the adaptation needs to be

applied on the chapters. So the real question is; which chapters contain known information to the

student and which chapters don’t?

In the following subchapters each chapter of the module will be examined.

5.5.1 Explanation

This chapter explains how the e-learning module works. Every e-learning module of InfoSecure works

in the same way, so if a student is already familiar to e-learning modules of InfoSecure, this chapter

can be skipped and adaptation can make this possible.

5.5.2 What is Information Security

In this chapter information security is explained. This chapter is very important and can therefore

only be adapted if it is a 100% certain the student is familiar to information security.

5.5.3 Status within the Company

These two pages gave some statements about information security at KPN and the conclusion is that

the employees of KPN do not think and act alike when it comes to security.

This page can be adapted/skipped if the student knows the status within the company, and therefore

knows why it is important to study this course. This page will be skipped if the previous page (What is

Information Security) is skipped as well.

5.5.4 About the Golden Rules

This chapter explains the golden rules. These are the most important rules that play a role in security

and information security within the company. There are nine roles each extensively explained in a

individual chapter. This chapter must be adapted in such a way that only the necessary golden rules

for the student will be explained and presented in the menu on the left. The 9 golden rules are:

• Rule 1: Keep to the law and KPN’s rules of conduct

• Rule 2: Allot the correct classification to company information

• Rule 3: Prevent improper use or theft of company information

• Rule 4: Prevent improper use or theft of company equipment such as laptop, PDA and

mobile phone

• Rule 5: Pay attention when using e-mail and the Internet

• Rule 6: Use information from and about others with care

• Rule 7: Prevent unauthorized access to our buildings and systems

• Rule 8: Your own safety and that of others is first and foremost!

• Rule 9: Report incidents directly to the Helpdesk of KPN Security: 0800 - 4040 442

If the student is familiar with all the rules it can be skipped, otherwise not.


5.5.5 Selftest

As described earlier the selftest will not be adapted at first, because the selftest must be followed by

every student. This selftest consists out of 12 questions and are all asked. But this selftest can be

adapted if the student fails for the selftest the first time and has to follow the course again. This time

only the questions that were answered incorrectly the first time need to be asked, and this will save

time (an adaptive selftest). So actually the chapter selftest, is the only chapter that is adapted within

the chapter instead of the chapter itself.

5.5.6 Conclusion

This chapter gives a summary and some important telephone numbers, so this chapter cannot be

adapted.

5.5.7 Relevant Links and Contacts

This chapter gives links and contact information, this chapter cannot be adapted, but the student is

not obligated to read it.

5.6 Module Adaptation Techniques

In the previous subchapters is clearly explained where the adaptation can be applied. In this chapter

will be explained how it is applied. A profile of the student is made before taking the course. This

profile keeps updating itself after every (or during the) course taken by the student. The current

course “E-learning awareness: Basic Module Employees” will be taken by a lot of new employees and

therefore the profile still needs to be created.

5.6.1 Student Profile

Every student has a profile. If the student doesn’t have a profile it will be created before starting with

a course. The profile will be filled with human resources (HR) information available for every

employee (education, diploma’s, department). If this information is not available (in the system), the

student has to answer this questions one time before starting with his first course.

The profile also contains answers to pretest questions, that are asked before every course. The

answer to these question are very important. The student should only answer if he’s absolutely sure,

otherwise the student might possibly skip vital information. And this will cost extra time, because the

student will then fail for his selftest and has to redo the course.

Information about courses already taken will also be saved in the profile of the student.

A good profile is the most vital part of the adaptation. The profile information is saved on the server,

and is only used for the adaptation of the courses. The information is encrypted and thus protected

against misuse by third parties.

5.6.2 Adaptation based on HR Information

As described in the previous chapter it is impossible to adapt on HR information without the results

of an analysis first. In this case you might suspect that a student with a background (diploma,

function, or certificate) in the Information Security field should pass this basic introduction module

without studying the information first. But again this conclusion cannot be drawn without an

analysis.


5.6.2.1 Accepted Diplomas and Certificates

After the analysis the results of the accepted diplomas and certificates will possibly look like this.

A possible list of diploma’s that satisfy the criteria to skip the course is given below.

Bsc Computer Science (Information Security)

Msc Computer Science (Information Security)

Postgraduate Diploma in Information Security

Ing. Computer Science (Information Security)

Ir. Computer Science (Information Security)

Table 23 Accepted Diplomas

A possible list of certificates that satisfy the criteria to skip the course is given below.

Postgraduate Certificate in Information Security

Certified Information Systems Security Professional (CISSP)

Table 24 Accepted Certificates

In this example only a list is considered, for which the student can skip the complete course. Ideally

the analysis is much more detailed, and for every HR attribute appropriate adaptation steps can be

executed. For instance a certain golden rule(s) can be skipped with a certain diploma.

5.6.3 Adaptation based on Pretest

Because HR adaptation is only possible after an analysis (of the pretest answers of the students) the

focus is on pretest adaptation. Therefore pretest answers need to be asked. For every module holds

that the benefit of time must be substantially larger than the time that it takes the student to answer

the pretest questions. In this case the module already contains small exercises that perfectly can be

used as pretest questions. Therefore it will not cost extra time and will only gain time.

All the nine golden rules have an exercise part, ask this question before explaining the rule, with the

option “I am not familiar with this golden rule and I’d like to study this rule first”. If the student

chooses this option, the course stays exactly the same and the same exercise is asked after finishing

the chapter about the golden rule (see Diagram 10 Page sequence).

If the student completes the chapter-exercise without mistakes, this chapter can be skipped, and

that will save time. The following diagram shows an example of a student that is familiar to the

golden rules 3, 6, and 9 according to his answers given at the chapter questions.


Diagram 11 Example adaptive page sequence

It is also an option to let the student make all these exercises at the beginning of the module (This is

why it is called a pretest), and then adapt the course according to the answers. This way the student

is first answering questions, than (possibly) studying information, and after that doing his final

selftest. Diagram 12 will show the sequence for the student that is familiar to the golden rules 3, 6,

and 9 according to his pretest answers.

Diagram 12 Example 2 adaptive page sequence

This option is used in the implementation in AHA! as described in the next chapter.


5.6.4 Adaptation Results

With the pretest (and later the HR information) it is clear that the course can be adapted, resulting in

a course containing different number of chapters for each student. The chapter “Explanation” will

only be presented if a student hasn’t taken a course yet. In other words if the student has a profile

already this chapter can be skipped.

For the following two chapters “What is information security” and “status within the company”

adaptation will be harder. If somebody is familiar with information security the entire course can be

skipped together with these pages.

For the latter, the status of the company is explained to raise questions to the student. It can be

skipped, but again this has to do with the didactics of the module. If the student follows most of the

course, it is better to leave this chapter intact.

The other chapters (conclusion, relevant links and contacts), except for the golden rules chapters,

and the selftest described above, will not be adapted.

The following diagram gives an overview of how in the first implementation the adaptation will be

applied throughout the course after analyzing the HR attributes, and assuming that there is only a

correct list of diplomas and certificates as stated in chapter 5.6.2.1, for which a student can skip the

entire course and starts immediately with the selftest. In this diagram the method explained in

Diagram 11 is used. Notice that this is a different implementation than chosen in the next module in

AHA! with only the pre-test and no HR adaptation.


Diagram 13 Adaptation Process

*In the diagram example the selftest is failed by the student on the questions about chapter 2,3, and 5. Only these parts of the course need to be repeated.

**Update Profile: actually the profile is possibly updated after every step, and not only at the end of the module.

5.6.5 Scenario’s

The fictional company JOHANSSON has over 100.000 employees, every employee has to have

sufficient knowledge about information security and therefore all the employees need to follow the

adapted Basic Module (Information) Security.

Adam, the chief information officer of the company, who just graduated for his CISSP certificate has

all the knowledge about information security you expect him to have.


Rachel, the secretary of Adam, has worked for him for several years and has quite some experience

working with computers and dealing with company secrets.

Joey is working in the sales department of the company, but has a background in Computer Science.

35 years ago he graduated in Computer Science, but he found a job in a complete different direction,

so his knowledge about information security is pretty out-of-date.

5.6.6 Time Benefits

The time benefits that are booked with the adaptation are hard to predict, but different scenario’s

can be sketched. In total there are 1025 several routes to go through the course (go to the selftest

immediately, or visit all the obligated pages together with the 10 optional pages (explanation, golden

rule 1 through 9), 102.).

The best case scenario is Adam, according to his HR information he’s familiar to the content and

apparently he was. He made the selftest with absolutely no mistakes.

Rachel had followed a different course before, so the explanation part was not necessary for her, and

according to her answers of the pretest, she was familiar to golden rules 3,4,5, and 9. At the end she

successfully ends the selftest.

The worst case scenario is Joey, according to his HR information he should be familiar to all the

information, but according to his selftest, he wasn’t familiar to any of the subjects what so ever.

Therefore he had to do the entire course, after failing his selftest. After finishing the course, Joey

successfully ended the selftest. Assuming Joey would have succeed for the selftest the first time, if he

first studied the course, the adapted course took more time than the non adapted course. But this

scenario will almost never occur.

Adapted Course (sec) Non-adapted Course (sec) Time benefit (sec)

Adam 352 2528 2176 (±36 min)

Rachel 1872 2528 656 (±11 min)

Joey 2801 2528 -273 (±-4,5 min)

Table 25 Time Benefits Scenario's

These time benefits are based on the HR adaptive module (assuming that there is a correct list of

diplomas and certificates as stated in chapter 5.6.2.1, for which a student can skip the entire course

and starts immediately with the selftest.). In case of the pretest adaptive module the adapted course

will take a little more time for the pretest questions. Possibly therefore the result of Joey will be

better. Also if a better analysis of the HR attributes is done, it is very well possible that Joey will not

pass for skipping the entire course, but only parts of it.

5.6.7 Conclusion

Adaptation can be applied for this course, but it is an introduction course, so most employees, will

probably follow the complete course. There will be gaining of time thanks to the adaptation, but

concrete numbers are not available till some testing with employees is done. However in the

previous chapter becomes clear that in the best case scenario 36 minutes is saved, and in an average

scenario 11 minutes. In a few cases the adapted module will take a little more time, but this is only

the case if the selftest needs to be done more than once.

The course will not change that much, only the sequence of presented information and exercises will

change. Therefore it is guaranteed that the module is still of high standard.


6 Module in AHA!

In the previous chapter it is clearly explained for which parts of the module adaptation is possible. In

this chapter the structure of the adaptive module is explained. How to build an adaptive model with

the help of AHA! and how to build this exact module is explained in this chapter.

The adaptive AHA! module is build for test purposes. Therefore the module works with pretest

questions and adapts according to the answers of them. After enough students have followed this

adaptive course, similarities between the pretest answers and the HR attributes of the student can

be found. But this is not the first test purpose. The main test purpose is to find out if the pretest

adaptive module has sufficient time benefits in comparison with the non-adapted module. The

adaptive module is followed by a test group that has never followed this module before, therefore

no additional information about the students is known, apart from the information that they entered

at the startup screen. Another similar test group will follow the non adaptive version of the course.

This course is also rebuild in AHA!. The building of the non adaptive AHA! module will not be

explained, because building this version is similar to building the adaptive version, except the

concept structure is less complicated.

6.1 Process of the Adaptive Module

In this subchapter first the process of the specific adaptive module is described. Afterwards this

process is visualized with a diagram that is suitable for all adaptive modules of this kind. This diagram

will be explained and initialized for this specific module.

This adaptive module is build with nine pretest questions, that are all directly linked with the nine

golden rules (as in Diagram 12). According to these answers the module is constructed. Possibly the

student can immediately go to the selftest, in case he answers all the pretest questions correctly. In

the other cases the according information (the golden rules) based on the pretest answers is shown,

before the student can make his selftest.

After the selftest (12 questions) is succeeded (at least 10 questions correct), the last two pages with

additional link and contact information are shown. If the students fails for his selftest, the module is

rebuild with only the according pages based on the selftest answers. After studying the pages again

the selftest must be done again. Only the selftest questions that were answered incorrectly the first

time are asked again (an adaptive selftest). If the student answers more than 9 questions correct

(together with his previous attempts) the selftest is succeeded, otherwise the module is again rebuild

with only the according pages based on the selftest answers. This process continues until the student

has answered at least 10 questions correct from the selftest.

This process is visualized with the help of

Diagram 14. This diagram describes the process of an adaptive module that uses pretest questions

and a selftest. The explanation and the initialization of the diagram for this specific adaptive module

is given next.


Diagram 14 Adaptive process generic module

First the initialization of the module needs to be done. In case of this module, the number of pretest

questions (N) is set to 9 and the number of selftest questions (M) is set to 12. The other variables;

number of pretest questions answered correctly (PQT), number of selftest questions answered

correctly (STT), and a Boolean variable that determines if the selftest is made at least once (m) are all

set to 0 at initialization.

After visiting the pages “Explanation Module” and “Explanation Pretest” (which can be replaced with

different pages in a different module of course) the temporarily variable n makes sure all pretest

questions are asked and the Boolean function PQ(n) determines if question n is answered correctly. If

all 9 (n<N) pretest questions are asked the process continues. In case all these questions are

answered correctly (PQT=N) the selftest starts, otherwise the pages “What is IS”, “Status within

KPN”, and “About Golden Rules” are visited before the temporarily variable i determines which pages

(in this case golden rules) have to be visited and which pages can be skipped. This decision factor is

based on the answers of the pretest questions (PQ(i)), the answers of the selftest questions (ST(i)),

and the Boolean variable m. The relationships between those are given in the next table for the

current module.

i PQs(i) (Bool) Q(i) (Bool) Page(i) (const)

1 PQ(1) ST(1) AND (ST9) Golden rule 1

2 PQ(2) ST(2) AND (ST10) Golden rule 2

3 PQ(3) ST(7) Golden rule 3






9 PQ(9) ST(4) AND ST(6) Golden rule 9

Table 26 Initializing table for current module


This table differs for every adaptive module, but in the current case the nine pages are the nine

golden rule pages and the pretest questions are all directly linked to the golden rules. Actually in this

case PQs(i) is the same as PQ(i), but if more pretest questions are used for one golden rule, this will

not be the case. Like with the selftest questions, multiple questions relate to one golden rule. F.i.

Selftest question 4 and 6 need to be answered correctly, only then golden rule 9 can be skipped. The

decision to visit page(i) depends on Q(i) if the selftest is done at least once (m=1) and depends on

PQs(i) if the selftest is not yet done (m=0).

After visiting the necessary golden rule pages the explanation of the selftest is given. Temporarily

variable m makes sure all the selftest questions are asked and Boolean function ST(m) determines if

question m is answered correctly. If all 12 (m<M) questions are asked the selftest results are

displayed. In case at least 10 selftest questions are answered correctly (STT>9) the pages

“conclusion” and “relevant links” are displayed before the module is ended. If less questions are

answered correctly the student has to visit some golden rules again (according to Table 26) and then

has to redo the selftest until at least 10 good answers are given.

To accomplishing such a process in AHA! first a conceptual structure needs to be created (see

chapter 6.2).

6.2 Conceptual Structure

Creating a conceptual structure is the most fundamental part of designing an adaptive hypermedia

application. First a concept structure/hierarchy needs to be designed (see chapter 6.2.1), afterwards

the concepts need to be created (see chapter 6.2.2). At last, and most important, the concept

relationships (see chapter 6.2.3) need to be created. The concepts, as well as the concept

relationships can be directly edited into an XML file that contains the application’s concept structure.

In chapter 6.2.4 will be explained for each concept how and which relations and attributes are

implemented.

AHA! has authoring tools (see chapter 6.5) that will make this process easier and will work less time

consuming than editing the XML file manually and leave little room for errors.

Before explaining how to create the conceptual structure, first the meaning of a concept is explained:

• something conceived in the mind : thought , notion ;

• an abstract or generic idea generalized from particular instances;

(Mer08).

AHA! uses concepts to implement the AHA! application on every level. So don’t be confused with the

meaning of the word concept, because AHA! uses concepts on every level, and for every page in

AHA! a concept should be created. Therefore in reference to AHA! it is better to speak of a concept

structure than a conceptual structure. Actually the conceptual structure of the application is given in

Diagram 14 and the concept structure is an overview of all the concepts and their hierarchy.

6.2.1 Design Concept Structure

Each AHA! application consists of a set of concepts. For each page there should be a concept, but

there may also be many other concepts. In the case of the adaptive module each page is linked with

exactly one concept. The concept structure of the adaptive module is the following, with the

concept name between brackets:

o Explanation (infosecure1)

o Pretest Questions (pretest)


• Question 1 of 9 (question1)









o What is Information Security (whatisism)

• Page 1 (whatisis)

• Page 2 (whatisis2)

o Status within KPN (statusm)

• Page 1 (status)

• Page 2 (status2)

o About the golden rules (about)

o Golden Rule 1 (goldenrule1m)

• Page 1 of 2 (goldenrule1)


































o Selftest (selftest)

• Question 1 of 12 (selftest1)

• Question 1 of 12 confirmation (selftest1c)























• Results (results)

o Conclusion (conclusion)

o Relevant links and contact (links)

The above table describes the concept structure of the adaptive module. As described in chapter

6.1, some concepts will not be showed in the menu, depending on the answers of the pretest

questions. This will be accomplished by creating concept relationships (see chapter 6.2.3). Every

concept is (in this case) linked to a page. This is described in detail in the next subchapter.

6.2.2 Creating Concepts

As explained above in this case every concept is linked to a page. A concept contains elements, the

necessary elements will be explained in this chapter. Each concept has a unique name (in AHA! this

must be a single word, alphanumeric and starting with a letter), a description, a resource, and a

concepttype. Description speaks for itself, this is a description of the concept. For linking the page to

the concept concepttype is set to “page” for every concept and for resource an URL is specified. For

instance: “file:/infosecure/page1.xhtml”.

Other elements are title and hierarchy, these elements are used to define the menu structure (see

chapter 6.2.2.1). Concepts also have attributes with their own properties (see chapter 6.2.2.2). After

all the concepts are created the xml file will have about 7000 lines and a fraction of this file is given in

chapter 6.2.2.3 to explain the representation in AHA!.


6.2.2.1 Menu Structure

As will be described in chapter 6.4 the module uses the StaticTreeView (standard in AHA!) to

generate a menu. Because of the automatic generation of the menu AHA! needs additional

information for each concept in the form of elements. Title is an element that holds the title of the

concept, that will be shown in the menu. And the element hierarchy has three sub-elements:

firstchild, nextsib, and parent. In order they describe the first child of the concept, the name of the

next sibling in the hierarchy and the name of the parent concept, if that exists.

6.2.2.2 Attributes

For each concept AHA! stores a number of attributes that may be different for every concept. Every

attribute has a number of properties and a set of adaptation rules. The properties are: name, type,

isPersistent, isSystem, isChangeable, description, default.

Name: the name of the attribute.

Type: the type of the attribute (Boolean, Integer, or String).

IsPersistent: a Boolean value that determines if the value of the attribute is remembered in the

user model or is only stored temporarily during the rule execution.

IsSystem: a Boolean value that determines if the attribute is system defined or not (only access

and visibility, these will be explained in chapter 6.2.3.1).

IsChangeable: a Boolean value that determines if the attribute can be changed with the help of

forms.

Description: a description of the attribute.

Default: the initial value of the attribute.

The attributes used for implementing the adaptive module will be explained in chapter 6.2.3.

6.2.2.3 Example .aha file

The complete .aha file which contains the concept structure with all his elements, attributes and

relationships is almost 7000 lines, a small fraction of this code is given below to explain the

representation of the concepts in AHA!:

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE conceptList SYSTEM '../generatelist4.dtd'>

<conceptList>

<name>infosecure2</name>

<concept>

<name>infosecure2.infosecure2</name>

<description>An abstract concept to bind the concepts of the module</description>

<resource>file:/infosecure2/infosecure2.xhtml</resource>

<concepttype>page</concepttype>

<title>Adaptive Module InfoSecure</title>

<hierarchy>

<firstchild>infosecure2.whatisism</firstchild>

<nextsib></nextsib>

<parent></parent>

</hierarchy>

<attribute name="visited" type="int" isPersistent="true" isSystem="true"

isChangeable="true">

<description>has this page been visited?</description>

<default>0</default>

</attribute>

<attribute name="suitability" type="bool" isPersistent="false" isSystem="false"

isChangeable="false">

<description>the suitability of this page</description>

<default>true</default>

</attribute>

<attribute name="showability" type="int" isPersistent="true" isSystem="false"


isChangeable="true">

<description>showability this concept</description>

<default>0</default>

</attribute>

<attribute name="hierarchy" type="bool" isPersistent="true" isSystem="true"


<description>the visibility of this concept in the hierarchy</description>

<default>true</default>

</attribute>

</concept>

………

</conceptList>

Figure 6 Example aha file

6.2.3 Concept Relationships

All the concepts described in the concept structure (see chapter 6.2.1) have relationships with each

other. As described in chapter 6.2.2.1 all the concepts have an element hierarchy with the explained

sub-elements to create the correct order of the concepts. The concepts also have an attribute

hierarchy that will make sure the concepts with this attribute set to true will be shown in the menu

(see chapter 6.4). This attribute and others will be explained in more detail in the next subchapter.

Each concept also has adaptation rules, these rules and how they are used to update the attributes

of concepts will be explained in chapter 6.2.3.2.

6.2.3.1 Used Attributes

As described in chapter 6.2.2.2 each concept stores attributes with different properties. AHA! uses

some standard attributes (see next subchapter) and some attributes are created especially for this

module (see chapter 6.2.3.1.2).

6.2.3.1.1 Standard Attributes

AHA! uses some standard attributes like Access, Knowledge, Visited, Suitability, Interest, Showability,

and Hierarchy. The ones used for this module are explained below:

Access: a non-persistent Boolean attribute that temporarily becomes true when the resource

associated with the concept is accessed.

Visited: a persistent integer attribute that counts the number of accesses to a concept.

Suitability: a Boolean attribute (that can be persistent or not) that determines whether the

concept is considered desirable or undesirable. AHA! can change the color of the links

to these concepts if they are desired or not. This feature is not used in this module.

However AHA! also has the option to implement a link to “next suitable concept” and

therefore this attribute is used.

Hierarchy: a Boolean attribute that determines if the concept is part of the hierarchy in the

menu.

6.2.3.1.2 Created Attributes

TempNumber1: For every pretest question Boolean attributes are created (temp1, temp2, temp3

etc.). The attributes determine which answer is given to the question. F.i. if temp2 of

concept question3 becomes 1, then the student answered option 2 for pretest

question3.

TempNumber2: It is also possible that integer attributes are created instead of Boolean ones, this

depends if multiple answer combinations are possible to one question. F.i. question1

consists out of 4 small questions, that each set a different temporarily attribute to a


different value. By adding these values together and compare this with a known

number, only one combination makes sure that all small questions are answered

correctly.

Done: For every pretest and selftest question a Boolean attribute is created that determines

if the question itself is been asked already.

Ff: For every selftest question an integer value is created that determines the answer of

the selftest question. If the value is set to 1 the answer was correct.

Temp1: For every selftest question this Boolean value determines if the selftest question was

answered correctly.

Complete: This Boolean value for selftest12 determines if the selftest is completed or not.

Result: The concepts results has an integer attribute that determines how many correct

answers are given so far to the selftest questions.

All above attributes are changeable, because with the help of forms these attribute values are

changed, and are persistent. The default value for all these attributes is false or 0.

6.2.3.2 Adaptation Rules

The updates to attributes of concepts in the user model are done through event-condition-action

rules. Every rule is associated with an attribute of a concept, and is "triggered" whenever the value of

that attribute changes. Every page has an access attribute which is (virtually) "changed" whenever

the end-user visits that page. This triggers the rules associated with this attribute. Every rule consists

of the following parts: condition, true-actions, false-actions, and propagation.

Condition: when the rule is triggered this Boolean condition is evaluated.

True-actions: when the condition is true, this set of actions is executed.

False-actions: when the condition is false, this set of actions is executed.

Propagation: when a rule is executed it updates some attribute(s) of some concept(s). The

propagation field indicates whether these updates will trigger the rules associated

with the updated attribute(s).

Adaptation rules are used in this module mainly if the access attribute was triggered, the

implementation of these rules and the use of the above explained attributes is explained in the next

subchapter.

6.2.4 Implementing Concept Structure

In the previous chapter is explained which attributes, elements, and adaptation rules can be used, in

this chapter will be explained how they are actually implemented in the concept structure for the

adaptive module.

6.2.4.1 Hierarchy and Suitability

These two attributes are very close to each other, as a matter of fact these two attributes are in this

module exactly the same, because there is no concept that is suitable and not in the hierarchy of the

menu or vice versa. Therefore for every concept that has another value than true for suitability, the

default value for hierarchy becomes suitability. This way it is sure that these two attributes have the

same value. It will also save time and leave little room for errors, if during programming the value of

suitability must be changed manually, the value of hierarchy will change automatically in the same

value.


Because of the adaptive module concepts are only shown (suitable/in the hierarchy) if these are

necessary for the user. And this mainly depends on the answers given to the pretest questions. Also

the order of the questions is determined with the help of the suitability attribute, because every

question needs to be asked and questions cannot be skipped. Therefore the module starts with only

the explanation starting page (this concept suitability is true, therefore always visible and in the

hierarchy) and the pretest explanation page. This concept has his suitability attribute set to:

infosecure1.whatisism.visited == 0 && infosecure1.selftest.visited == 0

Meaning that the concept whatisism (What is Information Security) must not be visited yet and the

concept selftest must not be visited yet. After the pretest is completed the next suitable concept will

be whatisism or selftest (according to yet to explain adaptation rules), therefore this concept

(pretest) will only be visible during the answering of the pretest questions thanks to this suitability

attribute.

Every pretest question is only suitable if it is not asked before and the previous question is asked

before. This is accomplished by setting the suitability attribute for every question. F.i. the suitability

attribute for pretest question2 is as follows:

infosecure1.question1.done && !infosecure1.question2.done

After finishing the pretest questions the next concept in line is whatism. This concept is suitable if

one of the golden rules is suitable or if the selftest is completed and not succeeded. This leads to the

following settings of the suitability attribute:

(infosecure1.goldenrule1m.suitability || infosecure1.goldenrule2m.suitability ||

infosecure1.goldenrule3m.suitability || infosecure1.goldenrule4m.suitability ||



infosecure1.goldenrule9m.suitability)

||

(infosecure1.results.result < 10 && infosecure1.selftest12.complete)

The concepts whatisis and whatisis2 are children of the concept whatisism and therefore

automatically only suitable if whatisism is suitable, and therefore no suitability attributes need to be

set here.

The concepts statusm and about get the same suitability attribute as whatisism, because the only are

suitable if whatisism is suitable.

The suitability attributes for the golden rules pages are most important, because they depend on the

answers given in the pretest and in case the selftest is complete they depend on the answers given in

the selftest. Therefore every golden rule has a unique suitability attribute, 2 golden rules concepts

are discussed in detail, goldenrule1m and goldenrule3m, because they are slightly different from

each other. The other suitability attributes of the golden rule concepts are constructed in the same

way.

In chapter 6.3.2 will be explained how certain temporarily attributes get certain values, but for now it

is enough to know that temporarily attributes get certain values that make it possible to know how

the pretest and selftest questions are answered (see chapter 6.2.3.1.2).

Concept question1 has among others the integer attributes temp1, temp2, temp3, and temp4. If

those four temporarily integer attributes together are 400 the question is answered correctly and

then concept goldenrule1m is not suitable. Also must be made sure that all pretest questions are

answered and the selftest is not yet made. In case the selftest is made, it must not be succeeded and

in this case question1 or question9 of the selftest must be answered incorrectly (because question 1


and 9 of the selftest are related with golden rule 1). Putting this all together the suitability attribute

of concept goldenrule1m becomes as follows:

(infosecure1.question1.temp1 +infosecure1.question1.temp2 + infosecure1.question1.temp3 +

infosecure1.question1.temp4 != 400 && infosecure1.question9.done &&

!infosecure1.selftest12.complete)

||

(infosecure1.results.result < 10 && infosecure1.selftest12.complete &&

(!infosecure1.selftest1.temp1 || !infosecure1.selftest9.temp1))

Concept question3 has among others the Boolean attributes temp1, temp2, temp3, and temp4. For

this question if temp1 and temp3 are true and the others are not true, than the question is answered

correctly and therefore goldenrule3m should not be suitable. Just as for question 1 it also must be

made sure that all pretest questions are answered and the selftest is not yet made. In case the

selftest is made, it must not be succeeded and in this case question7 of the selftest must be

answered incorrectly (because question 7 of the selftest is related with golden rule 3). Putting this all

together the suitability attribute of concept goldenrule3m becomes as follows:

(!(infosecure1.question3.temp1 && !infosecure1.question3.temp2 && infosecure1.question3.temp3

&& !infosecure1.question3.temp4) && infosecure1.question9.done &&

!infosecure1.selftest12.complete)

||

(infosecure1.results.result < 10 && infosecure1.selftest12.complete &&

!infosecure1.selftest7.temp1)

After the golden rules the selftest follows, the introduction page of the selftest is only showed if the

pretest is done, therefore the suitability attribute is:

Infosecure1.question9.done

The following selftest questions must be shown in order. Important is that no questions can be

skipped (therefore only one question at a time can be shown in the menu), which leads to a rather

long value for the suitability attributes of the selftest questions.

For selftest question 1 this is the following:

!infosecure1.selftest1.temp1 && !infosecure1.selftest1.done

The temp1 value is set in selftest1c, so this makes sure that the confirmation of the question is not

yet done and that the question isn’t asked already.

Selftest1c has his suitability attribute set to:

infosecure1.selftest1.done && !infosecure1.selftest1c.done

This makes sure the confirmation is not yet done and the selftest question is. This attribute is the

same for all selftest confirmation pages. The suitability attributes for the selftest questions becomes

longer for every question, because it must check if the previous questions are asked already.

Therefore for selftest2 the suitability attribute is set to:

!infosecure1.selftest2.temp1 && !infosecure1.selftest2.done && infosecure1.selftest1c.done

Until selftest12 that is set to:


!infosecure1.selftest12.temp1 && !infosecure1.selftest12.done &&

infosecure1.selftest1c.done && infosecure1.selftest2c.done && infosecure1.selftest3c.done &&



infosecure1.selftest10c.done && infosecure1.selftest11c.done

After all the questions are answered a result page comes up. The suitability attribute for this page is

set to the following, because all questions must be answered:




infosecure1.selftest10c.done && infosecure1.selftest11c.done && infosecure1.selftest12c.done

The last two pages (conclusion and links) are only displayed if the pretest is done and the selftest is

succeeded, this can easily be confirmed by:

infosecure1.question9.done && infosecure1.results.result > 9

In all the above values of suitability attributes are other attributes involved like results, done, and

complete that have a different value depending on time and answers given. How these values change

is explained in the next subchapter and in chapter 6.3.2 where is explained how adapted pages are

written and how these values change with the help of forms.

6.2.4.2 Access

Adaptation rules are used in this module if the access attribute of a concept is triggered. Every time

the confirmation of a selftest question is accessed, there is an adaption rule triggered that checks if

the result of the selftest is correct. If a question is answered correctly, the end result increases with

1. For selftest4c this leads to the following in the xml-file:

<concept>

<name>infosecure1.selftest4c</name>

…

<attribute name="access" type="bool" isPersistent="false" isSystem="true"


<description>triggered by page access</description>

<default>false</default>

<generateListItem isPropagating="true">

<requirement>infosecure1.selftest4.ff==1</requirement>

<trueActions>

<action>

<conceptName>infosecure1.results</conceptName>

<attributeName>result</attributeName>

<expression>infosecure1.results.result + 1</expression>

</action>

</trueActions>

</generateListItem>

…

</attribute>

</concept>

For further references it will be described in pseudo-code as follows:

IF infosecure1.selftest4.ff == 1 THEN

infosecure1.results.result := infosecure1.results.result + 1

After the selftest the results are displayed. When these results are displayed most attributes need to

be reset, because possibly a student has to redo parts of the test. A student only has to redo the

questions of the selftest for which he fails, therefore if the value of the temp1 attribute of the

selftest questions equals false, the done attribute of the selftest and selftest confirmation concept


needs to be set to false, so the question will be repeated during the next selftest. Also the value of ff

needs to be set to zero again.

At this point the selftest is completed, so this attribute needs to be set to true. This will create the

following adaptation rule: IF infosecure1.selftest1.temp1 == false THEN {infosecure1.selftest1.done=false

;infosecure1.selftest1c.done=false;infosecure1.selftest1.ff=0}

IF infosecure1.selftest2.temp1 == false THEN {infosecure1.selftest2.done=false






















infosecure1.selftest12.complete = true

In chapter 6.3.2 will be explained how the values of these attributes are changed.

6.3 Write Pages

In this chapter is explained how the actual pages are written. First is explained how the pages are

written that are exactly the same as in the non adapted version (see chapter 6.3.1). Afterwards is

explained how in some pages adaptation is encoded and why (see chapter 6.3.2).

All the pages are rewritten, because the styling of the original pages was done within the pages,

instead of using stylesheets. Also paragraphs (<p></p>) and divisions (<div></div>) are used, because

they are easy to adapt with the help of stylesheets. For more information about stylesheets, visit the

website of World wide web Consortium (W3C09). No information about writing XHTML is given in

this document, for more information, visit the same website (XHT09).

6.3.1 Write Standard Pages

As described in chapter 3 the import SCORM to AHA! program didn’t function as planned and

therefore the AHA! module needed to be build from scratch. While copying and pasting the pages

from the HTML pages to the for AHA! required XHTML pages, a few errors occurred. The HTML pages

used some tags with no end-tag, which is required for XHTML. E.g. <BR> was frequently used instead

of <BR></BR> or <BR />.

Another problem was the nesting of the tags. In HTML there is automatic endings of tags, and

therefore the following codes works fine:

<UL>

<LI>Option 1

<LI>Option 2

<LI>Option 3

<UL>

<LI>Suboption 1

<LI>Suboption 2

</UL>

<LI>Option 4


<LI>Option 5

But it has to be properly nested to work fine in XHTML, as shown in the next table:

<UL>

<LI>Option 1</LI>

<LI>Option 2</LI>

<LI>Option 3</LI>

<UL>

<LI>Suboption 1</LI>

<LI>Suboption 2</LI>

</UL>

<LI>Option 4</LI>

<LI>Option 5</LI>

</UL>

Another problem occurred while copying the text from the html source directly to the xhtml source.

Often the error: “invalid byte 1 of 1-byte utf-8 sequence” was displayed instead of the text. This error

occurs because rich characters, e.g. opening and closing quotations, were used in the text. These

characters needed to be replaced with the standard characters.

Because of these differences between HTML and XHTML basically all the pages were manually

rewritten, to make sure no errors were made. Also extra pages were necessary for the correct

adaptation, these pages will be explained in the following subchapter.

6.3.2 Write Adapting Pages

Most pages with adaptation were especially written and were no part of the non-adapted module,

like the pretest questions (see chapter 6.3.2.1) and the selftest (see chapter 6.3.2.3). Other pages

were only a little adjusted for a better learning experience, like the golden rules page (see chapter

6.3.2.2). In the next subchapters specific pages of the adaptive module are explained, with this

explanation it should be clear how to create your own adapting AHA! pages.

6.3.2.1 Pretest Questions

The pretest questions are asked at the beginning of the module. There are nine questions, that

means nine concepts, and nine pages. Each page is more or less the same, so by explaining two

pages (question1.xhtml and question3.xhtml) of respectively concept question1 and question3 the

principle of creating pretest questions becomes clear. Only the specific adaptations parts are

explained, the basic (X)HTML syntax is not explained. For creating the questions a form is created,

within this form AHA! attributes can be used, therefore the following is possible:

<form method="post" action="/aha/ViewGet/FormProcess?redirect=true">

<p>Fill in the correct terms. Use all terms exactly once.<br /><br />

<ul>

<li>The <select name="Element1.question1.temp1" size="1" >

<option value="0" name="0"> </option>

<option value="1" name="1">KPN company code</option>

<option value="100" name="2">KPN sub codes</option>

<option value="2" name="3">KPN key values</option>

<option value="3" name="4">KPN collective labour agreement</option>

</select> are concrete rules of behaviour on competition, integrity, safety and inside

knowledge.</li><br /><br />

<li>The <select name="Element1.question1.temp2" size="1" >

<option value="0" name="0"> </option>

<option value="100" name="1">KPN company code</option>

<option value="1" name="2">KPN sub codes</option>

<option value="2" name="3">KPN key values</option>

<option value="3" name="4">KPN collective labour agreement</option>

</select> describes that basic principles we apply within KPN in our daily

activities.</li><br /><br />


</ul>

</p>

<p class="nextlink"><input type="submit" value="Next Question"> </input>

</p>

<input type="hidden" name="Element1.question1.done" value="true"></input>

</form>

Table 27 Fraction of question1.xhtml

After submitting this form, by clicking on the “next question” button the attributes

infosecure1.question1.temp1 and infosecure1.question1.temp2 will be set to 0,1,2,3, or 100

depending on the answer selected. After submitting also the attribute infosecure1.question1.done

will be set to true thanks to the hidden input tag. With this information it is possible to adapt the

module as described in chapter 6.2.4.1.

The form will be sent to the internal FormProcess page of AHA!, this page will process the form in a

way AHA! can work with all the values of the attributes. Without the addition “?redirect=true” in the

action tag of the form a confirmation page is given, but with this addition, no confirmation is given

and the next suitable concept will show up in the same frame. In this case the next question.

Question 3 differs a little from question 1 because instead of selecting an answer, checkboxes are

used so multiple answers can be selected:

<p>

How can you best send confidential information by email? Select the correct answer(s)<br

/><br />

<input class="checkbox" type="checkbox" name="element1.question3.temp1" value="true"> Call

the receiver in advance</input><br />

<input class="checkbox" type="checkbox" name="element1.question3.temp2" value="true"> Via my

hotmail account</input><br />

<input class="checkbox" type="checkbox" name="element1.question3.temp3" value="true">

Encrypted</input><br />

<input class="checkbox" type="checkbox" name="element1.question3.temp4" value="true"> Does

not matter, as long as it is sent via the XP-working station</input>

</p>

This way the attributes infosecure1.question3.temp1 to infosecure1.question3.temp4 will be set to

either false or true, depending on selection.

6.3.2.2 About the Golden Rules

The page “about the golden rules” gives a summary about the golden rules. In case of the adaptive

test some golden rules are more important than others to specific users. Depending on the pretest

answers some golden rules can be skipped and therefore they need less attention on this page.

Therefore the important golden rules are displayed in bold, and the other rules just in plain test.

Because these rules differ per student adaptation is necessary. This adaptation is done with the help

of conditional fragment within the page. These conditional fragments let you conditionally include or

hide parts of a page. The condition is an expression using attributes of concepts and constants. In this

case the fragment looks like this:


<table border="0" width="100%">

<if expr="infosecure1.goldenrule1.suitability">

<block><tr><td width="40"><b>Rule 1: </b></td><td><b>Keep to the law and KPN's rules of

conduct</b></td></tr></block>

<block><tr><td width="40">Rule 1: </td><td>Keep to the law and KPN's rules of

conduct</td></tr></block>

</if>

<if expr="infosecure1.goldenrule2.suitability">

<block><tr><td width="40"><b>Rule 2: </b></td><td><b>Allot the correct classification to

company information</b></td></tr></block>

<block><tr><td width="40">Rule 2: </td><td>Allot the correct classification to company

information</td></tr></block></if>

…

</table>

This way the explanation of the golden rule is displayed bold if it’s concept is suitable.

6.3.2.3 Selftest

Before the selftest starts, a page with the explanation of the selftest is given. This page is also an

adapted page. If some golden rules are not read by the student, and these golden rules were advised

to read according to the pretest, the student will be reminded of that. If a student answers all pretest

questions correctly, he will immediately go to this page. In this case this will be displayed on the

page. If a student already made his selftest and he has to do the selftest again, he won’t need the

same explanation of the test as before. In this case a different (shorter) explanation of the selftest is

given. This page will look different depending on a lot of attributes, therefore a lot of conditional

fragments are used. Conditional fragments are also used inside other conditional fragments. To make

this more clear the complete page selftest.xhtml is given in Appendix A.

Some expressions within the conditional fragments contain & or < or > instead of &, <, or

>. Unfortunately this is necessary, otherwise it will confuse the XML parser.

As is visible in selftest.xhtml the “next page” button is also constructed with the help of conditional

fragments. This way the button is constructed in the complete module.

6.4 Look and Feel

Although the original module was in the style of KPN, the adaptive module is in the style of

InfoSecure, according to their demand. Each AHA! application defines its own look and feel. This

consists of a definition of layout, consisting of html frames and which information goes into which

frame, and a specification of concept presentation.

The files “ConceptTypeConfig.xml” and “LayoutConfig.xml” are created in such a way that the

module fits perfectly on a screen width a least a resolution of 800x600 pixels or more. The frame

structure looks like this with the corresponding pixels. If an asterisk instead of a number is given, this

means that the rest of the page is used, this depends per screen resolution:


Both the MainView, and the StaticTreeView are included in AHA!. The MainView automatically

presents the correct pages/concepts. The StaticTreeView automatically generates a menu in the form

of a tree according to the concept structure (see chapter 6.2). If the student clicks on a concept link

in the StaticTreeView, this concept will be shown in the MainView. To make sure these pages are

created in the InfoSecure style a stylesheet is automatically added to all the files and views that are

used in this program. This stylesheet is added to all the files by adding the following line into

LayoutConfig.xml:

<layoutconfig>

<stylesheet>infosecure.css</stylesheet>

</layoutconfig>

Figure 7 Frame Structure

*

Empty

88

*

20

190 610 *

HEADER

Main

View

FOOTER

Static

Tree

View

Empty


The end result looks like this:

Figure 8 Screenshot Module AHA!

6.5 Authoring Tools

Instead of writing all the code manually AHA! offers some authoring tools, which will lighten your

job. The Graph Author (see chapter 6.5.1) and the Concept Editor (see chapter 6.5.2) are both tools

for creating concepts, attributes, and concept relationships. The form editor (see chapter 6.5.3) lets

you create forms through which users can change part of their user model and at last the test editor

(see chapter 6.5.4) which lets you create multiple choice questions. For a complete description of

these authoring tools, visit the AHA! tutorial website (AHA08). In the following subchapters specific

aspects of the tools will be explained, also will be explained why this tool is used or not.

6.5.1 Graph Author

The Graph author is strongly recommended for creating the concept structure of an application. This

high level authoring tool is sufficient for almost all applications and is easy to use. Because of this


reason the Graph Author was first used to create the concept structure. But after creating the

structure the relationships need to established. Therefore the graph author was not ideally for this

specific module. As described in the previous subchapters the suitability and hierarchy attributes

depend on a lot of values and some concepts depend on a lot of other concepts. The graph author

works graphically. So literally all concepts that in one way or another depend on each other have to

be connected to each other with a specific relationship. This would be possible, but you probably

can’t see the forest for the trees anymore. Another important reason for not working with the graph

author anymore is that the module must be fully understand by the author. If the author wants to

make an adjustment in the XML file this must be possible. When working with the Graph author a

.gaf file is created which can be converted in the actual .aha xml file. The .aha file can’t be converted

to a .gaf file, so therefore changes made in the .aha file will not be visible in the Graph Author. And

after using the Graph Author again, changes manually made will be discarded. The Graph Author is

an easy to use tool, but for this purpose and better understanding of the concepts it is not used.

6.5.2 Concept Editor

The concept editor is a low level authoring tool which edits the .aha XML file with the help of concept

templates. It is an easy to use tool, but it leaves more room for errors than the graph author and it

doesn’t work that fast. If you are familiar to XML and if the same changes to a lot of concepts need to

be made, it is easier to work directly in the .aha file with the help of Notepad or another Editor.

While making the current module most changes are made directly in the XML file.

6.5.3 Form Editor

The Form editor is a tool that lets you create forms in an easy way. If you are already familiar with

the HTML forms this editor is unnecessary. But if you are unfamiliar with forms, this editor will make

your job easier. In this adaptive module, the pretest as well as the selftest consist of forms. The form

editor isn’t used while making these questions, because it was easier to make one question and copy

it for the other questions with some small adjustments.

6.5.4 Test Editor

AHA! has a very nice test editor for creating multiple choice questions with some great features, like

time spend, results analysis etc. which was perfectly suitable for this module. The only problem was

that the test made with this editor is a java applet and the author has no control over the

presentation of the test. Therefore all the test are made with forms instead of the test editor,

because that way the tests are in the same style as the rest of the module, which was a requirement

of InfoSecure.

6.6 Other Methods

In the previous subchapters is clearly explained how the module is build and which tools are used. Of

course there are multiple ways to create the same module in AHA!. The used method was that no

new templates or views were created and that AHA! his built-in views and relations were fully used.

Another possibility would have been to create a complete different view for the menu, instead of

using the built-in StaticTreeView. This would have saved time in setting all the suitability and

hierarchy attributes right, but again creating the new view would have cost time.

Another possibility would have been to use the NextView, this view will automatically display a

button to the next suitable page. This would have saved quite some time, because now on every

page the “next page” link is created with the help of conditional fragments. But again a button was

not in the correct style, but adjusting the NextView (and the corresponding ConceptTypeConfig.xml)

in such a way that the button becomes a link is a good option.


There is also the possibility to create templates. In the graph author tool you can create concepts and

concept relationships of different types. The list of available types is based on templates. The

functionality of the graph author can be extended by creating new templates for concept

relationship types and/or changing existing ones.

There are a lot of different methods for creating AHA! applications, no one is better than the other,

but be aware of the possibilities.

7 Extracting Test Data

Before explaining the results of the test in the next chapter, in this chapter will be explained how the

data from the AHA! log files is used, with the help of which programs and why.

7.1 Analyzing the AHA! Logs

In the aha subdirectory xmlroot/log are all the log files of all the persons that have done the test. The

log file is named after the name the user entered before the test started. For Instance John Doe his

log file is named “access_John Doe.xml” and looks like this:

<?xml version="1.0"?>

<!DOCTYPE log SYSTEM 'access.dtd'>

<log>

<user>John Doe</user>

<record>

<accessdate>Thu Sep 04 09:15:00 CEST 2008</accessdate>

<sessionid>6D3B224E69A3BE221E8803D0379032DD</sessionid>

<name>file:/infosecure1/infosecure1.xhtml</name>

<fragment>false</fragment>

</record>

<record>



<name>file:/infosecure1/pretest.xhtml</name>


</record>

<record>



<name>file:/infosecure1/question1.xhtml</name>


</record>

…

Table 28 Example access_John Doe.xml

As is visible from the example log file it is easy to calculate the time a person takes for each page by

subtracting the access dates from each other. In case of the example John Doe took 14 seconds at

the page “file:/infosecure1/pretest.xhtml”. These calculations can easily be made by a spreadsheet

program (f.i. Microsoft Excel 2007), this is why the xml log files were imported in Excel. An extra

column with the outcome of subtracting the two access dates from each other is created and the

following table is created for every user:

User Access date Name Time

John Doe Thu Sep 04

09:15:00 CEST

2008

file:/infosecure1/infosecure1.xhtml 00:00:20

John Doe Sep 04

09:15:20 CEST

2008

file:/infosecure1/pretest.xhtml 00:00:14


John Doe Thu Sep 04

09:15:34 CEST

2008

File:/infosecure1/question1.xhtml 00:00:22

… … … …

Table 29 Table with extra time column1

These tables (for every user) are imported in a database program (f.i. Microsoft Access 2007) in a

single table so the calculations can easily be made. This table (named p1) has 1720 records and all

the desired calculations with the help of SQL statements can be made.

7.1.1 Correcting Data

With the help of a simple SQL query (“SELECT time, name FROM p1 ORDER BY Kolom1 DESC;”) that

orders all the pages from most time consuming to less time consuming. The few pages that took

some users more than 5 minutes are replaced by the average time a user needed for this specific

page. Assumed is that the user was distracted (phone call etc.) and his time-value was incorrect. The

average time for every page is easily calculated by the following query:

SELECT AVG(SECOND(p1.Time)+60*MINUTE(p1.Time)) AS som, name FROM p1 GROUP BY name;

7.1.2 Data Group 1 vs. Data Group 2

With the help of the following two SQL statements:

SELECT SUM(SECOND(p1.Time)+60*MINUTE(p1.Time)) AS som, p1.[user]

FROM p1

WHERE name LIKE 'file:/infosecure1/*'

GROUP BY p1.[user];

And

SELECT SUM(SECOND(p1.Time)+60*MINUTE(p1.Time)) AS som, p1.[user]

FROM p1

WHERE name LIKE 'file:/infosecure2/*'

GROUP BY p1.[user];

the total time per user per group is displayed (these exact SQL statements are only capable of

calculating times with a maximum of 3599 seconds, which in this is case is more than enough). The

results are displayed in the following two tables, with the names of the persons anonymous.

Som

(sec)

User Som

(sec)

User

469 Person 1A 523 Person 2A

633 Person 1B 567 Person 2B

690 Person 1C 718 Person 2C

877 Person 1D 1355 Person 2D

1 As a matter of fact the actual table contained more columns, such as fragment, sessionid, and some

temporary columns. These columns aren’t used in the calculations and therefore left out in this document for a

better overview.


1015 Person 1E 1374 Person 2E

1107 Person 1F 1421 Person 2F

1166 Person 1G 1450 Person 2G

1256 Person 1H 1527 Person 2H

1586 Person 1I 1583 Person 2I

2018 Person 1J 1588 Person 2J

2206 Person 1K 1624 Person 2K

2318 Person 1L 1683 Person 2L

2401 Person 1M 2707 Person 2M

2653 Person 1N 3363 Person 2N

Group 1 Group 2

1456 Average 1535 Average

Table 30 Timings from all test persons

The figures in these tables are not the final figures, by further analyzing the log files, person 1A, 1B,

1C and 1D only made the pretest and quit the module afterwards, so for calculating the average time

for group 1 these figures need to be discarded. Person 1H didn’t make the final selftest, so his time

his discarded as well. Therefore the average time for group 1 will be 1830 seconds (30,6 minutes).

In group 2 only person 2A didn’t finish the complete module, so only his time is discarded. This gives

group 2 an average time of 1612 seconds (26,9 minutes).

Som

(sec)

User Som

(sec)

User

1015 Person 1E 567 Person 2B

1107 Person 1F 718 Person 2C

1166 Person 1G 1355 Person 2D

1586 Person 1I 1374 Person 2E

2018 Person 1J 1421 Person 2F

2206 Person 1K 1450 Person 2G

2318 Person 1L 1527 Person 2H

2401 Person 1M 1583 Person 2I

2653 Person 1N 1588 Person 2J

1624 Person 2K

1683 Person 2L

2707 Person 2M

3363 Person 2N

Group 1 Group 2

1830 Average 1612 Average

Table 31 Timings from correct test persons

These figures are not the figures that prove the adaptive test is less time consuming than the non-

adaptive test. On the contrary, the adaptive test takes on average almost 4 minutes more than the

non-adaptive test, according to the test results.

As will be described in chapter 8.2 most persons that made the test were all familiar to the content

and therefore the percentage that succeeded for the selftest the first time was pretty high. In the

next table the persons that succeed for the first time (1x), second time (2x) or third time (3x) are


given. There were also two persons that made the entire test, but didn’t succeed for the selftest the

first time, but neglected do to the selftest again (2x*). See next table.

Som

(sec)

User Selftest Som

(sec)

User Selftest

1015 Person 1E 1x 567 Person 2B 1x

1107 Person 1F 1x 718 Person 2C 1x

1166 Person 1G 1x 1355 Person 2D 1x

1586 Person 1I 1x 1374 Person 2E 1x

2018 Person 1J 1x 1421 Person 2F 1x

2206 Person 1K 2x 1450 Person 2G 1x

2318 Person 1L 1x 1527 Person 2H 1x

2401 Person 1M 2x 1583 Person 2I 1x

2653 Person 1N 1x 1588 Person 2J 1x

1624 Person 2K 2x*

1683 Person 2L 1x

2707 Person 2M 2x*

3363 Person 2N 3x

Group 1 Group 2

1830 Average All 1612 Average All

1695 Average 1x 1327 Average 1x

2304 Average 2x 2565 Average 2x* / 3x

The average time for persons that succeed for the selftest the first time is 1695 seconds in group 1

and 1327 seconds in group 2. Group 2, the persons that followed the non-adaptive test, are more

than 6 minutes faster than the persons who followed the adaptive test, and this is not the desired

result. For more information about these result see the next chapter.

The average time for persons that not succeed for the selftest the first time is 2304 seconds in group

1 and 2565 seconds2 in group 2. Group 1, the persons that followed the adaptive test, are 4,5

minutes faster than the persons who followed the non-adaptive test. This is the desired result. For

more information about these result see the next chapter.

2 This number is on the low end, because person 2K and 2M didn’t finish the test, therefore at least 240

seconds can be added to this number.


7.1.3 Pretest Ratio

During the analysis of the results the ratio between the pretest, the actual time persons study the

information pages, and the selftest is necessary, therefore first the average time of the pretest needs

to be established. This can be done with the following SQL statement:

SELECT AVG(SECOND(p1.Kolom1)+60*MINUTE(p1.Kolom1)) AS som, name

FROM p1

WHERE name LIKE 'file:/infosecure1/question*'

GROUP BY name

ORDER BY som DESC;

And the following table is the result.

Som

(sec)

Name

80 file:/infosecure1/question1.xhtml 55 file:/infosecure1/question2.xhtml 18 file:/infosecure1/question3.xhtml 53 file:/infosecure1/question4.xhtml 62 file:/infosecure1/question5.xhtml 37 file:/infosecure1/question6.xhtml 43 file:/infosecure1/question7.xhtml 49 file:/infosecure1/question8.xhtml 66 file:/infosecure1/question9.xhtml

463 Total

Table 32 Average time pretest

By executing a couple of queries the average time per golden rule is calculated as follows:

Average

time (sec)

Golden Rule

66 Golden Rule 1

129 Golden Rule 2

81 Golden Rule 3

99 Golden Rule 4

86 Golden Rule 5

34 Golden Rule 6

38 Golden Rule 7

102 Golden Rule 8

98 Golden Rule 9

733 Total

Table 33 Average time golden rules

The average time of the selftest can be calculated the same way and is:

Average

time (sec)

Selftest


34 Selftest 1

19 Selftest 2

48 Selftest 3

21 Selftest 4

32 Selftest 5

27 Selftest 6

24 Selftest 7

27 Selftest 8

29 Selftest 9

15 Selftest 10

20 Selftest 11

24 Selftest 12

320 Total

Table 34 Average time selftest

7.2 Analyzing the AHA! Profile Logs

In the aha subdirectory xmlroot/profile are all the profile files of the persons that have done the test.

The profile files are named with a number. There is also an index file which has an overview of all the

numbers coupled with the names of the students. This way it is possible to know which profile file

belongs to which student. In these xml formatted files all the latest values of all attributes used are

stored. In this file it is easy to determine which selftest questions were answered correctly (f.i. if the

value of infosecure1.selftest1.temp1 is true, this question was answered correctly) and to find all

latest values of all attributes in the user model.

7.3 Testing the Significance

With the help of two test the significance of the data is tested.

7.3.1 F-test Two-Sample for Variances

The F-test Two-Sample for Variances analysis performs a two-sample F-Test to compare two

population variances. This test can easily be executed with the help of a Excel, this spreadsheet

program has the option to load the analysis toolpak, and with the help of this toolpak the f-test can

be executed. The tool provides the result of a test of the null hypothesis that these two samples

come from distributions with equal variances, against the alternative that the variances are not equal

in the underlying distributions.

The tool calculates the value f of an F-statistic (or F-ratio). A value of f close to 1 provides evidence

that the underlying population variances are equal. In the output table, if f < 1 “P(F<=f) one tail” gives

the probability of observing a value of the F-statistic less than f when population variances are equal,

and “F Critical one-tail” gives the critical value less than 1 for the chosen significance level, Alpha. If

f>1, “P(F<=f) one tail” gives the probability of observing a value of the F-statistic greater than f when

population variances are equal, and “F Critical one-tail” gives the critical value greater than 1 for

Alpha.

7.3.2 T-test Two-Sample

The Two-Sample t-Test analysis tools test for equality of the population means that underlie each

sample. A t-Statistic value, t, is computed and shown as “t Stat” in the output value. If t < 0, “P(T<=t)


one tail” gives the probability that a value of the t-Statistic would be observed that is more negative

than t. If t > 0, “P(T<=t) one tail” gives the probability that a value of the t-Statistic would be observed

that is more positive than t. “t Critical one-tail” gives the cutoff value, so that the probability of

observing a value of the t-Statistic greater than or equal to “t Critical one-tail” is Alpha.

“P(T<=t) two tail” gives the probability that a value of the t-Statistic would be observed that is larger

in absolute value than t. “P critical two-tail” gives the cutoff value, so that the probability of an

observed t-statistic larger in absolute value than “P critical two-tail” is Alpha.

8 Testing Adaptivity

As mentioned in chapter 6, the described module was build with test purposes. A test group (see

chapter 8.1 and 8.2) has followed the adaptive module and another similar test group has followed

the non-adapted module. In chapter 8.3 will be described what were the time benefits for the

adapted version in comparison with the non-adapted version. In chapter 8.4 the HR results will also

be investigated, so be it in a nutshell, because a much larger test-group is necessary for this

investigation.

8.1 Test Group

First of all, the larger the test group, the more reliable the test results will be. Because this will be the

first test, and minor errors are still possible, it is better to start with a test group that is large enough

for correct results, without the risks of showing errors to possible customers. These possible errors

will be detected during this first test. A good number of test persons will be around 40, this is large

enough for reliable information, and good conclusions for testing the pretest adaptation can be

drawn.

The actual test group consists out of InfoSecure employees and customers of InfoSecure. This test

group is divided in two similar test groups. One group follows the non-adaptive version and the other

test group will follow the adaptive version. These two test groups have to be similar, therefore the

InfoSecure employees and the customers of InfoSecure are equally distributed over the two groups.

Because the InfoSecure employees are all well known to InfoSecure, it is possible to divide persons

with the same function over the two groups. This makes sure the two test groups are as similar to

each other as possible. After the tests are done, it will be analyzed in chapter 8.2 if indeed both test

groups were similar to each other.

The groups are divided fifty-fifty, because the results of both tests are equally important. For both

tests the results are extracted from the time taken per page and the sequence order of the pages in

the module. The analysis of the results is completely anonymous. The name (and additional entered

information) of the person will not be used in the primary analysis. The results will be discussed in

chapter 8.3. The additional information entered by the test person (so not his/her name) will be used

for the analysis of the HR results in chapter 8.4. Again this will be a very concise analysis, because the

test group of 25 persons is too little for this purpose.

8.2 Test Group Analysis

25 Persons were invited to make the adaptive test and another 25 to make the non-adaptive test.

Because the test takes around 30 minutes not all persons complied to this invitation, but after

another friendly reminder at least 14 persons in each group made the test and this is sufficient to

draw some first conclusions. The first conclusions that can be made is about the high percentage of

persons that succeed for the selftest. This percentage is high on both test groups. For group 2, the

persons that made the non-adaptive test 10 out of 14 persons succeeded for the selftest the first

time. And in group 1, the persons that made the adaptive test, 7 out of 9 persons that made the


selftest succeeded the first time. These are both high percentages, which possibly are not average,

because most of the test persons are employees of InfoSecure and have an above average

knowledge about Information Security. Therefore the test results in the next subchapter are mainly

based on persons that have sufficient knowledge of Information Security.

8.3 Pretest Results

In this subchapter the results of group 1 (the adaptive test) and group 2 (the non-adaptive test) will

be analyzed. In the previous chapter is explained how the data is extracted and in this chapter the

results are discussed.

The first and most important conclusion is that the desired result of the adaptive test gaining a

overall time benefit is not the case according to the test results. The conclusion can be divided in two

parts. First the persons that immediately succeeded for the selftest (see chapter 8.3.1.1) and

afterwards the persons that didn’t immediately succeed for the selftest (see chapter 8.3.1.2). The

percentages explained in chapter 4.4.2 are discussed in chapter 8.3.1.3 Afterwards a statistical proof

of the results is given (see chapter 8.3.2) and at last the overall conclusion (see chapter 8.3.3).

8.3.1.1 Persons that succeeded for selftest first time

As described in chapter 7.1.2 the average time for persons that succeed for the selftest the first time

is 1695 seconds in group 1 and 1327 seconds in group 2. The test group consisted out of mainly

InfoSecure personnel and they were familiar to most of the topics. Therefore some persons (f.i.

person 2B and 2C) were able to literally click through the information (only briefly observing the

information, and sometimes on own insights take a little more time to really study the information),

and immediately succeed for the final selftest. The persons in group 1 had to follow the pretest. With

this pretest still a high percentage (7 out of 9) succeeded for the selftest immediately, but too much

time was taken for the pretest to get decent percentages as described in chapter 4.4.2.2.

As described in chapter 7.1.3 the ratio for the pretest and the actual information pages is too high. In

the test example in chapter 4.4.1 the ratio is only 240/1350≈0,18 and the current ratio is 0,63.

Therefore the conclusion is that the pretest is taking too much time (in comparison with the other

information pages). People take too much time reading/analyzing the pretest, while it actually is

important that people answer the pretest questions quickly and skip the question quickly if they are

not sure of the answer. This should become more clear to the users of the module, so they take less

time answering the pretest questions.

Some pretest questions on their own can do with little answer alternatives, therefore gaining more

time benefit. The question itself becomes a little easier to answer, but this shouldn’t be a problem if

users only answer in case the know the correct answer for sure.

The ratio is too high, also because the test persons didn’t take as much time as expected for every

golden rule. F.i. the average time of golden rule 7 is 38 seconds, while the business clip alone has a

duration of 93 seconds, not to speak of another complete page with textual information. It is clear

that not all users view the business clip and do not completely read the information on the pages.

That being said the conclusion is that persons that are familiar to the information (which the majority

of the test persons apparently was) click through the information rather quick and still succeed for

the selftest.

Another problem with the current module is that the subject is not too technical or specific, and with

some common sense most questions can be answered. Therefore possibly questions that ask for the

knowledge estimation instead of questions that actually test the knowledge of the student are

necessary. Important in this case is that the student has to estimate his own knowledge really

well/honest. If not, he will fail for the selftest and will definitely make no time benefit. In case of a

more technical module a pretest with knowledge questions is the way to go. In this case the

questions are too difficult to answer if the student is not familiar with the content and easy to

answer if the student is familiar with the information. Therefore the pretest will not take too much

time.


To summarize the pretest in his current form does not guarantee a time benefit for persons that are

familiar to most of the information because:

1) Persons that are familiar to all the information are able to click through the information

rather quick, and make more time profit than with the help of this pretest.

2) The pretest takes too much time in comparison with the information pages. The ratio is

around 63% and ideally should be around 10%. The solution for creating a better pretest is:

a. Make sure the people know that the pretest is not the selftest and that incorrect

answers are no problem. People have to really know the answers to the pretest

question, otherwise just skip it and save time;

b. Shorten the length of the pretest questions, by changing the question or the number

of answer alternatives.

c. Change its form depending on the module (technical or not)

8.3.1.2 Persons that didn’t succeed for selftest first time

Persons that didn’t succeed for the selftest the first time, have a time benefit with the adaptive test.

2206 seconds vs. 2565 seconds. This is because the selftest is adaptive as well and questions of the

selftest that were rightly answered the first time, aren’t asked again in the selftest the next time.

Another great advantage of the adaptive module is that only the golden rules related to the wrongly

answered selftest question are repeated in the module the second (or even third or more) time. A

big side note is that the test data for persons that didn’t succeed for the selftest the first time is

rather small, so an extra investigation is necessary to validate this outcome a 100%. But expected is

that for people not too familiar with all the content, a pretest makes a solid time benefit, especially

when the ratio of the pretest to the information pages as described above becomes smaller.

8.3.1.3 Success and Pre-test Question Correctness Percentage

The pretest correlation percentage and the success percentages (see chapter 4.4.2.2) are highly

depend on the time of the pre-test questions. Since all the questions of the pretest are taking too

much time in comparison with the Information Pages and the selftest, it’s no use to calculate these

percentages, because even without calculation it is clear that the percentages will be too high or (in

case the pretest takes more time than the Information page) negative/impossible. When the pre-test

is adjusted, these percentages need to be calculated as explained in chapter 4.

8.3.1.4 Selftest Adjustments

As explained in chapter 4.4.2.3, there is possibly a group of students that succeed for a pretest

question and fail for the corresponding selftest question(s), but succeed for the complete selftest.

This is not a desired situation and with the help of the AHA! profiles, this can be analyzed for the

current test sample (see chapter 7.2). Seven persons succeeded for the selftest, and 2 of them

succeeded with answering at least one question of the selftest incorrect, without ever viewing the

according information pages. According to these numbers, it is better to adjust the selftest and let

these specific students redo parts of the test, to improve their knowledge.

8.3.2 Statistical Proof of Results

All the conclusions are based on the fact that the results of the two tests are significant enough. This

of course needs to be proven, and this can be done with the help of the f-test explained in chapter

7.3.1 and t-test explained in chapter 7.3.1.


The results for the f-test of group 1 and 2 with alpha (significance level) equaling 0,05 are:

Group 1 Group 2

Mean 1830 1612,307692

Variance 388202,5 529627,5641

Observations 9 13

F 0,732972614

P(F<=f) one-tail 0,33724584

F Critical one-tail 0,304512355

As explained above, F is 0,73, so this proves that the population variances are not equal, but this

conclusion cannot be drawn from this sample, because the value of P(F<=f) one-tail is much to great

and therefore the population variances of the two groups need to be considered equal. But with

more test results, it might very well be possible that the population variances are different for both

groups.

More important is the equality of the population means, which can be calculated with the help of the

t-test explained in chapter 7.3.1. The results for this test are as follows:

Group 1 Group 2

Mean 1830 1612,308

Variance 388202,5 529627,6

Observations 9 13

T Stat 0,751672923

P(T<=t) one-tail 0,230729075

t Critical one-tail 1,729132792

P(T<=t) two-tail 0,46145815

t Critical two-tail 2,09302405

T Stat is 0,75. Therefore the population means of the two groups are not equal. But again P(T<=t)

one-tail is 0,23 and P(T<=t) two-tail is 0,46. Which both are way too high and therefore the null

hypothesis is rejected. Even though the mean of group 1 is 218 seconds more in this test sample,

this difference is no significant difference because of the high dispersion.

It would be possible to find a significant difference, but therefore the variation needs to be much

lower in the sample data.

8.3.3 Overall Conclusion

Overall the mean of the adaptive test is equal (according to t-test) to the mean of the non-adaptive

test. This has a couple of reasons fully described above. Another reason the adaptive test didn’t gain

time as expected, is actually also described above. The test group consisted out of many persons that

did succeed for the selftest the first time, and consisted out of few persons that didn´t. This ratio is

also a factor of the overall result, because persons that don´t succeed for the selftest the first time

will probably gain time by following the adaptive module.


8.4 HR Results

There are no Human Resources results with this current test, because the test group is too small.

Before analyzing the HR results, first the pretest should be adjusted with the above mentioned

improvements. During this analysis it is also necessary to investigate the HR data of the persons that

did the non adaptive test and clicked through the information really quickly and succeeded for the

selftest (in this case persons 2B and 2C). Because these persons are better of immediately making the

selftest as well.

For the adaptive test analyze the results of the pretest answers with the HR information of the

student as described in chapter 4.4.3.


9 Conclusion

After finishing this thesis multiple conclusions can be drawn, but there are also some conclusions

that cannot be drawn yet according to this research. Before investigating the similarities between the

HR attributes of a student and the answers of his pretest questions as explained in chapter 4, the

current pretest needs to be improved as suggested in chapter 8. The pretest in its current form didn’t

lead to time benefit, this is why it needs to be adjusted. But still even with the adjusted pretest the

question remains if the current module is suitable for this kind of adaptation. Better adaptation is

definitely possible for a more technical module, which actually was the first intention of InfoSecure,

but was changed because of marketing reasons as explained in chapter 2.

Definitely more time benefit can be gained with the help of adaptive E-Learning on technical

subjects, but even on the less technical subjects time benefits can be gained, albeit smaller ones. To

test these time benefits a larger test group than used in chapter 8 is necessary. This is because with a

larger test group, smaller time benefits can be found.

AHA! is a suitable program to develop the adaptive E-learning programs as described in chapter 6,

and because of the automatic creation of logs, it is also ideal for test purposes as described in

chapter 7. In chapter 3 is described that with the help of the navigation and sequencings possibilities

of SCORM it is also possible to create an adaptive e-learning module with a pretest. But

unfortunately this is not personalized, so adjustments in the actual code need to be made to store

values for each student. Eventually with this adjustment it will become possible to link the HR

attributes of the student to the sequencing and navigation possibilities of the module and create the

desired adaptive e-learning module. Using SCORM content doesn’t automatically create log files, so

for test purposes additional code needs to be written as well. Therefore AHA! has the upper hand in

creating and testing adaptive e-learning modules, because of its simplicity and built-in possibilities.


10 References

Advanced Distributed Learning (ADL) SCORM® 2004 3rd Edition Content Aggregation Model (CAM)

version 1.0 [Report]. - 2006.

Advanced Distributed Learning (ADL) SCORM® 2004 3rd Edition Run-Time Environment (RTE) version

1.0 [Report]. - 2006b.

Advanced Distributed Learning (ADL) SCORM® 2004 3rd Edition Sequencing and Navigation (SN)

Version 1.0 [Report]. - 2006c.

Advanced Distributed Learning (ADL) Sharable Content Object Reference Model (SCORM)® 2004 3rd

Edition Overview version 1.0 [Report]. - 2006a.

AHA! Tutorial [Online]. - 2008. - http://aha.win.tue.nl.

Cristóbal Romero Morales Sebastián Ventura Soto, Cesar Hervás Martínez, Paul de Bra Extending

AHA! [Report]. - 2005.

De Bra Paul [et al.] AHA! The Adaptive Hypermedia Architecture [Report]. - 2003.

Derek Stockley [Online]. - 2008. - http://derekstockley.com.au/elearning-definition.html.

Merriam-Webster [Online]. - 2008. - http://www.merriam-webster.com/dictionary/.

W3C's overview of Web style sheets: CSS. [Online] // World Wide Web Consortium - Web

Standards. - 2009. - http://www.w3.org/Style/CSS/.

XHTML 1.0: The Extensible HyperText Markup Language (Second Edition) [Online] // World Wide

Web Consortium - Web Standards. - 2009. - http://www.w3.org/TR/xhtml1/.


Appendices

Appendix A: Selftest.xhtml

<!DOCTYPE html SYSTEM "/aha/AHAstandard/xhtml-ahaext-1.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<link href="stylesheets/infosecure.css" rel="stylesheet" type="text/css" />

<title>Selftest</title>

</head>

<body>

<h1>Selftest</h1>

<div class="left">

<if expr="!infosecure1.selftest12.done">

<block>

<p><if expr="(infosecure1.goldenrule1.suitability || infosecure1.goldenrule2.suitability ||

infosecure1.goldenrule3.suitability || infosecure1.goldenrule4.suitability ||



infosecure1.goldenrule9.suitability) == false">

<block>

You have answered all the pretest questions correctly and therefore immediately can do the

selftest.

</block>

<block>

<if expr="(infosecure1.goldenrule1.visited == 0 &&

infosecure1.goldenrule1.suitability) || (infosecure1.goldenrule2.visited == 0 &&








infosecure1.goldenrule9.suitability)">

<block>

You've not read through all the golden rules yet, you're advised to read through the

following golden rules, before taking the test:<br />

<if expr="infosecure1.goldenrule1.visited == 0 &&

infosecure1.goldenrule1.suitability"><block><a href="infosecure1.goldenrule1"

class="conditional">Golden Rule 1</a><br /></block></if>


























</block>

<block>

Now that you have read through all the necessary information, you can do the selftest.

</block>

</if>

</block>

</if>

With the test you can establish for yourself whether you have understood everything, the

level of your knowledge and how you deal with information security in practice.</p>

<p>The test consists of 12 questions, if you have given too many incorrect answers, you will

have failed the test. You will have to do (part of) the test again, after you have finished

(part of) the module again. </p>

<p>If you answered enough questions correct, you have passed the test and your manager will

be informed.</p>

<p>After pressing the "next question" button, it is stated if you have answered the question

correctly or not.</p>

<p>Good Luck</p>

<p class="nextlink">

<if expr="infosecure1.selftest1.suitability"><block><a href="infosecure1.selftest1"

class="conditional">next page</a></block>

<block>



<block>



<block>



<block>



<block>



<block>



<block>



<block>



<block>



<block>



<block>



</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>


</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>.

</p>

</block>

<block>

You are familiar with the selftest, so no explanation is needed. Only the questions are

asked which were wrongly answered the previous time.<br /><br />

Answer enough questions correcly and you will succeed for the test.

<p class="nextlink">



<block>



<block>



<block>



<block>



<block>



<block>



<block>



<block>



<block>



<block>



<block>



</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>

</block>

</if>.

</p>

</block>


</if>

</div>

<div class="right">

<img src="images/conclusion.jpg"/>

</div>

</body>

</html>

eindhoven university of technology master personalized e … · vaessen, d. award date: 2009 link...

Documents