bpm seminar understandability of process models a user’s perspective on learning

BPM Seminar

understandability of process models

A user’s perspective on learning

Topics

1. Context of the research

2. Theoretical Background

3. Theoretical Design & Conceptualisation

4. Measurement

5. Questions?

1. Context of the research

Goal:

Defining understanding as a result of learning, how do differences in user

characteristics contribute to differences in understanding process models?

An example

Kick-off

• Curtis et al. (1992)

Goal:

“facilitate human understanding and communication”

Sub-goal:

“represent processes in forms understandable by humans”

Gap-analysis

• Independent variables

The effect of:

task complexity

technology/process model characteristics

on performance

Visualisation

Human

Data Model

Task

Performance

Gap-analysis II

• Understandability defined as an intrinsic property of a process model

• Complexity, learnability, usability, etc.

• Pragmatic quality

Why is this important?

• Practically:) helps representing processes in forms understandable by humans) additional insights allow for more effective

training

• Academically:) effects of users often averaged out by assuming homogeneity (or x vs. y)) Maturity of the discipline will benefit from an interdisciplinary approach

2. Theoretical background

“To learn is to be human”

(Goward)

Theory of Cognitive Load

• Define information integration as a learning process

• Control factors of:Content

&Content Presentation

An example

WOOL

MEDDOW

GRASS

GOAT

Learning Conceptualised

PRESAGE PROCESS PRODUCT

USERCHARACTERISTICS

LEARNING CONTEXT

KNOWLEDGECONSTRUCTION

LEARNINGOUTCOMES

Revisions

• Controlled Learning Context

• Limited Mutuality

USERCHARACTERISTICS

KNOWLEDGECONSTRUCTION

LEARNINGOUTCOME

Controlling the Learning Context

HOW?

3. Theoretical Design& Conceptualisation

a Cognitive Perspective onLearning

Structure

• Gap Analysis

• Introduction of a framework

• Integration into learning theory

Prior to Learning

USERCHARACTERISTICS

KNOWLEDGECONSTRUCTION

LEARNINGOUTCOME

User CharacteristicsGap analysis

distalvariables

affective variables psychosocial variables

skills & expertise

Agarwal et al., 1999 Cognitive Fit

Aranda et al., 2007 Domain ExpertiseLanguage Expertise

Bandara et al., 2005 User Competence

Chiew & Wang, 2004

Syntactic KnowledgeSemantic KnowledgeSchematic KnowledgeStrategic Knowledge

Maes & Poels, 2007 Perceived Ease of UsePerceived Usefulness

Perceived Semantic QualityUser Satisfaction

Mendling et al., 2007

ExperienceDomain Expertise

Patig, 2008 AgeGender

ExperienceDomain Knowledge

Reijers & Mendling,2008

EducationJob type

Domain KnowledgeCompany Experience

Field Experience

Input: User Characteristics

• Identification of a framework which conceptualises user characteristics in an integrative manner

• Problem-solving is inherent to human nature

Usage of goal-setting behavioural theory

Goal-setting Behavioural Theory

Eight VariablesPresage

User Characteristics

Learning Context

Self-Efficacy

Skills

Attitude

Traits,Beliefs

SubjectiveNorm

PositiveAnticipatedEmotions

NegativeAnticipatedEmotions

PerceivedBehavioural

Control

Process

LearningStrategies

GoalMotives

GoalIntention

Knowledge Construction

Product

LearningOutcome

NoLearning

MeaningfulLearning

FragmentedLearning

During Learning

USERCHARACTERISTICS

KNOWLEDGECONSTRUCTION

LEARNINGOUTCOME

Knowledge ConstructionGap analysis

• Static Approach; hence limited availability

Cardoso et al. (2006): short- and long-term memory

chunking and tracing

Hungerford et al. (2004):

Task Planning

Others, e.g.: Central vs. Peripheral, Queues

Throughput: Knowledge Construction

• Identification of a framework that describes Knowledge Creation

• Mandviwalla & Hovav (1995)

Motivational process

Motivational Hub (Locke, 1991)

Motives Goals Strategy

Personal Agency Beliefs &

Expectancy

Volition

Volitional dimension

Goal

Approach to Learning

Commitment

Three VariablesPresage

User Characteristics

Learning Context

Self-Efficacy

Skills

Attitude

Traits,Beliefs

SubjectiveNorm

PositiveAnticipatedEmotions

NegativeAnticipatedEmotions

PerceivedBehavioural

Control

Process

LearningStrategies

GoalMotives

GoalIntention

Knowledge Construction

Product

LearningOutcome

NoLearning

MeaningfulLearning

FragmentedLearning

After Learning

USERCHARACTERISTICS

KNOWLEDGECONSTRUCTION

LEARNINGOUTCOME

Learning OutcomeGap analysis

Cognitive Performance Behavioural Performance

Affective Performance

Agarwal & Karahanna, 2000

Behavioural Intention to Use

Aranda et al., 2007

Correctness of understanding Time to completion ConfidencePerceived Difficulty

Goodhue et al., 2000

Time to completion User Evaluation of consistencyUser Evaluation of training

Igbaria et al., 1997

System Usage

Mendling et al., 2007

Number of questions correct Perceived understandability

Recker & Dreiling, 2007

Number of questions correctCloze Recall testProblem Solving task

Time to task completion Ease of Understanding

Venkatesh & Bala, 2008

Use Behaviour

Output: Learning Outcome

• Identification of a framework which distinguishes between different types of

understanding

Motivational Hub (Locke, 1991)

Test Performance

Learning Outcome Cognitive Description Retention Transfer

No learning No knowledge Poor Poor

Rote learning Fragmented knowledge Good Poor

Meaningful learning Integrated knowledge Good Good

Three VariablesPresage

User Characteristics

Learning Context

Self-Efficacy

Skills

Attitude

Traits,Beliefs

SubjectiveNorm

PositiveAnticipatedEmotions

NegativeAnticipatedEmotions

PerceivedBehavioural

Control

Process

LearningStrategies

GoalMotives

GoalIntention

Knowledge Construction

Product

LearningOutcome

NoLearning

MeaningfulLearning

FragmentedLearning

Time Out

Variables Explained

• Eight Presage

• Three Process

• Three Product

User Characteristics

Categories Variables Sub-variables

Distal Variables Distal Variables Demographics & CultureAttitude towards targetsPersonality traits Exposure to mediaOther individual difference variables

Affective Variables AttitudePositive Anticipated EmotionsNegative Anticipated Emotions

Psychosocial Variables Subjective NormPerceived Behavioural ControlSelf-efficacy

Skills and Expertise Skills and Expertise Syntactic SkillsSemantic SkillsPragmatic SkillsExpertise

Knowledge Construction

Factors Variables Sub-variables

Approach to Learning Goal Motives Fear of failureAim for qualificationIntrinsic interestCommitment to work

Goals PerformanceLearning

Learning Strategies SurfaceDeepNon-directed

Learning Outcome

Variables

No learning

Rote learning

Meaningful learning

4. Measurement

Configuration

• What to measure?

• How to measure?

• Who to sample?

What to measure?• Due to model complexity,

test a part of the model:

a) Personal data prior to learning

b) Information of their learning approach during learning

c) Questions on understanding post learning

a) Personal Data

• What to collect?

Distal Variables: Personality?Demographics?Culture?

Skills: Reading skills?Studying skills?Modelling skills?

Examples of Frameworks

• Frederiks & Weide (2006) Analysis Skillsincl. Handle implicit knowledge, grammatical analysis, abstract sentence structure, think on an abstract level

• Bandara et al. (2007) Content for IS SubjectsFrom focus group: analytical skills, understand the problem, ability to communicate with client

• Lindland et al. (1994) QualitySyntactic, Semantic & Pragmatic Quality

• Vanderfeesten et al. (2007) Complexity metricsincl. Coupling, Cohesion, Modularity

Skills• Syntactic Skills a) Word comprehension

b) Vocabulary & syntaxII) Identify obstacles

• Semantic Skills c) Spatial abilityc) BMP modelling discoursed) experience/past behavioural skillsI) studying habitsII) identify obstacles

• Pragmatic Skills e) Working memorye) Integration capacityf) Real time self-evaluationI) Develop effective strategiesII) Problem-solvingIV) Emotional stability

b) Learning Approach

• Methods available

Yet, is this useful?

c) Understanding

• Methods available

• Only cognitive measurement?Indicators:Recall & Transfer

Question answered correctly Problem solving

Time?

• Incorporation of affective constructs

Cognitive Performance Behavioural Performance Affective Performance

How to measure?

Qualitative vs. Quantitative

+

Ability to measure all skills

Control over context

Scope

No additional layer of interpretivism

Fixed vs. Loose

• Should respondents be provided with a goal?

• Should meaningful learning be the outcome to aim for?

Tests, tests, tests

some examples• Spatial ability test• Working memory

integration test• Reading ability• Understanding of story• Problem solving• Cloze Recall test• Cognitive Coupling

Who to sample?

• Controlling the Learning Context

vs.homogeneity of population

• Assume group differences (e.g. expert/novice)

vs.differences by measurement

Encore un foisPresage

User Characteristics

Learning Context

Self-Efficacy

Skills

Attitude

Traits,Beliefs

SubjectiveNorm

PositiveAnticipatedEmotions

NegativeAnticipatedEmotions

PerceivedBehavioural

Control

Process

LearningStrategies

GoalMotives

GoalIntention

Knowledge Construction

Product

LearningOutcome

NoLearning

MeaningfulLearning

FragmentedLearning

5. Questions?

?