18/5/20071 some initial analysis and observations dsa review seminar stirling, friday, 18 may 2007...

18/5/2007 1

Some initial analysis and observationsDSA Review Seminar

Stirling, Friday, 18 May 2007

Seb Schmoller Sero

Baseline study of e-activity in Scotland’s colleges: DSA in

Scotland’s colleges

18/5/2007 2

Insights arising from DSA

1. Some throat-clearing (1 slide)2. A refresher on DSA terms (3 slides)3. Scotland’s Colleges as a whole,

and some general insights (3 slides)

4. Individual process areas, and some more specific insights (~6 slides)

5. Summary (1 slide)

18/5/2007 3

Some throat-clearing

1. 2-way process2. Tentative 3. Integration not yet done

between data from:– the Supplementary Questionnaire– Stakeholder Interviews– DSA

18/5/2007 4

Refresher 1Institutional capability…

“…‘Capability’ refers to the ability of a college to ensure that e-learning design, development & deployment is meeting the needs of the students, staff and institution, including the ability of an institution to sustain e-learning as demand grows and staff change.”

“… the assessment of capability is complex, and necessarily involves reducing large amounts of detail into a broad overview to support decision making in the college.”

18/5/2007 5

Refresher 2Five “process areas”:• Learning; Development; Support;

Evaluation; Organisation(5-3 process statements per process area)

Five “dimensions of capability”:• Policy; Planning; Delivery; Management;

Improvement(Up to 4 practice statements per dimension per process statement)

Each practice statement self-judged as:• Not adequate; Partially adequate; Largely

adequate; or Fully adequate

18/5/2007 6

Refresher 3

DSA

Po

licy

Pla

nn

ing

Del

iver

y

Man

agem

ent

Imp

rove

men

t

LearningL1: e-Learning is designed according to pedagogically sound principles N P L F PL2: Learning skills are developed explicitly & incrementally during delivery P F P P PL3: Peer & tutor interactions are provided explicitly during e-learning delivery P P N P PL4: Courses take account of student workload and e-assessment demands P P N L PL5: e-learning actively engage students P P P P NL6: College defined (optional) n/a n/a n/a n/a n/aDevelopmentD1: Course development is directed and supported P P L F PD2: Resources are maintained and re-used P F P P PD3: Courses meet the needs of all students P P N P PD4: The e-learning infrastructure is reliable, robust and sufficient P P N L PD5: The e-learning infrastructure is standardised and integrated P P P P ND6: College defined (optional) n/a n/a n/a n/a n/a

18/5/2007 7

A whole country overviewLearning     Development   Support     Evaluation   Organisation  

Pol Plan Del Mgt Imp Pol Plan Del Mgt Imp Pol Plan Del Mgt Imp Pol Plan Del Mgt Imp Pol Plan Del Mgt Imp3.40 3.00 3.40 3.20 3.20 3.20 3.20 3.60 3.20 3.40 3.40 3.40 3.80 3.40 3.60 4.00 3.00 4.00 4.00 4.00 3.60 4.00 3.80 4.00 3.40

3.00 2.80 3.40 2.80 3.40 3.40 3.20 3.00 3.40 3.80 3.40 3.60 3.80 3.60 3.80 2.67 3.33 3.33 4.00 4.00 3.40 3.40 3.40 3.20 3.40

3.00 3.20 3.40 2.80 3.60 3.00 3.40 3.60 2.80 3.60 3.80 3.40 3.40 3.20 3.20 2.67 3.00 3.00 3.00 3.33 2.80 3.00 3.00 2.80 2.80

3.40 2.80 3.80 3.20 2.60 3.80 3.20 2.60 3.20 3.40 3.33 3.00 3.40 2.50 2.50 4.00 2.67 3.33 3.67 2.67 3.25 3.20 3.20 2.60 2.80

2.80 3.00 3.40 2.60 3.00 3.20 3.20 3.00 3.20 3.80 3.00 3.20 3.80 3.80 3.80 2.00 2.33 2.33 2.33 3.00 2.75 3.25 3.25 2.25 3.50

3.00 2.60 3.20 3.00 2.40 3.40 3.00 3.40 3.20 3.00 2.80 3.00 3.40 2.80 3.40 2.33 2.67 3.00 2.67 2.67 3.40 3.60 3.80 2.40 3.00

3.00 3.00 3.40 2.60 3.20 2.00 2.00 2.20 2.20 2.40 3.20 3.40 3.40 3.00 3.00 3.33 3.33 2.67 3.33 3.00 2.80 3.40 2.80 2.40 2.40

2.20 2.60 3.20 2.60 2.40 2.40 2.40 2.80 2.60 2.80 2.60 2.80 3.00 2.80 2.80 2.67 2.67 3.00 3.00 3.00 3.00 2.40 2.80 2.60 2.40

2.20 2.20 3.00 2.60 1.80 2.20 2.00 2.60 2.40 2.40 2.80 2.00 3.00 2.80 2.40 3.67 3.33 3.67 3.33 3.00 2.60 2.60 3.00 2.80 2.40

1.40 1.60 2.80 2.00 2.40 1.80 2.00 2.40 2.40 2.20 2.80 2.80 3.20 3.50 3.25 2.00 2.33 2.33 2.67 2.00 4.00 3.20 3.60 3.40 3.00

2.20 2.20 3.00 2.20 2.60 2.20 2.20 2.80 2.20 2.60 2.40 2.60 3.60 2.40 2.60 1.67 2.67 2.67 2.67 3.00 2.60 2.80 2.60 2.40 2.40

1.60 2.20 3.40 2.20 2.80 2.40 2.40 2.80 2.00 2.20 2.75 3.00 3.40 3.00 2.20 2.00 2.00 2.67 1.67 2.00 2.80 2.80 2.60 2.40 2.80

1.20 1.80 2.40 1.80 2.20 2.00 2.00 2.60 1.80 2.40 2.80 2.60 3.20 2.20 2.40 3.33 1.67 2.33 1.67 2.33 3.00 2.80 2.60 2.60 2.60

1.60 2.20 3.40 2.60 2.00 2.20 2.00 3.00 2.40 2.20 1.80 2.40 3.20 2.60 1.60 3.00 2.67 2.67 2.67 2.00 1.80 2.20 1.80 1.60 1.80

2.20 2.20 2.80 2.20 2.40 1.60 1.60 2.20 1.00 1.20 2.40 2.40 3.00 1.75 2.40 2.67 2.67 2.33 2.67 3.00 2.60 2.40 2.60 2.40 2.60

1.60 2.00 3.00 2.00 2.20 2.20 2.00 2.40 1.40 1.80 2.80 2.60 3.20 2.00 1.60 2.00 3.00 3.33 2.33 2.67 2.00 2.00 2.20 1.80 1.60

2.20 2.80 3.40 2.40 2.60 2.20 2.00 2.60 1.60 2.00 1.60 2.20 3.40 2.60 2.20 2.00 2.00 2.00 1.67 1.67 2.40 2.60 2.20 1.40 1.80

2.00 2.80 3.40 2.40 2.60 2.00 2.20 2.80 2.00 2.20 2.80 2.80 3.00 1.80 2.00 2.00 2.00 1.67 1.00 1.00 2.00 2.20 2.20 1.60 1.60

1.60 2.00 2.40 1.80 2.00 2.80 2.00 1.80 1.80 2.00 2.75 3.00 3.25 2.75 2.75 1.00 1.33 1.67 2.00 2.00 2.20 2.40 2.20 2.00 2.20

2.00 2.25 2.80 2.00 2.50 2.50 2.75 3.00 1.00 2.00 2.00 2.40 3.40 3.00 1.25 1.00 1.33 1.33 1.00 1.33 2.67 2.40 2.60 1.40 1.75

1.50 1.80 2.40 1.40 1.40 2.00 2.00 2.40 1.60 1.60 2.75 2.60 2.40 2.00 2.00 2.50 2.00 2.00 2.00 2.00 3.00 1.75 2.00 2.00 2.00

1.40 1.60 2.60 1.60 2.00 1.40 1.80 2.20 1.60 1.80 1.80 2.00 2.60 1.80 1.60 1.33 1.33 2.33 2.00 2.67 2.00 2.20 2.20 1.20 2.00

2.40 1.60 2.00 1.60 1.40 2.00 2.20 2.00 1.40 2.20 3.00 2.00 2.60 2.60 1.80 1.67 1.67 2.00 1.33 1.33 2.00 1.80 1.40 1.20 1.40

1.80 2.20 2.60 2.00 1.80 1.40 1.60 1.80 1.80 1.60 2.20 2.60 3.00 1.80 1.40 1.33 1.67 1.67 2.00 1.67 1.40 1.80 2.00 1.40 1.20

1.40 1.60 2.20 1.40 1.60 1.80 2.20 2.20 2.00 2.00 2.20 2.40 3.00 2.00 2.20 2.00 1.33 1.33 1.00 1.67 2.00 2.00 1.25 1.20 1.20

1.20 1.40 2.80 1.60 1.80 1.40 1.60 2.00 2.00 1.60 2.20 2.20 2.80 2.60 2.00 1.33 1.33 1.67 1.33 1.67 1.60 2.00 1.80 1.40 1.80

1.60 1.80 2.80 1.40 2.25 2.40 2.20 2.00 1.40 1.60 2.00 2.00 2.60 1.80 1.20 1.33 1.00 2.00 1.00 1.33 1.80 1.80 2.00 1.40 1.60

1.40 1.80 2.20 1.40 1.40 2.00 2.20 2.60 1.20 1.80 2.00 2.20 3.00 2.20 1.80 1.33 1.33 1.67 1.00 1.00 2.00 1.60 1.80 1.60 1.60

1.40 1.20 2.60 1.80 2.00 1.20 1.20 1.60 1.60 1.80 2.00 2.00 2.20 2.00 1.80 1.67 2.00 1.67 1.33 1.67 1.40 2.00 2.20 1.60 1.80

1.40 1.60 2.00 1.80 2.20 2.00 2.00 2.00 1.40 2.00 2.00 2.20 2.60 1.40 1.00 1.33 1.00 1.67 1.00 1.67 1.60 1.80 2.40 1.80 1.40

1.60 2.00 2.20 1.20 1.80 1.60 1.60 2.20 1.20 1.75 2.40 2.00 2.80 2.00 1.40 1.67 1.67 1.33 1.00 1.00 1.50 1.80 2.00 1.60 1.60

1.20 2.00 2.60 1.60 2.40 1.00 1.40 1.80 1.40 1.80 1.60 1.60 2.00 1.60 2.20 1.33 2.33 1.67 1.67 2.33 1.00 1.20 1.20 1.40 1.80

1.80 1.80 2.40 2.40 2.00 1.40 1.60 1.60 1.20 1.20 1.60 1.80 2.20 1.60 1.40 1.33 1.33 1.00 1.33 1.67 1.00 1.20 1.80 1.20 1.40

1.20 1.00 2.80 1.20 1.60 1.20 1.40 1.60 1.20 1.00 2.20 2.40 3.00 2.20 1.80 1.00 1.00 1.00 1.00 1.00 1.80 1.80 2.00 1.00 1.40

1.80 1.40 2.20 1.80 1.40 1.00 1.00 1.40 1.20 1.20 2.20 2.20 2.60 2.40 2.20 1.33 1.00 1.33 1.00 1.00 1.00 1.20 1.20 1.00 1.00

1.00 1.20 1.80 1.40 1.80 1.00 1.60 1.80 1.20 1.20 1.40 2.20 2.00 1.80 1.60 1.00 1.00 1.33 1.00 1.33 1.40 1.60 2.00 1.40 1.80

1.40 1.40 1.40 1.00 1.00 1.40 1.40 1.60 1.40 1.20 1.40 1.60 2.00 1.40 1.40 1.33 1.00 1.00 1.00 1.00 1.40 1.40 1.80 1.40 1.60

1.20 1.00 1.80 1.00 1.00 1.00 1.20 1.40 1.00 1.20 1.40 1.40 2.00 2.00 1.00 1.00 1.00 1.00 1.00 1.00 1.20 1.00 1.00 1.00 1.00

1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.20 1.00 1.00 1.20 1.40 2.00 1.20 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

>3 Green; >2 White; >1 Ochre; = 1 Red

18/5/2007 8

A “landscape” view of the same data

1 5 9

13

17

21

25

29

33

37

Learning Pol

Learning Mgt

Development Plan

Development Imp

Support Del

Evaluation PolEvaluation MgtOrganisation PlanOrganisation Imp

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

A landscape view of the e-capability of Scotland's Colleges

3.50-4.00

3.00-3.50

2.50-3.00

2.00-2.50

1.50-2.00

1.00-1.50

0.50-1.00

0.00-0.50

18/5/2007 9

General observations

• Few colleges self-audit their capability at mainly 1; few at mainly >3; a substantial proportion do so mainly at 1-2

• No discernable variation by class of college (big/small, urban/rural, rebuilt or not, HE%, etc)

• Thus: a small college can be e-capable; a college that is not new can be e-capable; etc.

18/5/2007 10

Two individual process areasProcess - Support

0% 20% 40% 60% 80% 100%

Policy

Planning

Delivery

Management

Improvement

Dim

ensi

on

s

Overal assessment

>3.00

>2.00

>1.00

<=1.00

Process - Evaluation

0% 20% 40% 60% 80% 100%

Policy

Planning

Delivery

Management

ImprovementD

imen

sio

ns

Overal assessment

>3.00

>2.00

>1.00

<=1.00

18/5/2007 11

Looking deeper 1/5 – Development

Development Process Statements

1.0

1.5

2.0

2.5

3.0

3.5

D1: Coursedevelopment isdirected andsupported

D2: Resources aremaintained and re-

used

D3: Courses meetthe needs of all

students

D4: The e-learninginfrastructure isreliable, robustand suff icient

D5: The e-learninginfrastructure is

standardised andintegrated

Policy

Planning

Delivery

Management

Improvement

18/5/2007 12

Looking deeper 2/5 – Support

Support Process Statements

1.0

1.5

2.0

2.5

3.0

3.5

S1: Technicalsupport is

available forstudents

S2:Library/LearningResource Centre(LRC) servicesare available for

students

S3: Studentconcerns andcomplaints are

collected,analysed and

acted on

S4: Support inpersonal and

learning issues isavailable tostudents

S5: Support in e-learning delivery is

provided forteaching staff

Policy

Planning

Delivery

Management

Improvement

18/5/2007 13

Looking deeper 3/5 – Organisation

Organisation Process Statements

1.0

1.5

2.0

2.5

3.0

3.5

O1: e-Learningprovision is

managed throughformal business

processes

O2: Institutionallearning &

teaching policyexplicitly

addresses e-learning

O3: Students areinformed ofpedagogical

rationale,procedures and

technologies

O4: Administrativerequirements are

effectivelycommunicated to

students

O5: Electronicinformation

handling is subjectto policies and

guidelines

Policy

Planning

Delivery

Management

Improvement

18/5/2007 14

Looking deeper 4/5 – LearningLearning Process Statements

1.0

1.5

2.0

2.5

3.0

3.5

L1: e-Learning isdesigned

according topedagogically

sound principles

L2: Learning skillsare developed

explicitly &incrementally

during delivery

L3: Peer & tutorinteractions are

provided explicitlyduring e-learning

delivery

L4: Courses takeaccount of studentw orkload and e-

assessmentdemands

L5: e-learningactively engages

students

Policy

Planning

Delivery

Management

Improvement

18/5/2007 15

Looking deeper 5/5 – Evaluation

Evaluation Process Statements

1.0

1.5

2.0

2.5

3.0

3.5

E1: Students provide systematicfeedback on their experience of

e-learning

E2: Teachers providesystematic feedback on their

experience of e-learning

E3: Course teams carry outsystematic review s of theeffectiveness of e-learning

Policy

Planning

Delivery

Management

Improvement

18/5/2007 16

Summary

• Wide variation between colleges• Some of the variation due to colleges

differing on how hard they are on themselves

• No clear evidence of correlation between different “class” of college and e-readiness

• Interesting differences between aggregated results for process areas, and between individual process statements within these