Evaluating the Impact of an eLearning Strategy on the Quality of Teaching and

Learning.

Dr. Iain Doherty

Associate Professor

Director eLearning Pedagogical Support Unit

Centre for the Enhancement of Teaching and Learning

16th May 2012

Overview

• HKU’s eLearning Strategy.

• Quantitative reporting.

• What is quality in learning?

• What is quality in eLearning.

• What would a program of evaluation look like?

• Concluding remarks.

2

eLearning Strategy Level One

• Teaching and learning is assisted by technology for efficiency and cost-effectiveness. Students are provided with on-line access to: – Information regarding their programmes of study,

including administrative procedures relating to course enrolment, assessments, degree audit, student evaluation of teaching and learning (SETL); and

– Resources for learning, including all course outlines and digital materials, course learning outcomes and assessments.

eLearning Strategy Level One

eLearning Strategy Level Two

• Teaching and learning is enriched by technology through enhancing opportunities for active learning within and beyond the classroom, provision of links to digital library resources, provision of just-in-time formative and summative feedback and assessment for learning, fostering teaching-research nexus, establishing a closer link with schools, community partners and employers.

eLearning Strategy Level 2

6

eLearning Strategy Level 3

• “Innovative pedagogy, curriculum design and assessment are brought to new heights by technology, including, but not restricted to, internationalization of the curriculum, collaborative teaching and learning within HKU courses and with overseas universities, integration of campus-based and experiential learning, involvement of employers and community partners in the learning processes”.

7

eLearning Strategy Level 3

8

eLearning Strategy Level 3

9

Faculty Moodle Statistics

Faculty/Others

No. of Moodle courses

Total Available Created Enabled Active

821 265 87 82

10

1. “Total available” - the course templates created based on SIS data or the courses created explicitly as per requests from teachers

2. “Created” - courses that their teachers have created and can start editing its eLearning materials 3. “Enabled” - courses that their teachers have granted students’ access 4. “Active” – courses with both teacher logins > 5 and student logins > 5

• Right now the reality is Moodle!

Moodle Activity

• Moodle Activity Levels– Level 1 - one-way transmission of information from

teacher to students – the data is gathered as "the total count of Moodle files/ folders in the Moodle courses"

– Level 2 - two-way interaction between teachers & students – total count of Moodle assignment, quizzes and choices in the Moodle courses

– Level 3 - three way interaction between teacher & students and among students – total count of Moodle forum, chat and wikis in the Moodle courses

11

Faculty Moodle Statistics

Faculty

/Others

Number of active Moodle courses havingonly Level 1 activities

only Level 2 activities

only Level 3 activities

Level 1 and Level 2 activities

Level 1 and Level 3 activities

Level 2 and Level 3 activities

Level 1, Level 2 and Level 3 activities

0 0 0 0 48 0 34

12

• The news forum is created automatically which may account for the level 1 and level 3 activities. So the only “real figure” may be levels 1, 2 and 3.

Quantitative Reporting

• “Quantitative Performance Indicator (PI) data has a limited role to play in determining impact of new learning and teaching practices on the learner experience. PIs are ‘input’ and ‘output’ measures and TESEP [Transforming and Enhancing the Student Experience through Pedagogy] found that they were best used as ‘diagnostic tools’ which could be used on a number of levels.”

13

Quantitative Reporting

• These statistics tell us about activity levels in Moodle.

• The statistics do not tell us anything about the quality of eLearning e.g.– A lecturer on any particular course might be delivering

all the course content (level 1) via Moodle.– The same lecturer might be engaging students with

multiple choice assessments (level 2) to test understanding.

• What does this tell us about quality?– Nothing!

14

Quality Learning

• When we say that a course is a quality course “we” are broadly making a judgment that the course is fit for purpose where the purpose is institutional purpose.

• For example, university which efficiently helps students to achieve the educational aims and learning outcomes can be considered as a quality institution.

• The same is true at a course level.

15

Quality Learning QAC

• “The main objective of QAC quality audits is to assure the quality of student learning in UGC-funded institutions. The audits are intended to assure the UGC and the public that institutions deliver on the promises they make in their role and mission statements. A QAC audit is therefore an audit of an institution’s Fitness for Purpose in teaching and learning” (Quality Assurance Council, 2008, p.10).

16

Quality Learning QAC

• QAC is concerned with learning outcomes:

• “the audit process is particularly concerned with the ways institutions articulate and measure the student learning outcomes they expect or aspire to” (Quality Assurance Council, 2008, p.11)

17

Quality Learning

• Quality also has something to do with the student learning experience.

• In other words being fit for purpose is a necessary but not sufficient condition of a quality course.

• The reason for this is that students may achieve educational aims but have a very negative experience on the course.

• So, we need to include some criteria referenced to experience.

18

Quality Learning QAC

• The QAC, through its audit process, aims to give confidence, in general, to students and their parents, employers and sponsors, that our institutions provide a quality and internationally recognised student learning experience (Quality Assurance Council, 2008, p.3.).

19

Quality in Learning

• At HKU the criteria for a quality course are given in the Student Experience of Teaching and Learning (SETL) questionnaire and the Student Learning Experience Questionnaire (SLEQ).

• Both questionnaires consist of items that relate to achieving course aims, developing generic skills / attributes and having a positive experience on a course.

• So a quality course is a course that is rated positively with respect to these aspects.

20

Quality in eLearning

• If we know what constitutes quality in teaching and learning we can ask about whether things are different with eLearning.

• The short answer is “no” because at a macro level the teaching and learning conditions are the same:– The course must enable students to achieve ILOs;– The course must develop generic attributes / skills; and– Students must have a positive experience on the

course.

21

Quality in eLearning

• “I believe the distinction between face-to-face and online will soon merge in both quality standard setting and practice. As pedagogy and learning needs drive educational design, every possible mutation of physical and virtual meetings of minds will be created and be grounded in pedagogical purpose. It is a grand time to be an educator” (Frydenberg, 2002).

• There is an important point here – pedagogy and learning drive technology use – but this is not the whole picture for quality in eLearning.

22

Quality in eLearning

• Things are different at the micro level because there is a question of the use of technologies in teaching and learning:– Technical infrastructure;– Learning management system;– Technical support for students;– Use of additional technologies;– Use of a variety of media;

• The big question, “Did eLearning help?”

23

Quality in eLearning

• The following three questions have been put forward for use in HKUs Student Learning Experience Questionnaire:– The Learning Management System Moodle was easy

to access and use;– eLearning has been effective in aiding learning in

courses in my discipline; and– eLearning has been effective in aiding communication

in courses in my discipline.

24

Quality in eLearning

• What would a more robust quality assurance program look like?

• There are issues here:– A robust quality assurance program would be much

more comprehensive in terms of the range of questions asked about eLearning;

– A robust quality assurance program would move beyond questions that ask about “perceptions”;

– The program would look for evidence of e.g. improved communication, improved understanding.

25

Concluding Remarks

• There has to be a program of work if we want actual evidence that the eLearning strategy has resulted in improved quality in teaching and learning;

• Funding would be nice because in the end showing actual gains will require a lot of work.

• Don’t really see this program of work as optional because without it what do we have in terms of evidence for effectiveness of the eLearning strategy?

26

References

• European Association of Distance Teaching Universities (EADTU). (2007). Quality Manual for E-learning in Higher Education,. European Association of Distance Teaching Universities (EADTU). Retrieved from http://www.eadtu.nl/e-xcellenceQS/files/members/E-xcellenceManualGrey/index.html

27

References

• Frydenberg, J. (2002). Quality Standards in eLearning: A Matrix of Analysis. The International Review of Research in Open and Distance Learning, 3(2), 1-15. Retrieved from http://www.irrodl.org/index.php/irrodl/article/viewArticle/109

28

References

• Herrington, A., Herrington, J., Oliver, R., Stoney, S., & Willis, J. (2001). Quality Guidelines for Online Courses: The Development of an Instrument to Audit Online Units. In G. Kennedy & M. Keppell (Eds.), Meeting at the Crossroads.18th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education. (pp. 263-270). Melbourne, Australia. Retrieved from http://www.ascilite.org.au/conferences/melbourne01/pdf/papers/herringtona.pdf

29

References

• TESEP (Transforming and Enhancing the Student Experience through Pedagogy). (2007). TESEP: Re-thinking Quality Enhancement (pp. 1-4). Edinburgh, UK. Retrieved from http://www2.napier.ac.uk/transform/rethinking_qe.htm

30