Improving lecture capture through usability testing and research

Ilkka Kukkonen

@ilkkakukkonen #mlconf13

Media & Learning 2013, Brussels 12.11.2013

Motivation

• Create the best possible learning environment for our students and clients

• Improve overall quality, commitment and effects with optimal resource allocation

• Support personal learning paths and flexibility in participation

• Improve the pedagogical models and spesifically the lecture format

• Create multiaccess courses

INPUT & OUTPUT

Project approach to support the development(”MOVIE”, 2011-2012) N~800• The development of pedagogical quality and effectiveness

of lecture capture– How the students are using the recorded lectures to

support their studies?– How can we measure of the impact in various contexts?– How the pedagogical quality of the lecture capture can

be improved?• Data from teacher training and high school students,

collected in mathematical subjects and statistics

A B C D

57%49% 50%

64%

A. Possibility for independent studies

B. Retention and repetition of classes

C. Illuminated unclear themes from lectures

D. Helped to prepare for final exams

Encouraging results (N=113) *

• 81% would like to increase the use of lecture capture in their studies• 85% would recommend lecture capture for their peers

* www.aducate.fi/movie

Findings from the study

• Students seem willing to increase the use of lecture capture in their studies

• Use of LC is seen as democratic method for participation

• It takes time to develop personal study strategies and build trust for LC

• Impact on lecture attendance seems to depend on the personal study strategies and previous experiences

• Viewing a lecture capture can feel more personal pedagogical experience than attending live lecture

* www.aducate.fi/movie

Findings from the study

• Lecture capture seems to be more efficient for lower achieving students– Supporting conceptual understanding for content,

metacognition and efficient study strategies• Viewing volumes seem to have correlation to

experienced value and grades• Challenge is to create meaningful learning contexts for

students– Include lecture capture in learning design– Summarize, contextualize and engage interaction

* www.aducate.fi/movie

(CC) Liz Castro @ Flickr

INFORMATION DELIVERY VS. TEACHING

Usability study (3-6/2012, N=16)

Usability study (3-6/2012, N=16)

• Need to create realistic insight about LC use in genuine context

• Are there some problems related to the usability and content?

• How does the interface and features support viewing?• How students find the essential parts of the lecture?• 4 short lectures with different statistical assignments• Familiarity of the content• Pair work vs. individual performance

● Research setup records students’ eye movement, video and audio and captures also mouse clicks (Tobii Eye Tracker)

● Two similar and syncronized workstations for pair work● Downside is research setup is quite complex and

annotation and analyzing the final data is time consuming● Minor inaccuracies may occur within the people with

glasses● Supplemental interviews and survey

Eye-tracking for lecture capture

VIDEO EXAMPLE

SLIDES

LECTURER

NAVIGATION

TIMELINE

TEACHERS COMMENTS

PERSONAL BOOKMARK

S

Navigation

Timeline

Lecturer

% o

f to

tal vi

ew

ing

tim

e

Slides

Total viewing time: regions

% o

f u

sers

use

d a

t le

ast

on

ceUsed features

Scenes

Timeline

Forward / ReversePersonal bookmarksTeacher’s comments

Tim

e i

n m

inu

tes

Standarddeviation

Mean

Time with assignments

Recording 1

Recording 2

Recording 3

Recording 4

* ”Flip”

Question 3

Question 1

Question 2

Recording 1 Recording 2 Recording 3 Recording 4

% a

nsw

ers

corr

ect

Performance

* ”Flip”

Results from the video annotation (N=16)

• Teacher’s comments and remarks seem to support the viewing experience (scaffolding)

• Familiarity of the content and student’s personal notes and bookmarks speeded up completing the assignments

• Writing lecture notes while playing the video in the background gave lower scores in individual level

• Learning by copying the instructor, ”vicarious learning” (Bandura) seems to lead into surface level learning, which generated lower scores individual level and pair work

• Students who spent time with making sense of the subject got better grades than expected

● Students formed pairs independently based on their social relationships

● Pairs typically shared the tasks for viewing and making notes● ”Weaker” party asks and questions the answers, so the

talented one needs to answer with thought and provide a convincing explanation about the potential answer

● In this setting stronger one seems to develop the weaker student’s grades

● Also in the final exam the weaker students outperformed the expected grades, so working in pairs with lecture capture might have developed their skills and motivation

Working in pairs with lecture capture

www.aducate.fiwww.uef.fi

Thanks!

Talk more? Collaborate?

http://aducate.fi/movieilkka.kukkonen@uef.fi @ilkkakukkonen