evaluating learning designs through the formal representation of learning patterns
DESCRIPTION
Evaluating learning designs through the formal representation of learning patterns. Diana Laurillard and Dejan Ljubojevic London Knowledge Lab Institute of Education, London . A Learning Design Support Environment (LDSE). Oxford Liz Masterman (CoPI) Marion Manton (CoPI). Birkbeck/LKL - PowerPoint PPT PresentationTRANSCRIPT
• Diana Laurillard and Dejan Ljubojevic • London Knowledge Lab
Institute of Education, London
Evaluating learning designs through the formal representation
of learning patterns
A Learning Design Support Environment (LDSE)
IOE/LKLBrock Craft (Technical)Sarah Gelcich (Admin)Diana Laurillard (PI)Dejan Ljubojevic (RF)Javier Calzada-Prado (Intern)
OxfordLiz Masterman (CoPI)Marion Manton (CoPI)
Birkbeck/LKLGeorge Magooulas (CoPI)Patricia Charlton
LondonMetTom Boyle (CoPI)
LSESteve Ryan (CoPI)Ed WhitleyRoser Pujadas (Studentship)
RVCKim Whittlestone (CoPI)Stephen MayCarrie Driver (Studentship)
OUTLINETeachers are interested in learning design and the use of technology but lack the tools they need LDSE
Planning, advice, exemplars – could it also evaluate a design?
• Design principles in the literature
• Learning design patterns – to capture pedagogic principles?
• Representing learning theories as an evaluative framework
• Evaluating the pedagogy in a learning design
• Testing the evaluative capability of the framework
Evaluating learning designs
Design principles in the literatureGood feedback practice:
1. helps clarify what good performance is - goals, criteria, expected standards
2. facilitates the development of self-assessment (reflection) in learning
3. delivers high quality information to students about their learning
4. encourages teacher and peer dialogue around learning
5. encourages positive motivational beliefs and self-esteem
6. provides opportunities to close the gap between current and desired performance
7. provides information to teachers that can be used to help shape the teaching.
(Nicol & MacFarlane-Dick, 2006)
Design principles in the literature 1. Encourage contacts between students and faculty 2. Develop reciprocity and cooperation among students 3. Use active learning techniques 4. Give prompt feedback 5. Emphasize time on task 6. Communicate high expectations 7. Respect diverse talents and ways of learning.(Chickering and Gamson, 1991)
Where is the pedagogy that aids learning design?
What active learning techniques? What kind of feedback?
Can pedagogical patterns assist with designing and evaluating the quality of a learning design?
Learning design patterns
• The LDSE( learning design support environment) aims to evaluate the pedagogy in a learning design in terms of theory
• We look for pattern templates that will help to evaluate pedagogy in a design
• A learning design pattern as a learning activity sequence designed to lead to a specific learning outcome
Comparing pedagogy pattern templatesiCOPER Planet TELL
Author & Copyright Credits
Summary/ Thumbnail Context
Rationale Rationale Rationale
Subject/ Discipline Context
Learning outcomes Problem Problem
ForcesGroup size
Duration (part)
Learner Characteristics Audience
Type of Setting
Graphical Representation Diagram
Sequence of Activities
Solution SolutionRoles
Type of Assessment
Where is the effective pedagogy captured?
We need categories within the ‘solution’ or ‘sequence of activities’ to help teacher:• Develop reciprocity and cooperation among students• Use active learning techniques• Give prompt feedbackSome accounts of learning activity sequences are quite elaborate
Categories do not sufficiently discriminate the form of pedagogy that will make the critical difference to learners
Comparing pedagogy pattern templatesiCOPER Planet TELL
Rationale Rationale Rationale
Learning outcomes Problem Problem
Sequence of Activities
Solution SolutionRoles
Type of Assessment
Needs further detailed analysis to support learning design
Pattern for understanding authentic practice
Pattern for helping whole class benefit from individual experiences
Representing learning patternsStructured text-based accounts , e.g. iCoper, Learning Designs
Text cannot be easily interpreted by a program.Computationally defined activities can be (e.g. LAMS).Therefore can be evaluated against theory.
Box-and-arrow diagrams for a sequence of activities, e.g. LAMS, iCoper, ISiS-ECTEL
Representing learning theories
The Conversational Framework: developed to provide a representation of what it takes to learn in education
based on the main design principles for learning and pedagogy drawn from the literature
can test a learning design in terms of which aspects of the framework it covers.
The learner learning
LCTC
LPTP
Thoughts
Action plans
Guidance
OC
OP
Articulating ideas
Others’ ideas
Preparing Outputs
Others’ Outputs
Asking Questions
Investigating
Listening/ Reading
Reflection
Producing
Revising
Working to a goal
Feedback
LCTuition
Practice LP
Adaptation
Discussion
Imitation
being supported, actingthinking,
Learning theories represented in the Conversational Framework
LCTC
LPTP
Thoughts
Action plans
Guidance
OC
OP
Articulating ideas
Others’ ideas
Preparing Outputs
Others’ Outputs
Asking Questions
Investigating
Listening/ Reading
Reflection
Producing
Revising
Working to a goal
Feedback
LC
LP
Adaptation
1. Encourage contacts between students
and faculty
4. Give prompt feedback
2. Develop reciprocity and cooperation among
students
3. Use active learning techniques
Learning theories represented in the Conversational Framework
LCTC
LPTP
Thoughts
Action plans
Guidance
OC
OP
Articulating ideas
Others’ ideas
Preparing Outputs
Others’ Outputs
Asking Questions
Acting
Listening/ Reading
Reflection
Producing
Revising
Working to a goal
Information
LC
LP
Adaptation
Social constructivism
ReflectionAdaptation
Learning theories represented in the Conversational Framework
LCTC
LPTP
Thoughts
Action plans
Guidance
OC
OP
Articulating ideas
Others’ ideas
Preparing Outputs
Others’ Outputs
Asking Questions
Acting
Listening/ Reading
Reflection
Producing
Revising
Working to a goal
Information
LC
LP
Adaptation
Instructivism
ReflectionAdaptation
Learning theories represented in the Conversational Framework
LCTC
LPTP
Thoughts
Action plans
Guidance
OC
OP
Articulating ideas
Others’ ideas
Preparing Outputs
Others’ Outputs
Asking Questions
Acting
Listening/ Reading
Reflection
Producing
Revising
Working to a goal
Feedback
LC
LP
Adaptation
Situated learning
ReflectionAdaptation
Representing technologies in support of learning
LCTC
LPTP
Thoughts
Action plans
OC
OP
Reflection
LCAdaptive tutoring
programs
Virtual adaptive
immersive environments
LP
Adaptation
Wikis, blogs and Forums
User-generated
content sites
Evaluating a learning design
The Conversational Framework: can test a learning design in terms of which aspects of the framework it covers
- IF we can link each LAMS activity (or similar computationally defined activity) in the sequence to part of the framework
LAMS – representing learning design as a temporal sequence of computationally defined activities
Chat
Chat and Scribe
ForumForum and Scribe
MCQNotebookNoticeboardQ&AResources & ForumShare resources
SubmitVoting
To understand the processes within a system
Chat and Scribe
Mapping LAMS activities to theory via CF
LCTC
LPTP
Guidance
OC
OP
Articulating ideas
Others’ ideas
Preparing Outputs
Others’ Outputs
Asking Questions
Investigating
Listening/ Reading
Producing
Revising
Working to a goal
Feedback
LC
LP
Forum and ScribeMCQ
Notebook
Q&A
Share resourcesSubmitVoting
Chat and Scribe
Notebook
Noticeboard
Noticeboard
Q&A
Share resources
Submit
Forum and Scribe
MCQ
Voting
LAMS – representing learning design as a temporal sequence of computationally defined activities
To understand the processes within a system
Interpreting the quality of the design
LCTC
LPTP
Guidance
OC
OP
Articulating ideas
Others’ ideas
Preparing Outputs
Others’ Outputs
Asking Questions
Investigating
Listening/ Reading
Producing
Revising
Working to a goal
Feedback
LC
LP
Notebook
Chat and Scribe
Noticeboard
Q&A
Share resources
Submit
Forum and Scribe
MCQ
Voting
System can interpret design as scoring:4 out of 10 activities;No iterations;Time for each task is good;Sequence is good for ‘awareness’; less good for ‘understanding’
Evaluating the quality of the design
Improving the quality of the design
LCTC
LPTP
Guidance
OC
OP
Articulating ideas
Others’ ideas
Preparing Outputs
Others’ Outputs
Asking Questions
Investigating
Listening/ Reading
Producing
Revising
Working to a goal
Feedback
LC
LP
Chat and Scribe
Noticeboard
Q&A
Voting
System can propose:Adding in designs or patterns that complete the other cycles;Checking that the sequence follows the motivating cycles of iteration
Testing the framework
The Conversational Framework: can differentiate between sequences in terms of coverage of the framework, amount of iteration, and effective ordering of activities
But to what extent can it recognise the relative effectiveness of learning designs identified in the literature?
Pattern 1
Contrasting cases text
Analyse data
S analyses data
S produces graphs
S listens to lecture
S selects and predicts
Very accurate MCQs
High transfer to prediction
Pattern 2
Theory and examples text
Read text
S reads text
S produces summary
S listens to lecture
S selects and predicts
Very accurate MCQs
Much lower transfer to prediction
Comparing learning design patternsIntended learning outcome: To predict accurately the outcomes for a hypothetical experiment on memory [Schwartz and Bransford,1998]
Learning activities
Reading
T sets a task goal
Investigating
Producing output
Listening
Working to achieve goal
Actual outcomes
Pattern 1
Contrasting cases text
Analyse data
S analyses data
S produces graphs
S listens to lecture
S selects and predicts
Very accurate MCQs
High transfer to prediction
Pattern 2
Theory and examples text
Read text
S reads text
S produces summary
S listens to lecture
S selects and predicts
Very accurate MCQs
Much lower transfer to prediction
Critical differences between patterns are not just the nature of the activity, but the internal relations between contents of activities in the sequence
Interpreting the quality of the designs
LCTC
LPTextData
Guidance
3. Summarising3. Investigating
1. Reading
4. Producing
Revising
2. Working to a goalFeedback
LC
LP
System must be able to differentiate between producing graphs and summaries;And between providing Text and Data in the practice environment
5. Listening
4. ProducingReflectingAdapting
Asking Questions
Representing the range of good pedagogy
LCTC
LPTP
Guidance
OC
OP
Articulating ideas
Others’ ideas
Preparing Outputs
Others’ Outputs
Asking Questions
Investigating
Listening/ Reading
Producing
Revising
Working to a goal
Feedback
LC
LP
Notebook
Chat and Scribe
Noticeboard
Q&A
Share resources
Submit
Forum and Scribe
MCQ
Voting
A System like LAMS can represent good pedagogy in terms of type and range of learning activity
Cannot represent good pedagogy within the practice environment – simulation, model, digital library, data repositories, type of task and type of feedback
SummaryDesign principles in the literature
Learning design patterns
Representing learning theories as a framework for design
Evaluating the pedagogy in a learning design
Testing the evaluative capability of the framework
Too general to be helpful for teacher-designers
May miss categorising the critical pedagogy
Makes it possible to relate theory to computational activities
Needs mapping from framework to LAMS or similar system
May not capture all the key differences in effectiveness
Pedagogy operates both across and within activities• We need user-editable patterns for activity sequences
– One role for learning technologists is to create the means for teachers to adopt and adapt pedagogic patterns for activity sequences – e.g. LAMS
– One role for teachers is to these to adopt, adapt and share pedagogic patterns for activity sequences
• We need customisable programmed patterns for activities– Another role for the learning technologist is to discover and
create the means for teachers to adopt and adapt pedagogic patterns for practice environments – e.g. GLO tool, NetLogo
Evaluating learning designs through the formal representation of learning patterns
Made publicly available by ALT under a Creative Commons Attribution-Non-Commercial 2.0 UK: England & Wales license http://creativecommons.org/licenses/bync/2.0uk/