evidence-based work on learning and...

Evidence-based work on learning and instruction George Mason University June 6, 2012

2

•  Integrating the design, building, monitoring, and improvement of learning environments; individualize learning experiences using our scale; and, ultimately, drive greater student career success.

•  Former CLO for K12, Inc. – structured use of technology, cognitive science, on-line and off-line materials for 1,700 teachers, 55k students

•  Former Publisher and General Manager for DK Multimedia, Inc. •  Management consultant with McKinsey & Company •  Education:

-  Ph.D. in Electrical Engineering and Computer Science from MIT -  M.D. from Harvard Medical School -  M.A. in Electrical Engineering and Computer Science from MIT -  M.A. in Mathematics from Oxford University -  B.S. in Electrical Engineering and B.S. with Honors in Mathematics

from the University of Washington

Bror Saxberg Chief Learning Officer, Kaplan, Inc.

3

•  Kaplan University •  Kaplan Legal Education •  Kaplan Professional

Education •  Nursing •  Kaplan Continuing Education •  KNEXT

•  KTPA •  Kaplan Tutoring •  Kaplan Bar Review •  Kaplan Publishing

•  Kaplan Higher Ed – Europe •  Kaplan Professional – Europe

•  Kaplan Higher Ed – Asia •  Kaplan Professional – Asia •  Kaplan Higher Ed – Australia •  Kaplan Professional –

Australia •  In Country Pathways – China •  Franklyn Scholar •  Carrick Education •  Global Knowledge Solutions

Kaplan education spans domains and geography

Kaplan University Group

Kaplan Higher Education Campuses

Kaplan Test Prep

Kaplan Asia Pacific Kaplan United Kingdom •  Kaplan Int’l Colleges

•  Global Pathways

Kaplan International Colleges

4

What we’re trying to do

5


6


7


8


9

Agenda

• What evidence says about learning • What this means for the design of instruction • What happens when you do this for real

10

Agenda


11

Much research to guide us

Learning Events

(hidden - inside students’ minds)

Student Performance

(observable -indicates

knowledge)

Instructional Events

(in the learning environment)

Knowledge

•  Explicit: Information, Explanation, Examples, Demos

•  Implicit: Practice tasks/activities (prompts and response)

•  Diagnosis and feedback

•  Explicit/Declarative/Conceptual/What •  Implicit/Procedural/How •  Knowledge Components

(Procedures + Facts, Concepts, Principles, Processes)

•  Response accuracy/errors •  Response fluency/speed •  Number of trials •  Amount of assistance (hints) •  Reasoning

Koedinger, K.R., Corbe/, A.T., and Perfe5, C. (2010). The Knowledge-‐Learning-‐InstrucEon (KLI) Framework: Toward Bridging the Science-‐PracEce Chasm to Enhance Robust Student Learning (DraN manuscript from the Pi/sburgh Science of Learning Center)

12

5 types of outcomes determine TYPE of information and practice

Knowledge Component Definition Example

Procedure

Sup

porti

ve/C

once

ptua

l Fact

Concept

Process

Principle

13



Procedure Sequence of decision and action steps to perform tasks; when and how to do things

• Prosecuting a criminal • Deciding if capital gains tax applies

Sup

porti

ve/C

once

ptua

l Fact

Concept

Process

Principle

14





Sup

porti

ve/C

once

ptua

l Fact Isolated, unique piece of information; one instance

• 52 Grosvenor Place • 2+3=5

Concept

Process

Principle

15





Sup

porti

ve/C

once

ptua



Concept Sets of items that share common attributes, common name; multiple examples

• Dog • Money • Happiness

Process

Principle

16





Sup

porti

ve/C

once

ptua





Process Flow of events or procedures; how things work

• Workflow • Chemical process

Principle

17





Sup

porti

ve/C

once

ptua





Process Flow of events or procedures; how things work

• Workflow • Chemical process

Principle Guidelines, rules that govern, predict, explain events; relationships among concepts

• Supply and demand • 80/20 principle • Novices need structure

18

3 stages of learning determine instructional elements and sequence

Stage Characteristics Implications for Instructional Design

1.

2.

3.

Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, ExperEse: Its Structure and AcquisiEon.

Fitts & Posner, (1967), John Anderson (2004, 2007); Anders Ericsson (2006, 2007)

19



1. Declarative

• Knowledge “about”, “that”, “what” “why”;

• Can be stated verbally; • Conceptual network • Conscious

Design clear, relevant, and accurate information displays, job aids, examples, reference material for all knowledge components: facts, concepts, principles, processes, procedures

2.

3.



20



1. Declarative




2. Procedural

• Knowledge “how” • Sequence of “if-thens” • Potential to become unconscious

Design practice tasks to elicit student performance/ responses; monitoring systems to detect errors; and feedback/coaching to correct errors in performance

3.



21



1. Declarative




2. Procedural

• Knowledge “how” • Sequence of “if-thens” • Potential to become unconscious

Design practice tasks to elicit student performance/ responses; monitoring systems to detect errors; and feedback/coaching to correct errors in performance

3. Automated

• Fluency • Expert • Unconscious • “10,000 hours”

Design opportunities for repeated frequent practice on the job and monitoring of speed and accuracy



22


Learning Events


Student Performance


knowledge)



Knowledge







Motivation

•  Orientation/Inoculation •  Monitoring •  Diagnosis and treatment:

Persuasion, Modeling, Dissonance

•  Value beliefs •  Self-efficacy beliefs •  Attribution beliefs •  Mood/Emotion

•  Behavior related to •  Starting •  Persisting •  Mental Effort

•  Self-reported beliefs


23

4 beliefs influence mo/va/on

Sources: Bandura; Eccles & Wigfield; Pintrich & Schunk; Clark; Dweck

Beliefs

•  Value •  Self-Efficacy •  Attribution •  Mood

Motivated Behavior

•  Starting •  Persisting •  Mental Effort

Learning/ Performance

•  Practice •  Test

Self-Efficacy

Effo

rt

High Moderate Low

Motivation Low High

Performance High Low

• Design materials and interaction to foster positive mood, high perception of value, moderate confidence, and attribution of success and failure to effort

• Design system for monitoring and guidance (group and individual)

24


Learning Events


Student Performance


knowledge)



Knowledge







Motivation

•  Orientation/Inoculation •  Monitoring •  Diagnosis and treatment:

Persuasion, Modeling, Dissonance

•  Value beliefs •  Self-efficacy beliefs •  Attribution beliefs •  Mood/Emotion

•  Behavior related to •  Starting •  Persisting •  Mental Effort

•  Self-reported beliefs

Metacognition •  Structure •  Guidance

•  Planning, Monitoring •  Selecting, Connecting

•  Amount of guidance required/requested


25

Agenda


26

Instructional design: “Engineering” from learning science

Overviews Information Examples Practice Assessment Learning Outcomes

Motivational Guidance

Design

Deliver

Learning science strongly suggests an order to design and delivery

Clark, R.E., & Feldon, D. F. (2008). GEL (Guided Experiential Learning), Adaptable Expertise and Transfer of Training. Kirscher, P.A., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential and inquiry-based teaching. Educational Psychologist, 41, 75-86.

Knowledge Integration

27

Evidence-based instructional principles Accumulation of results from lab studies support: •  Structure and guidance for novices

(Kirschner, Sweller, & Clark, 2006) •  Demonstrations and worked examples

(Paas & van Merrienboer, 1994; Sweller, 2006) •  Practice and corrective feedback

(Mathan & Koedinger, 2005) •  Prompted self-explanation

(Aleven & Koedinger, 2002) •  Multimedia use that minimizes extraneous cognitive load

(Mayer, 2009) •  Targeting beliefs (value, confidence, and attributions) and

emotions (positive feelings) to influence motivation (Clark, 2004; Um et al., 2011)

28

Task-centered instruction

• Move from simple to increasingly difficult tasks – NOT “PBL” sink or swim • Teach everything needed for each task • Fade coaching/support over time

29

Knowledge Component

Presentation (Prepare) Practice/Assessment (Practice, Perform)

Info Example Remember Proxy for Remember Use** Proxy for Use **

Procedure

When to use; List of action and decision steps

Demonstration of when and how to perform

Recall when to use; Recall action and decision steps

Reorder steps; Recall next or missing steps

Decide when to use; Perform the steps (actions and decisions)

Critique performance or output of actions and decisions

Sup

porti

ve K

now

ledg

e

Fact * Statement of fact Statement of fact Recall fact Recognize fact when

presented with distractors Recall fact in task context

Concepts List of defining attributes

Examples; Non-examples

List defining attributes verbally or in writing

Recognize defining attributes when presented with distractors

Classify, identify or generate examples and non-examples

Critique someone else’s identification or generation of examples

Process/ System

List of phases, events and causes at each phase

Examples; simulations of phases, events, and causes

Recall phases, events, and causes

Recognize phases, events, and causes; Recall missing phases, events, and causes

Identify causes of faults in a process; Predict events in a process

Critique someone else’s description of causes or prediction of events in a process

Principle (cause and effect relationship)

Statement of cause and effect relationship

Examples, demonstration, simulation of cause and effect relationship

Recall the principle

Recognize the principle; Recall missing elements of the principle

Decide if principle applies; Predict an effect; Apply principle to solve a problem, explain a phenomenon or make a decision

Critique someone else’s application of the principle to solve a problem, explain a phenomenon or make a decision

Knowledge Integration

Explain the interconnections among conceptual knowledge components, or the conceptual foundation of procedures, or the procedural implementation of conceptual knowledge components

Opportunities (including instructions, templates, rubrics) to self-explain, discuss, present, describe or select their reasoning about interconnections among knowledge components, for example the principle(s) that justify the application of a procedure.

Knowledge Transfer

Multiple and varied contexts for examples

Multiple and varied contexts for practice and assessment. Opportunities for students to explain how they would use the knowledge in other contexts

*Facts are concepts with single instances ** All Use and Proxy for Use Activities develop/require procedural knowledge

Presentation and practice match objectives (knowledge components)

30

Agenda


31

ID can change instructional outcomes at scale

Principle Description Effect size (s.d. units)

Multimedia Use relevant graphics and text to communicate content 1.5

Contiguity Integrate the text nearby the graphics on the screen – avoid covering or separating integrated information

1.1

Coherence Avoid irrelevant graphics, stories, videos, media, and lengthy text 1.3

Modality Include audio narration where possible to explain graphic presentation 1.0

Redundancy Do not present words as both on-screen text and narration when graphics are present

.7

Personalization Script audio in a conversational style using first and second person 1.3

Segmenting Break content down into small topic chunks that can be accessed t the learner’s preferred rate

1.0

Pre-training Teach important concepts and facts prior to procedures or processes 1.3

Etc. Worked examples, self-explanation questions, varied-context examples and comparisons, etc.

??

Source: E-learning and the Science of Instruction, Clark and Mayer, 2nd ed., 2008

32

Impact is not small!

50% 1 sd

84%!

33

Redeveloping courses at scale

Read, Write, Discuss •  Outcomes and content not

precisely aligned •  Limited demonstrations, worked

examples, and practice •  General assessment rubrics •  High reliance on discussion boards

Existing courses

Prepare, Practice, Perform •  Outcomes and content aligned •  One lesson per objective •  Demonstrations and worked examples •  Practice, feedback before assessment •  Detailed scoring guides •  Less discussion/more practice •  Standard instructor materials •  Monitoring and support for motivation

Redesigned courses

34

Content Design

Items Prepare Practice Perform

Seminar Discussion Lessons Sets Overview

Course Level Outcome 1

Unit Outcome 1

Unit Outcome 2

Unit Outcome 3

Prepare 1

Practice 1

Perform 1

Prepare 2

Practice 2

Perform 2

Prepare 3

Practice 3

Perform 3

Lesson 1 Lesson 2 Lesson 3

Navigation

35

Narrated demonstrations

36

Explanation and demonstration of concepts

37

Practice with hints and feedback

38

Scenario-based practice

39

Prompted self-explanation

40

Motivation surveys

41

5

Student Preparation and Questions

Prior to attending the live seminar, or reviewing the seminar recording, complete the following tasks:

1. Check your grades and think about your performance in the previous unit:

•  Where did you do well and where did you struggle? •  What would help you do even better in the current unit?

2. Read the current unit Overview, Seminar, and Discussion pages.

3. Read the current unit recommended textbook pages.

4. Write down at least two questions you would like answers to in the seminar. CHAT: Type one of your questions into the chat window.

6

Overview

Seminar

Discussion Journal: A situation in your life where the guidelines for improving nonverbal communication could guard against misinterpretation.

Lesson 1 Identify verbal and nonverbal elements in personal and professional situations 1-2 hrs.

Lesson 2 Identify nonverbal communication principles in personal and professional situations 1-2 hrs.

Lesson 3 Explain instances of effective and ineffective communication in terms of how verbal and nonverbal elements work together 2-4 hrs.

UNIT 5 Review: What is Nonverbal Communication?

2. Identify nonverbal communication principles in personal and professional situations

15

UNIT 5 Review: Lesson 2 Practice

http://www.youtube.com/watch?v=bg0kSIJZiRQ&feature=related

PART 1: Which nonverbal communication principle is predominant in the woman’s reactions to her blind date?

Watch Item 2 video:

21

Overview (including Survey)

Seminar Discussion

Explain how improving your listening skills can increase the effectiveness of your communication in the workplace and in your personal life.

Lesson 1 Identify forms of nonlistening in personal and professional situations 1-2 hrs.

Lesson 2 Apply the principles of mindful listening to improve the effectiveness of communication in personal and professional situations 1-2 hrs.

UNIT 6 Preview: How Does Listening Enhance Our I.C.?

25

UNIT 6: Lesson 2 Practice

Watch the Online Dating video. Answer the three questions, referring to the scoring guide.

Online Dating

After answering a question, study the “Compare with Expert” response.

1. From the interaction does it seem to you that Chris’s mom is actively listening during the first third of this conversation? Why or why not?

2. Apply the principles of mindful listening to improve communication effectiveness

27

• The Practice and Perform activities are open book and untimed. Consult the textbook and online information in a separate browser window or tab as you do them.

•  If you don’t understand the feedback in a Practice item, ask me about it and I will try to explain it.

• To do well on the Lesson 2 Perform activity, reference the scoring guide and the Prepare and Practice example responses for each question.

• Use the transcripts of the Lesson 2 video scenarios to help you answer the questions:

- For the Practice, see Online Dating: p. 166

- For the Perform, see Alan O’Connor: p. 217

UNIT 6: Tips for Success

29

Agenda Minutes Opening 5 ! Student Questions 10 ! Review Unit 5 10 ! View Unit 6 25 ! Preview Unit 7 5 ! Wrap Up 5 !

Poll Question

How many of you still have questions?

Post your questions in �Course Questions��

discussion board

!  Yes !  No

(Link in Course Home menu)

Wrap Up

Instructor seminar materials more standardized and aligned with online content

42

= Unit 1 = Unit 3 = Unit 6

Provide materials to monitor and support motivation

43

What happened? Research design: quasi-experimental

Control Pilot Control Pilot Control PilotInterpersonal Communications 8 7 4 3 237 199Principles of Nutrition 6 4 3 2 148 89Medical Terminology 6 7 4 2 197 220

Total 20 18 11 7 582 508

n sections n instructors n studentsCourse

•  1,090 students (508 pilot; 582 control) •  87% female, average age 32; average household income $20,000 •  3 courses •  18 instructors •  20 sections (assigned to pilot or control) •  2 terms (Aug – Dec 2011)

44

Analysis

•  Logistic regression to examine effect of course design on student success

•  Success (1 or 0): Defined as: Pass (1 or 0) + Master course objectives (>=4 on 0-5 scale) + Stay (1 or 0)

•  Controlled for variation in •  Instructor prior student success rates •  Student background variables

•  Age, prior education, prior GPA, tenure, household income

•  Calendar-based success variation

45

Student success: results controlling for variables •  11% higher

success rate

•  28% increase

•  Students in redesigned courses were 1.6 times more likely to be successful

Wald Chi-Square: 10.42, df=1, n=895, Sig<.001.

39%

50%

20%

30%

40%

50%

60%

70%

Control Pilot Adj

uste

d st

uden

t suc

cess

rate

Adjusted student success rates with 95% confidence limits

46

More to do: adjusted results varied by course

•  Odds of success more than doubled (2.5 times more) in Interpersonal Communications

•  9% difference in Principles of Nutrition (not stat significant due to smaller sample) •  Small improvement in Medical Terminology due to difficulty level of early units

33%#45%# 39%#

55%# 54%#42%#

0%#10%#20%#30%#40%#50%#60%#70%#

Interpersonal##Communica;ons#

Principles##of#Nutri;on#

Medical##Terminology#

Adjusted

(stude

nt(success(ra

te(

Error#bars:#+/I#1.00#SE#

Adjusted(student(success(rates(by(course(

Control# Pilot#

47

Student quote on benefits of added practice

“Something I found to be interesting was the degree of understanding between me and another individual that wasn’t in this class. A girl I had met in a previous term that has a similar degree plan but ended up in a regular medical terminology course, still we would discuss the differences and similarities between are assigned classes. During our unit 8 test she called me hysterical about all the different elements of the final tests and couldn’t seem to grasp the concept of the 1st part of the test i.e., analysis diagram, creating new terms from word roots etc. I was mystified that something that had become 2nd nature to me mainly due to the time spent every week filling out the Analysis Tables was so difficult for her to comprehend. It was at that point I realized all the griping I had done was actually the reason my level of understanding is more evolved than somebody who never experienced it.”

48

Quote from a student who previously failed

“This course was difficult for me to do. I tried to do this course when I attended another school and I failed it. I think the way the course was set up and how it broke everything down really helped me to understand it and pass it this time. I would not change a thing about how this course was set up.”

49

Satisfaction (end of term survey, 5 point scale)

Instructor satisfaction •  Higher in redesigned courses (mean 4.6 vs 4.1) Why?

•  Detailed scoring guides for assignments •  Less time in discussions – more time to monitor and

communicate with at-risk students •  Standard seminar format and content •  Student materials: structure, clarity, practice

Student satisfaction

•  Lower on end of course survey in redesigned courses (mean 4.4 vs 4.8), but still greater than 4 on 5 point scale. High positivity scores in motivation survey.

Why? •  Courses more rigorous, more work to complete; this is a

common finding in other research (e.g., Clark, 1982)

50


51

Useful references:

•  Why Students Don’t Like School, Daniel Willingham – highly readable! ;-) •  Talent is Overrated, Geoffrey Colvin – highly readable! ;-) •  E-Learning and the Science of Instruction, Clark and Mayer, 2nd ed. •  “First Principles of Learning,” Merrill, D., in Reigeluth, C. M. & Carr, A. (Eds.),

Instructional Design Theories and Models III, 2009. •  How People Learn, John Bransford et al, eds. •  “Design factors for educationally effective animations and simulations,” Plass,

J.L., Homer, B.D., Hayward, E.O., J Comput High Educ (2009) 21:31–61 •  “The Implications of Research on Expertise for Curriculum and Pedagogy”,

David Feldon, Education Psychology Review (2007) 19:91–110 •  “Cognitive Task Analysis,” Clark, R.E., Feldon, D., van Merrienboer, J., Yates,

K., and Early, S.. in Spector, J.M., Merrill, M.D., van Merrienboer, J. J. G., & Driscoll, M. P. (Eds.), Handbook of research on educational communciatinos and technology (3rd ed., 2007) Lawrence Erlbaum Associates

evidence-based work on learning and...

Documents