1

Flip the Script to Improve Learning Measurably

How Learning Engineered Online Courses Coupled with Learning Analytics Improves Learning Outcomes.

Michaele J. Brown, PhDEducational Technology Consultant

2

AgendaTime Frame Topics

11:30 – 11:35 Trends in Higher Education11:35 – 12:00 Community College / OLI Project:

“Introduction to Biology” Course

12:00 – 12:05 Challenges to Adaptive Learning12:05 – 12:15 Q & A

3

Trends in Higher Education Greater emphasis on feedback &

formative assessment Flipped instruction Learning dashboards displaying real-time

student performance Adaptive instruction informed by real-time

student performance

4oli.cmu.edu

Learner-Centered Instruction

Learning results from what the student does and thinks and only from what the student does and thinks. The teacher can advance learning only by influencing what the student does to learn.

Herbert Simon, 2001

It’s not teaching that causes learning. Attempts by the learner to perform cause learning, dependent upon the quality of feedback and opportunities to use it.

Grant WigginsPresident, Center of Learning Assessment

5oli.cmu.edu

6

Practice and Feedback Clear, observable, measureable, performance

goals Models of goals Practice directed toward specific levels of

performance Feedback on performance against goals and

models Self-assessment and opportunities for further

practice

Flipped Learning

Ref: Flipped Learning Network http://fln.schoolwires.net/FLN

Founders: Aaron Sams and Jonathan Bergmann 7

• View video lecture• Read & reflect on text• Learn by doing

Before Class In-Class

Facilitated Instruction

Individualized Learning Collaborative Work

Flipped Learning

Four Pillars of F-L-I-P: Flexible environments

o Anytime, anywhere access, at your own pace.

Intentional contento Video lectures, text resources, and active learning

Learning cultureo Learner-centered instruction

Professional educatorso Coaches, guides, tutors, and facilitates

Ref: Foundations of Flipped Learning http://fln.schoolwires.net//site/Default.aspx?PageID=39

Founders: Aaron Sams and Jonathan Bergmann 8

9oli.cmu.edu

Transformation of Higher Education

Real-time Learning Analytics

Flipped Instructional Pedagogy

Goal Directed Practice & Feedback Adaptive

Learning

10oli.cmu.edu

Open Learning Initiative

11oli.cmu.edu

• Open online learning environments based on the science of learning

• An approach to designing, developing, delivering and improving learning experiences

• Science of Learning• Evaluation• ImprovementScience

• Platform• In-course Affordances

Technology

• Team-based Development• Communities of Research

and UseTeams• Capture• In-course Use• Iterative Improvement• Research

Data

What is the Open Learning Initiative?

12oli.cmu.edu

Goal Directed Practice & Targeted FeedbackLearning activities that enact instruction:1. Intelligent tutoring systems2. Simulations3. Videos & animation4. Virtual Labs

13oli.cmu.edu

Team-based Design and Development

14oli.cmu.edu

Data drives powerful Feedback LoopsFeedback: Changing the Productivity of Learners and Teachers

15oli.cmu.edu

OLI Research Results OLI students completed course in half the time with half

the number of in-person meetings

OLI students showed significantly greater learning gains on the national standard “CAOS” test for statistics knowledge

No significant difference between OLI and traditional students in:o Amount of time spent studying statistics outside of class

o Follow-up measures given 1+ semesters later

M. Lovett, O. Meyer, & C. Thille, C., “The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning,” Journal of Interactive Media in Education (2008).

16oli.cmu.edu

Domain experts outside of the core development team provide vital

feedback

Community College OLI

17oli.cmu.edu

“Improvement in post secondary education will require converting teaching from a solo sport to a community based research activity.” Herbert Simon

Nobel Laureate & CMU Professor

Multi-Institution Participation

Three levels of participation by faculty SMEs Institutional participants:

1. Butte Community College2. California Polytechnic State University3. National University4. Prince George Community College

5. Redwood Community College6. Salisbury University7. Towson University8. UMUC

18oli.cmu.edu

Aligning Learning Outcomes and Learning Objects

19oli.cmu.edu

The Course Design Triangle

Instructional Activities

Objectives

AssessmentsTasks that provide

feedback on students’ knowledge

and skills

Descriptions of what students should be able to

do at the end of the course

Contexts and activities that foster students’ active

engagement in learning

20oli.cmu.edu

OLI Used in Hybrid Mode

21oli.cmu.edu

Analyzing Student Performance

22oli.cmu.edu

Introduction to Biology

Course Key: bio-mjb-v1P

23oli.cmu.edu

Learning Activities

1. Learn by Doing2. Did I Get That3. Key Terms 4. Confidence Checks5. Assessments

24oli.cmu.edu

Learning Dashboard

25oli.cmu.edu

Student Performance

26oli.cmu.edu

Student Participation

High Performers

27oli.cmu.edu

Student Participation

Moderate Performers

28oli.cmu.edu

Interaction Level Data

Misconception between genes and chromosomes

oli.cmu.edu

OLI’s Data Driven Process

29

30oli.cmu.edu

LearnLab:Transforming Education Research

NSF Science of Learning Center• 10 years, ~$50 million• Tech enhanced courses, assessment, & research• School cooperation for data collection

Ed tech + wide use = “Basic research at scale”

31oli.cmu.edu

Learning Curve Analysis

DataShop: Pittsburgh Science of Learning Center

32oli.cmu.edu

Other Learning Curves

33oli.cmu.edu

Thousands of combinations,

what works best depends …

What’s best?

Focused practice

Distributed practice

Study examples

Test onproblems

50/50Mix

Concrete AbstractMix

Delayed No feedback

Immediate

Block topics in chapters

Interleave topicsFade

Explain Ask for explanations

Mix

Gradually widen

Study Test50/50 Study Test50/50Study

Concrete Abstract Mix

Immediate No feedbackDelayed

Block topics in chapters Fade Interleave

topics

Explain Ask for explanations

Mix

More helpBasics

More challenge Understanding

34oli.cmu.edu

Better Insight Better Courses

Past research• Careful “cognitive task analysis”

produces much better courses

New opportunity• Ed tech provides data for

automated analysis

OLI Chemistry, equilibrium topic:Previously: 20% correct on examAfter Redesign: 60%

Flat learning curve Discovery opportunity!

35oli.cmu.edu

The “Killer App” Data Collection & Feedback Loops for Continuous Improvement

36oli.cmu.edu

OLI Courses http://oli.cmu.edu/

Introduction to Biology

37oli.cmu.edu

OLI Summary Apply learning science research and scientific inquiry

to course development, implementation and evaluation.

Develop intelligent learning platforms collaboratively with a team of content experts and novices, learning scientists, HCI, software engineers.

Employ feedback loops for continuous improvement.

Establish communities of use, evaluation and improvement.

38oli.cmu.edu

Challenges to Implementing Adaptive Instruction

39

Challenges for Adaptive Learning

1. Deciding on clear and measurable learning goals and objectives

2. Providing good quality and timely feedback3. Visualizing and interpreting leaning analytics,

and implications for adapting instruction.4. Faculty development for productive uses of

class time for learner-centered activities

Adaptive Learning

40

Course End: • Summative Evaluation• Course Improvements

Course Delivery: • Learner-centered activities:

1. Opportunities for feedback & assessment

2. Authentic action based activities

3. Peer interaction and collaboration

4. Experiential, project and inquiry based activities

5. Self actualizing activities

41oli.cmu.edu

Questions?

42

Michaele J. Brown, PhD

Thank You

Learning by design of authentic, goal-based, online experiences.

43oli.cmu.edu

OLI Research Reports