Approaches to Evaluating Blended Learning Programs

• Mary Esselman, Deputy Chancellor, EAA, Michigan• Ernie Silva, Director of External Affairs, SIATech,

California• Elizabeth Hessom, Director of Education Services,

SIATech

January 2014

Webinar Format• Overview

• Introductions

• Presentation

• Type questions in the chat window

Blended Learning• “a formal education program in which a student

learns at least in part through online delivery of content and instruction with some element of student control over time, place, path, and/or pace and at least in part at a supervised brick-and-mortar location away from home…” - (Horn and Staker, 2013)

Tech-rich = blended

Teaching and Learning• What the student is doing

and where the student is.

What the teacher is doing and where the teacher is.

What and where the content is.

Webinar Focus

• Blended Learning Program Evaluation– What ongoing measures are used to measure

program effectiveness?– What evaluation tools/processes are used by

teachers to determine student and program success?

– What evaluation tools/processes are used by administrators and leadership to determine student and program success?

Today’s Presenters

• Ernie Silva, Director of External Affairs, SIATech, California

• Mary Esselman, Deputy Chancellor, EAA, Michigan

SIATech Overview

• Appropriate Accountability for Drop Out Recovery

Blended Learning &

Competency Based

Job training, mentors

& internshi

ps

= Educated Productive Graduates

+

Adaptive Assessments

+

Our programs are recovering drop-outs. It is that simple.

13,186 standards-aligned diplomas earned since 1998.

Key lessons have been learned in the past 15 years serving a 100% drop-out population.

Key lessons have been learned in the past 15 years serving a 100% drop-out population.

  Florida

[N=1241]

California

[N=2110]

Arizona

[N=640]

New Mexico

[N=302]

Arkansas

[N=261]

Average Age on Entry

19.31 19.44 19.81 19.06 18.77

STAR Math  EntryG.E.   6.4  [N=1128]

G.E.  6.4  [N=1805]

G.E.   6.0 [N=430]

G.E.   6.1 [N=270]

G.E.  6.0 [N=282]

STAR Reading EntryG.E.   6.9   [N=1133]

G.E.  7.2  [N=1806]

G.E.   6.7 [N=431]

G.E.   7.4 [N=272]

G.E.  6.6 [N=304]

Socioeconomically Disadvantaged 88% 100% 100% 98% 100%

Educational Achievement Authority (EAA) Overview

• Detroit, Michigan

“The mission of the EAA, as a catalyst for change, is to disrupt traditional public schooling and provide a

prototype for next generation learning.”

Year 1 Year 2 Year 3

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

EAA – Year 1 – Five Pillars

1.Students are grouped by readiness, not by age or grade.

2.Students create and assume ownership for their respective personalized learning paths and are able to communicate their progress relative to their individualized learning goals.

3.Students are allowed to work at their own pace using a blended delivery system to master rigorous standards aligned to next generation readiness.

4.Students provide evidence of mastery through relevant performance tasks and common assessments.

5.Continuous feedback is provided to students, teachers, administrators and parents.

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

EAA – Year 2

• Professional Development was redesigned to follow a blended, personalized delivery model

• Schools began to look at new ways to use time, space, talent and resources.

• PASE, the SCL Village, the Open Math Lab, and the Learning Pods were all created within EAA schools to allow students to move completely at their own pace and to start leveraging time, space, and talent more efficiently and effectively.

• Students began transitioning freely between learning spaces and time. Students are no longer tethered to one classroom and one subject.

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

EAA – Year 3:Next Generation High School

• Staffing structures will be redesigned to truly leverage time, talent, and space with a distributed leadership model rather than a traditional principal led school.

• Students will demonstrate mastery of Next Generation Competencies that truly prove they are Next Generation Ready.

• Staffing structures will allow for multiple adults, including full time and part-time teachers, tutors, paras, instructional assistants, mentors and interventions

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

“The mission of the EAA, as a catalyst for change, is to disrupt traditional public schooling and provide

a prototype for next generation learning.”

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

NGLC Seven Design Principles

Student Centered

High Expectations

Self-Pacing and Mastery Base.d

Blended Instruction

Student Ownership

Financial Sustainability

Scalable

EAA Year One

Five Pillars:

1.Students are grouped by readiness, not by age or grade.2.Students create and assume ownership for their respective personalized learning paths and are able to communicate their progress relative to their individualized learning goals.3.Students are allowed to work at their own pace using a blended delivery system to master rigorous standards aligned to next generation readiness. 4.Students provide evidence of mastery through relevant performance tasks and common assessments.5.Continuous feedback is provided to students, teachers, administrators and parents.

EAA Year One

Five Pillars:

1.Students are grouped by readiness, not by age or grade.2.Students create and assume ownership for their respective personalized learning paths and are able to communicate their progress relative to their individualized learning goals.3.Students are allowed to work at their own pace using a blended delivery system to master rigorous standards aligned to next generation readiness. 4.Students provide evidence of mastery through relevant performance tasks and common assessments.5.Continuous feedback is provided to students, teachers, administrators and parents.

.

EAA Year Two•Professional Development was redesigned to follow a blended, personalized delivery model

•Schools began to look at new ways to use time, space, talent and resources.

•PASE, the SCL Village, the Open Math Lab, and the Learning Pods were all created within EAA schools to allow students to move completely at their own pace and to start leveraging time, space, and talent more efficiently and effectively.

•Students began transitioning freely between learning spaces and time. Students are no longer tethered to one classroom and one subject.

.

EAA Year Two•Professional Development was redesigned to follow a blended, personalized delivery model

•Schools began to look at new ways to use time, space, talent and resources.

•PASE, the SCL Village, the Open Math Lab, and the Learning Pods were all created within EAA schools to allow students to move completely at their own pace and to start leveraging time, space, and talent more efficiently and effectively.

•Students began transitioning freely between learning spaces and time. Students are no longer tethered to one classroom and one subject.

EAA Year Three

Next Generation High School

•Staffing structures will be redesigned to truly leverage time, talent, and space with a distributed leadership model rather than a traditional principal led school.

•Students will demonstrate mastery of Next Generation Competencies that truly prove they are Next Generation Ready.

•Staffing structures will allow for multiple adult including full time and part-time teachers, tutors, paras, instructional assistants, mentors and interventions

EAA Year Three

Next Generation High School

•Staffing structures will be redesigned to truly leverage time, talent, and space with a distributed leadership model rather than a traditional principal led school.

•Students will demonstrate mastery of Next Generation Competencies that truly prove they are Next Generation Ready.

•Staffing structures will allow for multiple adult including full time and part-time teachers, tutors, paras, instructional assistants, mentors and interventions

SIA Tech Assessment

• Achievement scores on state mandated exams penalize SIATech students, because the majority enter our schools achieving at the 6th grade level in reading & math.  

• State tests do not work well for us

• Prescribed time table• Limited administrations• Too hard/too long for too 

many

• Achievement scores on state mandated exams penalize SIATech students, because the majority enter our schools achieving at the 6th grade level in reading & math.  

• State tests do not work well for us

• Prescribed time table• Limited administrations• Too hard/too long for too 

many

SIATech schools needed a different way to demonstrate accountable for student learning!

Advocating for Alternative Accountability

Reaching At Promise Students Association

http://www.rapsa.orgSave the date: November 14-15 San

Diego 3rd Annual Alternative Accountability

Policy Forum

• Adaptive

• Multiple administration times

• Short

• Correlated to state standards

• Reliable

• Measure & report growth and achievement

• Can use results to improve learning and teaching

• Adaptive

• Multiple administration times

• Short

• Correlated to state standards

• Reliable

• Measure & report growth and achievement

• Can use results to improve learning and teaching

Renaissance STAR Math and Reading*

270

299

349

400

Far Below Basic150

Below Basic

Basic

Proficient

Advanced

600

Grade 6 Grade 7

CS

T E

LA

SC

ALE S

CO

RE

305

330

Grade 8

34525 15

• Student learning growth is a statistical measure that estimates how much each student progresses from one test to the next compared to similar students

What is learning growth?

Benefit #1 – Holds schools accountable for the

things they have control over

Benefits of measuring learning growth -

Benefit #2 – Measuring learning growth fosters

student hope and engagement because gains are

based on individual progress – not unrealistic

targetsBenefit #3 – Measuring student learning growth sets

high expectations for every student, because every

student (high, middle, and low achieving) must

increase his or her learning every year

• Interpretation of Individual Student Growth Data

• Individual Student Goal Setting (weekly, biweekly, monthly)

• Student Accountability for Tracking and Adjusting Academic and Learning Progress

• Evaluation of Course Completion and Academic Growth

• Interpretation of Individual Student Growth Data

• Individual Student Goal Setting (weekly, biweekly, monthly)

• Student Accountability for Tracking and Adjusting Academic and Learning Progress

• Evaluation of Course Completion and Academic Growth

Students take ownership of personal learning as schools make efficient use of core processes

Annual Development of SITE Professional Development Plans

• Interpretation of Learning Data to determine general success and gaps in school outcomes.

• Site Goal Setting• Determination of a Site

Professional Development Focus

• Evaluation of Plan Implementation and Results (EOY)

• Interpretation of Learning Data to determine general success and gaps in school outcomes.

• Site Goal Setting• Determination of a Site

Professional Development Focus

• Evaluation of Plan Implementation and Results (EOY)

Annual Development of Individual Staff Member Professional Development Plans

• Interpretation of Learning Data as related to specific content area(s).

• Individual Staff Goal Setting

• Planning Pertinent Training (examples)

• Lesson Plan Development

• Differentiation

• Knowledge Acquisition

• Teaching Strategies

• Data Driven Instruction

• Evaluation of Plan Implementation and Outcomes (EOY)

• Interpretation of Learning Data as related to specific content area(s).

• Individual Staff Goal Setting

• Planning Pertinent Training (examples)

• Lesson Plan Development

• Differentiation

• Knowledge Acquisition

• Teaching Strategies

• Data Driven Instruction

• Evaluation of Plan Implementation and Outcomes (EOY)

• Monthly Content Teacher Conversations Wednesdays 3:30-4:30 pm

Math, ELA, Science, Social Studies, CTSP (4/year)

Sharing of best content practice with a focus on:

- literacy (reading and writing across content areas)

- small group lessons- opening and closing whole group

lessons/activities

• Monthly distribution of active student learning data

• Research into a computer based, individual student reading program (included into 2013-14 budget for all sites)

Multiple Measures Include Gallup Pollof Hope, Engagement and Well Being

Hopethe ideas and energy

we have for the future.

Hope drives:attendance, credits

earned, and high school GPA

Hope predicts: college GPA and

retention

Hope scores are better predictors of college

success than high school GPA, SAT, and

ACT scores.

Ways around a problem

Average student response on a five point scale, where 1=strongly disagree and 5=strongly agree.

Response Average

Possible Student Activities to Impact this Area:During direct instruction problem-solving, ask

students for other ways they would solve the problem or issue.

Remind students of the variety of ways they can get help and support for problem-solving

Help students learn that there are many ways to success and learning (not just innate talent-there are MANY talented but unsuccessful folks)

teach multiple problem solving techniques encourage students to learn how they ‘best learn of

learn best’ provide students with specific strategies to deal with

their ‘blocks’ (e.g. memorizing, planning, organizing, classifying, preparing, …)

Your Reflections / Actions for this Area:

Question Text:

I can find lots of ways around any problem.

(Five point scale)

Question Text:

I can find lots of ways around any problem.

(Five point scale)

HOPE

Engagement the involvement and

enthusiasm for school.

Engagement scores separate

high-performing from low-performing schools

Schoolwork important

ENGAGEMENT

Question Text:

My teachers make me feel my schoolwork is important.(Five point scale)

Question Text:

My teachers make me feel my schoolwork is important.(Five point scale)

Average student response on a five point scale, where 1=strongly disagree and 5=strongly agree.

Response AverageResponse Average

Possible Student Activities to Impact this Area:

Teachers provide frequent, sincere, specific feedback on schoolwork in a variety of ways

Teachers make relevant connections between schoolwork and goal attainment;

Students are provided with opportunities to discuss with teachers and/or mentors how their schoolwork is helping them…

Your Reflections / Actions for this Area:

Well-Being how we think about and experience

our lives.

Well-being is an indicator of how students are currently doing.

Well-being also predicts future success.

A thriving student earns 10% more credits and a 2.9 GPA compared to a struggling or suffering student who completes fewer credits and earns a

2.4 GPA.

Treated with respectWELL-BEING

Question Text:

Were you treated with respect all day yesterday?(Yes or no)

Question Text:

Were you treated with respect all day yesterday?(Yes or no)

Percent who said Yes

Possible Student Activities to Impact this Area:

• ALL staff knows and USES student names• Be and Do what you want to see from students• Explain the reasons/why or why not for

critiques of student work, not just X• Reward and/or acknowledge acts of respect to

build a culture• Demonstrate positive approach to classroom

management and student discipline (preserve dignity)

• Find ways to help students answer their own questions and then honor their success

Your Reflections / Actions for this Area:

EAA Assessment

Program Evaluation Questions

Question 1: To what extent is the student-centered system being executed as planned?

Question 2: To what extent is there sufficient support available to staff, parents, and students who are participating in a student-centered system of education?

Question 3: To what extent is implementation of the student-centered system achieving desired changes in student performance outcomes, socio-emotional well-being and teacher practice?

Question 4: To what extent is the program being scaled in terms of depth, sustainability, spread, shift, and evolution?

4-pronged Assessment System

Individual Growth (Performance Series) Performance Tasks to assess Deeper Learning

(3 pieces of evidence for each Learning Target) Learn (reflection) Practice (Understanding) Apply (real-world connections) Common Assessments

Achievement (State Assessment) Next Generation Ready—Graduation by Defense

Integrated Teaching & Learning

Platform.

Real-time Analytics

• Supports a robust collection of open sourced, licensed, and teacher created resources.

• Learning objects parsed in bite sized chunks.

• Digital assets mapped to standards--local, state, common core.

• Curriculum maps directly linked to the digital assets

• Students have access to choice-based learning pathways.

• Dashboards provide data for students, parents and teachers.

• Data is provided as a prescriptive newsfeed for immediate application.

• Integrated planner, profiles, personalized learning plans with student, teacher, parent, communication tools.

• Peer to Peer support system.

Part 2: Access to Information—monthly, weekly, real-time

Time & Topic Pace/Progress Badges for leadership, scholarship, character,

service personal milestones Student Profile Mastery:

School/Class/Individual by Level/Unit/Standard Mastered/In progress/Not Yet Attempted Digital Portfolio of Evidence

The teaching and learning platform:The student experience…

EDUCATION ACHIEVEMENT AUTHORITY OF MICHIGANEDUCATION ACHIEVEMENT AUTHORITY OF MICHIGAN

Student Progress Page

Creating a personalized learning path

The Teacher Experience:  3-D Classroom Radar Report of Student Productivity—who’s progressing and who is not vs. traditional 2-D, linear gradebook.

Color-coding on students represents self-assessment information

Color-coding on students represents self-assessment information

Click a student to drill down to view real-time personal performance

Click a student to drill down to view real-time personal performance

Classroom Summary ReportTeachers can click to select individual or groups of students to assign supplemental resources and/or send notes.

Teachers can click to select individual or groups of students to assign supplemental resources and/or send notes.

Click a student to drill down to view real-time personal performance

Click a student to drill down to view real-time personal performance

Pie charts indicate mastery of

individual learning targets.Pie charts indicate mastery of

individual learning targets.

Self-assessment resultsSelf-assessment results

Standards Report

Teachers, students and parents can review progress against individual standards as well as access supplemental study resources.

Teachers, students and parents can review progress against individual standards as well as access supplemental study resources.

Teacher Grading and Messaging Center

Item for grading are collected in the to-do list making it easy for teachers to provide grades and feedback.

Item for grading are collected in the to-do list making it easy for teachers to provide grades and feedback.

Just as teachers can monitor real-time student progress they also can monitor

their own progress.

Just as teachers can monitor real-time student progress they also can monitor

their own progress.

Teacher Personal Progress

Lessons Learned - EAA

Part 3: Other Assessments of Progress/Implementation End of Course for non-tested Subjects GRIT Value-added Growth Model for Pay for

Performance S ystem Instructional Practice—scaffolded

walkthroughs Classroom Set-up/Visual Cues Classroom Structures Rituals/Routines Student Ownership/Engagement

Note: We triangulate data across reports/surveys/observations/focus groups

Lessons Learned – SIA Tech

1. A Learning Growth model is imperative for School Evaluation/Monitoring of reengaged students.

2. Graduation Rate Calculations must be appropriate in terms of cohort.

3. Strong advocacy work at the local, state and federal level needs to be a priority.

4. Competency-based education needs engaging, specialized and customized curriculum to truly accelerate learning

5. Staff development and training is crucial to success with a dropout population

6. A robust data system needs to be in place in order to show success in multiple, meaningful ways.

Contact Information

• Mary Esselman, EAA. – MEsselman@eaaofmichigan.org

• Ernie Silva, SIATech– Ernie.Silva@siatech.org

• Elizabeth Hessom, SIATech– Elizabeth.Hessom@siatech.org