Measuring the Effectiveness of Virtual School Programs

Research and BeyondJuly 10 2007

2

Introductions and Program Overviews

bull VHS Global Consortium ndash Liz Pape

bull IDLA -Idaho Digital Learning Academy ndash Donna Vakili

bull eHigh School Cobb County GA ndash Becky Nunnally

3

What does an effective virtual school program look like

What are the indicators for measurement and how are they defined

How are virtual school programs held accountable

How do we communicate if we all define ourselves by different standards and definitions

4

EVALUATION is integral to a continuous quality improvement process

What do we evaluate

Online Course (Becky)Online Instruction (Donna)Student Learning (Liz)Online Program ( All)

5

ldquoFor many students online access has changed the way they see the world and the way they work and play Consideration of these student issues is now critical in designing and delivering quality online instructionrdquoStandards for Quality Online Courses SREB November 2006

Why do we evaluate an online course

To improve the course

To improve student learning

6

How do we evaluate an online course

Clearly defined objectives aligned to standards

Linking learning objectives to specific learning activities course

materials assessment and measurement

Opportunities for communication and interaction including student to

student student to instructor and student to content

Highly engaging activities that meet the needs of all learning styles

and engage the student

Developed in teams and meets standards and quality measures by

peers

7

Ex eHigh School Course Evaluation Process

Technical design

Curriculum alignment

Rigor depth and breadth

Student performance

Student participation and interaction

Ongoing

8

Why do we evaluate online course instruction

To improve course designTo improve delivery of course contentTo improve course facilitation and managementTo meet the needs of our customers

9

How do we evaluate online course instruction

Focuses on whether the objectives of the course were met amp informs practice

Pre-Mid Post-course surveys to all stakeholdersLearner reflectionsPeer and Administrator reviewInstructor self-evaluation

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

2

Introductions and Program Overviews

bull VHS Global Consortium ndash Liz Pape

bull IDLA -Idaho Digital Learning Academy ndash Donna Vakili

bull eHigh School Cobb County GA ndash Becky Nunnally

3

What does an effective virtual school program look like

What are the indicators for measurement and how are they defined

How are virtual school programs held accountable

How do we communicate if we all define ourselves by different standards and definitions

4

EVALUATION is integral to a continuous quality improvement process

What do we evaluate

Online Course (Becky)Online Instruction (Donna)Student Learning (Liz)Online Program ( All)

5

ldquoFor many students online access has changed the way they see the world and the way they work and play Consideration of these student issues is now critical in designing and delivering quality online instructionrdquoStandards for Quality Online Courses SREB November 2006

Why do we evaluate an online course

To improve the course

To improve student learning

6

How do we evaluate an online course

Clearly defined objectives aligned to standards

Linking learning objectives to specific learning activities course

materials assessment and measurement

Opportunities for communication and interaction including student to

student student to instructor and student to content

Highly engaging activities that meet the needs of all learning styles

and engage the student

Developed in teams and meets standards and quality measures by

peers

7

Ex eHigh School Course Evaluation Process

Technical design

Curriculum alignment

Rigor depth and breadth

Student performance

Student participation and interaction

Ongoing

8

Why do we evaluate online course instruction

To improve course designTo improve delivery of course contentTo improve course facilitation and managementTo meet the needs of our customers

9

How do we evaluate online course instruction

Focuses on whether the objectives of the course were met amp informs practice

Pre-Mid Post-course surveys to all stakeholdersLearner reflectionsPeer and Administrator reviewInstructor self-evaluation

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

3

What does an effective virtual school program look like

What are the indicators for measurement and how are they defined

How are virtual school programs held accountable

How do we communicate if we all define ourselves by different standards and definitions

4

EVALUATION is integral to a continuous quality improvement process

What do we evaluate

Online Course (Becky)Online Instruction (Donna)Student Learning (Liz)Online Program ( All)

5

ldquoFor many students online access has changed the way they see the world and the way they work and play Consideration of these student issues is now critical in designing and delivering quality online instructionrdquoStandards for Quality Online Courses SREB November 2006

Why do we evaluate an online course

To improve the course

To improve student learning

6

How do we evaluate an online course

Clearly defined objectives aligned to standards

Linking learning objectives to specific learning activities course

materials assessment and measurement

Opportunities for communication and interaction including student to

student student to instructor and student to content

Highly engaging activities that meet the needs of all learning styles

and engage the student

Developed in teams and meets standards and quality measures by

peers

7

Ex eHigh School Course Evaluation Process

Technical design

Curriculum alignment

Rigor depth and breadth

Student performance

Student participation and interaction

Ongoing

8

Why do we evaluate online course instruction

To improve course designTo improve delivery of course contentTo improve course facilitation and managementTo meet the needs of our customers

9

How do we evaluate online course instruction

Focuses on whether the objectives of the course were met amp informs practice

Pre-Mid Post-course surveys to all stakeholdersLearner reflectionsPeer and Administrator reviewInstructor self-evaluation

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

4

EVALUATION is integral to a continuous quality improvement process

What do we evaluate

Online Course (Becky)Online Instruction (Donna)Student Learning (Liz)Online Program ( All)

5

ldquoFor many students online access has changed the way they see the world and the way they work and play Consideration of these student issues is now critical in designing and delivering quality online instructionrdquoStandards for Quality Online Courses SREB November 2006

Why do we evaluate an online course

To improve the course

To improve student learning

6

How do we evaluate an online course

Clearly defined objectives aligned to standards

Linking learning objectives to specific learning activities course

materials assessment and measurement

Opportunities for communication and interaction including student to

student student to instructor and student to content

Highly engaging activities that meet the needs of all learning styles

and engage the student

Developed in teams and meets standards and quality measures by

peers

7

Ex eHigh School Course Evaluation Process

Technical design

Curriculum alignment

Rigor depth and breadth

Student performance

Student participation and interaction

Ongoing

8

Why do we evaluate online course instruction

To improve course designTo improve delivery of course contentTo improve course facilitation and managementTo meet the needs of our customers

9

How do we evaluate online course instruction

Focuses on whether the objectives of the course were met amp informs practice

Pre-Mid Post-course surveys to all stakeholdersLearner reflectionsPeer and Administrator reviewInstructor self-evaluation

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

5

ldquoFor many students online access has changed the way they see the world and the way they work and play Consideration of these student issues is now critical in designing and delivering quality online instructionrdquoStandards for Quality Online Courses SREB November 2006

Why do we evaluate an online course

To improve the course

To improve student learning

6

How do we evaluate an online course

Clearly defined objectives aligned to standards

Linking learning objectives to specific learning activities course

materials assessment and measurement

Opportunities for communication and interaction including student to

student student to instructor and student to content

Highly engaging activities that meet the needs of all learning styles

and engage the student

Developed in teams and meets standards and quality measures by

peers

7

Ex eHigh School Course Evaluation Process

Technical design

Curriculum alignment

Rigor depth and breadth

Student performance

Student participation and interaction

Ongoing

8

Why do we evaluate online course instruction

To improve course designTo improve delivery of course contentTo improve course facilitation and managementTo meet the needs of our customers

9

How do we evaluate online course instruction

Focuses on whether the objectives of the course were met amp informs practice

Pre-Mid Post-course surveys to all stakeholdersLearner reflectionsPeer and Administrator reviewInstructor self-evaluation

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

6

How do we evaluate an online course

Clearly defined objectives aligned to standards

Linking learning objectives to specific learning activities course

materials assessment and measurement

Opportunities for communication and interaction including student to

student student to instructor and student to content

Highly engaging activities that meet the needs of all learning styles

and engage the student

Developed in teams and meets standards and quality measures by

peers

7

Ex eHigh School Course Evaluation Process

Technical design

Curriculum alignment

Rigor depth and breadth

Student performance

Student participation and interaction

Ongoing

8

Why do we evaluate online course instruction

To improve course designTo improve delivery of course contentTo improve course facilitation and managementTo meet the needs of our customers

9

How do we evaluate online course instruction

Focuses on whether the objectives of the course were met amp informs practice

Pre-Mid Post-course surveys to all stakeholdersLearner reflectionsPeer and Administrator reviewInstructor self-evaluation

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

7

Ex eHigh School Course Evaluation Process

Technical design

Curriculum alignment

Rigor depth and breadth

Student performance

Student participation and interaction

Ongoing

8

Why do we evaluate online course instruction

To improve course designTo improve delivery of course contentTo improve course facilitation and managementTo meet the needs of our customers

9

How do we evaluate online course instruction

Focuses on whether the objectives of the course were met amp informs practice

Pre-Mid Post-course surveys to all stakeholdersLearner reflectionsPeer and Administrator reviewInstructor self-evaluation

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

8

Why do we evaluate online course instruction

To improve course designTo improve delivery of course contentTo improve course facilitation and managementTo meet the needs of our customers

9

How do we evaluate online course instruction

Focuses on whether the objectives of the course were met amp informs practice

Pre-Mid Post-course surveys to all stakeholdersLearner reflectionsPeer and Administrator reviewInstructor self-evaluation

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

9

How do we evaluate online course instruction

Focuses on whether the objectives of the course were met amp informs practice

Pre-Mid Post-course surveys to all stakeholdersLearner reflectionsPeer and Administrator reviewInstructor self-evaluation

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

10

Ex IDLA Teacher Evaluation External (Fall lsquo04-Fall lsquo05)

bull Student Surveys (N=715)ndash 87 of students had positive and rewarding interactions with their

online teacherndash 88 said that the online teacher was very supportive and helped

them learnndash 89 said that the teacher was good at moderating course

discussionsndash 86 felt that the teacher met their needsndash 85 felt that the teacher was prompt and helpful with feedbackndash 84 felt encouraged to express themselves throughout the coursendash 84 felt satisfied with the extra online help received from the

teacherndash Areas of Improvement Declining trend in experience with teachers

(up 11 for Summer rsquo06) Increase prof dev of teachers in moderating online discussions more teacher interaction

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

11

Ex IDLA Teacher Evaluation External (Spring 2006)

bull Administrators (N=152)ndash Overall 86 of administrators rate IDLA as good or

excellentndash 85 of administrators rate responsiveness of IDLA teachers

to needs of their school and students as good or excellentndash 64 rate the quality of the IDLA teachers as average to

above avg (30 Donrsquot Know)ndash Areas of improvement Improve student progress reporting

clear lines of communication and schools know their responsibilities

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

12

Ex IDLA Teacher Evaluation Internal Methods

bull Independent Contractorsndash Non-Renewablendash Teaching Expectations

bull Bonus Teaching Rubricndash Administrative Requirements (25)ndash Professional Development (20)ndash Communication and Intervention (45)ndash Student Evaluations (10)

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

13

Ex Evaluating Online Course Instruction Informs Practice at IDLA

bull Mandated Online Site Coordinator Coursebull Instituted Teacher Rubricbull Systemized and clarified role of online principal and

evaluationbull Established systems and consistency for reporting such

as phone logs and progress reportsbull Guided Online and Summer Conference Professional

Development for Facultybull Focused Emphasis on Quality amp Customer Servicebull Increased Value for External Evaluations

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

14

Why do we evaluate student learning

Mastery of course content

Application of course content to the real world

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

15

How do we evaluate student learning

Pre-MidPost-course surveysLearner reflectionsPeer reviewInstructor self-evaluationEnd-of-course assessments

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

16

Ex VHS Student Learning Measures

bull AP pass rates 70

bull AP exam take rates79

bull Course completion rates 79

bull Credit recovery rate (summer)88

bull TeacherSCPrincipalSuperStudent Survey input

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

17

Ex VHS AP exam take rate with of Students

AP Exam Take Rates

7989

60

49

80 302

124

68

1843

0102030405060708090

100

0001 0102 0203 0304 0405

0

50

100

150

200

250

300

350

AP ExamParticipationRate AP Students

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

18

Ex VHS Course Completion Rate

Course Completion Rate

818383

8179807979

0102030405060708090

100

9798 9899 9900 0001 0102 0203 0304 0405

CourseCompletionRate

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

19

Why do we evaluate an online program

AccountabilityStewardship of fundsImprove student achievementMeet the needs of the learning communityPlan for the future

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

20

How do we evaluate an online program

Program UtilizationEnrollment (seats)Number of students servedNumber of students served with no other opportunityCourseTeacher Evaluations

By studentsBy parents

Counselor and Administrator Surveys and Interviews

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

21

Ex VHS Growth GoalNumber of Students

VHS students (Projected 0707)

710 943

25163675 3736 4063

50696114

7604

9500

0

2000

4000

6000

8000

10000

9798 9899 9900 0001 0102 0203 0304 0405 0506 0607

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

22

Ex VHS Professional Development Measures ndashHard Data Points Research and Survey Data

bull Professional development grad rate 89bull 1st-semester teacher success rate 94bull of teachers remaining with FA 4bull of external evaluationsbull Satisfied with VHS professional development

ndash 99 principals

bull Applying VHS pedagogy to f2f teachingndash 79 teachers

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

23

Ex VHS Performance Metrics-Program and Services Quality

bull Membership retention rates 87bull Seat utilization 83bull of external evaluationsbull Satisfaction with electronic communications email

ndash Teachers 99 SCs 96

bull Websitendash Teachers 97 SCs 98

bull NewsletterAnnual Reportndash Superintendents 8668ndash Principals 7879ndash Teachers 7954ndash SCs 8454

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

24

Ex VHS Quality Benchmark Indicators

Indicators of Success

0

20

40

60

80

100

AP Pass Rate CourseCompletions

Satisfactionw ith VHS

High CourseQuality

01-05(cum)all APstudents

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

25

eHigh School Program Measures of Effectiveness

Student success - completions - passing - exceedsmeets EOCT Meets local school needs - scheduling conflicts - graduation requirements Meets student needs - flexibility - acceleration - credit recovery - hospitalhomebound - family obligations

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

26

Tools of Tools of EngagementEngagementWhole Course Whole Course

DeliveryDeliveryBlended Blended LearningLearningBlackboard

Live Classroom

BlogsWikis

Podcasts

Polar Watches

Turnitincom

AP Courses

Work Study Programs

Career Education

High School

Middle School

Faculty Collaboration

Mathematics

Language Arts

Science

Social Studies

Health amp PE

Business

Meeting our 21Meeting our 21stst Century Students Century Students OnlineOnline

Cobb County School Districtwwwcobbk12ehighschoolorg Marietta Georgia

ehighschoolcobbk12org

ISTE NETSS Standards 1 Creativity amp Innovation 2 Communications amp Collaboration

3 Research and Information Fluency 4 Critical Thinking Problem-Solving amp Decision-Making5 Digital Citizenship 6 Technology Operations amp Concepts

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

27

Ex Virtual Charter Schools QualityEvaluation

bull Similar to Supplementalndash Inputs Sourcing Structure Alignmentndash User Input Surveys Feedback ldquoPopularityrdquondash Results Successful Completion

bull Different from Supplementalsndash Total Student Progress Monitoring (and Intervention

with Multiple Tools) ndash NCLB Accountability ndash Test Resultsndash Systemic Improvement ndash Courses Curriculum

Platform Instruction

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

28

Measuring the Effectiveness of Virtual School Programs Research amp Beyond

ISSUESIndicators for measurementDefinition of indicatorsTimeframe for evaluationExample Completion Rates

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

29

Measure of Effectiveness

bull Policy Changesndash Establishment of State Virtual Schoolsndash Michigan Requirement for High School

Graduation

bull Fundingndash Changes to State Law and funding formulasndash Examples IDLA Georgia and Florida Virtual

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

30

And still more questions

bull Are we measuring what we should be measuring

bull What else should we measurebull Who should be measuringbull How do we publish results of measuresbull Where is the Consumer Reports for virtual

schools and online courses

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

31

Will the growing national investment in Kndash12 online learning particularly implementing and expanding the

use of online high school courses provide a basically sound high-quality educational

experience for online students Or will this new investment in Kndash12 online

learning be yet another waste of scarce national educational resources

on todayrsquos newest educational technologies that is destined to be abandoned

in some virtual tomorrowrdquo

(A Synthesis of New Research on Kndash12 Online Learning)

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg

32

References

bull eSchool wwwcobbk12ehighschoolorg bull Idaho Digital Learning Academy

wwwidahodigitallearningorg bull Keeping Pace with K12 Online Learning A Review of

State Level Policy and Practice bull North American Council for Online Learning (NACOL)

wwwnacolorg bull A Synthesis of New Research on Kndash12 Online Learning

httpwwwncrelorgtechsynthesissynthesispdfbull Virtual High School (VHS) wwwgoVHSorg