FINAL EVALUATION

REPORT

Effectiveness of Virtual Direct Instruction of Unit 6, MJUSH

Submitted to:

Esther Webb, Instructional Leader for Liberal Arts, Young Virtual School

Submitted by:

Kathryn Booth, Graduate Student, Boise State University

P a g e 1 | 15

Summary

The MJUSH team at Young Virtual School has been providing supplemental direct instruction

sessions to their students for a unit on Jacksonian Democracy over the past six weeks. The

program consists of three sessions for a total of three hours of supplemental, direct instruction.

The team created three new the sessions due to their general consensus that this unit was creating

an above average challenge to their students resulting in low unit exam scores. The team

implemented virtual, direct instructional session for students to complete as a supplemental

resource to the curriculum. Each session was conducted by a team instructor two times per week

in a live virtual classroom. The goal of the project was to provide a greater understanding of the

unit concepts to students which would then result in improved unit exam scores. The ultimate

goal was to increase student exam scores by five percentage points.

An evaluation of the program was requested by the Instructional Leader to determine if the

program objectives were being met. In addition to gathering data on oral and unit exam scores

prior to the program evaluation start date, pre and post program surveys with participating

instructors regarding program effectiveness were collected and analyzed. Pretest, oral exam, and

unit exam scores for participating students were gathered for students who completed the

program during the evaluation period. Additionally, post-program student surveys were also

collected in order to obtain a snapshot of the student perspective on program effectiveness.

Results of the evaluation reflect agreement amongst instructors the need for supplemental

resources in reaching the objectives. Instructors found the programs live sessions to meet the

current need. Suggestions for further development of resources is described in more detail later

in this report. Evaluation of student surveys found the program, from the student perspective, to

be beneficial. Qualitative data reflects the program objectives were met.

Description of the program evaluated

The supplemental direct instruction sessions on Jacksonian Democracy were created by the

Young Virtual School MJUSH team based on the unit objectives and curriculum standards. The

current curriculum and resources had been underway since July 2013. The Instructional Leader,

Team Lead, and team instructors were concerned with the low unit exam scores for Jacksonian

Democracy. All parties cited are individually evaluated on end-of-course exams which include

content on the program unit. As a team, they self-assessed the need for improved unit

six/Jacksonian Democracy unit scores in order to achieve higher end-of-course exam scores.

With the approval of the Instructional Leader, sessions were created by the team and then

conducted by a team instructor in a live, virtual classroom two days per week from June 1, 2014

through July 11, 2014.

Program Objectives

Increase student understanding of the unit concepts

Increase student unit exam scores by five percentage points

P a g e 2 | 15

Program Components

Students that were enrolled in the MJUSH course and had completed course content through unit

five were eligible to participate in the program. Three MJUHS Instructors conducted live, direct

instruction session two times per week in the MJUSH virtual classroom via Blackboard

Elluminate. Students joined one of the three program sessions at a time to complete a total of

three hours of supplemental direct instruction on Jacksonian Democracy.

Prior to the program sessions, all students completed a unit pretest in their online course

assessment area. Students were invited to live sessions via the Course Announcement page live

session schedule and an invitation emailed to students and parents on Sunday of each week.

Students selected from the session schedule the day(s) and time they would attend. Each session

included a five question post-quiz which attending students were required to complete. Prior to

exiting the final session, students were required to complete a student survey (Appendix C).

After completing all three sessions, students completed a one-on-one discussion of unit content

with their instruction via the telephone. After the discussion based assessment, students were

assigned the unit exam. Students who participated in all three sessions and completed the unit

exam were included in the program evaluation.

Evaluation Method

Participants

Primary data for the evaluation came from MJUHS active students who had taken the unit six

pretest, completed all three live, direct instruction sessions, a one-on-one content discussion with

their instructor, and completed their unit exam between June 1, 2014 and July 11, 2014. A total

of ninety-three students completed all components of the program during the evaluation period.

No sample was purposely used. Due to the self-paced nature of the course, only data gathered for

those students who completed all components of the program was considered.

Additional data was compiled from student and instructor surveys. All ten instructors for the

MJUSH course participated in the evaluation. Instructors completed pre and post surveys using a

Likert Scale (Appendices A and D, respectively). Three of the ten instructors delivered the live

instructional session during the evaluation period (Appendix B). Students completed surveys at

the end of the third synchronous session (Appendix D).

Variables include but may not be limited to varying student learning levels (i.e. gifted, gen-ed,

ESE), time spent on course content outside of program, time lapse between session completion,

and different instructor instructional delivery techniques.

P a g e 3 | 15

Procedures

Surveys were given to all ten instructors prior to beginning the program. Instructors responded to

questions regarding the strength of the curriculum resources and the need for supplemental

resources to meet the program objectives. Instructors also reported in the survey the previous

school year (July 1, 2013 - May 31, 2014) data for their classroom mean unit exam scores

(Appendix A).

Students who entered the unit during the evaluation period completed a pretest before

completing unit assignments and participating in the program sessions. The three live

instructional sessions were offered two times each week during the six week evaluation period.

Students were invited via email and a posted calendar of sessions on their course announcement

page. Students joined the session that best fit their schedule. Each live session included five post

session comprehension questions. The three instructors responsible for delivering the sessions

reported after each session via a Google Spreadsheet the mean of each sessions post quiz scores

(Appendix B).

At the end of the third live session, students completed a Likert scale survey to determine, from

the student perspective, the effectiveness of the live session components and the time lapse

between live session attendances (Appendix C). Following student completion of the pretest, unit

assignments, and all three direct instruction sessions, a one-on-one discussion took place

between each student and their instructor of record. Discussion were focused on the unit content

and instructors used the curriculum grading rubric to determine the student level of content

knowledge comprehension.

A post-program Google form survey using the Likert scale was completed by instructors in

which they reported to the evaluator their classroom mean unit pretest score, discussion-based-

assessment score, and unit exam score. Averages for only those students who completed all

components of the program between June 1, 2014 and July 11, 2014 were reported (Appendix

D). The survey included instructor attitudes regarding the effectiveness of the program.

Data Sources

Both qualitative and quantitative data were used in evaluating the effectiveness of the program.

A Google form survey, using the Likert scale, was sent through email to the ten participating

team instructors prior to program implementation (Appendix A). The surveys collected instructor

feedback on strengths of the curriculum resources and the need for supplemental resources to

reach program objectives. Quantitative data, in the form of the previous school year’s (July 1,

2013 – May 2014) mean unit exam scores, were also reported on the survey. Three instructors

assigned to delivering the synchronous sessions provided mean post-quiz scores for each session

in a Google spreadsheet (Appendix B). Students who completed all three of the programs

synchronous sessions completed a Google form survey (Appendix C) at the end of session three

and before exiting the sessions. The student surveys collected qualitative data regarding the

student perspective on program session effectiveness. The post-program survey completed by

P a g e 4 | 15

instructors provided to the evaluator their classroom quantitative data on pretest, discussion

based assessment, and unit exam mean scores (Appendix D).

Results

The program evaluation results are summarized and related to the program objectives. Survey

data compiled from students and their attitude toward the effectiveness of the program

synchronous sessions reflect positive results. All students who completed the program’s three

synchronous sessions completed an attitude survey.

Over half of the participants reported knowing very little about the unit content prior to

beginning the program unit with 76% knowing little to nothing about the unit.

Effectiveness of the program from the student standpoint indicates success with over 90 % of

participating students finding the sessions very beneficial to their understanding of the unit

content. Details of the student survey are elaborated in Appendix C.

P a g e 5 | 15

Similarly, 80% of the instructors

reported the program sessions were

beneficial to their students

understanding of the lesson content

and a mean discussion-based

assessment score of 88.6%. This is

an improvement from the pre-

program mean score of 84.2%. This

data indicates the first objective was

met from both the instructor and

student standpoint.

In evaluating the quantitative results collected, both the discussion-based-assessment scores and

the unit exam scores increased. The seven percentage point increase in unit exam scores suggests

objective number two, to “increase student unit exam scores by five percentage points,” has been

met. Appendices A and D illustrate additional data regarding quantitative data.

Discussion

The results of this study are used to address the program objectives of increasing learner

understanding of unit content and increasing student achievement scores on unit exams by five

percentage points. Post survey results and instructor reported discussion-based-assessment scores

were used to evaluate the level of program effectiveness for the former. Student unit exam scores

reported during the evaluation period were used to evaluate the program effectiveness for the

latter.

P a g e 6 | 15

Qualitative data collected from students and instructors displays a correlation between student

understanding of unit content and the program synchronous sessions. Quantitative results show

an overall increase of five percentage points for discussion-based-assessment scores with sixty

percent of instructors having improved scores, twenty percent having no change, and twenty

percent having decreased scores. Variables include a different sample group, subjectivity of

instructor when grading assessment with the rubric, and the time lapse between completion of

session and the discussion-based-assessment. Therefore, it can be suggested that program

sessions increased student understanding of the unit content but cannot absolutely attribute the

increase to the sessions entirely.

Data gathered on pre and post exam scores were compared to determine the programs impact on

the second objective – to increase student unit exam scores by five percentage points. Based on

the difference between pre-program unit exam scores and post-program unit exam scores, the

second objective was achieved. Records indicate a seven percentage increase which exceeds the

goal of a five percentage point increase. However, the data employed in the evaluation came

from different sample populations between pre and post program scores. It must be considered

that the seven percentage point increase in unit exam scores was possibly related to variables

outside of the program sessions. Variables to consider but not limited to include mixed student

learning levels (i.e. gifted, gen-ed, ESE), mixed learning styles, time spent on course content

outside of program, time lapse between session completion, and different instructional delivery

techniques among instructors.

Project Costs

P a g e 7 | 15

The project cost for the program evaluation on the effectiveness of virtual direct instruction of

Unit 6, MJUSH for Young Virtual School is outlined above. One principal evaluator was used at

the request of the Young Virtual School Liberal Arts Instructional Leader, Esther Webb. The

evaluator twice met with principal stakeholders at Young Virtual School’s administrative offices,

collected, analyzed, and compiled data to produce a formal written report. All documents were

delivered electronically to Ester Webb.

P a g e 8 | 15

Appendix A

How would you rate the curriculum resources for Unit 6 in helping your students comprehend the content?

1 0 0%

2 0 0%

3 1 10%

4 5 50%

5 4 40%

How would you rate the need for supplemental resources for Unit 6 in helping your students comprehend the content?

1 7 70%

2 3 30%

3 0 0%

4 0 0%

5 0 0%

P a g e 9 | 15

What was your classroom Discussion Based Assessment average score between July 1, 2013 and May 31, 2014?

92 90 82 80 86 84 76

What letter grade was your classroom average on the Unit 6 Discussion Based Assessment between July 1, 2013 and May 31, 2014?

A+ 0 0%

A 0 0%

A- 3 30%

B+ 1 10%

B 1 10%

B- 4 40%

C+ 1 10%

C 0 0%

C- 0 0%

D+ 0 0%

D 0 0%

D- 0 0%

F(50% or below) 0 0%

What was your classroom unit 6 exam score between July 1, 2013 and May 31, 2014?

71.2 71.25 67 69 71 73 62 74.1 74.6 50

P a g e 10 | 15

What letter grade was your classroom average on the Unit 6 Exam between July 1, 2013 and May 31, 2014?

A+ 0 0%

A 0 0%

A- 0 0%

B+ 0 0%

B 0 0%

B- 0 0%

C+ 0 0%

C 3 30%

C- 3 30%

D+ 2 20%

D 0 0%

D- 1 10%

F(50% or below) 1 10%

What suggestions or comments do you have regarding student comprehension Unit 6?

Students don't seem to take the time needed to understand all the content in this unit. They are rushing through and not too interested in Jacksonian Democracy. I just don't think the students care enough about doing well on this. They know they can keep their grade up if they do better on the next unit. There's so much information in the unit that they lose focus. Breaking the unit down into smaller parts might help. Give them handouts and video help. They don't take the time to do the work themselves. If we want them to be successful in the time they're willing to spend in the unit then we have to meet them halfway. Live lessons with some background information and then some follow up questions worded like the exam questions. Students have a hard time understanding what the questions are asking even when they know the content. Help needed with test taking strategies, and reading comprehension. Get some tips from reading coach and add strategies to some live instruction. Students don't come with the background knowledge needed to understand the unit content well enough. Perhaps creating some

P a g e 11 | 15

supplemental lesson handouts, video instructions, and/or doing some live lessons would help. These kids just don't have the background knowledge to be successful in this unit. There are a lot of different connections to make in this unit and they can't figure it out. We have to help them by guiding them through it one step at a time. Some kids do really well but, for most, this unit is just too much. Students struggle with the amount of complex text in the unit and on the exam. Possibly provide some help to students with understanding the text and unit content. We should offer some help with understanding the unit content. There's a lot of challenging text and students have trouble with it. Maybe live sessions and/or teacher-created video instruction. I think that giving students some one on one instruction or maybe a handout with a lower reading complexity level would help them understand the content a lot better. They struggle with the reading and miss the content.

Appendix B

Week 1 Week2 Week 3 Week 4

Average by session

(Liverotti)602 91.1 84.3 85.5 94 89.1 83.6 91.1 89.1 88.475

(Perez)604 88 85.5 82.2 90 84.4 90 91.1 86.7 87.2375

(Gaulier)606 82 84.4 90 92 91.1 86 90 87.2 87.8375

Average by week 87.03333 84.73333 85.9 92 88.2 86.53333 90.73333 87.66667

MJUSH Unit 6 Program Live Session Quiz Averages

P a g e 12 | 15

Appendix C

P a g e 13 | 15

Appendix D

P a g e 14 | 15

Student Note: Due to the featured online school’s regulations, publishing of exam questions

in any form is prohibited. For this reason, the evaluation includes only compiled data and

not the exam itself.