ed tech 505 dsis math tutorial evaluation
DESCRIPTION
Final evaluation project for EdTech 505 at Boise State UniversityTRANSCRIPT
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
1/17
DSIS Math Tutorial Evaluation
Submitted to: Davis School for Independent StudyCompleted by: Christine Simms
May 9, 2014
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
2/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 1 of 16
SummaryThe Math Tutorial Program at Davis School for Independent Study (DSIS) is an intervention thatwas put in place as a result of declining test scores and low course grades. It supplements thedirect instruction that occurs in weekly math classes to support improved academic achievement.It offers four hours a week of tutorial time staffed by the sites credentialed math teacher. The
tutorial hours give students opportunities to get help with their homework and clarify contentunderstanding through one-on-one interaction with the teacher. Student attendance is voluntaryuntil a students grade falls below a C-; at that time, the student is mandated to attend at least onehour of tutorial a week.
This formative evaluation was completed during the fourth year of the program to determine theextent to which it is meeting its established objectives. It presents an analysis of data gatheredfrom interviews with administrators and staff, standardized test scores, course grades, the 2011WASC Report, School Site Plans, program observations and a student survey as they relate tothe achievement of program goals to support students and increase academic achievement.
The evaluation found that the Math Tutorial Program is at least a contributing factor to student
success, but that academic achievement in mathematics remains a critical area of concern as datadoes not reflect the desired student outcomes. CST scores have not risen by the 5% a year goalsoutlined in the School Site Plans and many students are struggling to earn passing grades in theirmath courses. The report identifies areas where historic data is missing and suggestsimplementing tracking protocols that would support future evaluations.
Description of the program evaluatedDSIS is a K-12 site located in a small city flanked by a major university and surrounded byfarmland. Supervising teachers meet weekly with their students individually. Secondary mathcourses meet three times weekly (except Junior High Math which meets twice) and are taught bya credentialed math teacher. Because of the more limited contact time, math is a particularly
challenging subject to teach in an independent study setting.
The Math Tutorial Program was instituted in the 2010-2011 school year as one of the changes tothe structure of the math program after the schools Self-Study for their Western Association ofSchools and Colleges (WASC) accreditation renewal identified declining California StandardsTest (CST) scores in mathematics and low grades in math courses as critical deficits. Four hoursweekly FTE, divided across all five school days and distributed between morning and afternoonhours in order to accommodate the diversity of student schedules, is allocated to staffing thetutorial hours with the schools credentialed math teacher to better support student success in
mastering the course content delivered during math classes. (See Appendix A) This interventionprovides a less structured learning environment where students can get individualized help with
homework assignments, make up missed tests, and ask questions that support their developingunderstanding of concepts and processes.
The program serves all eight levels of secondary math courses taught at the site: Junior HighMath, Transition to High School Math, Pre-Algebra, Algebra 1, Geometry, Algebra 2,Trigonometry, and Business Math. It is open to all DSIS students on a drop-in basis and meets inthe school library. Once-weekly attendance is mandatory for students with Ds or Fs. The math
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
3/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 2 of 16
teacher circulates among the students, checking in to see how things are going and responds toindividual questions as they arise.
Program ObjectivesThe Math Tutorial Program is in place to improve student achievement in mathematics at DSIS.Based on the programs goals, the following program objectives were established.
Students with grades of D+ or below will receive the individual support they need toincrease their overall course grades to passing by attending a minimum of one tutorial perweek until their grades improve.
Students will have the individual help they need outside of class to have their questionsanswered and content clarified to support content mastery as measured by increasingoverall course grades.
Students will receive the individual attention needed outside of class to have questionsanswered and content clarified to improve content mastery to proficient or above asmeasured by the CST.
Students will develop the content mastery necessary to increase confidence andengagement to prevent them from dropping math courses without completing them.
Students will receive the individualized attention they need to increase content masteryand confidence necessary to support them in continuing to progress on the math pathwaythrough high school graduation.
Program ComponentsThe foundation for Math Tutorial Program is four hours of dedicated FTE for the credentialedmath teacher to provide individualized support in a casual group setting, on a drop-in basis, forstudents taking all levels of math at DSIS. Students typically bring their own math books andgraphing calculators, but they are available to borrow as needed.
Evaluation MethodIn order to determine the extent to which Math Tutorial is meeting its established objectives, agoal-based model was used to design this evaluation.
ParticipantsGiven the relatively small size of the school, all math students in grades 7-12 were surveyedduring their math classes and students attending tutorial were observed as a part of thisevaluation. Additionally, interviews were conducted with the DSIS math teacher and two formerprincipals who had been directly involved in establishing the program.
Procedures
The evaluator began by meeting with staff and administration to determine program history,goals, and objectives, and identify existing data sources to be reviewed for the evaluation.Existing data was gathered for review of program history and student achievement since the yearprior to the institution of the program to establish a baseline and examine progress towardachieving program objectives.
A student survey was designed and administered as a follow up to the student survey on programparticipation completed in the spring of 2013 as well as collecting additional data related tostudent attitudes and future plans for continuing on the math pathway. Observations of Math
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
4/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 3 of 16
Tutorial were conducted to see the program operating under normal, everyday conditions. It wasconfirmed with the math teacher that the observations during the evaluation period were typicalof normal use.
Finally, survey and academic achievement data were converted to excel for analysis and the finalevaluation report was produced for the school.
Data SourcesData was compiled from numerous sources to develop a comprehensive overview of theevolution of the program from its inception to current practice. The evaluator conductedinterviews with the DSIS 2010-current math teacher, Jamie Wales, 2012 Math Committee Chairand 2012-2013 principal, Laura Juanitas, and 2008-2012 principal, Kim Wallace. She reviewedthe 2011 WASC report, 2012 Math Committee meeting notes, School Site Plans for 2011-12,2012-13, and 2013-14, and math department syllabi.
Student achievement data was gathered from the 2011 WASC Accreditation Self-Study Report,School Site Plans, 2013-14 class marks recorded in Q, and 2013 California Standards Test (CST)
scores reported on Illuminate. Students also self-reported average grades on the student survey.
Data related to student understanding of and program use was gathered using a student survey(See Appendix B), evaluator observations, and previously utilized attendance trackingmechanisms/records. (See Appendix C) Existing data from a survey administered in the spring of2013 was also reviewed.
Student attitudes about math and future plans to continue on the math pathway were alsogathered using the student survey.
ResultsAcademic Achievement
Prior to changes made to the structure of math delivery at DSIS and the institution of MathTutorial, student achievement data was indicating a steady decline in the percentage of studentswho were performing at levels of proficient or above on the CST. (Table 1)
Though the state goal is for all learners tobe proficient or above, the 2010 resultsreported that 43.38% of DSIS studentswere in the proficient/advanced range.This prompted changes to the frequency
of math classes and the inception of theMath Tutorial Program. Moving forwardfrom this point, there was a noticeableupturn in CST score data.
Table 1
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
5/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 4 of 16
Scores from the spring of 2011 indicated that the percentage of students scoringproficient or above had risen to 47% andthose from 2012 were fairly consistent withthat at 46%. There was a significant declinein the 2013 results, with only 30% of students
achieving proficient or advanced. (Table 2)
Course grade data available was limited tothe current year, 2013-14. At the end of thefirst semester, 49% of all math course gradeswere in the A/B range, 33% were Cs, and18% were Ds or Fs. Progress marks for thesecond semester are recorded at the end ofthe third quarter. At that time, almost thesame percentage (48%) of students wereperforming with sufficient mastery to earnAs or Bs, 14% of students were in the C
range, and 38% of student marks were eitherDs or Fs. (Table 3)
Survey DataThe student survey provided data related to participation in theMath Tutorial Program as well as information about studentattitudes about mathematics and their long-term math goals.
Of the students who completed surveys in 2014, 56% reportedattending Math Tutorial at least once during the current schoolyear. (Table 4) 14 of 52 students said that there are times whenattendance at Math Tutorial is mandatory. Of those, only 6indicated that the condition that would mandate attendance ispoor grades. None of the students who identified failing grades
2009 42.00%
2010 43.38%
2011 47.00%
2012 46.00%
2013 30.00%
0%
10%
20%
30%
40%
50%
60%
70%
80%90%
100%
CST Mathematics 2009-2013
Percent of Students Scoring
Proficient or Advanced
Table 2
A/B
49%C
33%
D/F
18%
2013-14 Semester 1
Math Grades for All Courses
A/B C D/F
Table 3
A/B
48%
C14%
D/F
38%
2013-14 Quarter 3
Math Grades for All Courses
A/B C D/F
Yes
56%
No
44%
2013-14 Student Use
of Math Tutorial
Table 4
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
6/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 5 of 16
as triggering anattendancerequirementreported averaginggrades lower than
Cs.The top threereasons studentsreported choosing toattend Math Tutorialwere that theyneeded help with
homework, they didnt understand something in class, or a parent required them to come. (Table5) The three most frequently stated reasons for not attending were that they didnt need help, hadschedule conflicts, or that they get help elsewhere. (Table 6)
37% of studentssurveyed reportedhaving sourcesother than MathTutorial to supporttheir success inmath classes: 22 gethelp from parents, 8have UCD tutors, 7have other tutors, 7get help from
friends, and 4 have other sources of support. 76% of students reported that they always orusually understand content the content clearly enough to be successful by coming to class andcompleting homework/studying on their own.
A similar number of students weresurveyed in 2013. At that time, 63% ofresponses indicated having attendedtutorial at least once during the schoolyear. Of the students who were notelecting to attend, 58% cited scheduleconflicts and 21% reported that theyget help elsewhere.
The 2014 student survey alsogenerated baseline data related tostudent attitudes about math,themselves as math students, and their
plans for taking future math courses. (Table 7) The majority of students surveyed reportedfeeling pretty good at math and that they plan to continue taking math courses until high schoolgraduation. Students not only care about their math grades, but want to do even better.
1
0
4
65
13
20
3
11
0 5 10 15 20 25
Other
Help with Graphing
Like the Routine
Helps Me Feel ConfidentLike One-on-One Support
Didn't Understand
Needed Help with
Ms. Wales Required
Parent Required
Why Students Report They Attend Tutorial(could choose multiple responses)
Table 5
6
6
2
2
1
3
1
12
4
4
2
4
3
1
17
18
3
10
14
10
8
13
20
10
11
8
17
10
4
4
33
27
25
19
32
ENJOY MATH
GOOD AT MATH
CONTINUE MATH TO GRAD
CARE ABOUT GRADES
MATH IS IMPORTANT
TRY MY HARDEST AT MATH
WANT TO DO BETTER
Student Attitudes on a Scale of 1-5
1="no way" 5="definitely"
1 2 3 4 5
Table 7
3
9
3
9
24
19
23
0 5 10 15 20 25
Other
Might Help, but Not Motivated to Go
Don't Think it will Help
Satisfied with my Grade
Don't Need Help
Get Help Elsewhere
Schedule Conflict
Why Students Report They Do Not Attend Tutorial(could choose multiple responses)
Table 6
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
7/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 6 of 16
Attendance DataTwo attendance tracking mechanisms were included in the review of existing data. In the 2010-2011 year, supervising teachers received an email listing all students who should be attendingtutorial (or working with a UCD tutor) due to low math grades. Students and their parents hadalready been notified; the idea was that supervising teachers could remind their students when
they saw them at their individual meetings in order to improve student follow-through. In the2012-2013 school year, the math teacher tracked attendance each week in a binder, writing downthe names of students present at each tutorial session and also collecting data related to theirdegree of engagement during the hour.
Policy Related DataSyllabi received by parents and students at the beginning of the course outline the tutorial policyas being optional for students with grades of C- or better, but attendance is mandatory at leastonce a week for those with current grades of D+ or below. The penalty for not attending when itis required is a warning letter. (See Appendix D)
Both administrator interviews indicated that attendance is mandatory for students with Ds or
Fs. Interviews with the math teacher offered the same data, but revealed that there is currentlyno tracking mechanism in place because the things she has tried have been cumbersome yetunsuccessful. Responses on the student survey revealed that while 27% of students believe thattutorial attendance is sometimes mandatory, only 12% are aware of low academic performancebeing the condition that triggers mandatory attendance.
DiscussionThe purpose of this evaluation was to determine the extent to which the Math Tutorial Programis meeting its established objectives. After several interviews with stakeholders and a thoroughexamination of all available data, it seems evident that the Math Tutorial Program is acontributing factor in the overall success of DSIS math students. However, given that at least two
major adjustments were made at the same time, an increase in the number of hours each mathclass meets weekly and the institution of Math Tutorial, along with incomplete historic data forcomparison, it is impossible to determine the precise impact of the program.
In looking at the CST scores, there is a significant increase (+3.62%) in the percentage ofstudents scoring proficient or above in the first test cycle (2010-2011) after the tutorial programwas established. There is a drop of 1% the following year, but given the small size of the school,this is insignificant and both the 2010-11 and 2011-12 school years present as consistent. The2012-13 scores offer a different picture, with only 30% of students scoring in theproficient/advanced range.
A number of possibilities that have nothing to do with Math Tutorial, or the math program atDSIS, may account for the sudden dip in scores. Scores are reported based on students currentlyenrolled at DSIS rather than on students who took the test at DSIS last year. Because the schoolstarted the year with fairly low enrollment, many of the current students were not at the site lastyear, thus their CST scores do not reflect on the teaching or interventions available at DSIS.Additionally, even students who take the CST at DSIS may not be receiving their instruction atthe site. Split site students may take math at a comprehensive site, but because they areconsidered DSIS students, they are tested at and scores are attributed to DSIS. Another
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
8/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 7 of 16
possibility is a changing student population at the site. These scores should be further analyzed todetermine whether the decrease in scores is related to an increase in the population of theSocioeconomically Disadvantaged (SES) student subgroup. Historic data indicate that this grouphas performed significantly lower than the general DSIS population. If a change in population isresponsible for the dip in scores, it may be necessary to reconsider the best interventions for
meeting student needs.
Undoubtedly, academic achievement in math continues to be an area of concern at DSIS. CSTscores have failed to reach the 5% annual gains established in the school site plans and manystudents are receiving Ds or Fs in their math courses. Math Tutorial exists as an intervention tosupport student success, but is not being fully utilized to reach its potential. This may be due to agap in student perception. 10% of students surveyed self-reported D/F grades three weeks afterQ3 Progress marks indicated 38% of students were earning Ds or Fs.
Student survey data also indicated a lack of student understanding regarding the Math Tutorialattendance policy related to course grades. Though clearly stated in each course syllabus, only12% of students reported being aware of D/F grades making tutorial attendance mandatory.
Student confusion may be attributed to a failure to attend to that detail in the syllabus or may berelated to the disparity between the stated policy and current practice which does not trackattendance or penalize for lack thereof. The math teacher reported in her interview that she feelsthis is the biggest issue related to Math Tutorial. She has tried different methods, but has foundthem to be unsuccessful. She would like support in identifying a system that would work for bothher and the students, 44% of whom cite schedule conflicts as preventing them from attendingtutorial hours.
The 56% of students were have attended tutorial this year were well-served by the program. Farmore (60%) responses indicated reasons for attending related to seeking one-on-one support thanindicated attendance because a parent or the teacher required it (22%). This demonstrates a
conscious recognition of the value students place on having individual attention from the teacheroutside of class time.
67% of students report that they get help with their math from sources other than Math Tutorial.It may be appropriate to more closely examine the correlation between individual studentachievement and what sources they are using for support. Given the high percentage of studentswith low course grades and the number of students stating schedule conflicts as their reasons fornot utilizing the tutorial hours, it may be possible to provide more consistent, reliable support iftutorial hours can be configured differently or delivered in a different format, perhapssynchronously online.
Several program objectives relate to student attitudes about math as they affect decisions to drop
courses without completing them or long-term plans to continue progressing along the mathpathway beyond requirements for graduation. As there was no historic data in this area, it is notpossible to determine how the individual attention students receive as a result of Math Tutorialhas influenced changes in attitudes or plans. Additionally, there are no mechanisms in place totrack either course retention or how many/when/why students elect to stop taking courses inmathematics. If the objectives are to remain the same, it will be necessary to determine a plan forgathering the necessary data. The data collected for this evaluation may serve as a baseline in thearea of student attitudes for future evaluations.
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
9/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 8 of 16
Table 8Objective Identified Strengths Identified Concerns
Students with grades of D+or below will receive theindividual support they needto increase their overallcourse grades to passing byattending a minimum of onetutorial per week until theirgrades improve.
Support in place to servestudent need
Attendance policy for D/Fgrades
Support underutilized by this targetaudience
No effective mechanism in place fortracking or enforcing attendancerequirement
Communication of policy to studentsnot clear
Students will have theindividual help they needoutside of class to have theirquestions answered andcontent clarified to supportcontent mastery as measuredby increasing overall coursegrades.
Students who attend aregetting individualizedsupport to clarify content
No tracking mechanism in place tocompare course grades historically
Students will receive theindividual attention neededoutside of class to havequestions answered andcontent clarified to improvecontent mastery to proficientor above as measured by theCST.
Support mechanism is inplace
Though growth was indicated, it neverreached 5% annually and declinedconsiderably in 2013
Further evaluation of the CST data isnecessary to address possible reasonsfor the decline
Students will develop thecontent mastery necessary toincrease confidence andengagement to prevent themfrom dropping math courseswithout completing them.
Most students have generallypositive attitudes about math
Some students who attendtutorial report that they do sobecause it increases theirconfidence
No historic data is available todetermine the role of programparticipation in student attitudes. Thisevaluation collected baseline data onstudent attitudes that can be used goingforward.
There is no readily available datarelated to course completion or thereasons students have dropped mathcourses
Students will receive theindividualized attention they
need to increase contentmastery and confidencenecessary to support them incontinuing to progress on themath pathway through highschool graduation.
Students report feeingsupported by the individual
attention they receive attutorial
Most students have generallypositive attitudes about math
Most students are planningto continue taking mathcourses until graduation
No historic data has been kept related toDSIS math students continuing to take
math until graduation. In order tofollow up on the achievement of thisobjective, it would be necessary toinstitute a mechanism for tracking thisinformation.
This evaluation collected baseline datarelated to student confidence that can beused for comparison in the future.
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
10/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 9 of 16
The strength of the Math Tutorial Program is clearly in its ability to provide students with theindividual attention they need to address their own gaps in understanding. Because of thestructure of math instruction at DSIS, with the teacher having only two or three hours to deliverall of the content students will need to understand for the week, classes tend to be packed withdirect teaching and offer very few opportunities for student questions. Math Tutorial definitely
fills the gap for students by offering personal support and encouragement on an as-needed basis.
The weakness seems to emerge from the inability or lack of desire for some students to recognizetheir need for additional support, or from the schedule conflicts that prevent them from accessingthe resource. For this reason, it is necessary to establish a means of enforcing the attendancerequirement for those students who need help but are not coming on their own. It should bemanageable for both students and teacher. Rather than constant reevaluation, perhaps just use thefirst of each month as the cut-off date to mandate weekly attendance until a reevaluation thefollowing month. This would put all courses on the same cycle. A tracking sheet could be createdas a simple check-off that is marked when the student attends once during the week, no need totrack attendance for those who are not mandated to attend or for multiple days for those who are.
In looking at each of the objectives established for the Math Tutorial Program, this evaluationhas identified both strengths and concerns that can be used to begin conversations between staffand administration to chart a path for program improvement. (Table 8) It will be necessary torevise program objectives in light of the move to Common Core Standards and the transitionfrom the CST to the Smarter Balanced Assessment (SBAC). Until baseline scores are availableafter the 2014-2015 school year, achievement goals should be tied to course grades.
Project CostPrincipal Evaluator, Christine Simms, was on the project for a total of ten days of billable hours.She spent two days on site at the school and district office conducting interviews with staff andadministration, conducting observations of the program, and gathering existing data. She created
a student survey to be administered by the math teacher, then compiled the results for analysis.The balance of time was spent reviewing and analyzing data and producing the final evaluationreport.
Personnel Budget
Christine Simms [Principal Evaluator]: 10 days @ $500/day $5000
Total Personnel $5000
Miscellaneous
Supplies [paper, ink, copying, etc.]: $100
Total Miscellaneous $100
Total Budget $5100
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
11/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 10 of 16
Appendices
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
12/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 11 of 16
Appendix A
2013-2014 Secondary Class Schedule with Math Tutorial Indicated
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
13/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 12 of 16
Appendix B
MATH STUDENT SURVEY (please do not include your name on this paper)Math course ___________________________ My test grades are typically: [A/B] [ C ] [D/F]
Is attendance at Math Tutorial ever mandatory? [Yes] [No] If yes, when? _________________Have you attended Math Tutorial during the current school year? [Yes] [No]
If yes, what did you find valuable? (select all that apply)[Having a scheduled time to do my math] [Working with others from my class][Getting immediate feedback to my questions] [Other:]_____________________________
How frequently do you attend Math Tutorial in general? (select one)[2+ times/week] [1x/week] [every other week] [1x/month] [
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
14/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 13 of 16
Appendix C
2010-2011 Mandatory Attendance Tracking
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
15/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 14 of 16
2012-2013 Attendance Tracking Document
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
16/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 15 of 16
Appendix D
Algebra 1 Syllabus/Course Policies 2013-2014
-
5/19/2018 Ed Tech 505 DSIS Math Tutorial Evaluation
17/17
DSIS Math Tutorial Evaluation|Submitted to: DavisSchool for Independent Study
Page 16 of 16