de anza math pilot results, 7-9-07 1 evaluating a nonrandomized trial: a case study of a pilot to...
TRANSCRIPT
De Anza Math Pilot Results, 7-9-07 1
Evaluating a NonRandomized Trial: A Case Study of a Pilot to Increase Pre-Collegiate Math Course Success Rates
Andrew LaManque, PhDSupervisor, Institutional ResearchDe Anza College
De Anza Math Pilot Results, 7-9-07 2
• Overview of the Pilot• Purpose of the Evaluation• Results from Quantitative Analysis
• Demographics of Students• End of Class Survey• Course Success
• Findings
Outline
De Anza Math Pilot Results, 7-9-07 3
Overview of the Pilot
• A Partnership was formed to increase the success rates in Pre-Collegiate math courses
• Components:• Math and Counseling faculty• EnableMath – Course Software• Noel Levitz – College Student Inventory
• Using technology, provide students timely information on their progress as well as information about their motivations
De Anza Math Pilot Results, 7-9-07 4
Overview of the Pilot
• Developmental math sequence consists of pre, elementary, and intermediate algebra
• Offer 175 class sections per year of developmental math (37% of all math department courses)
• Historically, a 55% success rate in developmental math courses
• 31 full-time instructors, 45 part-time instructors
• Class size is 40
De Anza Math Pilot Results, 7-9-07 5
Program Evaluation
•Given that the pilot program provides an additional learning choice for students, the question from the Dean is:
“do the results demonstrate that pilot students do at least as well as other students”
•Using multiple forms of evaluation (e.g. student grades, student surveys) provides feedback to instructors as well as administration on implementation
De Anza Math Pilot Results, 7-9-07 6
Limitations
•Not a randomized trial – possible instructor (although, 12 different instructors participated) and student self selection bias
•No independent measure of success
•More research is needed to determine the relative effect of each intervention
Program Evaluation
De Anza Math Pilot Results, 7-9-07 7
Results: Demographics of Students
How would you describe your Racial / Ethnic Origin?
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
AfricanAmerican
Asian orPacificIslander
Latino White Prefer not torespond /
Other
De Anza College CSI Respondents Fall 2005-Spring 2007. N = 1,114.
Twice the % of Latino students as college, similar to non-pilot sections
De Anza Math Pilot Results, 7-9-07 8
Results: Demographics of Students
What is the highest level of education completed by your father?
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
high school or less 1-3 years of college 4-year degree or more
De Anza College CSI Respondents Fall 2005-Spring 2007. N = 1,028.
Many first-generation college students
De Anza Math Pilot Results, 7-9-07 9
•Similar with regard to Gender, Initial Education Goal and Ethnicity
•although Math114 EnableMath had a higher percentage of minority students (54%) compared with the other sections (34%).
•EnableMath students are slightly older on average (45% vs 37% born prior to 1987).
Results: Demographics of Students
De Anza Math Pilot Results, 7-9-07 10
Results: End of Class Survey
Time Spent Studying Math at Home Compared to Time Necessary to Get Desired Grade
72%
24%
4%
12%
45% 43%
0%
20%
40%
60%
80%
100%
0-4 hrs/wk outside class 5-9 hrs/wk outside class 10 plus hrs/wk outsideclass
De Anza College Winter 2006-Spring 2007 Term EnableMath Survey. N = 992 .
Time at Home Doing Math Time Necessary to Get Desired Grade
38% of respondents indicated that they needed to study more to get the grade they wanted, than they actually studied
De Anza Math Pilot Results, 7-9-07 11
Results: End of Class Survey
EnableMath helped me learn more math than I would have without the software
0%
20%
40%
60%
80%
100%
Strongly Agree Agree Disagree Strongly Disagree
De Anza College Winter 2006-Spring 2007 Term EnableMath Survey. N= 986.
77% Agree or Strongly
Agree
De Anza Math Pilot Results, 7-9-07 12
Results: End of Class Survey
Completing the CSI was worth the time
0%
20%
40%
60%
80%
100%
Strongly Agree Agree Disagree Strongly Disagree
De Anza College Winter 2006-Spring 2007 Term EnableMath Survey. N=959.
61% Agree or Strongly
Agree
De Anza Math Pilot Results, 7-9-07 13
Results: Course Success
Approx 1 of 5 students
Course Group # Sec HC *
Pre-Algebra Pilot 14 428Other 55 1,755
Elem Algebra Pilot 21 723Other 88 3,045
Interm Algebra Pilot 27 853Other 112 3,573
Total Pilot 62 2,004Other 255 8,373
* HC for the Pilot only includes studentscompleting the CSI.
De Anza College Math PilotSections / Students, Fall 05 - Spr 07
De Anza Math Pilot Results, 7-9-07 14
Results: First Course Success
Course Group Grades Percent Grades Percent Grades Percent Grades PercentPre Algebra
Pilot 308 72% 76 18% 44 10% 428 100%Other 1,012 58% 391 22% 352 20% 1,755 100%
Elementary AlgebraPilot 514 71% 138 19% 71 10% 723 100%Other 1,660 55% 664 22% 721 24% 3,045 100%
Intermediate AlgebraPilot 641 75% 140 16% 72 8% 853 100%Other 1,977 55% 675 19% 921 26% 3,573 100%
Includes end of term grades of students enrolled at census (3rd week). Note: Students in the Pilot group used the EnableMath Software and completed the CSI.
Did Not Pass Withdrew TotalPass
Course Success of Fall 2005-Spring 2007 EnableMath / Noel Levitz Sections Compared to All Other Sections, De Anza College
De Anza Math Pilot Results, 7-9-07 15
Results: Second Course Success
First CourseCourse Group Grades % Row Grades % Row Grades % Row Grades % Row
Pre to Elementary AlgebraPilot 94 45% 59 28% 57 27% 210 100%Other 305 55% 106 19% 148 26% 559 100%
Elementary to Intermediate AlgebraPilot 168 64% 45 17% 49 19% 262 100%Other 545 66% 137 17% 146 18% 828 100%
Intermediate Algebra to CollegePilot 85 50% 36 21% 48 28% 169 100%Other 301 56% 97 18% 142 26% 540 100%
Note: Considers only the first attempt of both courses.
Pass Did Not Pass W Total
De Anza College EnableMath Pilot, Fall 2005-Winter2007Second Course Pass Rate for Students Successful in First Course
De Anza Math Pilot Results, 7-9-07 16
Results: Second Course Success
First Course Second CourseCourse Group Group Grades % Row Grades % Row Grades % Row Grades % Row
Pre to Elementary AlgebraPilot Pilot 60 52% 36 31% 20 17% 116 100%
Control 34 36% 23 24% 37 39% 94 100%
Control Pilot 47 64% 14 19% 12 16% 73 100%Control 258 53% 92 19% 136 28% 486 100%
Elementary to Intermediate AlgebraPilot Pilot 97 80% 17 14% 7 6% 121 100%
Control 71 50% 28 20% 42 30% 141 100%
Control Pilot 95 74% 24 19% 9 7% 128 100%Control 450 64% 113 16% 137 20% 700 100%
Pass Did Not Pass W Total
De Anza College EnableMath Pilot, Fall 2005-Winter2007Second Course Pass Rate for Students Successful in First Course by Group
De Anza Math Pilot Results, 7-9-07 17
Findings
• Students currently in the project want to take next class in sequence
• These classes are filled before other math classes
• Participating instructors & counselors want to continue
• More faculty are wanting to participate
De Anza Math Pilot Results, 7-9-07 18
Findings
• No increased costs – taking existing resources and using them differently
• Regular class period / time
• Counseling time in the class instead of
counseling center
De Anza Math Pilot Results, 7-9-07 19
Findings
• Overall, the data suggests that there is value in offering a different approach to students, but claims of improved learning, especially the long term retention of material, can not be substantiated.
• Difficult to evaluate a non-randomized trial • Math department might focus on improving
the transition in pilot sections from Pre-Algebra to Elementary Algebra
De Anza Math Pilot Results, 7-9-07 20
Questions
For a copy of this presentation
please email
Andrew LaManque at: