ses report 07-08

December, 2009

Supplemental Educational Services in New Mexico: SY 2007-2008 PREPARED BY: DR. STEPHANIE AMEDEO MARQUEZ, STATISTICIAN, NMPED

DR. VERONICA C. GARCÍA CABINET SECRETARY OF EDUCATION DR. CATHERINE CROSS-MAPLE

DEPUTY SECRETARY, LEARNING AND ACCOUNTABILITY DR. TOM DAUPHINEE, INTERIM

INTERIM SUPERVISOR, ASSESSMENT AND ACCOUNTABILITY DIVISION

1

EXECUTIVE SUMMARY

Legislative History. The No Child Left behind Act of 2001 (NCLB) required school districts receiving Title I funds to provide Supplemental Education Services (SES) to parents. Economically disadvantaged children who attended Title I schools identified as “In Need of Improvement” by the state for two years in a row were eligible for SES services in School Year (SY) 2007-2008. The option required districts to set aside funds from school budgets which may reach 20% of the school budgets to provide parents with this opportunity to enroll their children in Supplemental Education Services (SES). Supplemental Education Services may include tutoring, remediation, or other academic instruction offered by a state-approved provider, and must be an addition to the regular daily instruction a student receives in the classroom.

Purpose of the Report. The main goal of this report is to examine whether the SES services had an impact on student achievement as intended by the Federal legislation. A secondary goal was to evaluate the effectiveness of various aspects of service delivery.

KEY FINDINGS

Completion Rates. Only 3,147 of 7,771 SES students in SY 2007-2008 could be evaluated for changes in achievement. The principal reason they were not evaluated was a lack of two years of valid SBA scores. Many SES tutored students were not in tested grades. About 41.2% of the SES tutored students could not be evaluated for academic achievement.

A second issue involves completion of services. The mean completion level was at 13.49 sessions completed, the median was 15, and the mode was zero (meaning more students in this category than any other). There were only 8 students at the highest end of completion, that is, only 8 completed between 51 and 81 sessions. There were 544 students for whom no information was reported to the State. Some 1,390 students had not completed any sessions, after applying for services.

No Statistically Significant Gains in Math or Reading Proficiency. SES participating students, when matched with non-participants on key characteristics (prior scores in reading and math, school, grade, ethnicity, gender, English Language Learner status, special education) experienced no significant gains in reading proficiency on the second administration (post-test) of the NMSBA. Overall, New Mexico students declined in math, and stayed the same in reading proficiency level in both years they participated. This trend was also true of SES students.

Analysis of Service Delivery Indicated Three Problem Areas. Three problem areas of service delivery were substantially agreed upon by both providers and districts: (1) completion rates, (2) increased communication between tutors and teachers, (3) improved achievement.

2

Introduction What is SES? Supplemental Educational Services (SES) were initiated by passage of the No Child Left Behind Act of 2001 (NCLB). NCLB required schools that received Title I funds and had not made Adequate Yearly Progress for three consecutive years to offer parents additional academic support in the form of SES services. The majority of SES services are provided by for-profit companies that contract directly with the districts to supplement the regular academic classroom with after-school one-on-one tutoring.

Research Results Are Inconclusive. Research results from other states for SES are mixed regarding student achievement outcomes (CEP, 2006 and 2007). In June 2007, a report published by the U.S. Department of Education (ED) found that students served by SES scored better on math and reading in the first year and even better in the second and subsequent years. This study was based on data collected from nine large, urban school districts.

Subsequent studies have not shown similar results, instead, most show student achievement outcomes that were modest (Betts, 2006; Burch, 2007). In 2007-2008, Tennessee’s statewide evaluation found no significant differences for SES compared to non-SES students. No providers were significantly better or worse in producing academic achievement in their students than the achievement in a group of comparable control students (Source: 2007-2008 Supplemental Educational Services in Tennessee Student Achievement Report).

An evaluation of Minneapolis Public Schools, for 2006-2007 found no statistical significance in reading achievement between students who participated in SES and those who were eligible but did not participate. This was true across all grade levels (Tan, 2007). The Los Angeles Unified School District (2007) reported statistically significant, yet substantively negligible performance gains in achievement for students in SES tutoring over those who applied for but did not receive services (used as the control group).

One study, conducted by the Chicago Public Schools (CPS), found students who started farthest behind gained most. Students in the most need of academic help (those with the lowest achievement scores) obtained a greater benefit from participating in the SES program.

Participation and Completion Rates. Participation and completion rates may impact findings. A substantial number of SES students fail to start services, or to complete them. During 2005-2006, in Massachusetts it was found that only 10% of SES instruction began before December (Morgan and Bracket, 2007). The Chicago Public Schools (CPS) evaluation found that students who completed the program made more gains on standardized math and reading tests than the eligible students who did not receive SES. New Mexico Public Education Department (2007) reported a similar finding that the more hours of tutoring resulted in greater growth gains in standardized test scores.

Summary of Research Findings. Common results found in many states and school districts included the following: 1. Only a small percentage of eligible families are taking advantage of the free tutoring; 2. Quality of tutoring programs or tutoring companies is varied, and 3. Achievement results are weak or not widely verified. The purpose of this report is to investigate the participation in tutoring, achievement results, and service delivery.

3

Background Eligibility and Receipt of Services. During SY 2007-2008, New Mexico districts reported a total of 80,819 students who were eligible for SES services. Only 6,825 students applied for SES services, and even fewer (5,959) received them. Districts varied in their SES expenditures per student. An expenditure of $6,220,154 for 5,964 students equates to the average amount spent in New Mexico per student for SES tutoring was approximately $1,043 per student.

TABLE 1: ELIGIBILITY, APPLICATIONS, AND RECIPIENTS OF SES SERVICES, BY DISTRICT

DISTRICT

NUMBER OF STUDENTS

ELIGIBLE FOR SES

NUMBER OF STUDENTS

APPLYING FOR SES

NUMBER OF STUDENTS

RECEIVING SES TOTAL EXPENDITURES

ALBUQUERQUE 25,988 2,844 2,844 3,193,695AZTEC 219 34 34 20,624BELEN 2,840 210 137 85,292

BERNALILLO 884 58 24 15,059BLOOMFIELD 375 26 21 19,794

CARLSBAD 526 44 26 27,360CENTRAL CONS. 5,851 367 302 358,687

CUBA 726 27 27 22,823DEMING 2,461 24 24 25,943DULCE 538 12 12 10,597

ESPANOLA 2,002 332 299 218,187FARMINGTON 1,174 25 24 28,703

GADSDEN 5,535 803 577 573,368GALLUP 8,872 897 780 889,775GRANTS 1,973 11 8 7,000HATCH 1,128 14 14 15,242

JEMEZ MOUNTAIN 67 13 11 12,582JEMEZ VALLEY 155 30 18 11,295

LAS CRUCES 4,546 152 88 70,007LAS VEGAS CITY 682 24 19 16,679

LOS LUNAS 2,173 103 101 87,357MAGDALENA 299 8 8 6,994

MORIARTY 332 17 15 12,760PECOS 164 19 2 2,250

POJOAQUE 292 45 45 38,155ROSWELL 2,145 115 104 130,032SANTA FE 5,290 473 316 243,191SOCORRO 576 36 26 13,027

TAOS 2,073 28 28 34,212WEST LAS VEGAS 933 32 25 29,467

TOTALS 80,819 6,825 5,959 6,220,154

4

Participation Rates. Findings in New Mexico reflect the national pattern of low participation. The average participation rate nationally in SES for SY 2005-06 was 14%. In SY 2006-07, Albuquerque Public Schools (APS) reported a participation rate of 12.1%. State data presented in Table 1 suggests that APS participation rates in 2007-08 were 10.94%, which was higher than the state average of 7.38% for all districts reporting in 2007-08. Participation rates varied from a low of .41% (Grants) to a high of 16.32% (Jemez Mountain). State totals were 7.38%, confirming the national pattern of low participation.

Reported Program Expenditures. Based upon data reported to the state, Table 1 provided a breakdown of district expenditures by number of students receiving SES for fiscal year 2007-08 (the latest year for which complete data are available). As the figure shows, some districts have a larger proportion of students receiving services per eligible students than others, partially explained by the fact that not all schools in a district are required to support supplemental educational services.

Supplemental Educational Service Providers Providers. Of 16 providers operating in both years, 13 service providers operated as for profit agencies, 1 was a non-profit agency, and 2 were community-based agencies. Locations where the tutoring services were delivered included: in-home services (3), at the child’s school (2), in the provider’s business center (2), and some combination of the aforementioned locations (3).

Two providers did not continue providing services in the second school year examined, 2007-08. Eleven providers were either new to operations in 2007-08 or did not provide information during 2006-07. Finally, 14 providers operating in both years were available for achievement analysis:

• A to Z In-Home Tutoring • Advantage Tutoring Services • ATF Teacher Tutoring Services • Club Z! In-Home Tutoring • Club Z! New Mexico • CompatibleLand, Inc. • Education Station • eProgress Academy • La Promesa A+ Tutoring • Northern New Mexico Network • One Room School House • Successylvan Farmington • Rio Grande Educational Collaborative (RGEC) • Youth Development Inc., (YDI)

Hours Recommended. The hours recommended by providers to complete their programs ranged between 22 to 40 hours, with two providers having variable recommended hours depending upon the students’ needs. Six of these providers recommended 26 hours or less, while the rest recommended 30 or more hours as completing a program of study.

5

Tutor Qualifications. Tutor qualifications fell into three categories: providers that hired only certified teachers (3), those that hired tutors with a Bachelor of Arts degree (3), and the rest hired tutors who had a high school diploma and some college courses, but no four-year degree.

Mission. All providers are dedicated to improving the academic success of students, as evidenced in their applications, where each was asked to describe their mission in providing tutoring services (12 of 14 responding). TABLE 2: SES SERVICE PROVIDER’S SELF-DESCRIPTION OF SERVICES

TUTORING SERVICE PROVIDER

PROVIDER’S SELF-DESCRIPTION OF TUTORING SERVICES *

A TO Z IN-HOME TUTORING “NOT ONLY HAVE CLASSROOM GRADES IMPROVED, BUT ALSO ATTITUDES, BEHAVIOR, ATTENDANCE, SELF-ESTEEM, AND OTHER LESS TANGIBLE FACTORS.”

ADVANTAGE TUTORING SERVICES “TUTORS EMPLOY EFFECTIVE STRATEGIES FOR ADDRESSING ESSENTIAL SKILLS, DRAWING UPON THEIR EXPERTISE AS EDUCATORS, IN THOUGHTFUL CONSIDERATION OF THE SKILLS EACH STUDENT POSSESSES…”

ATF TEACHER TUTORING SERVICES “PROFESSIONAL TEACHERS MAKE OUR PROGRAM EFFECTIVE AND SUCCESSFUL BECAUSE THEY HAVE THE TRAINING TO KNOW HOW TO HELP YOUR CHILD.”

CLUB Z! IN-HOME TUTORING “STUDENTS WHO COMPLETE CLUB Z! SUPPLEMENTAL EDUCATIONAL SERVICES SHOW AN AVERAGE OF A YEAR’S WORTH OF GROWTH, SO START YOUR CHILD ON THE ROAD TO ACADEMIC SUCCESS BY JOINING THE CLUB Z! CLUB TODAY!”

CLUB Z! NEW MEXICO “THE MAJORITY OF OUR STUDENTS IMPROVE NOT ONLY THEIR GRADES, BUT THEIR CONFIDENCE….WE HELP YOU HELP YOUR CHILDREN TO BE STUDENTS WITH THE NECESSARY ABILITIES TO MAKE THE EDUCATION SYSTEM WORK FOR THEM.”

COMPATIBLELAND, INC. “COMPATIBLELAND USES COMPUTERIZED CURRICULUM TO TEACH STUDENTS…. COMPATIBLELAND’S SES MODEL IS PROVEN EFFECTIVE.”

LA PROMESA A+ TUTORING “A+ TUTORING SERVICES USES COMPUTERIZED CURRICULUM TO TEACH STUDENTS….A+TUTORING SERVICES’ SES MODEL IS PROVEN TO BE EFFECTIVE.”

NORTHERN NEW MEXICO NETWORK “THE NORTHERN NEW MEXICO NETWORK PROVIDES SMALL GROUP TUTORING (NO MORE THAN THREE STUDENTS PER TEACHER) IN MATHEMATICS, READING, AND WRITING. OUR GOAL IS TO ACCELERATE ACADEMIC PERFORMANCE BY SIX MONTHS IN AN EIGHT WEEK PERIOD.”

ONE ROOM SCHOOL HOUSE “THE ONE ROOM SCHOOLHOUSE PROVIDES HIGH QUALITY TUTORING…THE PROGRAM WAS DESIGNED BY TEACHERS AND IS RESEARCH BASED…”

SUCCESSYLVAN FARMINGTON “THE FARMINGTON SYLVAN LEARNING CENTER, LOCATED IN THE SAN JUAN PLAZA, HAS BEEN HERE FOR 11 YEARS AND IT WORKS!”

RIO GRANDE EDUCATIONAL COLLABORATIVE (RGEC) “RGEC WILL HELP YOUR CHILD SUCCEED IN SCHOOL.” YOUTH DEVELOPMENT INC., (YDI) “YDI TUTORING OFFERS FREE TUTORING IN READING, LANGUAGE

ARTS, MATH, AND SCIENCE. TUTORS ARE TRAINED TO MEET YOUR CHILD’S NEEDS. YOUR CHILD WILL BE PROVIDED AN INDIVIDUALIZED PLAN FOR IMPROVEMENT.”

*SOURCE: PROVIDER APPLICATIONS TO THE STATE OF NEW MEXICO Complexity of Services. There are a wide variety of service delivery models in operation in New Mexico. Tutoring is offered before and after school, in homes and at schools, on computers and using more traditional methods of teaching. Individual tutors’ education, skills, and experiences can range from a young college student with his high school diploma and some college

6

coursework to a certified teacher with decades of experience. The wide variety in service provision complicates analysis of the effectiveness of tutoring on achievement.

Characteristics of SES Participating Students

SES Students With Valid Test Results. The numbers of SES students enrolled in grades that test using the New Mexico Standards Based assessment and receiving tutoring services was 4,573 out of a total of 7,771 SES students during SY 2007-08. Analysis of achievement was necessarily limited to those students that had academic test records.

TABLE 3: NUMBERS OF SES STUDENTS ENROLLED IN SBA TESTED GRADES

GRADE NUMBER OF STUDENTS

ENROLLED*

PERCENT OF SES ENROLLEES

NOT TESTED ON SBA 3,198 41.2

03 1,039 13.4

04 877 11.3

05 786 10.1

06 799 10.3

07 571 7.3

08 466 6.0

11 35 .5

TOTAL 7,771 100.0

*ENROLLED STUDENTS INCLUDES CONTINUOUS ENROLLMENTS, SUMMER ENROLLMENTS, AND IS NOT EQUAL TO ‘APPLIED’ IN TABLE 1.

Gender and English Language Learners. The percent of English Language Learners (ELL) among SES students compared to Non-SES SBA test-takers in 2008 is shown in Table 4.

Compared to Non-SES students, the English Language Learners among SES recipients were a greater proportion of the group than that observed in the eligible population (compare 33.9% SES to 27.5% Non-SES) English Language Learners.

There were slightly more males among the Non-SES, and correspondingly slightly more female students among the SES (Table 4).

7

TABLE 4: GENDER AND ELL OF THE SES TUTORED STUDENTS COMPARED TO NON-SESSTUDENTS

SES NON-SES

FREQUENCY PERCENT FREQUENCY PERCENT ENGLISH LANGUAGE

LEARNERS 2,614 33.9% 44,721 27.6%

MALE 2,235 48.9% 84,989 50.9%

FEMALE 2,338 51.1% 81,811 49.1%

Ethnicity. In looking at ethnicity as a demographic variable, the SES students compared to Non-SES were more heavily Hispanic (71.6% compared to 55.7%) and Native American (18.6% to 10.7%). The Non-SES students are more representative, that is, more like the general New Mexico population, according to the U.S. Census which states New Mexico has the following demographic characteristics: Gender: Male (49.4%); Female (50.6%); Ethnicity: Asian (1.4%); Black (2.2%); Hispanic (44.1%); White (68.7%); and Native American (9.2%).

Completion Rates. A major question raised by the research on SES effectiveness as discussed in the introduction above, is program completion rates (Table 5). The mean number of completed sessions was 13.49, the median was 15, and the mode was zero, indicating that there were more students in this category than any other. At the highest end of completion, 8 students completed between 51 and 81 sessions. The few students at the high end were receiving computer-based instruction. There were 544 students for whom the state received no information.

TABLE 5: SESSIONS COMPLETED BY NUMBER AND PERCENTAGE FOR SES STUDENTS

SESSIONS COMPLETED

NUMBER OF STUDENTS

PERCENT OF SES STUDENTS

0 1,390 17.9%

1-10 457 5.9%

11-16 1,690 21.7%

17-21 1,862 24.0%

22-51 1,820 23.4%

52-81 8 .1%

MISSING INFORMATION 544 7.0%

TOTAL 7,771 100.0

The research question under consideration was whether the service providers had effected the expected changes in the SES students’ Standards Based Assessments (SBA) scores for SY 2007-2008 over their test scores in SY 2006-2007 in reading and math. The expectation was to

8

observe a significant increase in proficiency levels for the tutored students over those who did not receive tutoring.

The following section reports on an analysis of tutoring effectiveness measured by student achievement on the state’s Standards Based Assessment, comparing test scores for the treatment group (SES receiving students) to the comparison group of similar students enrolled in the same schools.

EVALUATION OF ACADEMIC ACHIEVEMENT Research Question. This study examined one basic research question on the program effectiveness of SES: do SES students achieve higher academic gains than students who were not tutored?

Limitations of the Data. The first consideration was to assess whether the service providers had effected the expected changes in the students’ Standards Based Assessments (SBA) scores for SY 2007-08 over their performance in SY 2006-07. This goal was limited by the low participation in SES services, and by the large proportion of SES students that sign up for, but do not complete, these services.

Trends in the Underlying Student Population. Two and three year trends in the general New Mexico student population revealed statistically significant declines in math SBA scores. Comparing the SBA math mean scale score achieved in SY 2007-2008 for the entire NM Population of test takers to the level achieved in SY 2005-2006 represents a significant decline of -18.5 scale score points (t-value of -146.277, significant at .00).

Overall, New Mexico student achievement on the Standards Based Assessment in math (comparing the SY 2007-2008 mean for the entire New Mexico population of test takers to the SY 2006-2007 mean) represented a statistically significant contribution of approximately -5 scale score points to the earlier observed three year decline of -18 (Table 6).

TABLE 6: T-TEST RESULTS FOR NEW MEXICO STUDENTS TESTED ON SBA IN MATH COMPARING MEANS FOR SY 2007-2008 TO SY 2006-2007

N 2008

MEAN

2008

MEAN

2007 T-VALUE SIG T

160,607 642 641 -38.829 .000

Three year trends in reading show statistically significant declines: comparing the SBA mean in reading for the entire group of test takers (651.33) in SY 2007-2008 to the mean for the same population in SY 2005-2006 (a mean of 661.84) shows a statistically significant decline as measured by the t test of significance (t-value of -94.348, significant at p<.00). The scale scores for 2008 were not statistically significantly different from the scale scores for 2007 in reading, such that the two-year trends in reading represent a plateau.

9

Having established the significant underlying trends, the analysis proceeded to compare SES to Non-SES to determine if trends for SES are different than trends for Non-SES. Are differences in scale score means between the SES and the Non-SES large enough to be statistically significant?

SES VS NON SES DIFFERENCES IN SCALE SCORES. NMSBA reading and math scale score means for the SES, Non-SES were compared, to each other and to the overall population, looking for significant differences between the groups on the SY 2005-2006, SY 2006-2007, and SY 2007-2008 assessments.

Average math achievement, comparing SES, Non-SES, and the overall New Mexico population established that SES math achievement followed the same pattern of decline. The reading results also mirrored those of the overall student population (no statistically significant changes).

TABLE 7: AVERAGE ACHIEVEMENT IN MATH AND READING: MEAN MATH SCALE SCORES COMPARING ALL NEW MEXICO STUDENTS, SES TUTORED STUDENTS, AND STUDENTS WITH NO TUTORING (SY 2006, 2007, 2008)

MATH SCALE SCORE (MEAN) 2006



NM STUDENT POPULATION 660 647 642 SES STUDENTS 657 630 625 NON-SES STUDENTS 656 647 642 READING SCALE

SCORE (MEAN) SY 2005-2006

READING SCALE SCORE (MEAN) SY 2006-2007

READING SCALE SCORE (MEAN) SY 2007-2008

NM STUDENT POPULATION 662 651* 651 SES STUDENTS 632 637 631 NON-SES STUDENTS 638 650 652

Conclusion: trends for SES students are not significantly different from overall student population, there were no statististically significant gains in either reading or math, and there was a statistically significant decline in math. (Statistical significance was tested by use of the T-test to compare means).

TABLE 8: T-TEST RESULTS FOR SES STUDENTS SHOWING NON-SIGNIFICANCE AND LACK OF GAINS FOR READING (SY2005-06 COMPARED TO SY 2007-2008)

N 2008 MEAN 2008 MEAN 2006 T VALUE SIG T

4552 631.14 632 -1.409 .159

What Does This Analysis Reveal?

• Math scores for the entire group of New Mexico students have declined over time, and these decreases are statistically significant.

• Math scores for SES have experienced a similar decline over both the two and three year periods.

10

• In looking at reading changes in scale scores for all New Mexico students, there was a statistically significant three-year decline, with a plateau for the last 2 years examined.

• The gains in reading for SES from SY 2005-2006 to SY 2007-2008 were statistically insignificant, and the decline from SY 2006-2007 to SY 2007-2008 also not statistically significant. Testing from 2005-2006 to 2007-2008, the result was no significant changes (mean difference of -.864).

What Next? Given these results, the analysis examined whether SES students profited more from SES tutoring than their non-tutored counterparts. The results for reading suggest that perhaps SES, as compared to their peers, might have made stronger gains, in order to maintain the plateau seen for all students. Another possibility is that the SES students experienced a less sharp decline in math.

The major methodological question is how to effectively compare SES to Non-SES. It is necessary to control for various influences that can affect achievement differentially for the SES students over the non-SES, such as prior achievement levels, school, special education needs, or gender. The answer is to create a special sample, using a method called “propensity scores” to create a control group. This methodology is outlined in the section that follows.

Sample Selection Process. Given the wide disparity in numbers, the SES students must be compared to a group of matched comparisons.

A matched pairs design helps to diminish the effects of size in order to better analyze whether tutored students have greater achievement than similar peers who were not tutored. An additional need for our sample is to control various influences that might affect the SES group differentially. This need is especially acute given the observed differences in demographics between SES and Non-SES. Without such a matched, controlled comparison group, it cannot be established whether the SES students’ observed differences can be attributed to tutoring services, rather than to variations in factors such as demographics.

Propensity score matching (Rosenbaum and Rubin, 1985: Rubin, 1997; Joffe and Rosenbaum, 1999) provides an answer. It is a refined approach to a matched pairs design. Propensity scores provide a method of controlling for factors such as the quality of instruction in the school, entering differences in student abilities, or some combination of these factors, while creating a matched pairs sample.

TABLE 9: SES STUDENTS AS PERCENTAGE OF ALL STUDENTS WITH VALID SBA SCORES IN BOTH YEARS

TOTAL NUMBER OF

STUDENTS IN 2008

NUMBER OF STUDENTS WITH VALID SBA

SCORES IN BOTH YEARS (% OF TOTAL)

NON-SES 165,376 162,227 (97.2%)

SES 7,731 4,561 (2.8%)

TOTAL 173,107 166,788

11

The matching variables used to select a student similar to the SES student were:

• School • District • Grade • Economic disadvantage • English Language Learner Status as determined by the New Mexico English Language

Proficiency Assessment (NMELPA) • Student with Disabilities based on federal criteria • Ethnicity • Gender • Pre-Test NMSBA Scores (scale scores from the 2007 NMSBA)

All the SES students were economically disadvantaged (by definition, to be eligible for SES services). The first step in creating the group of potential matched controls was to eliminate those students who were not economically disadvantaged from the pool of potential matches. Economic disadvantage was measured by whether the student was eligible for the Free and Reduced Lunch Program (FRL). This meant removing a group of 58,823 or 35.3% percent of the total of 166,800 with complete information regarding FRL status, before continuing with the analysis of 107,977 who were not economically disadvantaged.

Next, the groups were matched on a variety of covariates using the following procedure:

1. The procedure started with a sample from treatment group of interest (SES tutored students). In our case, the sample of SES student with valid test scores on the SBA for the two years of interest, 2007 and 2008.

2. The comparison group was drawn from the larger database of students taking the SBA who were not tutored, but had valid test scores in the two years considered (2007, and 2008).

3. Logistic regression analysis predicted group membership (SES or non-SES) from the covariates that might affect achievement (school, grade, ethnicity, ELL, SPED, prior year’s math SBA scale score, prior year’s reading SBA scale score). (See Table 16 below)

Propensity scores created for each student reflect, in one variable, the desired combination of student characteristics (Table 10).

12

TABLE 10: AN EXAMPLE SHOWING PROPENSITY SCORE RESULTS MATH SCALE SCORE 2007

READING SCALE SCORE 2007

ENGLISH LANGUAGE LEARNER

MALE FEMALE BLACK CAUCASIAN

HISPANIC

NATIVE AMERICAN

PROPENSITY SCORE

SES

674 693 NO YES NO NO NO YES NO .01339 YES 707 683 NO YES NO NO NO YES NO .01261 NO 642 665 NO NO YES NO NO YES NO .01912 YES 662 688 NO NO YES NO NO YES NO .01821 NO 682 660 NO YES NO NO YES NO NO .01602 YES 704 725 NO YES NO NO YES NO NO .01261 NO 527 557 YES NO YES NO NO NO YES .00803 YES 571 619 YES YES NO NO NO NO YES .00493 NO 664 614 YES NO YES NO NO YES NO ,00316 NO

4. Propensity groups based upon the quartile distribution of propensity scores were created:

the quartile cut scores were: .0106883, .0191488, .0288963, .0530488, with a maximum of .77985 and a minimum that was close to zero.

5. Computed the probability of being in the treatment group using the logistic regression function based on the covariates.

6. Form propensity groups. Using quintiles to establish the groups within which students are matched most closely, assign each student to one of four propensity groups were created (Propensity Group 1, Propensity Group 2, Propensity Group 3, Propensity Group 4).

The results were four groups that are matched on the covariates of interest (see Table 11 below). The treatment effects (SES tutoring) that are observed can then be attributed to the difference in test scores due to tutoring, rather than to variations in ethnicity, prior achievement, etc.

TABLE 11: THE DISTRIBUTION OF SES AND NON-SES WITHIN THE FOUR PROPENSITY GROUPS PROPENSITY GROUP TREATMENT GROUP 1 (25%) 2 (25%) 3 (25%) 4 (25%) TOTAL

NON-SES 17,088 16,848 16,406 15,887 66,229

SES 146 379 831 1,356 2,712

TOTAL 17,234 17,227 17,237 17,243 68,941

Although the technique of propensity score matching allows a control for covariates, it was also necessary to first rule out issues of test-retest unreliability, since the analysis dealt with multiple measures, before continuing to analyze achievement.

Ruling Out Test-Retest Unreliability. Another aspect that might have affected differences in achievement for SES involved multiple measures. Perhaps the SES students performed better in test conditions during the second year, so that gains were not attributable to the tutoring, but rather to differences in the test. Correlations between the two test administrations for both content areas, reading and math, tested this assumption.

13

Table 12 shows the pattern of correlations for the entire group of matched pairs. Correlations observed are those that would be expected if solid test re-test reliabilities existed: that is, all the correlations achieve statistical significance.

Given these results, it is safe to conclude that differences in the test will not be a major factor affecting achievement results for the sample of SES and Non-SES matched pairs.

TABLE 12: CORRELATIONS BETWEEN TESTS FOR THE MATCHED PAIRS PROPENSITY SAMPLE TO TEST FOR TEST-RETEST RELIABILITY

MATH

SCALE SCORES

2007

READING

SCALE SCORES

2007

MATH

SCALE

SCORES

2008

READING

SCALE SCORES

2008

MATH SCALE SCORES

2007

PEARSON

CORRELATION 1.000 .791** .676** .422**

SIG. (2-TAILED) .000 .000 .000

N 5,425 5,425 5,425 5,425

READING SCALE

SCORES 2007

PEARSON

CORRELATION .791** 1.000 .605** .549**

SIG. (2-TAILED) .000 .000 .000

N 5,425 5,425 5,425 5,425

MATH SCALE SCORES

2008

PEARSON

CORRELATION .676** .605** 1.000 .428**

SIG. (2-TAILED) .000 .000 .000

N 5,425 5,425 5,425 5,425

READING SCALE

SCORES 2008

PEARSON

CORRELATION .422** .549** .428** 1.000

SIG. (2-TAILED) .000 .000 .000

N 5,425 5,425 5,425 5,425

**. CORRELATION IS SIGNIFICANT AT THE 0.01 LEVEL (2-TAILED). SES vs. NON-SES Differences in Math and Reading Achievement. Having established that neither characteristics of the SES versus the Non-SES students nor differences in the two tests are significantly impacting the sample, the analysis proceeded by looking at differences in achievement.

14

Reading Achievement Results. OLS regression is a method of analysis for assessing the strength of the relationship between each explanatory variable: whether the treatment group (tutored SES students) had statistically significantly gains in reading and math scale scores—holding the prior year’s scale scores constant.

The multiple regression first tested whether the variables in the analysis (as indicated in the correlation coefficient for the complete model) indicated a strong, statistically significant correlation between the observed math scores and those predicted by the regression model (factors, treatment group and prior reading scores). This was demonstrated in the R square statistic which was significant. The F ratio that was found reflected the ratio of regression sum of squares to the residual sum of squares, which compared unexplained to explained variance in the reading scores attributable to the independent variables entered in the equation, and was significant.

A significant result indicates that the analysis can proceed to estimations of the effects of single variables. The adjusted R square statistic indicated that changing the number of factors thought to predict the reading scores would not substantially improve estimation of R squared (Landau and Everitt, 2004).

TABLE 13. “MODEL FIT”-- TESTING THE SIGNIFICANCE OF SES AND PRIOR READING SCORES AS PREDICTORS OF

READING ACHIEVEMENT GAINS

R R SQUARE

ADJUS

TED R

SQUAR

E

STD. ERROR

OF THE

ESTIMATE

CHANGE STATISTICS

R SQUARE

CHANGE F CHANGE

D

F

1 DF2

SIG. F

CHANGE

.731A .534 .534 29.084 .534 3106.418 2 5421 .000

MODEL

SUM OF

SQUARES DF MEAN SQUARE F SIG.

1 REGRESSION 525304.794 2 2627652.397 3106.418 .000A

RESIDUAL 4585507.062 5421 845.878

TOTAL 9840811.855 5423

A. PREDICTORS: (CONSTANT), TREATMENTGROUP,

READINGSS2007

B. DEPENDENT VARIABLE:

SBA_READ_SS0708

The bivariate regression creates unstandardized beta coefficients which tested specifically the research question: do SES students display more achievement gains as compared to Non-SES

15

matched pairs, controlling for prior reading scores? The table displays these unstandardized beta coefficients, as the estimate for the reading gains in the SES group when prior year’s achievement is held constant. One interpretation of the unstandardized beta is that the Treatment Group (SES) gained .062 scale scores for every one scale score gained by the Non-SES students. As can easily be seen, this was a non-significant difference. TABLE 14: UNSTANDARDIZED BETA COEFFICIENTS SHOWING NO SIGNIFICANT DIFFERENCE IN GAINS IN READING FOR SES COMPARED TO MATCHED NON-SES STUDENTS

UNSTANDARDIZED COEFFICIENTS

STANDARDIZED

COEFFICIENTS

T SIG. B STD. ERROR BETA

1 (CONSTANT) 116.216 6.659 17.452 .000

SES GROUP .062 .790 .001 .079 .937

READINGSS2007 .840 .011 .731 78.801 .000

A. DEPENDENT VARIABLE: SBA_READ_SS0708

Math Achievement Results. The same OLS regression analysis was conducted to examine SES vs Non-SES achievement results for math. The model fit summary table showed that the R square value of explained variance was significant enough to proceed to test the effects of each variable separately considered. Testing the effects of SES tutoring on math achievement, unfortunately, the unstandardized beta coefficients for this model did not reach significance, although the unstandardized beta for treatment group shows that the tutored students gained 1.873 scale score points in math over the students that did not receive tutoring (controlling for prior math scores). This effect is stronger than that observed for SES students in reading, but of insufficient magnitude to reach statistical significance at the p<.01 level.

TABLE 15: UNSTANDARDIZED BETA COEFFICIENTS SHOWING NO SIGNIFICANT DIFFERENCE IN GAINS IN MATH FOR STUDENTS WHO WERE TUTORED OVER MATCHED STUDENTS WHO RECEIVED NO TUTORING

UNSTANDARDIZED COEFFICIENTS

STANDARDIZED

COEFFICIENTS

T SIG. B STD. ERROR BETA

1 (CONSTANT) 152.446 7.162 21.285 .000

SES GROUP 1.873 .893 .021 2.098 .036

MATHSS2007 .788 .012 .674 67.053 .000

A. DEPENDENT VARIABLE: SBA_MATH_SS0708

16

The results above reveal weak or moderate results for math and no significant results in reading for the SES. These results were not statistically significant in comparing SES to Non-SES. The analysis will not be able to proceed to compare individual tutoring service providers, nor to examine those aspects of the tutoring services that may have produced the results observed. The reasons are these: (a) the changes in scale scores are weak for all students, and (b) the difficulty of sorting out significant effects for each provider is complicated by the sample attrition noted earlier. Thus, this report moves to an examination of provider customer service and compliance.

EVALUATION OF SERVICE DELIVERY

In addition to evaluating student achievement, the state evaluation included supplemental core categories, including: Completion Rates, Communication, and Overall Performance. For each category, the evaluator synthesized several sources of available data (achievement analyses, surveys, monitoring tools, provider applications). Limitations of the Data. Ascertaining the extent of stakeholder satisfaction is key to effective program implementation. The satisfaction surveys developed for this and earlier evaluations provide valuable insight into the self-reported experiences of those stakeholders who responded. However, collection of on-site observational data will improve the nature and level of this data collection effort, and improve the data quality.

Descriptive statistics showed wide variations in programs, operational models, and hiring practices from one provider to another. Wide variations in program delivery, curriculum, methods, and practices create a methodological difficulty when it comes to analyzing which service providers created greater gains.

Completion Rates. During 2007-2008, districts that had SES service providers were required to submit a customer satisfaction survey for each of the providers operating in their district. The full survey is available online at www.ped.state.nm.us/SES.

Responses were obtained from the following districts (the number in parenthesis indicates the number of surveys per provider that were submitted): Albuquerque (10), Bernalillo (2), Bloomfield (9), Central Consolidated (2), Cuba (6), Des Moines (1), Dexter (2), Dulce (3), Espanola (8), Farmington (1), Gadsden (11), Gallup (9), Grants (2), Hatch (1), Jemez Mountain (1), Jemez Valley (1), Las Cruces (14), Las Vegas City (8), Los Lunas (1), Loving (1), Lovington (2), Portales (1), Ruidoso (1), Santa Fe (11), Taos (3), West Las Vegas (1).

Districts’ Explanation of Completion Rates. Completion rates have been shown to be an important prediction of overall provider success in raising achievement. Therefore, districts were asked during SY 2007-2008 to report whether any providers in their districts had promised to provide services, but had failed to do so, and to state the reason(s).

17

A few examples were:

• “Overall they did a good job, but there were students that were never served and not all students began service within 30 days of enrollment.”

• “They only provided at specifice (sic) sites. Many schools that had students who had chosen Advantage went unserved.”

• “Babbage Tutoring Company, stated that not enough studens (sic) had chosen them to make it feasible (sic) for them to tutor in our district.”

• “They are all online services and then they learned our kids don't have computers or internet at home. They failed to hire a person to stay after school with kids when we offered our computer lab.”

• “This company did not provide any services. They could not obtain tutors. Last year they had an onsight (sic) coordinator. This year they had no one and therefore did not realize the vastness of this district.”

• “Srvices (sic) were not provided to students until second semester due to lack of coordination of tutors.”

Providers’ Explanation of Completion Rates. All 27 providers operating in New Mexico during SY 2007-08 were administered a survey similar to that administered to the districts. They were asked functionally equivalent questions to provide a comparison to district responses. Some 26 providers responded, these were:

1st Place Tutoring A+ Tutoring A to Z In-Home Tutoring ABC Phonetic Reading Advantage Tutoring ATF Teacher Tutoring Services Chrysalis Club Z! Inc Club Z! New Mexico CompatibleLand, Inc Education Station FELC Tutors Huntington National Farm Workers SC Northern New Mexico Network Rio Grande Educational Collaborative (RGEC) Sylvan-Farmington (Successylvan) Tutorial Services Youth Development, Inc. (YDI)

Providers that did not respond to the survey (N=7) were; BabbageNet, CAIR, Catapult On-Line, CompassLearning, Corridor, eProgress Academy, One Room Schoolhouse.

Providers were asked about situations in which services that students signed up for were not completed: “Were you initially contracted to provide services to children in this school

18

district, but then were unable to do so?” All 27 providers offering services during SY2007-2008 answered the question: 8 stated “yes” and the remainder stated “no”. Those who provided comments listed the following reasons:

• “Families moved to other schools that were not on the list, or out of state, or out of the country.”

• “Some were no shows; some parents changed their minds about tutoring after school l(sic) (no transportation, no one at home to care for other siblings).”

• “Some students we were assigned to did not follow through with their tutoring. Our final of students who actually received tutoring from us was 87.”

• “Some students had to be returned to the district due to a lack of tutors.” • “Some students had to be returned to the district because they needed a bilingual tutor.” • “Some students had to be returned because of lack of tutors in the region.” • “One student required a Bilingual tutor which we didn't have at the time. That student

was given to another agency.” A comparison of the district’s responses to these provider results provided anecdotal answers to the question of why there were so many students with low completion rates.

Low Completion Rates. Table 16 shows the list of providers with the most students with low completion rates. Two providers were in the top five list two years in a row: Catapult On-Line and Advantage Tutoring Services, despite the fact that Catapult’s percentage reduced from more than one half to just above one third. Club Z! Inc. dramatically reduced the percentages of students with low completion rates.

TABLE 16: PROVIDERS WITH THE HIGHEST PERCENTAGES OF STUDENTS WITH ZERO COMPLETED HOURS TOP FIVE PROVIDERS WITH HIGHEST PERCENTAGES OF STUDENTS WITH ZERO COMPLETED HOURS DURING SY 2006-2007:

SY 2006-2007 PERCENTAGE

OF STUDENTS WITH ZERO COMPLETED

SESSIONS

SY 2007-2008 PERCENTAGE OF STUDENTS

WITH ZERO COMPLETED SESSIONS

CLUB Z! INC. 57.9% 8.4% CATAPULT ON-LINE 53.7% 36.5% EPROGRESS ACADEMY 31.6% NOT CONTINUED IN SY

2007-2008 COMPATIBLELAND, INC 18.7% 11.1% ADVANTAGE TUTORING SERVICES

13.7% 28.8%

SES providers who fit the criteria for having the most students completing their respective programs in SY 2007-08 are listed below. The criteria were that the provider was in the top quintile for having students complete programs, and that the highest percentage of students within the program fell within the category of most completed hours, that is, had completed between 22-82 hours.

19

High Completion Rates. Providers with high rates of students that best completed their respective tutoring programs were these:

• Successylvan —62.5% • BabbageNet Schools, Inc. —52.2% • Education Station LLC —45.9% • Northern New Mexico Network —38.7%

Whatever the reasons for non-completion of sessions, an area of concern emerges: how to increase levels of completion.

Communication. Providers and districts reported in parallel surveys during SY 2007-2008 on the communication that occurred between the tutoring service providers and districts, teachers, and parents. Communication is the key to cooperative relations between the districts, classroom teacher and the tutors. The reliability rating of the four item communication survey (See www.ped.state.nm.us/SES for district Survey), was quite high, at .87 with four items and 143 valid responses.

Provider’s Responses About Communication.. Of the 26 providers who answered the questionnaire, and were asked, “What is the average # of times during the tutoring process that tutors are expected to communicate with parents?” Very few providers (3.0%) responded that tutors were not required to communicate with parents, 27.3% reported communicating regularly with the parents, at the level of 3 or more times in total, 51.5% reported communicating with the parents once each session, and 21.2% reported once each week. 7 providers stated that their policy was to communicate with the parent after every 5th hour of instruction.

Providers (N=26) were also asked to respond to this question: “What is the average number of times during the tutoring process that tutors are expected to communicate progress to the child’s teacher?” Answers were as follows:

Not required —47.3% Twice (once at the beginning and once at the end of tutoring) —12.8% Regularly, say three or more times in total —33.5% Once each session —0.0% Once each week —6.4% Providers were asked to report on teacher’s co-operation and involvement, and they reported:

• Minimal—6 of 26 providers agreed with this assessment • A little bit—2 of 26 providers agreed with this assessment • Somewhat—15 of 26 providers agreed with this assessment • Very much—6 of 26 providers agreed with this assessment • A great deal—3 of 26 providers agreed with this assessment

20

District Responses About Communication. Districts responded as follows to the question(s) about communication:

% that answered provider contacted district ‘frequently’—23.6% (7 of 27)

% that answered provider contacted parents ‘frequently’—22.9% (6 of 27)

Communication as Contributions to Overall Provider Performance. The correlational analysis above demonstrates those aspects of communication that most contributed to the district’s overall satisfaction with the services of the provider. The only significant correlation is between “how often did provider communicate with parents…” This item correlated at a moderate .343, with significance at the lower level of p < .05 level.

TABLE 17: DISTRICTS’ EVALUATION OF QUALITY OF PROVIDER COMMUNICATION DURING SY 2007-2008

ASPECTS OF COMMUNICATION: “OVERALL, I AM SATISFIED WITH SERVICES OF

THIS PROVIDER.”

HOW OFTEN DURING THE SCHOOL YEAR DID THIS

PROVIDER COMMUNICATE STUDENT PROGRESS

WITH YOU?

PEARSON

CORRELATION .184

SIG. (2-TAILED) .152

N 62

HOW OFTEN DID PROVIDER COMMUNICATE WITH

PARENTS IN AN EASILY UNDERSTANDABLE FORM?

PEARSON

CORRELATION .343**


N 62

HOW OFTEN DID PROVIDER COMMUNICATE WITH

TEACHERS?

PEARSON

CORRELATION .208


N 62

HOW OFTEN DID PROVIDER MEET OBLIGATIONS

FOR CONDUCTING STUDENT SESSIONS?

PEARSON

CORRELATION .299*


N 62

** CORRELATION IS SIGNIFICANT AT THE 0.01 LEVEL (2-TAILED).

* CORRELATION IS SIGNIFICANT AT THE 0.05 LEVEL (2-TAILED).

21

With some differences in strength of endorsement, providers and districts agree on the levels of communication that occurred between the tutoring service providers and teachers and parents. The picture that emerges is that communication is an area identified for improvement, especially communication that occurs between tutors and teachers.

Perceptions of Achievement. The correlational analysis displayed in the table below confirms the importance of perceptions of SES providers’ achievement to the district’s assessment of overall satisfaction with providers. There is a strong relationship between the district’s overall assessment of the provider’s ability to positively impact student achievement (Pearson correlation coefficient=.667, Sig p <.01), responsiveness (Pearson correlation coefficient=.685, Sig p<.01), and overall satisfaction with the provider.

TABLE 18: DISTRICT’S EVALUATION OF TUTORING COMPANY SERVICES’ OVERALL EFFECTIVENESS

ITEMS CONTRIBUTING TO DISTRICT OVERALL SATISFACTION WITH PROVIDER:

“OVERALL, I AM SATISFIED WITH

SERVICES OF THIS PROVIDER.”

OVERALL, HAS POSITIVELY IMPACTED STUDENT

ACHIEVEMENT IN DISTRICT.

PEARSON CORRELATION .667**


N 59

OVERALL, I AM SATISFIED WITH THE

RESPONSIVENESS OF THIS PROVIDER.



N 53

OVERALL, HAS COMPLIED WITH SES DISTRICT

CONTRACT.



N 58

These results can be also interpreted as providing a rough estimate of district satisfaction overall with providers. The results show remarkably high rates of satisfaction among New Mexico districts, as compared to other state’s findings. For example, only half of the district coordinators who responded in Tennessee’s SY 2007-2008 evaluation (50.0%; n=30/60) agreed with this statement: “Overall, I am satisfied with the services of this provider.” (Source: 2007-2008 Supplemental Educational Services in Tennessee Evaluation Report).

Areas that Need Improvement Completion Rates. Districts noted extensive dissatisfaction when a provider signed a contract and then did not deliver services. Very low completion rates for a high number of students who apply for services initially and then do not complete sessions emerge as an area of concern.

22

Communication. Improving communication between the providers and districts, parents, and teachers of enrolled students is indicated. Contact between the provider and the child’s teacher remains the greatest challenge. Research shows the importance of school interventions having direct connection to regular classroom learning (Slavin & Fashola, 1998). In the case of SES, frequent communications with teachers and school leaders should increase the continuity, consistency, and relevance of the instructional program.

Student Achievement. Analysis comparing last year’s SBA scores to this year’s SBA scores for tutored versus comparable students who did not receive tutoring, the findings were these: (1) weak or moderate results for math, (2) no significant results in reading.

Suggestions for Improvement

State education agencies play a critical role in establishing, administering, and sustaining high-quality extended learning opportunities. This role has increased over time, largely attributable to state administration of the 21st Century Community Learning Centers (21st CCLC) Program and the Supplemental Educational Services (SES) provisions of the No Child Left Behind (NCLB) Act. An ongoing challenge is ensuring that students who need them most have access to high-quality programs. To achieve continuous improvement in SES program quality, steps to improve evaluation and enhanced monitoring ought to be considered.

Determine Reasons for Non-completion of Program. Without data on why students fail to show up for services, or fail to complete services, it is difficult to address the sources of the problem and improve retention rates. Implementation of a parent survey, to be conducted by PED, or an independent outside contractor, could include all parents of children that failed to “show” for services once applied, or that fail to complete a minimum number of hours. Contact information should be required from districts or providers, and supplied to PED, so that a telephone survey can be conducted by an independent source to determine the reasons that students or parents fail to complete tutoring services. In short, implementation of a parent survey for those students is warranted. This data can then be shared with providers to allow them to develop strategies to address this crucial issue.

Stronger District Monitoring of Providers. Contact between the provider and the child’s teacher remains the greatest challenge. District On-Site visits can improve communication, verify use of Student Instructional Plans, and create stronger data to support monitoring decisions. PED plans to assist districts with technical assistance to implement on-site monitoring in 2009-10.

Implement Program Quality Improvement Measures. The PED is moving towards the development of an instrument or tool, or perhaps a list of possible instruments, to provide technical assistance to SES providers in valid, reliable methods to create continuous improvement in program quality. State education agencies (SEAs) have the responsibility to play a critical role in developing and sustaining high-quality, effective programs.

23

Conclusions The success of the SES program will be determined as more information becomes available about how it is working and whether it is an option that parents find attractive. So far, few national or statewide evaluations have demonstrated conclusively the effectiveness of SES in improving student achievement performance on standardized state tests as required by and intended by NCLB.

Tutoring Service companies operated using greatly varied methods in New Mexico. Tutoring is a service that occurred before and after school, in homes and at schools, used entirely computer-based curriculum, was set in a traditional classroom, took place in small groups or one-on-one. Tutors varied widely in education, skill, and experience. Given such wide variety in tutoring services, it is difficult to quantify the effectiveness such tutoring services might have on achievement.

Acknowledgments:

This report was prepared by Dr. Stephanie Amedeo Marquez under the supervision of Dr. Cindy Gregory.

Technical Report:

To request the full technical report call (505) 827-6509 or email the PED helpdesk: [email protected]

This report and others are available on the PED Internet site at www.ped.state.nm.us .

ses report 07-08

Documents