Dr. Carol AlbrechtResearch Team

EXAMPLE of EVALUATION RESEARCH

18 public school districts in the state of Texas were involved in Service Learning Projects. The State of Texas Service Learning Center hired Dr. Carol Albrecht and her students to evaluate this program. These power point slides outline the steps they tookto complete this evaluation.

We met with them to identify their objectives. They wanted to know how the program impacted public school children, teachers, community partners and parents of students.

Click here to see the surveys and

codebooks.

Click here to see the surveys and

codebooks.

An Old Chinese Proverb States:

I HEAR, I FORGETI SEE, I REMEMBER

I DO, I UNDERSTAND

By “doing” this project we learned and really understood some important

components of valid and reliable evaluation research.

Timing of the Pre-testTiming of the Pre-test

Many programs are ongoing, and this canMany programs are ongoing, and this can

have a major impact on the pre-test.have a major impact on the pre-test.

In our study, many of the students had In our study, many of the students had

already participated in a Service Learningalready participated in a Service Learning

activity at some point in their school years.activity at some point in their school years.

So, we didn’t have a true “pre” test. TheSo, we didn’t have a true “pre” test. The

““pre” test scores were contiminated bypre” test scores were contiminated by

prior participation.prior participation.

Timing of the Post-TestTiming of the Post-Test

The “May” EffectThe “May” EffectOutside environmental/social factors needOutside environmental/social factors need

to be considered. to be considered.

In our study, we discovered that both teachers and In our study, we discovered that both teachers and students were more negative about almost EVERYTHING students were more negative about almost EVERYTHING related to school at the end of the year. This may berelated to school at the end of the year. This may be

explained by two factors: explained by two factors:

First, they were just TIRED of school, and lookingFirst, they were just TIRED of school, and looking

forward to vacation.forward to vacation.

Second, they had just taken standardized tests, Second, they had just taken standardized tests, TAKS. These tests were stressful for both TAKS. These tests were stressful for both

teachers teachers and students.and students.

Head Start Program In the 1960’s the head start programwas launched. The objectivewas to increase the IQ scores of underrepresented

populations including children living in poverty.Early research showed that standardized tests of IQ

increased for several years, and the decreased, until there was no difference between the experimental and control groups. While some felt this was evidence for

discontinuing the program, parents came forward arguing that the researchers

weren’t using the right measurements.

Head Start Program

A group of researchers call The Perry Preschool Consortium, with The input of teachers and parents, identified(1) social, (2) educational and (3)

socioeconomic indicators that differentiated preschool participants from a control group up to 19 years after participation in the program. The differences were compelling.

Furthermore, this group arguedthat the decreasing IQ scoresactually provided evidence that environmental factors CAN influence IQ – both positively and negatively. Thus being In an “enriched” environment (i.e., the Head

Start Program) can increase IQ but then being transferred to an impoverished environment

(i.e., public schools in poor neighborhoods) can decrease IQ.

Schooling SuccessHigh School Graduation or EquivalentCollege or Vocational TrainingFunctional Competence

Ever Classified as Mentally RetardedTime Spent in Special Education

Social Responsibility

Ever Detained or ArrestedTeen Pregnancies

EmployedReceiving Welfare

Using focus groups and intensive interviews,

we looked to

(1) Teachers,

(2) Parents

(3) And student participants

as well as past research, to

help us identify valid and accurate indicators.

Analysis of this qualitative data indicated

(1) Some students did experience the desired impact.

(2) We needed to use “control” variables to accurately assess the impact of Service Learning.

The following control variables were all highly significantly related to EVERY outcome measurement.

Student Success

Results from focus groups with students and intensive interviews with teachers

indicated that these were valid indicators of the quality and quantity of

participations were related to outcomes.

Mean Score on Outcome Measurements by Amount of Time Student Planned or Participated in Project

Students Planned -Four Hours or More

Yes No

Attitudes Toward Attending CollegeAttitudes Toward SchoolProblem Solving SkillsLeadership Skills

12.81***19.84***17.01***17.15***

12.50***18.45***15.58***15.85***

Students Participated - Four Hours or More

Yes No

Attitudes Toward Attending CollegeAttitudes Toward SchoolProblem Solving SkillsLeadership Skills

12.89***19.54***16.91***17.04***

12.18***18.38***15.09***15.46***

***p<.0001

* p< .05 **p< 0.01 ***p< 0.0001

Mean Scores on Outcome Measurements by Sense of Ownership – Whether or Not Students Made Decisions

Students Made Decisions

Yes No

Attitudes Toward CollegeAttitudes Toward SchoolProblem Solving SkillsLeadership Skills

13.04***19.59***

17.26***p17.51***

12.36***18.84***15.58***15.81***

* p< .05 **p< 0.01 ***p< 0.0001

Click here to see the Power Point Presentation.

CHART 1. High School Students’ Perception of How Good They are at Speaking in Front of Groups by Whether or Not They Made Decisions about Service Learning Projects

(p <0.0001)

CHART 2. High School Students’ Perception of How Good They are at Finding Resources by Whether or Not They Made Decisions about Service Learning Projects

11%7%

44%

13%7%

28%24%

9%

43%

17%

0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%

100.00%

StronglyDisagree

Disagree Undecided Agree Strongly Agree

Did

Did Not

(p <0.0001)

Beware of Social Desirabil

ity

In Evaluation Research, Participant Often Evaluate

a Program Positively – EVEN when the Program is “poor” and Ineffective.

It may not be seen as socially acceptable to do

otherwise.

Why did we become concerned?Why did we become concerned?

What are the “danger” signs? What are the “danger” signs?

How did we attempt to alleviate it? How did we attempt to alleviate it?

How did we modify the constructionHow did we modify the construction

of our surveys, our research design and of our surveys, our research design and

analysis of the data to deal with this analysis of the data to deal with this

problem?problem?

Danger SignsDanger Signs

Type of researchType of research

Past literature indicates that respondents tend to Past literature indicates that respondents tend to bebe

very positive when asked about their participation very positive when asked about their participation inin

a program even when it is a poor program. They a program even when it is a poor program. They don’ don’ want to believe they wasted their time, and want to believe they wasted their time, and they oftenthey often

feel an obligation to express appreciation for feel an obligation to express appreciation for thosethose

who implemented the program. who implemented the program.

Danger SignsDanger Signs

Self selection into the programSelf selection into the program

Students and teachers were not required to Students and teachers were not required to

participate in the program. Therefore the participate in the program. Therefore the programprogram

was more likely to attract participants who was more likely to attract participants who

already had positive attitudes toward thesealready had positive attitudes toward these

““types” of activities. types” of activities.

Danger SignsDanger Signs

Consistently high scores on everyConsistently high scores on every

aspect of the program – no variationaspect of the program – no variation

Response Set can occur. This is where respondents Response Set can occur. This is where respondents give you the same response (usually positive) give you the same response (usually positive)

without without seriously considering the question. seriously considering the question.

The “ceiling” effect is a similar problem. ThisThe “ceiling” effect is a similar problem. This

is when you get consistently highly positive is when you get consistently highly positive scoresscores

on the pre-test. In this case, there is little room on the pre-test. In this case, there is little room

for improvement in scores.for improvement in scores.

Check ListCheck List

Make participation voluntary and make answers anonymous or confidential.Make participation voluntary and make answers anonymous or confidential.

Vary negative/positive statements inVary negative/positive statements in

Your indexYour index

Avoid misleading/biased questionsAvoid misleading/biased questions

Make statements or questions very specificMake statements or questions very specific

Check List – continuedCheck List – continued

Make participation voluntary and make answers anonymous or confidential.Make participation voluntary and make answers anonymous or confidential.

Put “sensitive” questions at the endPut “sensitive” questions at the end

Ask how they would change program “under ideal circumstances”.Ask how they would change program “under ideal circumstances”.

Avoid (1) yes or (2) no answers – ask “degrees” of positive or negative.Avoid (1) yes or (2) no answers – ask “degrees” of positive or negative.

Ask for their input in improving the program – rather than simply Ask for their input in improving the program – rather than simply

evaluating the program for instance:evaluating the program for instance:

NOT – Is this a successful program, but rather - what factors increase or decrease the success NOT – Is this a successful program, but rather - what factors increase or decrease the success of this program.of this program.

Check ListCheck List

If possible, don’t evaluate your own programIf possible, don’t evaluate your own program

An “outsider” would tend to be more objective and participants would be more An “outsider” would tend to be more objective and participants would be more likely to provide unbiased answers.likely to provide unbiased answers.

Have a variety of participants evaluate the program soHave a variety of participants evaluate the program so

you can look for consistencies/inconsistencies in answers.you can look for consistencies/inconsistencies in answers. StudentsStudents TeachersTeachers Parents of participantsParents of participants Community PartnersCommunity Partners

Check List - continuedCheck List - continued

Use multi-methods so you can compare results across Use multi-methods so you can compare results across

to see if you get similar results and look for additionalto see if you get similar results and look for additional

insights. These could include:insights. These could include:

Focus groupsFocus groupsParticipant observationParticipant observationSurveys Surveys Intensive interviewsIntensive interviewsContent analysis Content analysis

Content analysis is especially important for researchers who Content analysis is especially important for researchers who identify tangible products (e.g., bushels of grains) as their identify tangible products (e.g., bushels of grains) as their outcomes. outcomes.

Check ListCheck List

Compare your program with other programsCompare your program with other programs

Compare across different levels of participation within Compare across different levels of participation within your sample to see if there are variationsyour sample to see if there are variations

Compare across different types of participation within Compare across different types of participation within your sample (i.e., in our study, we compared across your sample (i.e., in our study, we compared across types oftypes of

Service Learning projects).Service Learning projects).

Check ListCheck List

Compare across different “types” of participants (This Compare across different “types” of participants (This wouldwould

include males vs. females, parents vs. children, rural include males vs. females, parents vs. children, rural vs. vs.

urban dwellers). urban dwellers).

Compare scores across questions – especially questions Compare scores across questions – especially questions thatthat

measure the same outcomes.measure the same outcomes.

Compare answers across time Compare answers across time

fall vs. summer participants fall vs. summer participants The most important thing to remember here is to NOT just ask if

the program was successful, but rather, HOW and WHEN it is most successful.

Laboratory Experiments

Intensive Interviews

Examples of data Examples of data

that the program is producing the desired that the program is producing the desired

outcomesoutcomes

collected from Teachers collected from Teachers Using Telephone SurveysUsing Telephone SurveysUsing Focus Groups Using Focus Groups

Descriptive Statistics for Elementary and Middle/High School Service Learning Teachers: Descriptive Statistics for Elementary and Middle/High School Service Learning Teachers: Extent to Which Teachers Agree with the Following Statements about the Impact of Extent to Which Teachers Agree with the Following Statements about the Impact of

Service Learning in Their ClassroomService Learning in Their Classroom

ElementaryElementary Middle/High SchoolMiddle/High School

NumberNumber PercentPercent NumberNumber PercentPercent

Positive Addition to Classroom LearningPositive Addition to Classroom Learning Agree/Strongly AgreeAgree/Strongly Agree

Beneficial for the ALL StudentsBeneficial for the ALL Students Agree/Strongly AgreeAgree/Strongly Agree

Motivates Students to be InvolvedMotivates Students to be Involved Agree/Strongly AgreeAgree/Strongly Agree

Helps Students Learn CurriculumHelps Students Learn Curriculum Agree/Strongly AgreeAgree/Strongly Agree

Should be Required for All StudentsShould be Required for All Students Agree/Strongly AgreeAgree/Strongly Agree

7272

7575

6666

4747

4040

90.0090.00

93.7593.75

87.5087.50

58.7558.75

50.0050.00

7373 7474 6666 5656

5050

96.0596.05

97.3797.37

88.8988.89

73.6873.68

65.7965.79

Descriptive Statistics: Identification of GREATEST BENEFITS by Descriptive Statistics: Identification of GREATEST BENEFITS by Middle/High School and Elementary Service Learning TeachersMiddle/High School and Elementary Service Learning Teachers

ElementaryElementary Middle/High SchoolMiddle/High School

PercentPercent PercentPercent

Benefits for StudentsBenefits for Students Service to OthersService to Others Understanding of WorldUnderstanding of World Personal GrowthPersonal Growth Help Learn CurriculumHelp Learn Curriculum

50.0050.00 14.7114.71 30.89 30.89 4.414.41

32.4332.43 25.6825.68 32.4332.43 9.469.46

Benefits for TeachersBenefits for Teachers Student GrowthStudent Growth Service to OthersService to Others Involvement with StudentsInvolvement with Students Break in TAKSBreak in TAKS

82.2682.2616.1316.13 1.611.61

----

73.9173.9115.9415.94 7.257.25 2.902.90

As One Teacher Stated, “Service Learning is the most powerful and impactful thing I ever did in the classroom as a teacher. It hooked me, and I am a believer in the power.

Another Teacher Claimed, “I think this program has transcended anything that anyone expected when they began the program. It has extended beyond what they thought it could achieve.”

One Teacher Argued, “I could have never ever taught the lessons they learned about human nature.”

While Another Claimed, “It teaches kids the skills that are not book skills….skills like how to think, how to plan, how to organize, how to manage - stuff you can read about in a book, but until you do it, you don’t know you have the ability to do it.”

One Teacher Stated, “school is not as…engaging as when they learn through these projects…they are learning all of these things by action – their great public speaking skills, their writing skills, their marketing…”

Another Teacher Explained, “in the writing TAKS, we had to write with a prompt so it kind of helped with the writing and the reading TAKS too.”

Examples of data Examples of data

that the program is producing the desired that the program is producing the desired

outcomesoutcomes

collected from parents and community partners collected from parents and community partners using telephone surveysusing telephone surveysusing focus groupsusing focus groups

Descriptive Statistics For Parents and Community Partners;Descriptive Statistics For Parents and Community Partners;An Evaluation of the Service Learning ProgramAn Evaluation of the Service Learning Program

ParentsParents Community PartnersCommunity Partners

MeanMean RangeRange MeanMean RangeRange

Positive Addition to ClassroomPositive Addition to Classroom

Beneficial for All StudentsBeneficial for All Students

Motivates Students to be Involved Motivates Students to be Involved

Helps Agency Achieve GoalsHelps Agency Achieve Goals Is Valued by AgencyIs Valued by Agency

Is Valued by CommunityIs Valued by Community

*on 5 pts. Scale (SD to SA)*on 5 pts. Scale (SD to SA)**sample size is small**sample size is small

5.005.00

4.834.83

4.834.83

----

----

----

00

11

11

----

----

----

4.944.94

4.834.83

4.894.89

4.394.39

4.674.67

4.564.56

11

11

11

33

33

33

One Community Partner Described Their Relationship with the School, “We Actually came to the schools…and we were looking for assistance. It’s a great marriage. We are still married.”

And when Describing the Benefits for Students,“..we’ve watched students mature into more

socially aware students - much more mature. It’s amazing. It’s just amazing.”

Timing of data collection is important.

Selecting reliable/valid indicators is critically important. Spend some time doing this.

IF you are doing Evaluation Research, plan ways to reduce the impact of social desirability on your results.

Use multi-methods when feasibility to provide additional insights and greater support for your results.

Try to gather information from all the different groups that may be impacted by the program (i.e., parents, students etc.).

Dr. Carol Albrecht Assessment Specialist USU Ext carol.albrecht@usu.edu (979) 777-2421