pilot stem in ost evaluation preliminary report
DESCRIPTION
Pilot STEM in OST Evaluation Preliminary Report. March 2012 University of California , Irvine. Study Sample Selected. 17 programs in 9 Regions participating Pilot Evaluation Study Regions 1, 2, 3 4, 7, 8, 9, 10, 11 Total of 65 sites recruited 52 Elementary 9 Middle School 4 K-8 - PowerPoint PPT PresentationTRANSCRIPT
Pilot STEM in OST Evaluation
Preliminary Report
March 2012 University of California , Irvine
Study Sample Selected• 17 programs in 9 Regions participating
Pilot Evaluation Study• Regions 1, 2, 3 4, 7, 8, 9, 10, 11
• Total of 65 sites recruited• 52 Elementary• 9 Middle School• 4 K-8
• Criteria for Selecting Study Sample:
Range of STEM curriculum approaches Range of student age groups (grades 3-12) Diverse ethnic and socio-economic backgrounds,
representative of students in the State of California Internet access
Overview of Evaluation Activities:
Selected Study Sites engage in the following activities:
Administer pre (fall 2011) and post (spring 2012) online student surveys
Administer online staff surveys at two time points: pre (fall 2011), and end-of-year (spring 2012)
Document Weekly Stem Activities—to be recorded daily by site implementers on STEM Activity Documentation Forms
Provide UC Irvine with copies of program schedule and lesson plans
Pilot Study Launch Fall 2011ACTIVITIES TO DATE:
Selection of pilot study sample sites
Set up of staff pre-survey and student pre-survey online links
Communication with pilot study sites:
1. Email sent informing Program Liaison of selected sites and overview of evaluation activities to be carried out
2. Individual emails sent to STEM Implementers at each study site including:1) Individual staff IDs assigned & instructions distributed for staff
surveys2) Site IDs assigned & instructions distributed for student
surveys3) STEM Activity Documentation Form (electronic and hard
copies) distributed with instructions & envelopes for returning forms to UC Irvine
3. Packages sent to each study site or program liaison with:1) Hard copies of STEM Activity Documentation Forms and
instructions2) Prepaid and addressed envelopes completed forms to UC
Irvine
Winter 2012 Evaluation Activities
Follow-up Communications & Reminders:1. Program liaisons sent a list of study sites and
corresponding staff IDs and site IDs Nov. 30/Dec. 1, 2011
2. Reminders sent to Program Liaisons & STEM Implementers• Reminders to complete surveys and begin
sending STEM Activity Documentation Forms
• Accounting of data collected from their program sites
• All site and staff codes and survey instructions resent with email
Ongoing contact with program Program Liaisons & STEM Implementers by phone and email
Data Collected to Date—March 15 2012
90 staff pre-surveys have been completed
35 sites have completed 1,277 student pre-surveys
Preliminary results of Staff and Student Pre-Surveys summarized in slides that follow
103 STEM Activity Documentation forms received reporting on 310 Individual Stem Activities Items reported by STEM implementers about each activity include:
Date and Duration of Activity Name of activity STEM content area addressed Number of students and grade level 4 point ratings of
1. Level of student Engagement
2. Level of challenge
3. Overall assessment of success of activity
Data are being entered in the data base for analysis and will be correlated with staff survey data and student survey site level data.
Summary STAFF Pre-Survey Data
Data collected December 2011–March 2012
SUMMARY OF STAFF PRE-SURVEY RESULTS
90 staff surveys collected between November 2011 and March 2012
73% respondents female.
1/3 are between 18 and 25 years old.
72% are 35 or under.
18-2526-3536-4546-55Over 55
AGE OF STAFF
STAFF SURVEY—LEVEL OF EDUCATION ACHIEVED
Nearly half (48%) have attended college or have an AA degree
42% have B.A. Degrees or higher
High School Diploma or GED
Attended classes/training not related to a degree
Attended college
Completed two-year college degree (AA)
Completed four-year college degree (BA)
Attended graduate school
Completed Master’s degree
Completed Doctoral degree
0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00%
3%
1 %
29%
19%
29 %
12 %
7%
0.00%
Level of Education n = 90
16%1%
7%
1%
36%3%
34%
2%
Staff Ethnicity (n = 90)
African, African-American
American Indian or Alaskan Native
Asian, Asian-American
Filipino
Hispanic or Latino
Pacific Islander
White, Caucasian
Other
Staff Reflect the Diversity of Students and Communities Served
STAFF POSITION IN AFTERSCHOOL PROGRAM
Less than 6 mo.
6-12 mo.
1-2 years
2-3 years
3-5 years
5-9 years
10 years or more
0% 10% 20% 30%
24%
16%
15%
12%
21%
12%
0%
Time in Current Po-sition
(n =90)
54%42.5%
3.5%
Current Position (n = 87)
Site Coordina-torActivity Leader at one siteProgram Coach work-ing with several sites
40% staff have less than one year in current position33% have more than 5 years in position27% have 1-3 years in position
STAFF EXPERIENCE IN OTHER SCHOOL SETTINGS62% have some experience as a classroom aide or TA
23% 1-5 years 14% more than 5 years
37% reported having some classroom teacher experience
15% 1-5 years 10.5% more than 5 years
15% have some school administrative staff experience
8% student support staff experience
Few staff report any administrative experience (6%)
Professional Experience
n < 6 mo 6-12 mo 1-2 yrs 2-5 yrs 5-10 yrs 10+ yrs
School Administrator
74 3% 0 0 3% 0 0
Student Support Staff
74 1% 0 3% 1% 0 3%
Administrative Staff 76 3% 3% 5% 4% 0 0
Classroom Teacher 79 2.5% 9% 4% 11% 2.5% 8%
Instructional Specialist
75 4% 3% 3% 9% 3% 3%
Classroom Aide, TA 84 14% 11% 11% 12% 6% 8%
Staff Self-Identified Instructional Roles
Science
Engineering
Reading/Language
Arts
Athletics
Other
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
53%
31%
16%
67%
66%
37%
57%
81%
23%
Instructional Roles (n = 90)
Variety of STEM Activities Implemented
18%
15%
9%
11%
16%
9%
22%
Type ImplementedLife Science
Physical Science
Earth Science
Technology
Information & Commu-nication Technologies
Engineering
Mathematics
Prompt: In your current position, do you implement any STEM activities with students? If so, please specify which types of activities (select all that apply):
None
15-30 min.
30-60 min.
1-2 hours
2-3 hours
3-5 hours
5-10 hours
10 hours or more
0% 5% 10% 15% 20% 25% 30%
22%
10%
25%
24%
11%
3%
3%
2% n = 90
Prior Experience & Time per Week Implementing STEM
Prompt: In your current position, how much time per week do you spend implementing STEM activities with students?
22% spend NO time implementing STEM Nearly half (49%) do at least 30 minutes to 2 hours of
STEM per week
30 % have no prior experience implementing STEM at another program
Staff Meetings Around STEM
Never Less than once a month
Once a month
2-3 times a month
Once a week
2-3 times a week
4-5 times a week
0%
5%
10%
15%
20%
25%
30%
0%
5%
18%
28%27%
13%
9%
21%19%
23%
13%
19%
4%
0%
Program IssuesSTEM Issues
21 % report never discussing STEM at staff meetings
55% discuss STEM in staff meetings at least once a month to once a week
n =90
Frequency of Staff Discussion of Program and STEM Issues
Compensation for Staff Meetings
( n = 89)
$ 66% staff report being compensated for all meetings
$ 10% for most meetings
$ 7% for some meetings
$ 15% receive no compensation for meetings
2% staff report having no meetings
Training and Support
0%
10%
20%
30%
40%
50%
60%
70%
11%
62%
16%
7%2% 2%
42%
54%
4%0% 0% 0%
Any Training
A little over half (52%) of staff surveyed have had at least 1-4 sessions of Stem Related Training in the past academic year
42 % have had NO Stem Training
n = 90
STAFF RELATIONS WITH TEACHERS
Curriculum concepts being taught in school
STEM concepts being taught in school
Homework assignments
Needs or progress of individual student
Issues related to classroom space
8%
32%
3%
6%
22.00%
20%
29%
14%
9%
26%
32.00%
14%
11%
16%
17%
11%
11%
17%
18%
12%
7%
4%
26%
26%
9%
> 1x/wk1x/wk
2-3x/mo1x/mo
n = 90
During the past academic year, how often have you discussed…
32% never speak with classroom teachers about STEM concepts taught in classroom
STAFF RELATIONS WITH PARENTS
Talk with parents about STEM activities
Hold STEM-related events for parents
34%
76%
37%
17%
17%
7%
6%
> 1x/wk1x/wk2-3x/mo1x/mo< 1x/moNever
1%
n = 89
76% of staff report never holding STEM-Related events for parents
36% say they never speak with parents about STEM activities
Staff Outcome Measures Beliefs & Sense of Competency
n Range Mean SD Alpha
Staff Beliefs about STEM
90 1.60-4 2.85 0.49 .73
Staff Sense of Competency
89 2-4.17 3.12 0.46 .79
Two staff outcome measures:5 point rating scale
o strongly disagree, disagree, neither agree or disagree, agree, strongly agree [some reversed coded]
Staff beliefs about STEM in the afterschool program. Example Items
I think the students enjoy doing STEM Activities In general, I think these students [in the afterschool program] are very
capable of doing hands-on science activities I don’t think there is enough time at the program for students to learn
much about STEM
Staff sense of competency implementing STEM in the afterschool program Example Items
I have a strong background in at least one area of STEM I do not know enough about Science, Technology, Engineering and/or
Mathematics to teach any of them well I feel confident about teaching Science, Technology, Engineering and/or
Mathematics in the afterschool program
Summary STUDENT Pre-Survey
DataData collected
December 2011–March 2012
Student Survey Sample
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
51% 49% 51%56% 58%
47%49% 51% 49%44% 42%
53%
BoysGirls
1,277 in Full Sample
51% Boys and 49% GirlsGrade and Gender of Student
Respondents
Time to Complete Survey
Full sample Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
10% 11%8%
11%8%
13%
6%
19%
14% 13%
23%25%
31%
60%
22%
27%
20%17%
26%28%
20%
10%
14%
11%8%
6%
2% 3%
39%
34%
48%
42%
34%
26%
11%
< 5 min 5-9 min 10-15 min > 15 min Don’t know
51% students overall report taking 15 minutes or less to complete survey
55% Middle School & 28% Elementary students report taking 9 minutes or less
Survey taking Experience
6% 4% 4%
12% 12% 11%
19.17%26.% 25%
64%58% 60%
Not at all true
A little true
Mostly true
Really true
Full sample: n = 1200
The majority of students find the surveys Easy to Read, Understand
and Answer• 83-85% state “mostly true” or “really true”
STUDENT OUTCOMESPre-Survey Results
1,277 surveys completed by students grades 3-8
Eight scales:1. Work Habits
2. Misconduct
3. Social Competencies
4. Math Efficacy
5. Science Efficacy
6. Interest and Engagement in STEM
7. Future Outlook
8. Science Career Aspirations
Work Habits
Full Sample Grade 3 (n = 339)
Grade 4 (n = 374)
Grade 5 (n = 356)
Grade 6 (n = 132)
Grade 7 (n =40)
Grade 8 (n = 36)
0
0.5
1
1.5
2
2.5
3
3.5
2.91 2.99 2.92.74 2.64
2.42 2.42
Mean Sc...
Work Habits—mean of 6 items Assessed on a 4-point scale (1 = not at all true, 4 = really true)Sample items include:
“I work well by myself” “I finish my work on time.”
Full Samplen = 1,277SD = .67Alpha = .81
Misconduct
Full sample
Grade 3 (n =
334)
Grade 4 (n =
364)
Grade 5 (n =
350)
Grade 6 (n =
131)
Grade 7 (n =
40)
Grade 8 (n =
36)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.44 0.440.37
0.450.5 0.51
0.66
Full Samplen = 1,255SD = .52Alpha = .85
Misconduct—mean of 9 itemsAssessed on a 4-point scale: (0 = never, 3 = more than once a week)
• Lower score means LESS misconduct Sample items include:
“I have gotten into a fight at school” “I have taken something that belongs to someone
else.”
Full sample
Grade 3 (n =
331)
Grade 4 (n =
358)
Grade 5 (n =
349)
Grade 6 (n =
131)
Grade 7 (n =
39)
Grade 8 (n = 36)
2.75
2.8
2.85
2.9
2.95
3
3.05
3.1
2.95
2.912.93
2.99 3
2.86
3.08
Mean Sc...
Full Samplen = 1,244SD = .66Alpha = .73
Social CompetenciesSocial Competencies—mean of 7 itemsAssessed on a 4-point scale: (1 = not at all true, 4 = really true)Sample items include:
“I work well with other kids” “I can tell other kids what I think, even if they disagree with me.”
Full sample
Grade 3 (n =
327)
Grade 4 (n =357)
Grade 5 (n =
349)
Grade 6 (n =
131)
Grade 7 (n =
39)
Grade 8 (n =
36)
2.5
2.6
2.7
2.8
2.9
3
3.1
3.2
3.3
3.4
3.5
3.273.32
3.37
3.28
3.13
2.812.91 Mean Sc...
Full Samplen = 1,239SD = .80Alpha = .84
Math EfficacyMath Efficacy—mean of 4 itemsAssessed on a 4-point scale: (1 = not at all true, 4 = really true)Sample items include:
“I expect to do well in math” “I am interested in math.”
Full sample
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 82.5
2.6
2.7
2.8
2.9
3
3.1
3.2
3.3
3.4
3.13 3.13
3.27
3.12
2.892.94
2.76
Mean Score
Full Samplen = 1,239SD = .80Alpha = .84
Science EfficacyScience Efficacy—mean of 4 itemsAssessed on a 4-point scale: (1 = not at all true, 4 = really true)Sample items include:
“I expect to do well in science” “I am interested in science.”
Full sample
Grade 3 (n =
312)
Grade 4 (n =
355)
Grade 5 (n =
348)
Grade 6 (n =
131)
Grade 7 (n =
39)
Grade 8 (n = 35)
0
0.5
1
1.5
2
2.5
3
3.5
2.89 2.91 2.99 2.92.74 2.64
2.42
Mean Sc...
Full Samplen = 1,220SD = .56Alpha = .93
Excited, Engaged and Interested Science Learner
[PEAR-Harvard]Excited, Engaged & Interested Science Learner—mean of 24 itemsAssessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree)
• Some items reverse codedSample items include:
“Science is something I get excited about.” “I am curious to learn more about science, computers or
technology.” “Science is boring” “I pay attention when people talk about recycling to protect our
environment.” “I enjoy visiting science museums or zoos.”
Future Outlook
Full sample
Grade 3 (n =
307)
Grade 4 (n =
354)
Grade 5 (n =
347)
Grade 6 (n =
130)
Grade 7 (n =
39)
Grade 8 (n =
35)
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.61
3.44
3.63
3.74
3.61
3.54
3.63
Mean Score
Full Samplen = 1,212SD = .56Alpha = .93
Future Outlook—mean of 7 itemsAssessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree)Sample items include:
“I will go to college.” “I will have a job that I enjoy doing.”
Science Career Aspirations
Full sample
Grade 3 (n = 302)
Grade 4 (n = 354)
Grade 5 (n = 347)
Grade 6 (n = 130)
Grade 7 (n = 39)
Grade 8 (n = 35)
2.4
2.5
2.6
2.7
2.8
2.9
3
3.1
2.92
3.03 3.04
2.81
2.68
2.79
2.65
Mean Sc...
Full Samplen = 1,207SD = .82Alpha = .85
Science Career Aspirations—mean of 4 itemsAssessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree)Sample items include:
“I will have a career in Science, Technology, Engineering, or Mathematics.”
“I will graduate with a college degree in a major area needed for a career in science.”
Next StepsInitiate Post-Survey administration: April 15, 2012-June 15, 2012
Ensure staff have received information, instructions and forms for completing Activity Documentation Forms
UC Irvine to Request sample copies of site schedule and weekly lesson plan
Final analysis of pre/post student and staff outcome measures and program implementation data
Refinement of measures for 2012-13 STEM in OST evaluation study