talking talons youth leadership evaluation report · talking talons youth leadership report school...
TRANSCRIPT
Leiden LLC
Talking Talons Youth Leadership Report
School Year 2016
Dr. Carmen Sorge
The printed document includes:
Summary of Results
Talking Talons Evaluation Report
Talking Talons Structural Equation Model Report
In addition to digital copies of the above, the CD includes:
Quiz responses
Interviews (audio and transcript)
Photographs
Contact: Dr. Carmen Sorge
1
1
Summary of results
A statistically significant change in Attitude toward Science was seen in the treatment
group and not for the control group from pretesting to post testing. F(1,74)=4.62. p<.05 η2=.06.
This represents a small effect size.
A statistically significant change in Knowledge was seen in the treatment group and not
for the control group from pretesting to post testing. F(1,74)=4.63. p<.05 η2=.45. This
represents a very large effect size.
A statistically significant change in self-reported anticipated grade in science was seen in
the treatment group and not for the control group from pretesting to post testing
F(1,71)=4.05. p<.05 η2=.06. This represents a small effect size.
The treatment group exhibited a statistically significant positive change in attitude toward
science as the program progressed. F(3,30). p<.01 η2=.37 (large effect size). This
statistic uses only the treatment group, as the control group does not take the Talking
Talon’s quizzes.
2
2
Classroom teacher feedback was extremely positive and both teachers indicated the
program increased student science knowledge and attitudes, that the buddy class was
worthwhile and also indicated a strong willingness to have the program again. Mean
scores for all subscales measuring teacher feedback were all above 6 on a 7 point scale (a
higher number indicates more positive feedback).
Qualitative feedback from students was overwhelmingly positive, mentioning that the
program was a positive experience, that they enjoyed presenting to the buddy class and
found it both rewarding and educational and far useful than their standard science
curriculum, one student summarized by stating “I think it’s uh great because it teaches us,
well real-life actual skills and things we’ll need to know in the future unlike X-Cal which
is teaching us that we will probably never use again in my life. “. They pointed out that
the buddy class consisted of “them looking up to us and them actually learning something
new from someone else, cuz them might not pick it up from like a teacher or someone,
but they can probably pick it up from us” and “they also got to experience what we
experienced with the animals” The students also were enthusiastic about both the hands
on and presentation sections of the program and found them enjoyable and educational .
The main suggestion for improvements by the students (which is consistent across every
interview) a was the inclusion of even more animals, especially ones they can touch.
Students actually requested “dangerous” animals. Students indicated that the speech
practice was helpful because it “I got nervous but it got me over my stage fright.” All
students interviewed emphatically affirmed that they would like to participate in Talking
Talons again, and would like to do so even at higher grade levels.
3
3
Structural equation model
This Structural Equation Model is a preliminary investigation of the relative importance of
the educator and the animals upon the science attitudes of the students. The full report is
provided separately.
A theoretical structural model (the Saturated Model) had a good fit to the data The model fit
for this SEM is indicates that high level of confidence in the model.
4
4
Attitude toward Educator had
A large positive direct effect on Attitude towards Animals
A medium positive indirect effect on Science Attitudes
Attitude toward Animals had
5
5
A large positive direct effect Science Attitudes
Upstream variables in the Pruned Model predicted
42% for Science Attitude
65% for Attitude toward Animals
Students who had more positive Attitudes toward Animals had large statistically
significant impact on Science Attitudes. Students who had more positive viewpoints of Attitude
toward Educator had a large direct effect and a medium indirect effect on Science Attitudes for a
total large statistically significant impact on Science Attitudes. The direct impact of the Attitude
toward Educator on Science Attitudes was not significant. Attitude towards Animals has the
largest direct impact on Science Attitudes, however the Attitude toward Animals is directly
impacted by the Attitude toward Educators. Educators are changing Science Attitudes by
changing the student’s Attitudes toward Animals. This is a complex outcome, indicating
that the animals are the crux of the change in science attitudes for the students, but that the
educators are indirectly influencing science attitudes by working through impact on
attitudes toward animals.
6
6
LEIDEN LLC
Talking Talons Youth Leadership Evaluation Report
And
Structural Equation Model for Talking Talons Science Attitudes
School Year 2016
Dr. Carmen Sorge
7
7
Table of Contents
Summary of results ......................................................................................................................... 1
Guide to basic interpretation ......................................................................................................... 14
Summary of results ....................................................................................................................... 14
Data Collection and Research ....................................................................................................... 19
Research Design ........................................................................................................................ 19
Composite instrument ............................................................................................................ 19
The Composite instrument scoring .................................................................................... 21
Reliability of the instrument .............................................................................................. 22
Quizzes .................................................................................................................................. 23
Reliability and Scale development..................................................................................... 24
Talking Talons Quiz Knowledge ....................................................................................... 25
Talking Talons Quiz Attitudes ........................................................................................... 26
Teacher Feedback Form ........................................................................................................ 28
Qualitative Data Collection ................................................................................................... 29
Participants ................................................................................................................................ 29
Explanation of GLM Repeated Measures ................................................................................. 30
Examples ............................................................................................................................ 31
Science Attitudes .......................................................................................................................... 36
Research on Science Attitudes .................................................................................................. 36
Science attitude change during program for participants .......................................................... 37
Science Attitude change from pretest to posttest by group ....................................................... 43
Student Qualitative Feedback on Science Attitudes ................................................................. 49
Classroom Teacher feedback about Science Attitudes ............................................................. 50
Hands on science experiments and Animals in the classroom ..................................................... 50
Research on Hands on Science and Animals in classroom ....................................................... 50
8
8
Classroom Teacher feedback .................................................................................................... 52
Student Feedback on Animals in classroom and Hands on science .......................................... 53
Research on peer tutoring .......................................................................................................... 54
Classroom teacher assessment of Buddy class .......................................................................... 56
Student feedback on Buddy class .............................................................................................. 57
Role Model for younger students .......................................................................................... 58
Science Knowledge ....................................................................................................................... 62
Change in Science Knowledge by Group ................................................................................. 62
Quiz Knowledge ........................................................................................................................ 66
Self-perceived science grade in school ..................................................................................... 69
Quiz Attitude subscales ............................................................................................................. 73
Feedback on Educator ................................................................................................................... 76
Student Quantitative Feedback.................................................................................................. 76
Student Qualitative Feedback on Educators.............................................................................. 80
Classroom teacher feedback for Educator ................................................................................. 81
Classroom teacher feedback for Talking Talons Program ............................................................ 83
Teacher Attitude ........................................................................................................................ 83
Effectiveness by student ability: Feedback by Classroom teachers ........................................ 85
Classroom teacher subscales ..................................................................................................... 87
Other Results ................................................................................................................................. 89
Changes in pre posttest Composite ............................................................................................... 89
Structural Equation Model ............................................................................................................ 91
Summary ....................................................................................................................................... 94
References ..................................................................................................................................... 97
SEM
What is Structural equation modeling (SEM)? ....................................................................... 107
Study Design .............................................................................................................................. 110
Purpose of the Study ............................................................................................................... 110
Research Questions ................................................................................................................. 110
9
9
Limitations of the Study .......................................................................................................... 110
Definition of Constructs .......................................................................................................... 111
Attitude Theory ....................................................................................................................... 111
Method ....................................................................................................................................... 113
Analysis Plan ........................................................................................................................... 113
Analysis Sequence................................................................................................................... 113
Fit Indices ................................................................................................................................ 114
Models ........................................................................................................................................ 116
Measurement Model ................................................................................................................ 117
Examination of Latent Variables in the Measurement Model ......................................... 119
Impact of Outliers on the Measurement Model ................................................................... 119
Final Measurement Model ............................................................................................... 120
Saturated Model ...................................................................................................................... 121
Pruned Model .......................................................................................................................... 126
Comparison of Model Fit ........................................................................................................ 130
Direct and Indirect Effects ...................................................................................................... 132
Summary of Outcomes ............................................................................................................ 132
Variance of the Latent Variables ......................................................................................... 133
Global Summary ....................................................................................................................... 133
Bibliography .............................................................................................................................. 135
10
10
Tables
Table 1 Pretest and Posttest Cronbach’s alpha for subscales ....................................................... 22
Table 2 Participants by Group ...................................................................................................... 29
Table 3 Participants by Gender .................................................................................................... 29
Table 4 Science Attitude Change over Time ................................................................................ 39
Table 5 Science Attitude Change over Time Multivariate Tests .................................................. 39
Table 6 Science Attitude Change over Time within Subjects Effects .......................................... 40
Table 7 Science Attitude Change over Time Within subjects Contrasts ...................................... 42
Table 8 Science Attitude Change over Time between Subject Effects ........................................ 43
Table 9 Change in Science Attitude from Pretesting to Posttesting by Group ............................. 43
Table 10 Change in Science Attitude by Group Multivariate Tests ............................................. 44
Table 11 Change in Science Attitude by Group Within subjects Effects ..................................... 44
Table 12 Change in Science Attitude by Group within Subjects Contrasts ................................. 46
Table 13 Change in Science Attitude by Group between Subject Effects .................................... 46
Table 14 Change in Science Attitude by Group Estimated Means ............................................... 47
Table 15 Difference Change in Science Attitude by Group ........................................................ 47
Table 16 Attitude toward Science from Quizzes Mean ............................................................... 48
Table 17 Classroom Teacher assessment of Science Attitude change ......................................... 50
Table 18 Classroom teacher assessment of program components ................................................ 52
Table 19 Classroom teacher assessment of buddy class impact .................................................. 57
Table 20 Student answers by group to "I am a good role model for younger students" .............. 59
Table 21 I am a good role model significance tests ...................................................................... 60
11
11
Table 22 Change in Science Knowledge by Group ...................................................................... 63
Table 23 Change in Science Knowledge by Group Multivariate tests ......................................... 64
Table 24 Change in Science Knowledge by Group Means ......................................................... 65
Table 25 Mean scores on Knowledge Quiz .................................................................................. 68
Table 26 Group Comparison for I will get a good grade in science class this year ...................... 70
Table 27 Group Comparison for I will get a good grade in science class this year ...................... 71
Table 28 Quiz Attitude subscale descriptive statistics ................................................................. 75
Table 29 Student feedback on Educator Subscales ....................................................................... 77
Table 30 Student feedback on Educator from Quiz results .......................................................... 79
Table 32 Classroom Teacher Feedback on Talking Talons Educator .......................................... 82
Table 33 Classroom Teacher Feedback Personal ......................................................................... 84
Table 34 Classroom teacher feedback program effectiveness by student ability ......................... 86
Table 35: Classroom teacher subscales for Talking Talons program ........................................... 88
Table 36 Statistically significant Pre Post change by group ........................................................ 89
Table 37 ANOVA Statistically significant Pre Post change by group ......................................... 90
Table 38 Effect Size Statistically significant Pre Post change by group ...................................... 91
SEM
Table 1: Fit Indices ..................................................................................................................... 115
Table 2: Factor Path Loadings for Latents .................................................................................. 119
Table 3: Measurement Model Unstandardized Results** .......................................................... 120
Table 4: Correlations Between Latent Variables in the Measurement Model ............................ 121
Table 5: Measurement Model Covariances** ............................................................................ 121
Table 6: Regression Weights for Saturated Model** ................................................................. 122
Table 7: Squared Multiple Correlations for Saturated Model .................................................... 123
12
12
Table 8: Saturated Model Total Effects Standardized ............................................................... 124
Table 9:Saturated Model Direct Effects ..................................................................................... 124
Table 10: Saturated Model Indirect Effects ................................................................................ 125
Table 11: Fit Indices for Saturated Model .................................................................................. 125
Table 12: Unstandardized Regression Weights for Pruned Model** ........................................ 128
Table 13: Squared Multiple Correlations for Pruned Model ...................................................... 128
Table 14: Total Effects for Pruned Model .................................................................................. 128
Table 15: Direct Effects for Pruned Model................................................................................. 129
Table 16: Indirect Effects for Pruned Model .............................................................................. 129
Table 17: Fit Indices for Pruned Model ...................................................................................... 130
Table 18: Comparison of the Fit of the Models .......................................................................... 130
13
13
Figures
Figure 1 Longitudinal change in Science Attitude 2015 .............................................................. 37
Figure 2 Longitudinal change in Science Attitude 2016 .............................................................. 38
Figure 3 Change in Science Knowledge by Group ....................................................................... 62
Figure 4 Quiz Knowledge section means 2015 ............................................................................ 66
Figure 5 Quiz Knowledge means 2016 ......................................................................................... 67
Figure 6 Self perceived change in science grade by group ........................................................... 70
Figure 7 Quiz Attitude subscales .................................................................................................. 73
Figure 8 Student Perception of Educator ...................................................................................... 76
Figure 9 Student Perception of Educator ...................................................................................... 78
Figure 11 Classroom teacher feedback on Talking Talons Educator ........................................... 81
Figure 12 Classroom Teacher subscales on Educator ................................................................... 82
Figure 13 Classroom teacher feedback on Talking Talons program ............................................ 83
Figure 14 Classroom teacher assessment of Effectiveness of program by student ability ........... 85
Figure 15 Classroom teacher perception ....................................................................................... 87
Figure 16 Statistically significant Pre Post change by group ....................................................... 89
Figure 17: Final pruned model of impact of student attitude toward educator and animals on
science attitudes ............................................................................................................................ 91
SEM
Figure 1: Measurement Model .................................................................................................... 118
Figure 2: Saturated Model .......................................................................................................... 122
Figure 3: Direct and Indirect Effects .......................................................................................... 124
14
14
Guide to basic interpretation
The p value tells the researcher the probability that this data would be observed simply by
chance. A p value of .05 (5 percent chance that the difference was just due to random
fluctuation) is the statistical standard for the maximum while a p value of .01 (less than 1 percent
chance the difference is due to random fluctuation) is considered more conservative and thus a
better indication of actual results.
The effect size (η2) examines the magnitude of change or how MUCH difference
occurred. The standard in use for repeated measures is .02 is a small effect size, .13 is a medium
effect size and .26 is a large effect (Bakeman).
Summary of results
A statistically significant change in Attitude toward Science was seen in the treatment
group and not for the control group from pretesting to post testing. F(1,74)=4.62. p<.05 η2=.06.
This represents a small effect size.
A statistically significant change in Knowledge was seen in the treatment group and not
for the control group from pretesting to post testing. F(1,74)=4.63. p<.05 η2=.45. This
represents a very large effect size.
15
15
A statistically significant change in self-reported anticipated grade in science was seen in
the treatment group and not for the control group from pretesting to post testing
F(1,71)=4.05. p<.05 η2=.06. This represents a small effect size.
The treatment group exhibited a statistically significant positive change in attitude toward
science as the program progressed. F(3,30). p<.01 η2=.37 (large effect size). This
statistic uses only the treatment group, as the control group does not take the Talking
Talon’s quizzes.
Classroom teacher feedback was extremely positive and both teachers indicated the
program increased student science knowledge and attitudes, that the buddy class was
worthwhile and also indicated a strong willingness to have the program again. Mean
scores for all subscales measuring teacher feedback were all above 6 on a 7 point scale (a
higher number indicates more positive feedback).
Qualitative feedback from students was overwhelmingly positive, mentioning that the
program was a positive experience, that they enjoyed presenting to the buddy class and
found it both rewarding and educational and far useful than their standard science
curriculum, one student summarized by stating “I think it’s uh great because it teaches us,
well real-life actual skills and things we’ll need to know in the future unlike X-Cal which
is teaching us that we will probably never use again in my life. “. They pointed out that
the buddy class consisted of “them looking up to us and them actually learning something
16
16
new from someone else, cuz them might not pick it up from like a teacher or someone,
but they can probably pick it up from us” and “they also got to experience what we
experienced with the animals” The students also were enthusiastic about both the hands
on and presentation sections of the program and found them enjoyable and educational .
The main suggestion for improvements by the students (which is consistent across every
interview) a was the inclusion of even more animals, especially ones they can touch.
Students actually requested “dangerous” animals. Students indicated that the speech
practice was helpful because it “I got nervous but it got me over my stage fright.” All
students interviewed emphatically affirmed that they would like to participate in Talking
Talons again, and would like to do so even at higher grade levels.
Structural equation model
This Structural Equation Model is a preliminary investigation of the relative importance of
the educator and the animals upon the science attitudes of the students. The full report is
provided separately.
A theoretical structural model (the Saturated Model) had a good fit to the data The model fit
for this SEM is indicates that high level of confidence in the model.
Attitude toward Educator had
A large positive direct effect on Attitude towards Animals
A medium positive indirect effect on Science Attitudes
Attitude toward Animals had
17
17
A large positive direct effect Science Attitudes
Upstream variables in the Pruned Model predicted
42% for Science Attitude
65% for Attitude toward Animals
Students who had more positive Attitudes toward Animals had large statistically
significant impact on Science Attitudes. Students who had more positive viewpoints of Attitude
toward Educator had a large direct effect and a medium indirect effect on Science Attitudes for a
total large statistically significant impact on Science Attitudes. The direct impact of the Attitude
toward Educator on Science Attitudes was not significant. Attitude towards Animals has the
largest direct impact on Science Attitudes, however the Attitude toward Animals is directly
impacted by the Attitude toward Educators. Educators are changing Science Attitudes by
changing the student’s Attitudes toward Animals. This is a complex outcome, indicating
that the animals are the crux of the change in science attitudes for the students, but that the
educators are indirectly influencing science attitudes by working through impact on
attitudes toward animals.
18
18
19
19
Data Collection and Research
The Talking Talons program evaluation data collection consists of:
A control/treatment pretest and posttest called the Talking Talons Composite
A series of 10 quizzes measuring knowledge and attitude change for the
participants
Online teacher feedback
Qualitative interviews of a sample of the Talking Talons participants
Research Design
Composite instrument
Talking Talons program is evaluated using a pretest and posttest quasi- experimental
design with control and treatment groups for the composite instrument. As the program is
offered by classroom, true random assignment is not feasible. However, the control groups are
selected from the same school using teachers who are not currently receiving the program.
Although ideally a Solomon design would be used, the effort and expense involved in collecting
four sets of data points is prohibitive.
The use of a control group is imperative. Without control groups any changes cannot be
attributed to the program. Factors such as maturation, the effect of testing and other outside
factors cannot be eliminated as agents of change without the use of a control group.
20
20
Improvements seen in the treatment group therefore cannot be ascribed to the program unless a
control group is utilized.
For the composite instrument a General Linear Model Repeated measures utilizing SPSS
18 was run. The GLM repeated measures improves on the multivariate regression model by
allowing for linear transformations or linear combinations of multiple variables. This expansion
means that the GLM has important advantages over the multiple and the purported multivariate
regression models which are inherently univariate methods. The first advantage is that
multivariate test of significance may be use if the responses on multiple dependent variables are
correlated. This is helpful as separate univariate tests of significance for correlated dependent
variables (such as used in multiple t tests) are not independent and may not be appropriate.
Multivariate tests of significance of independent linear combinations of multiple dependent
variables may also yield information about which response variables are, and are not, actually
related to the predictor variables. A second advantage is the ability to analyze effects of repeated
measure factors. Linear combinations of responses reflecting a repeated measure effect can be
constructed and tested for significance using either the univariate or multivariate approach to
analyzing repeated measures in the general linear model. In this research, with pretest and
posttest measurements of control and treatment groups the GLM repeated measures is the most
appropriate test.
Effect sizes based on recent research on repeated measure designs based on Cohen (1988,
pp. 413–414), who did not consider repeated measures designs explicitly, defined an η2 of .02 as
small, one of .13 as medium, and one of .26 as large. “It seems appropriate
21
21
to apply the same guidelines to η2 as well.” (Bakeman)
The Composite instrument scoring
All composite questions are scored on a Likert scale from 1 to 6 ranging from Very False
to Very True. All questions and scales are renormed so that a higher value indicates a more
desirable outcome. Therefore a value of 5 indicates better self-control or less violent tendencies
than a score of 3. For all subscales a higher score is more desirable.
The Locus of Control measure reports a student’s self-reported ability to direct and
manage their own behavior. Ten questions addressing these issues such as “I do things I know
are wrong because my friends are doing them.” and “I am easily distracted.” are included.
The School Attitude scale consists of five self-reported feelings about school such as
“The work I do in school is important to me” and “I would skip school a lot if I knew that I
would not get caught”.
The Self-esteem attitude scale consists of twelve self-assessments of the student’s self-
value and perception of ability such as “I am proud of myself.” And “If I want to learn to do
something new I usually can learn it.”
The Attitude toward Violence scale consists of five self-reported actions or intentions of
violence such as “I fight a lot.” and “If people will not do what you want it is okay to threaten
that you will hurt them.” Questions 9, 11, 25, 42 and 20 from the composite instrument are
included in this subscale.
22
22
The Moral Judgment scale consists of eleven questions evaluating student attitudes
toward various ethical and unethical behaviors such as “It is okay to cheat on schoolwork if you
do not get caught.” And “It is wrong to lie in order to get what you want.”
Seventeen questions for the treatment group only are asked of the treatment
group. These questions address the participant’s evaluation of the program and the educator and
include questions such as “I looked forward to the days that Talking Talons came to my
classroom” and “My Talking Talons educator was easy to understand”. A full report of these
questions is included in the educator report.
Reliability of the instrument
Cronbach’s alpha was calculated for each subscale using the overall sample. Reliability
greater than .7 is considered good (Nunnaly, 1978). All subscales of the instrument have good to
excellent reliability. The 2015 and 2016 evaluations have a much smaller sample size than
previous years, which does impact reliability. School Attitude reliability fell below the
reasonable interpretation threshold and thus any significant results are disregarded.
Table 1 Pretest and Posttest Cronbach’s alpha for subscales
2003
-
2004
2003
-
2004
2004
-
2005
2004
-
2005
2005
-
2006
2005
-
2006
2006
-
2007
2006
-
2007
2015*
2015
2016 *
2016
Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post
Locus of
Control
.797 .807 .78 .83 .79 .81 .81 .82 .78 .76 .80 .73
23
23
2003-
2004
2003-
2004
2004-
2005
2004-
2005
2005-
2006
2005-
2006
2006-
2007
2006-
2007
2015*
2015
2016 *
2016
School
Attitude
.750 .779 .75 .75 .79 .78 .71 .74 .65 .52 .67 .78
Self
Esteem
.666 .798 .81 .84 .83 .86 .82 .86 .91 .87 .89 .88
Attitude
toward
Violence
.717 .747 .73 .73 .80 .85 .77 .83 .77 .71 .78 .81
Moral
Scale
.776 .799 .83 .84 .86 .86 .80 .86 .70 .87 .81 .82
Science
Attitude
Scale
.957 .965 .95 .97 .96 .97 .94 .95 .90 .94 .95 .96
Talking
Talons
Scale
.680 .782 .76 .80 .71 .89 .77 .76 .77 .71 .71
Posttest
only
Treatmen
t section
.94 .92 .76 .73 .73
*note: Sample was much smaller than previous years, this tends to reduce reliability
Quizzes
The treatment group students are also given a series of ten quizzes. These quizzes consist
of five knowledge questions and five attitude questions. The knowledge questions pertain to the
information taught by the program and the attitude questions cover attitudes towards the
program, the educator and science in general. Some science attitude questions are repeated
24
24
throughout the attitude section of the quizzes over the course of the program in order to measure
longitudinal changes in attitude scores.
These quizzes are used to determine both the changes in science knowledge and in attitudes
over the course of the program.
Reliability and Scale development
The quizzes were used to generate several subscales. For each subsection the internal
reliability was examined. Internal reliability is how much consistency is apparent in the answers
of one subtest. Internal consistency is measured with a scale called “Cronbach's Alpha”.
Technically speaking, Cronbach's alpha is not a statistical test - it is a coefficient of reliability (or
consistency). Alpha is calculated by one split-half reliability and then randomly dividing the
items into another set of split halves and recomputing until, all possible split half estimates of
reliability have been computed. In other words, each item is compared to the group in all
possible combinations of items. Cronbach's alpha measures how well a set of items (or variables)
measures a single unidimensional latent construct. When data have a multidimensional structure
(they are not all measuring the same construct) then Cronbach's alpha will usually be low. If the
number of items is increased and the consistency remains the same the Cronbach’s alpha will
increase. Additionally, if the average inter-item correlation is low, alpha will be low. As the
average inter-item correlation increases, Cronbach's alpha will also increase. This effect makes
sense intuitively - if the inter-item correlations are high, and then there is evidence that the items
are measuring the same underlying construct.
25
25
Recall that an alpha value of .70 is considered acceptable; however we would like it to be
higher. Reliabilities are presented in each section of the report.
Talking Talons Quiz Knowledge
All of the scales are mean scores. Thus if an educator did not give one of the quizzes
containing a question for that subset it is not entered into the mean. This can impact the results
because the score is then based on fewer data points. All attitude scales are recoded so that 6
represents the highest and best score and 1 is the lowest and worst score. For the quizzes 5 is the
highest possible score and 0 is the lowest. Remember that this is a very simple way at looking at
scores. With simple means one participant who has a negative change of 6 point will mask the
positive change of 1 point for 6 participants. This is especially true for small sample sizes (the
smaller classrooms). An educators with a score of 5.5 on the scale could have had half the kids
answer “6” and half answer “5” OR almost all kids answering 6 with a few answering “1”. Keep
this in mind when interpreting results.
Bar charts are good visual presentations of information. However, they may lead to over
interpretation. As the standard deviations are not listed on a bar chart it is difficult to ascertain
true significant differences. One must ask “Does that “3” on a bar chart mean that almost
everyone answered 3 or that half the participants answered “5” and half answered “2”? “
Significance tests indicate actually differences by utilizing the standard deviation as well.
26
26
Therefore two bars on a chart may look very different and not be statistically different or may
look similar but actually represent a statistical difference.
The knowledge sections relate to the topics studied in the program and includes the following
topics Introduction, Hawks, Raptors, Bats 1, Bats 2, Owls, Reptiles,Vocabulary General and
Energy. Five questions related to these topics are asked on each of the quizzes.
Talking Talons Quiz Attitudes
The back of each quiz includes an attitude section; these questions are then utilized for the
following subscales.
Overall perception of educator (Reliability α=.90 (2015) α=.86 (2016))
My Talking Talons educator (teacher) is easy to understand.
My Talking Talons educator (teacher) understands how kids think.
My Talking Talons educator (teacher) knows a lot about the animals.
My Talking Talons educator (teacher) speaks clearly and loudly enough for me to hear
and understand.
My Talking Talons educator (teacher) pays attention to all of the kids in my class.
My Talking Talons educator (teacher) treats me kindly.
My Talking Talons educator (teacher) is prepared to teach us when he/she comes to class.
27
27
My Talking Talons educator (teacher) was fair to everyone in my class.
Bonding with Animals (Reliability α=.68 (2015) α=.67 (2016)))
In the future, I would like to be a Talking Talons Educator and hold a bird.
I would like the chance to hold a Talking Talons reptile (like a snake or lizard).
I think about the Talking Talons animals when I am not in science class
I feel comfortable around the Talking Talons animals (eliminated due to small sample
size)
Seeing the Talking Talons animals makes me feel happy.
Perceived Enjoyment of Talking Talons program (Reliability α=.78(2015) α=.78 (2016)))
I like learning about animals.
I look forward to the days that Talking Talons comes to the classroom
I talk to kids who are not in the Talking Talons program about the program.
I would like the Talking Talons program to come to my science class next year.
Perceived Ability (Reliability α=.72 (2015) α=.62 (2016)))
The quizzes for Talking Talons were very hard.
I try my hardest to do well in school.
I am good at science.
28
28
I think I will get a better grade in science class this year than last year.
I understood most of the information in the Talking Talons program.
Attitude toward science (Reliability α=.82 (2015) α=.90 (2016)))
I like science class.
Science class is boring
I like science class.
Attitude toward environment (Reliability α=.68(2015) α=.62 (2016))
It is ok to shoot hawks because they kill smaller birds.
It is ok to throw trash out the window because the highway department picks it up.
I think teaching other kids about the environment is important
Teacher Feedback Form
At the end of the program an online form (to preserve teacher anonymity) is used to
collect data on the classroom teachers’ perceptions of the program, of changes in attitude of
his/her students and of the educator.
29
29
Qualitative Data Collection
A sample of five to six students from each group was interviewed by the evaluator for
feedback about the Talking Talons program. These interviews were recorded and transcribed.
Student names were removed from the transcription to preserve anonymity.
Participants
Participant demographic information is included below. For each subscale the total
number of student varies slightly due to outliers and participants who did not finish a
particular subscale. The total number can be found in the table of means for each subsection.
Because of the small sample size for the 2015 and 2016 program, missing posttest data for
the control group was imputed from representative previously collected control test data in
order to reach a statistically useable sample size. Students were all between 10 and 11 years
old at post testing.
Table 2 Participants by Group
Frequency Percent Valid Percent Cumulative Percent
Valid Control 40 44.0 44.0 44.0
Treatment 51 56.0 56.0 100.0
Total 91 100.0 100.0
Table 3 Participants by Gender
group Total
30
30
Control Treatment
gender Female 20 26 46
Male 20 25 45
Total 40 51 91
Explanation of GLM Repeated Measures
The data for the composite file were analyzed using SPSS General Linear Model
Repeated Measures (GLM). The GLM Repeated Measures procedure is based on the general
linear model, in which factors and covariates are assumed to have linear relationships to the
dependent variables.
This section explains how to interpret the graphs and data from the analysis of the
composite section. For simplicity of explanation sample graphs and data are used which make
the differences clear for interpretation.
A GLM repeated measures with control groups is used to examine the following possibilities:
QUESTION: Are the control and treatment groups similar to each other?
VISUAL ANSWER: Are the two lines far apart (be sure to look at the scale of the graph)
from each other, especially at pretesting?
STATISTICAL ANSWER: If the two groups are not similar to each other than the test
of between groups will be significant.
31
31
QUESTION: Does the score on the instrument change over time from pretesting to post testing
for one or both of the groups?
VISUAL ANSWER: Is the line flat or sloped for the groups?
STATISTICAL ANSWER: If the scores are changing overall in the same direction and
magnitude then the “Time” variable will be significant.
QUESTION: Is the change over time different for the two groups? (Either one group goes up
and one goes down or the magnitude of the change is different.)
VISUAL ANSWER: Is the slope of the line different (in direction or magnitude) for
each group?
STATISTICAL ANSWER: If the scores are changing, but differently for each group
then the “Time by Group” variable will significant overall. The difference can be either
in different directions or in different magnitudes.
These questions are answered statistically in the tables. However, for ease of interpretation the
statistical significance is listed under the graph showing change.
Examples
Groups are different but with no change over time
32
32
The first question is answered statistically in a table called “Between groups’ differences”
but can also be seen by the distance between the score values of the control and treatment
groups. Note the scale on the graph as well when interpreting the data.
The first graph shows the following
No change over time for either group (the slope of the lines are flat)
Groups are different from each other (there is a big gap between the Treatment and
Control Group lines)
No Change over Time
Groups are different
TIME
21
Estim
ate
d M
arg
ina
l M
ea
ns
4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
.5
GROUP
Treatm ent
Control
33
33
Groups are similar and both change over time in the same way
The second graph shows the following
both change from pretesting to post testing (the slope of the line is not flat)
Both change in the same manner (the slope is in the same direction and at the same
angle).
Change over Time
TIME
21
Estim
ate
d M
arg
ina
l M
ea
ns
7.0
6.5
6.0
5.5
5.0
4.5
4.0
3.5
GROUP
Treatm ent
Control
Groups are the same but change differently over time
34
34
The third graph shows
change over time for both groups
Change is not in the same direction.
Different Changes Over Time
TIME
21
Estim
ate
d M
arg
ina
l M
ea
ns
6
5
4
3
2
1
0
GROUP
Treatm ent
Control
Obviously with real data the differences are not always so clear. However, this
explanation should make visual interpretation of the results simpler. Statistical results are
presented in the tables below the charts for those who are interested in the means, levels of
significance, effect size and power for the results. Type III Sums of squares were used as the
35
35
hypotheses being tested involve only marginal averages of population cell means. Greenhouse-
Geisser epsilon was used to adjust degrees of freedom if sphericity was violated as per
assessment with Mauchly's test. These statistics are available from the researcher if desired.
Estimated marginal means of the dependent variables (with covariates held at their mean value)
for specified between- or within-subjects factors in the model are provided. In this research these
predicted means are equivalent to observed means as the covariates are categorical.
Outlying scores due to extreme responses may influence results. In order to identify
outliers the score distributions for the participants for all observed variables were examined
univariately. Scores that were more than three standard deviations from cell means and were
also discontinuous from their closest neighboring scores were considered univariate outliers and
were removed on a subscale by subscale basis.
For each subsection the results were examined by GLM repeated measures using the
following group memberships.
Overall
By gender
The results are then presented for results that yielded significant or interesting result for
the difference between the groups if there was no difference then overall results are presented
unless a trend that merits consideration was present. Recall that as the group is subdivided the
membership within that group becomes smaller and significance is more difficult to detect.
36
36
Science Attitudes
Research on Science Attitudes
Attitudes toward science have long been examined as an important factor in student
science education. Researchers have found that attitudes toward science impact future course
selection (Farenga and Joyce), motivation to study science (Slate and Jones), achievement in
science (Oliver and Simpson; Kose et al.; Smith, Pasero, and McKenna; Baker; Oliver;
Reynolds and Walberg; Warburton, Jenkins, and Coxhead; Willson) and general attitudes toward
school (Jarvis and Pell) ). Improving student’s attitudes toward science could therefore be said
to have many positive outcomes.
However as students enter the middle school years, their generally positive attitudes
about science decrease (Mattern and Schau; Desy, Peterson, and Brockman; Catsambis;
Mattern; Sorge) especially for girls (Backes; Lee and Burkam; Oliver; Papanastasiou and
Zembylas; Warburton, Jenkins, and Coxhead) .
Of particular relevance is research examining changes in attitudes exhibited after
exposure to “hands on” or inquiry based science. These programs have been found to improve
attitudes toward science with increased frequency of hands on experiments leading to most
positive attitudes (Ornstein) and with hands on laboratory experiment increasing knowledge
(Freedman). Interestingly, girls were found to be impacted more (Teshome, Maushak, and
Athreya) by hands on experimentation than boys.
37
37
Specific research on involving students in environmental science projects in order to
increase both knowledge and attitudes toward science has statistically positive outcomes on both
factors (Al-Balushi and Al-Aamri).
Science attitudes for the Talking Talons students were examined in multiple forms.
Pretest and posttest attitudes were compared for control and treatment groups. For the treatment
groups the same question about attitudes towards science was asked throughout the program in
order to measure longitudinal change. Information was collected from the classroom teacher and
from interviews.
Science attitude change during program for participants
In order to examine longitudinal changes in science attitude for the treatment group
(those students who participated in the Talking Talons program), science attitude questions were
asked at pretesting, on quiz 3, on quiz 7 and at posttesting. The chart below delineates the
change over time. This change was statistically significant (see table for results)
Figure 1 Longitudinal change in Science Attitude 2015
38
38
Figure 2 Longitudinal change in Science Attitude 2016
39
39
Table 4 Science Attitude Change over Time
Descriptive Statistics
Mean Std. Deviation N
Science is a topic which I enjoy studying. 4.818 1.1580 33
I like science class. (Quiz 3) 5.36 .822 33
I like science class. (Quiz 7) 5.39 .864 33
POST Science is fun 5.52 .667 33
Table 5 Science Attitude Change over Time Multivariate Tests
40
40
Multivariate Testsa E
ffec
t
Val
ue
F
Hypoth
esis
df
Err
or
df
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
erc
Attitude
change
Pillai's Trace .372 5.924b 3.000 30.00 .003 .372 17.773 .927
Wilks'
Lambda
.628 5.924b 3.000 30.00 .003 .372 17.773 .927
Hotelling's
Trace
.592 5.924b 3.000 30.00 .003 .372 17.773 .927
Roy's Largest
Root
.592 5.924b 3.000 30.0 .003 .372 17.773 .927
Table 6 Science Attitude Change over Time within Subjects Effects
Tests of Within-Subjects Effects
41
41
Source
Typ
e II
I S
um
of
Sq
uare
s
df
Mea
n S
qu
are
F
Sig
.
Part
ial
Eta
Sq
uare
d
Non
cen
t. P
ara
met
er
Ob
serv
ed P
ow
era
Attitude
Change
Sphericity
Assumed
9.515 3 3.17
2
8.70
3
.000 .214 26.11
0
.993
Greenhouse
-Geisser
9.515 2.465 3.86
0
8.70
3
.000 .214 21.45
3
.983
Huynh-
Feldt
9.515 2.686 3.54
2
8.70
3
.000 .214 23.38
0
.989
Lower-
bound
9.515 1.000 9.51
5
8.70
3
.006 .214 8.703 .816
Error(Attitud
e Change)
Sphericity
Assumed
34.98
5
96 .364
Greenhouse
-Geisser
34.98
5
78.87
7
.444
Huynh-
Feldt
34.98
5
85.96
4
.407
Lower-
bound
34.98
5
32.00
0
1.09
3
42
42
a. Computed using alpha = .05
Table 7 Science Attitude Change over Time Within subjects Contrasts
Tests of Within-Subjects Contrasts
Measure: MEASURE_1
Sourc
e
Att
itude
Chan
ge
Type
III
Sum
of
Squar
es
df
Mea
n S
quar
e
F
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
era
Attitude
Change
Linear 7.424 1 7.42
4
16.18
8
.00
0
.33
6
16.18
8
.97
4
Quadrati
c
1.485 1 1.48
5
4.314 .04
6
.11
9
4.314 .52
2
Cubic .606 1 .606 2.087 .15
8
.06
1
2.087 .28
9
Error(Attitud
e Change)
Linear 14.67
6
3
2
.459
Quadrati
c
11.01
5
3
2
.344
Cubic 9.294 3
2
.290
a. Computed using alpha = .05
43
43
Table 8 Science Attitude Change over Time between Subject Effects
Tests of Between-Subjects Effects
Measure: MEASURE_1
Transformed Variable: Average
Source
Type
III
Sum
of
Squar
es
df
Mea
n S
quar
e
F
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
era
Intercept 3669.818 1 3669.818 1735.092 .000 .982 1735.092 1.000
Error 67.682 32 2.115
a. Computed using alpha = .05
Science Attitude change from pretest to posttest by group
Both the treatment and control group were assessed for science attitude changes from
pretesting to postttesting. A statistically significant change in Attitude toward Science was seen
in the treatment group and not for the control group from pretesting to post testing. F(1,74)=4.62.
p<.05 η2=.06. This represents a small effect size.
Table 9 Change in Science Attitude from Pretesting to Posttesting by Group
44
44
Descriptive Statistics
group Mean Std. Deviation N
sci Control 4.3137 1.24026 38
Treatment 5.0816 .90460 38
Total 4.6977 1.14539 76
psci Control 4.0690 1.32331 38
Treatment 5.2842 .70731 38
Total 4.6766 1.21853 76
Table 10 Change in Science Attitude by Group Multivariate Tests
Multivariate Testsa
Effect
Val
ue
F
Hypoth
e
sis
df
Err
or
df
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
e
r Obse
rve
d P
ow
erc
science Pillai's Trace .001 .041b 1.000 74.000 .840 .001 .041 .055
Wilks' Lambda .999 .041b 1.000 74.000 .840 .001 .041 .055
Hotelling's Trace .001 .041b 1.000 74.000 .840 .001 .041 .055
Roy's Largest
Root
.001 .041b 1.000 74.000 .840 .001 .041 .055
science *
group
Pillai's Trace .059 4.623b 1.000 74.000 .035 .059 4.623 .564
Wilks' Lambda .941 4.623b 1.000 74.000 .035 .059 4.623 .564
Hotelling's Trace .062 4.623b 1.000 74.000 .035 .059 4.623 .564
Roy's Largest
Root
.062 4.623b 1.000 74.000 .035 .059 4.623 .564
a. Design: Intercept + group
Within Subjects Design: science
b. Exact statistic
c. Computed using alpha = .05
Table 11 Change in Science Attitude by Group Within subjects Effects
45
45
Tests of Within-Subjects Effects
Measure: MEASURE_1
Sourc
e
Type
III
Sum
of
Squar
es
df
Mea
n S
quar
e
F
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
era
science Sphericity
Assumed
.017 1 .017 .041 .84
0
.00
1
.041 .05
5
Greenhous
e-Geisser
.017 1.000 .017 .041 .84
0
.00
1
.041 .05
5
Huynh-
Feldt
.017 1.000 .017 .041 .84
0
.00
1
.041 .05
5
Lower-
bound
.017 1.000 .017 .041 .84
0
.00
1
.041 .05
5
science *
group
Sphericity
Assumed
1.901 1 1.90
1
4.62
3
.03
5
.05
9
4.62
3
.56
4
Greenhous
e-Geisser
1.901 1.000 1.90
1
4.62
3
.03
5
.05
9
4.62
3
.56
4
Huynh-
Feldt
1.901 1.000 1.90
1
4.62
3
.03
5
.05
9
4.62
3
.56
4
Lower-
bound
1.901 1.000 1.90
1
4.62
3
.03
5
.05
9
4.62
3
.56
4
Error(scienc
e)
Sphericity
Assumed
30.43
6
74 .411
Greenhous
e-Geisser
30.43
6
74.00
0
.411
Huynh-
Feldt
30.43
6
74.00
0
.411
Lower-
bound
30.43
6
74.00
0
.411
a. Computed using alpha = .05
46
46
Table 12 Change in Science Attitude by Group within Subjects Contrasts
Tests of Within-Subjects Contrasts
Measure: MEASURE_1
Sourc
e
scie
nce
Type
III
Sum
of
Squar
es
df
Mea
n S
quar
e
F
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
era
science Linear .017 1 .017 .041 .840 .001 .041 .055
science * group Linear 1.901 1 1.901 4.623 .035 .059 4.623 .564
Error(science) Linear 30.436 74 .411
a. Computed using alpha = .05
Table 13 Change in Science Attitude by Group between Subject Effects
Tests of Between-Subjects Effects
Measure: MEASURE_1
Transformed Variable: Average
47
47
Sourc
e
Type
III
Sum
of
Squar
es
df
Mea
n S
quar
e
F
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
era
Intercept 3339.323 1 3339.323 1764.335 .000 .960 1764.335 1.000
group 37.358 1 37.358 19.738 .000 .211 19.738 .992
Error 140.058 74 1.893
a. Computed using alpha = .05
Table 14 Change in Science Attitude by Group Estimated Means
group * science
Measure: MEASURE_1
group science Mean Std. Error 95% Confidence Interval
Lower Bound Upper Bound
C 1 4.314 .176 3.963 4.665
2 4.069 .172 3.726 4.412
T 1 5.082 .176 4.731 5.432
2 5.284 .172 4.941 5.627
Table 15 Difference Change in Science Attitude by Group
48
48
Student data from quizzes.
Table 16 Attitude toward Science from Quizzes Mean
group * science
Measure: MEASURE_1
group science Mean Std. Error 95% Confidence Interval
Lower Bound Upper Bound
49
49
Control Pre 4.314 .176 3.963 4.665
Post 4.069 .172 3.726 4.412
Treatment Pre 5.082 .176 4.731 5.432
Post 5.284 .172 4.941 5.627
Student Qualitative Feedback on Science Attitudes
Interview sections pertaining to science attitudes for the treatment group are excerpted
below. The full transcript is also included in the evaluation information.
Students found the Talking Talons more enjoyable and educational when compared to the
standard science curriculum, which is called X-Cal.
Well uh, it’s definitely uh more educating, We have uh, X-Cal, which is kind of like
science, once a week and we have this two times so it’s probably double that.
Um, it’s really helping us because it explains more.
think it’s because, in X-Cal like Darren said, uh some of, Talking Talons doesn’t, uh
covers more than what X-Cal does like, we can actually use in the rest of our lives.
Um, I think Talking Talons is good cuz they let us experience like we’re gonna do a
presentation and they let us experience like, like if we were talking to actual people and
holding animals and all that stuff.
Uh I think it’s uh great because it teaches us, well real-life actual skills and things we’ll
need to know in the future unlike X-Cal which is teaching us that we will probably never
use again in my life.
When asked how Talking Talons compares to their regular science program.
It’s a lot better, Yeah.
50
50
Yeah we learn more about science and stuff. It’s more hands on too. Much-Yeah. I
prefer- Much more interesting, as in it puts, it makes it more fun than it usually is for me.
Classroom Teacher feedback about Science Attitudes
The classroom teachers indicated that they perceived that the student’s both learned a lot
of science from the program (mean score 6.5/7) and that the program had a positive impact on
student attitudes toward science (mean score 6.5/7).
Table 17 Classroom Teacher assessment of Science Attitude change
The students learn a lot of
science from the TT program.
The TT program had a positive impact on
the student’s attitudes about science.
Mean 6.5000 6.50
Std.
Deviation
.70711 .707
Hands on science experiments and Animals in the classroom
Research on Hands on Science and Animals in classroom
Significant contribution of hands on learning to science knowledge has been identified as
making a significant contribution to peer interaction through cooperative learning, object-
51
51
mediated learning and embodied experience. (Satterthwait). The more hands on experience, the
more learning occurs “Specifically, students who engaged in hands-on activities every day or
once a week scored significantly higher on a standardized test of science achievement than
students who engaged in hands-on activities once a month, less than once a month, or never.”
(Stohr-Hunt)
When compared to textbook learning, hands on (inquiry) also improves achievement
“Students in the inquiry-based group reached significantly higher levels of achievement than
students experiencing commonplace instruction. This effect was consistent across a range of
learning goals (knowledge, reasoning, and argumentation) and time frames (immediately
following the instruction and 4 weeks later).” (C. D. Wilson et al.)
Hands on learning is more effective for student comprehension even when compared to
teacher demonstration “students in the hands- on laboratory class performed significantly better
on the procedural knowledge test than did students in the teacher demonstration class. These
results were unrelated to reasoning ability.” (Glasson). This impact has been shown to influence
traditionally underrepresented minorities attitudes toward science and career plans. (Kanter and
Konstantopoulos). This is especially important as the impact of No Child Left behind has led to
less teaching of science in the classroom. (Milner et al.)
Furthermore, the use of animals in the classroom has been found to teach humane values
(Zasloff et al.) while improving young students attitudes toward animals (Ellery Samuels, Nicoll,
and Trifone) and towards other humans (Arkow) . The use of animals also improves
52
52
environmental education for young children (Margadant-Van Arcken) and knowledge about
animals specifically (Kimble).
The specific use of more exotic animals (zoo outreach) found that students recorded more
observations, made more use of science facts and use more science vocabulary when writing
about more exotic animals rather than more ordinary pets. (Trainin et al.) and also used more
advanced literary concepts (K. Wilson et al.)
Classroom Teacher feedback
Classroom teachers gave the maximum score possible (7/7) to all factors relating to the
effectiveness of the hands on and animal sections of the Talking Talons program.
Table 18 Classroom teacher assessment of program components
The animals
are an
important part
of the TT
program for
the students.
The public
speaking
component of
TT is an
important part of
the program.
The hands on
activities are
an important
part of the TT
program.
Presenting to the
Buddy class made
the students take the
responsibility of TT
more seriously.
Mean 7.00 7.00 7.00 7.00
Std.
Deviation
.000 .000 .000 .000
53
53
Student Feedback on Animals in classroom and Hands on science
Sections of the interview pertaining to the animals and hands on learning are excerpted below.
The students clearly found the animals to be an important aspect of the program, they universally
ask for more touchable animals.
Um, one thing that I liked is that they let us hold snakes and taught us about like, uh the
diseases and stuff. A thing I think they could improve in bringing more touchable
animals, I guess, because a lot of the times I have half the mind to just steal Hyde, the
owl I am going to be performing on today.
I really like how like uh their teaching us like if one day like we get lost in a forest and
there’s a snake like we can know that it’s poisonous or not poisonous. And I think they
could improve by, yeah having more touchable animals cuz like whenever we see that
birds we all wanna them.
I really like that they allow us to hold these snakes and show us like what to do if like
you see a snake or if you see a bird um injured, you know what to do, instead you just see
a bird injured and like go against the rules.
And you know, and that’s what I like about it, but I think that we can improve on like all
of them said is bringing in more, more touchable animals.
I like the activities because it actually gives us a more hands on experience of what we
are actually doing and what the animal does.
I actually like the activities better because it gives us more of an experience about what’s
around us in the world and stuff I guess.
54
54
I would change the fact that they don’t allow us to, to, to, see the more dangerous
animals. As in like, as in like rattlesnakes. I thought that, that they could do it by putting
it into a glass container, so that it won’t be able to damage us.
Well I, I was gonna say the same thing that he did but um, what I wanted them to do
more was that they should bring in more dangerous animals of course.
Was that I like how they taught us how to present it and, to let us touch the snakes and
lizards and such like that. And they taught us, taught me more about who’s endangered
and the continents and kind of that, yeah.
The same but like you know, if uh, if we would be able to hold the birds that would be
pretty cool.
It’s um, with the bats. Um, I felt like they’re not really that dangerous of animals because
even the vampire bats they don’t really suck any human blood, they do animal blood, as
in like cows and pigs, for animals that have lots of blood that they can spare, that they can
spare so I felt like that the bat was, I thought, I thought we would be able to like hold it.
Research on peer tutoring
One aspect of the Talking Talons program is the peer interaction. Students in the Talking
Talons program learn the material with the goal of presenting to a younger group of students in
the school.
55
55
A meta-analysis of peer assisted learning found increases in achievement. Peer tutoring is
most effective with “PAL interventions were most effective with younger, urban, low-income,
and minority students.”(Rohrbeck et al.).
Peer tutoring helps reading fluency (Kamps et al.), reading comprehension (McMaster,
Fuchs, and Fuchs) mathematics learning (Fantuzzo, Polite, and Grayson) both mathematics and
social skills (Pigott, Fantuzzo, and Clement) English (Greenwood et al.) and social interaction
for rejected boys (Gumpel and Frank).
Meta analyses on impact on math found the effect is particular strong for elementary aged
students (Kunsch, Jitendra, and Sood). A hands on gardening program found improvement in
social relationships in the classroom (Kim, Park, and Son).
Specifically buddy lesson plans for healthy eating and exercise physical activity, healthy
eating, and self-esteem and body image where lesson plans were delivered by 9-12 year olds for
6 to 8 year olds found that “Reductions in waist circumference were particularly significant for
children who were younger, overweight or obese, or attending First Nations schools. No
difference in body mass index score was observed between groups. Self-efficacy, healthy living
knowledge, and dietary intake significantly improved in younger peers who received the
intervention compared with students from control schools.” (Santos et al.).
Meta-analysis also shows that this impact is positive for both the “teacher” and the
“student” in peer tutoring situations. “This review of the literature on peer and cross-age tutoring
emphasizes programs in mathematics and suggests that such programs have positive academic
outcomes for African American and other minority students as well as for White students who
56
56
participate as tutors, as tutees, or both”. This results was shown even with short programs
“tutors show academic gains even when they do not receive additional subject matter instruction,
why longer and/or more substantial tutoring programs may not foster greater immediate
academic gains than shorter programs,” (Robinson, Schofield, and Steers-Wentzell)
Teaching benefits the tutor specifically in science classes “Mental rehearsal of peer-tutoring
episodes helped them appreciate weaknesses in their own subject knowledge.” (Galbraith and
Winterbottom)
Meta analyses also supports that peer tutoring is effective regardless of many factors.
“This meta-analysis examined effects of peer tutoring across 26 single-case research experiments
for 938 students in Grades 1–12. The Tau effect size for 195 phase contrasts was 0.75 with a
confidence interval of 0.71 to 0.78, indicating that moderate to large academic benefits can be
attributed to peer tutoring. Five potential moderators of these effects were examined: dosage,
grade level, reward, disability status, and content area. This is the first peer tutoring meta-
analysis in nearly thirty years to examine outcomes for elementary and secondary students, and
extends previous peer tutoring meta-analyses by examining disability as a potential moderator.
”Findings suggest that peer tutoring is an effective intervention regardless of dosage, grade level,
or disability status. Among students with disabilities, those with emotional and behavioral
disorders benefitted most.” (Bowman-Perrott et al.)
Classroom teacher assessment of Buddy class
The classroom teachers all agreed very strongly (mean of 7/7) that having the buddy
class had a positive impact on their students.
57
57
Table 19 Classroom teacher assessment of buddy class impact
Presenting to the Buddy class made the students take the responsibility of TT more seriously.
N Valid 2
Missing 0
Mean 7.00
Std. Deviation .000
Student feedback on Buddy class
Students universally expressed that they enjoyed “being the teacher” and presenting to
younger students. They mentioned leadership directly as well. Students were also irriated by the
fact that the younger students were not always quiet, this concern stemmed from the Talking
Talons students taking seriously that the animals were disturbed by excess noise. Sections related
the buddy class are excerpted from interviews below.
Um but one thing that was really nice about it is that they also got to experience what we
experienced with the animals. So yeah
Also they talked a lot during the presentations and they wouldn’t follow much of the
rules that we put them out, so they would scare the animals a lot.
Um I really like that they can um look up to us as their leaders and we can like educate
them and it was also kind of annoying cuz they did scare the animals a lot.
58
58
I kinda like how, them looking up to us and them actually learning something new from
someone else, cuz them might not pick it up from like a teacher or someone, but they can
probably pick it up from us. But then again they do kinda scare the animals.
Uh, it was a good thing for, to help us present that to other people because again at the
end we had to present to at least a hundred.
I, the buddy class was OK for me so like it prepared me for the presentations. Yes.
Role Model for younger students
The change in role model was not significant, the power was low due to sample size.
However, the direction of change was positive for the treatment group and negative for the
control group. This trend was seen in the 2015 data as well. This information is included for
trends, rather than statistical significance.
I am a good role model for younger students 2016
59
59
Table 20 Student answers by group to "I am a good role model for younger students"
Descriptive Statistics
group Mean Std. Deviation N
I am a good role model for younger students. Control 4.472 1.4038 36
Treatment 4.611 1.2935 36
Total 4.542 1.3420 72
POST I am a good role model for younger students. Control 4.31 1.411 36
60
60
Treatment 4.86 1.175 36
Total 4.58 1.319 72
Table 21 I am a good role model significance tests
Multivariate Testsa
Eff
ect
Val
ue
F
Hypoth
esis
df
Err
or
df
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
erc
Role Model Pillai's Trace .001 .052b 1.000 70.000 .821 .001 .052 .056
Wilks' Lambda .999 .052b 1.000 70.000 .821 .001 .052 .056
Hotelling's
Trace
.001 .052b 1.000 70.000 .821 .001 .052 .056
Roy's Largest
Root
.001 .052b 1.000 70.000 .821 .001 .052 .056
Role Model *
group
Pillai's Trace .018 1.289b 1.000 70.000 .260 .018 1.289 .201
Wilks' Lambda .982 1.289b 1.000 70.000 .260 .018 1.289 .201
Hotelling's
Trace
.018 1.289b 1.000 70.000 .260 .018 1.289 .201
Roy's Largest
Root
.018 1.289b 1.000 70.000 .260 .018 1.289 .201
a. Design: Intercept + group
Within Subjects Design: Role Model
b. Exact statistic
c. Computed using alpha = .05
61
61
Tests of Within-Subjects Effects
Measure: MEASURE_1
Sourc
e
Type
III
Sum
of
Squar
es
df
Mea
n S
quar
e
F
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
era
Role Model Sphericity
Assumed
.062 1 .062 .052 .821 .001 .052 .056
Greenhouse-
Geisser
.062 1.000 .062 .052 .821 .001 .052 .056
Huynh-Feldt .062 1.000 .062 .052 .821 .001 .052 .056
Lower-bound .062 1.000 .062 .052 .821 .001 .052 .056
Role Model *
group
Sphericity
Assumed
1.562 1 1.562 1.289 .260 .018 1.289 .201
Greenhouse-
Geisser
1.562 1.000 1.562 1.289 .260 .018 1.289 .201
Huynh-Feldt 1.562 1.000 1.562 1.289 .260 .018 1.289 .201
Lower-bound 1.562 1.000 1.562 1.289 .260 .018 1.289 .201
Error(Role
Model)
Sphericity
Assumed
84.875 70 1.213
Greenhouse-
Geisser
84.875 70.000 1.213
Huynh-Feldt 84.875 70.000 1.213
Lower-bound 84.875 70.000 1.213
a. Computed using alpha = .05
62
62
Science Knowledge
Change in Science Knowledge by Group
A statistically significant change in Knowledge was seen in the treatment group and not
for the control group from pretesting to post testing. F(1,74)=4.63. p<.05 η2=.45. This
represents a small effect size.
Figure 3 Change in Science Knowledge by Group
63
63
Table 22 Change in Science Knowledge by Group
Descriptive Statistics
group Mean Std. Deviation N
science Control 4.3137 1.24026 38
Treatment 5.0816 .90460 38
Total 4.6977 1.14539 76
pscience Control 4.0690 1.32331 38
64
64
Treatment 5.2842 .70731 38
Total 4.6766 1.21853 76
Table 23 Change in Science Knowledge by Group Multivariate tests
Multivariate Testsa
Eff
ect
Val
ue
F
Hypoth
esis
df
Err
or
df
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
erc
science Pillai's Trace .001 .041b 1.000 74.000 .840 .001 .041 .055
Wilks'
Lambda
.999 .041b 1.000 74.000 .840 .001 .041 .055
Hotelling's
Trace
.001 .041b 1.000 74.000 .840 .001 .041 .055
Roy's Largest
Root
.001 .041b 1.000 74.000 .840 .001 .041 .055
science *
group
Pillai's Trace .059 4.623b 1.000 74.000 .035 .059 4.623 .564
Wilks'
Lambda
.941 4.623b 1.000 74.000 .035 .059 4.623 .564
65
65
Hotelling's
Trace
.062 4.623b 1.000 74.000 .035 .059 4.623 .564
Roy's Largest
Root
.062 4.623b 1.000 74.000 .035 .059 4.623 .564
a. Design: Intercept + group
Within Subjects Design: science
b. Exact statistic
c. Computed using alpha = .05
Table 24 Change in Science Knowledge by Group Means
Tests of Between-Subjects Effects
Measure: MEASURE_1
Transformed Variable: Average
Source Type III
Sum of
Squares
df Mean
Square
F Sig. Partial
Eta
Squared
Noncent.
Parameter
Observed
Powera
Intercept 3339.323 1 3339.323 1764.335 .000 .960 1764.335 1.000
group 37.358 1 37.358 19.738 .000 .211 19.738 .992
Error 140.058 74 1.893
a. Computed using alpha = .05
group * science
Measure: MEASURE_1
group science Mean Std. Error 95% Confidence Interval
Lower Bound Upper Bound
Control 1 4.314 .176 3.963 4.665
2 4.069 .172 3.726 4.412
Treatment 1 5.082 .176 4.731 5.432
2 5.284 .172 4.941 5.627
66
66
Quiz Knowledge
The information below indicates the mean score for the treatment group on the
knowledge sections of the quizzes. A table with the standard deviation is also included. The
educators have been given the results by question in order to examine areas which need more
Role Modelion. This document is available upon request and is included on the CD. A
comparison was also done by class. For all quizzes the results were not different by class. There
were no significant differences by gender. For both years student comprehension was lower for
the raptor quiz.
Figure 4 Quiz Knowledge section means 2015
67
67
Figure 5 Quiz Knowledge means 2016
68
68
Table 25 Mean scores on Knowledge Quiz
Descriptive Statistics
N Minimum Maximum Mean Std. Deviation
Q1 Intro. 43 1.00 5.00 4.1163 .95641
69
69
Q2 Hawks. 38 1.00 5.00 3.7105 1.13680
Q3 Raptors. 41 1.00 2.00 1.3902 .49386
Q4 Bats 1. 47 1.00 5.00 3.5957 1.20974
Q5 Bats 2. 47 1.00 5.00 3.2766 1.21050
Q6 Owls. 44 1.00 5.00 3.9773 1.10997
Q7 Reptiles . 44 1.00 5.00 4.1364 1.00211
Q8 Vocab. 42 2.00 5.00 4.3571 .98331
Q9 General. 42 1.00 5.00 3.4286 1.03930
Q10 Energy . 43 1.00 5.00 3.8605 1.18686
Valid N (listwise) 29
No significance difference was found by gender, males and females had equivalent
scores on the knowledge sections of the test.
Self-perceived science grade in school
A statistically significant change in self-reported anticipated grade in science was
seen in the treatment group and not for the control group from pretesting to post testing
F(1,71)=4.05. p<.05 η2=.06. This represents a small effect size.
70
70
Figure 6 Self perceived change in science grade by group
Table 26 Group Comparison for I will get a good grade in science class this year
Descriptive Statistics
group Mean Std.
Deviation
N
I will get a good grade in science class this year Control 4.946 .9985 37
71
71
Treatment 5.083 1.0247 36
Total 5.014 1.0068 73
POST I will get a good grade in science class this
year
Control 4.41 1.462 37
Treatment 5.25 .806 36
Total 4.82 1.251 73
Table 27 Group Comparison for I will get a good grade in science class this year
Multivariate Testsa
Eff
ect
Val
ue
F
Hypoth
esis
df
Err
or
df
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
erc
scigrd
2
Pillai's
Trace
.016 1.130b 1.000 71.000 .291 .016 1.130 .182
Wilks'
Lambda
.984 1.130b 1.000 71.000 .291 .016 1.130 .182
Hotelling'
s Trace
.016 1.130b 1.000 71.000 .291 .016 1.130 .182
72
72
Roy's
Largest
Root
.016 1.130b 1.000 71.000 .291 .016 1.130 .182
scigrd
2 *
group
Pillai's
Trace
.054 4.045b 1.000 71.000 .048 .054 4.045 .510
Wilks'
Lambda
.946 4.045b 1.000 71.000 .048 .054 4.045 .510
Hotelling'
s Trace
.057 4.045b 1.000 71.000 .048 .054 4.045 .510
Roy's
Largest
Root
.057 4.045b 1.000 71.000 .048 .054 4.045 .510
a. Design: Intercept + group
Within Subjects Design: scigrd2
b. Exact statistic
c. Computed using alpha = .05
Group Comparison for I will get a good grade in science class this year Within Subjects Contrasts
73
73
Tests of Within-Subjects Contrasts
Measure: MEASURE_1
Sourc
e
scig
rd2
Type
III
Sum
of
Squar
es
df
Mea
n S
quar
e
F
Sig
.
Par
tial
Eta
Squar
ed
Nonce
nt.
Par
amet
er
Obse
rved
Pow
era
scigrd2 Linear 1.275 1 1.275 1.130 .291 .016 1.130 .182
scigrd2 * group Linear 4.563 1 4.563 4.045 .048 .054 4.045 .510
Error(scigrd2) Linear 80.095 71 1.128
a. Computed using alpha = .05
Quiz Attitude subscales
Student mean attitudes about the program are presented below. The maximum possible
score is a 6/6 which indicates Strongly Agree.
Figure 7 Quiz Attitude subscales
74
74
75
75
Table 28 Quiz Attitude subscale descriptive statistics
Descriptive Statistics
N Minimum Maximum Mean Std.
Deviation
tt posttest student enjoyed TT 42 3.00 6.00 5.5952 .68880
TT postest overall student
perception of educator
42 3.50 6.00 5.6810 .50133
Program enjoyment 48 3.00 6.00 5.4062 .69246
Attitude Environment 45 4.00 6.00 5.7926 .40383
Valid N (listwise) 40
76
76
Feedback on Educator
Student mean attitudes about the Talking Talon’s educators are presented below. The
maximum possible score is a 6/6 which indicates Strongly Agree.
Student Quantitative Feedback Figure 8 Student Perception of Educator
77
77
Student Feedback
Table 29 Student feedback on Educator Subscales
Descriptive Statistics
N Minimum Maximum Mean Std. Deviation
Perception of Educator 50 4.43 6.00 5.7346 .34169
78
78
Bonding with Educator 49 4.67 6.00 5.7177 .41451
Educator Teaching Skills 50 4.33 6.00 5.7682 .36095
Valid N (listwise) 49
Figure 9 Student Perception of Educator
79
79
Table 30 Student feedback on Educator from Quiz results
Descriptive Statistics
N Minimum Maximum Mean Std.
Deviation
POST My Talking Talons educator
(teacher) cared a lot about what I learned.
41 5 6 5.76 .435
POST My Talking Talons educator
(teacher) was interested in what I had to
say.
42 4 6 5.71 .508
POST My Talking Talons educator
(teacher) was fair about giving everyone an
equal chance to do things.
42 2 6 5.60 .857
POST My Talking Talons educator knew a
lot about the animals.
42 5 6 5.86 .354
POST My Talking Talons educator
(teacher) was easy to understand.
42 2 6 5.48 .890
POST I think my Talking Talons educator
(teacher) liked coming to teach my class.
42 4 6 5.74 .544
POST My Talking Talons educator
understands how kids my age think.
42 1 6 5.50 .944
POST My Talking Talons educator treated
boys and girls equally.
42 2 6 5.74 .734
POST My Talking Talons educator helped
me to learn how to give speeches.
41 4 6 5.73 .549
Valid N (listwise) 40
80
80
Student Qualitative Feedback on Educators
The students enjoyed the Talking Talons educators and felt they were effective
instructors
Um, I like, like, because they know what we were thinking, or how we react to big birds,
they kind of break it down, or if we don’t know that word they break it down, or like
since they were our age they know how like to do stuff.
Well, that’s, they did, I, I can’t find, I don’t know what they can improve on but they did
pretty well.
kind of like uh, cuz Betsy and Lori they actually, they actually break it down in terms that
we can understand and they help us out in ways they know we would understand.
Um, I like that Lori is just like, whatever, and (Giggling and Laughing) and I like that
um, that Betsy like uh, she lets us understand, like when we’re learning about the animals
she like slowly, and then we catch on.
There’s nothing actually they can improve on they’re perfect.
Uh I like uh that Lori, well yeah I like that Lori she kind of like goes with the flow and
makes us laugh a lot, and I also like that how Betsy she explains to uh everything to us in
a way we can all understand.
Um, I thought that they did good about telling us how, how to hold it, how we could
change our speech to uh, improve it.
I think they did amazing.
81
81
They were fun, they, they were fun and they taught us a bunch of things.
They were really nice…and they did a good job
Classroom teacher feedback for Educator
The classroom teachers mean perception of the educator is presented below. These
questions are on a scale with a maximum positive score of seven.
Figure 10 Classroom teacher feedback on Talking Talons Educator
82
82
Table 31 Classroom Teacher Feedback on Talking Talons Educator
Statistics
The TT
educator
was
generally
prepared to
teach the
class.
The TT
educator
worked with
me to make
the program
run smoothly.
The TT
educator
used
classroom
time well.
The TT educator
appeared generally
knowledgeable
about content
presented
The TT
educator
appeared to
enjoy
teaching the
students.
N Valid 4 4 4 4 4
Missing 0 0 0 0 0
Mean 6.50 6.75 7.000 6.75 7.00
Median 7.00 7.00 7.000 7.00 7.00
Std.
Deviation
1.000 .500 .0000 .500 .000
Minimum 5 6 7.0 6 7
Maximum 7 7 7.0 7 7
Figure 11 Classroom Teacher subscales on Educator
83
83
Classroom teacher feedback for Talking Talons Program
The classroom teacher’s mean perceptions of the Talking Talon’s program are presented
below. These questions are on a scale with a maximum positive score of seven.
Teacher Attitude Figure 12 Classroom teacher feedback on Talking Talons program
84
84
Table 32 Classroom Teacher Feedback Personal
Statistics
Talking Talons is
a worthwhile use
of my classroom
time.
I was able to tie
the TT program
into my
curriculum.
I found the material
presented by the TT
program to be
interesting to me
personally.
I would be
willing to have
the TT program
in my classroom
again.
N Valid 4 4 4 4
Missing 0 0 0 0
Mean 6.75 6.25 6.25 6.75
Median 7.00 6.50 6.50 7.00
Std.
Deviation
.500 .957 .957 .500
85
85
Effectiveness by student ability: Feedback by Classroom teachers
The classroom teachers mean perception of the impact of the program by student ability
is presented below. These questions are on a scale with a maximum positive score of seven. The
teachers found the program generally very effective for average students and v for advanced
students. The program was found to be effective for those with learning disabilities. These
results mirror previous years feedback and indicate that the program is reaching the majority of
the student’s abilities at the correct level.
Figure 13 Classroom teacher assessment of Effectiveness of program by student ability
86
86
Table 33 Classroom teacher feedback program effectiveness by student ability
Statistics
The TT program was
effective for above
average students.
The TT program was
effective for average
students.
The program was effective
for students with learning
disabilities.
N Valid 4 4 4
Missing 0 0 0
Mean 6.25 6.50 5.50
Median 6.00 6.50 5.50
Std.
Deviation
.500 .577 1.291
Minimum 6 6 4
87
87
The teachers found the program generally very effective for average students and
effective for advanced students and those with learning disabilities.
Classroom teacher subscales
The classroom teacher’s mean perceptions for subscales are presented below. These
questions are on a scale with a maximum positive score of seven. Data for teachers for this year
and the previous year is included.
Figure 14 Classroom teacher perception
88
88
Table 34: Classroom teacher subscales for Talking Talons program
Statistics
Teacher
science
for
students
Teacher
effectiveness
by student
ability
Teacher
components
of TT
Teachers
personal
feelings
about TT
Teachers
perception
of educator
N Valid 4 4 4 4 4
Missing 0 0 0 0 0
Mean 6.3333 6.0833 7.0000 6.4000 6.8333
Median 6.3333 6.0000 7.0000 6.5000 7.0000
Std.
Deviation
.54433 .73912 .00000 .48990 .33333
89
89
Minimum 5.67 5.33 7.00 5.80 6.33
Other Results
Changes in pre posttest Composite
Figure 15 Statistically significant Pre Post change by group
Table 35 Statistically significant Pre Post change by group
Report
groupnum Change in Knowledge Change in Science Attitude
90
90
Treatment Mean .5758 .2026
N 37 38
Std. Deviation .61541 .70653
Control Mean .0255 -.2447
N 37 38
Std. Deviation .49282 1.07052
Total Mean .3007 -.0211
N 74 76
Std. Deviation .61911 .92862
Table 36 ANOVA Statistically significant Pre Post change by group
ANOVA Table
Sum of
Squares
df Mean
Square
F Sig.
Change in
Knowledge *
groupnum
Between
Groups
(Combined) 5.603 1 5.603 18.027 .000
Within Groups 22.378 72 .311
Total 27.980 73
Change in Science
Attitude * groupnum
Between
Groups
(Combined) 3.803 1 3.803 4.623 .035
Within Groups 60.873 74 .823
91
91
Total 64.675 75
Table 37 Effect Size Statistically significant Pre Post change by group
Measures of Association
Eta Eta Squared
Change in Knowledge * groupnum .447 .200
Change in Science Attitude * groupnum .242 .059
Structural Equation Model
A full separate report of a Structural equation model of the impact of student attitudes
toward the educator and the animals upon science attitudes is provided separately. This model
was designed to examine the impact of the educators and the animals upon science attitude. A
summary of the conclusions is presented below.
Figure 16: Final pruned model of impact of student attitude toward educator and animals on science attitudes
92
92
A theoretical structural model (the Saturated Model) had an good fit to the data The model fit
for this SEM is indicates that high level of confidence in the model. The sample size is small.
93
93
A second and simpler model (the Pruned Model) that eliminated the non-significant paths in
the Saturated Model also exhibited a close fit to the data
Attitude toward Educator had
A large positive direct effect on Attitude towards Animals
A medium positive indirect effect on Science Attitudes
Attitude toward Animals had
A large positive direct effect Science Attitudes
Upstream variables in the Pruned Model predicted
42% for Science Attitude
65% for Attitude toward Animals
Students who had more positive Attitudes toward Animals had large statistically
significant impact on Science Attitudes. Students who had more positive viewpoints of Attitude
toward Educator had a large direct effect and a medium indirect effect on Science Attitudes for a
total large statistically significant impact on Science Attitudes. The direct impact of the Attitude
toward Educator on Science Attitudes was not significant. Attitude towards Animals has the
largest direct impact on Science Attitudes, however the Attitude toward Animals is directly
impacted by the Attitude toward Educators. Educators are changing Science Attitudes by
changing the student’s Attitudes toward Animals. This is a complex outcome, indicating that
the animals are the crux of the change in science attitudes for the students, but that the
educators are indirectly influencing science attitudes by working through impact on
attitudes toward animals.
94
94
Summary
A statistically significant change in Attitude toward Science was seen in the treatment
group and not for the control group from pretesting to post testing. F(1,74)=4.62. p<.05 η2=.06.
This represents a small effect size.
In order to examine the longitudinal change, the science attitudes of the treatment group
were examined twice during the program as well as during pretesting and posttesting. The
treatment group exhibited a statistically significant positive change in attitude toward science as
the program progressed. F(3,30). p<.01 η2=.37 (large effect size). This statistic uses only the
treatment group, as the control group does not take the Talking Talon’s quizzes. Students in the
program also exhibited a statistically significant change in self-reported anticipated grade in
science. This change was seen in the treatment group and not for the control group from
pretesting to post testing F(1,71)=4.05. p<.05 η2=.06. This represents a small effect size.
A statistically significant change in Knowledge was seen in the treatment group and not for
the control group from pretesting to post testing. F(1,74)=4.63. p<.05 η2=.45. This represents a
very large effect size. A breakdown of the treatment group Knowledge scores on quizzes also
was provided.
Classroom teacher feedback was extremely positive and both teachers indicated the program
increased student science knowledge and attitudes, that the buddy class was worthwhile and also
indicated a strong willingness to have the program again. Mean scores for all subscales
measuring teacher feedback were all above 6 on a 7 point scale (a higher number indicates more
positive feedback)
95
95
Qualitative feedback from students was overwhelmingly positive, with students mentioning
that the program was a positive experience, that they enjoyed presenting to the buddy class and
found it both rewarding and educational.
This Structural Equation Model is a preliminary investigation of the relative importance of
the educator and the animals upon the science attitudes of the students. The full report is
provided separately.
A theoretical structural model (the Saturated Model) had an good fit to the data The model fit
for this SEM is indicates that high level of confidence in the model.
Attitude toward Educator had
A large positive direct effect on Attitude towards Animals
A medium positive indirect effect on Science Attitudes
Attitude toward Animals had
A large positive direct effect Science Attitudes
Upstream variables in the Pruned Model predicted
42% for Science Attitude
65% for Attitude toward Animals
Students who had more positive Attitudes toward Animals had large statistically
significant impact on Science Attitudes. Students who had more positive viewpoints of Attitude
96
96
toward Educator had a large direct effect and a medium indirect effect on Science Attitudes for a
total large statistically significant impact on Science Attitudes. The direct impact of the Attitude
toward Educator on Science Attitudes was not significant. Attitude towards Animals has the
largest direct impact on Science Attitudes, however the Attitude toward Animals is directly
impacted by the Attitude toward Educators. Educators are changing Science Attitudes by
changing the student’s Attitudes toward Animals. This is a complex outcome, indicating
that the animals are the crux of the change in science attitudes for the students, but that the
educators are indirectly influencing science attitudes by working through impact on
attitudes toward animals.
Even with a small sample size, and thus less statistical power, it is clear that the program
positivity impacted the participants. Science attitudes and knowledge are increased and
classroom teachers find the program effective and enjoyable for their students.
97
97
References
Al-Balushi, Sulaiman M, and Shamsa S Al-Aamri. “The Effect of Environmental Science
Projects on Students’ Environmental Knowledge and Science Attitudes.” International
Research in Geographical and Environmental Education 2014: 213–227. Web.
Aquino, Karl, and Americus Reed. “The Self-Importance of Moral Identity.” Journal of
personality and social psychology 83.6 (2002): 1423–1440. Web.
Arbuthnot, J, and D a Gordon. “Behavioral and Cognitive Effects of a Moral Reasoning
Development Intervention for High-Risk Behavior-Disordered Adolescents.” Journal of
consulting and clinical psychology 54.2 (1986): 208–216. Web.
Arkow, Phil. “Animals as Facilitators of Social Capital and Character Education.” Society for the
Study of Social Problems 2006. Web.
Backes, John S. “Bridging the Gender Gap: Self-Concept in the Middle Grades.” Schools in the
Middle 3.3 (1994): 19–23. Print.
Bakeman, Roger. “Recommended Effect Size Statistics for Repeated Measures Designs.”
Behavior research methods 37.3 (2005): 379–384. Web.
Baker, Dale R. “Predictive Value of Attitude, Cognitive Ability, and Personality to Science
Achievement in the Middle School.” Journal of Research in Science Teaching 22.2 (1985):
98
98
103–113. Print.
Berkowitz, Marvin W. et al. “Moral Reasoning and Judgments of Aggression.” Journal of
Personality and Social Psychology 51.4 (1986): 885–891. Web.
Bowman-Perrott, Lisa et al. “Academic Benefits of Peer Tutoring : A Meta-Analytic Review of
Single-Case Research.” School Psychology Review 42.1 (2013): 39–55. Print.
Catsambis, Sophia. “Gender, Race, Ethnicity, and Science Education in the Middle Grades.”
Journal of Research in Science Teaching 32.3 (1995): 243–257. Print.
Desy, Elizabeth A, Scott A Peterson, and Vicky Brockman. “Gender Differences in Science-
Related Attitudes and Interests among Middle School and High School Students.” Science
Educator 20.2 (2011): 23–30. Web.
Ellery Samuels, William, Kate Nicoll, and Cindy Trifone. “An In-Class, Humane Education
Program Can Improve Young Students’ Attitudes Toward Animals.” Society & Animals
16.1 (2008): 45–60. Web.
Fantuzzo, John W., Kenneth Polite, and Natalie Grayson. “An Evaluation of Reciprocal Peer
Tutoring across Elementary School Settings.” Journal of School Psychology 28.4 (1990):
309–323. Web.
Farenga, Stephen, and Beverly Joyce. “Science-Related Attitudes and Science Course
Selection...” Roeper Review 20 (1998): 247. Print.
Freedman, Michael P. “The Influence of Laboratory Instruction on Science Achievement and
99
99
Attitude toward Science across Gender Differences.” Journal of Women and Minorities in
Science and Engineering 8.2 (2002): 191–200. Web.
Galbraith, Jonathan, and Mark Winterbottom. “Peer‐tutoring: What’s in It for the Tutor?”
Educational Studies 37.3 (2011): 321–332. Web.
Glasson, George E. “The Effects of Hands-on and Teacher Demonstration Laboratory Methods
on Science Achievement in Relation to Reasoning Ability and Prior Knowledge.” Journal
of Research in Science Teaching 26.2 (1989): 121–131. Web.
Greenwood, Charles R et al. “Classwide Peer Tutoring Learning Management System:
Applications with Elementary-Level English Language Learners.” Remedial and Special
Education 22.1 (2001): 34–47. Print.
Guivernau, Marta, and Joan L. Duda. “Moral Atmosphere and Athletic Aggressive Tendencies in
Young Soccer Players.” Journal of Moral Education 31.1 (2002): 67–85. Web.
Gumpel, T P, and R Frank. “An Expansion of the Peer-Tutoring Paradigm: Cross-Age Peer
Tutoring of Social Skills among Socially Rejected Boys.” Journal of applied behavior
analysis 32.1 (1999): 115–118. Web.
Jarvis, Tina, and Anthony Pell. “Changes in Primary Boys’ and Girls' Attitudes to School and
Science during a Two-Year Science in-Service Programme.” The Curriculum Journal 13.1
(2002): 43–69. Print.
Kamps, D M et al. “Classwide Peer Tutoring: An Integration Strategy to Improve Reading Skills
and Promote Peer Interactions among Students with Autism and General Education Peers.”
100
100
Journal of applied behavior analysis 27.1 (1994): 49–61. Web.
Kanter, David E., and Spyros Konstantopoulos. “The Impact of a Project-Based Science
Curriculum on Minority Student Achievement, Attitudes, and Careers: The Effects of
Teacher Content and Pedagogical Content Knowledge and Inquiry-Based Practices.”
Science Education 94.5 (2010): 855–887. Web.
Kim, SeongSil, SinAe Park, and KiCheol Son. “Improving Peer Relations of Elementary School
Students through a School Gardening Program.” HortTechnology 24.2 (2014): 181–187.
Web.
Kimble, G. “Children Learning about Biodiversity at an Environment Centre, a Museum and at
Live Animal Shows.” Studies in Educational Evaluation 41 (2014): 48–57. Web.
Kose, Sacit et al. “The Effects of Cooperative Learning Experience on Eighth Grade Students’
Achievement and Attitude toward Science.” Education 131.1 (2010): 169–180. Print.
Kunsch, Catherine A, Asha K Jitendra, and Sheetal Sood. “The Effects of Peer-Mediated
Instruction in Mathematics for Students with Learning Problems: A Research Synthesis.”
Learning Disabilities Research & Practice 22.1 (2007): 1–12. Web.
Lee, Valerie E, and David T Burkam. “Gender Differences in Middle Grade Science
Achievement: Subject Domain, Ability Level, and Course Emphasis.” Science Education
80.6 (1996): 613–650. Print.
Maguin, Eugene, and Rolf Loeber. “Academic Performance and Delinquency.” Crime and
Justice 20 (1996): 145–264. Web.
101
101
Malti, Tina, and Tobias Krettenauer. “The Relation of Moral Emotion Attributions to Prosocial
and Antisocial Behavior: A Meta-Analysis.” Child development 84.2 (2013): 397–412.
Web.
Margadant-Van Arcken, Marjan. “Environmental Education, Children, and Animals.”
Anthrozoos 3:1 (1989): 14–19. Print.
Mattern, Nancy Page Garland. “Relationships between Attitudes toward and Achievement in
Science for Rural Middle School Students: Patterns across Gender.” Dissertation Abstracts
International Section A: Humanities & Social Sciences 60.12-A (2000): 4319. Print.
Mattern, Nancy, and Candace Schau. “Gender Differences in Science Attitude-Achievement
Relationships over Time among White Middle-School Students.” Journal of Research in
Science Teaching 39.4 (2002): 324–340. Web.
McMaster, Kristen L., Douglas Fuchs, and Lynn S. Fuchs. “Research on Peer-Assisted Learning
Strategies: The Promise and Limitations of Peer-Mediated Instruction.” Reading & Writing
Quarterly 22.1 (2006): 5–25. Web.
Milner, Andrea R. et al. “Elementary Teachers’ Beliefs About Teaching Science and Classroom
Practice: An Examination of Pre/Post NCLB Testing in Science.” Journal of Science
Teacher Education 23.2 (2012): 111–132. Web.
Oliver, Joseph S. “A Longitudinal Study of Attitude, Motivation and Self Concept as Predictors
of Achievement in and Commitment to Science among Adolescent Students.” Dissertation
Abstracts International 47.8-A (1987): 2983. Print.
102
102
Oliver, Steve J, and Ronald D Simpson. “Influence of Attitude toward Science, Achievement
Motivation, and Science Self Concept on Achievement in Science: A Longitudinal Study.”
Science Education 72.2 (1988): 143–155. Web.
Ornstein, Avi. “The Frequency of Hands-on Experimentation and Student Attitudes toward
Science: A Statistically Significant Relation.” Journal of Science Education and
Technology 15.3/4 (2006): 285–297. Web.
Papanastasiou, Elena C, and Michalinos Zembylas. “The Effect of Attitudes on Science
Achievement: A Study Conducted Among High School Pupils in Cyprus.” International
review of education 48.6 (2002): 469–484. Print.
Pigott, H E, J W Fantuzzo, and P W Clement. “The Effects of Reciprocal Peer Tutoring and
Group Contingencies on the Academic Performance of Elementary School Children.”
Journal of applied behavior analysis 19.1 (1986): 93–8. Web.
Raaijmakers, Quinten, Rutger Engels, and Anne Van Hoof. “Delinquency and Moral Reasoning
in Adolescence and Young Adulthood.” International Journal of Behavioral Development
29 (2005): 247–258. Web.
Reynolds, Arthur J, and Herbert J Walberg. “A Structural Model of Science Achievement and
Attitude: An Extension to High School.” Journal of Educational Psychology 84.3 (1992):
371–382. Print.
Reynolds, Scott J, and Tara L Ceranic. “The Effects of Moral Judgment and Moral Identity on
Moral Behavior: An Empirical Examination of the Moral Individual.” The Journal of
103
103
applied psychology 92.6 (2007): 1610–1624. Web.
Robinson, Debbie R., Janet Ward Schofield, and Katrina L. Steers-Wentzell. “Peer and Cross-
Age Tutoring in Math: Outcomes and Their Design Implications.” Educational Psychology
Review 17.4 (2005): 327–362. Web.
Rohrbeck, Cynthia a. et al. “Peer-Assisted Learning Interventions with Elementary School
Students: A Meta-Analytic Review.” Journal of Educational Psychology 95.2 (2003): 240–
257. Web.
Santos, Robert G et al. “Effectiveness of Peer-Based Healthy Living Lesson Plans on
Anthropometric Measures and Physical Activity in Elementary School Students: A Cluster
Randomized Trial.” JAMA pediatrics 168 (2014): 330–7. Web.
Satterthwait, Donna. “Why Are ‘Hands-on’ Science Activities so Effective for Student
Learning?” Teaching Science: The Journal of the Australian Science Teachers Association
56.2 (2010): 7–10. Web.
Slate, John R, and Craig H Jones. “Fourth and Fifth Grade Students’ Attitudes toward Science:
Science Motivation and Science Importance as a Function of Grade Level, Gender, and
Race.” Research in the Schools 5.1 (1998): 27–32. Print.
Smith, T. J., S. L. Pasero, and C. M. McKenna. “Gender Effects on Student Attitude Toward
Science.” Bulletin of Science, Technology & Society 34.1-2 (2014): 7–12. Web.
Sorge, Carmen. “The Relationship Between Bonding with Nonhuman Animals and Students’
Attitudes Toward Science.” Society & Animals 16 (2008): 171–184. Web.
104
104
Stohr-Hunt, Patricia M. “An Analysis of Frequency of Hands-on Experience and Science
Achievement.” Journal of Research in Science Teaching 33.1 (1996): 101–109. Web.
Tangney, June Price, Jeff Stuewig, and Debra J Mashek. “Moral Emotions and Moral Behavior.”
Annual review of psychology 58 (2007): 345–372. Web.
Tarry, Hammond, and Nicholas Emler^. “Attitudes, Values and Moral Reasoning as Predictors
of Delinquency.” British Journal of Developmental Psychology 25 (2007): 169–183. Web.
Teshome, Yalem, Nancy Maushak, and Krishna Athreya. “Attitude toward Informal Science and
Math: A Survey of Boys and Girls Participating in Hands-On Science and Math
(FUNTIVITIES).” Journal of Women and Minorities in Science and Engineering 7.1
(2001): 59–74. Web.
Trainin, Guy et al. “Extraordinary Animals and Expository Writing: Zoo in the Classroom.”
Journal of Science Education and Technology 14.3 (2005): 299–304. Web.
Warburton, S J, W L Jenkins, and P Coxhead. “‘Science Achievement and Attitudes’ and the
Age of Transfer to Secondary School.” Educational Research 25.3 (1983): 177–183. Print.
Willson, Victor L. “A Meta-Analysis of the Relationship between Science Achievement and
Science Attitude: Kindergarten through College.” Journal of Research in Science Teaching
20.9 (1983): 839–850. Print.
Wilson, Christopher D. et al. “The Relative Effects and Equity of Inquiry-Based and
Commonplace Science Teaching on Students’ Knowledge, Reasoning, and Argumentation.”
Journal of Research in Science Teaching 47.3 (2010): 276–301. Web.
105
105
Wilson, K. et al. “Our Zoo To You: The Link between Zoo Animals in the Classroom and
Science and Literacy Concepts in First-Grade Journal Writing.” Journal of Early Childhood
Literacy 11.3 (2011): 275–306. Web.
Zasloff, R L et al. “Animals in Elementary School Education in California.” Journal of Applied
Animal Welfare Science 2.4 (1999): 347–357. Web.
106
106
Structural Equation Model of
Talking Talons Variables Attitudes toward
Animals and Educator upon
Science Outcome Attitudes
Dr. Carmen Sorge
Leiden LLC
Jan 2017
107
107
What is Structural equation modeling (SEM)?
Behavioral research often involves attempting to answer questions about how variables
relate to each other. For example, we might want to know what factors influence the grade a
student gets in algebra. We often go about trying to find the answer by examining factors
individually.
Perhaps we might look at whether the following factors influence grades:
Intelligence
Previous Experience with math
Attitudes toward math
Experiences with math teachers
In linear statistics the researcher looks at how each of these variables affects the grade in
math. So the researcher might find that IQ is a good predictor or that effort is not a good
predictor of what grade the student gets in algebra.
This research would look like this:
or
What is missing is how those variables might be interacting with each other complexly.
Often we know that the two variables are related (such as experiences with math teacher and
attitude about math). But does Intelligence affect Effort that then in turn affects Attitude? These
questions can be answered by structural equation modeling.
IQ Algebra grade
Effort Algebra grade
108
108
First the researcher makes a measurement model. This model looks at how the variables
interrelate with each other. The researcher then creates a model. For example the research might
use “Effort” as one of the variables. Several measures might be made of “Effort” such as the
hours the student spends studying and class attendance.
The variable called Effort, which is a “latent variable”, will then look like this:
Hours
Studying
The researcher continues with the other variables, say Intelligence, and Attitudes.
The strength of a Structural Equation model is that we can look at complex relationships
BETWEEN the variables. We can examine how one variable influences another that in turn
affects yet another variable. In the example above we might have model that looks like this:
Effort Class
Attendance
109
109
In the SEM the researcher can see if Intelligence affects Attitude that then in turn affects
Effort that predicts the grade in the class. Of course there are many more connections in this
model and with SEM it is possible to examine many of them and find which ones are the best
predictors. The researcher can also look at several outcomes or how one outcome affects another.
A SEM model gives weights to each of the relationships and allows the researcher to examine
the strength of these interactions. This is a powerful tool, which allows the researcher to give
more useful feedback to the program because it pinpoints the variables which lead to the most
change overall. However, it requires a greater investment of time and effort and a larger set of
participants than more linear research.
Intelligence
IQ Grades
Final Grade
in Algebra
Attitudes
Attitude
about
Learning
Perception
of Math
Difficulty
Effort
Hours
studying
Class Attendance
110
110
Study Design
Purpose of the Study
The purpose of this study was to examine possible causal relationships between the
attitudes of students towards the educator and animals and the impact upon science attitudes.
Throughout the program a significant change in science attitudes of treatment groups has been
observed which is not seen in the control groups. This SEM is a preliminary investigation of the
relative importance of the educator and the animals upon the science attitudes of the students.
All scales are designed so that a higher number is a more positive score. The models of these
relationships were drawn from theoretical constructs primarily related to Expectancy Value
theories.
Research Questions
1. Which model better describes the empirical relationships between the Talking Talons Factors
with the outcome variables?
2. Does the better model adequately describe the empirical relationships between the Talking
Talons Factors?
Limitations of the Study
1. The sample students in this study were drawn from one geographic area near Albuquerque.
The sample was not random. However, the sample is drawn from the population served by
the program and thus represents the target group
111
111
2. The attitude and demographic information was self-reported by participants.
3. No demographic differences were examined. It is possible that they exist and impacted the
models.
4. Data collection for this model is over a two year period and therefore and an ideal sample
size has not yet been achieved.
Definition of Constructs
Attitude toward Educator and Attitude toward Animals are latent variable constructed
form student feedback on their attitudes toward the educator taken from the Talking Talons
Posttest. Science Attitude outcome variable is constructed from responses on the Talking Talons
posttest and quizzes.
Attitude Theory
Theories of attitude and behavior have been entwined throughout their development.
Many attempts have been made to clarify the relationships between attitude and behavior in spite
of measurement difficulties. The use of attitude to predict behaviors, including achievement,
links these constructs both in practical application and in theory development.
Three components of attitude have been commonly identified from ancient times to the
present: affective, cognitive and behavioral. The affect component consists of feelings of like
and/or dislike held by the individual toward the attitude object. The cognitive component
consists of beliefs and ideas that the individual holds about the attitude object. The behavioral
component consists of tendencies to respond to the object.
112
112
Despite the long-standing interest in attitude, social psychologists have failed to provide a
single, universally accepted definition of the construct. Gordon W. Allport’s classic definition
(1935) is often cited as a starting point for attitude definition. He characterized attitude as “a
mental and neural state of readiness, organized through experience, exerting a directive or
dynamic influence upon the individual’s response to all objects and situations with which it is
related.” (Allport, 1935). Current literature suggests that attitude can be conceptualized as a
tendency to evaluate a stimulus with some degree of favor or disfavor. This evaluation is usually
expressed in the classical cognitive, affective and/or behavioral responses (Eagly & Chaiken,
1993; Manstead, 1996; Olson & Zanna, 1993).
Several schools of attitude theories have emerged from research in the area. These
theories can be broken down into at least four classes based on the assumptions that each makes
about attitude formation. These four categories are: Learning theories, Expectancy Value
theories, Consistency theories and Attribution theories. Each of these groups has a basic
underlying principle in common and contains variations on the main principle; each also
presumes the same underpinnings of attitude development. Most of the classes of theories have
modern offshoots that are derived from the 1960s model by Rosenberg and colleagues
(Rosenberg, Hovland, McGuire, Abelson, & Brehm, 1960). Their model added moderators
between attitude and objects. They theorized that affect, cognition and behavior all act as filters
on attitude.
Expectancy Value theories are based on the assumption that in making decisions people
try to maximize their reward potential. This is called subjective utility. Subjective utility is
defined as the product of 1) the value of a particular outcome and 2) the probability that this
113
113
alternative will produce that outcome. Examples of Expectancy Value theories are those by
Fishbein (Fishbein, 1979), Edwards (Edwards, 1954) and Rosenberg and colleagues (Rosenberg
et al., 1960).
The long term goal of this SEM research is to develop a model based on expectancy
theory that addresses the impact of various attitudes on risk and resiliency factors. This initial
research examines the impact on Science Attitudes. This section was chosen as a beginning
point due to the fact that five years of research indicated that the Talking Talons program has a
positive effect on Science Attitudes.
Method The participants were students enrolled in schools in a rural area near Albuquerque, New
Mexico.
Analysis Plan
SPSS 23 for Windows were used in the preliminary analysis of the data set to score the
composite instrument and quizzes, identify non-participants and examine score distributions.
AMOS 23 was used to estimate the Structural Equation Model (SEM) solutions.
Analysis Sequence
A two-step approach was used for the structural equation modeling. First, estimations of
the measurement models for the latent constructs were made. Second, the structural
relationships between the latent constructs were tested. This method is recommended by many
researchers (e.g. (Anderson & Gerbing, 1988; Joreskog & Sorbom, 1993). The following
analysis plan is based on that developed by Mattern (Mattern, 1999).
114
114
1. Analyzed preliminary sample for non-participants, outlying scores, missing data and
incomplete information.
2. Estimated solutions for the Measurement Model.
3. Estimated solutions for the Saturated Model.
4. Estimated solutions for the model nested within the full model (Pruned Model).
5. Compared chi-square value differences to determine the better fitting model.
6. Evaluated the fit of the better fitting model.
Fit Indices
Amos 23 provides a plethora of fit indices for each model tested. Fit indices measure the
fit of the data used to the hypothesized model. As different indices have been developed to
measure different parts of a model’s fit, it is customary to report a cross section of these indices
from different categories.
Fit indices can be classified as either absolute or incremental. Absolute indices assess
how well the model being tested fits the sample data. Incremental (or comparative) fit indices
measure improvement in fit for a model when compared to a second model. This second model
is usually some form of a baseline model and is often the null model.
The most universally used index of fit is the chi-square measure (or CMIN). Chi-square
is an absolute fit index and as such tests the extent to which the data fit all aspects of the model
together including factor loadings, factor variances/covariance’s and error variances (Byrne,
2001). Although commonly used, chi-square is problematic due to its sensitivity to sample size.
Large sample sizes are needed in SEM in order to have distributions that are well behaved.
115
115
Unfortunately, these same large sample sizes cause an inflation of chi-square (relative to the
sample size), thereby making it difficult to achieve a reasonable probability for accepting the
model fit. Due to these problems inherent with chi-square, researchers have developed a
multitude of goodness of fit indices (Byrne, 2001). A summary of these fit indices is presented
in Table 1.
Table 38: Fit Indices
Fit index Type of
index
Compares or tests Acceptable or good fit Sample size
issues
2
(Chi -square
or CMIN)
Absolute Likelihood ratio test
statistic
Small, non-significant
values
Large sample
sizes inflate
chi square and
increase
probability of
a Type II error
Cmin/df
(Relative chi-
square)
Absolute Chi square (taking into
account the df)
Lower values are better
values < 3 are
acceptable (Kline,
1998).
TLI
(Tucker-Lewis
fit index)
Comparative Fit compared to the null
model
Values closer to 1 are better
values close to or
greater than .95 (for
large samples)
indicative of good fit
(Hu & Bentler, 1999)
Less affected
by sample
size, penalizes
for model
complexity
116
116
Fit index Type of
index
Compares or tests Acceptable or good fit Sample size
issues
CFI
(Comparative
Fit index, also
called the
Bentler CFI)
Comparative Fit compared to the null
model
Values closer to 1 are better
values > .90 are
acceptable (Bentler,
1992)
values close to or
greater than .95 are
acceptable (Hu &
Bentler, 1999)
values should > .90 to
accept the model
(Garson, 2001)
Takes sample
size into
account
PRATIO
(Parsimony
Ratio)
Comparative
Ratio of the degrees of
freedom in the test
model to the degrees of
freedom in the null
model
Higher values are better
values > .5 are
reasonable fit
(Byrne, 2001)
Rewards
parsimonious
models
Models
Four models were of special interest in this research. They included the Measurement
Model, the Saturated Model, the Pruned Model and the Null Model. Descriptions of these
models are provided below.
117
117
Measurement Model
Measurement models are confirmatory factor analysis models that define the
relationships between the observed (indicator) variables and the unobserved (latent) variables.
The latent variables represent underlying constructs (for example, “Risk and Resiliency”) that
are measured by the observed variables (for example, “pself” and “pmoral”). Each indicator
variable has associated with it an error term which is indicated by the small circle containing an
“e” and a number attached to it (e.g. “e19” represents the error associated with the observed
variable “Risk and Resiliency”). This error consists of both random error and error which might
be part of the measurement. In these measurement models, a specific indicator variable’s error
variance is not allowed to covary with any of the errors associated with other indicator variables
for the latent corresponding construct. Measurement Models (unlike structural models) cannot
include an error term for the latent variable. In Measurement Models the latent variable is
considered perfectly reliable and errors are associated with the indicator variables.
Using Joreskog’s method the measurement model for each of the latent constructs for
attitude and achievement was tested for model fit separately. Factor loading patterns relating the
latent variables to their observed indicator variables were examined for size and direction. Each
latent construct model was estimated repeatedly (with each indicator’s path to the latent set to
“1” in turn) so that each indicator served as the scaling factor for one of the estimation runs.
A baseline Measurement Model was then estimated including all latent constructs, which
were allowed to intercorrelate. Each latent construct was connected by a regression path to its
observed indicators. The Measurement Model was then estimated repeatedly with a different
observed variable used to scale each latent in order to examine possible resulting differences (see
118
118
Figure 1). Because outlying scores can unduly affect statistical results, the baseline Measurement
Model was estimated three times: with outliers included, with outliers eliminated, and with
outliers reset to three standard deviations from their mean in the direction of their scores
Figure 17: Measurement Model
119
119
Examination of Latent Variables in the Measurement Model
The Measurement Model for each latent construct individually was estimated with the
participants including the original outlier scores. Some participants have missing data and are
not included in portions of the model. Unfortunately if the student is missing one part of the
data to be used in the SEM they will not be included in the model as the software needs full data
sets to fit the model.
The referent factor path loading (the path set to the value of 1) for each latent construct
was rotated among the indicators for that construct. The range of results obtained for all of the
indicator variables for each of the latents is given below in Table 2. Each factor loading was
positive, statistically significant, and at least moderate in size.
Table 39: Factor Path Loadings for Latents
Data Range of
Standardized
Factor path
loadings
Attitude
toward
Educator
.38-.64
Attitude
toward
Animals
.53-.66
Science
Attitude .58-.77
Impact of Outliers on the Measurement Model
The preliminary Measurement Model was specified with the three latent variables
intercorrelated. It was estimated three separate times; once with outliers included, once with
120
120
outliers removed, and once with outliers reset to three standard deviations from the distribution’s
mean in the direction of the original score. The results from the Measurement Model indicated
little difference in fit, regardless of the status of the outliers. Therefore, each subsequent model
was estimated including all participants and their original scores.
Final Measurement Model
The final Measurement Model also was specified with the three latent variables
intercorrelated. This model had an acceptable fit to the data. Each factor path loading was
positive, statistically greater than zero and at least moderate in size (see Figure 2). The
unstandardized results for the Measurement Model are provided in Table 3. The Measurement
Model was also reestimated using different individual variables as referents. Once again, all
factor path loadings were statistically greater than zero
Table 40: Measurement Model Unstandardized Results**
Regression Weights: (Group number 1 - Default model)
Estimate S.E. C.R. P
tt6q2 <--- Attitude toward Educator 1.000
tt5q1 <--- Attitude toward Educator .884 .268 3.294 ***
tt8q4 <--- Attitude toward Educator 1.627 .665 2.447 .014
tt3q4 <--- Attitude toward Animals 1.000
tt8q1 <--- Attitude toward Animals .881 .240 3.673 ***
tt9q5 <--- Attitude toward Animals 1.006 .252 3.988 ***
tt7q1 <--- Science Attitude 1.000
rtt5q3 <--- Science Attitude 1.321 .308 4.286 ***
tt10q1 <--- Science Attitude 1.035 .264 3.922 ***
*Path loadings of observed referent variables were set to 1.
**Amos output estimated values. The critical ratio (CR) is equal to the path value divided by its standard error. A
value greater than 1.96 indicates statistical significance at p<. 05.
121
121
Table 41: Correlations Between Latent Variables in the Measurement Model
Estimate
Attitude toward Educator <--> Attitude toward Animals .813
Science Attitude <--> Attitude toward Educator .517
Science Attitude <--> Attitude toward Animals .655
Table 42: Measurement Model Covariances**
Estimate S.E. C.R. P
Attitude toward Educator .050 .023 2.178 .029
Attitude toward Animals .145 .058 2.513 .012
Science Attitude .273 .099 2.772 .006
e1 .108 .022 4.869 ***
e2 .058 .014 4.075 ***
e3 .806 .136 5.911 ***
e4 .250 .051 4.912 ***
e5 .230 .044 5.235 ***
e6 .186 .041 4.509 ***
e7 .364 .080 4.552 ***
e8 .335 .107 3.139 .002
e9 .582 .111 5.237 ***
**Amos output estimated values. The critical ratio (CR) is equal to the path value divided by its
standard error. A value greater than 1.96 indicates statistical significance at p<. 05.
***Amos output estimated values. p<.001
Saturated Model
In this fully saturated model Attitudes toward Educator and Attitudes toward Animals
served as an exogenous variable which impacted Science Attitude. This model allowed for
each latent upstream to directly impact the latents downstream. The model assigned a latent
residual to each of the endogenous latent constructs. The theoretical structural model tested in
this project draws largely from Expectancy Value theory. However, many of the latents and
links between them have support in other theoretical models as well.
122
122
Figure 18: Saturated Model
The regression weights for the relationships between the latent variables and their
observed indicators and the direct relationships between the latents are presented in Figure 3 and
in Table 6. Squared multiple correlations for the Saturated Model are presented in Table 7
Table 43: Regression Weights for Saturated Model**
Estimate S.E. C.R. P
Attitude toward Animals <--- Attitude toward Educator 1.381 .481 2.874 .004
123
123
Science Attitude <--- Attitude toward Educator -.109 1.104 -.099 .921
Science Attitude <--- Attitude toward Animals .951 .681 1.396 .063
tt6q2 <--- Attitude toward Educator 1.000
tt5q1 <--- Attitude toward Educator .884 .268 3.294 ***
tt8q4 <--- Attitude toward Educator 1.627 .665 2.447 .014
tt3q4 <--- Attitude toward Animals 1.000
tt8q1 <--- Attitude toward Animals .881 .240 3.673 ***
tt9q5 <--- Attitude toward Animals 1.006 .252 3.988 ***
tt7q1 <--- Science Attitude 1.000
rtt5q3 <--- Science Attitude 1.321 .308 4.286 ***
tt10q1 <--- Science Attitude 1.035 .264 3.922 *** **Amos output estimated values. The critical ratio (CR) is equal to the path value divided by its standard error. A
value greater than 1.96 indicates statistical significance at p<. 05.
***Amos output estimated values. p<.001
Table 44: Squared Multiple Correlations for Saturated Model
Estimate
Attitude toward Educator .000
Attitude toward Animals .660
Science Attitude .430
tt10q1 .335
rtt5q3 .587
tt7q1 .429
tt9q5 .441
tt8q1 .329
tt3q4 .368
tt8q4 .142
tt5q1 .404
tt6q2 .317
Several of the variables in the structural model can impact each other through multiple
routes or paths. For example, a subsection of the Saturated Model is shown in Figure 3. Note
that Attitude toward Educator can impact Attitude towards animals directly (the thick line) and
indirectly through Attitude towards animals (the thin line). In other words, Attitude toward
Educator affects the Attitude towards animals directly (the thick line) and by changing Attitude
124
124
towards animals (dashed thin line), which then in turn affects the Science Attitude (thin line).
Attitude toward Educator total effect on Science Attitude is the sum of these two effects, the
direct and the indirect effects.
Figure 19: Direct and Indirect Effects
Table 45: Saturated Model Total Effects Standardized
Attitude toward
Educator
Attitude toward
Animals
Science
Attitude
Attitude toward
Animals
.813 .000 .000
Science Attitude .517 .693 .000
tt10q1 .299 .401 .579
rtt5q3 .396 .531 .766
tt7q1 .338 .454 .655
tt9q5 .539 .664 .000
tt8q1 .466 .573 .000
tt3q4 .493 .606 .000
tt8q4 .376 .000 .000
tt5q1 .636 .000 .000
tt6q2 .563 .000 .000
Table 46:Saturated Model Direct Effects
Attitude toward
Educator
Attitude toward
Animals
Science
Attitude
Attitude
toward
Educator
Science
Attitude
Attitude
toward
Animals
125
125
Attitude toward
Animals
.813 .000 .000
Science Attitude -.047 .693 .000
tt10q1 .000 .000 .579
rtt5q3 .000 .000 .766
tt7q1 .000 .000 .655
tt9q5 .000 .664 .000
tt8q1 .000 .573 .000
tt3q4 .000 .606 .000
tt8q4 .376 .000 .000
tt5q1 .636 .000 .000
tt6q2 .563 .000 .000
Table 47: Saturated Model Indirect Effects
Attitude toward
Educator
Attitude toward
Animals
Science
Attitude
Attitude toward
Animals .000 .000 .000
Science Attitude .563 .000 .000
tt10q1 .299 .401 .000
rtt5q3 .396 .531 .000
tt7q1 .338 .454 .000
tt9q5 .539 .000 .000
tt8q1 .466 .000 .000
tt3q4 .493 .000 .000
tt8q4 .000 .000 .000
tt5q1 .000 .000 .000
tt6q2 .000 .000 .000
The fit indices for the Saturated Model are presented in Table 11. These values indicate a
reasonably good (but not excellent) fit of the Saturated Model to the data (see Table 1 of fit
indices).
Table 48: Fit Indices for Saturated Model
Model 2 df p 2/df TLI CFI PRATIO
126
126
Saturated Model
40.80 24 .017 1.67 .73 .85 .53
Pruned Model
The Pruned Model is nested within the Saturated Model. This trimmed model eliminates
the paths from the Saturated Model that were not statistically significant or close to significant .
The following non-significant paths (from the Saturated Model) were eliminated in the Pruned
Model: Attitude toward Education impact on Science Attitude
The remainder of the paths remain the same as in the Saturated Model. Standardized estimates
are presented in Figure 6.
Figure 4 Pruned Model Standardized Estimates
127
127
The regression weights for the relationships between the latent variables and their
observed indicators as well as the direct relationships between the latents are presented in Table
10. All of the regression weights were statistically significant since the insignificant paths were
128
128
eliminated from the Pruned model. Squared multiple correlations for the Pruned Model are
presented in Table 10.
Table 49: Unstandardized Regression Weights for Pruned Model**
Estimate S.E. C.R. P
Attitude toward Animals <--- Attitude toward Educator 1.361 .472 2.886 .004
Science Attitude <--- Attitude toward Animals .897 .284 3.159 .002
tt6q2 <--- Attitude toward Educator 1.000
tt5q1 <--- Attitude toward Educator .877 .266 3.293 ***
tt8q4 <--- Attitude toward Educator 1.620 .662 2.446 .014
tt3q4 <--- Attitude toward Animals 1.000
tt8q1 <--- Attitude toward Animals .888 .242 3.663 ***
tt9q5 <--- Attitude toward Animals 1.017 .256 3.974 ***
tt7q1 <--- Science Attitude 1.000
rtt5q3 <--- Science Attitude 1.317 .307 4.289 ***
tt10q1 <--- Science Attitude 1.038 .264 3.931 ***
**Amos output estimated values. The critical ratio (CR) is equal to the path value divided by its standard error. A
value greater than 1.96 indicates statistical significance at p<. 05.
***Amos output estimated values. p<.001
Table 50: Squared Multiple Correlations for Pruned Model
Estimate
Attitude toward Educator .000
Attitude toward Animals .652
Science Attitude .423
tt10q1 .337
rtt5q3 .585
tt7q1 .429
tt9q5 .446
tt8q1 .331
tt3q4 .365
tt8q4 .141
tt5q1 .401
tt6q2 .320
Table 51: Total Effects for Pruned Model
Standardized Total Effects (Group number 1 - Default model)
Attitude toward
Educator
Attitude toward
Animals
Science
Attitude
129
129
Attitude toward
Animals
.807 .000 .000
Science Attitude .525 .650 .000
tt10q1 .305 .378 .581
rtt5q3 .402 .497 .765
tt7q1 .344 .426 .655
tt9q5 .539 .668 .000
tt8q1 .464 .575 .000
tt3q4 .487 .604 .000
tt8q4 .376 .000 .000
tt5q1 .633 .000 .000
tt6q2 .565 .000 .000
Table 52: Direct Effects for Pruned Model
Standardized Direct Effects (Group number 1 - Default model)
Attitude toward
Educator
Attitude toward
Animals
Science
Attitude
Attitude toward
Animals
.807 .000 .000
Science Attitude .000 .650 .000
tt10q1 .000 .000 .581
rtt5q3 .000 .000 .765
tt7q1 .000 .000 .655
tt9q5 .000 .668 .000
tt8q1 .000 .575 .000
tt3q4 .000 .604 .000
tt8q4 .376 .000 .000
tt5q1 .633 .000 .000
tt6q2 .565 .000 .000
Table 53: Indirect Effects for Pruned Model
Standardized Indirect Effects (Group number 1 - Default model)
Attitude toward
Educator
Attitude toward
Animals
Science
Attitude
Attitude toward
Animals
.000 .000 .000
Science Attitude .525 .000 .000
130
130
tt10q1 .305 .378 .000
rtt5q3 .402 .497 .000
tt7q1 .344 .426 .000
tt9q5 .539 .000 .000
tt8q1 .464 .000 .000
tt3q4 .487 .000 .000
tt8q4 .000 .000 .000
tt5q1 .000 .000 .000
tt6q2 .000 .000 .000
The fit indices for the Pruned Model for both years are presented in Table 13. These fit
indices indicate a good level of fit of the Model to the data. (see Table 1 of fit indices).
Table 54: Fit Indices for Pruned Model
Model 2 df p 2/df TLI CFI PRATIO
Pruned Model
40.814 25 .024 .76 ..88 .56
Comparison of Model Fit
Table 18 presents the fit indices for the three Models from this research plus that of the
Null Model. The Null Model represents a model in which all of the observed variables are
uncorrelated and as such represent a “baseline” lower limit for the fit of the model to the data.
Table 55: Comparison of the Fit of the Models
Model 2 df p 2/df TLI CFI PRATIO
Null Model 74.1 27 2.746 .24 .60 .32
Measurement Model 40.08 24 .017 1.67 .73 .85 .53
131
131
Saturated Model
40.80 24 .017 1.67 .73 .85 .53
Pruned Model
40.814 25 .024 1.67 .76 ..88 .56
Table 15 below compares the 2 and the df for the models used in this research. A
statistically significant 2 for the df (as indicated by p< .05) indicates that the fit of the two
models were significantly different from each other. As can be seen in Table 14, the
Measurement, the Saturated and the Pruned Models all fit the data significantly better than the
Null Model. There is no statistically significant difference in fit between the Saturated Model
and the Pruned Model. Because there is no difference in fit and the Pruned Model is simpler
than the Saturated Model, it is selected as the best fitting model.
Model fit for the Measurement Model was similar for all levels of inclusion for outliers.
Results from the comparison of model fit indicated that the Measurement, the Saturated and the
Pruned Models exhibited better fit than the Null Model. There was no significant difference in fit
between the Saturated and the Pruned Models. Therefore the answer to first research question
posed in this study is that the Pruned Model is the better fitting of the two models because it is
simpler.
132
132
Direct and Indirect Effects
This section discusses the standardized estimates of the path coefficients for the Pruned
Model. Standardized path coefficients estimate how much a downstream variable would change
assuming a change of one standard deviation in the upstream variable. For example, Attitude
towards Animals in the Pruned Model had a direct impact on Science Attitudes of .81. This
value means that a change of one standard deviation in Attitude towards Animals would produce
a change of .81 standard deviations in Science Attitudes (controlling for the rest of the upstream
latent variables). Attitude toward Educator had an indirect effect on Science Attitudes use of .53.
This value means that a change of one standard deviation in Attitude toward Educator would
produce a change of .53 standard deviations in Science Attitudes through its impact on other
latent variables that are upstream, in this case Attitudes toward Animals.
According to Kline (Kline, 1998); standardized path coefficients with values of less than
.10 can be interpreted as small effects, values of around .30 can be interpreted as medium effects
and values above .50 can be interpreted as large effects. All the effects in this model were larger
than .65, indicating large effects.
Summary of Outcomes
Every increase of one standard deviation in Attitude toward Animals produced a Direct effect
of .65 standard deviations in Science Attitude. Every increase of one standard deviation in
Attitude toward Educator factors produced a Direct effect of .81 standard deviations on Attitude
towards Animals and an Indirect effect of .53 standard deviations.
133
133
Variance of the Latent Variables
The magnitude of the variance for a latent variable indicates how much of the latent
variable's variance is accounted for by its upstream latent variables. The amount of variance for
each of the endogenous latent variables accounted for by its upstream latent predictors was 42%
for Science Attitude and 65% for Attitude toward Animals.
Global Summary
A theoretical structural model (the Saturated Model) had an good fit to the data The model fit
for this SEM is indicates that high level of confidence in the model. The sample size is small.
A second and simpler model (the Pruned Model) that eliminated the non-significant paths in
the Saturated Model also exhibited a close fit to the data
Attitude toward Educator had
A large positive direct effect on Attitude towards Animals
A medium positive indirect effect on Science Attitudes
Attitude toward Animals had
A large positive direct effect Science Attitudes
Upstream variables in the Pruned Model predicted
42% for Science Attitude
65% for Attitude toward Animals
134
134
Students who had more positive Attitudes toward Animals had large statistically
significant impact on Science Attitudes. Students who had more positive viewpoints of Attitude
toward Educator had a large direct effect and a medium indirect effect on Science Attitudes for a
total large statistically significant impact on Science Attitudes. The direct impact of the Attitude
toward Educator on Science Attitudes was not significant. Attitude towards Animals has the
largest direct impact on Science Attitudes, however the Attitude toward Animals is directly
impacted by the Attitude toward Educators. Educators are changing Science Attitudes by
changing the student’s Attitudes toward Animals. This is a complex outcome, indicating that the
animals are the crux of the change in science attitudes for the students, but that the educators are
indirectly influencing science attitudes by working through impact on attitudes toward animals.
135
135
Bibliography
Allport, G. (1935). Attitudes. In C. Murchison (Ed.), A handbook of Social Psychology (pp. 789-
844.). Worcester, Mass.: Clark University Press.
Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review
and recommended two-step approach. Psychological Bulletin, 103(3), 411-423.
Bentler, P. M. (1992). On the fit of models to covariances and methodology to the bulletin.
Psychological Bulletin, 107(238-246).
Byrne, B. (2001). Structural equation modeling with AMOS: Basic concepts, applications and
programming. New Jersey: Lawrence Erlbaum Associates, Inc.
Eagly, A. H., & Chaiken, S. (1993). The psychology of attitudes. Ft Worth, TX, USA: Harcourt
Brace Jovanovich College Publishers.
Edwards, W. (1954). The theory of decision making. Psychological Bulletin, 51, 380-417.
Fishbein, M. (1979). A theory of reasoned action: Some applications and implications. Nebraska
Symposium on Motivation, 27, 65-116.
Garson, G. (2001). PA 765 Statnotes: An Online Texbook. Available:
http://www2.chass.ncsu.edu/garson/pa765/structur.htm [2001, January 31, 2000].
Hu, L.-t., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:
Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55.
Joreskog, K. G., & Sorbom, D. (1993). LISREL 8: Structural equation modeling with the
SIMPLIS command language. Chicago: Scientific Software International.
Kline, R. B. (1998). Principles and practice of structural equation modeling. New York: Guilford
Press.
136
136
Manstead, A. S. R. (1996). Attitudes and behaviour, Applied social psychology (pp. 3-29).
London, England UK: Sage Publications.
Mattern, N. (1999). Relationships between attitudes toward and achievement in science for rural
middle school students : patterns across gender., University of New Mexico,
Albuquerque.
Olson, J. M., & Zanna, M. P. (1993). Attitudes and attitude change. Annual Review of
Psychology, 44, 117-154.
Rosenberg, M. J., Hovland, C. I., McGuire, W. J., Abelson, R. P., & Brehm, J. W. (1960).
Attitude organization and change: An analysis of consistency among attitude components
(Yales studies in attitude and communication Vol III ). New Haven, CT: Yale Univer.
Press.