gangitracydissertation10-20-09
DESCRIPTION
Gangi Tracy dissertationTRANSCRIPT
-
1
SCHOOL CLIMATE AND FACULTY RELATIONSHIPS: CHOOSING AN
EFFECTIVE ASSESSMENT MEASURE
A dissertation submitted in partial
fulfillment of the requirements
for the degree of
DOCTOR OF PSYCHOLOGY
to the faculty of the department of
SCHOOL PSYCHOLOGY
at
St. Johns University
New York
by
Tracy A. Gangi
Date Submitted: _________________ Date Approved: ________________
_______________________________ ______________________________ (Students Signature) (Mentors Signature)
-
2
Acknowlegements
Id like to thank several people for their role in helping me get from Point A to the
final Point B on this road to attain the doctoral degree in psychology. First and foremost,
Id like to thank my father, Dr. William Gangi; without his positive influence, I would
not have had the desire to also become a Dr. Gangi, nor believe it could be done. My
mother, Patricia DeLeo Gangi, for her unflagging support and desire for me to find
happiness with my life choices. To my sister, Alessandra Gangi Long, and my brother
William P. Gangi, my deepest heartfelt thanks for being my biggest champions
throughout life, regardless of many unconventional decisions I have made along the way.
To my husband, David, whom came on board later in my quest, but my most loyal
fighter, I love you with my whole heart. To my entire extended family, my good
friends and close colleagues, thank you for your patience, your kindness, and your love
and friendship. To Dr. Mary Macedonio, your courage and perseverance have inspired
me from the first day I met you, I doubt I would have made it through this without your
help.
I would like to thank several respected researchers in the field of school climate
whom honored me with their feedback, input, and guidance, especially: Dr. Jonathan
Cohen, Dr. John Shindler, Dr. Megan Tschannen-Moran, Dr. N. Scott Norton, Dr. Wayne
Hoy, Joshua Freedman, Dr. Carina Fiedeldey-Van Dijk, Dr. C. John Tarter, Dr. Martha
Alberg, and Dr. Barry Fraser.
To the members of my committee, both former: Dr. Pauline Magee-Egan, my
original mentor, thank you for your enthusiasm for my original topic (which included the
-
3
Myers-Briggs Type Inventory), Dr. Jim Curley, Dr. John Hogan, and current: Dr.
Raymond DiGiuseppe, my mentor, and Dr. Marlene Sotelo-Dynega and Dr. Kate Walton,
thank you for agreeing to take on my project and bringing it to fruition. Finally, Id like
to thank past, present, and future researchers of school climate for their assistance in its
evolution, and furthering its benefit to educators and students everywhere.
-
4
Table of Contents
Chapter I: Introduction1
Chapter II: Literature Review..6
The Evolution of School Climate....6
Impact of Negative School Climate/Low Levels of Trust: Students, Teachers, and
the Community.8
Teachers: Emotional Aspects of School Climate and Student Impact9
Trust Research in Schools..11
School Administrators and School Climate Intervention..12
Systems Level Change and Continuing Professional Development..13
School Psychologists as Effective Consumers of Research and Needs
Assessment.14
Organizational Development, School Climate, and the Use of Team Building15
Team Building and Communication.16
Statement of Problem17
Purpose..17
Research Question18
Chapter III: Methods.19
Procedures.19
Step 1 (Criteria 1): Literature search21
Step 2 (Criteria 2): Faculty relationship factor.22
Step 3 (Criteria 3): Additional school climate variables..22
Step 4 (Criteria 4): Direct measures...23
-
5
Step 5 (Criteria 5): Viewable test items.23
Step 6 (Criteria 6): Current published instruments23
Step 7 (Criteria 7): Reliability25
Step 8 (Criteria 8): Validity...........26
Step 9 (Criteria 9): School Level..28
Step 10: Expert Panel Procedures.....28
Step 11: Comparative ranking system.31
Reliability..31
Validity.31
Sample size, recency of norms, and representative norms...32
Chapter IV: Results...33
Step 10: Expert Panel Procedures.45
Step 11: Comparative Ranking System.50
Review of Point System50
Reliability..50
Validity.51
Sample size, recency of norms, and representative norms....51
Critical Test Review of the Comprehensive School Climate Inventory (CSCI)...51
Description51
Development.54
Technical...54
Norming....54
Internal consistency reliability..54
-
6
Test-retest reliability.....55
Inter-rater reliability..55
Construct validity..55.
Convergent validity...55
Divergent/discriminant validity.55
Content validity.55
Criterion-related validity...56
Concurrent validity56
Predictive validity.56
Use of multiple informants56
Critical Test Review of the School Climate Inventory-Revised (SCI-R).56
Description56
Development..58
Technical...58
Norming.58
Internal consistency reliability..59
Test-retest reliability.60
Inter-rater reliability..60.
Construct validity..60
Convergent validity..61
Divergent/discriminant validity61
Content validity61
Criterion-related validity...61
-
7
Concurrent validity....61
Predictive validity..61
Use of multiple informants....61
Critical Test Review of the Western Alliance for the Study of School Climates
School Climate Assessment Instrument.61
Description.61
Development..64
Technical66
Norming.66
Internal consistency reliability..67
Test-retest reliability.68
Inter-rater reliability..68
Construct validity..68
Convergent validity...69
Divergent/discriminant validity.69
Content validity.69
Criterion-related validity69
Concurrent validity69
Predictive validity..69
Use of multiple informants70
Chapter V: Discussion...72
Expert Panels Confirmation of Criteria72
Comparative Ranking System Outcome73
-
8
Commentary and Summary of the Comprehensive School Climate Inventory
(CSCI)....74
Strengths74
Weaknesses75
Commentary and Summary of the School Climate Inventory-Revised (SCI-R)..76
Strengths76
Weaknesses77
Commentary and Summary of the Western Alliance for the Study of School
Climate- School Climate Assessment Inventory (WASSC- SCAI).77
Strengths...77
Weaknesses...78
Chapter VI: Implications for Practice...81
Suggestions for School Climate Improvement Efforts.83
Suggestions Regarding Follow-Up Research85
Limitations of This Study..85
A Final Note..86
References..87
Appendix A: Expert Panel Questionnaire105
Appendix B: Additional School Climate Measures109
Appendix C: Expert Panel Biographies..113
-
9
Chapter 1
Introduction
School climate has been studied for decades. It has been defined variously as the
atmosphere, ethos, tone, ideology, community, personality, or milieu of a school (Hoy,
2008), how one feels about the school and the people involved in the school (Davis &
Peck, 1992), or how one feels about their experiences in a school (Lindelow, Mazzarella,
Scott, Ellis, & Smith, 1989).
Only recently, however, has there been a consolidated, formally agreed upon,
definition of school climate. The definition refers to four areas of school functioning: 1)
physical safety, 2) relationships of those in the school environment, including faculty,
students, and parents, 3) teaching and learning methods, and 4) the actual physical
environment of the school. This definition of school climate was agreed upon during a
consensus-building meeting in April, 2007 by the National Center for Learning and
Citizenship, Education Commission of the States, and the National School Climate
Center at the Center for Social and Emotional Education (Cohen, McCabe, Michelli, &
Pickeral, 2009). The findings from these collaborative organizations determined that
there was an enormous discrepancy between current empirical research on positive
school climate, and what is actually currently being done in the schools, in state and
federal education departments, as well as in the colleges and universities in which
teachers are educated. Their overall consensus deemed that the gap between school
climate research, policy, practice, and teacher education is socially unjust and a violation
of childrens human rights (Cohen, McCabe, Michelli, & Pickeral, 2009).
-
10
School climate research has indicated that poor school climate is correlated with
absenteeism (Reid, 1983), suspension rates (Wu, Pink, Crain, & Moles, 1982), and higher
school drop-out rates (Anderson, 1982). Research has further confirmed that students
who do not graduate from high school face increased risk of unemployment, poverty,
poor health, and involvement in the criminal justice system (Barton, 2005). Conversely,
research indicates that schools that demonstrate a positive school climate had better
attendance, higher morale (Bryk & Driscoll, 1988) and more academic effectiveness
(Borger et al., 1985, Shindler, et al., 2009). These schools are also a place where self-
worth, pride, respect, and trust are fostered (Kaplan & Geoffroy, 1990; Kelley, Thornton,
& Daugherty, 2005). Cohen, Pickeral, and McCloskey (2009), reiterated the importance
of the learning climate and the various populations that it is comprised of. Specifically,
not only do children thrive academically in a positive school climate but, so too, do
teachers. For example, teachers tend to stay employed longer at schools with positive
climate, and teacher continuity and consistency benefit student academic achievement
(Ingersoll, 2001, Wynn, 2008). Although there has been an abundance of research done
in the area of student/peer relationships and physical safety, especially in regards to
violence and bullying, a growing amount of research has been done in the area of
teacher/colleague relationships and how the quality of their relationships impact students
success (Tschannen-Moran, Parish, and DiPaola, 2006).
Recent school climate research has indicated that teachers and other adults in the
school strongly influence student character, morals, and ethics. The articles urge that
these adults serve as models and moral compasses to students (Lehr & Christenson, 2002;
NASP Blueprint, 2006; Steegman, 2006; Sandberg, 2007; Pantaleno, 2007). They
-
11
concluded that the way in which adults in schools treat each other is an important
variable in predicting school climate outcomes. Further, they concluded that to build a
positive social-emotional climate within an entire school, it is necessary that all members
are involved (Desrochers, 2007; Ninan, 2006; Sandberg, 2007; Pantaleno, 2007 and
Stewart, 2007). Trust has also been reported to be an important factor in the quality of
teacher relationships and their impact on a positive school climate. Where trust was
higher among teachers, there was a higher trust in students and their parents
(Tschannen-Moran, 2009, p. 240).
As the primary mental health experts in schools, school psychologists are
uniquely suited to facilitate and implement positive school climate changes and invest in
the under-utilized areas of school mental health and social-emotional learning (NASP,
2006; Desrochers, 2007; Lehr & Christenson, 2002; Center for Social-Emotional
Education, 2009; and Steegman, 2007).
In 1987, the Northwest Regional Educational Laboratory (NWREL) released a
report identifying 42 school climate assessments known to exist (as cited in Shindler et
al., 2003). To date, this author has documented more than 100 school climate measures.
A review of the literature indicated that there is a dearth of rigorous, peer-reviewed
studies that instruct educators how to improve the overall school climate specifically by
strengthening faculty relationships. (In the literature, these relationships are also
called: morale/cohesion/affiliation/ collegiality/trust). Surprisingly, no single cohesive
review exists of the more than 100 school climate measures that have been found, nor is
there a review of school climate measures that might specifically help school
psychologists assess the quality of faculty relationships. Finally, there is no known list of
-
12
critically reviewed school climate measures that have been held to the standards for
educational and psychological tests (American Psychological Association, American
Educational Research Association, and the National Council on Measurement in
Education, 1999). These standards refer to, test construction and development,
specifically: validity, reliability, and norms.
Before embarking on a systems change/ intervention that could assist in
enhancing faculty relationships, as well as other school climate variables, an appropriate
school climate measure needs to be selected. Moving forward, without choosing the
most appropriate assessment tool that will measure critical school climate variables could
waste time, energy, and funds. This concern was also verbalized in Bulach and Malones
(1994) report on school climate and educational reform, [Educational reform] is far too
expensive in terms of money and expended human energy to continue implementation in
the haphazard manner that occurs in many school districts (p. 6). Complicating the
matter of choosing the appropriate measure of school climate are the various emphases
each instrument has and which constructs are desired to be assessed. There is a demand
to locate a school climate measure that not only assesses the academic variables that
impact student success but also assesses the (adult) social-emotional factors, illustrated
here: There is a need for assessment tools that permit schools to monitor, report on, and
improve practice in addressing a wider range of educational outcomes (responsibility,
caring, engagement, and problem solving), as well as linking them to academic
outcomes (CASEL, 2006, p. 5).
Therefore, the focus of this paper is a search for school climate measures that not
only effectively assesses faculty relationships in primary and secondary schools, but also
-
13
assesses particular school climate predictors that have found consensus in current
research. Presumably, this/these instrument(s) can then be utilized with best practices
currently being applied to school action plans addressing school reform, social-emotional
learning goals, citizen and character education initiatives (Cohen, 2007).
This study proceeded in three stages. First, school climate assessment
instruments were selected according to a set of criteria. These included measures that
could confidently be used as a pre- and post-test in a faculty team-building intervention
or perhaps as part of a larger school climate assessment/intervention. Second, these
selected instruments were ranked using a point system, and third, the chosen instruments
underwent a critical test review, adhering to the standards put forth by the American
Psychological Association, American Educational Research Association, and the
National Council on Measurement in Education (1999).
-
14
Chapter 2
Literature Review
The Evolution of School Climate
One thing has remained constant throughout the decades of school climate
research: the inability to delineate the school climate literature into specific constructs
that are agreed upon among researchers. Van Horn (2003) reported that due to
measurement difficulties, the selection of variables and controls, and ambiguity
concerning which statistical analyses to use, school climate has been a discouraging area
of research for some. He explained that there are large differences between the various
school climate theories. Anderson (1982) brought up this concern about school climate
researchthat the view of school climate varied significantly among researchers
depending on which theory of school climate was used by the researcher.
Some researchers conceptualized school climate from a school/organizational-
level, one in which all the children in the same school are affected by the same climate
(Tagiuri & Litwin, 1968; James, 1982; Van Horn, 2003). Others conceptualized school
climate from an individual-level, in which a compilation of individual differences would
be expected to influence the overall school climate (Miller & Fredericks, 1990; Dixon,
Johnson, & Toman, 1991; Raudenbush, Rowan & Kang, 1991). Still others viewed
school climate in terms of organizational health. For example, Miles (1969), believed a
healthy organization was one that copes well over a long period of time; it has a good
long-term prognosis for thriving in good times and enduring through bad times.
Organizational health was later described by Hoy and Fedman (1987) as one in which
-
15
there is harmony in the relationships among, students, teachers and administrators in their
quest to fulfill their schools mission.
Arthur Perry, a principal in a school in Brooklyn, made the first observations
concerning the importance of not only the physical environment of the school but the
esprit (translated from French to mean spirit) of the school (Perry, 1908). In the
following hundred years, school climate has undergone an evolution, as more and more
research is added to our understanding of what it is and its impact on student
achievement.
In 1963, Halpin and Croft developed the Organizational Climate Descriptive
Questionnaire (OCDQ). It focused on teacher-teacher and teacher-administrator
interactions from the teachers point of view. In the 1970s, Moos and Insel (1974)
conceptualized school climate as the interaction of human environments and humans
within the social and physical aspects of the environment, according to three dimensions:
interpersonal relationships, goal orientation/ personal growth, and system
maintenance/system change. Rutter (1979) and his colleagues established the notion of
ethos as an important quality of schools (Rutter et al, 1979). This included caring relationships among faculty members and administrators as positive role models, high
expectations for student success, and an emphasis on positive rewards. In the 1980s school climate was largely conceptualized as the whole of a school
is more than the sum of its parts; the school was viewed as a community. Bryk and
Driscoll (1988) found that schools that demonstrated the importance of school
community had better attendance, higher morale, and better mathematics achievement. It
was further determined that teachers sense of responsibility for student learning, or self-
-
16
efficacy, was a factor of schools that viewed themselves as a community (e.g., Bandura,
1992, 1997; Tschannon-Moran et al., 1998).
In the 1990s the phrase academic press was utilized as a theory of school
climate. Academic press refers to the extent to which the school community, including
students and teachers, experience a strong emphasis on academic success and conformity
to school values and practices that bolster high standards for student performance (Lee,
Smith, Perry, & Smylie, 1999). This notion has been recognized in the literature as
predictive of relatively high achievement (Evans, 1997; Hoy and Sabo, 1998; Newmann,
1992).
Now, in a seemingly much needed organization of theories and definitions, the
21st century has finally turned out a consensus-determined definition of school climate
highlighting four areas of school functioning (as noted previously): physical safety,
relationships of those in the school environment, including faculty, students, and parents,
teaching and learning methods, and the actual physical environment (Cohen, McCabe,
Michelli, and Pickeral, 2009).
Impact of Negative School Climate/Low Levels of Trust: Students, Teachers, and the
Community
Hoy, Tarter and Kottkamps (1991) research confirmed that a great many schools
across the nation fit the description of an unhealthy school. Schools with a negative
school climate, prevent teachers, support staff and administration from being able to
model positive citizen-forming behaviors for our students (Collaborative for Academic,
Social, and Emotional Learning (CASEL), 2008).
-
17
Damico, Roth, Fradd, and Hankins, (1991) indicated that schools with a negative
climate send out negative signals to at-risk students, instilling the perception that they are
unworthy and unable to continue in the educational process. Beardon, Spencer, and
Moracco (1989) confirmed that students perception of themselves, along with the school
experience, are paramount to school values and practices, since a negative school climate
could actually decrease a students motivation for learning.
Mowday, Porter, and Steers (1982) and Wynn, Carboni, and Patall (2007)
reported that schools with negative school climates had high teacher absenteeism and
turnover. Also, Haynes, Emmons, and Ben-Avie (1997) state that any setting that has a
negative climate, in which members do not relate well to each other, is a psychologically
unsound environment and contributes to poor mental health for all. In fact, in schools
that were run with an authoritarian, rule-bound manner, teachers were less likely to go
above and beyond contractual expectations for students and have less trust with their
administration (Tschannen-Moran, 2009).
Teachers: Emotional Aspects of School Climate and Student Impact
Echoing Borger (1985), Hoy (2008) stressed the importance of meeting teachers
emotional needs. Hoy posited that if a school has open, healthy and collegial interactions
among the adults in the school along with high academic expectations, that teachers on
the whole will feel empowered and effective. In fact, Murray (2005) suggested that the
ability of new teachers to build quality relationships with other adults in the school
setting (specifically administrators and colleagues) is a significant factor that molds their
decision on whether or not to remain in the teaching profession. Wynn, Carboni, and
-
18
Patall (2007) corroborated that good principal leadership and positive school climate are
significant factors in teacher retention.
Neither staff cohesion nor staff conflicts exist in a vacuum. Ninan (2006)
illustrated that children learn by example and that the moral fabric of a school is set and
nurtured by the teachers in the environment. Children view teachers values and
(consciously or unconsciously) emulate those behaviors. The manner in which teachers
treat other teachers is also recognized and internalized. Similarly, research has evidenced
that when teachers were more committed to their students, and went beyond typical
contractual expectations, there was a distinct, positive relationship with student academic
success (Tschannen-Moran, Parish, and DiPaola, 2006).
Similarly, McLoughlin, Kubick, and Lewis (2002) believed that to ensure that our
schools are safe, teachers play pivotal parts in communicating equal respect among all
students. They pointed out that teachers send this message to students in their classrooms
by modeling appropriate behavior, as well as teaching social skills and citizenship
directly.
Pantaleno (2007) noted that the Social Emotional Learning (SEL) movement
currently making its way into the public school curriculum (as witnessed by the recent
spate of character-education programs and social-emotion learning initiatives being
considered in state education legislation and supported by federally subsidized grants
[U.S. Department of Education, 2008]), will be the vehicle to teach emotionally
intelligent behavior and that important task will lie on the shoulders of our classroom
teachers. Of note is that results of these pilot programs indicated that successful
-
19
strategies included modeling character traits by all adults in the school (CASEL, 2008;
Haynes & Thomas, 2007; U.S. Department of Education, 2008).
The importance of having good role models for our children in schools cannot be
over-emphasized. Sandberg (2007), asserted that the most important action for schools is
to shift the focus from teaching just our childrens moral character to creating entire
school environments, (and all who function within that environment), that brim with high
moral character. She suggested that instead of solely teaching character-building lessons
in the classroom, that all adult school members: teachers, administration, and support
staff, build a moral climate within entire schools. In a similar fashion, Cohen (2009)
emphasized that to the extent that students feel safe, cared for, appropriately
supported, and lovingly pushed to learn, [ultimately] academic achievement should
increase (p. 186).
Trust Research in Schools
Several studies indicate that fostering relationships, particularly trust, among
colleagues in schools is not only of utmost importance, but indeed critical to the
development of a successful learning environment (Bulach & Malone, 1994, NWREL,
2003; Homana, Barber, & Torney-Purta, 2005; Hoy, Gage & Tarter, 2006; Tschannen-
Moran & Hoy 1998; Bryk & Schneider, 2002; Norton, 2008). It has also been suggested
that school leaders who demonstrate a strong trust in their faculty, have a reciprocal
trusting relationship with teachers, and parents. As a result, for the greater school
community, teachers will likely evidence greater professionalism, including a
stronger commitment to their students, [and] greater cooperation with colleagues
(Tschannen-Moran, 2009, p. 244).
-
20
School Administrators and School Climate Intervention
School leaders have a critical amount of influence as to a schools climate,
whether it be clearly expressed or implied (Cohen, et al., 2009). Although research
supports a series of positive consequences of educating students in a school with a
healthy climate, there are few states that have either a climate specialist or have
developed school climate policies (McCabe & Cohen, 2006). In a recent policy scan,
Cohen, et al. (2009) discovered that, of those states that indeed had school climate policy
written into their accountability plans (22), only six of them had done so in a limited
manner, and the other 22 had school climate subscribed to either special education, health
or school safety concerns. This lack of state support to enhance school climate
significantly curtails a school leaders ability to provide for positive changes.
NWREL (2003) offers suggestions to administrators to help foster a positive
school climate in their schools. They include: facilitating and modeling effective
communication, supporting the right and importance to have dissenting views, and
reducing teachers feelings of vulnerability between and among all faculty members.
They also suggest that administrators work hard to make relationship building of
paramount importance and to select a professional development model that encourages
relationship-building (NWREL, 2003). Similarly, Gibbs (2005) and Homana, Barber,
and Torney-Purta (2005), respectively, believe that building the capacity of school
personnel, as well as holding schools and their administrators accountable to teach
students to become more civic-minded, are the best methods to attain a positive school
climate which then encourages student achievement. Tschannen-Moran (2009) believes
that principals with a professional leadership style who vigorously encourages stronger
-
21
standards of trust throughout the school community will help instill a more professional
demeanor in their teachers. She suggested that, Creating conditions that strengthen
faculty trust in colleagues within the school may in turn allow greater faculty trust in
studentsto emerge (p. 243).
Systems Level Change and Continuing Professional Development
School psychologists are encouraged to be pivotal catalysts/supporters in system
level change in their schools (Curtis & Stoller, 2002; NASP, 2000). Curtis and Stoller
(2002) believe that, due to specialty training in human behavior and collaborative
relationships within the school environment, school psychologists are especially qualified
to work at this level in the school organization. They believe that collaborative
relationships within the schools requires, among other things, mutual respect and trust, in
which school psychologist are qualified to facilitate. Some examples they refer to
include: communication and effective listening skills, as well as team and consensus
building skills, in order to bolster mutual respect and trust among members. In a recent
(June, 2009) article published in the National Association of School Psychologists
(NASP) periodical, the Communiqu, school psychologists were encouraged to learn
about various communication styles among the people they work with and work in teams
with. They believe that doing this can help reduce conflict among team members
because the focus can be on what is being communicated, rather than the way it has come
across. The authors also suggested that being educated about the various communication,
personality, and personal styles, fosters a positive work environment, greater
understanding of others, and collaboration (Insert section, p. 2). In the same issue of the
Communiqu, Zuhkle and Mussman (2009) also describe the many different people from
-
22
various populations with whom they will need to communicate effectively with-- [I]t is
how these challenges and daily tribulations are handled that determines the value of
school psychologists to our constituents and personal career success (p. 15).
Both NASP (2006) and APA (2003) emphasize that school psychologists are
ethically bound and responsible for their own ongoing professional development. In a
guide provided to NASP leaders responsible for organizing members professional
development activities (Armistead, 2006), school climate was listed as an area in which
school psychologists could help create and maintain positive, supportive environments
for all school members. Additionally, Brown (2002), encouraged school psychologists to
utilize professional experiences to further long-range career goals, e.g., obtain
administration positions or expand their current roles.
School Psychologists as Effective Consumers of Research and Needs Assessment
According to the latest revision of the American Psychological Associations
Ethical Principles of Psychologists and Code of Conduct (2002), psychologists must:
(a) administer, adapt, score, interpret, or use assessment techniques, interviews, tests,
or instruments in a manner and for purposes that are appropriate in light of the research
on or evidence of the usefulness and proper application of the techniques and (b) use
assessment instruments whose validity and reliability have been established for use with
members of the population tested. When such validity or reliability has not been
established, psychologists describe the strengths and limitations of test results and
interpretation (American Psychological Association, 2002). These principles are reiterated by Keith (2002), in Best Practices in School
Psychology IV, all psychologist need to be effective consumers of research; that is,
-
23
they should be able to critique research that has implications for their practice and
incorporate the findings of that research into their practice (p. 100).
A diagnostic evaluation of the schools strengths and weaknesses in which school
psychologists may involve themselves with, often referred to as a needs assessment-- is
particularly useful for gathering info about the organization (Curtis & Stoller, 2002).
Nagle (2002), defines needs assessment in schools as, a systemic process of
collecting and analyzing data in order to identify needs and problems to be addressed in
program planning, development, and modification. Nagle goes further by emphasizing
important parts of conducting a needs assessment in a school: Determining what the exact
purpose and goal of the assessment is, monitoring how open to change the school is, and
working closely with school faculty and administration to ensure commitment to
proposed and agreed upon changes. He states, as schools become increasingly
involved in reform and restructuring activities (see U. S. Department of Education, 2008),
school psychologists will need system-based skills such as needs assessment.
Organizational Development, School Climate, and the Use of Team Building
There are ways to improve the learning environment at the school building level.
Although experts differ on measuring techniques, many agree that organizational
development and school climate improvement teams are successful interventions to
employ (Lindelow, Mazzarella, Scott, Ellis, & Smith, 1989). Yeager (1994) highlighted
the need for organizational development techniques in the schools:
If it is true, that the major hurdles to high organizational
performance are dysfunctional conflicts, poor communication, structured
rigidity, failure of members to know themselves and how they have an
-
24
impact on others, then organizational development techniques should be
designed to help to improve a [schools] effectiveness (p. 417).
Organizational development interventions, such as team building, help to
improve or alter the way the organization operates, and has been proven to be a
very effective and highly regarded intervention strategy (Yeager, 1994;
McNamara, 2006). Understanding the multidimensional components of team-
building is important in creating effective teams. Appreciating and nurturing
personal differences and addressing personal diversity, allow for an environment
where creative choices and decisions can occur (Rideout & Richardson, 1989, p.
62).
Team Building and Communication
An educational objective often overlooked in schools is a counseling component
that can include numerous personal growth opportunities for all members. Rome (1990)
noted that school climate becomes more positive when there is effective communication
and understanding among its members. Norton (2008) states that success in todays
schools, is possible if the environment supports: the generation of ideas, collaboration
and team effort from all members. Chandler, Kern, and Durodoye (1996) agree that an
essential element in an integrated team building approach to improving school climate is
communication and understanding, and [t]he future of education depends on innovative
integration and team building in schools (Rome, 1990, p. 42). Tschannen-Moran (2009) suggests, for example, that having teachers collaboratively plan instruction will
aid in their development and commitment towards their students, which may invigorate
them to go above and beyond what is typically expected of them. Organizational
-
25
development, staff development, and team building have all been successful methods of
increasing effectiveness in schools. Effectiveness in schools is in part, as noted earlier, a
result of a positive school climate. It is essential then, that school psychologists focus
their time and attention on this enhancement. It is universally understood that,
[s]tudents will be the ultimate beneficiaries of enhanced learning environments
(Chandler, Kern, & Durodoye, 1996, p. 44).
Statement of Problem
To this researchers knowledge, no cohesive source of critically reviewed school
climate measures exist that specifically assess the faculty-to-faculty component, or one
that also assesses school climate variables including: safety, teaching and learning, and
external environment. To engage in a comprehensive and time-consuming faculty
intervention without first choosing the most empirically supported and valid tool by
which to measure progress wastes both time and limited financial resources. Or worse, if
an ineffective/ inappropriate tool is used to measure broad-based school climate
constructs for pre-and post-intervention, misleading findings can lead to poor
intervention results. Also, the effort that personnel would expend on inefficient or
inaccurate projects would be crucial time spent on those that are supported and proven by
research.
Purpose
This study identified the most empirically supported broad-based school climate
measures that fit the following criteria: possesses a gauge of relationships (especially
faculty relationships), safety (physical and emotional), teaching and learning, and
external environment; have viewable test items (that this researcher is able to view), and
-
26
are direct measures (e.g. questionnaires) of primary and secondary education levels
(elementary through high school).
Research Question
Which are the most empirically supported broad-based school climate
assessments that fit the following criteria: possesses a gauge of faculty relationships;
safety; teaching and learning; relationships, external environment components; as well as
have viewable test items and are direct measures of primary and secondary education
levels?
-
27
Chapter Three
Methods
Procedures
The study included nine criteria review and eleven selection steps. Each
inclusionary criteria was considered a step in the sequence of instrument selection, and
there were two additional steps that were taken to complete this study. Table 1 reviews
the instruments bearing of the following components: Assessment of school climate
(Criteria 1), assessment of faculty relationships (Criteria 2), current research components
that measure school climate (Criteria 3), direct teacher measure (Criteria 4), viewable test
items (Criteria 5), current and representative normative samples (Criteria 6), published
technical characteristics (reliability and validity) (Criteria 7 & 8, respectively), and
school level (Criteria 9). Step 10 describes the expert panel procedure utilized in this
study, and step 11 will introduce and apply a ranking system and critical test review to
those measures that remain after criteria 9 has been applied.
Table 1. Criteria Presented in Sequential Order _____________________________________________________________________
Criteria 1: Literature Search
This will entail a search for measures that specifically assess school climate.
Criteria 2: Faculty relationship factor
This will entail a search for school climate measures that specifically assess faculty-to-
faculty relationships.
Criteria 3: Additional school climate variables.
-
28
Those that also assess these additional school climate factors: Safety; teaching and
learning; and external environment.
Criteria 4: Direct teacher measures.
Direct measures that include information that is gathered directly from those individuals
that work at the school.
Criteria 5: Viewable test items.
The ability to view individual items of the narrowed list of school climate assessment for
appropriate content.
Criteria 6: Current published instruments.
School climate measures published before 1990 would not adequately represent the
populations in which a measure would be recommended by this study.
Criteria 7: Reliability.
There are different types of reliability estimates that help the test user judge how
confident he/she can be in the amount of measurement error the test has. Adequate
reliability coefficients range from .70 or higher.
Criteria 8: Validity.
There are different types of validity estimates that help the test user judge if the test
assesses what it purports to assess.
Criteria 9: Both Primary and Secondary School Levels.
All assessment tools need their data to have been drawn only from faculty in primary
through secondary schools which, by necessity, excluded colleges/ universities,
preschools, clinics, hospitals, etc.
________________________________________________________________________
-
29
Step 1 (Criteria 1): Literature Search
First, to ensure that as many school climate measures that were available would
be found, a broad search of the literature was conducted. The following databases were
searched: Education Resources Information Center (ERIC) (1966-2009), PsychLit, the
Buros Institute of Mental Measurements Yearbook, the Mental Measurements Yearbook,
PsycARTICLES (EBSCO & APA), and PsychINFO. Proquest Psychology sites were
also searched, which include APA approved journals, an online search of school climate
tests published by ETS and PsychCorp. Additionally, as a catch-all, the World Wide
Web was searched using search engines (such as Google), using the keywords and
phrases: school climate, affiliation, faculty relationships, cohesion, communication,
collegiality, trust and morale. A search of the names of authors in the field of school
climate was also conducted. Following this strategy, 102 measures were found. In
addition, an informal questionnaire was presented to an expert panel of school climate
researchers via personal email asking them to rate the importance of several of the criteria
applied to the school climate measures in this study (Step 11; see Appendix A).
Step 2 (Criteria 2): Faculty relationship factor
The second step entailed a search for school climate measures that specifically
assessed faculty-to-faculty relationships. The quality of the relationships that teachers
with whom they work have been found to be important climate variables (Rutter et al.,
1979; Phi Delta Kappa study, 1980; Wynne, 1980; Anderson, 1982; Van Horn, 2003).
For this study, instruments were reviewed and retained if they possessed a faculty
(colleague) relationship component, i.e., gave the adult staff an opportunity to report on
-
30
their perceptions of relations between faculty members in the school environment they
worked in.
Step 3 (Criteria 3): Additional school climate variables
In this step, a table was created for the measures that meet the primary criteria for
review, i.e., those that also assess these additional school climate factors: Safety; teaching
and learning; relationships, and external environment (operationalized below in detail in
Cohen, 2006, p. 245):
[S]afety (physical, social-emotional); teaching and learning (quality of instruction,
expectations for student achievement, leadership, professional development and social-
emotional-ethical education); relationships (respect for diversity among students and
adults, school outreach to parents, school-home partnerships, morale), and external
environment (quality and structure, physical plant, structure of time and space).
These particular factors are supported in the research by Howard, 1987; Knoff,
2002; Lehr & Christenson, 2002; National School Climate Council, 2007; Stichter, 2008.
Likewise, Felner and Felner (1989) also noted the importance of sensitivity to cultural
diversity, and Gotffredson and Gottfredson (1985) emphasized that school is to be an
environment where students felt safe. Uline and Tschannen-Moran (2008) also
emphasized clearly that human comfort, pleasing appearance, adequacy of space,
functional furniture and equipment, a clean and orderly environment, and regular
maintenance effects occupants sense of well being and thus their ability to teach and
learn (p.4).
-
31
Step 4 (Criteria 4): Direct teacher measures
Lehr and Christenson (2002) and Freiberg (1999) reported that there are two
different kinds of measures of school climate: indirect and direct. Indirect measures are
those in which information is not directly gathered from informants but instead, data is
collected. Such data could include suspensions, discipline referrals, absenteeism, teacher
rate of retention, etcDirect measures, however, include information that is gathered
directly from those individuals that attend, work at, or have children that attend the
school. Questionnaires, in particular, are preferred as a survey method when large
numbers of individual responses are needed (Nagle, 2002). For the purposes of this
study, it was important that the original information was gathered directly from those
individuals associated with the schools. School climate measures in which teacher
questionnaires were utilized were selected for this study.
Step 5 (Criteria 5): Viewable test items
According to the Code of Fair Testing Practices (2004), Test users
shouldevaluate representative samples of test questions or practice tests, directions,
answer sheets, manuals, and score reports before selecting a test ( p. 5), and Rudner
(1994), You should gather the information you need to evaluate a test. ..[g]et a specimen
set from the publisher. It was determined that, for this study, it was important to have
the ability to view individual items of the narrowed list of school climate assessment in
order to evaluate each item.
Step 6 (Criteria 6): Current published instruments
According to the American Psychological Association, the American Educational
Research Association, and the National Council on Measurement in Education (1999),
-
32
normative samples must be of sufficient size and adequately represent the population it
purports to measure in order to support statements of validity. The instrument also needs
to have adequate norm samples in order for conclusions to be supported regarding its use
for its intended purpose. The normative samples need to include individuals for whom the
test was intended for (e.g., age, experience, and background (Rudner, 1994; Anastasi,
1988). However, Hunsley and Mash (2008) explain that certain psychometric elements
may not have bearing on an assessment (e.g., a program evaluation) and that
psychometrics in regards to assessments used in clinical practice for evidenced-based
assessments (e.g., of symptoms of anxiety or depression) are inherently different than
measuring the constructs of a non-living organization (like a school). Hunsely and Mash
also describe that clinical assessment usually uses both idiographic and nomothetic
instruments. Idiographic instruments track changes in individuals, whereas nomothetic
measures are more global and can track changes across populations. It follows logically,
then, that school climate assessments are nomothetic instruments that are used to track
changes across populations.
Regarding norms, Hunsley and Mash reported that regardless of the population
to which comparisons are to be made, [and] [I]deally, whether conducted at the
national level or the local level (p. 8), .at a minimum, clinicians need to be
provided with an indication of the quality and likely accuracy of the norms for a
measure (p. 9). Hunsley and Mash go on to define adequate norms as those that
consist of measures of central tendency and distribution for the total score (and subscore
if relevant) based on a large, relevant, clinical sample (p. 8), good norms as measures
of central tendency and distribution for the total score (and subscores if relevant) based
-
33
on several large, relevant samples (must include from both clinical and non-clinical
samples)(p. 8), and excellent as measures of central tendency and distribution for
the total score (and subscores if relevant) based on one or more large, representative
samples (must include data from both clinical and nonclinical samples) (p. 8).
Information from Bond (1996) was used as encouragement to strive for norm
samples within, at most, seven to ten years. In support, the Joint Committee (1999) also
emphasized the importance of renorming tests to be sure that accurate and appropriate
score interpretations (p. 59) can be obtained. For this study it was determined that
school climate measures with norms obtained before 1990 would not adequately
represent the populations in which a measure would be recommended by this study. This
was based on evidence that use of outdated norms in a measure is not best practice
(Reschly & Grimes, 2002). It may be necessary to note however, that psychological
constructs, such as IQ, may be more sensitive to outdated norms than school climate
predictor variables (Beaujean & Gulling, 2006). Norms for school climate instruments,
and their correspondence to current U.S. census statistics, is discussed further in the
Results section of this study.
Step 7 (Criteria 7): Reliability
It is important that test scores are minimally impacted by measurement error
and that they are constant from one administration to another. There are different types of
reliability estimates that help the test user judge how confident he/she can be in the
amount of measurement error the test has. The different types of measurement error are
estimated by:
1. Inter-rater reliability - errors due to differences in judgment between raters
-
34
2. Alternate-form reliability- errors due to how the individual is expected to rank
on an alternate form of the test
3. Internal consistency reliability - error that is due to content sampling,
4. Test-retest reliability- error due to inconsistent repeat performance after a
specific amount of time has passed (Anastasi, 1988; Joint Committee, 1999;
Rudner, 1994).
In regards to test-retest reliability, Hunsley and Mash (2008) noted, ...not all
constructs or measures are expected to show temporal stability (e.g., measure of state-like
variables, life stress inventories) (p. 10). This reflects the varying interpretations that
some school climate experts have regarding the potential impact (or lack of impact) of
test-retest has on school climate.
As for internal consistency, they reported that adequate internal consistency is
when the preponderance of evidence indicates a values of ...70 - .79, good internal
consistency is when a values [are] ...80-.89, and excellent internal consistency is
when a values [are greater than or equal to] .90 (p. 8). The criteria selection below will
explain how reliability will be rated in this study.
Step 8 (Criteria 8): Validity
There are different types of validity estimates that help test users judge if the
test assesses what it purports to assess. Predictive validity refers to how well a test can
correctly predict something that it is theoretically believed to be able to predict (Anastasi,
1988; Rudner, 1994). Content, construct and criterion validity are widely believed to
measure the same concept (Education Testing Service (ETS) Standards for Quality and
-
35
Fairness, 2002; Joint Committee, 1999). They refer to how well the test measures specific
content of interest and how well the test items represent the test content.
Hunsley and Mash (2008) report that adequate content validity is when the the
test developers clearly defined the domain of the construct being assessed and ensured
that selected items were representative of the entire set of facets included in the domain,
good content validity is when in addition to the criteria used for an adequate rating,
all elements of the instruments (e.g., instructions, items) were evaluated by judges (e.g.,
by experts or by pilot research participants, and excellent content validity is when in
addition to the criteria used for a good rating, multiple groups of judges were employed
and quantitative ratings were used by the judges (p.9). Hunsley and Mash (2008) also
stated that adequate construct validity is when some independently replicated
evidence of construct validity (e.g., predictive validity, concurrent validity, and
convergent and discriminant validity), that good construct validity is when there is a
preponderance of independently replicated evidence, across multiple types of validity
(e.g., predictive validity, concurrent validity, and convergent and discriminant validity)
and excellent construct validity is when in addition to the criteria for a good rating,
evidence of incremental validity with respect to other clinical data (p. 9).
In addition, multiple informants (considered another form of concurrent or
predictive validity) are utilized when measuring different perceptions of the same
construct(s). Contemporary studies commonly use two or three informants as data
sources. . one anticipates discordant reports. If [there is] no discordance, the
additional reports provide no new information. Multiple sources are used to provide data
either on independent variables (predictors), or dependent variables (outcomes), or both
-
36
(Horton, 2004). Particularly relevant to this study, multiple informant surveys have been encouraged to be utilized if rigorous and reliable data are to be used in providing evidence-based policy recommendations or reform in public sector
organizations (Kerr, Lunkenheimer, & Olson, 2007, Abstract). The criteria below will
indicate how validity will be rated in this study.
Step 9 (Criteria 9): School Level
All assessment tools needed their data to have been drawn only from faculty in
primary through secondary schools which, by necessity, excluded colleges/ universities,
clinics, hospitals, etc.The target audience for this research study consists of school
psychologists and administrators in primary and secondary school system; other
environments are beyond the scope of this study.
Step 10: Expert Panel Procedure
In addition to a critical review of the three instruments that met this studys
inclusionary criteria, a questionnaire was presented to an expert panel of researchers in
the field of school climate. Several prolific researchers in the area of school climate were
contacted via email and asked if they would participate in a dissertation study on school
climate. These particular researchers were selected according to either the frequency in
which their publications were cited in the school climate literature (at least four or more),
or their affiliation with a school climate research center. Nine out of nine potential panel
members responded favorably. Of the nine expert panel members, eight wished to be
identified in this study: Dr. Jonathan Cohen [Co-founder and Director of The Project for
Social Emotional Learning (PSEL) at Teachers College, Columbia University and the
Center for Social and Emotional Learning (CSEE). The CSEE is the creator of the
-
37
Comprehensive School Climate Inventory [CSCI]); Dr. John Shindler (Co-director of the
Western Alliance for the Study of School Climate, and the developer of the WASSC
Climate Assessment Instrument, WASSC); Dr. C. John Tarter (Professor, University of
Alabama, Department of Educational Leadership, Policy, and Technology Studies;
Author/co-author of several books and articles regarding school climate); Dr. N. Scott
Norton (Professor Emeritus, Arizona State University; Author of several books and
articles regarding organizational/school climate); Dr. Martha Alberg (Co-author of the
School Climate Inventory-Revised (SCI-R) and co-director of the Center for Research in
Educational Policy (CREP) at the University of Memphis in Tennessee); Dr. Megan
Tschannen-Moran (Associate Professor, The College of William and Mary in Virginia;
Author/co-author of several books and articles regarding school climate); Joshua
Freedman (Chief Operating Officer, Six Seconds, The Emotional Intelligence Network;
Co-Author of the Six Seconds Emotional Intelligence Assessment and the Organizational
Vital Signs Climate Assessment); and Dr. Carina Fiedeldey-Van Dijk (Senior Research
Scientist; a significant contributor to the development, psychometric validation, and
statistical analysis of the Six Seconds Emotional Intelligence Assessment (SEI), the
Organizational Vital Signs Climate Index and the Assessment of School Climate (ASC)).
(Please see Appendix C for a brief biography of each expert panel member).
Although the ninth panel member is also a prolific researcher and educator in the
field of school climate and provided extremely helpful feedback on the questionnaire,
more qualifications of the proposed study were needed in order for this persons
identification on the panel to occur.
-
38
Each panel members was emailed a questionnaire as an attachment. The
questionnaire contained twelve questions eliciting their perception of the level of
importance of some technical qualities; specifically, the reliability, validity, and norms of
school climate measures (See Appendix A for the questionnaire). The response set
included a five-point Likert-scale: 0-Not important at all, 1-Somewhat important, 2-
Moderately important, 3-Very important, 4-No opinion, 5- Do not know. Six of the nine
panel members answered each of the twelve questions, one member answered five of the
twelve questions with modified levels of importance than were given on the original
Likert-type questionnaire, and two members answered one of the twelve questions with
modified levels of importance than were given on the original Likert-type questionnaire.
In order to find out how much the raters agreed with each other, some modifications to
responses were made by this author (with rater permission). For example, three raters
gave a score of 2.8 (rather than 2-moderately important or 3-very important) to two
questions, this score was reassigned as a 3. These same three raters also gave a score
of 2.5; this score was reassigned as a 2. The rationale was that these score
reassignments would not significantly skew the agreement/non-agreement of the raters
responses. In addition, the following changes were made: Three raters responses were
discarded from the statistical calculation of interclass correlation if they used either 4-No
opinion (or did not answer the question), or 5-Do not know. The rationale was that these
two responses were not part of the ordinal scale: 0-Not important at all, 1-Somewhat
important, 2-Moderately important, or 3-Very important. As a result, only six of the nine
expert rater responses were utilized in the intraclass correlation calculation (ICC).
Step 11: Comparative ranking system
-
39
As each of the school climate measures reviewed in this paper have various
components and psychometrics obtained, and essentially, only one is needed to serve the
purpose of this study, a ranking system was developed. In determining this ranking
system, a point system was applied to three components: reliability correlation
coefficient; validity information, and current and representative (adequate) normative
samples.
Reliability. In examining the internal consistency, inter-rater, alternate-form, and
test-retest correlations for each measure, the following points were assigned: If the
average of the measures subscales had a correlation below 0.70, then zero points were
assigned; if the average of all subscale correlations was between 0.70 0.79, one point
was assigned; if between 0.80 0.89, two points were assigned; and if 0.90 or above,
three points were assigned. The same point assignment was followed for the faculty
relationship subscale. If there was evidence that test-retest, inter-rater, or alternate-form
reliability gathered for the measure, one point was assigned for each.
Validity. All the measures reviewed in this paper had some form of validity. One
point each was assigned if the following type of validity was applied in review: content,
construct, concurrent, convergent, divergent/discriminative, criterion, and/or predictive.
Anastasi and Urbina (1997) recommend that at least three forms of validity need
to be evidenced in an educational/psychological test: content description [content
validity], criterion prediction [criterion-related validity], and construct identification
[construct validity]. An additional point was added if three forms of validity were
demonstrated in the research of the measure(s). One point (for each informant group)
was added if the instrument utilized multiple informants.
-
40
Sample size, recency of publication/norms, and representative norms. Lastly,
sample size, recency of publication/norms, and representativeness of norms were
examined in this quantification of assessment tools. The following conversion was
utilized: Zero points for samples under 150, one point for sample sizes between 150-500,
two points for samples between 500 and 1,000, and three points for samples over 1,000
subjects. For recency of publication/norms: Zero points if the measure had norms before
1990, one point if the measure had norms from 1991-1998, and two points if the measure
had norms from 1999-present. If efforts were documented to cull a normative sample
representative of the U.S. Census, an additional point was also added.
-
41
Chapter Four
Results
The previous chapter focused on the cursory review of each of the climate
instruments culled from several decades of school climate research. A description of the
steps that followed was discussed in detail, as well as specific criteria that the measures
were held to for final critical review. In this chapter the data collected will be presented,
analyzed, and interpreted. Descriptive statistics, specifically reliability coefficients,
claims of validity and norm information, were examined for each of the final three
measures included in the critical test review.
In step one, the original list of school climate measures (see Table 1) were chosen
from the literature and resources if they were simply recognized/considered through title
and/or description to be tests/measures/questionnaires measuring school climate. No
other criterion was needed here. One hundred and two instruments were found.
Table 1 Step1: Literature Search ________________________________________________________________________ 1. Alaska Association of School Boards (AASB) School Climate and Connectedness
Survey (SCCS)
2. Academic Setting Evaluation Questionnaire
3. Affective Work Competencies Inventory
4. Assessment of School Climate, The
5. Assessment of School Needs for Low-Achieving Students: Staff Survey
6. Attitudes and Beliefs on Classroom Control (ABCC) Inventory
-
42
Table 1 (Continued) Step1: Literature Search ________________________________________________________________________ 7. Barclay Classroom Climate Inventory
8. Building Level Effectiveness Survey
9. California Health Kids Survey
10. California School Climate and Safety Survey
11. Character in Action Survey
12. CharacterPlus, The
13. Charles F. Kettering, Ltd. School Climate Profile
14. Chicago Public Schools Student Connection Survey
15. Civic Mission of Schools Self-Assessment
16. Class Activities Questionnaire
17. Classroom Environment Scale
18. Classroom Level Effectiveness Survey
19. Classroom Practices Inventory
20. Classroom and School Community Inventory
21. College Characteristics Index (CGI)
22. College Student Experiences Questionnaire
23. College and University Classroom Environment Inventory
24. Communication Climate Questionnaire
25. Comprehensive Assessment of School Environment
26. Comprehensive School Climate Inventory
27. Connecticut Effective Schools Questionnaire (CESQ)
-
43
Table 1 (Continued) Step1: Literature Search ____________________________________________________________________
28. Developmental Studies Center Child Development Project Scales
29. Diagnostic Assessment of School and Principal Effectiveness
30. Dimensions of Excellence Scales
31. Education, Training, Research Character Education Survey
32. Effective School Battery, The
33. Effective School Function Inventory
34. Effective Schools Survey
35. Emotional Climate Inventory
36. Elementary School Environment Survey
37. Elementary Student Opinion Questionnaire
38. Enjoyment of Class Scale
39. Environmental Assessment Technique
40. Glendale School Effectiveness Survey, The
41. High School Survey of Student Engagement
42. High Performance Learning Community Assessments (HiPlaces Assessments)
43. Illinois Quality Schools Index
44. Individualized Classroom Environment Questionnaire
45. Institutional Functioning Inventory
46. Instructional Climate Inventory for Teachers
47. Inventory of School Climate, The
48. K12 School Climate and Diversity Surveys
-
44
Table 1 (Continued) Step1: Literature Search ________________________________________________________________________
49. Learning Climate Inventory
50. Learning Climate Questionnaires
51. Learning Environment Inventory (LEI)
52. Liking for School Scale
53. Listening to Student Voices Self-Study Toolkit
54. My Class Inventory
55. My School Inventory
56. National Study of School Evaluation: Teacher Opinion Survey
57. Organizational Description Questionnaire-Revised (OCDQ-R)
58. Organizational Health Inventory (OHI)
59. Paces College and University Environment Scales
60. Parent-School Communities Questionnaire
61. Playground and Lunchroom Climate Questionnaire, The
62. Profile of a School, The
63. Psychological Sense of School Membership
64. Pupil Control Ideology/ Pupil Control Behavior
65. Quality of School Life Scale, The
66. Robustness Semantic Differential (RSD) Scale, The
67. Safe and Responsive Schools Survey
68. School Assessment Survey
69. School Attitude Measure
-
45
Table 1 (Continued) Step1: Literature Search ________________________________________________________________________
70. School as Caring Community Profile (SCCP) II
71. School Characteristics Inventory: Study of Instructional Improvement
72. School Citizenship Education Climate Assessment
73. School Climate Questionnaire
74. School Climate Survey (Yale)
75. School Culture Scale
76. School Description Inventory, the (SDI)
77. School Diversity Inventory, the (SDI)
78. School Discipline Climate Survey
79. School Effectiveness Questionnaire (SEQ)
80. School Environment Preference Survey
81. School Improvement Program Needs Assessment Questionnaires
82. School Level Environment Questionnaire (SLEQ)
83. School Participant Empowerment Scale
84. School Improvement Program Needs Assessment Questionnaires (Elementary
Schools, 1970)
85. School Survey of Interpersonal Relationships (SSIR)
86. School Work Culture Profile
87. Secondary School Moral Atmosphere Questionnaire, The
88. Social Outcomes Survey
89. Staff Development and School Climate Assessment Questionnaire (SDSCAQ)
-
46
Table 1 (Continued) Step1: Literature Search ________________________________________________________________________
90. Staff morale questionnaire (SMQ)
91. Statements About Schools Inventory
92. Student Satisfaction Inventory
93. Survey of Instructional and Organizational Effectiveness
94. Teacher Involvement and Participation Scale
95. (Tennessee) School Climate Profile, The
96. Thoughts About School Measure
97. Tribes TLV Assessment Questionnaire
98. Vessels School Climate Questionnaire
99. Western Alliance for the Assessment of School Climate (WASSC)
100. What Kids Can Do: Students as Allies in Improving Their Schools
101. WINSS School Climate Survey
102. World Health Organizations Psycho-Social Environmental Profile
_______________________________________________________________________
Second, in step two (Table 2), each of the 102 measures from step one were
reviewed and retained if they possessed a faculty component, i.e., gave the adult staff an
opportunity to report on their perceptions on the school climate of the environment they
worked in/relations between faculty members. Forty-seven measures remained after this
second inclusionary criteria was met.
-
47
Table 2 Step 2: Faculty Relationship Component ________________________________________________________________________
1. Alaska Association of School Boards (AASB) School Climate and Connectedness
Survey (SCCS)
2. Academic Setting Evaluation Questionnaire
3. Assessment of School Climate, The
4. Assessment of School Needs for Low-Achieving Students: Staff Survey
5. CharacterPlus (The)
6. CFK, Ltd. School Climate Profile
7. Civic Mission of Schools Self-Assessment
8. College Characteristics Index (CGI)
9. Comprehensive Assessment of School Environment
10. Comprehensive School Climate Inventory
11. Dimensions of Excellence Scales
12. Education, Training, Research Character Education Survey
13. Effective School Battery, The
14. Effective School Function Inventory
15. Emotional Climate Inventory
16. Glendale School Effectiveness Survey, The
17. Illinois Quality Schools Index
18. Institutional Functioning Inventory
19. Instructional Climate Inventory for Teachers
20. Inventory of School Climate, The
-
48
Table 2 (Continued) Step 2: Faculty Relationship Component ________________________________________________________________________
21. Learning Climate Inventory
22. National Study of School Evaluation (NSSE): Teacher Opinion Survey
23. Organizational Description Questionnaire-Revised (OCDQ-R)
24. Organizational Health Inventory (OHI)
25. Profile of a School, The
26. Robustness Semantic Differential (RSD) Scale, The
27. School Assessment Survey
28. School as Caring Community Profile (SCCP) II
29. School Characteristics Inventory: Study of Instructional Improvement
30. School Citizenship Education Climate Assessment
31. The School Diversity Inventory (SDI)
32. School Discipline Climate Survey
33. School Effectiveness Questionnaire (SEQ)
34. School Improvement Program Needs Assessment Questionnaires
35. School Level Environment Questionnaire (SLEQ)
36. School Survey
37. School Survey of Interpersonal Relationships (SSIR)
38. School Work Culture Profile
39. Staff Development and School Climate Assessment Questionnaire (SDSCAQ)
40. Staff morale questionnaire (SMQ)
41. Survey of Instructional and Organizational Effectiveness
-
49
Table 2 (Continued) Step 2: Faculty Relationship Component ________________________________________________________________________
42. Teacher Involvement Participation Scales
43. (Tennessee) School Climate Profile, The
44. Tribes TLV Assessment Questionnaire
45. Western Alliance for the Assessment of School Climate (WASSC)
46. WINSS School Climate Survey
47. World Health Organizations Psycho-Social Environmental Profile
_______________________________________________________________________
Third, in step three (Table 3), each of the 47 measures from step two were
reviewed again and retained if they possessed the additional school climate variables that
have been found in the research to be considerable predictors of positive/negative school,
i.e., 1) safety, 2) teaching and learning, and 3) external environment. Only eight measures
of the 47 possessed these additional school climate variables.
Table 3 Step 3: Additional School Climate Components ________________________________________________________________________ 1. CFK, Ltd. School Climate Profile
2. Comprehensive Assessment of School Environment
3. Comprehensive School Climate Inventory
4. Institutional Functioning Inventory
5. Learning Climate Inventory
6. School Effectiveness Questionnaire (SEQ)
7. (Tennessee) School Climate Profile, The
-
50
8. Western Alliance for the Assessment of School Climate (WASSC)
_______________________________________________________________________
In step four, the eight measures from step three were reviewed again and retained
if they were direct teacher measures of climate. All of the eight measured remaining
possessed at least one teacher questionnaire.
Table 4 Step 4: Direct Teacher Measures ________________________________________________________________________ 1. CFK, Ltd. School Climate Profile
2. Comprehensive Assessment of School Environment
3. Comprehensive School Climate Inventory
4. Institutional Functioning Inventory
5. Learning Climate Inventory
6. School Effectiveness Questionnaire (SEQ)
7. (Tennessee) School Climate Profile, The
8. Western Alliance for the Assessment of School Climate (WASSC)
_______________________________________________________________________ In step five (Table 5), the eight measures retained from step four were reviewed
and retained if the actual school climate measure was able to be obtained online or
secured via the publisher. Out of the eight measures, only four measures had individual
items that could be directly observed by this examiner. These included: The Charles F.
Kettering, (CFK) Ltd. School Climate Profile, the Comprehensive School Climate
Inventory (CSCI), The (Tennessee) School Climate Inventory-Revised, and the Western
Alliance for the Assessment of School Climate (WASSC). The primary reason for four
-
51
of the measures (Comprehensive Assessment of School Environment (CASE),
Institutional Functioning Inventory, Learning Climate Inventory, and School
Effectiveness Questionnaire [SEQ]) to have been dropped from this study for review, was
that they were no longer available/in print. This was determined by contacting the
publishers directly and inquiring.
Table 5 Step 5: Viewable Test Items ________________________________________________________________________
1. CFK, Ltd. School Climate Profile (via Howard book)
2. Comprehensive School Climate Inventory (via publisher directly)
3. (Tennessee) School Climate Inventory, The
4. Western Alliance for the Assessment of School Climate (WASSC)
_______________________________________________________________________ In step six (Table 6) each of the four measures from step five were reviewed and
retained if the measures were based on normative samples collected in 1990 or later.
This step is similar to step five because much of the reason why the tests were
unavailable/no longer in print was presumably due to the advanced age of the measure.
One measure was dropped at this step, although still available and in print, due to original
norms dating back to 1973 (Charles F. Kettering (CFK), Ltd. School Climate Profile,
1973).
-
52
Table 6 Step 6: Recently Published (1990 or higher) ________________________________________________________________________ 1. Comprehensive School Climate Inventory (via publisher directly)
2. (Tennessee) School Climate Inventory-Revised (via publisher directly)
3. Western Alliance for the Assessment of School Climate (via publisher directly)
_______________________________________________________________________
In step seven each of the three measures from step six were reviewed and retained
if they possessed statistical information regarding reliability in the climate measures
manual or research. All measures reviewed in step seven possessed this statistical
information and so all were retained for further review.
Table 7 Step 7: Reliability ________________________________________________________________________
1. Comprehensive School Climate Inventory
2. School Climate Inventory-Revised
3. Western Alliance for the Assessment of School Climate
_______________________________________________________________________
In step eight, each of the three measures from step seven were reviewed and
retained if they possessed statistical information regarding validity in the climate
measures manual or research. All measures reviewed in step eight possessed this
statistical information and so all were retained for further review.
-
53
Table 8 Step 8: Validity ________________________________________________________________________
1. Comprehensive School Climate Inventory
2. School Climate Inventory-Revised
3. Western Alliance for the Assessment of School Climate
_______________________________________________________________________
In step nine (Table 9), the three measures retained after steps 7 and 8 were
reviewed and retained if they measured only primary and secondary education settings,
i.e., no preschool, and university-level measures, as well as business-related/hospital-
related measures. All measures reviewed in steps seven and eight measured only
primary/secondary educational settings and so were retained for further review.
Table 9 Step 9: School-Level ________________________________________________________________________
1. Comprehensive School Climate Inventory
2. School Climate Inventory-Revised
3. Western Alliance for the Assessment of School Climate
_______________________________________________________________________ Step 10: Expert Panel Procedures
In step ten (Table 10), an informal questionnaire was presented to an expert panel
of school climate researchers. (See Appendix A for the questionnaire). The response set
included a five-point Likert-scale: 0-Not important at all, 1-Somewhat important, 2-
-
54
Moderately important, 3-Very important, 4-No opinion, 5- Do not know.
The results of the questionnaire responses are listed in Table 10.
Table 10 Expert Panel Questionnaire Responses __________________________________________________________________
1. How important is the construct of safety in a broad-based assessment of
school climate?
3 out of 9 --- Very Important
4 out of 9 --- Moderately Important (*one rater gave a score of 2.5; this
score was reassigned [with permission from the rater] to a 2 by the
author).
2 out of 9 --- I do not know
2. How important is the construct of teaching and learning in a broad-based
assessment of school climate?
6 out of 9 --- Very Important (*one rater gave a score of 2.8; this
score was reassigned [with permission from the rater] to a 3 by the
author).
3 out of 9 --- Moderately Important
3. How important is the construct of relationships in a broad-based assessment
of school climate?
9 out of 9 --- Very Important
4. How important is the construct of external environment in a broad-based
assessment of school climate?
3 out of 9 --- Very Important
-
55
4 out of 9 --- Moderately Important
2 out of 9 --- I dont know
5. How important is it that a broad-based assessment of school climate possesses
inter-rater reliability?
3 out of 9 --- Very Important
5 out of 9 --- Moderately Important
1 out of 9 --- Not Important At All
6. How important is it that a broad-based assessment of school climate possesses
test-retest reliability?
6 out of 9 --- Very Important
2 out of 9 --- Moderately Important
1 out of 9 --- Somewhat Important
7. How important is it that a broad-based assessment of school climate possesses
content validity?
8 out of 9 --- Very important
1 out of 9 --- Did not answer the question.
8. How important is it that a broad-based assessment of school climate possesses
concurrent validity?
7 out of 9 --- Very Important
2 out of 9 --- Moderately Important (*one rater gave a score of 2.5; this
score was reassigned [with permission from the rater] to a 2 by the
author).
9. How important is it that the assessments possess criterion validity?
-
56
6 out of 9 --- Very Important
2 out of 9 --- Moderately Important (*one rater gave a score of 2.5;
this score was reassigned [with permission from the rater] to a 2 by the
author).
1 out of 9 --- Somewhat Important
10. How important is it that a broad based assessments of school climate
possess norms within the last 7-10 years?
4 out of 9 --- Very Important
4 out of 9 --- Moderately Important
1 out of 9 --- Answered the question but without choosing a qualifier (and
instead mentioned that local norms seem to be more important than
national norms in the measuring of school climate, and that the nature and
purpose of the work being done would play a role).
11. How important is the sample size in a broad-based assessment of school
climate?
7 out of 9 --- Very Important
2 out of 9 --- Moderately Important
12. How important is it that the standardization sample of a broad-based
assessment of school climate match the U.S. Census?
2 out of 9 --- Very Important
3 out of 9 --- Moderately Important
1 out of 9 --- Not Important At All
1 out of 9 --- Answered the question but without choosing a qualifier (and
-
57
instead mentioned that it was unrealistic due to the lack of resources)
2 out of 9 --- I dont know * Scores given a 2.5 were rated downward to a 2 (Moderately important) and scores given a 2.8 were rated upward to a 3 (Very important)[with permission from this rater]. This was done to facilitate input and calculations of inter-rater agreement using a computer statistical program (please see next paragraph). __________________________________________________________________
To measure the consistency of raters agreement, an intraclass correlation
coefficient (ICC) was computed using SPSS 11.0.1 (Table 11). Intraclass correlation
coefficients are typically computed when the differences in ratings among judges is an
important consideration. The most common method when using ICC is one in which the
judges are assumed to be a random sample of a larger possible sample of judges, and one
in which each judge only rates the item being rated once (Howell, 2006). This was the
method chosen in reference to the expert raters because it was important to find out how
much the experts agreed on the relevance each item on the questionnaire possessed when
considering a school climate assessment. Responses for three of the nine raters were
discarded from the ICC because they utilized in their responses a non-ordinal rating that
could not be quantified for agreement purposes (either 4-No opinion or 5- I dont know).
Because the remaining six