web viewevery few years, international organizations conduct and publish results from international...
Post on 04-Feb-2018
221 Views
Preview:
TRANSCRIPT
Compare Globally, Interpret Locally:
International Assessments and News Media in Israel
Oren Pizmony-LevyTeachers College, Columbia University
ABSTRACTScholars have documented the global spread of international large-scale assessments of student achievement (ILSAs), such as the Trends in International Mathematics and Science Study (TIMSS), and the Programme for International Student Assessment (PISA). Research to date has focused on the influence of national and world characteristics on the likelihood of a country to participate in these assessments. This article investigates the role of local public discourse on the institutionalization of ILSAs “on the ground.” To do so, I use a sample of articles (n=97) from two newspapers in Israel. The analyses reveal three key patterns. First, public discourse about ILSAs developed gradually, and not immediately after the release of the first few reports. Second, policy makers are more active in the public discourse about ILSAs than other actors (e.g., students, teachers, parents, and education scholars). Third, public discourse about ILSAs constructed an achievement crisis that calls for change in policy/practice. These patterns facilitated the institutionalization of ILSAs in the Israeli context, beyond the contribution of transitional organizations.
Total word count: 8,525
Key words: globalization, international large-scale assessments, public discourse, news media
Compare Globally, Interpret Locally | Pizmony-Levy
Compare Globally, Interpret Locally: International Large-Scale Assessments of Student
Achievement and News Media in Israel
INTRODUCTION
Every few years, international organizations conduct and publish results from
international large-scale assessments (ILSAs) of student achievement. The International
Association for the Evaluation of Educational Achievement (IEA) coordinates the Trends in
International Mathematics and Science Study (TIMSS) and the Progress in International Reading
Literacy Study (PIRLS). TIMSS is administered every four years to students in 4th and 8th grades,
whereas PIRLS is administered every five years to students in 4th grade. In addition, the
Organisation for Economic Co-operation and Development (OECD) coordinates the Programme
for International Student Assessment (PISA), which is administered every three years to 15
years-old students. These ILSAs draw on data from large number of participants in many
countries/economies worldwide. More than half a million students in 65 countries/economies
took part in the most recent cycle of PISA (2012).
ILSAs are considered to be an advanced scientific and technical endeavor (National
Research Council, 2002). They involve complicated sampling techniques of schools and
students, development of reliable questionnaires and instruments, translation and adaptation to
various languages. Further, ILSAs are based on state-of-the-art statistical methods aimed to
produce comparable and valid scores (e.g., Item-Response Theory [IRT]). Results from each
cycle of ILSAs are reported in thick and dense volumes that include numerous charts, graphs,
and tables as well as technical notes.
2
Compare Globally, Interpret Locally | Pizmony-Levy
Nevertheless, shortly after the release of a new ILSA report, the news media begins with
frenzied attention. For some time, newspapers worldwide carry articles highlighting the
standings of nations in international ranking, and commenting on the implications of these
results. Following the release of PISA 2012, for example, the New York Times published an
article titled “American 15-year-olds lag, mainly in math, on international standardized test”
(Rich, 2013), and the Israel-based Ha’aretz published an article titled “Israeli student
achievement does not leave much room for optimism” (Kashti, 2013). These articles challenge
educational systems and schools to implement changes and mobilize stakeholders to improve the
quality of education.
In addition to describing, benchmarking, and monitoring education systems, ILSAs are
intended to enrich the public discussion about education (Howie and Plomp, 2005). Although
both IEA and OECD make the results of ILSAs available to the public for free (digital versions
of the reports are available on the organizations’ websites), they are not necessarily accessible to
the public due to their scientific/technical language and jargon. Given this, and the fact that
people are more likely to seek news and information from the news media, it is important to
understand how the media “filter and translate scientific information to the public” (Epstein,
1996:22). Moreover, leaders of IEA and OECD view public visibility of ILSAs as “key factor in
shaping the public policy debate (Wagemaker, 2011[2004]: 259; see also Wagemaker, 2013) and
argue that “the media had a key role in raising awareness of policy implications of the PISA
research results […]” (Schleicher, 2006: 267).
This article will focus on two research questions: (a) What role the news media play in
informing the public about the results of ILSAs and in framing these results as meaningful
indicators of educational quality? (b) Which actors participate in the translation, interpretation
3
Compare Globally, Interpret Locally | Pizmony-Levy
and filtering of results of ILSAs? Or which actors are granted “speech acts” or “claim making” in
the public discourse on ILSAs? To investigate these questions in a systematic fashion, I draw on
content analyses of articles published between 1996 and 2014 in two Israeli newspapers –
Ha’aretz and Yediot Ahronoth – on ILSAs (sample size is 97). Using data from both newspapers,
I am able to investigate the ways in which quality-elitist (Ha’aretz) and mass-popular (Yediot
Ahronoth) newspapers engage with ILSAs.
In so doing, this article speaks to global and transnational sociology, specifically World
Society theory that examines institutionalization of global rationalized models and scripts
(Meyer, Boli, Thomas & Ramirez, 1997). To date, scholars have focused on institutionalization
in terms of organizational practices that result from top-down processes. David H. Kamens and
Connie L. McNeely (2010), for example, examined the institutionalization of ILSAs by looking
at the immense growth in the number of countries participating in these assessments worldwide.
However, we know less about the incorporation of these organizational practices into the local
context and discourse (Phillips, Lawrence & Hardy, 2004). Public discourse on ILSAs provides a
good window to the “friction” (Tsing, 2005) between transnational organizations (i.e., IEA and
OECD) and local actors who interact with these organizations beyond organizational practice of
participation in ILSAs. In other words, analysis of the public discourse on ILSAs could shed
light on the social process of institutionalization “on the ground.”
BACKGROUND
Theoretical Framework
World Society theory holds that nation-states enact policy scripts diffused and legitimized by a
network of international organizations (Meyer et al., 1997). This theory, which is an extended
4
Compare Globally, Interpret Locally | Pizmony-Levy
version of Neo-Institutionalism, emerged as a response to the postwar rise in actor-centered
theories of modern societies (Meyer, 2010). Scholars writing within this framework have
observed the dramatic rise of national and supranational non-governmental organizations (NGOs
and INGOs) as well as international governmental organizations, such as the United Nations and
its affiliated organizations (IGOs; Boli & Thomas, 1999). These organizations promote global
scripts (models) for nation-states on “how to be better actors in light of general principles”
(Meyer, 2010: 7). These global scripts, as World Society theory posits, are transmitted to nation-
states through various mechanisms (e.g., intergovernmental conferences, professional
associations, international standards, and epistemic communities), and over time nation-states
adopt them.
The majority of World Society research has relied on cross-national time-series datasets,
using nation-states, organizations, and other artifacts as the unit of analysis. In the field of
education, we find research on the worldwide diffusion of mass schooling and the enormous
increases in enrollment in primary and secondary education (Meyer, Ramirez & Soysal, 1992),
and in post-secondary and higher education (Schofer & Meyer, 2005). Other examples include
research on the worldwide diffusion of curricular models (Meyer, Kamens & Benavot, 1992),
international schools (Suarez, Ramirez & Koo, 2009), and pedagogical movements such as
human rights education (Meyer, Bromley, & Ramirez, 2010) and environmental education
(Bromley, Meyer & Ramirez, 2011; Pizmony-Levy, 2011). More recently, scholars have
examined the emergence of national and international assessments of student achievement as a
new global policy script (Kamens 2015; Kamens & Benavot, 2011; Kamens & McNeely, 2010;
Pizmony-Levy, 2013).
5
Compare Globally, Interpret Locally | Pizmony-Levy
One major criticism of World Society theory is that it is overlooking local contexts:
discourses and actors. Scholars argue that local discourse and actors shape whether or not global
scripts become reality (see Anderson-Levitt 2003, 2012). Further, while World Society scholars
assume that global scripts are legitimized at the transnational level, they neglect the social
process of legitimacy that might take place “on the ground.” Drawing on literature in social
psychology and organizations, Johnson, Dowd and Ridgeway (2006) argue that legitimacy
processes, which are important to the survival and success of social objects, begin with local
validation. Legitimacy, they argue, is dependent on consensus among actors in the local arena as
well as links between new social objects and existing, widely accepted culture. In this paper, I
examine what happens to a global script – ILSAs – once it “lands” in a local setting.
Literature Review
Following the immense expansion of ILSAs, scholars have begun to explore public
discourse about these assessments. Schmidt (2003) provides a broad overview of reactions to
PISA 2000 in eight countries – Canada, France, the Netherlands, the Russian Federation,
Sweden, Switzerland, the United Kingdom, and the United States. Michelle Stack (2007)
examines U.S. media coverage of the 1999 TIMSS and both 2000 and 2003 PISA results. She
finds that the media endorses these assessments, with little critique, as valid indictors of system
performance. Further, Stack argues that the media interpreted the results of these assessments in
concert with business and political elites. Takayama (2008, 2009) shows how conservatives and
progressives in Japan interpreted and appropriated results from PISA to support their political,
economic, and cultural needs. He argues that shifts in the political and social conditions in Japan
affected media responses to the Japanese performance in PISA 2000 and PISA 2003 and thus
6
Compare Globally, Interpret Locally | Pizmony-Levy
created biases in the report. Xavier Pons (2012), drawing on six national case studies (Belgium,
France, Hungary, Portugal, Romania, and Scotland), find that media coverage of PISA focuses
on the ranking itself more than anything else. He demonstrates how local actors – including
policy makers and teachers unions – use PISA results for political instrumentalization. He also
points to missing voice of local scholars in the interpretation of PISA results. Dixon and her
associates (2013) compare reactions to ILSAs and biased coverage in four countries (Britain,
Finland, France, and Germany). They illustrate how media coverage includes negative reactions
to performance in ILSAs across countries, including Finland that performs high in these
assessments.
While important, extant studies have methodological and conceptual limitations. For
instance, Stack’s (2007) does not examine whether and how media coverage of ILSAs changed
over the years, while Dixon et al. (2013) overlooks the role of different local actors in the
construction of public discourse about ILSAs. Takayama (2008, 2009) and Pons (2012) examine
the ways in which local actors engage with ILSAs, but they focus on newspaper articles that
cover the release of new PISA reports. Finally, most studies draw heavily on qualitative analysis
of small samples of newspaper articles, thereby limiting the generalizability of the findings.
THE CASE: ISRAEL
Israel offers a fruitful case study for exploring public discourse on ILSAs. Despite its relative
newness among the world’s states (independent since 1948), Israel has developed rapidly and is
considered a Western industrialized democracy with a very high level of human development.
Israel’s Human Development Index in 2015 is .888 and it is among the top-twenty countries with
respect to this measure (UNDP, 2015). As such, the education system in Israel is well-developed.
7
Compare Globally, Interpret Locally | Pizmony-Levy
Public spending on education in Israel is 5.9 percent of the gross domestic product (slightly more
than in the United States 5.6 percent). The average years of schooling is 11.9, and the average
expected years of schooling is 15.7 years (in the United States these figures are 13.3 years and
16.8 years, respectively).
Israel is among the early participants in ILSAs, and its engagement began with the pilot
study, titled Educational Achievements of Thirteen-Year-Olds in Twelve Countries (Foshay et al.
1962). Therefore it is no surprise that Israel joined the new cycle of studies that was launched in
the mid-1990s. Because this article examines public discourse about ILSAs, it is important to
review the performance of Israeli students, as reported in the official reports of TIMSS, PIRLS,
and PISA (Figure 1). These assessments suggest that the performance of Israeli students is close
to the average of participating countries. The results vary by domain (mathematics, science and
reading literacy) and the year in which assessments were conducted. Results also vary by the
method of presentation (average score versus location in ranking tables). This is important
because fluctuation in the average scores is a result of changes in student achievement, whereas
fluctuation in the ranking is affected by changes in student achievement in other countries and by
changes in the composition of countries in the ranking tables (e.g., countries joining or leaving
the ranking tables).
Figure 1 – About Here
The overall performance of Israeli students in mathematics and science has not changed
since the mid-1990s. Specifically, TIMSS reports show that the average achievement scores did
not change significantly between 1999 and 2003, but declined significantly between 2003 and
2007. PISA reports further confirm that achievement in mathematics and science did not change
8
Compare Globally, Interpret Locally | Pizmony-Levy
between 2000 and 2009. Although the average achievement did not change over time, the
location of Israel in the ranking tables has been fluctuating.
As for reading literacy, international reports show improvement in the overall
performance of Israeli students since the early 2000s. PIRLS show statistically insignificant
improvement between 2001 and 2006 in the average achievement scores in reading literacy. This
pattern is further supported by international reports of PISA that show significant improvement
between 2000 and 2009. Yet, the location of Israel in the ranking tables, however, has decreased
over time.
These mixed results provide local actors with ample opportunities to engage with ILSAs.
On the one hand, drawing on the average achievement scores, one could characterize the Israeli
educational system as mediocre, but stable in terms of performance. On the other hand, drawing
on the ranking tables, one could characterize the Israeli educational system as going downward
with regard to performance. Feniger, Livneh and Yogev (2012) argue that the low achievement
of Israeli students in ILSAs were to be expected, given the economic and demographic
characteristics of Israel. They conclude that the public debate in Israel regarding the low
achievement in ILSAs “has been magnified out of all proportion.” This article systematically
analyzes the public discourse in Israel and the actors involved in its creation.
DATA AND METHODS
Data Collection
In order to examine the public discourse about ILSAs in Israel, I constructed an original
dataset of newspaper articles published between 1996 and 2010. I used the Bet Ariela Catalog for
Daily Newspapers (the Catalog), which is part of the Israeli Center for Digital Information
9
Compare Globally, Interpret Locally | Pizmony-Levy
Services, to identify relevant newspaper articles. The Catalog contains four news sources in
Hebrew, and it is the only one to cover publications from the early-1990s. The Catalog does not
include all news articles; rather it includes a sample of news articles that Bet Ariela’s librarians
deemed important or meaningful. I use all the news articles referenced in the Catalog. I searched
for articles that were published in two newspapers in Israel: Yediot Ahronoth and Ha'aretz. These
newspapers represent different journalistic style and cater to different audience. Yediot Ahronoth,
the most popular newspaper in Israel, is a daily tabloid and is considered to be the “people's
newspaper” aimed at mass consumption and favors simplistic, dramatic coverage. In contrast,
Ha'aretz is a quality-elitist broadsheet that targets a smaller, more educated audience who expect
more depth in their news coverage. In many ways, Yediot Ahronoth is comparable to USA
Today, whereas Ha'aretz is comparable to The New York Times. Using both newspapers,
therefore, provides a broad and inclusive perspective to the public discourse in Israel.
The search protocol includes the following key-words: “international studies,”
“international tests,” and “international achievement tests.” In addition, I searched for the full
title and acronym of each ILSAs (in English and in Hebrew): “Third International Mathematics
and Science Study” (TIMSS), “Trends in International Mathematics and Science Study”
(TIMSS), “Progress in International Reading Literacy Study” (PIRLS), and “Program for
International Student Assessment” (PISA). This search protocol yielded a dataset with 97
newspaper articles. More than three-fourths of the articles (78 percent) were published in
Ha’aretz and less than one-fourth of the articles (22 percent) were published in Yediot Ahronoth.
The dataset is limited in two ways. First, because the Catalog began indexing newspaper
articles in 1993, public discourse about early ILSAs is neither indexed nor accessible. Second,
because the Catalog is restricted to news articles published in Hebrew, the analysis to follow
10
Compare Globally, Interpret Locally | Pizmony-Levy
applies to public discourse among the Jewish majority in Israel more than it applies to public
discourse among the Arab minority in Israel who predominantly speak Arabic. Nevertheless, for
the purpose of this paper, the dataset provides an opportunity to explore public discourse in a
critical time point when the field of ILSAs is dramatically expanding worldwide.
Coding
In order to analyze the structure and content of the public discourse, I developed a detailed code
book. The code book was informed by open coding and theoretical coding. Open coding began
by reading the articles in the dataset and immersing myself in issues and happening in the data.
This preliminary reading resulted in a list of questions and ideas about the development of public
discourse about international assessments in Israel. Then, I began to identify themes and
categories that repeated across articles. For example, I found several articles where achievement
of Israeli students in ILSAs was discussed in the context of national security/defense. This
became a category titled “transnational practices and local logics.” I found articles that draw a
comparison between achievement of Israeli students in ILSAs in the 1960s and in the past
decade (a comparison that is not advanced by international organizations). This also became a
category titled “extending the global message.” I found articles that included critical comments
about ILSAs in general and in Israel specifically. This became a small, but telling, category titled
“critiques.”
The next step involved theoretical coding. For example, I used Steiner-Khamsi’s
typology (2003) for political reaction to ILSAs (i.e., scandalization, glorification, and
indifference). I searched for articles that used different linguistics and visual techniques to shame
– or scandalize – the Israeli government/society for the low achievement in ILSAs. Inspired by
11
Compare Globally, Interpret Locally | Pizmony-Levy
the literature on transnational transfer of educational policy (Steiner-Khamsi, 2004), I searched
for articles that discuss and/or call for policy reaction to results in ILSAs. Following research on
media coverage of contentious issues (Ferree, Gamson, Gerhards, and Rucht, 2002), I identified
a list of speakers (individuals and their organizational affiliation and roles) who were given
“speech acts” in the articles. The list includes seven groups: Ministry of Education (e.g.,
Minister, Executive Director, and Chief Scientist), politicians (e.g., Knesset Members), ILSAs
practitioners (e.g. national research coordinator), scholars in academy, social-movements
organizations, parents and students, educators (e.g., teachers and head teachers), and others. For
each articles I counted how many types of speakers were mentioned.
The final dataset includes three additional background variables: year of publication,
location of article (1=news section, 2=feature articles, editorials, opinions, or letters to the
editor), and objective of article (1=covers the release of new ILSAs report or focus on ILSAs, 2=
refers or cites ILSAs as part of a larger news story). All coding sheets were entered to data
management software.
12
Compare Globally, Interpret Locally | Pizmony-Levy
RESULTS
Descriptive Patterns
I begin by presenting descriptive statistics of the public discourse. Figure 1 shows the usage of
ILSAs-related search terms in two daily newspapers in Israel during the period under
investigation. The number of newspaper articles increased over the years, especially after 2003.
Between 1996 and 2002, the volume of articles covering ILSAs was small, with an average of
1.8 articles per year. Between 2003 and 2010, however, the volume of articles covering ILSAs
accelerated, with an average of 10.4 articles per year. Another measure for this growth is the
ratio between number of ILSAs reports released and the number of articles in each period. In the
first period (1996-2002) two ILASs reports were released and the ratio is 6.5 articles/per ILSA
report, whereas in the second period (2003-2010) seven ILASs reports were released and the
ratio is 11.8 articles/per ILSA report. It is clear from Figure 1 that the public discourse about
ILSAs was not automatically diffused to Israel, but rather it took several years (or cycles of
ILSAs) before it evolved in the Israeli context.
Figure 2 – about here
Almost all of the newspaper articles in the sample are original. Only two articles are
translated from a foreign source (both published in Ha’aretz). The first article, which is titled
“Who’s top?” and was originally published in The Economist in 1997, is devoted to TIMSS 1995
(then the Third International Mathematics and Science Study). After brief overview of the
ranking tables, the article describes the political context of TIMSS and the growing interest and
role of governments in international assessments. The second article, which is titled “The last
time you used algebra was in...” and was originally published in the New-York Times in 2004,
offers much more critical perspective on the meaning and implications of international
13
Compare Globally, Interpret Locally | Pizmony-Levy
assessments. The author argues that while American students are doing poorly in these
assessments, the United States is still a superpower in different domains related to mathematics
and science.
Figure 2 also shows an association between the release of ILSAs reports and the usage of
ILSAs-related search terms in newspaper articles. The number of newspaper articles is higher in
years when new ILSAs reports are released than in other years. This pattern is confirmed in an
independent t-test (M1=9.0, M2=3.4, p<.05). The public discourse about ILSAs is somewhat
dependent on the release of new ILSAs reports. Yet, it is important to note that while more than
half of the articles (54 percent) were published in the news section of the newspaper, the rest (46
percent) were published in other sections of the newspaper (e.g., daily magazine, editorial,
opinions, and letters to the editor). In other words, the public discourse about ILSAs is reactive
to transnational developments and expansive with regard to the contexts in which articles are
published.
Further, less than half of the articles (43 percent) actually cover the release of new ILSAs
reports or focus explicitly on ILSAs. These articles often include general information about the
organization that conducted the assessment, details about the domain and the target population, a
visual representation of the results, and key-points from press release of the Ministry of
Education. The following titles exemplify the orientation of these articles: “Study: level of eighth
graders in mathematics similar to the international average” (Ha’aretz 1996) and “Failed:
Evidence of poverty, international study finds Israeli students are ranked 31st out of 41 in reading
literacy, mathematics and science” (Yediot Ahronoth 2003). Other articles in this group include
interviews with key figures in organizations conducting ILSAs, critical comments about the
implications of international assessments, and commentary about the implementation of these
14
Compare Globally, Interpret Locally | Pizmony-Levy
assessments. For example, the article “The world is an educational laboratory” (Ha’aretz 2004) is
devoted to an interview with Dr. Ina Mullis, Co-Director of the TIMSS and PIRLS International
Study Center. This article directly corresponds with the original idea behind ILSAs as expressed
in the First International Mathematics Study (Husén, 1967). Although Dr. Mullis situates TIMSS
within the broader context of IEA research and refers to the idea of learning from differences,
she offers a new interpretation that emphasizes the importance of learning from ranking tables as
a way to reduce differences: “The world is an educational laboratory, and we are trying to look at
the countries with the best performance and learn from them so that others can learn and emulate
them.”
Majority of the articles (57 percent), however, only cite and refer to ILSAs reports. These
references are made by the author of the article as well as by interviewees who participate in the
article. For example, the article “One, two and we failed” (Ha’aretz 2003 ID16) presented the
story of the Israeli Foundation for Mathematical Achievement for All (IFMAA), a local non-
governmental organization. IFMAA was established in 2002 by a group of scholars committed to
improving and revising the mathematics curriculum in Israel. Throughout the article,
interviewees keep referring to ILSAs as the context and the main motivation behind the creation
of IFMAA. Another example is an article titled “Money yes, education no” (Yediot Ahronoth
2003), which was published on the occasion of the International Children's Rights Day. In this
article, the authors contrast the high governmental investment in education and the low return on
this investment as reflected in student achievement. Among other interviewees, the authors bring
quotes from members of the Citizens for Education Committee (local non-governmental
organization) citing the poor achievement of Israeli students in ILSAs.
15
Compare Globally, Interpret Locally | Pizmony-Levy
Voice in Public Discourse
Although ILSAs pertain to various stakeholders, not every stakeholder has an equal chance to
have a voice in the public discourse about these assessments. Because the interpretation and
translation of these assessments is done by actors that participate in the public discourse, it is
important to examine who are these actors. Figure 3 shows the representation of different
stakeholders in the news articles. Majority of the articles (83.5 percent) included at least one
speaker. The most common participants in the public discourse about ILSAs in Israel are the
Ministry of Education and scholars in the academy. Slightly more than two-fifths of the articles
(41.2 percent) include speakers from the Ministry of Education, and about two-fifths of the
articles (39.2 percent) include scholars affiliated with academic institutions. Other common
participants are individuals involved with the planning and implementation of ILSAs. Slightly
more than one-fifth of the articles (22.7 percent) include ILSAs practitioners. Interestingly, other
important and relevant stakeholders are represented less frequently in the public discourse about
ILSAs. For example, less than one-fifth of the articles (16.5 percent) include teachers or head
teachers, and one-eighth of the articles (12.4 percent) bring the perspective of parents and
students.
Figure 3 – About Here
The representation of different stakeholders in the public discourse about ILSAs varies
across the years, as illustrated in Figure 4. Between 1996 and 2002, when the public discourse
was small in volume (n=13), more than half of the articles include scholars affiliated with
academic institutions (53.8 percent). This figure, however, dropped to less than two-fifths (36.9
percent) between 2003 and 2010. In earlier news articles, scholars were the source of information
about ILSAs. For example, in the article “Survey: The achievement level of 8th graders in
16
Compare Globally, Interpret Locally | Pizmony-Levy
mathematics is similar to the international average” (Ha’aretz 1996 ID200) the author indicates
that “results were reported by the President of The Academic College of Tel Aviv-Yafo,
Professor Elazar Kochva.” Interestingly, this article was published before the official press
release of TIMSS 1995. And in another article “Not good in numbers” (Yediot Ahronoth 1997
ID2) is based on a lecture by Professor David Chen, Faculty of Education at Tel-Aviv
University.
Figure 4 – About Here
Except for the Ministry of Education, all other stakeholders saw a meaningful growth in
their representation in news articles about the about ILSAs. The representation of teachers and
head teachers, for example, grew from 7.7 percent to 17.9 percent of the articles. Also, the
representation of ILSAs practitioners grew from 7.7 percent to 25.0 percent of the articles.
Between 2003 and 2010 new stakeholders joined the public discourse about ILSAs: politicians,
parents and students, and social-movements organizations. These patterns suggest that ILSAs
became a legitimate and useful tool to make claims about education and schools in Israel.
Making ILSAs Taken-For-Granted
Local newspapers and stakeholders in the education domain contributed to the further
institutionalization of ILSAs in three interconnected ways. First, they embedded ILSAs in the
Israeli context by discussing these assessments according to dominant logics in Israel (i.e., the
logic of national security and defense). Second, they constructed ILSAs as reaction-worthy by
dramatizing the results and their implications, and by taking different actions in order to improve
the location of Israel in the international ranking tables. Third, they presented structured and
coherent discourse, with limited questions and critique of ILSAs themselves.
17
Compare Globally, Interpret Locally | Pizmony-Levy
In the following paragraphs I demonstrate these processes.
Embedding ILSAs in the Israeli Context
The public discourse about ILSAs embraced a critical tone and framed the performance of Israeli
students as an ongoing national crisis. Newspaper articles focused on the international “horse
race” and often overlooked more meaningful comparisons based on the average achievement
scores. This critical tone is reflected articles headlines: “Not good in numbers” (Yediot 1998
ID2), “Israel repeats a grade” (Yediot Ahronoth 2007 ID157), and “Failing” (Ha’aretz 2010
ID112). Out of the articles covering the release of new international assessments, three-fifths (60
percent) had a headline with a critical and negative tone. Other articles had a neutral headline (34
percent) or a positive headline (6 percent).
In addition to the international “horse race,” newspaper articles also paid attention to
domestic inequalities in student achievement. Slightly more than one-third of the articles (35
percent) point not only to differences between countries, but also to differences within Israel
based on students’ social location: gender, ethnicity (Jewish versus Arabs), socioeconomic
background, and sector (Jewish secular sector versus Jewish religious sector). For example, one
headline indicated that “In Israel: World's largest gap between boys and girls in eighth grade
mathematics” (Ha’aretz 1998 ID3) and another headline indicated that “Barely enough:
Alarming international test, low achievement in all subjects, gaps between rich and poor”
(Yediot Ahronoth 2010 ID159).
The significance of the Palestinian-Israeli conflict, coupled with the central role that the
military play within the Israeli culture and society, is implicated in the public discourse about
ILSAs. Local actors in Israel used the discourse of national security and national defense in three
18
Compare Globally, Interpret Locally | Pizmony-Levy
different ways. First, local actors argue that low performance in international assessments will
gradually translate into national security threat. The following excerpt suggests that low
performance in ILSAs will negatively affect the development of future cadre of scientists and
innovative technologists. In turn, this will have impact on the national capacity to develop
necessary weapons and intelligence systems:
“Israel was ranked in 39th place out of 53 countries […] This gives a hint of the nation's future ability to absorb, utilize and develop new technologies that are the primary ingredients underlying an economy's ability to grow, and of its future ability to operate and invent the advanced weapons and intelligence systems necessary for maintaining the existence of a country with our national security problems" (Ha’aretz 2005 ID56)
Second, local actors argue that the negative consequences of low performance in
ILSAs are more dangerous than the consequences of military conflict and terror. The
quotes in the following excerpt are especially telling because both speakers are affiliated
with the left-wing of the political spectrum in Israel. Although these speakers’ political
agenda emphasizes issues of peace and demilitarization of Israeli society, their statements
connect low performance in ILSAs with national defense/security discourse:
“Knesset Member Yossi Sarid, former Minister of Education and member of the Knesset Education Committee said: ‘the level of the Israeli student is a real existential threat, and Israel's ranking in education is more dangerous than all types of terrorism together’. […] Knesset Member Shimon Peres said that the harsh and severe report is prejudicial to the security of Israel” (Ha’aretz 2003 ID23)
Third, local actors contrast the centrality of defense/security issues in public agenda with
the marginality of education in the public agenda. They also compare the high priority and
resources allocated to defense/security policy with the relatively low priority and resources
dedicated to education policy. The following excerpts exemplify this pattern:
The Executive Director of the Ministry of Education: “This is a national problem that the Prime Minister needs to decide on. Enough talk about national security
19
Compare Globally, Interpret Locally | Pizmony-Levy
and wars. In the name of the threat of war, they are streaming budgets to defense items. It comes on the expense of our children's education. Only decision to make education a national priority will get us out of the mess” (Yediot Ahronoth 2003 ID26)
Constructing ILSAs as Reaction-Worthy
The public discourse about ILSAs constructed these assessments as reaction-worthy by further
scandalizing and dramatizing the results. Scandalization took place by comparing between
achievement of Israeli students in ILSAs in the 1960s and in the past decade. Almost one-tenth
of the articles (9 percent) framed the results to suggest that over the years the performance of
Israeli students in ILSAs eroded. Quotes such as “Who is to blame for the fact that Israeli
children fell from first place in the same test in 1964 to 28th, 25 years later?” (Ha’aretz 2003
ID16) and “Forty years ago Israel was the first in the world in an international test in
mathematics. Twenty years ago, Israel fell to the sixth place and in 1999 - was ranked 28th”
(Ha’aretz 2003 ID19) exemplify this theme. This frame was employed by various actors,
including the Ministry of Education, scholars in the academy, and journalists. Importantly, this
frame was not advanced by international organizations. Nor was this frame accurate, as
illustrated in the article “The myth behind the excellence in mathematics: The assumption that
Israel was once an educational superpower in mathematics is unfounded” (Ha’aretz 2007 ID50).
In an opinion column, the author argues that early ILSAs were based on a non-representative
sample of students in Israel, which excluded Arab schools and many Jewish schools serving new
immigrants. He concludes:
“The importance of international comparative studies for the local education system should not be minimized. However, for the discussion to be to the point and informed, it must be based on examined facts. The time has come to bid farewell to the myth that once, many years ago, when we were young and beautiful and innocent, we were a light unto the nations in mathematics education.”
20
Compare Globally, Interpret Locally | Pizmony-Levy
Further, scandalization took place by comparing the performance of Israeli students with
developing countries that are located in the periphery of the world system. By doing so, local
actors are able to manipulate public perception of performance in ILSAs by shaming the Israeli
government/society. This approach is reflected in the following excerpts in which the author
links between performance in ILSAs and location in the global chain of production. Countries in
which the educational system is ranked low international assessments are destined to be
exporters of cheap textiles and/or labor:
“[Israel’s ranking is] below every industrialized country participating in the study and below other countries much poorer than Israel, including Thailand and Romania from which we import laborers. Even the children of Malaysia, a country so poor that its textile workers are willing to work for miniscule wages that no Israeli would be willing to accept, have better grades than us” (Ha’aretz 2001 ID98).
“We ranked below all of the Western countries and below additional countries such as Malaysia, Thailand and Romania who provide us – still, though apparently not forever – with cheap textiles and labor” (Ha’aretz 2006 ID86).
Dramatization of the results took place by presenting manipulated visuals that emphasize
the low performance of Israeli students in ILSAs. Slightly less than one-third (29 percent) of the
articles included visual representation of results from ILSAs. Most of the visuals (20 out of 27)
presented some form of an international ranking table. Often, these visuals articles included
visuals with a partial list of participating countries that create a misleading impression that Israel
is at the bottom of the ranking table. For example, while 41 countries participated in TIMSS
1995, the visual that accompanied a news article in Yediot included only 20 countries. Given the
ranking of Israel (17th), this visual creates the impression that Israel is located closer to the
bottom of the list rather than to the middle of the list. This practice was used in different time
points and across assessments, as illustrated in Figures 5A and 5B.
21
Compare Globally, Interpret Locally | Pizmony-Levy
Figure 5 – About Here
The response of local actors, especially the Ministry of Education, to ILASs also
contributed to the construction of these assessments as reaction-worthy and thus to their
institutionalization. After all, responses suggest that ILSAs are meaningful, important and thus
deserve attention. I found two types of responses to ILSAs in Israel: (1) system-oriented changes,
and (2) ILSAs-oriented changes.
System-oriented changes. Following the release of ILSAs reports, newspaper articles
presented a wide range of initiatives by the Ministry of Education. These initiatives include
revision of national curriculum, establishment of system-wide standards, revision of textbooks,
and the establishment of the National Task Force for the Advancement of Education in Israel.
The following excerpt demonstrates this type of policy actions:
“The Ministry of Education announced yesterday the establishment of a committee to examine ways to improve the teaching of mathematics in schools [...] The timing is not coincidental. The release of Israeli student achievement in international assessments of mathematics and science is scheduled for tomorrow” (Ha’aretz 2000 ID137).
ILSAs-oriented changes. In several cases, newspaper articles presented a set of initiatives
that were intended to improve Israel’s location in the international ranking tables. In contrast to
system-oriented changes, these actions are less concerned with teaching and learning processes.
These initiatives include short-term preparation of students in cohorts that are scheduled to be
assessed by ILSA, and increasing students’ motivation to invest energy and time when
participating in ILSAs. In an article titled “The solution of the Ministry of Education for the poor
results in international knowledge tests: introducing a sense of national mission to the students”
(Ha’aretz 2009 ID115) the author reports on a memo sent by the Ministry of Education to
22
Compare Globally, Interpret Locally | Pizmony-Levy
students in which participation in ILSAs is situated in the context of national pride and
international competition:
“Dear Student, not every day you are given the opportunity to represent our country in an international study. This time it happens! […] All that is expected from you is serious attention to any question, and your effort to provide the best answer. Consistent, serious and responsible answers will allow obtaining a true picture of the achievements.”
Additional analysis shows that ILSAs-oriented changes of students was endorsed in times
when the Minister of Education was affiliated with the Likud Party, which is a right-wing
leaning party (Limor Livnat, 2001-2006 and Gideon Sa’ar, 2009-2013). This activity, however,
was criticized and rejected in times when the Minister of Education was affiliated with the Labor
Party, which is a left-wing leaning party (Professor Yael Tamir, 2006-2009). This pattern may
suggest that ideological orientation play a role in the way in which the Ministry of Education
engages with ILSAs.
Presenting Structured and Coherent Discourse
Overall, the public discourse about ILSAs in Israel was receptive. In many newspaper articles,
journalists and speakers engaged with ILSAs in un-critical fashion, thus constructing a coherent
discourse that contributed to the institutionalization of ILSAs. The lack of critical perspective is
evident in two articles that feature interviews with leaders of international organizations that
conduct ILSAs. Both articles recognize the immense power of ILSAs and professionals, but fail
to offer any critique whether this power is appropriate in a democracy. For example, in an
interview, Dr. Ina Mulis, Co-Director of the TIMSS and PIRLS International Study Center,
explains the process in which assessments are developed and the role of experts from different
countries in this process (Ha’aretz 2004 ID94). She also indicates the power of assessments and
their impact on sovereign countries:
23
Compare Globally, Interpret Locally | Pizmony-Levy
“Curricula are determined, of course, by each state individually, but the uniformity of tests requires matching the content [of the curricula] to questions […] In the 1980s [experts] decided to focus on the core issues, reading literacy, which is the basic proficiency needed for attaining knowledge, mathematics and science, which are the means for understanding the subjects that will contribute to growth in the economy, welfare and health. In the 1990s there were several series of consultations, which formed a consensus among experts about the level of knowledge required of the students”
Further, several articles pointed to the importance of continued participation in ILSAs. In
one article, for example, the Chief Scientist of the Ministry of education said “Not participating
in these tests [assessments] will kill the messenger […] I do not imagine that Israel decide not to
participate in international studies, even if it costs a quarter million" (Ha’aretz 2003 ID16). And
in another article, the Executive Director of the Ministry of Education reiterated this sentiment:
“Not participating in these tests [assessments] will cause severe damage to the image of Israel”
(Ha’aretz 2006 ID45). The latter excerpt echoes the notion of normative factor that is
emphasized in World Society Theory.
Nonetheless, I found several articles that challenge ILSAs and the public discourse that
has evolved around them. In a letter to the editor, “A student’s witness,” the author provides a
glimpse to students’ engagement (or lack of) with ILSAs:
“Results from tests, such as PISA, show the Israeli students are lazy, ignorant and stupid. As a former student who experienced these tests, I would like to argue that these tests do not represent the ability of students. The system works as follow: teachers explain the tests are [important] for the pride of Israel and will rank students from different countries worldwide. […] The thought that goes through the mind of the average high school student is this: this test is not affecting my report card. I do not mind the honor of Israeli students and Israeli pride worldwide. […] If I finish the test quickly, I can get out of here fast. I do not like tests. I marked answers randomly and left the classroom. I spent the next two hours reading a book. So did many of my friends. Is it really possible to take these tests as an indication of something?” (Ha’aretz 2009 ID300)
In this letter, the author describes the main reason for students’ disengagement with
international assessments and teachers’ efforts to motivate students (i.e., national pride).
24
Compare Globally, Interpret Locally | Pizmony-Levy
Further he argues that instead of spending time on assessments (that often measure
achievement in literacy) students choose to spend their time on actual reading. The fact
that this critical perspective is limited to the opinion section and is not echoed in the news
section of the newspaper highlights the broad acceptance of international assessments in
the public discourse.
25
Compare Globally, Interpret Locally | Pizmony-Levy
CONCLUSION
By analyzing a sample of newspaper articles published in two Israeli newspapers between 1996
and 2010, this study sheds light on the local reception of ILSAs, such as TIMSS, PIRLS, and
PISA. Rather than assuming that transnational developments (e.g., the creation of new cycles of
assessments) and country participation are sufficient indications for the legitimacy and
institutionalization of ILSAs, I analyze the ways in which news media and local actors enhance
the taken-for-grantedness of these assessments.
The descriptive part of this study finds that the public discourse about ILSAs in Israel
evolved over the years and increased dramatically in 2003, soon after the release of PISA 2002.
Although it is not possible to discern whether the discourse took-off because of PISA and its
affiliation with the OECD or due to the accumulation of social indicators about the education
system in Israel (i.e., two rounds of TIMSS), it is clear that after 2003 the public discourse
became more vibrant. Overall, the discourse includes not only coverage of new ILSAs report, but
also numerous references to and citations of ILSAs reports.
The participation of various stakeholders in the public discourse varies across the years.
In the first period (1996-2002), scholars in the academy dominated the discourse as they were the
main brokers of information about ILSAs. In the second period (2003-2010), however, scholars
in the academy were less central in the public discourse and most often provided explanations for
the results and/or commented about the possible implications of the results. This pattern echoes
similar developments at the transnational level. In other work (Pizmony-Levy et al., 2014), I
show how the transnational field of ILSAs emerged in the late-1950s from the collaborative
work of scholars in the academy (among them Benjamin Bloom and Torsten Husén). In the late-
1990s, however, the field was transformed into inter-governmental collaboration.
26
Compare Globally, Interpret Locally | Pizmony-Levy
The expansion of the public discourse about ILSAs in Israel created an opportunity for
new local actors to use these assessments as a tool to advance claims. The Ministry of Education
capitalized ILSAs results to support their request to break teachers’ strike. The Israeli Teachers
Union supported questionable pedagogical initiatives (that involved salary increases) as a way to
improve the location of Israel in the international ranking table. And other local actors leveraged
ILSAs results to justify the creation of common core curriculum across all schools.
The qualitative part of this study shows how the news media and local actors enhanced
the legitimacy and institutionalization of ILSAs in Israel. This cultural work necessitates three
simultaneous tasks or processes. The first task involves the “indigenization” of ILSAs in the
local context. Local actors did so by highlighting achievement gaps between social categories
that are well established in Israel (e.g., ethnicity and religiosity). That is, rather than focusing
solely on the international “horse race” and thus “flattening” differences within Israel, public
discourse paid attention to differences and inequalities between groups. Local actors also
employed jargon taken from public discourse on national security/defense to “make sense” of
ILSAs results and their implications. The second task involves the construction of ILSAs as
reaction-worthy rather as something to ignore. This task was accomplished by dramatizing and
scandalizing the results, and by taking actions aimed to affect the education system and future
results of ILSAs. The third task involves the creation of coherent public discourse that accepts
and supports ILSAs. This task was accomplished by marginalizing critiques and questions
regarding the efficacy and accuracy of these assessments.
These findings suggest some interesting directions for future research. First, will these
discursive patterns vary by countries? This question is beyond the scope of this study, but it is
important to consider that Israel is a relatively small polity. It may be that in larger and more
27
Compare Globally, Interpret Locally | Pizmony-Levy
complex polities – for example: the United States, Canada, and Germany – the discourse will
evolve differently. A comparative study is required could show how country-level characteristics
affect public discourse about ILSAs. Second, does public discourse matter for public opinion
toward education and schooling? This study might provide some interesting results that would
complement this article. Scholars have documented decrease in public trust in the education
system in the United States (Loveless, 1997; Smith, 2008); they conclude that this trend might be
related to the release of ILSAs reports. Yet, this hypothesis was not tested. Third, how do local
actors – Ministry of Education, politicians, ILSAs practitioners, scholars in academy, social-
movements organizations, parents and students, educators, and journalists – view their role in
disseminating ILSAs reports to the public?
This study of public discourse about ILSAs in Israel increases our understanding of what
happens to global rationalized models after countries adopt them. This study complements Neo-
Institutionalism (i.e., World Society Theory) by exploring the social process “on the ground” and
beyond formal commitment to global rationalized models. Findings suggest that public discourse
(i.e., ILSAs) emerge over time and not immediately after a country adopt global rationalized
models. This study enriches scholarship in sociology of education by exploring the politics
beyond large-scale datasets.
28
Compare Globally, Interpret Locally | Pizmony-Levy
REFERENCES
Dixon, R., Arndt, C., Mullers, M., Vakkuri, J., Engblom‐Pelkkala, K., & Hood, C. (2013). A lever for improvement or a magnet for blame? Press and political responses to international educational rankings in four EU countries. Public Administration, 91(2), 484-505.
Epstein, S. (1996). Impure science: AIDS, activism, and the politics of knowledge. University of California Press.
Feniger, Y., Livneh, I., & Yogev, A. (2012). Globalisation and the politics of international tests: the case of Israel. Comparative Education, 48(3), 323-335.
Ferree, M. M. (2002). Shaping abortion discourse: Democracy and the public sphere in Germany and the United States. Cambridge University Press.
Foshay, A. W. (1962). Educational Achievements of Thirteen-year Olds in Twelve Countries: Results of an International Research Project, 1959-1961. UNESCO.
Gamoran, A., & Porter, A. C. (Eds.). (2002). Methodological advances in cross-national surveys of educational achievement. National Academies Press.
Howie, S., & Plomp, T. (2005). International comparative studies of education and large-scale change. In International handbook of educational policy (pp. 75-99). Springer Netherlands.
Kamens, D. H., & McNeely, C. L. (2010). Globalization and the growth of international educational testing and national assessment. Comparative Education Review, 54(1), 5-25.
Meyer, J. W., Boli, J., Thomas, G. M., & Ramirez, F. O. (1997). World society and the nation-state. American Journal of sociology, 103(1), 144-181.
Phillips, N., Lawrence, T. B., & Hardy, C. (2004). Discourse and institutions. Academy of management review, 29(4), 635-652.
Pizmony-Levy, O. (2011). Bridging the global and local in understanding curricula scripts: The case of environmental education. Comparative Education Review, 55(4), 600-633.
Pizmony-Levy, O., Harvey, J., H. Schmidt, W., Noonan, R., Engel, L., J. Feuer, M., ... & Torney-Purta, J. (2014). On the merits of, and myths about, international assessments. Quality Assurance in Education, 22(4), 319-338.
Schleicher, A. (2006). Chapter 14: How can international organizations work with the media to manage the results of cross-national studies. In: Cross-National Studies of the Quality of Education: Planning Their Design and Managing Their Impact (pp. 265-275). International Institute for Educational Planning (IIEP) UNESCO.
29
Compare Globally, Interpret Locally | Pizmony-Levy
Stack, M. (2007). Representing School Success and Failure: media coverage of international tests. Policy Futures in Education, 5(1), 100-110.
Steiner-Khamsi, G. (2003). The politics of league tables. Journal of Social Science Education, 2(1).
Takayama, K. (2008). The politics of international league tables: PISA in Japan’s achievement crisis debate. Comparative Education, 44(4), 387-407.
Tsing, A. L. (2005). Friction: An ethnography of global connection. Princeton University Press.
Wagemaker, H. (2011[2004]).
Wagemaker, H. (2013). International Large-Scale Assessments: From Research to Policy. Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis (pp.11-33).
30
Compare Globally, Interpret Locally | Pizmony-Levy
Figure 1: Performance of Israeli Students in TIMSS, PIRLS, and PISA 1995-2009
Study Mathematics Science Reading
Ranking Score Ranking Score Ranking Score
TIMSS 1995a 21/41 522 23/41 524 -- --
TIMSS 1999 29/39 466 27/39 468 -- --
TIMSS 2003 19/46 496 23/46 488 -- --
TIMSS 2007 24/49 463* 25/49 468* -- --
PIRLS 2001 -- -- -- -- 23/35 509
PIRLS 2006 -- -- -- -- 31/40 512
PISA 2002 [Reading]b 31/41 433 33/41 434 30/41 452
PISA 2006 [Science] 40/57 442 39/57 454 40/57 439
PISA 2009 [Reading] 41/64 447 41/64 455 36/64 479*
Notes:* Significant difference between this year and previous year a In TIMSS 1995, Israel sampled students in both fourth and eighth grade. In subsequent
assessments, however, Israel sampled only students in the eighth grade. Due to unapproved sampling procedures in TIMSS 1995, comparison between this assessment and later assessments is not available in official publications of IEA.
b Similar to ten other non-OECD countries, Israel joined the first wave of PISA in 2002.
31
Compare Globally, Interpret Locally | Pizmony-Levy
Figure 2: Number of Articles Published, by Year
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20100
2
4
6
8
10
12
14
16
18
20
0
1
12
3
0
4
21
19
9
3
5
17
5
12
14
Press Release Total
Num
ber o
f Arti
cles
32
Compare Globally, Interpret Locally | Pizmony-Levy
Figure 3: Representation of Stakeholders in Newspaper Articles
33
Compare Globally, Interpret Locally | Pizmony-Levy
Others
No Speaker
Social-Movements
Parents and Students
Politicians
Teachers and Head Teachers
ILSAs
Academy
Ministry of Education
22.7%
16.5%
6.2%
12.4%
13.4%
16.5%
22.7%
39.2%
41.2%
Figure 4: Representation of Stakeholders in Newspaper Articles, by Time Period
34
Compare Globally, Interpret Locally | Pizmony-Levy
Others
No Speaker
Social-Movements
Parents and Students
Politicians
Teachers and Head Teachers
ILSAs
Academy
Ministry of Education
23.1%
15.4%
0.0%
0.0%
0.0%
7.7%
7.7%
53.8%
38.5%
22.6%
16.7%
7.1%
14.3%
15.5%
17.9%
25.0%
36.9%
41.7%
2003 to 2010 (n=87) 1996 to 2002 (n=13)
35
Compare Globally, Interpret Locally | Pizmony-Levy
Figure 5: Visual Representation of ILSAs Item A: TIMSS 1995 Item B: TIMSS 2007
36
Compare Globally, Interpret Locally | Pizmony-Levy
Appendix Table 1: Ministers of Education in IsraelBegin End Minister of Education Political Party Leaning
June 1993 June 1996 Amnon Rubinstein Meretz Left
June 1996 January 1998 Zevulun Hammer Mafdal Right
February 1998 June 1999 Yitzhak Levy Mafdal Right
June 1999 June 2000 Yossi Sarid Meretz Left
June 2000 March 2001 Ehud Barak Labor Left
March 2001 January 2006 Limor Livnat Likud Right
January 2006 March 2006 Meir Shitrit Kadima Center
March 2006 March 2009 Yuli Tamir Labor Left
March 2009 March 2013 Gideon Sa'ar Likud Right
37
top related