journal of research in science teaching volume 50 issue 7 2013 [doi 10.1002_tea.21090] price, c....

JOURNAL OF RESEARCH IN SCIENCE TEACHING VOL. 50, NO. 7, PP. 773801 (2013)

ResearchArticle

Changes inParticipants ScientificAttitudes andEpistemologicalBeliefsDuring anAstronomicalCitizenScienceProject

C. Aaron Price1 and Hee-Sun Lee2

1American Association of Variable Star Observers, Museum of Science and Industry, Chicago,

Illinois2University of California, Santa Cruz, California

Received 21 March 2013; Accepted 11 May 2013

Abstract: Citizen science projects provide non-scientists with opportunities to take part in scientific

research. While their contribution to scientific data collection has been well documented, there is limited

research on how participation in citizen science projects may affect their scientific literacy. In this study, we

investigated (1) how volunteers attitudes towards science and epistemological beliefs about the nature of

science changed after sixmonths of participation in an astronomy-themed citizen science project and (2) how

the level of project participation related to these changes. Two main instruments were used to measure

participants scientific attitude and epistemological beliefs and were administered before they registered for

the program and six months after their registration. For analysis, we used pre- and post-test data collected

from 333 participants who responded to both tests. Among them, nine participants were randomly chosen for

interviews. Participants responses were analyzed using the Rasch Rating Scale Model. Results show that

overall scientific attitudes changed positively, p < 0.01. The change was strongest in attitudes towardsscience news and citizen science projects. The scientific attitudinal change was related to participant social

activity in the project. There was a negative change in their evaluation of their knowledge. The interviews

suggest that this is due to a greater appreciation for what they have yet to learn. Epistemological beliefs about

the nature of science significantly improved from the pre- to the post-tests, p < 0.05. Overall, we foundvolunteers participation in social components of the programwas significantly related to their improvement

in scientific literacy while other project participation variables (such as amount of data contributed to the

project)was not.# 2013 Wiley Periodicals, Inc. J Res Sci Teach 50: 773801, 2013

Keywords: citizen science; informal science; nature of science (NOS); science literacy

Most people spend the majority of their lives outside of school, yet science learning in non-

school settings is often overlooked (National Research Council [NRC], 2009). Informal learning

opportunities can support lifelong learning (Dierking, Falk, Rennie, Anderson, &Ellenbogen, K.,

2003; Falk, Storksdieck, & Dierking, 2007), engage underrepresented populations in science

(Center for Informal Learning and Schools [CILS], 2005) and create personal relationships with

science (NRC, 2009) in ways different from formal science education opportunities. Informal

science education is a rapidly growingfield of research.Recent calls formore research on informal

science learning have been made by the National Science Board (NSB, 2008) and the National

ResearchCouncil (NRC, 2009).

Contract grant sponsor:National ScienceFoundationDRLaward;Contract grant number: 0840188.

Correspondence to: C.AaronPrice;E-mail: [email protected]

DOI10.1002/tea.21090

Publishedonline 11 June 2013 inWileyOnlineLibrary (wileyonlinelibrary.com).

# 2013 Wiley Periodicals, Inc.

Citizen science, defined as research collaborations between scientists and volunteers (Cornell

Ornithology Lab, 2009), is an increasingly popular venue for informal science education

(Cohn, 2008). While the scientific contributions of citizen science participants have been

documented, such as contributing to the large-scale database of migrating bird populations

(Hand, 2010) and discovering new galaxies (Christian, Lintott, Smith, Fortson, &

Bamford, 2012), there is limited research addressing how citizen science projects impact

volunteers science literacy (Conrad & Hilchey, 2011; Mueller, Tippins, & Bryan, 2012;

Silvertown, 2009).

In this study, we investigate volunteers involved in an astronomical citizen science project

calledCitizen Skywhere participants collaboratewith scientists tomonitor and analyze data about

a bright variable star. Aligned with Miller (1983, 1998, 2004), we use a civic oriented scientific

literacy framework and measured two scientific literacy elements called attitudes towards

science-related activities and epistemological beliefs about the nature of science. The research

questions of this study are:

How did volunteers attitudes towards science-related activities and epistemologicalbeliefs about the nature of science change over six months of participation in an

astronomy-themedcitizen science project?

Howdopatterns of project participation account for these changes?

We first introduce the role citizen science plays in informal science education and categorize

citizen science projects in terms of their participants level of involvement. We then describe

Citizen Sky as our research context and provide rationale for measuring scientific literacy as an

important outcome for citizen science projects. Next, we characterize our research methods

including subjects, test designs, data collection and analysis. In the results section, we compare

pre-/post-test differences in volunteers attitudes towards science-related activities and epistemo-

logical beliefs about the nature of science and use interview data to explain some of those

differences. Finally,we discuss findings and implications for citizen science participants, program

designers, and researchers, alongwith limitations of the study.

Citizen Science and Science Education

Citizen science projects, at their core, provide an organized venue for non-scientists to take

part in scientific endeavors ranging from passive (such as running a computer screen saver to

process scientific data) to active engagement (such as amateur astronomers searching for and

discovering newplanets). The spectrumof citizen science projects has been categorized intomany

different and often overlapping categories (Brandt, Shirk, Jordan, Ballard, & Tomasek, 2010;

Ely, 2008; Wiggins & Crowston, 2011). Citizen science can also be seen as a type of

crowdsourcingthe process of using an extremely large number of participants to accomplish a

focused task (Howe, 2006). Some citizen science projects, such as the large-scale GLOBE project

(GLOBE, 2011; Penuel &Means, 2004), take place in classrooms. However, this study is focused

on out-of-classroomprojects as their implementations are considerably different due to the lack of

environmental controls and limitations on instructional strategies.

Most citizen science projects are driven by their scientific goals with little attention to their

educational impact on participants (Mueller et al., 2012). As a result, there are a limited number of

educational research studies available in the literature. The vast majority of published articles are

descriptions of projects and/or their targeted scientific accomplishments. This need for empirical

research drives our research reported in this paper. With limited research about participants

Journal of Research in Science Teaching

774 PRICE AND LEE

learning through citizen science projects, there is lack of evidence in guiding the design of future

citizen science projects (Jordan,Ballard,&Phillips, 2012).

Citizen science plays an important role in the expanding field of informal science

education (NRC, 2009; Ucko, 2010) and is seen as a leading trend in new projects funded under

the informal science education program at the National Science Foundation (NSF, 2010). It

allows participants to get involved in many different scientific practices while remaining

focused on a single goal or outcome (Roth & Barton, 2004). It can often promote sustained

engagement over decades of a persons life (Price & Paxson, 2011). These characteristics

resonate well with free-choice learning, a critical element in development of lifelong scientific

literacy (Falk & Dierking, 2010; Falk et al., 2007). At the same time, this flexibility creates

unique challenges in sustaining participant interest (Nov, Arazy, & Anderson, 2011a),

balancing the reality of data collection in uncontrolled environments with the fidelity to

established scientific standards (Fore, Paulsen, & OLaughlin, 2001; Ottinger, 2010) and, for

education researchers, developing rigorous research methodologies that are not intrusive into

the participant experience.

One of the unique characteristics of citizen science is in its focus on individual agency.Unlike

traditional laboratory-style science education programs, citizen science projects allow partic-

ipants to build their own view of the project rather than seeing it exclusively through the

eyes of science (Roth & Barton, 2004). It is also an example of personal-curiosity science,

a term coined by Aikenhead (2005) to describe scientific learning that is of personal interest of

the student. Citizen science, with no ties to any classroom and no direct profit for the participant,

relies on this personal interest to drive participation and sustain activity. Aikenhead (2005)

believes that this type of personal connection is central to the success of science education writ

large.

This individual empowerment increases the value of social opportunities in citizen

science projects. Since there is no formal classroom or physically shared space, it is difficult to

build a community of practice among fellow participants. Communication between project

scientists and participants can be challenging (Greaves, 2012). The scientists often have to

serve as the teacher or mentor, a role for which they may or may not be trained, based on very

little interaction with participants. This sometimes leads to a topdown flow of instructions

from scientists to participants without requiring much independent thought and contribution

by the participants (Ely, 2008; Mueller et al., 2012). The Internet has helped to overcome this

barrier by making communication easier, at least for projects that are based online. An

increasing number of online citizen science projects have turned to asynchronous discussion

forums to create such virtual communities of practice among scientists and participants

(Khare, Zevit, & Shirk, 2012a; Raddick et al., 2010; Roy et al., 2012). Other projects have

begun partnering with museums and science centers to allow participants to interact directly

with project scientists (Khare, Zevit, & Shirk, 2012b; Time Out Chicago, 2012). As citizen

science increases in popularity, both in terms of diversity of projects and number of

participants, finding creative ways to build supportive communities becomes more and more

important.

The role of their participants differs substantially between citizen science projects, which

may determine the type and quality of participants experience and the learning outcomes after the

projects are completed. The Center for the Advancement of Informal Science Educations Public

Participation in Scientific Research inquiry group proposed three models of citizen science

projects based on the participation level of the volunteers (Brandt et al., 2010). The models are

idealized in that not all citizen science projects perfectly fit into a specific category, but they are

useful as benchmarks along the continuumof citizen science projects.


CITIZEN SCIENCE LITERACY 775

The Contributory Model

Contributory citizen science projects use participants mostly as data collectors in distributed

networks. This is the most common type of citizen science project (Karrow & Fazio, 2010). We

further divide this category into two types of contributory models because the participant

experience can be significantly different based on a single factor: whether they were actively or

passively contributing data to the project.

Passive ContributoryModel:After the initial recruitment phase, participants are asked tomonitor equipment that automatically collects data and transmits them to a central

repository. One of the most successful of projects in this type is the Berkeley Open

Infrastructure for Networked Computing (BOINC; Anderson, 2003) where the typical

participant runs a computer screen saver to process project data.

Active Contributory Model: Projects in this category actively engage participants in theprocess of data collection and/or data processing. Participants are required to make

decisions such as how often to collect data and when to deviate from suggested protocol.

The most popular projects in this category involve monitoring wildlife, including birds

(Brossard, Lewenstein, & Bonney, 2005; Evans et al., 2005; Wee & Subaraj, 2009),

insects (Howard & Davis, 2004; Phillips, 2008) and turtles (Somers, Matthews, &

Carlone, 2009). But it is also common in astronomy (Ferris, 2002; Percy, 1999) and

meteorology (Cifelli, 2005).

To locate empirical studies on how the contributory model impacts citizen science

participants, we used Google Scholar to look for a variety of specific and vague terms. Examples

are scientific literacy citizen science, scientific literacy informal science education, citizen

science education research, citizen science studies, and finally, simply citizen science by

itself. However, our search did not lead to published empirical educational studies of projects

within thismodel.

The Collaborative Model

In collaborative citizen science projects, participants are involved in developing descriptions

and explanations, or performing basic forms of initial analysis. The most popular example of this

type of model is the Galaxy Zoo project. The participants categorize photographic data to classify

galaxies into groups based on their appearance, and later their results are further analyzed by

professional astronomers. In addition, volunteers are encouraged to comeupwith explanations for

anomalous galaxy shapes. One volunteers comments on a picture led to the famous discovery of a

uniquegalaxy structure known asHannysVorweep (Cardamore et al., 2009).

Most of the citizen science empirical research literature involves projects that belong to the

Collaborative Model. In a study of The Birdhouse Network, researchers measured change in

scientific attitudes, understanding of the scientific process and the knowledge of birds by project

participants and found no change in attitudes or understanding of the scientific process (Brossard

et al., 2005). However, Brossard et al. (2005) did detect an increase in knowledge of ornithology.

In another ornithology project called The SeedPreferenceTest, researchers analyzed communica-

tion between participants and project organizers to look for evidence of scientific thinking

(Trumbull, Bonney, Bascom, & Cabral, 2000). While the researchers at the project did find

evidence of scientific thinking in the communication, they could not isolate the project impact

from other influences due to their study design. In addition, the researchers did not find any

relationship between scientific thinking and the amount of data contributed by the participant.

Another study analyzed e-mail communications between participants and project staff (Evans


776 PRICE AND LEE

et al., 2005) and found that changes in participants knowledge of birds were not apparent in email

communications butwas indicated through interviewswith a sample of the participants.

The Co-Created Model

The co-created model is sometimes referred to as participatory action research (Cornwall

& Jewkes, 1995). In this model, volunteers do everything from defining research questions to

publishing results. These more demanding citizen science projects are often used as examples of

distributed thinking (Hand, 2010). This is a relatively new category pioneered by groups such as

the Bossa project (BOINC, 2009). There are relatively few active projects in this field (Bonney

et al., 2009; Curren, 2013). This category is likely to grow (Roy et al., 2012) and there have been

calls for involving citizens in these types of more authentic research roles (Cooper, Dickinson,

Phillips,&Bonney, 2007; Lakshminarayanan, 2007;Wright, 2010).

We found some empirical studies about the scientific output of co-created model projects on

participants but none about their educational impact. One group of volunteers was trained to both

take samples and do their own analysis (alongside professionals who did their own independent

analysis) of water quality in a river shed over 3 years, with published results of their analysis

(Wilderman, 2004). And recall in the Galaxy Zoo project one participant discovered their own

galaxy type to great fanfare (Wright, 2010). That particular participants initiative elevated their

experience to one similar to the co-created model in that they discovered anomalous data,

investigated it and published the results (with professional guidance). However, there has been no

published studies about how these projects, or any other Co-Created project, impacted learning.

The focus has been on the scientific results.

Research Context: The Citizen Sky Project and e AurigaeCitizen Sky (www.citizensky.org)was a 3-year, astronomical citizen science project launched

in June 2009. The hosting organization was the American Association of Variable Star Observers

(AAVSO), a citizen science organization that has been collecting astronomical data from amateur

astronomers since 1911. The project was organized according to a set of design principles that

were codified based on literature and the collective experience of AAVSO staff in running citizen

science projects.

Design Principle 1: Use a Context Where Volunteers Contribution Is Necessary and

Meaningful for Their Scientific Inquiry

The scientific goal of the project was to engage the public in monitoring a rare eclipse of a

very bright starepsilon Aurigae. The star was so bright that the majority of professional

observatories could not observe it with their sensitive instruments, but it is ideal for public

observationeven from bright cities. The educational goal of the project was to increase general

scientific literacy by involving participants in every stage of the scientific process. That is,

participants were asked to do more than simply collect data. They made their own hypotheses,

perform data analysis, tested theories andmodels, and eventually even published papers in a peer-

reviewed astronomical research journal (Percy, 2012).

Design Principle 2: Provide Internet Resources to Help Volunteers Interact With Peers

and Scientists

TheCitizen Sky project wasmostly coordinated via a central web site. Anyone could read the

sites content, but participants who wished to actively contribute to the project had to register for

an account. There were a number of ways participants could interact with the project. They could

supply data by making brightness estimates of epsilon Aurigae and submitting it to a central



database. They could explore the data using a variety of analysis and modeling tools available on

the web site. For social interactivity, the site had nine asynchronous online forums, which were

moderated and facilitated by project staff. Also, synchronous live chats were held roughly every

month. These free-for-all style chat sessions were scheduled around special topics and/or guests

(usually professional astronomers). Ultimately, the participants were given the tools and other

support to form collaborative teams focused around a mini-research project. Sometimes project

topics were suggested by staff and sometimes the teams came up with their own. Their mission

was to identify a topic, do their own research and develop a paper for submission to a peer-

reviewed astronomical publication. A professional liaison was provided to each team to answer

questions and give advice. Example team topics include developing a software package to

statistically analyze submitted data, testing the use of consumer DSLR cameras to acquire data

and designing visualizations based on the acquired data for use in press releases and other outreach

venues. In the citizen science classification regime described in this paper, we place Citizen Sky in

the co-created category.

Design Principle 3: Actively Involve Scientists in a Role of Teaching and Communication

One of the most important motivating factors in citizen science is direct and frequent

interaction with professional researchers (Doering, Miller, & Veletsianos, 2008). Citizen Sky

applied this principle by creating an environment with many channels of communication with the

professional astronomical world. A project scientist (Ph.D. astronomer)maintained an interactive

blog to keep participants informed of the latest research of epsilon Aurigae from professional

journals and conferences. Also, a graduate student spent roughly 20 hours aweek interactingwith

participants via the project web siteproviding themwith feedback, advice, and general support.

Additional external scientists were invited to contribute blog posts about their own research into

the star. Finally, the online chats provided a chance to interact directly, in a synchronous manner,

with scientists (and each other).

Design Principle 4: Support Participants for Analyzing and Presenting Their Own Data

This principle was based on the concept of experiential learning (Kolb, 1984), which

postulates that effective learning is based on a transformative experience. The experience here is

the collection and analyzation of data by participants working on a real and pressing research

problem. Using simulated or modeled data in an educational context can introduce misleading

ideas about the scientific experience (Hollow, 2010). With their own data, participants had the

same connection to the data as a researcherin fact, they became researchers in every sense of the

term. In Citizen Sky, after they submitted data, participants were shown real-time graphs of their

data superimposed over the data of other participants, so they could see how their data compared.

They were also provided with tutorials and GUI tools to analyze the data, either separately or

combined with data from others. Ultimately, many published their results at amateur and

professional astronomymeetings and in journals.

Design Principle 5: Encourage Participants to Become an Active Member of a Research

Community

The most advanced and successful citizen science projects in recent AAVSO history were

ones in which the participants worked as a team. Constructivist approaches to science education

also have shown the benefits of group work in classroom laboratory projects (Cummings &

Kiesler, 2005), but remote projects need extra attention to foster this type of collaboration (Corter

et al., 2007). As a result, the Citizen Sky web site had a section dedicated to the formation of

different teams working on their own research project. These were private areas where team


778 PRICE AND LEE

members could communicate, share documents and chat. The formation of teams was fostered by

staff during the second year of the project. But team participation was not mandatory. About 23

teams eventually formed.

These design principles are not unique to the Citizen Sky project. The first two principles are

now considered as common sense for the construction of new, online citizen science projects. The

third design principle, working closely with scientists, is becoming more common with many

online citizen science projects. The Galaxy Zoo project, which has spawned a collection of other

projects beneath the umbrella ZooUniverse organization, has maintained a very active section of

their web sitewhere scientists interact with participants. The fourth design principle is common in

active citizen science projects where participants generate their own data, such as the Christmas

Bird Count. The fifth principle, working as a community, is perhaps themost unique to the Citizen

Sky project. Many projects involve teams to collect data, such as the Mount Rainier Citizen

Science Team (Bacher, 2011), but the teams tend to be highly localized and focused (Roy

et al., 2012).We found none that supported teams involved in the broader process of creating their

own research questions, analyzing data and ultimately publishing their results. None of these ideas

are individually unique to Citizen Sky, collectively they are unique only because all of the design

principleswere applied to a single project.

Theoretical Framework: Scientific Literacy

One of the most heralded promises of citizen science projects is to support the application of

scientific thinking to everyday life. This is mainly because participants are engaged in a project

that has become a part of their life outside of school or profession (Bonney et al., 2009; NRC,

2009). This may help overcome some of the significant transfer issues that exist in science

education (Gilbert, Bulte, & Pilot, 2011) by establishing the home as part of the participants

learning environment extended beyond formal science education settings. That home is also both

its own community and a part of a larger community. Roth and Barton (2004) propose an idea that

scientific literacy and citizen science is linked through these communities, which are better able to

address complex scientific issues than relying on individuals who were taught merely to follow

directions. The complexity alsomakes learningmore authentic: . . . expecting one set of relations(institutional school) to prepare students for a world of many relations does not make sense, (p.

2237). The participants role in the project is influenced not just by personal interest but also by

their community (e.g., amount of time they can spend on the project is related to family and work

demands). In this way, citizen science is unique in that it is an authentic scientific learning

experience that takes advantage of the support structure tailored for the participant in their home,

and thus can overcome the limitations of classrooms or laboratories.

Scientific literacy in a citizen science context is a collective concept (Roth & Lee, 2005) that

requires the integration of non-scientific considerations (Sadler & Zeidler, 2009) within a

personally meaningful context (Holbrook & Rannikmae, 2007). Citizen science projects are

generally created by organizations outside of the classroom, adapted to fit within non-scientific

limitations provided by everyday life and ultimately fated to the personal interest of the

participants. Thus, we have chosen to focus on a definition of scientific literacy closely aligned

with the concept of civic engagement (Shen, 1975) and with an interest in its collective

development by participants in the project. This differs from more traditional definitions that

describe scientific literacy as a combination of scientific knowledge and awareness of scientific

practices (NRC, 1996) or broad definitions such as that in the Project 2061 Benchmarks for

Scientific Literacy that include civic elements, but as one component of a much larger picture that

includes habits ofmind and the ability to observe and reflect on science (AmericanAssociation for

theAdvancement of Science, 1993).



Miller (1983, 1998, 2004), described civic-based scientific literacy through three elements:

(1) Vocabulary of science (science content): The vocabulary of basic scientific constructs

needed to read and understand competing views from a popular science news source

(e.g.,TheNewYorkTimesTuesdayScienceSection).

(2) Understanding of scientific inquiry (nature of science): The process or nature of

scientific inquiry.

(3) Attitudes towards organized science and knowledge (attitudes towards science): The

social impact of science on the individual and society.

A 2004 meta-analysis of the international scientific literacy studies shows that about 17% of

the US population was scientifically literate as described by this definition (Miller, 2004). Civic-

based scientific literacy has been shown to be especially influenced by education in informal

settings (Falk&Dierking, 2010).

We assume that most participants of this project already had a command of scientific

vocabulary at the level of fluency used byMillers (1998, 2004) studies (e.g., correctly being able

to categorize astrology as scientific or not or know whether lasers focus sound or light waves). In

addition, the participants were able to critically read the science section of a newspaper, or else it

would not have been possible to understand any of the recruitment materials or sign up for the

project. Thus, our measures were focused on the 2nd and 3rd elements of Millers definition: the

understanding of scientific inquiry (epistemological beliefs about science) and attitudes towards

organized science and knowledge (science-related activities).

According to Bonney et al. (2009), in order to measure scientific literacy, citizen science

projects can utilize project participation data (e.g., data submission logs), pre- and post-surveys,

analysis of e-mail and listserv messages, self-report surveys, focus groups and interviews. This

study collected participation data via the web site and collected data from pre- and post-surveys

and interviews. We treated interviews as a secondary data source to explain some of the findings

from the pre-/post-test analysis from the participants point of view. In another study, we analyzed

online discourse of participants to identify emergent patterns of inquiry in their participation

(Price, Borland,&Lee, 2012).

Methods

Instrument Design

Scientific Attitude Instrument. An attitude instrument was assembled to match the unique

characteristics of an older, informal science audience. It needed to be constrained in length, focus

on the use of science in everyday life, and include questions that would measure behavior unique

to a citizen science audience such as the pursuit of science news and attending scientific talks

two hobbies not often found in the general population. There are a total of nine items answered

with a 5-point Likert scale consisting of Strongly Disagree, Disagree, Neutral, Agree, and

Strongly Agree categories. See Table 1 for item details. The reliability of the instrument was high,

a 0.95.NSKS Instrument. Items for measuring epistemological beliefs about nature of science

(Table 2) were based on the Nature of Scientific Knowledge Scale (NSKS) established by Rubba

and Andersen (1978) (hereafter referred to as the NSKS instrument). We use the term

epistemological belief becausewe feel it is flexible enough to reflect that attitudes, feelings and

understanding change and are somewhat subjective. Other words such as knowledge or

awareness imply a hard reality the participant is being judged against and oversimplifies what


780 PRICE AND LEE

constitutes nature of science, a term that stirs strong emotions inmany academics.While it is not

a perfect term, we feel it best represents what we are attempting to measure in this study. There is

no agreed upon definition of the nature of science, but there is consensus that . . . it is related toepistemology and values and beliefs for scientific knowledge (p. 409) and how that knowledge is

developed, refuted, and changed (Ozgelen, 2012). The original NSKS instrument was validated

by scientists and teachers and is commonly used in science education research (Bloom, 2008). It

was chosen over other NOS instruments because of its extensive pedigree and also because it is a

survey instrument. We piloted an open ended instrument which was strongly resisted by the

participants due to its length. In fact, they rebelled on our public discussion forums at the length of

the instrument. This is a common problem with informal science settings when participants are

expected to be somewhat entertained in addition to educated. The items in the original NSKS

included 48 items grouped into six categories of the nature of science (amoral, creative,

developmental, parsimonious, testable, and unified).Each category included four positively stated

items and four negatively stated items. To constrain the length of the overall test (in response to the

pilot study), we omitted all negative items leaving four items per category for a total of 24 items in

this study. See Table 2. The NSKS used a 5-point Likert scale consisting of Strongly Disagree,

Disagree, Neutral, Agree, and StronglyAgree categories. The overall reliability for theNSKS test

for this study was high, Cronbachs a 0.94, and was in general agreement with previousvalidation work on the original NSKS instrument, despite its shortened length (Rubba &

Andersen, 1978;Meichtry, 1994).

Interview Protocol. The interview protocol included questions about past participation in

other citizen science projects (Have you participated in any other astronomical project similar to

Table 1

Scientific attitude individual item difficultiesbased on Rasch analysis

Item (Abbreviation)

Item Difficulty (Logits) SErasch Outfit

Pre Post Pre Post Pre Post

I actively seek out stories aboutastronomy in the news. (SEEK)

1.05 1.23 0.09 0.09 1.03 0.90

I am likely to attend a scienceseminar, class or talk. (ATTEND)

0.81 1.17 0.10 0.09 1.45 1.21

I plan to participate in other citizenscience projects in the future.(OTHER)

0.69 1.54 0.10 0.09 0.86 0.87

I am knowledgeable about science.(KNOWLEDGE)

0.55 0.29 0.12 0.09 1.26 0.83

I use knowledge of science toevaluate claims made aboutscience. (EVALUATE)

0.04 0.29 0.11 0.11 0.94 0.82

I will pay attention if an astronomynews item crops up in a mediasource I am already following.(ALREADY)

0.46 0.49 0.11 0.11 0.75 0.82

I use knowledge of science ineveryday life. (EVERYDAY)

0.67 0.28 0.11 0.09 1.25 1.23

I am interested in news aboutastronomy. (NEWS)

0.84 0.17 0.11 0.11 0.67 0.68

I am interested in science.(INTEREST)

1.07 0.85 0.12 0.11 0.5 0.48



Table 2

NSKS individual and sub-category item difficultiesbased on Rasch analysis

Item Categories and ItemsWith Abbreviations



(a) Amoral 0.0 0.32 0.15 0.14The applications of scientific knowledgecan be judged good or bad; but theknowledge itself cannot.

0.30 0.14 0.07 0.07 0.90 0.85

Even if the applications of a scientifictheory are judged to be good, weshould not judge the theory itself.

0.17 0.92 0.07 0.06 1.24 1.14

A piece of scientific knowledge shouldnot be judged good or bad.

0.06 0.07 0.08 0.07 0.91 0.82

It is incorrect to judge a piece ofscientific knowledge as beinggood or bad.

0.41 0.27 0.07 0.07 1.04 0.89

(b) Creative 0.50 0.95 0.14 0.13Scientific knowledge is a product ofhuman imagination.

0.78 1.11 0.06 0.06 1.31 1.25

A scientific theory is similar to a workof art in that they both expresscreativity.

0.49 1.0 0.07 0.06 1.16 1.02

Scientific laws, theories, and conceptsexpress creativity.

0.39 0.85 0.07 0.07 1.08 1.02

Scientific knowledge expresses thecreativity of scientists.

0.35 0.85 0.07 0.07 1.05 1.10

(c) Developmental 0.22 0.08 0.17 0.16

We accept scientific knowledge eventhough it may contain error.

0.57 0.87 0.07 0.06 1.42 1.22

Those scientific beliefs which wereaccepted in the past and since have

been discarded should be judged intheir historical context.

0.06 0.28 0.08 0.07 1.29 1.11

Scientific knowledge is subject toreview and change.

0.58 0.74 0.09 0.09 0.75 0.71

Todays scientific laws, theories, andconcepts may have to be changed inthe face of new evidence.

0.93 0.71 0.09 0.09 0.85 0.89

(d) Parsimonious 0.39 0.74 0.15 0.13There is an effort in science to keepthe number of laws, theories, andconcepts to a minimum.

0.82 1.29 0.07 0.06 1.14 1.20

Scientific knowledge is comprehensiveas opposed to specific.

0.53 1.07 0.08 0.06 1.17 1.44

Scientific knowledge is stated as simplyas possible.

0.18 0.86 0.07 0.06 1.15 1.22

If two scientific theories explain ascientists observations equally well,the simpler theory is chosen.

0.04 0.71 0.07 0.07 1.17 0.96

continued


782 PRICE AND LEE

Citizen Sky?), level of participation in this project (What do you feel your role is in the Citizen

Sky project?), views on the various categories of the NSKS (e.g., Do you see the creative aspect

of the nature of science on display in the Citizen Sky project?) and included a few questions

tailored to any anomalous responses each participant had to specific items. Therewere a total of 11

planned questions, plus the tailored follow up questions. Interview transcripts were analyzed by

listing verbatim transcripts of all answers to each common question separately and looking for

emergent trends and differences. For the individually tailored questions, the transcripts were

analyzed separately by comparing their interview response to their survey responses to see if there

is any explanation for the anomalous responses.

Subjects and Data Collection

As of February 1, 2012, the projects web site had 6,491 registered users (participants). Of

them 3,180 had taken the pre-test as part of the project registration process. They self-identified as

78% male, 19% female, and 3% did not choose a gender. The mean age was 41 years old

(SD 16). That the gender ratio and age skews towards an older, more male audience is typicalfor the amateur astronomy community. Sky& Telescopemagazine, the magazine of record for the

community, reports 95% of their subscriber readership is male with a mean age of 51 (New Track

Media, 2010). About a quarter of participants in this study reported no prior experience in

astronomy. About 61% of the participants had a bachelors or higher degree, which was below that

of subscribers reported by Sky&Telescopemagazine (77%).

Item Categories and ItemsWith Abbreviations



(e) Testable 0.66 0.33 0.18 0.16

Consistency among test results is arequirement for the acceptance ofscientific knowledge.

0.48 0.21 0.09 0.08 0.86 0.90

A piece of scientific knowledge will beaccepted if the evidence can beobtained by other investigatorsworking under similar conditions.

0.58 0.07 0.09 0.07 0.88 0.82

Scientific laws, theories, and conceptsare tested against reliableobservations.

0.65 0.45 0.09 0.08 0.77 0.79

The evidence for scientificknowledge must be repeatable.

0.93 0.58 0.09 0.09 0.70 0.77

(f) Unified 0.02 0.21 0.17 0.15

Biology, chemistry, and physics aresimilar kinds of knowledge.

0.53 0.32 0.08 0.07 0.98 1.08

The various sciences contribute to asingle organized body of knowledge.

0.20 0.32 0.07 0.07 1.07 1.39

The laws, theories, and concepts ofbiology, chemistry, and physicsare interwoven.

0.15 0.22 0.09 0.08 0.79 0.82

The laws, theories, and concepts ofbiology, chemistry, and physicsare related.

0.65 0.60 0.09 0.08 0.84 0.84

Standard error of all items in thegroup added in quadrature.

Table 2

Continued



When participants in the project first registered via the web site, they were given the

opportunity to take the pre-test that included the Scientific Attitude instrument and the shortened

NSKS instrument along with many other instruments. Theywere compensated with placement in

an annual drawing for a gift card. Most participants knew very little about the project at this point

and had not formally participated in it at any level. The pre-test was offered to 6,491 participants,

of whom 3,180 opted to complete at least a portion of it (49%). Theywere invited to take the post-

test during their first login to theweb site after 6months had passed since registration. Ultimately,

365 participants were offered the post-test and 333 opted to complete at least a portion of it (91%).

The difference between the number of participants offered the pre-test and post-test is likely due to

two factors. First, there was a high dropout rate and many people did not return to the project

6 months later. Second, participants were not required to log into the site unless they needed to

submit data or post to a forum. And web site login was a requirement to take the post-test (so we

could match pre- and post-tests). So most participants who did return later did not need to log in.

The gender distribution between the post-test group and the pre-test group is similar, but the post-

test groups mean age was about 6 years older. Six months of project activity was chosen as a

measurement point because, based on project staff experience,most participantswill have already

reached their peak involvement at that point. In order to reach participants who were no longer

active in the project, we also sent private e-mail invitations to thosewho had registered for theweb

site 6months prior but had not logged in during the previous 3months.

Fourteen participants who took both the pre- and post-tests online were randomly invited to

participant in an interview session, of which nine accepted. They ranged in age from 18 to 64.

Eightweremale and one female. Their education experience ranged fromhigh school graduates to

one with a Ph.D. Their astronomy experience ranged from novice to professional, however, most

were in the intermediate category. The interviews were conducted via the telephone or Internet

communication software such as Skype, Google Voice, and Yahoo Messenger. They were each

compensated with a gift card. The interview durations ranged from 25 minutes to 1 hour and

15 minuteswith an average duration of 40 minutes.

Data Analysis

Coding. For each item on the two instruments, we scored responses by assigning a 1 for

StronglyDisagree, 2 for Disagree, 3 for Neutral, 4 for Agree, and 5 for Strongly

Agree.Unanswered questionswere treated asmissing data.

Rasch Analysis. Likert scores were set on an ordinal, non-interval scale. This non-interval

nature presents many complications for parametric analysis (Carifio & Perla, 2008;

Jamieson, 2004; Knapp, 1990) which assumes equal intervals between two adjacent scores. To

address this complication, the responses to the Likert scale were converted into an single interval

scale through Rasch analysis (Rasch, 1960) based on the Rating Scale Model (RSM;

Andrich, 1978;Muraki, 1990;Wright&Masters, 1982;), which is often used by science education

researchers (Boone & Scantlebury, 2005). The RSM can be described through the following

equation (Andrich, 1978; Linacre, 2002):

logPnik=Pnik1 Bn Di FkwherePnik is the probability that participant n, on encountering item iwould be observed (orwould

respond) in category k (item response) while Pni(k1) is the probability that the response would bein category k 1. Bn shows the amount of the trait the participant n has on a single numericalscale. In the measurement community, the amount of a trait is often referred to as an ability

estimate even though some constructs such as attitudes towards science are not directly associated


784 PRICE AND LEE

with the concept of ability. We use ability or ability estimates hereafter in this paper to mean the

amount of the trait an individual has for scientific attitude and nature of science constructs. Di is

the difficulty of item i, on the same single numerical scale. And Fk is the difficulty of producing a

category k response relative to a category k 1 on the same scale. Participant ability estimates(ability) and item difficulty estimates are placed on the same scale with normalized values

typically ranging from4.0 to 4.0 logits (log-odds unit).The application of the RSMRasch analysis generates an ability estimate for each person and

an item difficulty estimate for each item. The higher the person ability estimate the more the

person has the ability to endorse an item. That is, the person with an ability estimate of 2.0 on the

scientific attitude scale has higher scientific attitude than a person with 1.0. Similarly, higheritem difficulties are associated with increasing difficulty for an item to be endorsed. That is, an

item with an item difficulty estimate of 2.0 is more difficult and thus requires a greater amount of

ability to endorse than an item with 1.0. Therefore, whether an item, i, will be endorsed by aperson, n, (endorsement likelihood) with a response category of k depends on the difference

between the persons ability and the difficulties associated with the item and the response

category.

Rasch analysiswas conducted using theWinsteps software program (Linacre, 2010).We only

included responses from those who took both the pre- and post-tests. We first applied the RSM to

pre-test responses. Then, we applied RSM to the post-test responses using the item and the person

scales as anchors to equate both tests on the same scale,which is the traditionalmethod to compare

pre- and post-tests usingRasch analysis (Bond&Fox, 2007).

To check for psychometric validity and reliability on the two instruments,we usedfit statistics

to evaluate how well the data fit the Rasch RSM model. For this analysis, the outfit statistic was

used. A proposed range for acceptable fit statistics for polytomous data is 0.61.4 (Wright &

Lincare, 1994) for rating scale items. No items were omitted from either test according to this

criterion. For person ability estimates, 17 participants on the attitude test and seven participants on

the NSKS test were omitted because their person outfit statistics were1.4. We kept participantswith outfit statistics

Chat Joinwas assigned a 0 if they never visited a live, synchronous online chat sessionor 1 if they havevisited at least one live chat.

PostCountDiwas assigned a 0 if the participant had never posted amessage to aCitizenSky online, asynchronous discussion forum and a 1 if they had posted at least once to a

forum.

Since participating in chat sessions or posting messages on forums are considered social

activities, we combined these two variables into variable called Participant Communication to

reflect how active participants were in communicating with other participants. This variable had

three categories: low,medium, and high. The lowcategory consisted of participantswhohad never

joined a chat or posted a message in a forum. The medium category consisted of participants who

have either joined a chat or posted at least one message to a forum. The high category consisted of

participantswho have both joined a chat and also posted to a forum. For the analysis, we used three

variables: Team, Active Observer, and Participant Communication to represent various levels of

project participation.All project participationvariableswere categorical.

Repeated Measures ANCOVA Analysis. Repeated measures ANCOVAs were used to

investigate the main and interaction effects of project participation variables on differences in

participants scientific attitudes and NSKS ability estimates between pre and post-tests. The

independent variables in the analysis were related to the Participant Communication, Team, and

Active Observer variables. Covariates were related to individual characteristics such as the

AstronomyExperience, Gender, andAgevariables.

Interviews.Post-test interviewswere transcribed into a spreadsheet so patterns could be easily

identified among responses to the same question(s). The patterns were analyzed to characterize

participants perspectives on results from the pre- and post-tests. For example, we looked for

comments that may explain why some items showed significant change between pre- and post-

tests while other items did not. We also asked questions about any anomalous answers in the

participants survey results as compared to the rest of their responses. This was to examine if the

differences were due to personally held epistemological beliefs or misunderstanding of the items

on the tests. Every interviewed participant was asked about each category of the nature of science,

but we only present analysis of the responses in which we detected a clear pattern or link to the

survey data.

Results

Pre-Test Descriptive Statistics

Our descriptive analysis includes all 3,180 responses to the pre-test. Overall responses to the

survey questions were skewed. About 78% of all raw scores across all items lie between neutral

and strongly agree on both instruments. This is not surprising considering thesewere volunteers in

a citizen science project who are naturally motivated to participate in scientific activities and have

strong epistemological beliefs about science. According to the Rasch analysis results on the

attitude pre-test (Table 1), the easiest item with an item difficulty value of 1.05 was I activelyseek out stories about astronomy in the news. (hereafter referred to as the SEEK itemsee

Table 1 for code words used for subsequent items). The INTEREST item was the most difficult

item with the item difficulty of 1.07.While still quite positive, the responses to the NSKS pre-test

(Table 2) itemswere less negatively skewed than the attitude pre-test items. On theNSKSpre-test,

the easiest item to endorse was the There is an effort in science to keep the number of laws,

theories, and concepts to a minimum, which had an item difficulty value of 0.82. The mostdifficult NSKS item was Todays scientific laws, theories, and concepts may have to be changed


786 PRICE AND LEE

in the face of new evidence and The evidence for scientific knowledge must be repeatable, both

ofwhich had itemdifficulty values of 0.93.Table 2 also lists an average itemdifficulty value for each

of the six sub-categories of NSKS epistemological beliefs. The easiest NSKS sub-category was

creative with an average item difficulty value of0.50 and the most difficult was testable withthat of 0.66. See detailed itemdifficulty values andotherRasch analysis results inTables 1 and2.

Changes in Participants Scientific Attitudes and Epistemological Beliefs About Nature of

Science

Analysis of change between the pre- and post-test is based only on responses from the 333

participantswho tookboth tests.

Attitude Towards Science. To identify significant changes between pre- and post-tests in

scientific attitudes, we compared the difference in logit units between person ability and item

difficulty in each of the nine scientific attitudes items (Figure 1). When the difference variable

becomes zero the item will be endorsed by 50% of the participants. Positive values in the

difference variable mean more than 50% chance of participant endorsement and negative values

mean less than 50% chance of participant endorsement. Figure 1 indicates that, at the pre-test, the

SEEK item was the most likely item to be endorsed by participants and the INTEREST item was

the least likely. At the post-test, the differences significantly decreased, meaning the SEEK,

ATTEND, OTHER, EVERYDAY, NEWS, and INTEREST items became easier to endorse,

p < 0.05. There were no significant changes for ALREADY. There was a significant increase forKNOWLEDGE and EVALUATE, meaning they became more difficult to endorse. The item with

Figure 1. Participants endorsement likelihood is determined by the distance between person ability and item difficultyon the scientific attitudes scale. The bars indicate the average distance in each of the nine scientific attitude items. Thesmaller the bar, themore likely the participants endorsed the item.



the biggest overall change is the OTHER item, which is about the participants likelihood to join

other citizen science projects.

As seen in Table 3, ANCOVA results indicate that there was a significant change from pre to

post in overall person ability estimates for the scientific attitude variable. The Active Observer

variable was not significant, and neither was the Team variable. There were no significant

interaction effects for either with the Time variable. However, Participant Communication had a

significantmain effect on the scientific attitudes variable. ATukeys post hoc test indicated that the

high participant communication groupwas significantly different, both at thep < 0.05 level, fromlow andmedium communication groups. This means that, coming into the project, thosewho had

higher Participant Communication tendencies (i.e., became most active in the chat room and

online discussion forums) had more positive attitudes at both testing times. There is a significant

interaction effect between the Participant Communication variable and the Time variable (the

repeatedly measured scientific attitude variable), meaning that those who were actively involved

with Participant Communication changed to a greater extent from pre-test to post-test than those

who did not. There was no significant difference between the low and medium Participant

Communication groups on either test.

Of the covariates, only Astronomy Experience was significantly related to the overall

scientific attitudes variable. The Age and Gender variables were not significantly related to the

overall scientific attitudes variable. A Tukeys post-hoc analysis of the Astronomy Experience

variable on the scientific attitude variable showed significant differences between the low and

medium experience groups, p < 0.05, and between the low and high experience groups,p < 0.01. Therewas no difference between themediumand high experiencegroups, p 0.76.

Epistemological Beliefs About the Nature of Science. Figure 2 shows what NSKS item

categories changed significantly between pre- and post-tests. There were significant positive

changes in creative, parsimonious, amoral, and testable categories. There was no significant

change in the developmental category. The overall NSKS ability estimates of thosewho took both

pre- and post-test showed significant improvement between the two time points, according to

Table 4, p < 0.05.

Table 3

Analysis of covariance on scientific attitudes

Source df F Partial eta Squared r

Fixed effectsTime (M) 1 18.6 0.166 0.000

Participant communication (P) 2 3.76 0.075 0.027

Active observer (AO) 1 0.353 0.004 0.554Team (T) 1 1.49 0.016 0.225M P 2 3.38 0.076 0.025M AO 1 2.55 0.058 0.061M T 1 0.072 0.001 0.789

CovariatesAge 1 0.311 0.003 0.578Astronomy experience 1 15.0 0.138 0.000

Gender 1 0.575 0.006 0.45

p < 0.05.p < 0.01.


788 PRICE AND LEE

As with the attitude test, repeated measures ANCOVA was used to look for differences in

various groups of participants (Table 4). No significant main or interaction effects of project

activity or individual characteristic variableswere detected.

Post Interviews

Social Communication.The interviewdata provide insight into someof the results detected in

the tests. The attitude itemwith the greatest change is the OTHER item, which could be due to the

Figure 2. Participants endorsement likelihood is determined by the distance between person ability and item difficultyon the scientific attitudes scale. The bars indicate the average distance in each of the six NSKS subcategories. The smallerthe bar, themore likely the participants endorsed the item.

Table 4

Analysis of covariance on beliefs in nature of science

Source df F Partial eta Squared p

Fixed effectsTime (M) 1 4.02 0.042 0.048

Participant communication (P) 2 0.216 0.005 0.806Active observer (AO) 1 0.149 0.001 0.106Team (T) 1 0.000 0.000 0.997M P 2 0.458 0.010 0.634M AO 1 0.760 0.008 0.428M T 1 0.363 0.004 0.548

CovariatesAge dichotomous 1 0.694 0.008 0.407Astronomy experience 1 0.074 0.001 0.786Gender 1 1.95 0.021 0.166

p < 0.05.



selection effect related to participants who are already interested in citizen science. However, this

increase could also be related that their positive experience in Citizen Sky. Further validation of

this interpretation can be done by comparing with other citizen science projects, for which we do

not have data. The item with the second most change is the NEWS item. The other news related

item, SEEK, also showed an increase. Our interviewdata shed some light on a possible connection

between the two due to participants sharing news stories with each other via our online forums.

When asked about changes in their news reading activities, three participants referred to posts to

our discussion forums as new sources of news:

Participant 4: I tend to read specific [news sources] that allow me to gain as much info as

quickly as possible . . . I will tend to readCitizen Sky [forum] posts at nightwhen things are abit quieter.

Participant 5 (referring to news updates from project staff): Ive always eagerly read any of

those [forum] posts fromcitizen sky.

Participant 9: To some degree CS has led me more into the blogosphere and the web with

regard to news. At the same time Ive had to enhancemyway of critically reading such news

and be able to try to deal with the sources it is coming from . . . the forums have haddiscussionswith regard to thevalidity of sources andmethods.

In addition to the items related to news reading, the interviews suggest that working with

others makes the project more interesting and also allows them to look at things from new

perspectives. Regarding this increased interest, two participants commented on the importance of

community and the collaborative nature of the Citizen Sky project, which was one of the design

principles:

Participant 2: Just in participating in it Ive learned things about interacting with other

people and assumptions about sort of the knowledge and the interpretation skills of other

people.

Participant 5: [The project has] become a pretty important part of my life (laugh). These

peopleits partly the people too its not just the science, its the combination of the two.

One thing Ive really learned is that when you are doing science it is really helpful to have

a team. It is really helpful to be able to throw ideas out there and bounce them around

each other and have people with different expertises that can clarify things that you

might now have understood completely or to see something in a different way than

someone else did.

Recall that the social aspect of the project was related to change in attitudes on the surveys.

These two interview responses suggests that it was also an important role in helping participants

see themselves as part of the scientific team.

Self-Perception Regarding Their Knowledge. The KNOWLEDGE and EVALUATE

items show the only decreases in endorsement. However, six of the nine interviewees stated

that their knowledge increased or was otherwise unaffected through participation in the

project:

Participant 1: Im learningmore about variable stars as awhole.


790 PRICE AND LEE

Participant 4: I have a better appreciation of the different kinds of variable stars and some of

the characteristics of their light curves. Im learningmore about data analysis in this context

as a result of asking questions and experimentingwith data.

Participant 7: I always felt before that variable star observingwas too advanced forme. That

alongwith the thought that [it] sounds boring. Its not so boring now.

Participant 9: I dont think my knowledge of astronomy has been affected, no. But there has

been a big effect onmyknowledge ofhow to do astronomy. (emphasis theirs)

Therewas no statement in any interviewwhere a participant suggested theywere not learning

or having trouble learning. An explanation for the discrepancy between what they said and what

the test results show could be in how the KNOWLEDGE item was worded (I am knowledgeable

about science.). That could be interpreted as a question about efficacy as opposed to knowledge.

That is, participants are gaining more knowledge but this is also opening their eyes to how much

more they have to learn (the more you know, the more you dont know). This was suggested by

one of the participants:

Participant 2: There have been many instances where I think peoples awareness of

limitations of past knowledge has been increased. One of the overwhelming feelings I get is

how much we dont know. So, many times its more like in what direction to move your

boundaries of ignorance, as opposed to your boundaries of knowledge.

Creativity in Science. The NSKS Creative category had the greatest increase in endorsement

likelihood. The interview responses show consistent evidence that the project does involve broad

creativity. All nine participants interviewed gave different examples of creativity in the project. Of

them, one of the participants described their own problem solving skills as creative. But most of

the examples of creativity involvedwatching other peoples application of creativity:

Participant 3: In the part I participated in, I was just gathering data so other people could do

the creative part of explaining the data. So I didnt domuch of the creative stuff but there are

definitely other peoplewho did.

Participant 4: In general, discussions in the CS forums in which people are coming up with

alternative explanations for aspects of eps aur.

The interviewer noted in their field notes that when asked about the creative items,

participants often had to pause and think more so than in the other categories. Also, early in the

project a particularly heated discussion occurred in the project forums over the topic of creativity

in science. Investigation of the individual items found that in general the creative category of items

were easiest to endorse in both pre-test and post-test (Table 2). Also, participant endorsement

likelihood indicators changed the most with the creative category (see Figure 2). One participant

in the online debate described the difficulty reconciling knowledge and creativity:

Online Forum Participant 1: Scientists are creative when they develop theories, but is

knowledge creative? No, knowledge is knowledge. Its like saying water freezes at 32 F,

Imbeing creative by telling you this.



This participants focus on the specific terminology is likely an example of issues many had

with those questions. However, they also establish a difference between the presence of creativity

at the beginning of the scientific process as opposed to the end. This is supported by many of the

other participants in the discussion:

Online ForumParticipant 2:While the scientic (sic.)method calls for strict reasoning and logic

when checking hypotheses and making inferences, at the very begging (sic.) of a new theory,

there is alwaysan informedguess.Apieceof intuition, imagination, somecreative event.

Online Forum Participant 3: Of course, logic and mathematics are the means by which the

world can be abstracted, represented and understood as models. But before models can be

created and tested against evidence, surely imagination and creativity play a role. Einstein

imagined (emphasis theirs) what it would be like to ride on a light beam, wondering how

theworldwould look at close to the speed of light.

Online Forum Participant 4: No matter how mundane, devising experiments (and creating

(sic.) models) must surely often require creativity also. So, I think science is a mix of both

creativity and logic. Its a human endeavour (sic.) after all.

Parsimony in Science

The other NSKS item category that was most easily endorsed by participants was the

parsimonious category of items. The interviews suggest participants did not completely

understand thewords definition:

Participant 1: . . . this ismore difficult question than I thought . . .(laugh)

Participant 5:What does thatmean? (laugh)

Participant 7: Thats a bit harder towrapmyhead around.

To better describe the concept, the interviewer referred to Occams Razor (summarized as

when confronted with two explanations of equal accuracy, choose the simpler.), which is a

common quotation in popular astronomy literature. After the definition was cleared up, everyone

interviewed exhibited support for the parsimonious nature of science. Most often they quote the

evolving theory of epsilonAurigae and also the ongoing development of a Theory of Everything

in the physics community.

Reinforcement. The change in the epistemological beliefs detected in the surveys is

translational,meaning the center of the distributionmoved from lower to higher points in the scale

without changing the shape of the distribution or changing the locations of items on the scale. The

relative ranking of the six sub-categories of the nature of science did not change. This suggests a

reinforcement of beliefs, which is also evidenced in language used in the interviews. The language

illustrateswhere participants built on prior knowledge in phrases such as Im learningmore about

variable stars as a whole, I have a better appreciation [of experimentation], [my interest] in

science is greater, and I was reading about astronomy and physics and other science stuff even

before Citizen Sky. In none of the interviews was there any discussion about fundamental

changes in howparticipants view the nature of science. Instead, the commentswere about filling in

gaps of knowledge and supporting prior epistemological beliefs.


792 PRICE AND LEE

Discussion

There are two main overall findings addressing each of the two elements of scientific literacy

we studied. First, attitudes towards science increased and this increasewas significantly related to

participant communication. Second, epistemological beliefs about the nature of science also

increasedbut we did not detect a relationship with project participation levels. These two points

together indicate that the participants attitudes towards science-related activities can be

influenced by their positive social experience with others in citizen science projects. However,

epistemological beliefs about science are less subject to the project participation experience

possibly because epistemological beliefs are personal beliefs and thus harder to change after

participating in only one citizen science project. Finally, whether the participant submitted data or

was a member of a formal team did not have a direct relationship with changes in scientific

attitudes or beliefs about the nature of science.

Attitude Towards Science

Interest in astronomy and science was very high on all of our attitude measures, as

expected for a volunteer science project. Yet we still detected a significant change in the scientific

attitude ability estimate as a whole from pre to post. Considering that other citizen science

projects have not reported any change in scientific attitude (Brossard et al., 2005), our finding is

noteworthy. The difficulty with finding changes in attitudinal constructs has been claimed to

be related to the sensitivity of the instruments used in other citizen science projects (Brossard

et al., 2005) and we agree. The change we detected is mostly through reinforcement of

existing epistemological beliefs and our detected change is likely due to the use of a more

sensitive analysis procedure. Thus we conclude that participation in a citizen science

project alone is not likely to change overall attitudes towards science at a very high level,

probably due to the fact that those entering such projects already have very positive

attitudes. However, it can build on existing positive attitudes towards science and improve them

further.

The analysis procedure we used in this study, involving Rasch analyses of test responses

combined with ANCOVAs using project participation variables, allowed us to uncover more

significant item-related effects and could serve as an important analytical framework for future

projects. Six of our attitude items became significantly easier to endorse. The OTHER items

increase suggests they are more likely to participate in other citizen science projects in the

future. The NEWS items increase is related to how participants share news stories and news

sources with each other via the project web site and online forums. This is supported by

the relationship between the survey scores and the Participant Communication variable. Also,

the interviews provide insight into how that relationship manifested itself in the project (e.g.,

through sharing of online news sources). Modern conventional wisdom is that collaboration and

social factors are key to learning about science (Vygotsky, 1964), including online science

learning (Linn, Davis, & Bell, 2004). The KNOWLEDGE survey item became more difficult to

endorse. Interview data suggests that this is not due to participants losing (forgetting) knowledge

but rather that they are becomingmore aware of howmuch they do not know. This important result

speaks towards the fundamental relationship one has with science as the participants scientific

horizons expand. The EVALUATE survey item also became more difficult to endorse, which we

believe is related to the lowered self-perception found in the KNOWLEDGE item. That is,

participants trust in their ability to apply scientific thinking to daily life decreased because

they feel they understand a smaller fraction of the scientific field than they did when they took the

pre-test.



Epistemological Belief About the Nature of Science

Epistemological beliefs about the nature of science increased significantly in this project.

However, the relative rankings of the various NSKS categories did not change much between the

pre- and post-test. This suggests that previous epistemological beliefs are being reinforced, rather

than restructured. If they were being restructured, we would expect to see different levels of

change between the various categories to such a degree that their relative importance would

changefor example, creativity may suddenly become themost important element in participant

epistemological beliefs. Instead, categories with the strongest beliefs remained the strongest and

viceversa. Instead of structural difference, therewas an across-the-board difference.

The creative category showed the greatest amount of change. A discussion about creativity

was one of the most active and controversial threads in our online forums. That debate was more

about where creativity exists in the scientific process (restricted to the beginning or infused

throughout) rather thanwhether it is important at all. The amoral category also showed significant

change and also was an active topic in the online forums, mainly through a heated discussion

thread about global warming. Because our interview responses show confusion with vocabulary,

we believe the parsimonious change is largely due to syntax and itemwording rather than anything

related to the project. The change in the testable category may be associated with the ongoing

presence of a staff blog on theweb site that continually discussed currentmodels of the star system,

while comparing themwith new data and slowly constraining the cloud of uncertainty around the

source of the eclipse. Through this process, participants saw their data in action in a very visible

way and how itwas being used to solve the core scientific question of the project.

There was a drop in epistemological belief that the goal of science is to create universal laws

across various domains. This was a topic that was not addressed much at all in the projectwhich

was focused almost exclusivelywithin the domain of astronomy.

Our results are the first in the literature to show a change in epistemological beliefs in the

nature of science through a citizen science project. One previous study reported high

epistemological beliefs about the scientific process by their participants, but they could not

attribute the beliefs to participation in the project (Trumbull et al., 2000) so it could be a reflection

of the same selection effect we found in our pre-test results. Another study found no difference in

understanding of the scientific process between those who participated in a citizen science

project and a control group (Brossard et al., 2005). They suggested that since the primary

motivation to join the project was an interest in the projects content (birding), participants did

not view the project in a scientific light but rather as a knowledge-building exercise (i.e., an

educational hobby). This is backed by Raddick et al. (2010) which reported only 12% of the

Galaxy Zoo participants joined due to an interest in science or discovery. Themain reason to

join was interest in the subject matter, astronomy, at 46%. On the other hand, the Citizen Sky

project specifically refers to the scientific process throughout its training procedures. In fact,

Citizen Sky recruitment materials focus on the opportunity to participate in all steps of the

scientific method as a selling point to differentiate itself from other citizen science projects. So

this level of meta-cognition may explain why this study finds more of an increase in

epistemological beliefs, since participants went into the project more sensitive to how the entire

process unfolds.

The Role of Social Activity Involvement

One of the important results of this study is the value of the social component of the project.

After testingmany different variables, we found that the Participant Communication variablewas

the only one related to change in scientific attitudes. Interview transcripts suggest this is due to


794 PRICE AND LEE

participants communicating with each other and using the online forums as a source of

information and news. This could be a very positive development for the emerging field of

citizen science. In the past, one of the challenges of such projects was the isolation in which

participants worked. Nov, Arazy, and Anderson (2011b) found that establishing a community

of volunteers is second only to intrinsic motivation in terms of sustaining commitment to a

project.

One explanation for this link between change in attitude and Participant Communication

could be in the highly personalized nature of citizen science projects. The sense of agency that

drives interest in citizen science also serves as an illustration of Aikenheads personal-curiosity

science, which is part of a enculturation of science education into communities that are directed by

the learners themselves. Citizen science projects have a greater opportunity to build a social

community (as evidenced in the forums) and to empower its participants more than individual or

even classroom-based science projects. This agency stems both from a closer sense of ownership

of the process and its products and, for collaborative and co-created projects, also in the influence

the participant has over the project structure. This sense of increasing control is important to

scientific literacy and leads to citizen science becoming a strand of the learning process, one based

on personal relationships (Roth & Lee, 2005). The sense of community and personal

empowerment is fostered by an active community where the participant has a role beyond that of

an anonymous data collector or processor. Also, online and interactive forums can support a more

integrated community by narrowing the barrier between professional staff and participants, which

has been recognized as an important goal by the citizen science community (Raddick et al., 2009).

The Zooniverse project, arguably the most rapidly growing citizen science project ever, has

reported 11,000members registered for their forums (Raddick et al., 2010). It is a rich community

that supports a deeper investigation of the various Zooniverse projects than the original project

goals intended. And participants have shown the initiative to investigate ideas on their

own (Cardamore et al., 2009; Raddick et al., 2009). This has created its own challenges,

such as determining how to effectively support such a large and active group when faced with

limited resources. Our study suggests simply providing a forum to let participants communicate

with each other is an easy and effective first step. Yet it is one that most citizen science

projects are not yet taking. This stepwould go a longway to help put online citizen science projects

in line with the greater science education communitys emphasis on collaborative and social

learning.

Other Project Participation Measures

The Citizen Sky project was designed to give participants a chance to participate in every

stage of the scientific process with the belief that this greater engagement will increase their

scientific literacy. Participants were not required to engage at these other stages, but many did.

Over a dozen papers have been published in a peer-reviewed scientific journal dedicated to

participant projects. However, evidence of how participation levels and type affected change in

literacy has been elusive. For example, one may expect to find a relationship between the number

of variable star observations submitted to the database (the Active Observer variable) and

epistemological belief in the testable aspect of theNSKS test, since the very nature of variable star

observing is to combine lots of data from independent sources. However, we find no such

statistical relationship in our data. Other research on citizen science projects have also found that

contributing data did not increase participants epistemological beliefs in the nature of science

(Trumbull et al., 2000). It may take more than just participation in data collection for participants

to gain new insight into science.



Implications

Citizen science is a field experiencing explosive growth. It has great potential to not just help

scientists but it also can help educators. This study suggests a number of ways a citizen science

project can be designed to enhance scientific literacy of its participants. But education must be

established early in a projects design as a primary goal of the project in order to implement many

of these suggestions. First, we recommend a dedicated social aspect to the project. This data show

that participant attitudes were most affected by direct communication with other participants.

Online forums are simple and easy to install on almost any web site platform these days. We

recommend using them along with live chats with project scientists to create a community of

shared knowledge. This social component has the potential to help empower participants, which

others have shown to have a significant impact on overall scientific literacy. This empowerment is

somewhat unique to the personal nature of citizen science projects and additional research in this

area would be compelling. Also, we suggest a direct illustration of how the participant is involved

in the overall scientific process the researchers are conducting (perhaps using metacognitive

strategies embedded in training materials). Other citizen science projects have found that

participants sometimes have trouble understanding their role in the entire project (Evans et al.,

2005) whereas this project emphasized and defined their role very early oneven in the training

materials. Informal science has the luxury of not being restricted by classroomwalls, but it cannot

completely ignore the importance of framing the issue. Finally, practitioners should not expect to

see an increase in scientific attitudes or epistemological beliefs simply by having volunteers

participate in data collection. They need to be more involved in other aspects of the project. Our

results, along with that from other projects, shows that data collection alone has no established

relationship with changes in scientific attitudes or epistemological beliefs about the nature of

science. As for researchers, we showed how Rasch analysis can uncover results in coarse Likert

datathe type often found in informal science education surveys. Past citizen science projects

may need to apply amore sophisticated analytical model such as the Raschmodel to their data and

re-analyze them for hidden results. These relatively easy steps should go a long way in turning a

passive citizen science project into an active one that has the opportunity to change how its

participants view science.

Limitations

This study is limited in a number of ways. First, both scientific attitudes and epistemological

beliefs about nature of science instruments show certain degrees of a ceiling effect because

participants already had an interest in science before joining the project. It is likely that this ceiling

effect has depressed the measured range of responses and may be responsible for the relatively

minor amounts of change detected. Most scientific attitude instruments in the literature are

developed for children and adolescents in formal educational settings and are validated

accordingly. In order to advance research in informal settings, new and targeted instruments need

to be developed that take into consideration the unique aspects of informal science education

audiences. For example, theymust not takevery long to complete since such audiences are usually

volunteering their personal time (as opposed to formal environments which can require

completion of an instrument). Also, they need to be sensitive to a broad range of prior enthusiasm

for science. A second limitation is related to the representativeness of the sample used in this

study. While we did make a significant effort to contact those who were no longer active in the

project, themajority of the post-testswere filled out by thosewhowould be consideredmore active

than the average participant. Third, the project did not have a control group to estimate the effect of

the studied project as compared to other citizen science projects. Fourth, the attitude instrument


796 PRICE AND LEE

was created for this project and has not been externally validated in othersmeasurement research,

however, the strong Rasch alpha and fit statistics reflect that the underlying latent variable is

associated with the items chosen for the scientific attitudes instrument. Fifth, our conclusion that

the change in epistemological beliefs is due to reinforcement rather than restructuring of beliefs is

based on inferences in the survey data and phrases used in the interviews. A more in depth

interview process would be helpful to provide direct evidence of this. Sixth, the post-test group

was slightly older than the pre-test group, meaning that it may reflect a slightly more mature

population compared to the general population of the project. Finally, the authors of this study

were affiliatedwith theCitizen Skyproject.

Conclusion

This study found that overall attitudes towards science increased through participation in this

citizen science project. The change was greatest in a few specific attitudes related to scientific

news gathering and self-awareness of participants knowledge.We also found a positive change in

overall epistemological beliefs in the nature of science, but it was slight and mainly through the

reinforcement of previously held beliefs. The change in attitudes towards scientific news gathering

was related to social participati

journal of research in science teaching volume 50 issue 7 2013 [doi 10.1002_tea.21090] price, c....

Documents

museum of science

science teaching

citizen science projects

national science board

scientific attitudinal

scientific data collection

informalscience learning

sixmonths of participation