assessing undergraduate sustainability knowledge campus wide :
DESCRIPTION
Environmental and Social Sustainability Lab. Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath - UMD. Assessing Undergraduate Sustainability Knowledge Campus Wide :. From design to implementation and analysis. Overview. - PowerPoint PPT PresentationTRANSCRIPT
Sustainable Attitudes and Behaviors at OSU
Assessing Undergraduate Sustainability Knowledge Campus Wide:Adam Zwickle - OSUTomas Koontz - OSUAndrew Bodine - OSU
Mark Stewart UMDNicole Horvath - UMDEnvironmental and Social Sustainability Lab
From design to implementation and analysis
1OverviewHow we developed our Assessment of Sustainability Knowledge (ASK)
Why an ASK is important & how it can help
An aside on Knowledge and Literacy
Conducting an assessmentThinking long term
22Developing an assessmentBuilt upon the triple bottom line, the three legged stool, the 3 ps Environmental (planet)Economic (prosperity)Social (people)
3knowledge domains
3Developing an assessmentReplicated questions used in the pastCoyle, 2005. Environmental Literacy in America.Solicited topics and questions from expertsHeld expert focus groupsPilot tested among professors, graduate, and undergraduate studentsNarrowed down to 30 questions
4Developing an assessmentDistributed those 30 to OSU studentsUsed IRT to throw out 14Added UMDs 16Distributed those to OSU and UMD studentsUsed IRT to throw out 2Current ASK has 28 items:ess.osu.edu
5Conceptualizing sustainability knowledgeSocialEconomicEnvironmentalSustainability6Which of the following is the most commonly used definition of sustainable development?
Meeting the needs of the present without compromising the ability of future generations to meet their own needsWhat is the most common cause of pollution of streams and rivers?
Surface water running off yards, city streets, paved, lots, and farm fieldsMany economists argue that electricity prices in the U.S. are too low because
They do not reflect the costs of pollution from generating the electricityADD EXAMPLE QUESTIONS
6How is this helpful?7
7Need for measuring knowledgeUniversity goals-More along the lines of:Become carbon neutral by 2050
Less common:Create sustainably minded citizens of tomorrow
Can help track improvement over time
8
By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.8Need for measuring knowledgeServes as an evaluation for specific academic efforts:Interdisciplinary programsSustainability majors/minors
STARS Credit:ER 6 (STARS 2.0) 3 points available
9By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.9Knowledge vs. LiteracyKnowledgeCan be objectively measuredCan be used to evaluate academic programs
Literacy= KnowledgeKnowledge + values, attitudes, and behaviorsCan be used to evaluate outreach efforts, sustainability campaigns
10By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.10Measuring Behaviors, Values, and AttitudesMake sure you are asking the right questions
Collaborate with sustainability departments to target specific behaviors (e.g., leaving lights on)Include questions on behavioral barriersOther behaviors that may get at the same concept
11By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.11Measuring Behaviors, Values, and AttitudesCollaborate with academic departments to develop a good survey designNo need to reinvent the wheelEach survey could be a Masters thesis
12By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.12Conducting an Assessment
13Conducting an Assessment14Find your partners!
IRB approvalRequired for publicationExempt status
Registrar approval*Students emails, majors, and demographics
Survey software*We use Qualtrics, but there are others
*If a large scale assessment is plannedMaximizing Response RatesMaximizing response rates is important to reduce uncertainty about how well your completed sample matches the population of interest.
Dillman (2008) and others have long studied how to maximize response rates for surveys that were telephone, mailed, or in-person.
There is growing research on electronic survey response rates.
15Research MethodsWe compare response rates from five different survey implementations:1: 2012 spring OSU (n=10,000)2: 2013 spring OSU sample A (n=10,000)3: 2013 spring OSU sample B (n=10,000)4: 2013 spring OSU School of ENR (n=538)5: 2013 spring UMD (n=10,000)
We tried different treatments and tracked survey responses with survey software (Survey Monkey and Qualtrics)
16Key Variables Affecting Response Rates among College StudentsTiming When to send the invitation and reminders
Incentives
Email textWho it is from
Questionnaire formatLong list of questions vs. more page clicks
17Timing MattersTime of Semester:Last week of semester and into finals weekMiddle of semester
Time of Day:6:00 am6:00 pm
Reminders are criticalbig spike in completed surveys after each reminder, with a fast decay
18Timing MattersOverall, you want students sitting at their computers but wanting to be distracted
19
IncentivesSome disagreement in research on best way to provide incentivesahead of time (Dillman 2008)randomly select winner of 1 big prizegive more/all respondents smaller prizes
May impact the validity of the data20Survey Responses21
UMDn=1,556OSUn=2,621DayDayEmail text
How the invitation emails are phrased affects response rates.
Besides making the invitation personal, clear, and as short as possible, prior research has found that who the invitation comes from matters. An appeal from a trusted authority increases response rates.
22Email text: Appeal from Authority
We compared an appeal from authority versus an appeal from a peer (student).
Two surveys, A and B, had an appeal from a higher authority (University VP) for first 3 contacts.
For the 4th contact kept the higher authority for A, but switched B to have to an appeal from a grad student
23
Email text: Appeal from Authority
Results:
= additional 538
= additional 348
over the next 65 hours
24
Questionnaire FormatTrade-off:Long list of questions to scroll downShorter lists with a page click to get to the next page
We analyzed respondent drop-outs spotsHighest spots were just after clicking to the next page
We recommend finding a balance25Non-Respondent BiasEven after efforts to maximize response rates, are the non-respondents different from the respondents?
We conducted a non-respondent short survey (5 questions + some demographics).
Results indicate that non-respondents are slightly but significantly less knowledgeable about sustainability topics, but no difference in GPA or pro-environmental behaviors.
26Planning aheadFor longitudinal studies:Write a multiple year IRB Let registrar know this is a yearly surveyFinding partnersSustainability officeAcademic departments with survey expertise (interdisciplinary social science, communications, environmental studies, sociology, political science, psychology)
27AcknowledgementsFunded by:The Ohio State Office of Sustainabilityhttp://sustainability.osu.edu/
OSUs School of Environment & Natural Resourceshttp://senr.osu.edu/
28Thank You!Environment and Social Sustainability Labess.osu.eduContains:This presentation The 28 question ASKForthcoming article
[email protected]@osu.edu
Questions?2929