sound food uprising: lakewood 2019 evaluation preliminary ......marketing messages, and cook from...

1

AUGUST 2, 2019

The Beecher’s Foundation

Tacoma-Pierce County Health Department: Office of Assessment, Planning & Improvement

Sound Food Uprising: Lakewood

2019 Evaluation Preliminary Results Final Results

2

Acknowledgments

We would like to thank the students, parents, and staff of Clover Park School District for taking the time to participate in Sound Food Uprising: Lakewood. Additionally, thank you to the following partners for their collaboration and continued support of this initiative: The Beecher’s Foundation Boys & Girls Club Lakewood Branch City of Lakewood Mayor Don Anderson Clover Park School District Grocery Outlet Pierce County Executive Bruce Dammeier Tacoma-Pierce County Health Department Tacoma Rainiers

3

Table of Contents

Executive Summary ........................................................................................... 4

Introduction ......................................................................................................... 6

Design and Methods ........................................................................................ 10

Formative Research .............................................................................................. 10

Evaluation Design ................................................................................................. 11

Participant Sample and Inclusion Criteria ............................................................. 13

Protection of Human Subjects .............................................................................. 13

Results ............................................................................................................... 16

Pure Food Kids: 4th/5th Grade................................................................................ 16

Recommendations ........................................................................................... 24

Appendix A .............................................................................................................. 26

Pure Food Kids - Surveys ...................................................................................... 26

Sound Food Uprising Teens – Surveys ................................................................... 29

Sound Food Uprising Adults - Surveys ................................................................... 33

Appendix B ............................................................................................................... 37

Literature Review ................................................................................................. 37

Appendix C ............................................................................................................... 49

Evaluation SMART Objectives (by age group) ....................................................... 49

4

Executive Summary

The mission of The Beecher’s Foundation is to build a better food future for all. By

directly operating programming for youth and adults, the foundation educates and

inspires people to eat real food and vote with every food dollar. The Beecher’s

Foundation was established shortly after Seattle food entrepreneur Kurt Beecher

Dammeier founded his company, Sugar Mountain, in 2004. The foundation directly

operates two programs, both designed to influence behavior and inspire action around

food choice, thereby improving health outcomes.

The Beecher’s Foundation offers nutrition education programs to youth and adults in

the greater Seattle area. Pure Food Kids is offered in-class to 4th and 5th graders and is

offered for free to public and private elementary schools. Sound Food Uprising is

offered to adults across the Puget Sound Area. Both programs were designed to

explore the state of today’s food system and provide tools and resources for

individuals, family units and employee peer groups to make healthy food choices and

impact family buying choices.

Sound Food Uprising: Lakewood is the first Beecher’s Foundation program to be

offered in Pierce County. The Beecher’s Foundation partnered with Tacoma-Pierce

County Health Department to conduct an impact evaluation of this pilot program,

which was offered to students, parents and staff within the Clover Park School District

in Lakewood. As part of this program, Beecher’s offered its first Sound Food Uprising:

Teen program to Clover Park High School students.

Sound Food Uprising: Lakewood had very successful learning gains, a standard

measure of direct-education outcomes. Learning gains were at, or above, 30% for all

age groups reflecting excellent curricula. Behavioral outcomes for all age groups, such

as purchasing decisions and cooking meals at home, were also excellent and exceeded

our goal metrics. Among adults these outcomes were less conclusive due to low survey

response rates at 30- and 90-day points in time. The behavioral results of the few

participants that took the post-class surveys were excellent, so stronger recruiting with

incentives should improve future performance measures for behavioral outcomes. We

5

also recommend future interventions to include a Policy, Systems and Environment

(PSE) evidence-based approach to yield more successful behavioral change results,

with retention over time.

Structural issues with the survey could be prevented in the future with limiting survey

questions (<10 questions) and consistency of Likert Scale questions on pre- and post-

surveys. Pilot-testing surveys with the intended target audience before programming

begins would also inform readability and structural errors while involving community

stakeholders in the process.

Our final recommendation includes altering curricula based on Sound Food Uprising’s

target population, including socio-economic status, military status, race/ethnicity,

reading level and English proficiency. Many of our adult participants had very high

nutrition knowledge and cooking skills before the class started, leaving less room for

improvement in scores (i.e. self-selection bias). Community engagement for future

programs should be directed to adults who have a higher need for the education,

which ideally would result in greater impact.

6

Introduction

The Beecher’s Foundation partnered with Pierce County Executive Bruce Dammeier,

the City of Lakewood, and the Clover Park School District to implement a targeted food

and nutrition initiative directly reaching up to 2,000 Lakewood residents between

October 2018 – February 2019. The Beecher’s Foundation asked the Tacoma-Pierce

County Health Department to be a research partner and conduct an evaluation of the

Sound Food Uprising: Lakewood initiative.

A nutrition zone was created around the Tyee Elementary, Lakewood Hope Academy,

and Clover Park High School communities.

7

Beecher’s Foundation programming was provided at no cost to elementary (4th and 5th

grade) and high school (9th grade) students, their parents, teachers, and school staff.

The intervention was direct-education provided in a classroom setting including hands-

on cooking, nutrition education, culinary skills building, and a full meal.

The aim of the program was to promote deep and lasting change in the community’s

food-based health, with the following overarching goals in mind:

• Achieve a positive effect on residents’ shopping and eating behaviors.

• Build awareness of food marketing tactics and shift expectations around food.

• Serve a larger community based on the success of this initiative.

Pierce County is often an outlier in terms of health outcomes compared to other

Washington State counties. 65.9% of Pierce County adults are overweight and obese,

30.3% of 8th, 10th, and 12th graders are overweight and obese, and diabetes prevalence

is concentrated in the lowest income zip codes.1,2 Neighbors living less than a mile apart

can have up to 8 years difference in life expectancy.3 These disparate health outcomes

are often driven by diet-related disease.

Environmental and social determinants are risk factors for obesity, such as poverty,

unemployment, mental illness, substance abuse, or lack of proximity or access to

healthy food options (ex. grocery stores, farmer’s markets, community gardens). For

this pilot project, TPCHD and The Beecher’s Foundation will measure outcomes related

to individual behavior change and knowledge gains resulting from a direct-education

approach to health promotion. Recommendations for future food initiatives include

using a more holistic approach that encompasses policy, systems and environmental

change in addition to measuring individual impacts.

1 Behavioral Risk Factor Surveillance System (BRFSS) years? Pierce County 2 Healthy Youth Survey 2016, Pierce County (8th, 10th and 12th graders).

https://www.doh.wa.gov/DataandStatisticalReports/DataSystems/HealthyYouthSurvey/Reports 3 2015 Health Equity Assessment, Tacoma-Pierce County Health Department. https://www.tpchd.org/home/showdocument?id=196

https://www.doh.wa.gov/DataandStatisticalReports/DataSystems/HealthyYouthSurvey/Reports

https://www.tpchd.org/home/showdocument?id=196

8

Program Specifics:

4th and 5th graders: Tyee Elementary and

Lakewood Hope Academy. Pure Food Kids is a

commercial-free, learning standards-aligned food

education class. Students become “food

detectives” who can read labels, understand how

ingredient lists are ordered, see-through

marketing messages, and cook from scratch.

• 392 students (total) at Tyee Elementary and Lakewood Hope Academy

• 14 workshops, all conducted the week of October 8, 2018

• 2.5-hour curriculum Teens: Clover Park High School freshman and upper-class DECA students. Sound Food

Uprising: Teen explores the state of today’s food system and teens’ influence within

their families and peer groups to change their food environment for the better. The

hands-on workshop curriculum includes cooking, eating, and small group work.

• 350 9th graders and DECA students at Clover Park High School

• The first round of workshops was conducted the week of October 8, 2018; the final round the week of February 15, 2019

• 3-hour curriculum divided into two 90-minute sessions conducted approximately one week apart

Adults: Parents, school staff and community members were invited and incentivized to

participate in adult programming, hosted at The Boys and Girls Clubs Lakewood Branch.

Sound Food Uprising explores power and influence in our food system; lifts the veil of

the industrialized food state and uncovers practical ways to improve eating habits and

our communities’ collective well-being.

Pure Food Kids: Lakewood

9

• 370 adults (parents & school staff)

• 20 people per workshop (24 max)

• 4-hour curriculum

• Workshops held at the Boys & Girls Clubs Lakewood Branch (childcare provided)

• Workshops held October through February, Tuesdays 4-8pm and Saturdays 10 am – 2 pm

• Childcare for 6+ provided (includes a meal)

• $10 Grocery Outlet gift card for all participants

• Option to purchase a cooking kit (chef’s knife, stainless mixing bowl, cutting board) at 80% discount

• Refer-a-friend program: $25 Grocery Outlet gift card for each person who attends a workshop

Sound Food Uprising: Lakewood (adult workshop)

10

Design and Methods

Formative Research

A literature review and two key informant interviews with content experts were

completed as formative research for evaluation planning. Current evidence-based

practice around direct nutrition education in school settings include a variety of

interventions and outcome measures. Peer-reviewed articles emphasize the

importance of grounding an intervention design in theoretical models which consider a

variety of influences on human behavior. For example, from a perspective of the Social

Ecological Model, one study’s findings suggest that children’s eating behaviors need to

be considered at the intersection of children’s health literacy/attitudes, community

efforts, and media/screen time use.4 5(see Appendix B for full literature review).

Figure 1: Social Ecologic Model

We chose a combination of the Social Ecological Model and the Theory of Planned

Behavior to ground our evaluation design. In the context of diet education, the theory

of planned behavior assumes eating behavior and the intention of consumption are

both personal and environmental.

11

Figure 2: Theory of Planned Behavior

The literature suggests barriers to children’s consumption of healthy food include cost,

logistical barriers, unfamiliarity with produce, and parental opposition.

Both literature and key informant interviews confirmed that multi-component

interventions (those that include education, cooking, physical education, gardening,

etc.) and interactive learning are more successful than educational-only and/or didactic

styles of learning. Specifically, art projects and collaborative activities, like games, were

very successful.

Formative research revealed common use of one pretest and one to two post-tests for

data collection (rarely more than two post-tests). Best practices are a

survey/questionnaire at the beginning and end of the intervention, then possibly one

post-test further out. Peer-reviewed studies showed parents of school children are

more likely to respond to surveys that are sent via text message.

Evaluation Design

The Tacoma-Pierce County Health Department recommended a type of longitudinal

quasi-experimental study approach called a pretest-posttest design. This is a type of

“impact” evaluation, where we are measuring the impact of an intervention on

4 Bissell K, Baker K, Pember S, et al. Changing the Face of Health Education via Health Interventions: Social Ecological Perspectives on

New Media Technologies and Elementary Nutrition Education. Health Communication. Aug 24, 2018: 1-10

12

participant behavior. Based on our formative research and The Beecher’s Foundation’s

prior experience offering nutrition education programs in King County, we chose in-

class surveys as the primary instrument for data collection. In this evaluation, data

collection occurred at the point class began (pretest) and at several follow-up time

points, which allowed the evaluators to analyze the relationship between the

intervention and changes in participants’ food knowledge and behaviors. Pretest-

posttest design also enabled learning gain and normalized learning gains to be

calculated, describing the efficacy of the curricula.

An experimental design (where participants receive an assignment to either an

intervention or comparison group and in which most variables affecting the outcome

are controlled) is the most rigorous method for determining if a program’s services

elicit a certain outcome. Since an experimental design would entail withholding

education from a portion of students, we did not deem this ethically appropriate or

helpful to the students and the larger community.

Figure 3: Logic Model for Sound Food Uprising: Lakewood Evaluation

13

Participant Sample and Inclusion Criteria

To be included in the final report.

Data Collection and Analysis Table 1: Indicators Collected at Time Points

Time point Indicators collected Baseline/Beginning of the first workshop Antecedent/demographic variables (adults)

Baseline knowledge/output indicators End of the first workshop (in-class) Skills gained/output indicators (Learning)

Behavior/outcomes indicators (Purchasing) Goal setting indicator (Cooking)

Follow-up #1 30-days after workshop completion

Skills gained/output indicators (Learning) Behavior/outcomes indicators (Purchasing) Goal completion indicators (Cooking)

Follow-up #2 90-days after workshop completion

Skills gained/output indicators (Learning) Behavior/outcomes indicators (Purchasing) Goal completion indicators (Cooking)

Protection of Human Subjects

It is necessary to obtain human subjects’ protections approval and participant consent

to demonstrate respect for participants and their rights. These permissions help ensure

that information is collected safely and ethically. Children are especially vulnerable;

therefore, protecting their rights and information is of utmost concern.

For this evaluation, The Beecher’s Foundation worked with Clover Park School District

and followed the district’s existing parental consent procedures for the workshop pre-

and post-testing. For this evaluation, a formal review by an institutional review board

(IRB) was deemed unnecessary due to the evaluation not being deemed as “research”

under human subjects’ review guidelines; the data collected is not generalizable to a

larger population outside of the Clover Park School District community.

14

Limitations Impact evaluations aim to determine the degree to which a program’s services cause an

outcome, but rarely can they draw a conclusion that is 100% precise. Evaluators use the

concepts of reliability and validity to describe how precise the conclusions are.

Reliability is the extent to which the inference would be true if the evaluation were

tested again (that is, does the study give the same results on repeated applications) and

validity is the extent to which the inference is accurate (that is, does the evaluation

actually measures the concept(s) it intends to measure, rather than some other

concept).6 Any evaluation includes at least some challenges or “threats” to achieving

full reliability and validity. Our proposed study design and analysis tries to avoid major

threats to improve the confidence in results. Limitations that were experienced over

the course of this pilot project are described below, along with recommendations for

future consideration.

Attrition: A threat to validity based on drop-off of survey respondents at the 30- and

90-day posttest, which is quite common in survey research, and results in a smaller

sample size and less generalizable results. Attrition was experienced in the adult survey

results.

Missing Values: In our teen classes, approximately 20% of the participants did not

complete a pre-survey but did complete post-surveys. The baseline was not set

adequately because of the missing values resulting in about a 20% variance that could

be present in the calculated figures. The data meets the criteria for us to assume that

the “missing surveys” come from a random sample that would not affect the overall

trends.7

Instrumentation error: Two Likert scales on our adult survey were unfortunately

inconsistent as the pre-test question measured frequencies (ex. Always, Very Often,

Sometimes, Rarely, Never) while our post-test measured the same question in

6 Grembowski D. The Practice of Health Program Evaluation. Second. Los Angeles; 2016. 7 Acock, A. C. (2005), Working With Missing Values. Journal of Marriage and Family, 67: 1012-1028. http://people.oregonstate.edu/~acock/growth-curves/working%20with%20missing%20values.pdf

15

confidence intervals (ex. Very confident, confident, not confident, etc.) This error has

the potential to reduce our confidence in the results for these two points of data

collected since the participant was asked the same question in a different manner with

different multiple-choice answers to select from. To improve the confidence in results,

we chose to combine the negative and positive outcomes. For example, in the question

“How confident do you feel cooking at home?” we combined “very confident” and

“somewhat confident” to show an overall level of confidence.

Participant bias: Many 4th and 5th grade participants reported that they already read

nutrition labels and ingredient lists on the pre-test given before the workshop, which

resulted in little behavior change seen in outcomes. It is, of course, possible that we

have very nutrition-savvy participants, but we also need to consider participant bias in

this instance (when participants act or respond in ways they believe correspond with

what researchers (or teachers) are looking for).

Self-selection bias: Some of our adult participants self-selected themselves to

participate in the workshops based upon their existing interest and knowledge in

nutrition and cooking. Since these participants likely came in with an already high

knowledge level, our measurements for ‘learning gains’ from the workshop may not be

as high.

16

Results

Pure Food Kids: 4th/5th Grade

Response rate (to 30 days post-intervention): 38%

(demographics data not available for this age group)

Learning Gains

Learning gains are derived from the correct number of answers scored on tests by 4th

and 5th graders after completing the workshop. Preliminary results show a 31%

learning gain from pre to post (end of class), and 21% from pre to post (30-day). Our

goal metric is 45% based on King County outcomes.

Research on normalized gains, a key measure of student learning, reveals that ≥30%

improvements from pre to post testing show excellent curricula. Normalized gains

consider the mean gain for all students in the class, taking into respect where the

student starts from. Learning gains do not consider where students start- but reports

straight percentage changes.8,9

Normalized gains almost meet .30 which is satisfactory according to this research.

Table 2: Learning gains and Normalized Gains – Pure Food Kids

From pre to post

From pre to 30-day

(retained)

Goal Notes

Learning Gain 31% 21% 45% • High metric for age range; based on King County baseline data.

• Compare demographics data for King and Pierce to determine whether goal metric is realistic and achievable for this population.

• Consider curricula changes for this age category.

Normalized Gain g = (post-pre) (100%-pre)

0.42 0.29

17

Behavior Outcomes and Goal Completion

Outcomes for all behavior measures are excellent and exceed our goal metrics

(except for label reading, see below summary for this measurement detail).

• 50% of kids said they will change their food choices (purchasing decisions) after

the workshop.

• 68% shared what they learned in the workshop with their families.

• 100% said they were confident or very confident cooking or making snacks in

the kitchen after attending the workshop (note: 86% were confident at pretest

which is quite high).

• 21% achieved their goal of cooking chili at home.

Figure 4: Results - Pure Food Kids

Outcomes for label reading frequencies started out high, 70% of 4th and 5th graders

said they read nutrition labels before the workshop. While the goal was to see a 40%

increase in this frequency, we currently show a decrease of 8%. This may be due to

8 Melzer DE: Normalized learning gain: A key measure of student learning. (Addendum to Melzer DE: The relationship between mathematics preparation and conceptual learning gains in physics: a possible ‘hidden variable’ in diagnostic pretest scores. Am J Phys 2002;70:1259–1267)

9 Education in Chemistry, Learning Gains 2016. https://eic.rsc.org/feature/learning-gains/2000094.article. Accessed March 13, 2019.

https://eic.rsc.org/feature/learning-gains/2000094.article

18

participant bias (see limitations section for description) or challenges understanding

the test question in its current form.

Table 3: Label Reading – Pure Food Kids

Category Pre-

test Post-

30 days

change Goal Notes

Label Reading

Frequency

70% 62% -8% 40% increase from pre to 30-day

Possible participant bias in pre-test label reading frequency or misunderstanding of the question in post-survey. Explore test questions and review existing data.

Sound Food Uprising: Teens

Final Survey Response Rate:

Pre-survey: 301

Post-survey: 346

30-day survey: 306- 88% of post-survey respondents

90-day survey: 119- 34% of post-survey respondents

(demographics data not available for this age group)

Learning Gains

Learning gains are derived from the correct number of answers scored on tests by

high school students after completing the workshop. Results show a 30% learning

gain, which meets our goal metric of 30%. Results also show a very high retention rate

of 97% at 30 days and 98% at 90-days. The students showed a strong understanding

of the definition of a food system and the impact of the consolidation of our food

system.

19

Normalized gains are .43 which reflects excellent curricula.10

Table 4: Learning Gains – Sound Food Uprising (teens)


Outcomes for all behavior measures are excellent and exceed our goal metrics

(except one category – food advertising). Teens started out very confident in

cooking/making snacks at home. In almost all categories, the students retained

behavior changes for 3 months after taking the class (i.e. excellent 30- and 90- day

retention).

• 29% increased frequency in reading labels.

• 19% increased frequency in making purchasing decisions based on food labels.

The retention rates for this behavior were 100% for both the 30-day and 90-day

survey respondents.

• 60% shared what they learned in the workshop with their families, a

percentage that increased over time.

• 86% said they were confident or very confident cooking or making snacks in the

kitchen after attending the workshop (note: 85% were confident at pretest

Pre to post

Retained from pre-survey

Goal Notes

Learning Gain 30% 97% 30% increase pre-post


=.43

Normalized gain reflects success.

10 Melzer DE: Normalized learning gain: A key measure of student learning. (Addendum to Melzer DE: The relationship between mathematics preparation and conceptual learning gains in physics: a possible ‘hidden variable’ in diagnostic pretest scores. Am J Phys 2002;70:1259–1267)

20

which is quite high). The retention rate for confidence in the 30-day and 90-day

surveys were 100% and 99%, respectively.

• 3% increased frequency in thinking about how companies advertise food. This

increase had excellent retention over 30- and 90-day follow-up.

• 59% of students report achieving their goals they set.

Figure 5: Results – Sound Food Uprising (Teens)

Adults

(110 attendees; 78 pre-survey respondents)

Final Survey Response Rate:

• Pre-survey: 83%

• Post-Survey: 65%

• 30-day survey: 6%

• 90-day survey: 9%

10% 10%

50%

75%

10%

50%

29%

19%

60%

86%

3%

59%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Label Reading PurchasingDecisions

SharingKnowledge

KitchenConfidence

Food Advertising Goals (achieved)

n=346 (96% of goal)

Goal Actual

21

Demographics:

Race/Ethnicity Percent Count

Asian 2.60% 2

Black/African American 30.80% 24

White 43.60% 34

Hispanic/Latino 14.10% 11

American Indian/Alaska Native 7.70% 6

Prefer not to answer 1.30% 1

Totals 78

African American (31%), Hispanic (14%), and American Indian/Alaska Native (8%)

participants were represented at disproportionately higher rates compared to the

Pierce County average (7%, 11%, and 2% respectively). The number of white

participants (44%) were significantly lower than the Pierce County average (75%).11 The

30- and 90-day survey respondents were mostly white, which is not a representative

sample of the original cohort. 90% of adult participants were female.

Income Percent Count

Less than $20,000 24.40% 19

$20,000 to $34,999 14.10% 11

$35,000 to $49,999 28.20% 22

$50,000 to $74,999 14.10% 11

$75,000 to $99,999 3.80% 3

Over $100,000 6.40% 5

Don't know/Prefer not to say 9.00% 7

Totals 78

11 United States Census Bureau, Population Estimates July 1, 2018. https://www.census.gov/quickfacts/fact/table/piercecountywashington/PST045218. Accessed July 8, 2019.

https://www.census.gov/quickfacts/fact/table/piercecountywashington/PST045218

22

78% of our adult participants earn well below the Pierce County median income

($63.881).12 More than 65% of the people in these classes earn less than $50,000

annually.

Learning Gains

Learning gains are derived from answers adults gave on tests reflecting their

understanding of the US food system (Farm Bill, FDA approval process) after

completing the workshop. Final survey results show a 57% learning gain from pre to

post (end of class), which exceeds our goal metric of 20%.

Normalized gains are .77 which reflects excellent curricula.13

Table 5: Learning Gains (Adults)


Very few responses were received for 30- and 90-day post surveys (6% and 9%,

respectively) so we are limited in our ability to compare some of the adult behavioral

data post-workshop. Therefore, these data do not accurately represent our population

and cannot reflect retention rates of knowledge gained. Adult participants started out

12 United States Census Bureau, Population, Median Household Income, 2017. https://www.census.gov/quickfacts/fact/table/piercecountywashington/INC110217. Accessed July 8, 2019. 13 Melzer DE: Normalized learning gain: A key measure of student learning. (Addendum to Melzer DE: The relationship between mathematics preparation and conceptual learning gains in physics: a possible ‘hidden variable’ in diagnostic pretest scores. Am J Phys 2002;70:1259–1267)

pre to post (end of class)

Goal Notes

Learning Gain 57% 20% increase (from start of

workshop)

Made the metric


0.77 Excellent

https://www.census.gov/quickfacts/fact/table/piercecountywashington/INC110217

23

strong in their purchasing decisions and cooking skills, which left little room for growth

in these areas (this is an example of self-selection bias, see limitations section for full

explanation). Recommendations for improving enrollment, response rates, and survey

tools in the future can be found in the recommendations section.

• 30% increased frequency in reading ingredient lists.

• 28% increased frequency in making purchasing decisions based on food labels

(at the start of the workshop 82% reported they regularly purchase food based

on reading food labels).

• 100% shared what they learned in the workshop with their families.

• 97% said they were confident or very confident cooking or making snacks in the

kitchen after attending the workshop (note: 97% were confident at pretest no

change).

Figure 6: Behavior Outcomes (Adults)

20%

35%

50%

75%

30% 28%

100% 97%

0%

20%

40%

60%

80%

100%

120%

Reading Ingredients Purchasing Decisions Sharing Knowledge Kitchen Confidence

n = 110 attended (37% of goal)

Goal Actual

Adults started high at 97%

confident at pre – test.

Adults reported

high at pre-test,

only increasing

by 28% after the

workshop.

24

Recommendations

Pure Food Kids

• Learning Gains: With normalized gains of 30% (post) of 21% (30-day), these

results are satisfactory according to evidence-based research but fell short of our

program goal metric of 45% based on results in King County. After comparing

King County and Pierce County results, the number of correct answers given by

Pierce County students closely follow the trend of correct answers given by King

County students. With more classes in Pierce County, these numbers would likely

rise closer to the King County trendline.

Current results differ based on questions asked (King County youth scored higher

on food-additive and label-reading specific questions, while Pierce County youth

scored higher on food-marketing questions). We recommend further reviewing

student demographics including race/ethnicity, socio-economic status, English

proficiency and reading level of students to determine whether curricula changes

may be necessary to accommodate differences in student populations.

Label Reading: The decrease seen in label reading frequency from pre- to post-

test among youth may be attributed to a misunderstanding of the test question,

or participant bias. We recommend testing survey questions on a sample of

students in the future before official programming begins, to let students weigh-

in on the process and gain feedback regarding survey readability and to further

reduce possible survey instrumentation errors.

Sound Food Uprising – Teens

• Kitchen Confidence: Explore adjusting kitchen confidence goals (and curricula) based on teens starting with high confidence cooking meals and snacks at home.

• Pre/post-survey discrepancy: This was an extremely successful pilot program for this age bracket, despite the inconsistency in pre-post survey distribution. To avoid this discrepancy in the future, consider distributing pre- and post-surveys in-class only to those in attendance, to guarantee baseline pre-data for all

25

participants. 30- and 90-day follow-up surveys should then only be sent to those same attendees.

• Target curricula toward population (demographics): Due to the teen program

being a pilot, we had no baseline data to compare or qualify our results. Despite

lacking this data, there are other sources of data we can look to, such as

demographics, and qualitative data. We recommend researching demographics

of your student population (using American Community Survey by zip code) or

collecting qualitative data through interviews or focus groups with students, and

adjust curricula based on the population you’re serving. For example, Lakewood

has many working military families. How might that impact a teenager’s eating

and cooking behaviors?

Adults

• Recruiting: Adult participants were biased toward already having healthy

behaviors and knowledge. Explore how to reach parents, staff and community

members who may benefit more from the workshops. Increase community

engagement and marketing toward these populations by increasing incentives.

• Target curricula toward population (demographics): We recommend researching

demographics of your participant population (using American Community Survey

by zip code) to adjust curricula based on the population you’re serving.

• Attrition: To achieve a 100% response rate at the end of the workshop post-test,

hold incentives (gift cards) until participants have completed the survey. To

increase the post-survey response rate, consider increasing incentives and using

Mobile First applications only for post-30 and 90-day surveys.

• Instrumentation error: To prevent structural errors in pre-post survey

development, limit total survey questions (best practice <10) and make Likert

scale questions as consistent as possible on all surveys (and across participant

groups).

26

Appendix A Pure Food Kids - Surveys

29

Sound Food Uprising Teens – Surveys

33

Sound Food Uprising Adults - Surveys

The Beecher’s Foundation and Pierce County Building A Better Food Future For All

09/13/2018 of 51

Appendix B Literature Review

Literature review objective/question: What is the current evidence-based practice around direct nutrition education in school settings? Include methodology (direct education and data collection) and outcomes.

Summary

Current evidence-based practice around direct nutrition education in school settings include a variety of interventions and outcome measures. Peer-reviewed articles emphasized the importance of grounding an intervention design in theoretical models (such as a social cognitive theory or social-ecological model) which consider a variety of influences on human behavior. For example, from a perspective of the social-ecological model, one study’s findings suggest that children’s eating behaviors need to be considered at the intersection of children’s health literacy/attitudes, community efforts, and media/screen time use. We have included an overview of theories often used in direct nutrition education interventions in section III.

The literature suggests barriers to children’s consumption of healthy food include cost, logistical barriers, unfamiliarity with produce, and parental opposition. These should be considered moving forward.

Multi-component interventions (those that include education, cooking, physical education, gardening, etc.) and interactive learning are more successful than educational-only and/or didactic styles of learning. Specifically, art projects and collaborative activities, like games, were very successful.

Most of the literature reviewed include one pretest and one to two post-tests for evaluation (rarely more than two post-tests). Common practices are a survey/questionnaire at the beginning and end of the intervention, then possibly one post-test further out. Some studies used focus groups and interviews as formative research pre-intervention and/or to supplement quantitative surveys post-intervention.

Key informant interviews with our own local nutrition educators revealed some lessons learned for best practices for direct-ed nutrition interventions, including evaluation planning and community engagement. These lessons learned can be found in section IV.

I. BRFSS and Healthy Youth Survey data for Clover Park Schools and Pierce County/Lakewood

Pierce County is often an outlier in terms of health outcomes compared to other Washington State counties.

65.9% of Pierce County adults are overweight and obese, 30.3% of 8th, 10th, and 12th graders are overweight


09/13/2018 of 51

and obese, and diabetes prevalence is concentrated in the lowest income zip codes.14,15 Neighbors living less

than a mile apart can have up to 8 years difference in life expectancy.16 These disparate health outcomes are

often driven by diet-related disease.

II. Peer-reviewed Literature

1. Systematic Review of direct education nutrition programs to increase vegetable intake. Appleton K, Hemingway A, Hartwell H, et al. Increasing vegetable intakes: rationale and systematic review of published interventions. European Journal Of Nutrition. April 2016;55(3):869-896.

The problem/purpose: Increasing work suggests health benefits from a high consumption specifically of vegetables, yet intakes remain low, and barriers to increasing intakes are prevalent making intervention difficult. A systematic review was undertaken to identify from the published literature all studies reporting an intervention to increase intakes of vegetables as a distinct food group.

(Review of 77 studies including 140 interventions (133 of which were on children) aimed at increasing vegetable intake. Interventions used a range of techniques from providing information and education on nutrition or nutrition-related skills, providing education plus a demonstration, or gardening experience, providing tailored information, providing information on social norms, invoking choice, and invoking a memory.

Results: Greater percent success is currently found from environmental, educational and multi-component interventions, and long-term effects and cost-effectiveness are rarely considered. A focus on long-term benefits and sustained behavior change is required. Knowledge accumulates over time and experience, and it can be difficult for individuals to pinpoint the exact source/ sources of beneficial education.

2. The effectiveness of health literacy intervention on nutritional knowledge, attitudes toward healthy eating, food preferences, and healthier eating behavior. (Includes the influence of screen time/media on healthy eating; methods: iPads, kids-friendly apps for a survey).

Bissell K, Baker K, Pember S, et al. Changing the Face of Health Education via Health Interventions: Social-Ecological Perspectives on New Media Technologies and Elementary Nutrition Education. Health Communication. Aug 24, 2018: 1-10.

14 Behavioral Risk Factor Surveillance System (BRFSS), 2013-2017, Pierce County

15 Healthy Youth Survey 2016, Pierce County (8th, 10th and 12th graders). https://www.doh.wa.gov/DataandStatisticalReports/DataSystems/HealthyYouthSurvey/Reports

16 2015 Health Equity Assessment, Tacoma-Pierce County Health Department. https://www.tpchd.org/home/showdocument?id=196

https://www.doh.wa.gov/DataandStatisticalReports/DataSystems/HealthyYouthSurvey/Reports

https://www.tpchd.org/home/showdocument?id=196


09/13/2018 of 51

Purpose: Employ a social-ecological perspective in determining the effectiveness of a longitudinal health literacy/health intervention program on children’s nutritional knowledge, attitudes toward healthy eating and exercise, food preferences, and eating behavior. This study examined the effectiveness of the health literacy/health intervention program on children’s nutritional knowledge, attitudes toward healthy eating, food preferences, and healthier eating behavior.

Grades 2, 3 and 4. using new media technology to record their food intake at home and at school.

Data collection methods: Pre-post; pretest questionnaire (Qualtrics) with iPad program specific to kids with easy-to-understand measurements and images for food intake reporting. Participants reported demographic information, household dieting information, and household screen time use at meal times. The questionnaire also included questions about nutritional knowledge, attitudes toward eating and exercise, and food preferences.

Three points of collection: Time 1 represented an initial pretest and initial health intervention; the second round of data collection to the successfulness of the intervention program was conducted 1 month later, followed by data collection 1 year later.

Results: The health literacy program was successful with positive increases observed for cognition, attitudes, and behavior. From a perspective of the social-ecological model, the findings suggest that children’s eating behaviors need to be considered at the intersection of children’s health literacy/attitudes, community efforts, and media/screen time use.

3. Using a Theory of Planned Behavior to focus on attitude and perceived control over behavior change. Arrizabalaga-López M, Rada-Fernández de Jáuregui D, Rodríguez V, et al. A randomized controlled trial of a program based on the theory of planned behavior to promote fruit and vegetable intake among schoolchildren: PROFRUVE study protocol. BMC Public Health. July 4, 2018;19(1)


09/13/2018 of 51

Study objective: to evaluate the effectiveness of an intervention program in increasing fruits and vegetable consumption in schoolchildren aged 8 to 10 and based on TPB.

Participants took part in school activities (education; cooking classes), outside activities (visits to local markets and farms) and home activities (food diary for fruit and vegetable consumption reviewed with families). Using the Theory of Planned Behavior, the intervention focuses on attitude, subjective norms and perceived control in behavior change.

Methods: Cluster randomized control trial. Data-collection using validated surveys, 7-day food records, 24-hour reminders, and questionnaires. Pretest before, post during, post 1 year.

Results: Study in progress (use for methods only).

4. Communication preferences in reaching parents – text messaging. Luesse, H.B., Paul, R., Gray, H.L. et al. Matern Child Health J (2018) 22: 958

The problem/purpose: The most common barriers to parents providing healthy foods to their children were accommodating child preferences and familial opposition.

This research aimed to improve strategies to reach parents and directing tailored nutrition education to them is needed. The study identifies communication preferences to inform intervention strategies for effectively reaching low-income urban minority families.

Data collection methods: Semi-structured focus groups with parents of elementary children.

Results: Multi-media supports such as workshops, flyers, and text messaging may be useful to facilitate the sharing of information to minimize the tensions between intentionality and reaching desired goals to be healthy. Some parents thought that information received through text messaging could be easily shared and would act as a voice of authority to support child behavior change.

5. Interactive learning (vs. didactic) has more successful outcomes for children. Gandhi M, Feller E, El Rayess F. Art and movement in nutrition education. Medical Education [serial online]. November 2016;50(11):1160.

The problem/purpose: Studies performed on nutrition curriculum (reading and math exercises) demonstrated limitations in its effectiveness, even showing a decline in healthy eating and exercise habits across a semester.

The study aims to modify an existing curriculum to encompass a broader diversity of learning styles.

A 5-week program involving twice weekly nutrition classes for fourth and fifth graders. This initiative incorporated visual art projects, kinesthetic learning tools, and collaborative activities. Students


09/13/2018 of 51

created projects that would help them remember what they had learned in each lesson and keep them engaged in the course.

Data collection methods: pre-post surveys (at beginning and end of 5-week course).

Results: Statistically significant improvements in exercise, eating fruits and vegetables, and cutting out sugary beverages. An interactive learner-based initiative is potentially more effective than a mainly sedentary didactic education program. The impact of this pilot curriculum was limited because of its short (5-week) follow-up time.

6. (Methodology only – study in progress) Barriers of low-income families to healthy eating, produce/agriculture.

Seguin R, Morgan E, Jilcott Pitts S, et al. Farm Fresh Foods for Healthy Kids (F3HK): An innovative community supported agriculture intervention to prevent childhood obesity in low-income families and strengthen local agricultural economies. BMC Public Health. April 8, 2017;17:1-20.

The problem/purpose: Barriers to healthy eating. Cost, logistical barriers, and unfamiliarity with produce items inhibit low-income families consuming fresh produce.

The study aims to assess the impact of subsidized, or “cost offset,” community supported agriculture participation coupled with tailored nutrition education for low-income families with children to improve access to and consumption of fresh produce.

Low-income families recruited with at least one child ages 2-12 years old. Nine skill-based and seasonally-tailored healthy eating classes, and the provision of basic kitchen tools. The primary outcomes are children’s intake of fruits and vegetables and foods high in sugar and/or (solid) fat, as well as diet quality; secondary outcomes include physical, behavioral, psychosocial, and environmental variables.

Data collection methods: (4-year study) Pre-post survey data collected at baseline, and in the fall and spring for 3 years; Plus, Quarterly dietary recall. Physical components measured at baseline and in the Fall of years 2-4. Height, weight and Dermal carotenoid. Qualitative data collection (interviews) with stakeholders as formative research pre-intervention.


7. Integrated self-efficacy and planning tools more successful than education-only interventions. Arrizabalaga-López M, Rada-Fernández de Jáuregui D, Rodríguez V, et al. A randomized controlled trial of a program based on the theory of planned behavior to promote fruit and vegetable intake among schoolchildren: PROFRUVE study protocol. BMC Public Health [serial online]. July 4, 2018;19(1): N.PAG.


09/13/2018 of 51

Study objective: Randomized control trial compared a theory-based intervention with a health education session. Primary outcomes were fruit and vegetable consumption, intent to consume more fruits and vegetables, planning to consume more fruits and vegetables and dietary self-efficacy.

The intervention was a training program including general health and nutrition education plus psychological program focusing on self-efficacy enhancement and planning skills; the control group received only the general health and nutrition education lesson.

Methods: data collection – pretest before the intervention, post 1 – 1 week following, and post 2 – 6 weeks.

Results: A social-cognitive intervention to improve fruit and vegetable consumption was superior to a knowledge-based education session. Self-efficacy and planning seem to play a major role in the mechanisms that facilitate dietary changes.

*Successful use of incentive: To prevent attrition, eight gift vouchers were offered in sweepstakes as an incentive to those who completed either the intervention or health education program, and who also responded to all questionnaires at three points in time.

8. (Methodology only) Including the use of theoretical models in study design – Social ecological model and

social cognitive theory.

Hopkins L, Fristad M, Gunther C, et al. Camp NERF: methods of a theory-based nutrition education recreation and fitness program aimed at preventing unhealthy weight gain in underserved elementary children during summer months. BMC Public Health. October 26, 2016;16(1):1-12. Available from: Academic Search Alumni Edition, Ipswich, MA.

Study objective: This is the first, evidence-based intervention to address the issue/prevent excess weight gain during the summer months, among this population.

Camp NERF (partner USDA Summer Food Service Program) - 8-week multicomponent (nutrition, physical activity, and mental health) theory-based program for underserved school-age children (K-5) given in summer months.

Methods: Data- collection – pre/post survey at beginning and end of the 8-week course (nutrition, physical activity, and mental health questionnaires), dietary recall, physical measurements (height, weight, blood pressure, and waist circumference). Also, home interviews with child and parent/caregiver. Use of social ecological model (SEM) and social cognitive theory as theoretical models.



09/13/2018 of 51

9. Successful use of interactive learning in direct nutrition ed. (board game) Viggiano A, Viggiano E, Amaro S, et al. Kaledo, a board game for nutrition education of children and adolescents at school: cluster randomized controlled trial of healthy lifestyle promotion. European Journal Of Pediatrics [serial online]. February 2015;174(2):217-228

Study objective: To assess whether the game could be an effective educational tool to learn healthy eating habits.

Kaledo is a board game developed to promote nutrition education and improve dietary behavior. Participants 9-19 years old from 20 schools, including one control group. Primary outcomes measured were scores on “Adolescent Food Habits Checklist”, scores on a dietary questionnaire, and BMI z-score. Questionnaires included measures of nutrition knowledge, beliefs about healthy and unhealthy food, physical activity and lifestyle, food habits, self-efficacy (student’s ability to assume attitudes and behaviors can improve their health status), barriers to change.

Methods: Data collection -Food habits checklist, Questionnaire and BMI z-score. Pre-test at intervention (baseline data). Post 1 @ 6months Post 2 @ 18 months.

Results: Kaledo improved nutrition knowledge and dietary behavior over 6months and had a sustained effect on the BMI z-score. Therefore, it may be used as an effective tool in childhood and adolescence obesity prevention programs.

III. Theoretical models (commonly used as a framework in the design of behavior change interventions)

In a meta-analysis that studied the influence of behavioral theory-based interventions on children’s FV intake, interventions were more effective when it was theory-based than not–theory based.

The above-referenced literature used the following models

o Social Ecological Model The Social-Ecological Model helps us understand how layers of influence can shape a person's personal behavior. For example, strong evidence demonstrates that “school policies designed to enhance the school food setting leads to improvements in the purchasing behavior of children, resulting in a higher


09/13/2018 of 51

dietary quality of the food consumed during the school day.”17

o Social Cognitive Theory (also referred to as Social Learning Theory) The core constructs of SCT are goals, perceived self-efficacy, outcome expectancies, facilitators, and impediments. “Goals direct the behavior. Perceived self-efficacy is the belief that one is capable of performing the goal behavior in spite of obstacles. Outcome expectancies are the perceived costs and benefits of the behavior. Facilitators and impediments are social structural factors that include environmental aspects that could potentially predict goals and behavior.”18 19

17 https://health.gov/dietaryguidelines/2015/guidelines/chapter-3/social-ecological-model/ 18 Plotnikoff R, Lippke S, Courneya K, Birkett N, Sigal R. Physical Activity and Social Cognitive Theory: A Test in a Population Sample of Adults with Type 1 or Type 2

Diabetes. Applied Psychology: An International Review [serial online]. October 2008;57(4):628-643. 19 Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory: Englewood Cliffs: Prentice-Hall, Inc.

https://health.gov/dietaryguidelines/2015/guidelines/chapter-3/social-ecological-model/


09/13/2018 of 51

Theory of Planned Behavior

Attitude, subjective norms and perceived control are determinants that influence both the intention to consume and the action of consuming food. In the context of diet education, the theory of planned behavior assumes eating behavior and the intention of consumption determinants are both personal and environmental.

V. Evaluation – survey response rates and common practices

Industry standard response rates for health-based surveys or questionnaires can range anywhere from 20%-80% but most commonly lie between 30% - 40% for internal surveys (meaning surveys are given to a selected


09/13/2018 of 51

sample or cohort of participants within an organization, study, etc.) A successful response is very dependent on things like study design, survey execution, the motivation of participants, and incentives given. Well-executed surveys may reveal 80% or higher response rates, which has become the ‘gold standard’ in public health research.

Studies and references to standard rates and how to improve them:

1. Survey Gizmo. What’s a good survey response rate? 2018. Web. Accessed 12 September 2018. http://www.surveygizmo.com/resources/blog/survey-response-rates/

2. McMaster et al. An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC Medical Research Methodology (2017) 17:73

3. Trespalacios J, Perkins R. Effects of Personalization and Invitation Email Length on Web-Based Survey Response Rates. Techtrends: Linking Research & Practice To Improve Learning [serial online]. July 2016;60(4):330-335.

4. Millar M, Dillman D. Improving Response to Web and Mixed-Mode Surveys. Public Opinion Quarterly [serial online]. June 2011;75(2):249-269.

5. Shu-Fang S, Chieh-Hsing L, Liao L, et al. Health literacy and the determinants of obesity: a population-based survey of sixth grade school children in Taiwan. BMC Public Health [serial online]. March 22, 2016;16(1):1-8.

6. Chaintarli K, Ingle S, Bhattacharya A, Ashiru-Oredope D, Oliver I, Gobin M. Impact of a United Kingdom-wide campaign to tackle antimicrobial resistance on self-reported knowledge and behaviour change. BMC Public Health [serial online]. May 12, 2016;16(1):1-9

IV. SNAP-Ed resources (key informant interviews)

Elizabeth Kimball, Public Health – Seattle & King County and Hope Roberts, MultiCare Health Systems (MHS). Direct-Ed school nutrition programs in King (PHSKC) and Pierce County (MHS).

Measurement tools/Evaluation: Due to resource restraints, neither organization did follow-up tests. Both Elizabeth and Hope suggested them, saying it would have been ideal if they could have done some kind of follow-up.

Follow-up is always hard (attrition). Creative incentive ideas for follow-up post collection:

• Get consent from parents to send them a text with the survey at 1-month, 3-month or 6-month follow-up.

• Paper surveys on a table – give a $2 bill once completed.

• Online survey – Promo code is given at the end ($2-$3?) for use on Amazon or a similar site.

http://www.surveygizmo.com/resources/blog/survey-response-rates/


09/13/2018 of 51

Interviewees commented on the difficulty in measuring behavior change in kids because there is little they can control (environment, policy, their parents/family/culture).

PHSKC measures: Outcomes measured – knowledge, self-efficacy (confidence questions), and consumption of foods.

Nutrition literacy and knowledge can be measured as behavior change, “We like to measure knowledge because it leads to behavior change, though DOH would disagree with us on this.” – Hope Roberts

For kids, be sure to ask about choices on the survey, “What choices would you make? If given a choice between these 4, which would be the best food choice?” This measures their decision-making and goals. Ex. “If you have $20 to buy lunch and dinner, what should they do? (multiple choice where would you find the best options for meals? Subway, Corner Store, Safeway, etc.)

MHS uses Con-Q Kansas behavioral measurement tool - Measures food intake through asking questions such as, “yesterday how much of this food did you eat?” It also includes four questions that measure knowledge.

MHS uses unique identifiers/kids to write down their Student ID number. Tracks individuals. No age, gender, and demographic questions.

PHSKC uses 6-digit ID codes/unique identifiers for each participant in order to follow their results. --Using the first letter of the first name, the first letter of the last name, month born (2 digits), day born (2 digits), for example:

ET0408

Timeline Q: What’s your usual project timeline? Late Oct/early Nov. Semester ends late Jan. Once weekly.

• Pre-test on the first day of class (if time allows, otherwise have them take it the week before the sessions start so it doesn’t cut into class time). Post-test in January at the end of the session.

• Teachers are always happy to administer the pre/posttest in sessions before or after the actual workshops, so it doesn’t cut into workshop time (less they need to plan for their classes!)

• Carnival on last day – kids love it!

Connecting with Parents/ethics/consent MHS requested an invite to back-to-school open-house – they set up a table with “goodies”, so parents can get to know them personally and learn about the program.


09/13/2018 of 51

• Tacoma School district – can’t call it a survey (call it a pre and post-test) make sure the survey says OPTIONAL for ethics purposes.

• Neither MHS or PHSKC programs warranted IRB humans subjects research approval. Successful initiatives – King County

Q. Tell me about a successful program and why it was successful? (For the purpose of this interview we defined success by study completion and results of some kind – negative or positive- that you learned from).

1. 9 schools, measuring kids’ preference for fruits and vegetables Introduced foods that they may have perceived they didn’t like. For example, spinach – let’s plant spinach and watch it grow. Then use it in a recipe in three different ways Kids received the intervention (K-5) every year. Do kids like the veggie now after the program? – results increased over time. (They realized that they used Spinach consecutively as one of the veggies for 10 years of the program. Analyzed the data and saw the kids’ preference increase over time. 2010-10% like spinach 2011 – 15% liked spinach 2012 – 20% liked spinach. (This was because the kids learned more and more about spinach each year and grew to like it!)

2. The campaign around sugary beverages (PSE, not direct education) Taught how to read labels and make choices when faced with a different option. For example White milk vs. chocolate milk in the school cafeteria. Had cafeteria workers put white milk in front of chocolate milk. School district’s ordering logs showed a reduction in chocolate milk orders = the kids were buying less of it.

Although these two programs were successful, they were not sustainable over time unless the SNAP-ED team stayed in the schools. i.e. they were not sure the kids would continue on this healthy eating path without being prompted with more intervention (a study with longer follow-ups would reveal this). “In order to make a change, be a constant presence in the school.” - EK “We brought in staff and expertise, which the schools appreciated.” - EK Multi-component initiatives were more successful. For example, education plus getting staff involved, and school-wide campaigns – gardens, PE classes, etc. Also, the schools had a dedicated nutrition educator paired with an AmeriCorps staff member which helped. (For more SNAP-ED resources and interventions, contact Emily Turk [email protected] or Katie Tong directly at [email protected]).

mailto:[email protected]

mailto:[email protected]


09/13/2018 of 51

Appendix C Evaluation SMART Objectives (by age group)


09/13/2018 of 51

sound food uprising: lakewood 2019 evaluation preliminary ......marketing messages, and cook from...

Documents