Western Extension Risk Management Education Center2019 Project Director Orientation

1:00PM Pacific Time – May 2, 2019

2019 Project Orientation

Documenting Producer Success through Performance MeasuresA Systems Evaluation Model

Shannon Neibergs, Director

Jo Ann Warner, Associate Director

Heather Schriver, Grants & Communications Coordinator

Webinar Logistics

• Type questions in chat box• Mute computer mic• Mute phone if calling in by phone• Recording and posted on website• Performance Measure/Evaluation resources on web

• Networking – emails

2019 Funded ProjectsTheresa Kiehn, theresa@agsafe.org, AgSafe, Labor Considerations for Scaling Up Production

Michael Patrick, jmpat@nmsu.edu, NMSU, Navajo Ag & Natural Resource Sustainability Project

Ryan Larsen, ryan.larsen@usu.edu, USU, Risk & Profitability in Beef Production

Max Bennett, max.bennett@oregonstate.edu, OSU, Risk Mgmt Ed for Forest Producers in SW OR

Hernan Tejeda, htejeda@uidaho.edu, Regents UI, Wkshps to Improve RM & Fin. Deci-Mkg for Dairy Frmrs

Justin Oldfield, justin@calcattlemen.org, CA Cattlemen's Assoc., Livestock Traceability

Megan Van Emon, megan.vanemon@montana.edu, MSU, MT Master Cattle Producer's Program

Natalia Pinzon, natalia@mesaprogram.org, MESA, CA Farmers Build Fire Resilience

Tracy Mosley, tmosley@montana.edu, MSU Ext. Park Cty, Farm & Ranch Estate & Transition Planning

Ruby Ward, ruby.ward@usu.edu, USU, RME for Native American Farmers & Ranchers

Ruth Dahlquist-Willard, rdahlquistwillard@ucanr.edu, UCANR, Bilingual Food Safety Trng for SE Asian & Latino Small-Scale Farms

Jean Brokish, jean.brokish@oahurcd.org, Oahu RC&D, Skill Up! Utilizing Agritourism to Grow Farm Income

Kirk Hanson, kirk@nnrg.org, NNRG, Reducing Prdcr Rsk in the San Juan Islands Thru Improved Forest Mgmt

Jay Olsen, jay.olsen@snow.edu, Snow College, Anlyzng Contributions of Entrprss Dvrsfctn to Frm Prftblty for

Chris Brown, chris@albafarmers.org, ALBA, Self-directed Mktg Approaches for Small Holder farms (SMASH)

What We Are Going to Cover

Alignment of ERME Goals with your Program Goals Using a Systems Evaluation Model

Risk Management OverviewSystems Evaluation ApproachReporting – Communicating Success with Evaluation

FindingsNew ERME Reporting Requirements to Establish a

National “Understanding” Benchmark MetricInvoicing and Reporting Logistics

USDA/NIFA Programs

Grounded in USDA Strategic GoalsRequire planning and accountability measuresRely upon streamlined data collectionReduces reporting burdenCreates a more transparent reporting process

Robust evidence based system Shared knowledge base

Alignment of your Project Goals with ERME Program Goals

Extension Risk Management Education (ERME)Mission:Educating America’s farmers and ranchers to manage the unique risks of producing food for the world’s table. Overarching Goal:Helping farm and ranch families improve their economic viability through targeted risk management education.

Extension Risk Management Education

Economic Viability:

Legislative Authority: To educate agricultural producers with training that improves their economic viabilityEconomic viability refers to the farm’s or ranch’s

ability to improve profitability and to increase opportunities for equity growth over time.Successful projects provide evidence of economic

gains for producers.

ERME Program Goals – Linking the Five Risk Areas to Improved Economic Viability

1. Production: Ensure profitability through increasing yields and reducing input costs

2. Market: Ensure increased revenues by increasing sales and execution of optimal market pricing

3. Financial: Ensure profitability by increasing cash flow and growing business equity

4. Legal: Ensure profitability by reducing legal risk exposure and increasing compliance

5. Human: Ensure profitability by developing labor and human capital for worker and family business success.

**Economic or profit indicator of reduced risk which is constant across all 5 risk areas

Performance Measures & Documenting Producer Success Systems Evaluation ModelIntegrates application content

1. Results – what you want participants to accomplish2. Rationale – Project Results Narrative3. Curriculum/Tools/Resources/Collaborators4. Activities (Trainings/workshops/field trips)

Establish a successful monitoring and performance measurement systemInput into the Results Verification System (RVS)

1. Quarterly Progress Reports2. Final Report

Article: Program Evaluation and Results Verification: https://westrme.wsu.edu/news/winter-2017-newsletter/program-evaluation/

A Systems Evaluation Approach:Creating a Road Map for your Project

Insert into a framework that:Aligns producer outcomes with ERME economic

program goalsProvides a rationale for how your project activities

will lead producers to the desired resultsMonitors project performanceMeasures results across the life of the project (short-

medium-long term)Uses different types of data (from

participants/industry and government) to measure progress towards improved profitability

End Goal for Producer Actions is Improved Profitability

Five levels of producer actions in order of increasing producer engagement:

1. Understand2. Analyze3. Develop4. Decide5. Implement

**

KnowledgeShort Term

Necessary Pre-ConditionsActions

Medium

ConditionsLong Term

A Systems Evaluation Approach –What You Will Measure

1. Short term learning – understanding (immediately after initial or introductory training)

2. Short term objectives – develop/analyze (changes in behavior through undertaking specific actions within 3-6 months)

3. Medium term strategies – decide (changes in practice that result in a measurable reduction of risk within 6-12 months)

4. Long term outcome measures – implement(achievement of improved economic viability with evidence of realized economic gains within 12-18 months)

A Systems Evaluation Approach –How You Will Verify

Conduct a series of evaluations that provides evidence of achieved outcomes:

a) Retrospective Evaluation for “Understanding” (Near Term)

b) Follow up Evaluations – within 3-6 months (Short - Medium Term)

c) Follow up Evaluations – within 6-18 months (Long Term)

A Systems Evaluation Approach –Questions that Measure Results

Develops detailed evaluation questions from your proposed results that answer the expectation of improved profitability for your participants.

Do they adequately describe and can they effectively measure the producer results identified in the proposal?

**Program Resources:Survey Design Techniques: https://s3.wp.wsu.edu/uploads/sites/2162/2017/03/Program-Evaluation-and-Survey-Design-Techniques.pdf

Measuring Understanding – A Short Term Learning Goal

Result: Livestock producers will learn how to develop a data-driven price strategy for direct-to-consumer markets.

Evaluation question that measures level of understanding: How do you rate your understanding of data-driven price strategies for direct to consumer markets after participation in this workshop? (Likert numbered scale)

Retrospective Evaluation for Measuring “Understanding”• An example of the retrospective evaluation

question:

• How do you rate your understanding of market risk management for developing a data driven price strategy for direct to consumer markets as a result of participating in this risk management workshop (series)?

No A A CompleteUnderstanding Little Neutral Lot Understanding

a. Before the meeting 1 2 3 4 5 b. After the meeting 1 2 3 4 5

Measuring Short Term Actions (Develop/Analyze)

Result: Livestock producers will initiate a marketing plan specific to their own operation.

Evaluation question that measures adoption of knowledge/skills, actual changes in behavior and practice. (Short term objectives)Please indicate your level of agreement with the following statement(s) (Likert rating scale): I adopted a data driven marketing strategy for interaction with specific meat processing outlets.

Measuring Medium Term Actions (Decide)

Result: Livestock producers will transition their herds from commodity to differentiated markets.

Evaluation question measures changes in practice that result in measurable reduction of risk. Please indicate your level of agreement with the following statement(s) (Likert rating scale): I followed a data driven marketing strategy to convert my herd to a certified management and marketing plan.

Measuring Long Term Outcomes (Implementation)

Result: Livestock producers will transition their herds from commodity to differentiated markets.Evaluation question measures achievement of improved economic viability or realized economic gainPlease estimate the economic gain from higher prices received for products in differentiated markets:$0-$2,000$2,000-$5,000$5,000-$10,000$10,000-$25,000$25,000-$50,000

Benefits Accrued – Long Term Economic Improvement for Producers

Evaluation question measures achievement of improved economic viability or realized economic gainWhat is the value to you of attending this workshop series; or the benefits you estimate will accrue to your operation from changes you have made as a result of attending these workshops?Less than $100$100-$500$500-$1000More than $1,000

Evaluation FindingsContribute to program developmentEvidence Based

Collection of data that is useful and validates change (measures what is really happening)

Links producer success stories (anecdotal and testimonials) with data for accountability.

Builds upon continuous collaboration with participants to earn their trust.

Produces a return on investment (ROI) for your target audience for improved economic viability. (Benefits accrued to Producers)

Fulfills ERME goal for economic viabilityProvides justification for ROI of NIFA fundsWorthwhile investment of taxpayer dollars

Systems Evaluation Model Within a Performance Measure Framework –

Your Final Report & Public Value Level Impacts

Demonstrates: Achievement of participant outcomes and program success in a public reporting system available to all stakeholder partners

USDA GovernmentalCongressionalPublic/Private Sector StakeholdersCommunitiesFunding AgenciesUpper Administration at Universities

NEW Reporting Requirements

1. ERME Benchmark Metric for “Understanding”

a) Each project will be asked to assess changes in understanding using a retrospective design.

b) A retrospective evaluation measures participants’ previous versus current knowledge on a risk topic area(s) at the end of a workshop or training series.

**All ERME projects are expected to conduct this retrospective evaluation at the end of their workshop or training series

ERME Retrospective Evaluation Across the Project Portfolio

• Prepare a retrospective evaluation for each area of risk that your project covered (from the five broad risk areas)Develop a separate evaluation question for each risk.

• Use a 5 point numbered (Likert) scale1 will indicate No Understanding to5 indicating Complete Understanding

• Administer to participating producers at the end of a workshop or training series.

Retrospective Evaluation for Measuring “Understanding”• The basic retrospective evaluation question for each risk area

is as follows:

• How do you rate your understanding of ___(insert risk area)___ risk management (including ___(insert relevant topics)___) as a result of participating in this risk management workshop (series)?

• The response for each evaluation question is:No Complete

Understanding Understandinga. Before the meeting 1 2 3 4 5 b. After the meeting 1 2 3 4 5

Retrospective Evaluation for Measuring “Understanding”• An example of the retrospective evaluation

question:

• How do you rate your understanding of market risk management for developing a data driven price strategy for direct to consumer markets as a result of participating in this risk management workshop (series)?

No CompleteUnderstanding Understanding

a. Before the meeting 1 2 3 4 5 b. After the meeting 1 2 3 4 5

Retrospective Evaluation for Measuring “Understanding”• An example of the retrospective evaluation

question:

• How do you rate your understanding of financial risk management for developing a business plan and enterprise budgets; and for using livestock decision aids as a result of participating in this risk management workshop (series)?

No CompleteUnderstanding Understanding

a. Before the meeting 1 2 3 4 5 b. After the meeting 1 2 3 4 5

NEW Reporting Requirements

2. Write at least one story about an individual producer in the following format:a) Describe the situationb) Include outcomes accomplished by that producerc) Describe any overarching impacts; i.e. to the

family/community/environment etc.i. Include evidence or indicators of improved profitability or

economic gaind) Include a testimonial from this individual producer or

family member

Story Format ExampleSituation: This family owned enterprise was running a fairly traditional cow calf operation, growing their own hay to feed their livestock. Constraints on their time was an ongoing issue…..Outcomes: Conducting a comprehensive evaluation of the their operation through the use of QB’s resulted in the following outcomes:

1. Significantly reduced feed costs…2. Acquired additional income through…3. Transitioned herd from….

Impacts: As a result of these changes, family relationships improved....and their average economic gain for the year was $....Producer Quote: “This program provided me with the tools I needed to convert my herd from a conventional operation to include differentiated markets….”Link to Producer Story: https://s3.wp.wsu.edu/uploads/sites/2162/2019/03/WCRME-Outstanding-Project-Success-Story-2019-1.pdf

RVS Under Development

Data input for Retrospective Evaluation

Story input for Individual Producer Story

Questions/Discussion?

Impact Reporting – Communicating your Success in the Results Verification System (RVS)

Reflect back on your Systems Evaluation Model – At the end of 18 months: What strategies did producers put into practice that resulted in

measurable reductions of risk? How many producers were able to reduce input costs/increase

sales/ensure profitability by growing business equity?a) Include indicators of profitability and;b) Evidence of realized economic gain

Impact:

Describe how your project led to improved economic resilience and viability for your producer participants.

Impact Reporting – Communicating your Success in the Results Verification System (RVS)

Summary of Evaluation Findings1. Project Overview – PAST Tense2. Proposed Results – ACTUAL Results and # of participants who

achieved

Reflect back on your Systems Evaluation Model1. Project Comments

a. Unexpected resultsb. Project Improvementsc. Keys to Success

Data collection methods and instruments that capture producer stories and indicate improved economic resilience and viability

a) Individual Story(s)/Quotes

Award Management1. http://westrme.wsu.edu/award-management/tools-

for-project-directors/2. http://westrme.wsu.edu/award-

management/reporting/education-projects/

**Bookmark these pages!

Award Management(Tools for Project Directors Tab)

• Results Verification – Documenting Risk Management Outcomes through Performance Measures

• Evaluation Resources• Funding Acknowledgement and Logo Requirements• InvoicingUniversityNon-UniversitySample Invoice

Award Management

• Acknowledgement of Support:• This material is based upon work supported by

USDA/NIFA under Award Number 2018-70027-28587.

• Logos – USDA/NIFA and Western ERME

Award Management

– Acknowledgment of Support• Proper Naming Convention

Western Extension Risk Management Education Center

OrWestern ERME Center

Invoicing

Invoicing – General

– Submit invoices monthly– Email to SPS@WSU.EDU– Invoice Certification Statement

• Include on your invoice– Sample Subcontract Invoice:

https://s3.wp.wsu.edu/uploads/sites/2162/2017/03/2016-9.-Attach-6-Sample-Invoice.pdf

Invoicing

• Invoicing Certification Statement:

• Include on your statement

InvoicingInvoicing Procedures

• Cost Reimbursable Subcontract• Supporting documentation

• Accounting of salaries and wages• Copies of receipts etc. from vendors• Travel detail

**ERME 2019 Subaward Attachment 4 – Non University or University

Reporting & Invoicing PolicyInvoice payment Current and satisfactory reporting

Progress Notes Section of Online Report No Program Delivery in a quarter? Write a short

note in the Progress Notes Section of report. Submitting invoices in the first 3 months of your

project?a) Check off the project steps that have been

completed or;b) Write a brief description of the program delivery

that has occurred to date in the Progress Notes Section of the report.

Questions?

Reporting Dates and Timeline – (18 Month Period of Performance)

Online Reporting

Online quarterly reports (2019/2020 Timeline)

• July 1, 2019• October 1, 2019• January 2, 2020• April 1, 2020• July 1, 2020• Final Report is due October 15, 2020.

October 15, 2019 Proposed Results, Project Steps, Overview

Auto populated in the online report system

**Additional progress reports may be submitted or report info entered at any time.

Reporting Resources for the RVS System

• http://westrme.wsu.edu/award-management/reporting/education-projects/

• http://westrme.wsu.edu/award-management/reporting/exploratory-grants/

• http://extensionrme.org/Resources.aspx#RVS

Reporting Tutorialhttp://extensionrme.org/pubs/ReportingTutorial.PDF

Entering Your Report

Important Information

Reporting Portal

Instructions Window

Submitting A Progress Report

Reporting Status

Final Report

Thank you!Questions?

Jo Ann Warner 509-477-2168warnerj@wsu.eduShannon Neibergs 509-335-6360sneibergs@wsu.eduHeather Schriver 541-891-9190hschriver@wsu.edu