improving final report writing in california’s tobacco control projects jeanette treiber, phd...
TRANSCRIPT
![Page 1: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/1.jpg)
Improving Final Report Writing in California’s Tobacco Control Projects
Jeanette Treiber, PhD
Robin Kipke, MS
UC Davis
![Page 2: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/2.jpg)
Background
CDPH funds approx. 100 local programs for tobacco control (mainly for policy work)
CDPH also funds evaluation capacity building Center (TCEC)
3 yr funding cycles require evaluation plan and reporting
CDPH provides reporting guidelines Reporting guidelines are not well adhered to 2007 reports received relatively low scores
![Page 3: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/3.jpg)
The Problem
Evaluators do not think that their work is being used (Torres et al. 1997)
“Evaluators’ most long-standing concern about the lack of use of their work has to do with the dustgathering qualities of unread final reports.” (Preskill and Russ-Eft 2005)
Utilization focused evaluation relies in part on good reporting
![Page 4: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/4.jpg)
Objectives
Build reporting capacity of local California tobacco control projects
Identify ECB practices that can help in the improvement of report writing
Reasons Weak evaluation reporting Reporting procedures not followed Aggregate data hard to generate from reports Questionable use as formative evaluation tools
![Page 5: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/5.jpg)
Scoring Criteria (Excerpt)
Evaluation Design Section
Type of design (experimental, quasi-experimental, or non-experimental)
0 1 2
Reason for selecting the design used 0 1 2
# of times data are collected, when data are collected (pre-tests, during intervention, and/or post-tests), # of groups compared (if any), and whether activities varied by group
0 1 2
Any limitations of the design as a way to assess the intervention process and/or outcome
0 1 2
![Page 6: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/6.jpg)
Capacity Building Methods & Measures
Regional one-day training events throughout California
Webinar One-day training satisfaction survey Compare scores of:
Overall mean 2007 vs 2010 Paired t-test of comparable sub-group of 2007
vs. 2010 reports Score improvement rate of those that attended
vs. those that did not attend training events
![Page 7: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/7.jpg)
Results
Attendance: One-day training: 79 projects; 103 participants* Webinar: 21 projects One-day training + webinar: 20 projects
![Page 8: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/8.jpg)
Training Agenda
Why Final Evaluation Reports Matter Anatomy of a Final Evaluation Report (FER)
a review each section, scoring criteria, common pitfalls hands-on exercise: score and discuss a sample FER
Writing the Results Section a review of several examples hands-on exercises taken from actual progress reports
Making evaluation reports work for multiple audiences How to repackage information in the FERs
Office Hours Individual consultation about your Final Evaluation
Report or evaluation activities
![Page 9: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/9.jpg)
Webinar Agenda
Q & A session on commonly asked questions Available guidance/resources Identifying primary report audience The writing process How to handle changes to evaluation plan How to report negative results, lost data Reporting on multi-site evaluation Privacy issues Report needs involvement of project director
![Page 10: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/10.jpg)
Training Satisfaction Survey Results (partial)
“The training will help me write a high quality Final Evaluation Report”
74% strongly agreed 23% agreed
“The presentations were informative” 71% strongly agreed 24% agreed
“The interactive exercises were helpful.” 71% strongly agreed 23% agreed
![Page 11: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/11.jpg)
Training Satisfaction (continued)
Learning tool most appreciated:
Hands-on exercise / role play of scoring report
Hands-on eval. training
is most useful
(Trevisan 2004;
Patton 2005)
![Page 12: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/12.jpg)
Overall Score Comparison
![Page 13: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/13.jpg)
Pre/Post Score Comparison
Intervention Pre mean score in %
Post mean score in %
Percent point change
None 61.9 63.8 + 1.9
One-day training
65.6 78.3 + 12.7
One-day training and webinar
62.2 70.6 + 8.4
![Page 14: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/14.jpg)
Limitations
Possible scorer effect Limited measure of training methodology
effect Insufficient sample size to measure webinar
effect
![Page 15: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/15.jpg)
Lessons Learned
One-day training events have good potential of success in improving report writing
Hands-on exercises are preferred methods of learning
Training satisfaction surveys provide limited evidence of ECD effect
It may take repeated or longer training to produce better results
Final Evaluation Report scoring tool is a useful measure of capacity building
![Page 16: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/16.jpg)
Next Steps/Conclusions
Study characteristics of those who attended versus those who did not
Develop measures for webinar training Develop measures for various training
methods and activities Develop measures for capacity building in
general
![Page 17: Improving Final Report Writing in California’s Tobacco Control Projects Jeanette Treiber, PhD Robin Kipke, MS UC Davis](https://reader030.vdocument.in/reader030/viewer/2022032600/56649dac5503460f94a9c626/html5/thumbnails/17.jpg)
References
CTPH (California Department of Public Health). 2.2007.Tell Your Story. Guidelines for Preparing a Complete, High Quality Final Evaluation Report. http://www.cdph.ca.gov/programs/tobacco/Documents/CTCPTellYourStory.pdf
Preskill Hallie and Darlene Russ-Eft (2005). Building Evalution Capacity: Thousand Oaks (Sage Publications)
Patton, Michael Quinn. (2005). Teaching Evaluation Using the Case Method. New Directions for Evaluation. Volume 2005, Issue 105, pages 5–14, Spring
Trevisan, Michael S. (2004). Practical Training in Evaluation: A Review of the Literature. American Journal of Evaluation 25;
Torres, Rosalie, Hallie Preskill and Mary Piontek. (1997). Communicating and Reporting Practices and Concerns of Internal and External Evaluators. Evaluation practice, Vol. 18, No. 2, 105-125.