Texas Department of State Health Services Texas Cancer Prevention and Control Program
CDC-RFA-DP17-1701-01CONT18 Texas Cancer Registry
Program 3: National Program of Cancer Registries
Texas Cancer Registry
Program Evaluation Plan
January 31, 2018
Texas Department of State Health Services
Division for Disease Control and Prevention Services
Texas Cancer Registry
P.O. Box 149347, Mail Code 1928
Austin, Texas 78714-9347
Table of Contents Plan Overview ...................................................................................... 1
Intended Use and Evaluation Users ......................................................... 1
Program Description ............................................................................. 2
Overview .......................................................................................... 2
Activities........................................................................................... 2
Priority Populations ............................................................................ 2
Beneficiary Impact from Programmatic Activities ................................... 2
Logic Model ....................................................................................... 2
Evaluation Focus .................................................................................. 4
Plan for Collecting Data ......................................................................... 4
Methods ........................................................................................... 4
Relevant Indicators and Data Sources .................................................. 4
Data Collection Responsibilities ............................................................ 5
Analysis and Interpretation .................................................................... 5
Dissemination and Use of Findings .......................................................... 6
Communication Strategies .................................................................. 6
Formats and Audience ........................................................................ 7
Dissemination Responsibilities ............................................................. 7
Evaluation Timeline ............................................................................... 9
Conclusion ......................................................................................... 10
Appendix ........................................................................................... 11
Texas Cancer Registry Evaluation Plan Page 1 of 13
Plan Overview
As required for DP17-1701 National Program of Cancer Registries
(NPCR) Grantees, the Texas Cancer Registry (TCR) is providing an evaluation and performance measurement plan for grant Years 1–5. The
general approach follows the CDC Framework for Program Evaluation in Public Health as the primary guide for evaluation questions, overall planning,
and implementation.1 Using this validated, well-established CDC framework fosters TCR performance improvement by 1) engaging key stakeholders in
evaluation design, monitoring, and decision making, 2) describing and benchmarking TCR strengths and weaknesses, 3) using multiple varied
credible data sources, data gathering, analyses, and evaluation methods, 4)
justifying conclusions, 5) communicating results, and 6) establishing future goals with an explicit emphasis on program improvement and accountability.
Evaluation questions are focused on the inputs (resources), core registry and specialized project activities, corresponding outputs, outcomes,
and desired impacts described in the NPCR Logic Model.2 More specifically, questions will evaluate the TCR on the NPCR 2017–2022 Program Standards
and Strategies: program collaboration, external partnerships, cancer data and surveillance, community level interventions and patient support, health
systems change, and program monitoring and evaluation (data completeness, timeliness, and quality).
Intended Use and Evaluation Users
The overall purpose of the TCR Program Evaluation is to systematically
and transparently evaluate TCR program inputs, grantee activities, outputs, outcomes, and impacts, as described in the NPCR Logic Model. The
evaluation will be used by stakeholders to improve TCR efficiency, effectiveness, and accountability in carrying out its core and specialized
activities. The subsequent improvement in these activities will in turn result in improved short, intermediate, and long-term outcomes, as well as positive
program impact.
Consistent with guidance provided by the CDC Framework for Program
Evaluation in Public Health, evaluation users with access to results for decision-making include persons and organizations having an investment in
what will be learned from the evaluation, and what will be done with that knowledge. To minimize the risk of stakeholders ignoring, criticizing, or
resisting the evaluation and results, stakeholders will be engaged in various aspects of the evaluation. The three principle evaluation stakeholder groups
include 1) those involved in program operations (e.g., NPCR, TCR and other DSHS staff), 2) those served and effected by program (e.g., researchers,
various types of cancer reporters), and primary evaluation users (e.g., TCR
managers, DSHS leadership, TCR Advisory Committee).
Texas Cancer Registry Evaluation Plan Page 2 of 13
Program Description
Overview—Supporting a comprehensive and coordinated approach to
inform policy, systems, and environmental changes to decrease the cancer
burden remains a leading priority for public health in Texas, and requires
intensive, high quality cancer data and surveillance from a broad variety of
sources. Currently, the TCR represents one of the largest and most
comprehensive high quality population-based cancer data resources and
surveillance systems in the Nation. The TCR collects cancer data for over
28.3 million Texas residents, including significant special populations, and
contributes a large amount of non-resident cancer cases to other state
cancer registries and the national cancer surveillance system.3
Activities— The Texas Cancer Registry (TCR) is a combination active
and passive surveillance system responsible for the collection, maintenance,
and dissemination of high quality population-based cancer data. The TCR
engages in all grantee activities listed in the NPCR Logic Model and Program
Standards. The TCR consists of 54 staff, a central office and two regional
offices, located in Austin, Arlington, and Houston, respectively. Over
260,000 reports of cancer were received in 2017 from over 550 hospitals,
cancer treatment centers, ambulatory surgical centers, and pathology
laboratories located across the state.
Priority Populations— Texas provides a broad array of population
characteristics important to national cancer surveillance and thought to
impact effective cancer prevention, diagnosis, and treatment. It is diverse,
with no ethnic majority. Texas has the second largest Hispanic, third largest
Black, and fourth largest Asian populations. According to the U.S.
Department of Agriculture Economic Research Service, Texas also has the
largest rural population in the U.S. (3,048,574).
Beneficiary Impact from Programmatic Activities—TCR data are
used in all seven CDC data use categories for assisting clinicians,
researchers, policy makers, cancer patients, and others in making better
informed decisions, recommendations, and new discoveries to improve
cancer prevention, diagnosis, treatment, monitoring of clinical conditions,
delivery of care, survival, and quality of life at the patient, health care
system, and population levels. On average, over 80 peer-reviewed journal
articles are published annually using TCR data, and the TCR currently
supports 63 DSHS-IRB approved studies funded for $88,049,520. In 2017,
15,253 queries were completed through the Cancer-Rates Info query tool,
and 388 additional data requests were completed using TCR data.
Logic Model—The TCR will use the NPCR Logic Model as primary
guides for evaluation questions and the general approach to evaluation.
Texas Cancer Registry Evaluation Plan Page 4 of 13
Evaluation Focus
The evaluation purpose, stakeholder interests, the feasibility of answering questions, and in turn, being able to use that new knowledge in
an actionable way for program improvement were all taken into consideration in determining the evaluation focus. As stated previously in
the Plan Overview, all evaluation questions are tied to and based on the inputs (resources), core registry and specialized project activities,
corresponding outputs, outcomes, and desired impacts described in the NPCR Logic Model. Specifically, questions will evaluate the TCR on the NPCR
2017–2022 Program Standards and Strategies. These strategies include program collaboration, external partnerships, cancer data and surveillance
(legislative authority; data collection, content and format; data quality and education; data submission; data use and monitoring; electronic data
exchange), community level interventions and patient support, health
systems change (linkages), and program monitoring and evaluation (data completeness, timeliness, and quality).
Plan for Collecting Data
Methods—Both quantitative and qualitative data collection methods
will be used in conducting the TCR program evaluation, and these methods will align and vary according to the program strategy and grantee activity.
Web Plus, Prep Plus, CRS Plus, and eMaRC Plus, SAS, Tableau, Access, and Excel will serve as the primary software applications used for collecting
quantitative performance measure data used in the evaluation. Qualitative data will be collected through notes and captured using Survey Monkey,
Word, and Excel.
Relevant Indicators and Data Sources—The following relevant,
credible indicators and data sources will be used for the TCR evaluation: • TCR compliance, timeliness, completeness, and data quality reports
(for ongoing evaluation of timeliness, completeness, and data quality) • TCR passive and active case finding and data quality audit reports
• TCR business process maps • TCR data use and dissemination reports
• National quality assurance studies (SEER EOD, TNM Reliability Study) • NPCR data quality audits
• NPCR Hospital, Pathology Laboratory, and Physician Reporting Progress Report (to assess status and improvements in electronic case
capture) • NPCR Program Evaluation Instruments
• NPCR and North American Association of Central Cancer Registries (NAACCR) annual data submission data quality indicator and timeliness
reports (e.g., the NPCR-CSS Data Evaluation Reports, to assess NPCR
standards for data submission)
Texas Cancer Registry Evaluation Plan Page 5 of 13
• NPCR work plan interim and annual progress reports (includes measures for effective collaboration with NCCCP, NBCCEDP, and other
chronic disease programs, such as Tobacco Control) • Special studies presented at international/national meetings (e.g.,
NAACCR Annual Conference, National Cancer Registrar’s Association Annual Conference)
• Articles in the scientific literature (e.g., Journal of Registry Management)
The TCR will also ensure that all essential business functions are included
(see the Appendix, Texas Cancer Registry Work Flow Diagram, that documents these functions).
Data Collection Responsibilities—Data collection responsibilities will
be shared across TCR management. The TCR Epidemiology Group serves as
the lead for developing and maintaining the majority of current compliance, timeliness, completeness, data quality, and data dissemination/use reports.
However, the Operations, e-Reporting, and Quality Assurance Managers, team leads, and staff provide input when changes or additional information
in these reports is needed. Quality Assurance and Registry Operations Managers are primarily responsible for participating in and evaluating the
results for national quality assurance studies (e.g., SEER EOD, TNM Reliability Studies), TCR, and NPCR data quality audits. The Branch Manager
is the lead for the NPCR Program Evaluation Instruments, as well as grant interim and annual progress reports.
Analysis and Interpretation
A critical component for this evaluation plan is ensuring that
conclusions are justified, not only using accurate credible data sources and evidence, but also accurate analysis and interpretation. If appropriate
process, analysis, and interpretation are not used, the evaluation could in fact be harmful, due to inaccurate results and faulty conclusions. The
process for drawing appropriate, data-based conclusion includes engaging the right stakeholders at the right time, using standard high quality credible
data sources and methods of analysis. Analyses will focus on attempting to identify important patterns of evidence (e.g., certain types of reporters with
a high percentage of certain missing information), and isolating important findings (e.g., looking at each criterion used to determine NPCR’s 12 and 24-
month data standards). Analysis and interpretation responsibilities will be shared across TCR
managers. Primary responsibility for analysis and compilation will reside with
the TCR Branch and Epidemiology Managers. Primary interpretation responsibilities will vary according to the strategies and activities being
evaluated, and the stakeholder expertise. For example, Registry Operations,
Texas Cancer Registry Evaluation Plan Page 6 of 13
e-Reporting, and Quality Assurance managers who are Certified Tumor Registrars are the most qualified and appropriate subject matter experts to
review, analyze, and interpret national data quality study results and TCR/NPCR audit findings.
Key external program partners will also participate in the TCR program evaluation analysis and interpretation processes. The Texas Health and
Human Services Commission and Department of State Health Services have quarterly Coordination Meetings that include the Texas Comprehensive
Cancer Control Program (TCCCP) and Breast and Cervical Cancer Early Detection Services Program (BCCS). These meetings provide a regularly
recurring opportunity for the TCR to share analyses and request input on interpretation on evaluation questions and activities important to the
collaborations with their programs. The Advisory Committee to the Texas Cancer Registry (ACTCR), as
well as the Cancer Alliance of Texas (CAT) include a broad group of primarily
external stakeholders with expertise that can assist primarily with multiple aspects of the evaluation, including interpretation. The TCR convenes the
Advisory Committee at least twice annually and communicates with members monthly on items of interest and TCR updates. Stakeholders
represented on ACTCR Members include entities external to the DSHS such as the American Cancer Society, Texas Medical Association, Texas Hospital
Association, and Texas Tumor Registrar’s Association, and academic research institutions. They also include entities within the DSHS such as the
Health Promotion and Chronic Disease Prevention Section which administers the Comprehensive Cancer Control Program and the Texas Health and
Human Services Commission’s Breast and Cervical Cancer Services program. The full membership listing can be found at
https://www.dshs.texas.gov/tcr/actcr-members.aspx.
Dissemination and Use of Findings
To best ensure the use of TCR Program Evaluation findings and share
lessons learned, evaluation results must be shared using effective
communication strategies with the appropriate audience and using a variety
of formats. Responsibilities for dissemination and use of findings must be
clearly defined, and audience feedback considered.
Communication Strategies—Disseminating information on the
program evaluation plans, progress, preliminary, and final results to
stakeholders will use tailored communication strategies that will most
appropriately meet their particular needs. All TCR communication must be
transparent and accurate to foster trust in the evaluation process,
conclusions, and recommendations for improvement.
Texas Cancer Registry Evaluation Plan Page 7 of 13
Formats and Audience—The TCR will rely primarily on written
formats for evaluation plan dissemination and will tailor the formats and
content for the different stakeholder audience. The TCR will engage in
dissemination activities, including but not limited to:
• Emails to TCR staff, DSHS leadership, TCCCP and BCCS collaborators,
cancer reporters, data requestors, the Advisory Committee to the
Texas Cancer Registry, and the Cancer Alliance of Texas
• Updates to TCR staff and DSHS leadership at various regularly
occurring staff meetings (recurrence ranges from bi-weekly to bi-
monthly)
• TCR newsletter articles primarily targeted for cancer reporters
• Web page updates primarily targeted for TCR staff, cancer reporters,
and data requestors
• Internal Share Point updates primarily targeted for internal DSHS and
TCR staff/leadership
• Discussion and presentations at the quarterly Collaboration meetings
with the Texas TCCCP and BCCS programs
• Discussion and presentations at Advisory Committee to the Texas
Cancer Registry and Cancer Alliance of Texas meetings
• Progress reports to NPCR staff
Dissemination Responsibilities—As with data analysis and
interpretation, dissemination responsibilities will be shared across TCR
managers. Primary responsibility will reside with the TCR Branch and
Epidemiology Managers. Responsibilities will be shared according to the
intended audience, format, and internal TCR manager/staff/stakeholder
expertise and responsibilities within the TCR. For example, web updates are
mainly the responsibility of the Epidemiology group, but national data quality
study-related activities included in the evaluation plan require CTR expertise
from the Operations, e-Reporting, and/or Quality Assurance areas.
Audience Feedback and Action Steps—In order to maximize the
effectiveness of this program evaluation, mechanisms must be in place for monitoring and documenting audience feedback and following-up with any
resulting action steps. Feedback is the communication that occurs among all stakeholders engaged in the evaluation process. CDC program evaluation
guidance stresses that giving and receiving feedback creates an atmosphere of trust among stakeholders and keeps an evaluation on track by letting
those involved stay informed regarding how the evaluation is proceeding.4 TCR management will ensure that all primary program evaluation
participants, users, and other stakeholders have the right and ability to comment on decisions that impact actions steps and might affect the
likelihood of obtaining useful information. Stakeholder feedback is an
Texas Cancer Registry Evaluation Plan Page 8 of 13
integral part of evaluation, particularly for ensuring use. Obtaining feedback will be encouraged by keeping stakeholders regularly updated using the
variety of formats and mechanisms described in the Formats and Audience section, holding periodic discussions during each step of the
evaluation process, and routinely sharing interim findings, provisional interpretations, and draft reports. Feedback documentation and action steps
for the TCR program evaluation will be documented in specific discussion/meeting minutes, emails, and notes, saved in a specific TCR
Program Evaluation shared folder accessible to TCR management who have the primary responsibility in carrying out the evaluation.
Texas Cancer Registry Evaluation Plan Page 9 of 13
Evaluation Timeline
The TCR evaluation and performance measurement plan for grant
Years 1–5 will follow the recommended timeline provided by the NPCR.
Texas Cancer Registry Evaluation Plan Page 10 of 13
Conclusion
In conclusion, the TCR Program Evaluation seeks to systematically and
transparently evaluate TCR program inputs, grantee activities, outputs,
outcomes, and impacts, as described in the NPCR Logic Model. The
evaluation will be used by stakeholders to improve TCR efficiency,
effectiveness, and accountability in carrying out its core and specialized
activities. The subsequent improvement in these activities will in turn result
in improved short, intermediate, and long-term outcomes, as well as positive
program impact.
Cited References
1 Centers for Disease Control and Prevention. (1999, September).
Framework for program evaluation in public health. Morbidity and Mortality
Weekly Report, 48(RR-11). Available from:
ftp://ftp.cdc.gov/pub/Publications/mmwr/rr/rr4811.pdf
2 NPCR Evaluation Plan Guide For DP17-1701 NPCR Grantees Provided by
the CDC Division of Cancer Prevention and Control Cancer Surveillance
Branch
3 Quick Facts United States. U.S. Census Bureau. Internet:
https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xht
ml?pid=PEP_2017_PEPANNRES&src=pt . Accessed 1/29/2018.
4 Developing an effective evaluation plan. Centers for Disease Control and
Prevention, National Center for Chronic Disease Prevention and Health
Promotion, Office on Smoking and Health; Division of Nutrition, Physical
Activity, and Obesity, 2011. Available from:
http://www.cdc.gov/obesity/downloads/cdc-evaluation-workbook-508.pdf/