consequences of evaluation: research program improvement & decision-making dale pahl* and hugh...

Consequences of Evaluation:Consequences of Evaluation:

Research Program Improvement & Decision-makingResearch Program Improvement & Decision-making

Dale Pahl* and Hugh TilsonDale Pahl* and Hugh TilsonOffice of Research and DevelopmentOffice of Research and Development

U.S. Environmental Protection AgencyU.S. Environmental Protection Agency

November, 2006 November, 2006

American Evaluation Association 2006 Evaluation Conference: The Consequences of Evaluation

Research, Technology, and Development Workgroup Session 813: Improving the Utilization of Evaluation for Research Policy Decision-Makers

*contact address [email protected]

Pahl & Tilson Nov 4th, ’06 2006 Conference of the American Evaluation Association

AcknowledgementAcknowledgement

• Publicly-available documents that communicate details Publicly-available documents that communicate details about theabout the research and concepts in this presentation research and concepts in this presentation are accessible through EPA websites:are accessible through EPA websites:

Program design & management www.epa.gov/osp/HH%20MYP%20Final.pdf

Program evaluation www.epa.gov/osp/bosc/reports.htm Formal decisions about program improvement www.epa.gov/osp/bosc/reports.htm Communication about the results of human www.epa.gov/ord beginning in January

2007health research

Colleagues who have contributed to information Colleagues who have contributed to information described in this presentation:described in this presentation:

see slides 30 - 32see slides 30 - 32 2


Presentation FocusPresentation Focus

Developing a systematic process for research

managers & policy decision-makers that integrates information about

•Program design & management•Evaluation feedback•Program improvement, and •Communicating about research results

3


Why is this important?Why is this important?

• The Office of the President and Congress require federal executives, policy-makers & research managers to communicate about

How programs are aligned with the• Government Performance and Results Act • Research & Development Investment Criteria, • Research & Development PART, and • Recommendations from independent expert panels

How evaluation results are used for program improvement and decision-making

When research knowledge and applications help achieve When research knowledge and applications help achieve outcomesoutcomes

4


OverviewOverviewA Systematic Process with Annual & Periodic A Systematic Process with Annual & Periodic

ElementsElements

Using Evaluation Results Using Evaluation Results for Program for Program

Improvement Improvement

Communication about Communication about Program ResultsProgram Results

Program Program Evaluation Evaluation

Communication with targeted audiences

Formal Decisions about Formal Decisions about Program ImprovementProgram Improvement

Program Design & Program Design & ManagementManagement

5


The EPA Research ContextThe EPA Research Context

EPA’s Office of Research and Development

Multi-disciplinary, integrated research relevant to Multi-disciplinary, integrated research relevant to EPA’s mission of protecting human and EPA’s mission of protecting human and environmental healthenvironmental health• Rigorously peer reviewed at publication-, project-, and program-

levels; accepted as state-of-the-art

• Research informs risk assessment, EPA policy decisions, & regulations

Combination of in-house research and an external Combination of in-house research and an external grants program to provide credible, relevant, and grants program to provide credible, relevant, and timely science and research applicationstimely science and research applications

Periodic assessment of research progress to Periodic assessment of research progress to ensure that sound science is used in EPA decisionsensure that sound science is used in EPA decisions

6


Pilot Study:Pilot Study:

EPA’s Human Health Research ProgramEPA’s Human Health Research Program






Formal Decisions about Formal Decisions about Program ImprovementProgram Improvement

Program Design Program Design & Management& Management Slides 8 - 16

7


Program Design & Management for Human Health Program Design & Management for Human Health ResearchResearch

Human health research Human health research reduces uncertaintyreduces uncertainty11

• In critical links across the exposure-to-effect paradigm• For health risk assessment & regulatory decision-making

Human health research knowledge & risk assessment are Human health research knowledge & risk assessment are applied in policy decision-applied in policy decision-makingmaking• Deciding whether or not to take regulatory action,

(Is there an environmental hazard?)

• Deciding what actions are most effective, (What actions do we take to protect human health?)

• Evaluating the ultimate effectiveness of our actions (Were we effective?)

AmbientConditions

Fate andTransport

SourceEmissions

ExposureAnd Dose

Early Signsof Effects

HealthImpacts

81 1 For thoughtful discussion about the artificial distinction between basic and applied research, see Donald Stokes, For thoughtful discussion about the artificial distinction between basic and applied research, see Donald Stokes, Pasteur’s Quadrant: Basic Research and Technological InnovationPasteur’s Quadrant: Basic Research and Technological Innovation . An excellent review of these concepts is presented in the 2005 AEA . An excellent review of these concepts is presented in the 2005 AEA presentation by presentation by Bhavya Lal and Lisa Messeri, “Operating in Pasteur’s Quadrant: Use Inspired Basic Research”Bhavya Lal and Lisa Messeri, “Operating in Pasteur’s Quadrant: Use Inspired Basic Research”


Program Theory Integrates Human Health Program Theory Integrates Human Health Research, Research,

Risk Assessment, & Decision- Risk Assessment, & Decision-

makingmaking Adapted from Risk Assessment in the Federal Government: Managing the Process (NRC, 1983); 1997 Update to ORD’s

Strategic Plan (EPA, 1997); and OIG-ORD Presentation to EPA’s Deputy Administrator (Pahl & Norland, March 2002)

Dale Pahl Nov ‘02

EXPOSURE RISK ASSESSMENT

RISKMANAGEMENT

Accountability Developing and measuring appropriate environmental indicators demonstrates whether environmental decisions result in improved human and environmental health.

PollutionSources

Emissions

Transport &Transformation

Environmental Concentrations

Exposures Exposure Assessment

RiskCharacterization

Legal Considerations

Ecosystem and Human HealthConsiderations

Social, Economic,& PoliticalFactors

Exposure-Dose Relationships

Dose-ResponseAssessment

RiskManagementOptions

Implementation Decisions Managers make decisions about how to implement, comply with & enforce regulations or how to remedy environmental problems.

Internal Dose

Biological Effect

Adverse Health Effect

Health Assessment

Hazard Identification Environmental Decisions to Protect

Human Health

15


Program Design & Management for Human Health Program Design & Management for Human Health Research:Research: Long-Term Goals & Research Long-Term Goals & Research

Questions Help Organize the ProgramQuestions Help Organize the Program

Long-Term Goal 1Use of Mechanistic Data to

Reduce Uncertainty in Risk Assessment

Long-Term Goal 2Tools to Characterize Cumulative Risk

Long-Term Goal 3Protect Susceptible Subpopulations

Long-Term Goal 4Approaches to Assess

Risk Management Decisions

Risk assessors/managers use ORD’s methods, models or data to reduce uncertainty in risk assessment using mechanistic (or mode of action) information

Risk assessors/mangers use ORD’s methods, models or data to characterize aggregate and cumulative risk in order to manage risks of humans exposed to multiple environmental stressors

Risk assessors/mangers use ORD’s methods, models or data to characterize and provide adequate protection for susceptible subpopulations

Risk assessors/mangers use ORD’s methods, models or data to evaluate the effectiveness of risk management decisions

9


Legislative and Legislative and CongressionalCongressional• Clean Air Act• Food Quality Protection Act• Safe Drinking Water Act • Children’s Health Act

External Scientific AdvisorsExternal Scientific Advisors • National Research Council

Office of the PresidentOffice of the President• Presidential Commission on Risk

Assessment/Management

Policy & decision-makersPolicy & decision-makers• EPA Program and Regional Offices

Drivers for Long-term Goals & Research Drivers for Long-term Goals & Research QuestionsQuestions

10


What methods and models are needed to identify modes or What methods and models are needed to identify modes or mechanisms of action that can be used for risk assessment? mechanisms of action that can be used for risk assessment?

How can knowledge of toxicity pathways inform the development of How can knowledge of toxicity pathways inform the development of PK and PD models for risk assessment? PK and PD models for risk assessment?

How can knowledge of toxicity pathways (or mode of action) be used How can knowledge of toxicity pathways (or mode of action) be used to reduce uncertainty in extrapolation in risk assessment, includingto reduce uncertainty in extrapolation in risk assessment, including

• Extrapolation from high-to-low dose • Extrapolation from animals to humans• Extrapolation from in vitro data to in vivo exposures• Harmonization of cancer and non-cancer risk assessments

Long-term Goal 1: Risk assessors and risk managers use ORD’s methods and models to reduce uncertainty in risk assessment, using mechanistic (or mode of action) information.

Key Human Health Research Questions

Long-term Goals & Research QuestionsLong-term Goals & Research Questions

11


How can biomarkers be used in cumulative risk assessment?How can biomarkers be used in cumulative risk assessment?• What tools are needed to identify biomarkers for cumulative risk assessment?• How can those biomarkers be applied for cumulative risk assessment?

What source-to-dose models are needed for cumulative risk?What source-to-dose models are needed for cumulative risk?• What methods and models are available for assessing cumulative risk?

How can tools be used to conduct cumulative risk How can tools be used to conduct cumulative risk assessments on chemical mixtures? assessments on chemical mixtures?

How can cumulative risk at the community level be How can cumulative risk at the community level be evaluated?evaluated?• What tools are necessary for community-based risk assessments?• How can those tools be applied for community-based risk assessments?

Long-term Goal 2: Risk assessors and risk managers use ORD’s methods, models, and data to characterize aggregate and cumulative risk in order to manage risk of humans exposed to multiple environmental stressors.



12


Is there differential life-stage responsiveness or exposure to Is there differential life-stage responsiveness or exposure to environmental agentsenvironmental agents• What are the long-term effects of developmental exposure to chemicals?• How does aging affect responsiveness to environmental chemicals?• How can we model exposure and effects to protect susceptible subpopulations?

Which methods and models are appropriate for longitudinal Which methods and models are appropriate for longitudinal research with children? research with children?

What are predisposing factors for asthma as a function of life What are predisposing factors for asthma as a function of life

stage and how do they interact with indoor air environments?stage and how do they interact with indoor air environments?

Long-term Goal 3: Risk assessors and risk managers use ORD’s methods, models, and data to characterize and provide adequate protection for susceptible subpopulations.



13


What are the trends in health status in the US?What are the trends in health status in the US?

What tools are available to determine the impact of regulatory What tools are available to determine the impact of regulatory decisions on exposures to environmental stressors that lead decisions on exposures to environmental stressors that lead to adverse health outcomes?to adverse health outcomes?

Long-term Goal 4. Risk assessors and risk managers use ORD’s methods and models to evaluate the effectiveness of risk management decisions.



14


Program Design & Management for Human Health Program Design & Management for Human Health Research Research Source: Human Health Research Multi-Year Plan (May 15, 2006) page 13. U.S. Environmental Protection Agency

Research Program

ResearchOutputs

Outreach & Transfer

Clients IntermediateOutcomes

Long -TermOutcomes

Short -TermOutcomes

Short -Term Outcomes

Externalities that May Impact the HHR Program

Congressional appropriations and administration budget decisions, changes in EPA strategic goals, science priorities, or regulatory requirements, availability of investment capital, court decisions & consent agreements

Research Topics &Activities

Inputs and Resources

•Strategic guidance

•Research Coordination Teams:client needs, research questions, research priorities, outcomes, multi -year plan

•Partnerships

•ORD Intramural Research Resources

•ORD Extramural (STAR) Research Resources

•Program -level funding & FTE

ORD conducts research to develop methods, models and data to:

•Develop approaches to characterize dose -response extrapolation for risk assessment

•Develop approaches to characterize extrapolation from single to multiple pathways for risk assessment

•Develop approaches to characterize populations at risk for risk assessment

•Develop approaches to evaluate effectiveness of risk management decisions

ORD research provides:

•Characterization of exposure/dose -response extrapolation for risk assessment

•Characterization of extrapolation from single to multiple chemical/pathways of exposure for risk assessment

•Characterization of intraspeciesextrapolation for risk assessment

•Tools to evaluate effectiveness of risk management decisions

•Scientific leadership in disciplines related to high priority environmental problems

•Scientific conferences and workshops

•Peer -reviewed publications

•Technical and decision support

•Health advisories

•EPA (OPPTS, OW, OCHP, OEI, OAR, Regions

•Risk assessors (IRIS, RAF)

•Other Federal agencies (FDA, ATSDR, NIH, CDC)

•States

•Tribes

•International health organizations (WHO, IPCS)

•Regulated community

•Academic community

•Targeted communities

•American Public

Outcomes

Independent experts, risk assessors, or others in the field:

•Use ORD’smethods, models and data to characterize exposure/ dose -response extrapolation for risk assessment:

•Use ORD’smethods and models to characterize extrapolation from single to multiple chemical/pathways for risk assessment

•Use ORD’smethods and models to characterize populations at risk for risk assessment

• Use ORD’smethods and models to characterizeeffectiveness of risk management decisions

•Environmental stressors that pose an unreasonable risk are identified

•Exposure to environmental stressors by multi -pathways is reduced

•Exposure of populations at risk is reduced

•Effectiveness of risk management decisions is improved

•Risk to humans from environmental stressors is reduced or prevented

Intermediate Outcomes

Long -Term Outcomes

Research Outputs & Reach

Program Design & Research QuestionsProgram Implementation &

Management

16



Using Using Evaluation Evaluation Results forResults for

Program Program

ImprovementImprovement

Slides 18-22


Program Program EvaluationEvaluation


Formal Decisions about Program Formal Decisions about Program ImprovementImprovement


&

17


Evaluation of Human Health Research Evaluation of Human Health Research

ProgramProgram Independent Expert Independent Expert ReviewReview

The National Academies, OMB, and OSTP recommend The National Academies, OMB, and OSTP recommend independent expert review as ‘best practice’ for independent expert review as ‘best practice’ for federal research programsfederal research programs

OMB/OSTP evaluation criteria: OMB/OSTP evaluation criteria: • Relevance, quality, performance, • An additional criterion—leadership—is recommended• OMB guidance describes alignment between R & D Criteria, PART, and

program-level review by independent expert panels

ORD charters independent expert panels (Board of ORD charters independent expert panels (Board of Scientific Counselors) under FACA Scientific Counselors) under FACA • EPA Deputy Administrator recommends charter to OMB in 1994• OMB approval—1995 • FACA charter updated in 2004• FACA procedures include extensive opportunities for stakeholder

participation; safeguards against COI; consistent, open, and transparent process; participation by designated federal official (DFO); consultation with FACA attorney

18


Evaluation of Human Health Research ProgramEvaluation of Human Health Research Program BOSC Subcommittee for Human Health ResearchBOSC Subcommittee for Human Health Research

Subcommittee review: February-May 2005Subcommittee review: February-May 2005 Review procedures coordinated with DFO:Review procedures coordinated with DFO:

• Subcommittee agrees on charge statement and extensive documents / evidence about the program

• Subcommittee holds 2 conference calls to organize review of documents & evidence

• Subcommittee holds 2 ½ day face-to-face meeting Program design overview (ORD presentations) Research poster presentations (ORD & STAR presentations) Research applications at key decision-points (Program Office, Regional Office, and extramural

presentations) Extensive Subcommittee review, questions, and discussion Subcommittee report-out at end of meeting

• Subcommittee prepares draft report for Executive Committee review• Executive Committee reviews, approves, & transmits final report to EPA in

May 2005

BOSC report available at www.epa.gov/osp/bosc/reports.htmBOSC report available at www.epa.gov/osp/bosc/reports.htm

19


Evaluation of Human Health Research Evaluation of Human Health Research Program Program BOSC Report on Program Relevance, Quality, BOSC Report on Program Relevance, Quality,

Performance, LeadershipPerformance, Leadership

• Overall program-level comments

Has made significant progress toward each of its long-term goals

Research has informed risk assessment Achieved the goal of reducing uncertainty Decisive propensity to encourage mining of data to inform

risk assessment of stakeholders Research has allowed EPA to be a leader in conducting

credible, nationally and internationally accepted risk assessments on chemicals of environmental concern

• Additional comments focus on program’s long-term goals 20


Using Evaluation Results for Program Using Evaluation Results for Program ImprovementImprovement

Assessing research contributions to outcomes: Assessing research contributions to outcomes: guidance & policy decision-makingguidance & policy decision-making

Revising Multi-Year Research Plan (MYP):Revising Multi-Year Research Plan (MYP):• Extensive interaction with Research Coordination Team

Membership from Regional Offices, Program Offices, senior scientists and research managers from research labs/centers

Preparing for the PART:Preparing for the PART:• BOSC results help address several questions related to relevance,

quality, performance, and scientific leadership• BOSC results help respond to PART Sections 2.6 and 4.5—which

focus specifically on independent expert evaluation

21


Thinking about Alignment between GPRA, the R & D Thinking about Alignment between GPRA, the R & D Investment Criteria, and PART for Research Programs Investment Criteria, and PART for Research Programs 11

Improved Knowledge… (Quality of

the Science)

Performance 2

… and its Application

(Responsiveness to Agency Decisions)

Research

(activities, topics, questions)

Approach to a specific environmental problemResources, partners, & collaborationMechanisms & extent to which findings and products are disseminated to scientific audiences in a timely manner Mechanisms & extent to which findings and research contributions to outcomes are communicated to clients in a timely manner

Extent to which progress (across each topic, based on clearly stated performance measures) is being made for each LTGExtent to which progress is being made by an activity/topic that is representative of the LTG and program

Responsiveness to agency needs, decisions, strategic goalsCoordination with federal partners (research topics)•Mechanisms & extent to which findings and research contributions to outcomes are communicated to and used by clients in a timely fashionExtent to which clients communicate about research contributions

Outcomes (Research topics,

questions, LTGs)

Appropriateness of the research questions for each LTG (prospective and retrospective)

Extent to which measures and data for each LTG demonstrate improved knowledge and either utility or uncertainty reduction (retrospective)

Extent to which research for each LTG contributes to EPA strategic goals, outcomes, and decisions related to utility or to uncertainty reduction (prospective and retrospective)

Leadership (PIs, activities,

questions, LTGs)

Leadership role in the scientific community (influences on agendas, decisions, priorities of other researchers, national & international organizations)

Extent to which measures and data for each LTG demonstrate leadershipExtent to which bibliometric analysis (every 4-5 years) demonstrates scientific leadership and impact

Leadership role in the Agency and with federal partners (influence on strategic goals, research priorities, CENR objectives and topics, agency decisions, and federal accountability)

1 Charge questions for program review focus on program-level relevance, quality, leadership, performance, structure, coordination & communication, and outcomes2 Information and data about performance help to align narrative findings and adjective ratings in the report by the independent expert panel with evidence for PART question 4.1. Performance measures respond to definitions and guidance from GPRA and PART. 22

Examples of traditional focus for independent expert panels Extending the focus to include program-level performance








Formal Decisions about Program ImprovementFormal Decisions about Program Improvement

Slides 24-25


23


Decisions about Program ImprovementDecisions about Program Improvement

Formal EPA Decision & Response to the BOSCFormal EPA Decision & Response to the BOSC

1. Letter from EPA (senior research executive) to BOSC

2. Narrative response to Subcommittee report

3. Table of action items to improve program relevance, quality, performance, leadership

• Table forms measurable basis for mid-cycle review

Continued on next page . . .

24


Decisions about Program ImprovementDecisions about Program Improvement continued from previous page

Formal EPA Decision & Response to the BOSC Formal EPA Decision & Response to the BOSC

4. Proposes process and timing for mid-cycle review (~ 2 years) and next program review (~4 years) Mid-cycle review—assessment of EPA action items and progress to

improve program relevance, quality, performance, and leadership

5. Information reported also to EPA GPRA—evaluation findings, changes to performance goals/measures

6. Information reported also to GSA

7. EPA letter to BOSC available at www.epa.gov/osp/bosc/reports.htm

25


EPA’s Human Health Research EPA’s Human Health Research ProgramProgram



Communication Communication about Program about Program

Results Results Slides 27-29Slides 27-29



Formal Decisions about Formal Decisions about Program Improvement Program Improvement


26


Communicating about Program Communicating about Program Results Results

1.1. Communication informed by evaluation Communication informed by evaluation recommendationsrecommendations

2.2. Extensive interaction with scientists & Extensive interaction with scientists & managers in regulatory programs and managers in regulatory programs and regional offices about regional offices about

New research knowledge & applicationsNew research knowledge & applications Tools for risk assessment Tools for risk assessment Use of research for guidance & decision-making Use of research for guidance & decision-making

CContinued on nexton next page . . .27


Communicating about Program Communicating about Program Results Results

Continued from previous page . . .Continued from previous page . . .

3.3. Communication via internet site facilitates Communication via internet site facilitates public and client access topublic and client access to

• Reports about research program design & managementReports about research program design & management• Evaluation reports Evaluation reports • Fact sheets and articles aboutFact sheets and articles about

New research knowledge & applications New research knowledge & applications Use of research in guidance & decision-making Use of research in guidance & decision-making Significant research contributions that lead to public and Significant research contributions that lead to public and

environmental healthenvironmental health

4.4. Communicating results of human health Communicating results of human health research:research:

www.epa.gov/ord beginning ~ Jan/Feb www.epa.gov/ord beginning ~ Jan/Feb 20072007 28


Summary of Key PointsSummary of Key Points

Benefits of Systematic Process:Benefits of Systematic Process:

1. Helps align GPRA, R & D Criteria, R & D PART

2. Best practices for research—independent expert panels

3. Evaluation recommendations & decisions for program improvement

4. Meaningful engagement of program scientists, managers, & clients

5. Improves communication about research contributions to outcomes

6. Public access to documents about design, evaluation, & outcomes

29


AcknowledgementsAcknowledgements

EPA Office of Research & Development—Contributors to the Ideas in this Presentation

Jerry Blancato Ann Brown Howard Cantor Mimi Dannel Alva Daniels Kathy Driver William Farland Jon Herrmann Virginia Houk John Jones Lek Kadelli Lori Kowalski Mike Moore Jeff Morris Jack Puzak Larry Reiter Donna Roa Jennifer Robbins Kevin Teichman

AEA – Contributors to Evaluation Best Practices for Research Programs

Claude Bennett Huey-Tsyh Chen Steve Montague Gretchen Jordan Emma Norland Cheryl Oros M.Q. Patton Rosalie Ruegg George Teather

30


AcknowledgementsAcknowledgements Contributors to the Human Health Research Multi-Year Research Plan

EPA Regulatory Programs and Regional Offices

EPA Office of Research and Development

Valerie Blank (OW), Jeff Evans (OPPTS), Lee Hofmann (OWSER),Michael Firestone (OCHP), Scott Jenkins (OAR), Amal Mahfouz (OW),Dennis Pagano (OAR), Nicole Paquette (OEI), Randy Pefetti (OPPTS),Ravi Rao (R4), Rita Schoeny (OW), Hiba Shurairy (OGWDW)

Stan Barone Jr. (NCEA), Larry Cupitt (NERL), Alva Daniels (NRMRL),Kathleen Deener (NCER), Nigel Fields (NCER), Jim Jetter (NRMRL),Andrew Geller (NHEERL), Zhizhi Guo (NRMRL), Ross Highsmith (NERL)Julian Preston (NHEERL), Bruce Rodan (NCEA), Chris Saint (NCER),Hugh Tilson (ORD), Hal Zenick (NHEERL)

31


AcknowledgementsAcknowledgements Contributors to Research Tracks for the Human Health Research Multi-Year

Plan

Research LTG Track Team Lead(s) Contributing Team Members

1Methods and Models to Characterize MOA

Robert Devlin (NHEERL) C. Corton, S. Nesnow, W. Mundy, S. Hunter, I. Gilmour, G. Klinefelter, R. Ramabhadran (NHEERL)

1 Linkages Between PK and PD Models Hugh Barton (NCCT) M DeVito, M Evans, E Kenyon (NHEERL); C Thompson, K Guyton, W Chiu (NCEA); M Tornero (NERL)

1 MOA Information to Address Extrapolation in Risk Assessment

Oxidative Stress TeamReeder Sams (NCEA) and Gary Hatch (NHEERL)Conazole Team: Stephen Nesnow (NHEERL)Arsenic Team: David Thomas (NHEERL)Neuroendocrine Team:Ralph Cooper (NHEERL)Brominated Disinfection By-Products Team:Rex Pegram (NHEERL)

A DeAngelo, D Delker, R Devlin, J Gallagher, S Hester, K Kitchin, P Kodavanti, U Kodavanti, R Macphail, J Ross, S Nesnow, J Royland, B Veronesi, W Winnik (NHEERL); A Jarebek, A Rooney (NCEA) J Allen, LClaxton, D Delker, S Hester, L King, M Narotsky, J Ross, S Thai, W Ward, D Wolf (NHEERL); H Barton, D Dix, A Richard (NCCT); D Ekman (NERL); S Barone (NCEA); D Juberg (Dow-Elanco-US Triazole Task Force); V Dellarco (OPP) D Wolf, S Nesnow, K Kitchin, D Demarini, A Kligerman, S Thai, D Delker, B Adair, M Hughes E Kenyon (NHEERL); J Creed, P Creed, H Ozkaynak (NERL); R Sams (NCEA); R Connelly (NCCT) E Gray, T Stoker, J Goldman, S Laws, K Crofton, M Devito, M Gilbert (NHEERL)S Hunter, M Narotsky, T DeAngelo (NHEERL)

2 Biomarkers for Cumulative Risk Joachim Pleil (NERL) M Morgan (NERL); M Dellarco (NCEA); C Saint (NCER); R Devlin, K Crofton (NHEERL)

2 Source-to-Dose Models for Cumulative Risk

Valerie Zartarian (NERL) Z Guo (NRMRL); M Tornero, F Power, C Dary, M Okino, D Vallero (NERL); C Saint (NCER); H Ozkaynak (ORD)

2 Application of Tools for Cumulative Risk of Chemical Mixtures

Jane Ellen Simmons and Stephanie Padilla (NHEERL)

D Herr, G Moser, H El-Masri, I Gilmour, JE Simmons, K Crofton, M Madden, M Devito, P Bushnell, P Kodavanti, S Hunter, S Padilla, T Shafer, W Boyes (NHEERL); W Setzer, J Blancato (NCCT); M Tornero-Velez (NERL); R Hertzberg (NCEA)

2 Tools for Assessing Community Risk Stephen Graham (NERL) K Thomas (NERL); V Benignus (NHEERL)

3 Research on Life-Stage Long-Term Effects:John Rogers (NHEERL)Effects and Exposure in Children:Nicole Tulve(NERL)Aging:A Geller, L Birnbaum, R MacPhail (NHEERL)

C Lau, N Chernoff, S Fenton, M Rosen, E Massaro, B Abbott, J Andrews, M Gilbert, D Schreinemachers (NHEERL); S Barone (NCEA) S McMaster, R Daniels, G Moser (NHEERL); R Fortmann, K Thomas, D Stout (NERL), C Saint, (NCER); J Moya, S Makris (NCEA); M Firestone (OCHP); J Evans (OPP); E Hubal (NCCT); Z Guo (NRMRL) K Thomas (NERL); A Geller, M DeVito, J Royland, C Corton, C Gordon, V Benignus, L Birnbaum (NHEERL); B Glenn, J Moya (NCER); B Sonawane (NCEA)

3 Methods for Longitudinal Research Pauline Mendola (NHEERL) J Gallagher and S Fenton (NHEERL); J Quackenboss (NERL); B Sonawane and R Brown (NCEA); L Blackburn (OCHP); N Fields (NCER)

3 Research on Asthma Hillel Koren (NHEERL) I Gilmour, L Neas, S Gavett, M Ward, R Devlin (NHEERL); S Vesper, R Williams (NERL); J Jetter, M Menetrez, T Dean (NRMRL); C Saint, T Katz (NCER)

4 Approaches to Evaluate Risk Management Decisions

Rebecca Calderon(NEERL)

H Ozkaynak, V Zartarian, P Egegy (NERL); D Lobdell, D Schreinemachers (NHEERL); D Petersen (NRMRL), P Murphy (NCEA); C Saint (NCER); M Smuts (OAQPS)

4 Report on the Environment Danelle Lobdell (NHEERL) D Shaw, P Murphy (NCEA)

32

consequences of evaluation: research program improvement & decision-making dale pahl* and hugh...

Documents

research results

evaluation results

evaluation conference

pdfprogram evaluation

utilization of evaluation

integrated research

health research colleagues

hugh tilsonoffice of