Overview of OPASI’sOverview of OPASI’sDivision of Evaluation Division of Evaluation

andandSystemic Assessments Systemic Assessments

(DESA)(DESA)Deborah Guadalupe Duran, Ph.D.Deborah Guadalupe Duran, Ph.D.

Chief, Systemic Assessments BranchChief, Systemic Assessments Branch

DESA: Division of Evaluation and Systemic Assessments • Evaluation Feedback • System Assessments (Sci of Sci)• Evaluability• Organization Performance

DRDA: Division of Resource Development and Analysis • Knowledge Management• Portfolio Analysis• Public Health Need• Centralized tools

DSC: Division of Strategic Coordination• Strategic Coordination-High Risk-High Impact• Trans-NIH Initiatives

gaps-potentials

data

-evi

denc

e

strengths-improvements

DRDADRDA

DSCDSC

DESADESA

OPASI Overview

Two branches

SystemicAssessments

Branch(SAB)

EvaluationBranch

(EB)

DESA Structure

(DESA)

Coordinate NIH wide evaluations and assessments

Provide technical assistance for required performance reporting

Facilitate the development of adaptive measures for science

Provide findings, integrated with portfolio analyses that foster scientific planning

SAB EB

DESA Mission

Evaluation Branch

Evaluate NIH-wide programs, key initiatives, and strategic objectives (formative and summative)

Track outcomes/recommendations

Maximize the effective use of the NIH Evaluation Set-Aside Program SAB EB

DESA - EB Mission

Systemic Assessments Branch Coordinate program performance activities across NIH

Comply and respond to federally-mandated performance reporting mechanisms• GPRA• PART• PAR• Scorecard

Foster the development of measures for • Adaptive science assessments• Organization assessments

Facilitate science of science activities

SAB EB

DESA – SAB Mission

• Strategic Plans System Level- HHS/IC/Office/Program

• Science of Science Management• Pre-evaluation Assessments• Visual Analytic Tools• e-Monitoring Systems

• Strategic Plans Project Level• Independent Evaluations• Evaluation Set-Aside Funds

• GPRA• PART• PAR• Audit• Biennial Report• Scorecard

Coordinating-Planning-Managing-Tracking-Reporting

SystemPerformance

& Assessments

IndependentExternal

EvaluationsRelated Processes

Resources

StrategicPlanning

& Evaluability

SAB SABEB

DESA Activities

Evaluation Branch Manage Evaluation Set-Aside Program which provides

dedicated funding for ICs and OD offices

Provide hands-on technical support and online guidance

Consult with ICs and OD offices on evaluation planning

Coordinate two trans-NIH committees to evaluate proposals

Report evaluation results SAB EB

DESA – EB Role

Systemic Assessments Branch Monitor NIH performance (not evaluation)

Report program performance information in lay language

Ensure Budget/Performance Integration

Participate in OMB / Secretary / HHS / Federal workgroups

Liaison to OMB, HHS, ICs performance issues

Develop System Assessments

Conduct internal assessments (evaluability)

Coordinate Science of Science activities

Strategic Planning

SAB EB

DESA – SAB Role

Systemic Assessments Branch: • Evaluability Assessments (internal)• Science of Science (internal)• Monitoring & Tracking Performance• Organization Performance Reporting

Evaluation Branch:• Funding for independent evaluations (external)

SystemicAssessments

BranchEvaluation

Branch

SAB/EB--related (ie IC Planning & Evaluation Community support) but unique functions

DESA Functions

DESA (SAB and EB) offer comprehensive evaluations/assessments – Internal and External

SAB–internal & organization assessments• Evaluability Assessments• Monitoring & Tracking Performance• System Assessments (Science of Science)• Adaptive evaluations for science

EB–external • Independent Evaluations of projects/programs/key initiatives

Internal(SAB)

External(EB)

Evaluations/Assessments

EvaluationExternal- best conducted by source outside

of the project/program

Types Formative and Summative Evaluations (Mid-term and End-of-Project Reviews) Needs assessments Process

Standard Methodologies

Standard Evaluation Processes

Program/project as the unit of analysis Upfront utilization focus / cause and effect Linear logic models Focus on achieving pre-planned goals Findings assessed in relation to plans Time specific Methodologies

• Static designs• Preference for quantitative data• Randomized Control Trials-Gold Standard

Conditions that Challenge Evaluation

High innovation Developmental High uncertainty Dynamic (complex) Emergent

SCIENCE and SCIENCE SYSTEMS

Science Programs

• New field• Developing theories & tools• Developing metrics & analytic methodologies• NIH among the leaders (national & international)

Program & Management

Organization

project

technologies

communication administration

program

humancapital

management

A pictorial representation System Assessments

System Premises

Systems neutrality: A system is functioning as observed for some reasons, fulfilling some functions.

In whose interests is a system functioning?

Who benefits?

What does the homeostasis offer?

In a well-functioning system…

no sub-system operates at maximum

dynamic/systems change• Relationships are interdependent and non-linear, rather than

simple and linear (cause-effect)• Transitions - worse before better

complex - science• Highly emergent (difficult to plan and predict)

Healthy System

Internal preparationfor external evaluations

Independentexternal evaluations

Organizational andsystem assessments

SAB/EB work together to coordinate assessments of program/project performance and NIH overall performance

SystemPerformance

& Assessments

IndependentExternal

EvaluationsRelated Processes

Resources

StrategicPlanning

& Evaluability

SAB SABEB

DESA Synergy

• Providing Technical Assistance• Teaching Strategic Planning course• Developing and implementing evaluability survey• Providing results feedback• Preparing for GPRA and PART• Creating visual analytic tools

• Offering resources• Providing technical assistance• Guiding application process• Reviewing proposals• Monitoring awards• Promoting innovative methodologies

• Monitoring and Reporting- PART- GPRA- Audits- PAR/MD&A

DESA--implemented through a number of steps/tasks

SystemPerformance

& Assessments

IndependentExternal

EvaluationsRelated Processes

Resources

StrategicPlanning

& Evaluability

SAB SABEB

DESA Implementation

Be prepared for •Evaluations•Performance reporting•Strategic planning

Utilize analysis tools for decision makers

Conduct•Needs Assessments• Feasibility Studies• Process evaluations• Outcome evaluations• Evidence related activities

Score well in• PART• GPRA• Audits• PAR/MD&A

SAB and EB supports ICs to

SystemPerformance

& Assessments

IndependentExternal

EvaluationsRelated Processes

Resources

StrategicPlanning

& Evaluability

SAB SABEB

DESA Outcomes

SAB SABEB

Demonstrates appropriate federal stewardship

Provides a systematic process for conducting evaluations/ assessments that produce evidence-based results, which can be used to improve or adapt programs and/or management of science comply with performance assessment requirements plan

StrategicPlanning

& Evaluability

IndependentExternal

EvaluationsRelated Processes

Resources

SystemPerformance

& Assessments

DESA Summation