learning to care for those in harm’s way measures of effectiveness in defense engagement and...

Learning to Care for Those in Harm’s Way

Measures of Effectiveness in Defense Engagement and Learning (MODEL) Study

Principal Investigator: CAPT Glendon Diehl, PhD

UNCLASSIFIEDSeptember 2015


Background

Purpose: The MODEL study was developed to determine the effectiveness of Global Health Engagements (GHEs) as a Theater Security Cooperation (TSC) tool

Timeline

• Initial Driver: Constrained resources and need to prioritize

• Requirement: 2013 National Defense Authorization Act (NDAA) Section 715

• Resources: Funded by the Office of the Assistant Secretary of Defense for Health Affairs (ASD(HA))

• Implementation: Executed through USUHS and conducted at CDHAM

May 12 Aug 12 Oct 12 Nov 12 – Jul 13 Aug – Oct 13 Nov 13 – Jan 14

HASC passes 2013 NDAA

ASD(HA) funds MODEL study

Start of period of performance

Conducted retrospective assessment

Specified analytical approach

Produced initial analyses

Feb – Dec 14

Validated models, obtained additional data sources, and submitted articles for publication

Jan – Mar 15

Specified different levels of assessment

2


Engagements in OHASIS

The need to prioritize resources is evident in historical data trends, indicating that from 2001-2012 DoD conducted 2,654 GHEs in 140 Countries1

1. The data depicted on this slide is from the Overseas Humanitarian Assistance Shared Information System, a Defense Security Cooperation (DSCA) authoritative database that includes all Overseas Humanitarian, Disaster, and Civic Aid (includes Humanitarian Assistance, Disaster Relief, and Humanitarian Mine Action) and Humanitarian Civic Assistance Program (Title 10, Sec 401) engagements

0

50

100

150

200

250

300

350

0

200

400

600

800

1000

1200

1400

FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12

Tota

l Fun

ding

Allo

cate

dM

illio

ns

Coun

t of

Eng

agem

ents

OHASIS Health and Non-Health Project Trends FY01-FY12

Total Projects Health Non-Health Total Funding Health Funding Non-Health Funding

Over the period from FY01-FY12, approximately $1.13 billion was budgeted for 7,530 OHASIS engagements, of which $343 million (30%) was allocated to 2,654 health

engagements (health engagements made up 35% of all OHASIS engagements)

3


Magnitude of GHEs by Combatant Command (CCMD)

About Tableau maps: www.tableausoftware.com/mapdata

134 164

115

225

315

246

186

166126

106

123

203

117108

338

130160

100

102

102

342

64

14

74

44

14

14

55

45

65

25

25

25

85

65

15

15

45

86

86

1616

56

36

73

23

43

23

73

28

17

18 18

37

38

38 17

18

98

4887

18

17

67

49

29

19

2949

4050

40

40

50

10

50

10 10

20

12

12

92

22

72

22

42

52

52

72

31

71

51

11

11

4

4

4

4

4

4

5

5

5

5

5

6

6

3

3

3

33

7

78 8

8

8

8

8

9

9

2

2

2

2

2

2

2

2

21

1

1

1

1

1

11

1

1,584

1,595

2,840

975

842

Total Number of Engagements by CCMD

$289.0M

$316.3M

$309.3M

$162.1M

$94.2M

Total Funding by CCMD

COCOMAFRICOM

CENTCOM

EUCOM

PACOM

SOUTHCOM

4


Objective 1Levels of Assessment

5

Da

ta (

MO

Ps

an

d M

OE

s)

* MOE categories in backup slides

MODEL has developed analytic methods to assess the effectiveness of GHEs at the strategic, operational, and tactical levels

National

COCOM

Component or Service

Tactical Engagement

Operational• DOTMLPF Assessment: Provides a standardized tool to

baseline partner nation capabilities, relative value and track progress over time across a number of key capability areas

• Balanced Risk Scorecards: Calculates an indexed risk score for proposed GHEs that can be used to prioritize activities and funding

Tactical• Impact Evaluation: Determines the effectiveness of

individual engagements in terms of mission planning and execution, impact on the partner nation, benefit to US personnel, and provides evidence for process improvement

Strategic• Econometric Assessment: Generates a quantitative return

on investment of GHE funding on a variety of strategic, health and readiness Measures of Effectiveness (MOEs)*

• Qualitative Methods: Identifies overarching capability gaps, thematic problems/successes, and confounding conditions impacting GHE effectiveness

Po

lic

y a

nd

Do

ctr

ine


Hypotheses and Theory

6

Let “Y” be the “Variable of Interest” (or the Outcome specified by the

Commander’s Intent)

Let “X” be the “Treatment Variable”(or the Military Activities being

undertaken)

Let “” be read as “causes”

(or, for the moment, “correlated with”)

A theory tells us what to expect and why by predicting associations, which can be articulated as hypotheses, or statements of fact and written accordingly (i.e., if x then y)

(MOP)GHE MMR1

(MOE)

1. Maternal Mortality Rate (MMR) is an example of a health-related MOE


Strategic Assessments

7

Collaborate with USUHS and PKSOI5 to

Develop SOLLIMS Analysis Approach

Econometric Results

Generated

Measures Validated

by CCMDs

Collaborate with DSCA4 to Access, Clean, and Sort OHASIS Data

MODEL Assessment

Approach Developed

JAN 13 APR 13 JUL 13 SEP 13 DEC 13 SEP 14 JAN 15

Econometric Methods

Established

AFHSC ProMIS7

Data Collection

Qualitative Methods

Identifies overarching gaps and lessons learned• Status

‒ Conducting qualitative review of all GHE lessons learned (approximately 2,800) in the Joint Lessons Learned Information System (JLLIS)

• Current Data Sources‒ Stability Operations Lessons Learned Information

System (SOLLIMS) and JLLIS

• Next Steps: Continue to test and validate methods; incorporate data and techniques; produce analyses

Econometric Assessment

Generates a quatitative return on investment• Status

‒ Producing econometric analyses for a range of stakeholders1

‒ Working with Joint Staff to task the CCMD Surgeon Generals to identify and prioritze MOEs

• Current Data Sources‒ DoD GHE data (MOPs): OHASIS2, G-TSCMIS3,

USAID Overseas Loans and Grants‒ MOEs: World Bank, United Nations, Global Burden

of Disease

• Next Steps: Continue to test and validate methods; incorporate data and techniques; produce analyses

1. Combatant Commands, DoD HIV/AIDS Prevention Program, Humanitarian and Civic Assistance, Armed Forces Health Surveillance Center, Office of the Chief of Naval Operations International Engagement, U.S. Army Headquarters Stability Operations2. Overseas Humanitarian Assistance Shared Information System3. Global Theater Security Cooperation Management Information Systems4. Defense Security Cooperation Agency5. Peacekeeping and Stability Operations Institute6. DoD HIV/AIDS Prevention Program7. Armed Forces Health Surveillance Center (AFHSC) Proposal Management Information System (ProMIS)

Status: Relatively Mature

MAR 14

DHAPP6 Analysis

MOE Categories Established

MAY 14 MAY 15

G- TSCMIS Data

Acquired

JUL 15

Completed G-TSCMIS Codebook

AUG 15

G-TSCMIS Analysis/

JLLIS Analysis


8

Econometric and Qualitative Results

Qualitative Results

Econometric ResultsTable 1. All CCMDs OHASIS Health Engagements Stoplight Regression Table

Table 2. PACOM OHASIS Health Engagements Stoplight Regression Table

The nine themes below were derived from the GHWG Capabilities Committee’s review of 70+ documents to identify overarching GHE capabilities

SOLLIMS Example

Approach: Reviewed the SOLLIMS database and identified nine (9) overarching lessons learned themes


Operational Assessments

9

Refined OHASIS

Templates

Developed Balanced Risk Scorecards

Provided DSCA with Initial

Recommendations

Researched Existing Efforts

JUL 13 OCT 13 JAN 14 APR 14

JUL 14 OCT 14

FEB 15

Identified Operational Assessment

Gaps

Socialized DOTMLPF at Health TSC

Planning Group

Balanced Risk Scorecards

Prioritizes activities and funding by a risk score• Status

‒ DSCA is reviewing scorecard with intent to incorporate into OHASIS submission template

• Intended Data Sources‒ Organizations with funding approval authority over

GHEs

• Next Steps‒ Incorporate into OHASIS and G-TSCMIS

DOTMLPF AssessmentProvides a tool to track capabilities over time

• Status‒ PACOM is leading the effort to mature and validate

DOTMLPF assessments for draft Patient Movement, Blood Program, Force Health Protection and Basic First Responder

‒ Adapted the DOTMLPF assessment tool for DPP’s1 West African Disaster Management Initiative

• Intended Data Sources‒ US and Partner Nation personnel with expertise in

the associated capability area

• Next Steps‒ Standardize and validate the tool, expand to other

capability areas, disseminate to other CCMDs

Status: Developing

DEC 14

Updated Health TSC Planning Group on DOTMLPF Testing

and Initial Validation

DSCA Reviewing Recommendations

MAR 15

Initiated Development of

DOTMLPF Scorecards

Established OHASIS

Working Group with DSCA

JUN 15

Adapted DOTMLPF tool for DPP’s West African Disaster Management

Initiative

1. Disaster Preparedness Program


Tactical Assessments

10

USUHS Proposed Tactical Level Study

to Health Affairs

Further Developed

and Refined Methods

MODEL Produced Initial DHAPP Results

MODEL Met with DHAPP

JUL 14 SEP 14APR 14 MAY 14 OCT 14 JAN 15

Identified Tactical

Assessment Gaps

ASD HA Awarded Funding to USUHS for Best Practices

Study

Impact EvaluationDetermines effectiveness of individual missions in terms of planning/execution, impact on the partner nation,

benefit to US personnel, and provides evidence for process improvement• Status

‒ In the initial phases of standing-up a study to conduct tactical level assessments

• Intended Data Sources‒ Implementing organization (e.g., CCMD, Component) planners and executers‒ Partner nation ministries, agencies, and Subject Matter Experts‒ Mil group at embassy

• Next Steps‒ Identify partner(s), refine methodology and identify metrics, conduct data collection down range, perform analysis

Status: Initial

DHAPP funded MODEL to perform

tactical assessment


Collaborations

11

MODEL’s relationships with key stakeholders continue to grow and develop

• Met with DHAPP Analysts (13-14 JAN) to confirm the MOEs identified, identify information for data entry into G-TSCMIS and OHASIS, and link expenditure data with SOWs

• Working with DSCA to update the OHASIS templates based on DHAPP requests

• Discussed conducting an impact evaluation to evaluate DHAPP engagements at the intra-country level

Defense HIV/AIDS Prevention

Program (DHAPP)

• Proposed three changes to OHASIS including adding a balanced risk scorecard, fields to capture level of effort (man hours), and Joint Capabilities Areas to link engagements to Title 10 requirements

• Worked with DSCA to obtain engagement data from G-TSCMIS

Defense Security Cooperation

Agency (DSCA)

• Collaborated with the DPP’s Operation Unified Assistance Transition Disaster Preparedness Project to develop assessment tools and objectives

• Assisted in the development of a Disaster Management Capability Assessment Tool and Survey for assessing Partner Nations’ respective disaster preparedness and management capabilities over time

Disaster Preparedness Program (DPP)

• Supported the GHWG MOE Committee review and analysis of 70+ policy and doctrine documents to identify GHE capabilities

• Compared and contrasted multiple assessment efforts from across DoD, the Interagency, academia, and NGOs

• Provided strategic level analytic capabilities that met the intent of the GHWG Charter to develop “A process to ensure DoD GHEs are meeting the national security goals of the United States”

Global Health Working Group

(GHWG)

• Obtained access to data stored in AFHSC’s Proposal Management Information System (ProMIS)

• Discussed options for sorting the data within ProMIS by AFHSC’s strategic pillars – prevent, detect, respond, capacity building, and studies

Armed Forces Health

Surveillance Center (AFHSC)


MODEL Accomplishments

12

MODEL has made significant progress towards the achievement of the 2013 NDAA Section 715 A and C by developing (1) a process that ensures DoD GHEs

are effective and efficient and (2) a MOE learning tool

MODEL Accomplishments to Date:

• Assessment Capabilities

• Employed both quantitative and qualitative methods to identify extant assessment capabilities

• Designed various assessment methods in response to the demands of specific stakeholders

• Validated regression models and dataset with various stakeholders

• Discovered additional capability gaps, particularly at the operational and tactical levels

• Data Management

• Accessed, collected, reviewed, cleaned and sorted data from various data sources

• Created a unique dataset with data from 2001 to present that can be employed in a variety of econometric models

• Grouped MOEs into categories like strategic, health and readiness


Return on Investment (ROI)

13

MODEL delivers a ROI by: (1) evaluating the efficiency of GHEs; (2) comparing the efficiency of types of GHEs; (3) identifying capability gaps and providing recommendations

for closing the gaps; and (4) improving data management and accountability

Data

Stan

dard

ization

GHE Efficiency Evaluation by Type

of Engagement

Cap

abili

ty G

ap

An

alys

is

GHE Efficiency Evaluation

ROI

• Provide recommendations for improving data collection and database management

• Illustrate the benefits of good data collection before, during, and after program implementation

• Hold programs and agencies accountability for maintaining a high level of data quality

• Identify DOTMLPF capability gaps within a program and provide recommendations for improving capabilities

• Identify common capability gaps across types of GHEs and provide support for policy changes to address these gaps

• Illustrate which type of GHEs (i.e., disaster preparedness, infrastructure, health capacity building, direct health care) have the largest impact on each MOE

• Ex: As OHASIS spending increases, infrastructure GHEs have the greatest (statistically significant) impact on ideal policy point differences

• Quantifies the impact of GHEs on strategic, health, and readiness MOEs as GHE spending increases

• Ex: A 1% increase in the OHASIS budget returns an “x” unit reduction in ideal policy point differences


MODEL Next Steps

14

For the remainder of the study, MODEL will undertake the following tasks:

• Data Validation: validate data sources; ensure MODEL proxy measures and sources are appropriate

• Analysis: Identify MODEL assessment capability(ies) for potential collaboration

• Product Development: Develop products/deliverables, provide policy recommendations, and identify opportunities for future publication

• Address 2013 NDAA Section 715 B:

Verbiage from 2013 NDAA MODEL Proposed Way ForwardSec. 715 (B) (1) Assesses the operational mission

capabilities of the health engagementLeverage MOE analytic methods to inform capability development, requirements, and planning and formalize in policy/doctrine

Sec. 715 (B) (2) Uses the collective expertise of the Federal Government and non-governmental organizations to ensure collaboration and partnering activities

Establish an integrated organization with authority and a governance structure to coordinate GHE efforts

Sec. 715 (B) (3) Assesses the stability and resiliency of the host nation of such engagement

• Continue to mature, validate, and promulgate analytic methods

• Improve data entry, data quality, and information sharing across DoD, Interagency, and NGOs/IGOs


Measures of Effectiveness in Defense Engagement and Learning (MODEL) Study

Principal Investigator: CAPT Glendon Diehl, PhD

UNCLASSIFIEDSeptember 2015


Back Up Slides

16


17

Publication Title Description Journal Status Outcome

Measuring the Impact of Global Health Engagements, an Econometric Approach

Describes the application of econometric techniques to measures of effectiveness research and evaluation

MOR (Military Operational Research) Journal

Submitted APR 2014

Returned R&R JUN 2014

Provides a 2-stage least squares, fixed effects regression models to consider the impact of global health engagement (GHE) on a range of partner nation (PN) health outcomes and on the capacity of GHE to assist the geographic combatant commands (GCCs) to achieve a number of “strategic” and soft-power goals

MOE vs. M&E: Considering the Difference between Measuring Strategic Effectiveness and Monitoring Tactical Evaluation

Introduces empirical techniques, long-established as best-practices within the development economics literature, into the military assessments field

Military Medicine Journal

Published JAN 2015

Development of measures of effectiveness (MOEs) process to evaluate the impact of global health engagements on strategic end-states

A Qualitative Content Analysis of Global Health Engagements in PKSOI's SOLLIMS

Evaluates global health engagements (GHEs) using a summative qualitative content analysis approach to identify overarching lessons-learned for GHEs.


Published APR 2015

Illustrates how the SOLLIMS lessons-learned align with the GHE capabilities developed by the DOD's Global Health Working Group

Finding a Healthy Match: A Discussion on Applying the Appropriate Level of Analysis for GHE Activities

An op-ed discussing the importance of collecting and recording high quality data for strategic-level assessments


Submitted MAR 2015

Returned R&R JUL 2015

Illustrates the importance of data collection when determining the level of assessment and measuring the impact of engagements

Healthy End States? A Descriptive Analysis of OHASIS

A retrospective analysis of TSC data available in the Overseas Humanitarian Assistance Shared Information System (OHASIS), inclusive of GHE data

Joint Force Quarterly

Submitted MAR 2015

Published OCT 2015

Serves as an initial step in understanding how humanitarian assistance activities have been used by the USG over the past decade.

White Hull or White Elephant? Soft Power and the Chinese Hospital Ship, the 'Peace Ark'

Explores the soft power impact of the humanitarian outreach missions of the Chinese hospital ship, the Peace Ark

Defense and Security Analysis Journal

Submitted JUL 2014

Published DEC 2015

Determines whether the regions and countries visited by the first two Harmonious Missions (2010 and 2011) witnessed shifts in policy outcomes consistent with Chinese interests

Presentation Event Description Conference Status Outcome

MODEL Poster for Global Health Metrics and Evaluation (GHME) Conference

Overview of MODEL methods in categorizing OHASIS projects GHME Conference

Presented JUN 2013

Demonstrated MODEL’s initial methods through descriptive statistics on OHASIS from FY08-FY12

MODEL Abstract for Association of Military Surgeons United States (AMSUS) Conference

Overview of MODEL study AMSUS Conference

Presented NOV 2013

Presented MODEL study’s lines of effort and descriptive statistics on OHASIS from FY08-FY12

MODEL Poster for Consortium of Universities for Global Health (CUGH) Conference

Updated OHASIS descriptive statistics with initial analysis on health and strategic MOEs CUGH Conference

Presented MAY 2014

Descriptive statistics on OHASIS from FY02-FY12 and presented potential MOEs for analysis on GHEs

MODEL Abstract for AMSUS ConferencePresenting the challenges and solutions to GHE assessment

AMSUS Conference

Presented DEC 2014 Demonstrating MODEL’s relevance in GHE assessment

MODEL Poster for Consortium of Universities for Global Health (CUGH) Conference

Updated OHASIS descriptive statistics with initial analysis on health and strategic MOEs CUGH Conference

Presented MAR 2015

Descriptive statistics on OHASIS from FY02-FY1; collaboration with DHAPP – initial results and impact evaluation study

CDHAM Analytic Capabilities Poster or Presentation at the International Committee on Military Medicine (ICMM) 41st World Congress

Overview of MODEL and Best Practices grants describing strategic, operational, and tactical assessment capabilities available at CDHAM

ICMM 41st World Congress

Presented MAY 2015

Provide the international military community with an overview of DoD’s spectrum of assessment capabilities

Publications and Presentations


MEANSReadiness• Interoperability• Retention• Training• Provision of Care

WAYSHealth• Disability Adjusted Life Years• Health Services Strengthening• Health Systems Outcomes• Potential Hazards• Resiliency

ENDSStrategic• Policy Preferences• Security• Stability• Access• Partnership• Capability

One size does NOT fit all: Assessing GHE effectiveness across the three broad categories will provide decision-makers with information to prioritize resources

Note: Bulleted items represent MODEL’s Measures of Effectiveness examples

18

MOE Categories


OHASIS Analysis (1 of 2)

DV = POLICY IDEAL PT. DIFF. 2SLS(FE)

DV = STATE FRAGILITY 2SLS(FE)

OHASIS ALL HEALTH -0.5243*** -4.6354***(0.1589) (1.1535)

NON-MIL USA AID -0.0168** 0.2207***(0.0081) (0.0587)

INTERNATIONAL AID/pc 0.1898**(0.0944)

TOTAL RENTS -0.0114 -0.2406*(0.0202) (0.1408)

GDP/pc -0.095*** -1.2161***(0.0227) (0.1615)

POLITY -0.0032 -0.064**(0.0037) (0.0266)

CONFLICT INDEX -0.0064*** 0.0034(0.0022) (0.0159)

POPULATION -0.6839*** -0.5398(0.0994) (0.694)

n 1374 1336Countries 154 149

1st stage F statistic 28.7000 25.7800

Endogeneity test (χ2) 0.0001 0.0000

Standard errors in (parentheses). p < .10 = ‘*’ ; p < .05 = ‘**’ ; p < .01 = ‘***’.

TABLE 1. Policy MOEs & the Impact of Global Health Engagement

19


OHASIS Analysis (2 of 2)

DV = INFANT MORTALITY2SLS(FE)

DV = MATERNAL MORTALITY RT.2SLS(FE)

DV = TB DALYs2SLS(FE)

OHASIS GHE -0.6419*** -0.2689** -0.2327***(0.1334) (0.1338) (0.064)

NON-MIL USA AID -0.0076 -0.0193** -0.0008(0.0063) (0.0076) (0.0033)

INTERNATIONAL AID 0.0328*** -0.0059 0.0169***(0.0102) (0.0107) (0.0048)

GDP/pc -0.2149*** -0.1681*** -0.1629***(0.0438) (0.0534) (0.0242)

HN HEALTH/pc -0.0324 0.0068 -0.0368*(0.0436) (0.0528) (0.0237)

GOVT CONSUMPTION 0.023 0.4407** 0.1193(0.2049) (0.2234) (0.1004)

PCT. WATER 0.002 -0.0117*** -0.002(0.0034) (0.0039) (0.0018)

POP. DENSITY -0.1841*(0.1025)

POPULATION -0.2078** -0.199* -0.0835(0.0918) (0.1129) (0.1123)

n 1252 1035 1113Countries 141 140 140

1st stage F statistic 14.846 15.468 15.122

Endogeneity test (χ2) 0.6372 0.9114 0.2102

Sargon test (χ2) 0.0000 0.0328 0.0000

Standard errors in (parentheses). p < .10 = ‘*’ ; p < .05 = ‘**’ ; p < .01 = ‘***’.

TABLE 2. Health MOEs & the Impact of Global Health Engagement

20

learning to care for those in harm’s way measures of effectiveness in defense engagement and...

Documents

defense engagement

effectiveness of ghes

kind of engagement

countries3magnitude

proposed ghes

harms way2engagements

harms way1backgroundpurpose

ccmd total funding