uxo risk assessment methods: critical review

1Arroyo CenterRAND

UXO Risk Assessment Methods: Critical Review

Jacqueline MacDonald, Debra Knopman, J. R. Lockwood, Gary Cecchine,

Henry Willis

RAND

2Arroyo CenterRAND

Briefing Outline

1. Need for UXO risk assessment methods Prioritization Site-specific assessment

2. RAND review of existing methods: tasks, approach

3. Design features and limitations of existing methods

4. RAND recommendations for improving UXO risk assessment

Prioritization Site-specific assessment

3Arroyo CenterRAND

Need for UXO Risk Assessment

Prioritization: mandated by Congress Site-specific assessment

Cost differences among alternative response options are significant

Total current cost estimate of $14 billion assumes one scan of site and excavation of anomalies only

Alternative approaches proposed by regulatory agencies could cost much more

DOD needs to understand risk reduction differences among alternative approaches

4Arroyo CenterRAND

Alternatives for UXO Response

Surface clearance only Scan with metal detector, and excavate each

anomaly to a specific depth (e.g. 2 ft, 3 ft) Scan, excavate each anomaly, scan bottom of

hole, and excavate again if anomaly is detected Scan and excavate anomalies, and then repeat the

process two or more times Excavate the entire site in one-foot lifts to depths

of 2 ft, 4 ft, or more; sift the excavated soil to remove UXO

5Arroyo CenterRAND

Alternatives Proposed at Ft. X

Scan land (surface & 1ft) Excavate entire site to 2 ft

Scan land (2ft level) Excavate entire site to 3 ft

Scan land (3ft level) Excavate entire site to 4 ft

Excavate anomalies found



Scan land (4ft level) Excavate anomalies found

Scan land with best available

technology (at surface, one scan

effort)

Provide construction support to property

developers

Excavate all anomalies found to maximum depth (4

feet)

Army’s Preferred Approach

State EPA’s Preferred Approach

6Arroyo CenterRAND

Cost Differences Among Ft. X Options Are Hundreds of Millions of Dollars

0

200

400

600

800

1000

0 2 3 4

Clearance Depth

To

tal

Co

st (

$ M

illi

on

s)

Army

State

7Arroyo CenterRAND

Briefing Outline


2. RAND review of existing methods




8Arroyo CenterRAND

RAND Tasks

Client: Army Chief of Staff for Installation Management

Tasks:

1. Conduct a preliminary analysis of ongoing efforts in UXO risk assessment, including:• Ordnance and Explosives Cost-Effectiveness

Risk Tool (OECert)• Interim Range Rule Risk Methodology (IR3M)• Ordnance and Explosives Risk Impact

Analysis (very similar to IR3M)• Risk Assessment Code (RAC)

9Arroyo CenterRAND

Tasks, continued

2. Study methods used by the Department of Energy, National Aeronautics and Space Administration, and others to evaluate and measure risk of low-probability and high-consequence events.

3. Recommend how the Army could develop a risk assessment/risk management protocol for UXO sites.

10Arroyo CenterRAND

Study Approach

1. Develop criteria for a technically sound risk assessment, based on risk assessment literature survey and consultations with experts

2. Read all documentation for available methods; test software if available

3. Interview method developers

4. Evaluate extent to which each method satisfies the evaluation criteria

11Arroyo CenterRAND

Evaluation Criteria Are in Three Categories

1. Risk calculation features, e.g.• Is the logic sound?• Are assumptions reasonable?• Is the output reproducible?

2. Implementation features, e.g.• Is the method practical for intended use?• Are instructions clear?• Is it adaptable to different sites?

3. Communication features, e.g.• Can stakeholders provide input about

assumptions, scenarios?• Is the method transparent?

12Arroyo CenterRAND

Briefing Outline






13Arroyo CenterRAND

Interim Range Rule Risk Methodolgy (IR3M)

Purpose:

To provide “a comprehensive process for managing, assessing, and communicating risks on former ranges”

Four tools:

1. Explosives Safety Risk Tool

2. Other Constituent Risk Tool for Human Health

3. Ecological Risk Tool

4. Comparative Analysis Tool

14Arroyo CenterRAND

Explosives Safety Risk Tool

Provides a risk value on a dimensionless scale of A to E

The A to E value is determined from decision rules that consider three input factors:

1. Accessibility (A)

2. Overall hazard (O)

3. Exposure (E)

15Arroyo CenterRAND

Explosive Safety Risk Tool Output

A

Risk Value Accessibility Overall Hazard Exposure

A < 2 < 3 < 2

B < 2 < 5 < 2

< 3 < 3 < 3

C < 4 < 3 < 4

< 3 < 5 < 3

= 5 < 3 < 2

< 2 < 3 = 5

D < 4 < 5 < 4

< 5 < 3 < 5

E < 5 < 5 < 5

O E

16Arroyo CenterRAND

RAND Evaluation of Tool

Considers main factors that determine explosion risks from UXO: hazard severity and likelihood of exposure

However, these factors are poorly quantified The risk calculation features do not satisfy the RAND

evaluation criteria: Interactions among risk contributors are poorly

modeled Output can be illogical Assumptions often are not explained Uncertainties are not described Output is not reproducible

17Arroyo CenterRAND

Hypothetical Example Illustrates IR3M Weaknesses

Conditions for hypothetical site:

• Climatic conditions are such that buried UXO is unlikely to surface

• Significant digging at the site is unexpected

• All other conditions are such that O = 5, E = 2

Two UXO response options:

• Option 1: Excavate UXO to 0.99 ft of depth

• Option 2: Excavate UXO to 1.01 ft of depth

18Arroyo CenterRAND

Calculate A for Options 1 and 2

Depth below land surface 1) All UXO > 10 feet 2) All UXO > 4 feet 3) All UXO > 2 feet 4) All UXO > 1 foot 5) Any UXO < 1 footMigration /Erosion Potential 1) Very stable 2) Minor migration 3) Moderate migration 4) Significant migration 5) Highly dynamic

Intrusion Level of Activity 1)Non-intrusive: surface only 2) Minor: to 1 ft 3) Moderate: to 2 ft 4) Significant: to 4 ft 5) Highly intrusive

Option 1: A = 5

Option 2: A = 3

A Depth Migration Intrusion

1 = 1 < 2 < 2

2 = 1 < 5 < 5

= 2 < 3 < 3

3 = 2 < 5 < 5

= 3 < 4 < 4

= 4 < 2 < 2

4 < 4 < 5 < 5

5 = 5 < 5 < 5

19Arroyo CenterRAND

IR3M Output for Options 1 and 2 Is Not Logical

A = 5

Risk Value Accessibility Overall Hazard Exposure

A < 2 < 3 < 2

B < 2 < 5 < 2

< 3 < 3 < 3

C < 4 < 3 < 4

< 3 < 5 < 3

= 5 < 3 < 2

< 2 < 3 = 5

D < 4 < 5 < 4

< 5 < 3 < 5

E < 5 < 5 < 5

O = 5 E = 2A = 3

20Arroyo CenterRAND

Participants in IR3M Test Exercise Reached Similar Conclusions

• IR3M was tested by two independent teams in a hypothetical site evaluation exercise

• The teams found problems similar to those encountered by RAND:

“Intuitively, the scores did not make sense. . . . If action removes all known UXO, the tool does not reflect a change in protectiveness and therefore there is a flaw in the tool.”

“With a 5 on Overall Hazard and Accessibility, reducing exposure does not alter the overall risk. If there is one hand grenade at land surface, the overall risk is E. Thus, there is no way to reduce risk by reducing exposure.”

21Arroyo CenterRAND

Army Science Board (1998) Also Reached Similar Conclusions

“The existing R3M model is not sufficiently grounded or developed to accompany promulgation of the Range Rule in FY99.”

“Do not highlight the existing risk formulation in working toward promulgation of the Range Rule.”

22Arroyo CenterRAND

Ordnance and Explosives Cost-Effectiveness Risk Tool (OECert)

Purpose:

“The . . . OECert . . . provides the capability to estimate risks and cost associated with given levels of remediation and to prioritize among any set of sites by life cycle cost and the amount of reduction in public risk expected to be realized from the remediation process.”

23Arroyo CenterRAND

OECert Risk Calculation

Three methods:

1. Dispersed sector method (ranges, impact areas)

2. Localized sector method (burial pits, trenches)

3. Water sector method (submerged UXO)

For all methods:

Risk = (Number of Expected UXO Exposures)

x (UXO Hazard Factor)

24Arroyo CenterRAND

Dispersed Sector Method: UXO Hazard Factor

UXO Type Sensitivity Factor

Consequence Factor

Product Adjusted Hazard Factor

UXO 126 80 10,080 29

UXO light motion sensitive

327 80 26,160 76

UXO white phosphorus

126 36 4,536 13

Controlled chemical, biological, and radiological

126 273 34,398 100

25Arroyo CenterRAND

Dispersed Sector Method: Exposure Calculation

Site is divided into homogeneous “sectors” UXO density is assumed homogeneous in each

sector Density is calculated using statistical sampling

OR based on densities of similar sites in a database of Formerly Utilized Defense Sites

Within each sector, number of exposures to UXO is calculated for 19 activities

26Arroyo CenterRAND

Dispersed Sector Method Considers 19 Activities

Activities with Risks of Surface Exposures Only

Activities with Risk of Surface and Subsurface Exposure

Hiking Fishing Child playing Crop farming

Horseback riding

Swimming Off-road driving

Archaeology

Hunting Surveying Picnicking

Mountain biking

Ranching Camping

Motor biking Jogging Metal detection

Short cuts Construction

27Arroyo CenterRAND

Example: Camping

Total number of exposures is calculated for

1. hiking to campsite

2. setting up camp

Number of expected campers is determined from

1. demographic data for surrounding area

2. generic (nation-wide) estimates of percentage of people in each age category who camp

28Arroyo CenterRAND

Exposures During Hiking to Campsite

Distance of path to campsite is estimated as radius of a circle with area equivalent to area of sector

Area traversed by single individual:

traversed area = (sector area/π)1/2 x (path width)

Number of UXO exposures for single individual:

exposures = (traversed area) x (surface density)

29Arroyo CenterRAND

Exposures At Campsite

Assumed area of campsite = 2,500 ft2

Assumed subsurface intrusive area = 0.75 ft2

Exposures within campsite:

exposures = 2,500 ft2 x (surface density) +0.75 ft2 x (subsurface density)

30Arroyo CenterRAND

Total Exposures for UXO Site

sectors activities

activity individualexposures Total N

activityin ingparticipat sindividual ofnumber

personper dencountere items UXOofnumber

activity

individual

N

31Arroyo CenterRAND

RAND Evaluation of OECert

Overall structure of modeling UXO explosion risk has merit (exposure probability times exposure consequences)

However, there are major limitations: Contains many unjustified assumptions (e.g.,

campsite size of 2,500 ft2, camper intrusion area of 0.75 ft2)

Army staff report that output is usually 100,000 exposures

Software crashed when we tried to use it Output is not meaningful outside OECert context Process is not transparent Density estimation procedures need to be improved

32Arroyo CenterRAND

Risk Assessment Code (RAC)

Purpose: Prioritize sites in the Formerly Utilized Defense Sites (FUDS) program based on UXO explosion risk

Two factors considered:

1. Hazard severity

2. Hazard probability

33Arroyo CenterRAND

Hazard Severity Score

Based on five broad categories of ordnance:

1. Conventional ordnance and ammunition

2. Pyrotechnics

3. Bulk high explosives (uncontainerized)

4. Bulk propellants

5. Chemical warfare materiel and radiological weapons

A score is developed for each category based on types of ordnance present

Total score is based on sum of scores for the 5 categories (maximum possible is 61)

34Arroyo CenterRAND

Example: Conventional Ordnance Score

Ordnance Type Score

Medium/large caliber (20 mm and larger) 10

Bombs, explosive 10

Grenades, hand or rifle, explosive 10

Landmine, explosive 10

Rockets, guided missile, explosive 10

Detonators, blasting caps, fuzes, boosters, bursters

6

Bombs, practice (w/spotting charges) 6

Grenades, practice (w/spotting charges) 4

Landmine, practice (w/spotting charges) 4

Small arms, complete round (.22 cal-.50 cal) 1

Largest single value is

chosen

35Arroyo CenterRAND

Hazard Probability Score

Based on five subscores related to probability of human exposure to UXO:1. UXO location (surface, subsurface, etc.)2. Distance to nearest inhabited location3. Number of buildings within 2-mile radius4. Types of buildings (schools, residential,

industrial, etc.)5. Site accessibility (barriers, guards, etc.)6. Site dynamics (e.g., erosion, future

development)Total score is sum of these six subscores

(maximum possible is 30)

36Arroyo CenterRAND

RAC Score Is on 1-5 Scale

Hazard Severity

Hazard Probability

Frequent (>26)

Probable (21-26)

Occasional (15-20)

Remote (8-14)

Improbable (<8)

Catastrophic (>20)

1 1 2 3 4

Critical (10-20)

1 2 3 3 5

Marginal (5-9)

2 3 4 4 5

Negligible (1-4)

3 4 4 5 5

37Arroyo CenterRAND

RAND Evaluation of RAC

Model of estimating risk based on exposure probability and explosion severity is sound

Clear, easy-to-use instructions Appropriate for intended use (priority setting) Not detailed enough for site-specific assessment Limitations

UXO density is not considered in the score Does not address uncertainty Does not address issues of stakeholder

involvement and communication

38Arroyo CenterRAND

Summary: Existing Methods Do Not Satisfy Criteria

OE-CERT

Communication features

Implementation features

Risk calculation features

OERIAIR3MRAC

= does not satisfy evaluation criteria= satisfies some criteria

= satisfies all criteria

39Arroyo CenterRAND

Briefing Outline






40Arroyo CenterRAND

We Recommend a Two-Stage Process for Prioritization

Two stages: Stage 1: sort sites into explosion risk “bins” Stage 2: within bins, sort by munitions

constituent risks

Key design feature: Preserves separate information about explosion,

other constituent risks Separation is necessary; attempts to combine

these risks involve value judgments that should be explicit and transparent in decision-making

41Arroyo CenterRAND

Munitions constituent risk




No explosion risk sites

Low explosionrisk sites

Moderate explosionrisk sites

High explosionrisk sites

Determine explosion risk

Determine munitionsconstituent risk for eachsite in category

1

2

Rank Rank Rank Rank

Explosion risk None Low Moderate High













Policymakers decide how to allocate resources among bins

Modified RAC?

Hazard Ranking System or Relative Risk Site Evaluation Primer

42Arroyo CenterRAND

Briefing Outline






43Arroyo CenterRAND

We Searched Other Agencies for Models

Agency Model Applicability to UXO

EPA Risk Assessment Guidance for Superfund

Munitions constituent risks

NRC Probabilistic risk assessment (PRA)

Explosion risks

NASA PRA Explosion risks

FAA PRA Explosion risks

Army (Chemical Stockpile Disposal)

PRA Explosion risk

44Arroyo CenterRAND

Example: Army Uses PRA to Assess Risk of Chemical Weapons Release

Fault tree used in PRA for Tooele Chemical Agent Disposal Facility

45Arroyo CenterRAND

Potential Benefits of Using PRA for Explosion Risk Assessment

Widely and successfully used by other agencies and private industry

Could allow for meaningful quantitative evaluation of risks and benefits of response alternatives

Requires systematic analysis of elements of risk, which provides formal structure for decision-making and negotiations

46Arroyo CenterRAND

Army Science Board Made Similar Recommendation

Army Science Board reviewed IR3M in October 1998

Army Science Board recommended that the DOD and Army:

“consider the probabilistic and step (I.e., ON or OFF) nature of explosive events and consequences”

“consider using an approach analogous to the successful formulation of design codes based on the probability of structural failure”

47Arroyo CenterRAND

Summary of Recommended Approach for Site-Specific Risk

Assessment

1. Use EPA Risk Assessment Guidance for Superfund methods to assess risks of munitions constituents

2. Develop probabilistic risk assessment method specific to UXO to assess explosion risks Development should be overseen by a

technical advisory committee The method should be independently peer

reviewed Template “trees” should be developed Trees then could be modified at individual

sites, with substantial stakeholder input

48Arroyo CenterRAND

Summary

Existing methods for UXO risk assessment do not satisfy criteria for technical credibility

New prioritization method should include a two-tier screen:

Tier 1: sort by explosion risks (using new method—possibly a modified RAC)

Tier 2: sort by constituent risks (using existing method from EPA or Defense Environmental Restoration Program)

New site-specific assessment method should Use EPA’s RAGS for constituent risks Use new PRA method for explosion risks

49Arroyo CenterRAND

“Fear of harm ought to be proportional not merely to the gravity of the harm, but also to the probability of the event.”

--Antoine Arnauld, 1660

uxo risk assessment methods: critical review

Documents