1 arroyo center rand uxo risk assessment methods: critical review jacqueline macdonald, debra...
TRANSCRIPT
1Arroyo CenterRAND
UXO Risk Assessment Methods: Critical Review
Jacqueline MacDonald, Debra Knopman, J. R. Lockwood, Gary Cecchine,
Henry Willis
RAND
2Arroyo CenterRAND
Briefing Outline
1. Need for UXO risk assessment methods Prioritization Site-specific assessment
2. RAND review of existing methods: tasks, approach
3. Design features and limitations of existing methods
4. RAND recommendations for improving UXO risk assessment
Prioritization Site-specific assessment
3Arroyo CenterRAND
Need for UXO Risk Assessment
Prioritization: mandated by Congress Site-specific assessment
Cost differences among alternative response options are significant
Total current cost estimate of $14 billion assumes one scan of site and excavation of anomalies only
Alternative approaches proposed by regulatory agencies could cost much more
DOD needs to understand risk reduction differences among alternative approaches
4Arroyo CenterRAND
Alternatives for UXO Response
Surface clearance only Scan with metal detector, and excavate each
anomaly to a specific depth (e.g. 2 ft, 3 ft) Scan, excavate each anomaly, scan bottom of
hole, and excavate again if anomaly is detected Scan and excavate anomalies, and then repeat the
process two or more times Excavate the entire site in one-foot lifts to depths
of 2 ft, 4 ft, or more; sift the excavated soil to remove UXO
5Arroyo CenterRAND
Alternatives Proposed at Ft. X
Scan land (surface & 1ft) Excavate entire site to 2 ft
Scan land (2ft level) Excavate entire site to 3 ft
Scan land (3ft level) Excavate entire site to 4 ft
Excavate anomalies found
Excavate anomalies found
Excavate anomalies found
Scan land (4ft level) Excavate anomalies found
Scan land with best available
technology (at surface, one scan
effort)
Provide construction support to property
developers
Excavate all anomalies found to maximum depth (4
feet)
Army’s Preferred Approach
State EPA’s Preferred Approach
6Arroyo CenterRAND
Cost Differences Among Ft. X Options Are Hundreds of Millions of Dollars
0
200
400
600
800
1000
0 2 3 4
Clearance Depth
To
tal
Co
st (
$ M
illi
on
s)
Army
State
7Arroyo CenterRAND
Briefing Outline
1. Need for UXO risk assessment methods Prioritization Site-specific assessment
2. RAND review of existing methods
3. Design features and limitations of existing methods
4. RAND recommendations for improving UXO risk assessment
Prioritization Site-specific assessment
8Arroyo CenterRAND
RAND Tasks
Client: Army Chief of Staff for Installation Management
Tasks:
1. Conduct a preliminary analysis of ongoing efforts in UXO risk assessment, including:• Ordnance and Explosives Cost-Effectiveness
Risk Tool (OECert)• Interim Range Rule Risk Methodology (IR3M)• Ordnance and Explosives Risk Impact
Analysis (very similar to IR3M)• Risk Assessment Code (RAC)
9Arroyo CenterRAND
Tasks, continued
2. Study methods used by the Department of Energy, National Aeronautics and Space Administration, and others to evaluate and measure risk of low-probability and high-consequence events.
3. Recommend how the Army could develop a risk assessment/risk management protocol for UXO sites.
10Arroyo CenterRAND
Study Approach
1. Develop criteria for a technically sound risk assessment, based on risk assessment literature survey and consultations with experts
2. Read all documentation for available methods; test software if available
3. Interview method developers
4. Evaluate extent to which each method satisfies the evaluation criteria
11Arroyo CenterRAND
Evaluation Criteria Are in Three Categories
1. Risk calculation features, e.g.• Is the logic sound?• Are assumptions reasonable?• Is the output reproducible?
2. Implementation features, e.g.• Is the method practical for intended use?• Are instructions clear?• Is it adaptable to different sites?
3. Communication features, e.g.• Can stakeholders provide input about
assumptions, scenarios?• Is the method transparent?
12Arroyo CenterRAND
Briefing Outline
1. Need for UXO risk assessment methods Prioritization Site-specific assessment
2. RAND review of existing methods
3. Design features and limitations of existing methods
4. RAND recommendations for improving UXO risk assessment
Prioritization Site-specific assessment
13Arroyo CenterRAND
Interim Range Rule Risk Methodolgy (IR3M)
Purpose:
To provide “a comprehensive process for managing, assessing, and communicating risks on former ranges”
Four tools:
1. Explosives Safety Risk Tool
2. Other Constituent Risk Tool for Human Health
3. Ecological Risk Tool
4. Comparative Analysis Tool
14Arroyo CenterRAND
Explosives Safety Risk Tool
Provides a risk value on a dimensionless scale of A to E
The A to E value is determined from decision rules that consider three input factors:
1. Accessibility (A)
2. Overall hazard (O)
3. Exposure (E)
15Arroyo CenterRAND
Explosive Safety Risk Tool Output
A
Risk Value Accessibility Overall Hazard Exposure
A < 2 < 3 < 2
B < 2 < 5 < 2
< 3 < 3 < 3
C < 4 < 3 < 4
< 3 < 5 < 3
= 5 < 3 < 2
< 2 < 3 = 5
D < 4 < 5 < 4
< 5 < 3 < 5
E < 5 < 5 < 5
O E
16Arroyo CenterRAND
RAND Evaluation of Tool
Considers main factors that determine explosion risks from UXO: hazard severity and likelihood of exposure
However, these factors are poorly quantified The risk calculation features do not satisfy the RAND
evaluation criteria: Interactions among risk contributors are poorly
modeled Output can be illogical Assumptions often are not explained Uncertainties are not described Output is not reproducible
17Arroyo CenterRAND
Hypothetical Example Illustrates IR3M Weaknesses
Conditions for hypothetical site:
• Climatic conditions are such that buried UXO is unlikely to surface
• Significant digging at the site is unexpected
• All other conditions are such that O = 5, E = 2
Two UXO response options:
• Option 1: Excavate UXO to 0.99 ft of depth
• Option 2: Excavate UXO to 1.01 ft of depth
18Arroyo CenterRAND
Calculate A for Options 1 and 2
Depth below land surface 1) All UXO > 10 feet 2) All UXO > 4 feet 3) All UXO > 2 feet 4) All UXO > 1 foot 5) Any UXO < 1 footMigration /Erosion Potential 1) Very stable 2) Minor migration 3) Moderate migration 4) Significant migration 5) Highly dynamic
Intrusion Level of Activity 1)Non-intrusive: surface only 2) Minor: to 1 ft 3) Moderate: to 2 ft 4) Significant: to 4 ft 5) Highly intrusive
Option 1: A = 5
Option 2: A = 3
A Depth Migration Intrusion
1 = 1 < 2 < 2
2 = 1 < 5 < 5
= 2 < 3 < 3
3 = 2 < 5 < 5
= 3 < 4 < 4
= 4 < 2 < 2
4 < 4 < 5 < 5
5 = 5 < 5 < 5
19Arroyo CenterRAND
IR3M Output for Options 1 and 2 Is Not Logical
A = 5
Risk Value Accessibility Overall Hazard Exposure
A < 2 < 3 < 2
B < 2 < 5 < 2
< 3 < 3 < 3
C < 4 < 3 < 4
< 3 < 5 < 3
= 5 < 3 < 2
< 2 < 3 = 5
D < 4 < 5 < 4
< 5 < 3 < 5
E < 5 < 5 < 5
O = 5 E = 2A = 3
20Arroyo CenterRAND
Participants in IR3M Test Exercise Reached Similar Conclusions
• IR3M was tested by two independent teams in a hypothetical site evaluation exercise
• The teams found problems similar to those encountered by RAND:
“Intuitively, the scores did not make sense. . . . If action removes all known UXO, the tool does not reflect a change in protectiveness and therefore there is a flaw in the tool.”
“With a 5 on Overall Hazard and Accessibility, reducing exposure does not alter the overall risk. If there is one hand grenade at land surface, the overall risk is E. Thus, there is no way to reduce risk by reducing exposure.”
21Arroyo CenterRAND
Army Science Board (1998) Also Reached Similar Conclusions
“The existing R3M model is not sufficiently grounded or developed to accompany promulgation of the Range Rule in FY99.”
“Do not highlight the existing risk formulation in working toward promulgation of the Range Rule.”
22Arroyo CenterRAND
Ordnance and Explosives Cost-Effectiveness Risk Tool (OECert)
Purpose:
“The . . . OECert . . . provides the capability to estimate risks and cost associated with given levels of remediation and to prioritize among any set of sites by life cycle cost and the amount of reduction in public risk expected to be realized from the remediation process.”
23Arroyo CenterRAND
OECert Risk Calculation
Three methods:
1. Dispersed sector method (ranges, impact areas)
2. Localized sector method (burial pits, trenches)
3. Water sector method (submerged UXO)
For all methods:
Risk = (Number of Expected UXO Exposures)
x (UXO Hazard Factor)
24Arroyo CenterRAND
Dispersed Sector Method: UXO Hazard Factor
UXO Type Sensitivity Factor
Consequence Factor
Product Adjusted Hazard Factor
UXO 126 80 10,080 29
UXO light motion sensitive
327 80 26,160 76
UXO white phosphorus
126 36 4,536 13
Controlled chemical, biological, and radiological
126 273 34,398 100
25Arroyo CenterRAND
Dispersed Sector Method: Exposure Calculation
Site is divided into homogeneous “sectors” UXO density is assumed homogeneous in each
sector Density is calculated using statistical sampling
OR based on densities of similar sites in a database of Formerly Utilized Defense Sites
Within each sector, number of exposures to UXO is calculated for 19 activities
26Arroyo CenterRAND
Dispersed Sector Method Considers 19 Activities
Activities with Risks of Surface Exposures Only
Activities with Risk of Surface and Subsurface Exposure
Hiking Fishing Child playing Crop farming
Horseback riding
Swimming Off-road driving
Archaeology
Hunting Surveying Picnicking
Mountain biking
Ranching Camping
Motor biking Jogging Metal detection
Short cuts Construction
27Arroyo CenterRAND
Example: Camping
Total number of exposures is calculated for
1. hiking to campsite
2. setting up camp
Number of expected campers is determined from
1. demographic data for surrounding area
2. generic (nation-wide) estimates of percentage of people in each age category who camp
28Arroyo CenterRAND
Exposures During Hiking to Campsite
Distance of path to campsite is estimated as radius of a circle with area equivalent to area of sector
Area traversed by single individual:
traversed area = (sector area/π)1/2 x (path width)
Number of UXO exposures for single individual:
exposures = (traversed area) x (surface density)
29Arroyo CenterRAND
Exposures At Campsite
Assumed area of campsite = 2,500 ft2
Assumed subsurface intrusive area = 0.75 ft2
Exposures within campsite:
exposures = 2,500 ft2 x (surface density) +0.75 ft2 x (subsurface density)
30Arroyo CenterRAND
Total Exposures for UXO Site
sectors activities
activity individualexposures Total N
activityin ingparticipat sindividual ofnumber
personper dencountere items UXOofnumber
activity
individual
N
31Arroyo CenterRAND
RAND Evaluation of OECert
Overall structure of modeling UXO explosion risk has merit (exposure probability times exposure consequences)
However, there are major limitations: Contains many unjustified assumptions (e.g.,
campsite size of 2,500 ft2, camper intrusion area of 0.75 ft2)
Army staff report that output is usually 100,000 exposures
Software crashed when we tried to use it Output is not meaningful outside OECert context Process is not transparent Density estimation procedures need to be improved
32Arroyo CenterRAND
Risk Assessment Code (RAC)
Purpose: Prioritize sites in the Formerly Utilized Defense Sites (FUDS) program based on UXO explosion risk
Two factors considered:
1. Hazard severity
2. Hazard probability
33Arroyo CenterRAND
Hazard Severity Score
Based on five broad categories of ordnance:
1. Conventional ordnance and ammunition
2. Pyrotechnics
3. Bulk high explosives (uncontainerized)
4. Bulk propellants
5. Chemical warfare materiel and radiological weapons
A score is developed for each category based on types of ordnance present
Total score is based on sum of scores for the 5 categories (maximum possible is 61)
34Arroyo CenterRAND
Example: Conventional Ordnance Score
Ordnance Type Score
Medium/large caliber (20 mm and larger) 10
Bombs, explosive 10
Grenades, hand or rifle, explosive 10
Landmine, explosive 10
Rockets, guided missile, explosive 10
Detonators, blasting caps, fuzes, boosters, bursters
6
Bombs, practice (w/spotting charges) 6
Grenades, practice (w/spotting charges) 4
Landmine, practice (w/spotting charges) 4
Small arms, complete round (.22 cal-.50 cal) 1
Largest single value is
chosen
35Arroyo CenterRAND
Hazard Probability Score
Based on five subscores related to probability of human exposure to UXO:1. UXO location (surface, subsurface, etc.)2. Distance to nearest inhabited location3. Number of buildings within 2-mile radius4. Types of buildings (schools, residential,
industrial, etc.)5. Site accessibility (barriers, guards, etc.)6. Site dynamics (e.g., erosion, future
development)Total score is sum of these six subscores
(maximum possible is 30)
36Arroyo CenterRAND
RAC Score Is on 1-5 Scale
Hazard Severity
Hazard Probability
Frequent (>26)
Probable (21-26)
Occasional (15-20)
Remote (8-14)
Improbable (<8)
Catastrophic (>20)
1 1 2 3 4
Critical (10-20)
1 2 3 3 5
Marginal (5-9)
2 3 4 4 5
Negligible (1-4)
3 4 4 5 5
37Arroyo CenterRAND
RAND Evaluation of RAC
Model of estimating risk based on exposure probability and explosion severity is sound
Clear, easy-to-use instructions Appropriate for intended use (priority setting) Not detailed enough for site-specific assessment Limitations
UXO density is not considered in the score Does not address uncertainty Does not address issues of stakeholder
involvement and communication
38Arroyo CenterRAND
Summary: Existing Methods Do Not Satisfy Criteria
OE-CERT
Communication features
Implementation features
Risk calculation features
OERIAIR3MRAC
= does not satisfy evaluation criteria= satisfies some criteria
= satisfies all criteria
39Arroyo CenterRAND
Briefing Outline
1. Need for UXO risk assessment methods Prioritization Site-specific assessment
2. RAND review of existing methods
3. Design features and limitations of existing methods
4. RAND recommendations for improving UXO risk assessment
Prioritization Site-specific assessment
40Arroyo CenterRAND
We Recommend a Two-Stage Process for Prioritization
Two stages: Stage 1: sort sites into explosion risk “bins” Stage 2: within bins, sort by munitions
constituent risks
Key design feature: Preserves separate information about explosion,
other constituent risks Separation is necessary; attempts to combine
these risks involve value judgments that should be explicit and transparent in decision-making
41Arroyo CenterRAND
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
No explosion risk sites
Low explosionrisk sites
Moderate explosionrisk sites
High explosionrisk sites
Determine explosion risk
Determine munitionsconstituent risk for eachsite in category
1
2
Rank Rank Rank Rank
Explosion risk None Low Moderate High
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Munitions constituent risk
Policymakers decide how to allocate resources among bins
Modified RAC?
Hazard Ranking System or Relative Risk Site Evaluation Primer
42Arroyo CenterRAND
Briefing Outline
1. Need for UXO risk assessment methods Prioritization Site-specific assessment
2. RAND review of existing methods
3. Design features and limitations of existing methods
4. RAND recommendations for improving UXO risk assessment
Prioritization Site-specific assessment
43Arroyo CenterRAND
We Searched Other Agencies for Models
Agency Model Applicability to UXO
EPA Risk Assessment Guidance for Superfund
Munitions constituent risks
NRC Probabilistic risk assessment (PRA)
Explosion risks
NASA PRA Explosion risks
FAA PRA Explosion risks
Army (Chemical Stockpile Disposal)
PRA Explosion risk
44Arroyo CenterRAND
Example: Army Uses PRA to Assess Risk of Chemical Weapons Release
Fault tree used in PRA for Tooele Chemical Agent Disposal Facility
45Arroyo CenterRAND
Potential Benefits of Using PRA for Explosion Risk Assessment
Widely and successfully used by other agencies and private industry
Could allow for meaningful quantitative evaluation of risks and benefits of response alternatives
Requires systematic analysis of elements of risk, which provides formal structure for decision-making and negotiations
46Arroyo CenterRAND
Army Science Board Made Similar Recommendation
Army Science Board reviewed IR3M in October 1998
Army Science Board recommended that the DOD and Army:
“consider the probabilistic and step (I.e., ON or OFF) nature of explosive events and consequences”
“consider using an approach analogous to the successful formulation of design codes based on the probability of structural failure”
47Arroyo CenterRAND
Summary of Recommended Approach for Site-Specific Risk
Assessment
1. Use EPA Risk Assessment Guidance for Superfund methods to assess risks of munitions constituents
2. Develop probabilistic risk assessment method specific to UXO to assess explosion risks Development should be overseen by a
technical advisory committee The method should be independently peer
reviewed Template “trees” should be developed Trees then could be modified at individual
sites, with substantial stakeholder input
48Arroyo CenterRAND
Summary
Existing methods for UXO risk assessment do not satisfy criteria for technical credibility
New prioritization method should include a two-tier screen:
Tier 1: sort by explosion risks (using new method—possibly a modified RAC)
Tier 2: sort by constituent risks (using existing method from EPA or Defense Environmental Restoration Program)
New site-specific assessment method should Use EPA’s RAGS for constituent risks Use new PRA method for explosion risks
49Arroyo CenterRAND
“Fear of harm ought to be proportional not merely to the gravity of the harm, but also to the probability of the event.”
--Antoine Arnauld, 1660