office of security assessments appraisal process …
TRANSCRIPT
OFFICE OF SECURITY ASSESSMENTS APPRAISAL PROCESS PROTOCOLS
January 2017
Office of Security Assessments Office of Cyber and Security Assessments
Office of Enterprise Assessments U.S. Department of Energy
Office of Security Assessments Appraisal Process Protocols
January 2017
OFFICE OF SECURITY ASSESSMENTS APPRAISAL PROCESS PROTOCOLS
Approved by: ______________________________________ Date: ________________
James Lund, Acting Director
Office of Security Assessments
Office of Cyber and Security Assessments
1/31/2017
Office of Security Assessments Appraisal Process Protocols Preface
January 2017 i
Preface
The U.S. Department of Energy (DOE) Office of Security Assessments (EA-22), within the Office of
Enterprise Assessments (EA) Office of Cyber and Security Assessments, is responsible for conducting
independent oversight appraisals of safeguards and security (S&S) programs at DOE sites and other
organizations. These assessments focus on high consequence activities, such as high hazard nuclear
operations, and the protection of high value security assets, such as Category I quantities of special nuclear
material and classified information assets, as mandated by DOE Orders 227.1A, Independent Oversight
Program, and 226.1B, Implementation of Department of Energy Oversight Policy.
The EA-22 Appraisal Process Protocols, in conjunction with the Independent Oversight Program Appraisal
Process Protocols issued by EA in December 2015, describes the principal activities for implementing the
DOE independent oversight program. This document also describes the processes, techniques, and
procedures that EA-22 uses to evaluate DOE (including National Nuclear Security Administration) and
contractor organizations’ programs (referred to as “sites” throughout the document) designed to protect
national security assets, including special nuclear material and sensitive and classified information. EA-22
accomplishes these evaluations through rigorous and objective appraisals of the effectiveness of DOE
security policies and the performance of DOE line management in implementing those policies. EA-22
assessments are also key in identifying emerging trends throughout DOE.
Major additions in this revision of EA-22 Appraisal Process Protocols include referencing the limited-notice
testing program, processes for sharing lessons learned from appraisal activities, and greater focus on the
insider threat. Limited-notice performance testing focuses on specific S&S elements, in contrast to multi-
topic assessments that provide a more complete evaluation of overall performance. A goal of this approach
is to gather data with minimal advance notice to the personnel and facilities being tested, in order to
provide a realistic appraisal under actual operating conditions. EA-22 has also initiated mechanisms for
sharing lessons learned with line security managers and the National Training Center to enhance local
programs and S&S training for all of DOE. EA-22’s approach for evaluating the insider threat uses
enhanced lines of inquiry to determine the effectiveness of site administrative and physical security
measures.
This protocol document is part of an ongoing effort to ensure quality and consistency in DOE independent
oversight activities while providing the flexibility to address all activities being evaluated. To ensure that
this protocol remains current, all users of this protocol are encouraged to provide comments and
recommendations to EA-22 for consideration.
Office of Security Assessments Table of Contents Appraisal Process Protocols
ii January 2017
Table of Contents Acronyms ..................................................................................................................................................... iii
Definitions ................................................................................................................................................... iv
Section 1. Introduction ................................................................................................................................. 1
Mission ............................................................................................................................................ 1
Roles and Responsibilities ............................................................................................................... 1
Appraisal Types ............................................................................................................................... 3
Augmentation and Observer Program ............................................................................................. 4
Section 2. Appraisal Process ........................................................................................................................ 6
2.1 Appraisal Planning .................................................................................................................... 6
Appraisal Scheduling Process .......................................................................................................... 6
Appraisal Scoping ............................................................................................................................ 6
Announcement Memorandum ......................................................................................................... 7
Logistics Planning............................................................................................................................ 8
Site Scoping Visit ............................................................................................................................ 8
Performance Test Scoping Visit ...................................................................................................... 8
Site Cooperation and Coordination .................................................................................................. 8
Assessment Plan Development ........................................................................................................ 9
Planning Meetings ........................................................................................................................... 9
2.2 Appraisal Conduct .................................................................................................................... 9
Data Collection ................................................................................................................................ 9
Validation ........................................................................................................................................ 9
Data Analysis ................................................................................................................................. 10
End-of-Day Team Meetings and Morning Manager’s Meetings ................................................... 10
Report Writing ............................................................................................................................... 10
Exit Meeting .................................................................................................................................. 11
2.3 Post-Appraisal Activities ........................................................................................................ 12
Post-Appraisal Validation Review ................................................................................................. 12
Comment Resolution and Technical Edit ...................................................................................... 12
Final Report ................................................................................................................................... 12
Briefings ........................................................................................................................................ 12
Sharing Lessons Learned ............................................................................................................... 13
Appendix A: Multi-Topic Assessment Timeline ..................................................................................... A-1
Appendix B: Example Document Request .............................................................................................. B-1
Appendix C: Example of Multi-Topic Assessment Plan ......................................................................... C-1
Appendix D: Appraisal Scheduling Process ............................................................................................ D-1
Appendix E: Factors for Determining a Site’s Appraisal Priority, Type, and Scope .............................. E-1
Appendix F: Data Analysis, Deficiencies, Findings, Opportunities for Improvement,
Recommendations, Best Practices, and Ratings................................................................. F-1
Office of Security Assessments Appraisal Process Protocols Acronyms
January 2017 iii
Acronyms
CAS Central Alarm Station
CFR Code of Federal Regulations
CMPC Classified Matter Protection and Control
COMPASS Comprehensive Analyses of Safeguards Strategies
DOE U.S. Department of Energy
EA Office of Enterprise Assessments
EA-22 Office of Security Assessments
FCL Facility Clearance
FOCI Foreign Ownership, Control or Influence
FOF Force-on-Force
FY Fiscal Year
HRP Human Reliability Program
IOSC Incidents of Security Concern
LNPT Limited-Notice Performance Test
LSPT Limited-Scope Performance Test
MAA Material Access Area
MBA Material Balance Area
MC&A Material Control and Accountability
MOA Memorandum of Agreement
MOU Memorandum of Understanding
NNSA National Nuclear Security Administration
NTC National Training Center
OPSEC Operations Security
PAP Performance Assurance Program
POC Point of Contact
PPM Protection Program Management
PSS Physical Security Systems
QRB Quality Review Board
S&S Safeguards and Security
SAS Secondary Alarm Station
SCI Sensitive Compartmented Information
SECON Security Conditions
SNM Special Nuclear Material
SPO Security Police Officer
SSP Site Security Plan
TAP Training Approval Program
TID Tamper-Indicating Device
TSCM Technical Surveillance Countermeasures
UFV&A Unclassified Foreign Visits and Assignments
VA Vulnerability Assessment
VAR Vulnerability Assessment Report
VTR Vault-Type Room
Office of Security Assessments Definitions Appraisal Process Protocols
iv January 2017
Definitions
Deficiency: An inadequacy in the implementation of an applicable requirement or performance standard
identified during an appraisal. Deficiencies may serve as the basis for one or more findings. [DOE Order
227.1A]
DOE Oversight: DOE oversight encompasses activities performed by DOE organizations to determine
whether Federal and contractor programs and management systems, including assurance and oversight
systems, are performing effectively and complying with DOE requirements. Oversight programs include
operational awareness activities, onsite reviews, assessments, self-assessments, performance evaluations,
and other activities that involve evaluation of Federal and contractor organizations that manage or operate
DOE sites, facilities, or operations.
Evaluators: Individuals assigned the responsibility of formally assessing the readiness and performance
effectiveness of safeguards and security (S&S) system elements during the conduct of a performance test
as part of an independent oversight appraisal activity. Evaluators must be trained to perform their assigned
duties.
Limited-Notice Performance Test (LNPT): A performance test that is coordinated and scheduled with
one or more site trusted agents but is conducted without prior announcement to other site personnel.
Limited-notice tests are intended to elicit the most accurate information regarding an individual’s
knowledge or the performance of S&S programs.
Limited-Scope Performance Test (LSPT): A performance test designed to validate or evaluate specific
skills, equipment, or procedures instead of testing an entire system or multiple subsystems. An LSPT may
be interrupted to facilitate data collection, and the LSPT may be revised or restarted in order to achieve
specific goals.
Trusted Agents: Individuals who have appropriate operational authority or a compartmented role
necessary to provide administrative and logistical support for coordinating and conducting independent
oversight and LNPT activities. Trusted agents are responsible for maintaining strict confidentiality of
performance testing information in the interest of test validity. Trusted agents must remain impartial in
validating and developing performance test parameters and events necessary to evaluate identified
objectives. The utmost diligence must be applied to limit the number of trusted agents to the minimum
needed to administratively and logistically conduct the test.
Office of Security Assessments Appraisal Process Protocols Introduction
January 2017 1
Section 1 – Introduction
Mission
The mission of the U.S. Department of Energy (DOE) Office of Security Assessments (EA-22), within the
Office of Enterprise Assessments (EA), is to provide feedback to internal and external stakeholders by
conducting independent evaluations of the effectiveness of safeguards and security (S&S) policies and
programs throughout DOE. The major intention of these assessments is to foster continuous improvement
in the Department’s security program by identifying deficiencies, positive attributes, and unacknowledged
program risks and communicating them effectively to DOE management. To perform this mission, EA-22
plans and conducts a variety of announced and unannounced appraisal activities and performance testing
that incorporate a broad range of threats and scenarios in order to provide a complete and realistic evaluation
of site S&S system’s readiness to protect DOE assets, with an emphasis on protection of Category I special
nuclear material (SNM) and highly sensitive or classified information. EA-22 develops and validates
appraisal results in reports that may identify best practices, findings, deficiencies, and opportunities for
improvement. EA-22 also performs follow-up assessments to ensure that site-specific corrective actions
are effective and conducts complex-wide reviews of S&S programs to enhance the overall effectiveness
and efficiency of these programs.
Roles and Responsibilities
EA-22 Director (or Deputy Director)
In addition to the roles and responsibilities set forth in the EA Independent Oversight Program Appraisal
Process Protocols, the EA-22 Director:
Provides overall direction and manages the S&S independent oversight program.
Directs self-assessment program/activities.
Ensures that upcoming assessment activities are coordinated with line management and
stakeholders in a collaborative manner, well in advance of execution.
Briefs senior stakeholders, including the Under Secretaries, Secretarial Officers, the EA Director,
and DOE policy organizations, on the results of appraisal activities.
Ensures that appraisal feedback sessions are conducted with line management as appropriate.
Notifies senior EA management and the DOE Office of the Inspector General when appraisal
activities identify concerns that may have criminal or waste/fraud/abuse implications.
Develops and maintains assessment guides for conducting S&S appraisals.
Works with cognizant DOE line managers to resolve disagreements on appraisal schedules, results,
findings, or ratings, and works with cognizant policy organizations to ensure proper application
and characterization of DOE policies and directives in appraisal reports.
Ensures that the quality of independent oversight activities and reports is maintained.
Serves as appraisal leader for assessments as needed or required.
Coordinates appraisal team structure, designates appraisal Topic Team Leaders, and works with
stakeholders to define assessment scope.
Serves on the Quality Review Board (QRB).
Office of Security Assessments Introduction Appraisal Process Protocols
2 January 2017
Team Chief or Deputy Team Chief
Plans and leads assessments of information/physical security programs or topics as directed.
Provides input on recommended assessment scope.
Provides direction and guidance to team members on the approach to specific data collection
activities.
Oversees the development of assessment plans.
Provides feedback on proposed assessment team structure and makes recommendations for allocation of resources needed to accomplish the scope.
Makes arrangements with the site for document requests and other logistics as needed.
Communicates upcoming assessment activities to appropriate field and program office
management well in advance of the scheduled activity.
Seeks feedback on special interest items, implementation, and scope.
Establishes the schedule of events for security assessments and makes specific assignments.
Ensures that team members perform assigned data collection and performance testing duties in
accordance with DOE Order 227.1A, Independent Oversight Program, relevant DOE policy, and
EA protocols.
Addresses site concerns associated with data collection activities.
Provides daily feedback to site personnel to validate assessment information and clearly
communicate areas of concern.
Oversees the development of draft assessment reports.
Briefs site Federal and contractor management and security personnel on assessment results and
provides a written synopsis (letter form) of assessment results to site management prior to departure
if a draft report has not been developed.
If a significant vulnerability is identified, notifies EA management and the site Federal S&S
Director in accordance with DOE Order 227.1A, Independent Oversight Program.
Topic Team Leaders
Support Team Chief/Deputy in the assessment of physical/information security programs.
Lead specific topic teams during appraisal activities as needed or required.
Provide direction and guidance to topic team members on the approach to physical/information
security data collection activities, including performance testing.
Update and maintain assessment guide(s) for assigned topic(s).
Maintain current knowledge of applicable assigned S&S topic requirements and conduct training
with team members as required.
Provide input to the Team Chief/Deputy on topic area document requests (see Appendix B for a
baseline document request list), lines of inquiry (see Appendix C, Example of Multi-Topic
Assessment Plan, for generally used lines of inquiry), and other necessary logistics to support the
team.
Provide feedback on proposed topic team structure and make recommendations for allocation of resources needed to accomplish the scope.
Establish a topic team assessment schedule and make specific data collection assignments.
Ensure that team members perform their assigned duties.
Address site concerns associated with data collection or performance testing activities.
Provide daily feedback to site personnel to validate assessment information and clearly
communicate areas of concern.
Participate in briefing site management and security personnel on data collection results, as required.
Coordinate the preparation of topic input to the draft assessment report.
Office of Security Assessments Appraisal Process Protocols Introduction
January 2017 3
Work with the Team Chief/Deputy to resolve site comments on the appraisal report.
Team Members
Support the Team Chief/Deputy and applicable Topic Team Leader in conducting appraisals of
security programs.
Provide input to the Topic Team Leader on topic scope and potential approaches for accomplishing
security appraisals.
Conduct appraisal process activities following the direction and guidance of Topic Team Leaders.
Assist in preparing the schedule of interviews to accomplish during the onsite visit.
Review key site security documents before the onsite visit.
Conduct thorough and fair evaluations and assessments in accordance with the assessment plan.
Validate assessment data and conclusions with site personnel on a daily basis to ensure factual
accuracy.
Participate in briefing site management and information/physical security personnel on assessment
results, if requested.
Provide written input for topical sections of draft reports as directed by the Topic Team Leader.
Work with the Topic Team Leader to resolve site comments on the draft report.
Appraisal Types
EA-22 conducts a broad array of appraisals, including:
Multi-Topic Assessments: Multi-topic assessments evaluate the security functional areas related
to the protection of SNM or classified and sensitive unclassified information, and site actions for
deterring, detecting, and mitigating the insider threat. Multi-topic assessments typically require the
largest amount of resources, as well as time on site with two weeks of data collection activities and
another week for developing a draft report. A key aspect of EA-22 assessments is the conduct of
a wide range of performance tests, including large-scale force-on-force (FOF) performance tests
consistent with the Department’s threat policy (i.e., DOE Order 470.3B, Graded Security
Protection Policy, and DOE Order 470.3C, Design Basis Threat Policy, as applicable) and limited-
notice performance tests (LNPTs) and/or limited-scope performance tests (LSPTs). See Appendix
C for an example of a Multi-Topic Assessment Plan for more details.
Targeted Assessments: Targeted assessments are conducted to address S&S concerns that
transcend performance at a specific site or location. They might address the effectiveness of
program elements as implemented across DOE by analyzing DOE complex-wide program issues,
or they might analyze the implementation of a specific policy item throughout the complex.
Targeted assessments are also performed to address an area, concern, deficiency, or weakness
within a program, focusing (for example) on the status of a specific program element, the adequacy
of specific policies, or the implementation status of specific policies throughout DOE.
Follow-up Assessments: EA conducts follow-up assessments to determine the status and progress
of corrective actions and other actions taken in response to deficiencies identified during previous
EA appraisals or DOE line management oversight activities, focusing on evaluating the
effectiveness and sustainability of corrective actions. Follow-up assessments can include
announced and limited-notice performance testing and FOF performance tests. The scope and team
size for these assessments can vary, and data collection activities are tailored to assess the overall
effectiveness of corrective actions.
Office of Security Assessments Introduction Appraisal Process Protocols
4 January 2017
Limited-Notice Performance Testing: EA-22 uses limited-notice testing and conducts
evaluations outside of pre-scheduled appraisal activities to enhance DOE senior leadership’s and
line managers’ understanding of protection program readiness. LNPT activities related to physical
and information security are necessarily narrow in scope and limited in team size. Accordingly,
EA-22 maintains a set of physical and information security LNPT objectives and associated tasks,
conditions, standards, and evaluation criteria. The EA Limited-Notice Performance Test Guide
provides specific guidance for physical security LNPTs.
Protective Force Assessments: These assessments focus primarily on the implementation of the
DOE requirements for armed protective forces at Departmental facilities that do not possess SNM.
These reviews may also include specific aspects of other topical areas, particularly protection
program management and physical security systems, as deemed necessary. Data collection
activities are tailored to the assessment’s scope, including document reviews, interviews, and
LSPTs. The assessment teams generally consist of a small number of subject matter experts who
conduct onsite activities over the course of several days. Assessment planning, execution, and
post-appraisal activities are generally conducted in accordance with the processes described in
Section 2.
Augmentation and Observer Program
EA-22 recognizes the importance of working closely with Headquarters program offices, field offices, and
DOE and National Nuclear Security Administration (NNSA) physical and information security
organizations to ensure that appraisal results are clearly communicated and identified deficiencies are
adequately addressed. In addition to pre-coordination, EA has implemented an augmentation and observer
program that includes DOE Federal or contractor subject matter experts as augmentees or observers on
assessment and review teams.
The augmentee program allows the participants to observe the inner workings of the appraisal process and
return to their home organizations with information on oversight and protection program best practices.
The augmentee is considered an assessor and member of the EA-22 appraisal team. The field augmentee
program has proven to be mutually beneficial, developing a high level of trust between EA-22 and
participating sites and fostering improved relations during future appraisals.
The observer program offers benefits similar to the augmentee program; however, the observer is not
involved in data collection activities and is not considered an assessor.
Requesting organizations must follow these general program concepts to ensure the integrity of EA-22’s
appraisal process:
The DOE/NNSA augmentee is recommended in writing (emails are acceptable) by the applicable
Headquarters or field office and is selected and approved for participation by the EA-22 Office
Director or Deputy Director. Recommendations must come from the senior Federal S&S manager
and must include the specific objective and overall intention of the augmentee’s participation.
o Augmentees cannot participate in appraisals at their own site or organization; contractor
augmentees are further restricted from participating in appraisals at other sites operated by
their employer.
Office of Security Assessments Appraisal Process Protocols Introduction
January 2017 5
o Augmentees are fully integrated into the appraisal team and participate in the data
collection activities of the topic team to which they are assigned.
DOE and other government agency observers are recommended in writing (emails are acceptable)
by the applicable Federal agency manager and then are approved by EA-22 Office Director or
Deputy Director. Recommendations must come from the senior Federal S&S manager and must
include the specific objective and overall intention of the observer’s participation.
Observers are assigned to one or more topic teams during an appraisal activity but do not conduct
data collection activities.
Office of Security Assessments Appraisal Process Appraisal Process Protocols
6 January 2017
Section 2 – Appraisal Process
Each EA-22 assessment is a continuous process involving a myriad of activities at the strategic, program,
and individual appraisal level. Appraisal activities focus on identifying and executing activities that
collectively address the most significant security vulnerabilities faced by the Department, in order to
provide recommended actions to the Secretary and other senior managers for their use in allocating
resources and targeting remedial actions in a timely manner.
The appraisal process consists of pre-appraisal activities (planning, including scoping visits), execution
(conducting and documenting the assessment), and post-appraisal activities (closeout). This section of the
protocol describes these activities for multi-topic assessments in a normal sequence and an approximate
timeline within which they are completed. It should be noted that the other appraisal types (targeted, follow-
up, and protective force assessments and LNPTs) generally mirror this process. For some assessment
activities, EA management may determine that ratings will be assigned. In such cases, the appraisal team
is responsible for recommending ratings, and the EA Director approves the final ratings, with input from
the director of EA-22 (see EA Independent Oversight Program Appraisal Process Protocols for additional
information).
From the start of the scoping to the closeout and final publication of the appraisal report, the typical multi-
topic assessment is a 30-week process. The actual amount of onsite planning and data collection varies
with the type of appraisal being conducted; however, for multi-topic assessments, major onsite assessment
activities occur in two visits constituting three weeks (supplemented with short-duration scoping and
planning visits by a small number of planners and managers to ensure efficient and effective planning of
major assessment activities). The appraisal planning, conduct, and follow-on activities are described in
Sections 2.1, 2.2, and 2.3, respectively. A detailed timeline diagram indicating the sequence and timing of
these activities for a typical multi-topic assessment is provided in Appendix A. Appendices B through F
provide supplemental detail for selected phases of the process.
2.1 Appraisal Planning
Appraisal Scheduling Process
EA-22 develops its annual schedule of appraisals by using the overall process described in Appendix D of
this protocol. Throughout this process, EA-22 engages Headquarters program office and field management
and security officials. Although this process results in a formal schedule for the year, ongoing interaction
between EA-22 and DOE/NNSA line managers often results in modifications to either the scope or the
schedule of the appraisals. While these modifications can be challenging for EA-22 managers and staff,
this flexibility allows for optimization of appraisal scope and timing and, ultimately, the appraisal’s value
to line managers.
Appraisal Scoping
EA-22 determines the appraisal type best suited to accomplish line management and EA’s objectives, as
well as the scope of the appraisal activity (topics evaluated, level of detail, and lines of inquiry), by
coordinating with senior DOE/NNSA leadership and line management. EA-22 uses the data gathered
during the scoping process to begin developing a detailed appraisal plan. The list below shows some of the
factors that EA-22 considers when making these determinations (see Appendix E, Table E-1, Factors for
Determining a Site’s Appraisal Priority, Type, and Scope, for additional information on how these factors
relate to a site’s priority for evaluation). This risk-based assessment scoping methodology provides a
Office of Security Assessments Appraisal Process Protocols Appraisal Process
January 2017 7
mechanism for determining the emphasis to be placed on particular requirements and assets during EA-22
appraisal planning, data collection, analysis, and reporting.
Asset characterization and site mission: The types and attractiveness of security interests located
at a DOE site and the associated risks (or perceived risks) are significant considerations in
determining the scope and type of appraisal activity to be conducted.
Changes in site mission and operations: Significant changes in site mission/operations, site
geography, or the site security program’s contract structure are factors that EA-22 evaluates in the
appraisal scoping process.
Changes in policy: DOE policy and other applicable national standards are the fundamental
measure by which EA-22 conducts its appraisal activities. Changes in policy can significantly
impact site operations and thus are important considerations in determining the appraisal scope.
Previous evaluation results: Previous EA assessment results and regulatory enforcement actions,
DOE Inspector General inspections, site/operations office surveys, and contractor assurance
activities, as well as evaluation results from external agencies, provide valuable insights that
EA-22 considers during the appraisal scoping process.
Incidents of security concern: Recent/ongoing major vulnerabilities, significant incidents, or
chronic recurrence of lesser incidents influence EA-22’s scoping and prioritization process.
Line management input: EA-22 is committed to supporting Headquarters and field line managers
in accomplishing their S&S responsibilities and achieving DOE mission objectives. Therefore,
input regarding requested focus areas from Headquarters and field line management is an especially
important factor in determining appraisal priority, type, and scope.
Announcement Memorandum
Once EA senior management has approved the annual schedule and EA-22 has determined the scope of an
appraisal activity, an announcement memorandum is issued to the affected site approximately 90 days
before the start of the appraisal. The memorandum provides the dates, scope, and any special considerations
for the appraisal. EA-22 ensures that key program office officials receive a copy of the memorandum.
Separately, the EA-22 appraisal Team Chief communicates a data call request to the designated site point
of contact (POC) to ensure that the site has ample time to compile and provide the requested materials. The
data call request includes a detailed listing of documents, drawings, diagrams, and other materials being
requested to support planning and conduct of the appraisal activity. Appendix B provides examples of data
call document requests for each topical area.
Data call items are typically of two types: (1) the initial materials for EA-22’s review during the planning
process, and (2) other materials that should be available for onsite review. The assessment team reviews
the initial data call items during the Headquarters planning meeting before the team travels to the site. The
results of these initial document reviews may lead one or more teams to request additional data and
documentation from the site. The site sends the initial data call items to the EA-22 office in Germantown,
Maryland. Approximately four weeks before the onsite FOF performance testing begins, EA sends these
materials back to the site.
Office of Security Assessments Appraisal Process Appraisal Process Protocols
8 January 2017
Logistics Planning
The EA administrative staff (normally two individuals) is instrumental in planning for and executing
various tasks associated with onsite appraisal activities. Activities include identifying team workspace(s);
coordinating team lodging and issuing the appraisal team lodging announcement memo; arranging with the
site to provide the requisite team access, classified storage containers, copiers, and shredders; coordinating
EA computer support; and communicating any additional support requirements to the site.
Site Scoping Visit
EA-22 contacts site managers several times early in planning and conducts a one-day site scoping visit so
the Team Chief and Deputy, along with members of the administrative staff, can meet face to face with
field DOE/NNSA line management and S&S personnel to discuss the upcoming appraisal, including the
appraisal process and the tentative scope and focus of the appraisal. Topic Team Leaders may also
participate in these visits. The site scoping visit helps clarify the details of data collection and performance
testing activities, the site’s safety and security training requirements, and workspace availability.
Performance Test Scoping Visit
As the start of the onsite appraisal activity approaches, a performance test scoping visit brings together the
EA performance test director and coordinator for large-scale performance FOF tests and supporting LSPTs
and the site’s trusted agents and senior controller. The objectives of the visit are to identify valid and
realistic scenarios/tests and establish performance criteria. These objectives also necessitate interaction and
discussion of targets with site vulnerability assessment staff and local Federal line management officials
and staff.
Site Cooperation and Coordination
The cooperation and assistance of DOE/NNSA site representatives are essential for an effective appraisal.
Local representatives and trusted agents provide detailed knowledge of the site and systems, arrange
administrative and logistical support, expedite appraisal activities, and provide valuable feedback on factual
accuracy.
Relations between the appraisal team and representatives from the assessed organization must be cordial,
open, and professional. It is in the interest of both EA-22 and the local representatives to approach security
appraisals in partnership to ensure that these activities lead to potentially useful and effective program
improvements. Positive relationships are especially important in performance testing, where the trusted
agents are relied on to maximize realism while maintaining the confidentiality of the scenario or test content
and the timing of the scheduled and limited-notice tests. EA keeps the number of trusted agents as low as
possible and marks all performance test documents (including knowledge tests) with a header or disclaimer
that reads: “Trusted Agent Use Only, Not to be Disseminated.” All trusted agents also sign EA’s
Memorandum of Understanding and Agreement Regarding Trusted Agent Duties and Responsibilities form
before being briefed on sensitive test information. Finally, EA shares performance test materials with
trusted agents only in person or, when necessary, by encrypted emails.
Office of Security Assessments Appraisal Process Protocols Appraisal Process
January 2017 9
Assessment Plan Development
For each appraisal, EA-22 develops an assessment plan that includes scope, schedule, team responsibilities
and assignments, and process. The assessment plan also provides a topic schedule of appraisal activities
(data collection, performance testing, analysis of results, and report development), team composition, and
lines of inquiry. See Appendix C, Example of a Multi-Topic Assessment Plan, for more detail. Lines of
inquiry are derived from policy requirements (must, shall, will), with a focus on elements identified during
the scoping process. The lines of inquiry help topic teams better understand how the site’s protection
programs implement S&S policy, site procedures, and Federal regulations.
Planning Meetings
Two separate but concurrent planning meetings take place during pre-appraisal activities. One brings the
topic teams together, usually at Headquarters but sometimes at a field site, to review the data call, begin
identifying potential topic strengths and weaknesses, validate the topic scope, select and schedule data
collection methods, and share information with Office of the Associate Under Secretary for Environment,
Health, Safety and Security (on potential policy concerns) and the National Training Center (NTC) (onsite
training history and concerns). Topic teams often identify additional data call and support requirements
during this meeting and communicate these to their site POCs. The protective force topic team is most
likely to conduct part of its planning meeting at the site due to the complexity of data collection activities
in this topical area and the benefits gained from being on site. In exceptional cases, site, program office,
and EA senior managers may approve some limited onsite data collection activities during the planning
meeting.
The other planning meeting, conducted on site by EA FOF test planners, focuses on meeting with trusted
agents, adjusting test scenarios, receiving briefings, conducting safety walkdowns, and developing safety
risk assessments. The other major meeting objectives are determining the rules of engagement, identifying
“out-of-plan” areas, finalizing the test schedule and scenarios, and obtaining trusted agent concurrence on
test plans. Toward the end of the meeting, the EA performance test director contacts the Composite
Adversary Team Coordinator to discuss the number of adversary team members required for the test and
the Engagement Simulation Systems Support Coordinator to determine the types and quantities of
equipment needed for the test.
2.2 Appraisal Conduct
Data Collection
The topic teams collect data while on site, using techniques that typically include document reviews, tours
and observations, interviews, knowledge tests, and performance testing. All data collection techniques
exhibit elements of compliance and performance, and the full spectrum of techniques is necessary for an
effective appraisal. The goal in appraisal conduct is to accomplish all planned data collection activities
fairly, impartially, and professionally and to validate the technical accuracy of the collected data. The
Independent Oversight Program Appraisal Process Protocols provide additional information on data
collection techniques.
Validation
EA uses a continuous validation process to ensure the factual accuracy of the collected data and to
communicate any identified deficiencies, and their impacts, to the responsible managers and organizations.
Office of Security Assessments Appraisal Process Appraisal Process Protocols
10 January 2017
Topic teams validate data collection results (favorable and unfavorable) as soon as they occur, at the end
of the day or week, and near the end of appraisal execution. This continuous process verifies that the
collected data is factually correct and can legitimately be used to evaluate the effectiveness of the program,
project, or activity. It also provides an opportunity for the assessed site to provide additional data that may
mitigate any emerging concerns. The Team Chief and Deputy Team Chief validate the previous day’s
information in morning meetings with field Federal and site S&S contractor managers, described further
below.
Data Analysis
Although EA-22 actually starts analyzing data before the onsite visit begins, the extensive data collected
on site allows a greater focus on emerging performance and compliance trends. Data analysis can also
generate additional data collection activities and validation discussions with site security officials. The
sooner data analysis can be validated with the site, the better.
Topic teams use a team approach vetting each strength and weakness. Appendix F provides a tool to help
topic teams determine whether a weakness is best characterized as a deficiency or a finding.
End-of-Day Team Meetings and Morning Manager’s Meetings
End-of-day team meetings are an opportunity to communicate and integrate topic team data collection
results and to highlight key points and/or potential concerns for the appraisal team management’s
awareness. Each topic team prepares and briefs a list of narrative bullets summarizing data collection
activities and results for the day. After the conclusion of the end of day meeting, senior EA managers
and staff are provided the final version of the bullets. When briefing team management, Topic Team
Leaders indicate which of their topic’s summary bullets were validated with the site. This critical action
ensures that the Team Chief is aware of validated information during the next day’s morning manager’s
meeting. The Team Chief and Deputy Team Chief discuss the previous day’s information in morning
meetings with field Federal and site S&S contractor managers.
The primary purposes of the morning manager’s meeting are to openly communicate the progress of
appraisal activities and potential concerns, and to verify that the appraisal team’s information and the site’s
information regarding key data collection results are consistent. Another important objective of this
meeting is to discuss the day’s data collection activities and follow-on meetings. These meetings ensure
that site POCs and their managers are aware of the data that has been collected; the intention is for the site
to acknowledge factual accuracy, provide corrected information, request further data collection, or provide
mitigating information.
Report Writing
In the report writing phase, the topic teams thoroughly analyze the data, develop conclusions, and, based
on the analysis and conclusions, prepare a report that accurately reflects the status of the program, program
element, facility, or activity that was examined. For larger multi-topic assessments, report writing typically
begins mid-week of the second onsite data collection trip and continues through the remainder of the
appraisal.
Appraisal reports include a one-page summary, executive summary, introduction, and topical appendices.
The principal writer is responsible for the one-page summary, executive summary, and introduction. The
one-page summary is of special importance because it communicates the overall results of the appraisal to
Office of Security Assessments Appraisal Process Protocols Appraisal Process
January 2017 11
the Secretary, other senior DOE leaders, and other interested external stakeholders, such as Congress. To
facilitate communication of the results, the one-page summary typically contains only unclassified or
controlled unclassified information.
The topic team writers support the Topic Team Leader and are responsible for drafting their topic’s
appendix. Each topical appendix consists of an introduction; the identified results, best practices, findings,
and deficiencies (when applicable); conclusions; and opportunities for improvement. Within topical
appendices, the topic team must exert care to communicate all relevant results and the status of program
effectiveness, all in support of the conclusions. A key report element is identifying both best practices and
topic weaknesses and determining whether the weaknesses are deficiencies or findings (warranting high-
level management attention), as described in Appendix F. EA-22 normally assigns a finding for significant,
systemic, or recurring/chronic deficiencies that have a demonstrated or highly likely negative impact on the
protection of high consequence security assets. The final section of an appendix is opportunities for
improvement, which informs the assessed site of possible ways to improve program performance in areas
where findings have been identified. Reports may also provide recommendations (focused on management
systems rather than a specific deficiency) and best practices observed during an assessment that merit
consideration by other DOE/NNSA and contractor organizations.
The EA process ensures that all report elements are thoroughly reviewed before transmittal of the draft
report to site personnel. Topic team reviews are the first step in this phase of developing the report. After
topic teams have generated a draft document, the Management Review Board (consisting of the Team
Chief, Deputy Team Chief, and technical advisor, and other EA-22 managers as desired) reviews it to ensure
that all appendices are consistent with each other and that each appendix is readable, logical, and adequate
to support the conclusions. Subsequently, the QRB reviews all elements of the report. The QRB, composed
of senior EA managers and senior personnel (usually former senior DOE managers) who support EA,
provides comments that result in appendices undergoing a second review to ensure the report meets EA’s
high quality standards.
NTC representatives also participate as observers in QRB deliberations to identify findings and
recommendations that may have training implications for the NTC. During this phase of an appraisal, NTC
representatives note QRB participants’ observations, findings, and opportunities for improvement involving
training concerns, and also meet informally with team members to gain added insights into potential training
concerns.
Following the QRB review, the EA-22 Team Chief schedules a factual accuracy review with the site’s
security management team to help identify and resolve potential concerns about the draft report. These
reviews typically last no more than four hours and generate initial site comments that may be accepted,
accepted with modification, or not accepted by EA-22 management. Site management is informed of the
resolution of the comments before the closeout briefing (described below). The objective of the factual
accuracy review and comment period is to ensure that EA’s draft report is factually correct and presents
an accurate evaluation of the effectiveness of the facility’s protection program.
Exit Meeting (on the last day of onsite assessment activity)
The final step is the onsite exit meeting, which includes a closeout briefing. The EA-22 Team Chief
typically conducts this briefing, using presentation slides, to convey the team’s observations and
preliminary results to the field Federal and contractor managers. Topic Team Leaders and team members
attend the closeout briefing as needed or requested. At the end of the closeout briefing, EA-22 formally
provides a copy of the draft report to the responsible DOE/NNSA field office manager to begin the ten-day
factual accuracy review (see Section 2.3). Either before or after the closeout briefing, the EA-22 Director
Office of Security Assessments Appraisal Process Appraisal Process Protocols
12 January 2017
or Deputy Director, or a senior EA manager not assigned to the assessment team, meets with senior site
line managers and security officials to obtain their feedback on the appraisal process.
2.3 Post-Appraisal Activities Post-Appraisal Validation Review
EA-22 formally provides a copy of the draft report to the applicable Headquarters program office
representative immediately after the assessment team returns to Germantown Headquarters. The site and
program office have ten calendar days (beginning on the day of the onsite closeout briefing) to complete
their post-appraisal review of the report and provide factual accuracy comments to EA-22; EA-22 asks the
program office to consolidate all comments before forwarding them. EA-22 then reviews each comment
and develops a response. EA-22 typically accepts the comments as provided, accepts them with
modification, or declines to accept them due to technical accuracy. EA provides an explanation for any
comment that is not accepted and coordinates additional meetings and/or teleconferences as required to
complete resolution of the comments. EA also reviews the comment resolution comments with the program
office, either concurrently with or separately from site line management. If disagreements on comment
resolution between the EA Office Director and the applicable Headquarters program office representative
persist, they proceed to higher management levels, up to and including the Deputy Secretary, for resolution.
For multi-topic assessments, the cognizant Program Secretarial Officer has the opportunity to submit a
written management response to the draft report’s conclusions and recommendations. If a Secretarial
Officer provides a response, EA reflects this information in an appendix to the report.
Comment Resolution and Technical Edit
The Team Chief and Deputy Team Chief, working with Topic Team Leaders, resolve all program office
comments, incorporate needed changes into the draft report, and provide a summary of the rationale for any
rejected comment to the program office. The draft report then undergoes a technical edit, typically via the
classified network before becoming final.
Final Report
Within 60 days after the exit meeting, the report is submitted to the EA Office Director and undergoes a
final review before being released to the Office of the Secretary for approval. The approval process includes
briefing the Secretary’s staff via a one-page summary of the report. Once approved, final reports are
uploaded into the Safeguards and Security Information Management System.
Briefings
EA may conduct briefings for key Headquarters managers before or after the final report is approved.
External briefings to key stakeholders, such as Congressional staff and the Defense Nuclear Facilities Safety
Board, are conducted, if appropriate, only after issuance of the final report and completion of all internal
DOE briefings.
Office of Security Assessments Appraisal Process Protocols Appraisal Process
January 2017 13
Sharing Lessons Learned
EA-22 conducts quarterly teleconferences with Headquarters and field security officials to share lessons
learned from recent appraisals. Lessons learned are also routinely briefed at meetings of senior Federal and
contractor security directors and during various DOE working group meetings, and EA-22 prepares an
annual report identifying S&S trends and lessons learned. The NTC and the Office of the Associate Under
Secretary for Environment, Health, Safety and Security regularly receive information on lessons learned
from EA-22 as well.
Office of Security Assessments Appendix A Appraisal Process Protocols
A-2 January 2017
CA
T =
Com
posi
te A
dver
sary
Tea
m
C-L
AN
= C
lass
ifie
d L
oca
l A
rea
Net
work
E
A =
Off
ice
of
En
terp
rise
Ass
essm
ents
E
A-2
2 =
Off
ice
of
Sec
uri
ty A
sses
smen
ts
ES
S =
En
gag
emen
t S
imu
lati
on
Syst
ems
FO
F =
Forc
e on
Fo
rce
IT =
In
form
atio
n T
ech
nolo
gy
L
SP
T =
Lim
ited
-Sco
pe
Per
form
ance
Tes
t
NT
C =
Nat
ional
Tra
inin
g C
ente
r P
OC
= P
oin
t of
Con
tact
S
SIM
S =
Saf
egu
ard
s an
d S
ecu
rity
In
form
atio
n M
anag
emen
t S
yst
em
Mu
lti-
To
pic
Ass
essm
ent
Tim
elin
e
Appendix A – Multi-Topic Assessment Timeline
Office of Security Assessments Appraisal Process Protocols Appendix B
January 2017 B-1
Appendix B – Example Document Request (Data Call)
The following is a template for each topical area and should be revised according to the specific scope for
each Office of Enterprise Assessments (EA) appraisal conducted by the EA Office of Security Assessments
(EA-22). Topic Team Leaders must coordinate to ensure that the same documents are not requested in
multiple topical areas unless operationally necessary.
Protection Program Management (PPM)
The following items are normally provided to Headquarters to support planning meeting activities.
Safeguards and Security (S&S) Management and Administration
1) Organizational charts with names for all Federal and contractor site elements (including significant
subcontractors) that have S&S responsibilities.
2) Federal field office and contractor missions and function manuals and/or other reference material
describing the roles and responsibilities of current site organizations, including deliverables and
accountability within the S&S program.
3) Fiscal year (FY) (insert date range) Performance Evaluation Report(s) and FY (insert date range)
Performance Evaluation Plans for each prime contractor.
4) Current Federal and contractor delegation of authority memoranda.
5) Program office guidance and local instructions for the implementation of S&S programs.
6) Federal field office procedures for incorporating new/revised U.S. Department of Energy (DOE)
directives into the contract(s).
7) List of requirements documents in the current contracts with S&S responsibilities (e.g.,
management and operating contractor, S&S services support contracts).
8) Implementation plans for any S&S-related directives not fully implemented.
9) Approved and submitted equivalencies/exemptions (deviations), including justification, risk
analysis, and approvals as applicable.
10) Federal and contractor procedures/process documents for equivalency/exemption (deviation)
submissions and approval.
11) FY (insert date range) Annual Training Plan(s) for Federal and contractor S&S staff (non-
protective force).
S&S Planning
12) List of all Federal and contractor entities with a facility clearance supporting the site’s operations.
13) Current approved Federal, contractor, and any other possessing entity facility/site security plans
(SSPs).
14) Current vulnerability assessment (VA) report (VAR) supporting the most current approved SSP,
and the associated VA procedure.
15) The most current analytical basis that addresses the security assets present at the site, including
(but not limited to) hazardous materials and special nuclear material (SNM) rollup.
16) Compensatory measures and approvals as applicable.
17) Security Incident Response Plan and Tactical Doctrine implementing documents.
18) Security Conditions (SECON) Plan(s) and supporting implementing documents (e.g., procedures,
checklists).
19) Regional and local threat assessments.
20) Current applicable memoranda of understanding (MOUs) and memoranda of agreement (MOAs)
with local, state, and Federal law enforcement agencies.
Office of Security Assessments Appendix B Appraisal Process Protocols
B-2 January 2017
21) Federal and contractor procedure/process for controlling changes/updates in relevant documents
(e.g., plans, procedures).
S&S Program Assurance
22) Federal survey/assessment reports and contractor self-assessment reports for FY (insert date
range).
23) Guidance on Federal oversight, survey procedures, FY (insert date range) contractor self-
assessment procedures, and FY (insert date range) schedules.
24) Federal and contractor procedures for resolution of findings (external inspections, surveys and
self-assessments, observations, and/or other action items related to the mitigation of identified
weaknesses in the S&S program).
25) Federal and contractor corrective action plans for previous EA appraisals.
26) Records, other than those in the DOE Safeguards and Security Information Management System,
that reflect Federal verification, validation, and closure of deficiencies, findings, concerns, and/or
observations for FY (insert date range).
27) Performance Assurance Program (PAP) Plan and listing of essential elements.
28) Most current essential element test plans.
The following items are typically available in the PPM team workspace when they arrive at the site.
29) Briefings/discussions by Federal/contractor personnel (as coordinated and scheduled with the
EA-22 PPM Topic Team Leader) that include:
a) Overview of contract management structure.
b) Description of changes in program management since the last EA appraisal.
c) Overview of the VA and SSP development processes, including:
i. Developing adversary strategies/tactics.
ii. Evaluating insider adversaries, both working alone and in collusion with the outsider.
iii. Conducting computer modeling/performance testing analyses and developing
neutralization values.
iv. Developing and evaluating upgrade/efficiency packages.
v. Chronological efforts and events leading up to the current/approved SSP (including VAs,
validations, peer reviews, Headquarters visits, etc.).
vi. Mechanisms for performance-tested validation of VA assumptions and values.
d) Overview of the Federal and contractor survey/self-assessment programs, including the process
for conducting risk-based evaluations.
e) Overview of the corrective action/issue management and causal analysis processes to prevent
recurrence of deficiencies.
f) Summary of the status of corrective actions for prior EA findings.
30) Data from performance testing that supports the most recent/current VA or performance
assumptions in the VAR.
31) Files used to develop the VA and the associated evidence files for the following types of data:
a) Modeling inputs.
b) Protective force response.
c) Adversary capabilities.
d) Blast effects.
e) Sabotage data (if appropriate).
f) Timeline data.
g) Neutralization data.
h) Special weapons effectiveness.
32) Simulation reports and capability to review sample scenarios.
33) FY (insert date range) PAP test reports for all topical areas.
Office of Security Assessments Appraisal Process Protocols Appendix B
January 2017 B-3
34) Evidence files supporting Federal surveys and contractor self-assessments for FY (insert date
range).
35) Federal and contractor qualifications and training records for S&S staff (non-protective force).
Physical Security Systems (PSS)
The following items are normally provided to Headquarters to support planning meeting activities.
1) A site plan drawing (11"x17" or larger) indicating security areas and target locations associated
with the site. Include maintenance and communications facilities and the locations of the central
alarm station (CAS) and secondary alarm station (SAS). Include a separate drawing (11"x17" or
larger) for the Protected Area and perimeter intrusion detection and assessment system zones.
2) Compensatory plans associated with PSS equipment that is out of service.
3) Description of explosive detection program.
4) Copies of approved/pending PSS-related deviations.
5) List of PSS-related documents approved by the Officially Designated Security Authority.
6) List of PSS-related documents approved by the Officially Designated Federal Security Authority.
7) Copies of PSS-related self-assessments completed during the past 12 months.
8) Copies of PSS-related surveys completed during the past 12 months.
9) Copies of corrective actions associated with findings resulting from the previous EA assessment.
10) Information system security plans (or other terminology, such as cyber security plans) for
computer-controlled PSS.
11) List of privileged users (job title only) for all computer-controlled PSS.
Alarm Management and Control Systems (CAS/SAS operations, auxiliary power)
12) System descriptions and diagrams for alarm monitoring and control systems. For alarm
monitoring systems, schematic diagrams should include locations of major equipment (data
gathering panels, activation panels, and communication panels) for interior and exterior systems.
13) System descriptions and diagrams for primary and auxiliary power systems associated with PSS,
including generators, automatic failover systems, uninterruptible power supply systems, etc.
14) Procedures for testing auxiliary power systems, and documentation indicating the results of tests
conducted during the past year.
15) Description of all types of line supervision and tamper alarms in use, and their locations.
16) Procedures for conducting CAS/SAS failover tests, and documentation indicating the results of
tests conducted during the past year.
17) Configuration management policies and procedures for PSS hardware and software.
18) Alarm system operator training and testing documentation.
19) List of all vaults, Closed Areas, and/or vault-type rooms (VTRs).
Communications (radio communications, alternate communications)
20) System descriptions and diagrams for security-related voice communications, which should
include a VA. A description of the emergency communications plan (may be part of the Tactical
Defense Plan or Protective Force Response Plan).
Barriers
21) Descriptions and diagrams for active or passive barrier systems used to direct or control the
movement of personnel and vehicles through security area boundaries.
22) Procedures for identifying potential barrier degradation.
Office of Security Assessments Appendix B Appraisal Process Protocols
B-4 January 2017
Access Control Systems, Entry/Exit Inspections, and Badging
23) System descriptions and diagrams for access control systems, including interfaces between the
badge system, access control system, and alarm monitoring system, and the procedures for badge
fabrication and personnel enrollment. A VA addressing these interfaces should be included. Also
provide specific procedures for removing or changing access authorization in the access control
system for individuals whose need for authorization has changed, and destruction procedures for
badges returned to the badge office.
24) Description of screening procedures used at security boundaries to identify prohibited and/or
controlled articles.
Testing and Maintenance and PAP
25) Procedures and program description for false/nuisance alarm rate review, analysis, and corrective
action development.
26) False/nuisance alarm rate data and analysis for security system sensors for the most recent six
months available.
27) Security systems preventive and corrective maintenance and testing program documentation and
procedures. List of maintenance activities associated with PSS within the last 12-month period.
28) Performance testing criteria for alarm monitoring and video assessment systems, access control
systems, and security-related voice communications systems.
29) List of PSS-related performance testing activities completed during the past 12 months.
30) Acceptance testing procedures for installation of new or replaced security systems.
Lock and Key Program
31) Program documentation and associated procedures related to security locks and keys. Also
provide security key accountability documentation and audit results generated during the past
year.
System Upgrades
32) Description of new and/or upgraded systems installed and declared operational since the last EA
assessment.
33) Description of planned upgrades relating to PSS, including status of authorization and funding, as
well as the expected date the planned upgrades will be in service. Also provide descriptions and
dates of implementation for upgrades and modifications to alarm monitoring and video assessment
systems, access control systems, and security-related voice communications systems completed
in the last two years (hardware and software), and the acceptance testing documentation associated
with these modifications and upgrades.
Protective Force
The following items are normally provided to Headquarters to support planning meeting activities.
In addition to the requested items below, EA will require access to protective force training records once
onsite data collection begins.
Management Documentation 1) Current protective force contract, including all applicable modifications.
2) List of DOE orders cited in the protective force contract and any applicable implementation plans.
3) Latest approved SSP.
4) VAR and/or risk analysis report.
5) Latest Performance Assurance Plan, if separate from the SSP.
6) Latest approved SECON Plan, if separate from the SSP.
Office of Security Assessments Appraisal Process Protocols Appendix B
January 2017 B-5
7) Delegation letters for the Cognizant Security Office, the Officially Designated Federal Security
Authority, and any applicable Officially Designated Security Authority (Federal or contractor).
8) Any current or pending deviations (equivalencies/exemptions) from DOE S&S directives.
9) Supervisor post/patrol duty check logs for the (insert date range) time period.
10) Armorer maintenance records for the last two cycles (one-year period).
Protective Force Survey and Self-Assessment Documentation
11) Most recent Federal survey report.
12) Most recent protective force self-assessment report.
13) Corrective action plans, including the associated evidence files for any open findings resulting
from the most recent survey and self-assessment. (Note: If evidence files are too cumbersome to
scan, EA-22 can be review them on site during assessment activities.)
Protective Force Plans and Contracts
14) Security Incident Response Plans.
15) SECON implementing procedures or post orders.
16) Protective force work stoppage contingency plan.
17) Site map indicating protective force fixed posts and patrol areas.
18) Current and projected protective force strength report, including figures for the last four quarters.
19) Current protective force contract, including any modifications since the contract was originally
awarded.
20) Local law enforcement authority, including implementation/re-certification plans.
Protective Force Orders and Procedures
21) Protective force organization chart.
22) Protective force general orders.
23) Protective force post orders.
24) Protective force supervisory orders if separate from post/patrol orders.
25) Protective Force standard operating procedures.
26) A current “on-shift” and “off-shift” protective force duty roster that includes shift schedules,
limited-duty rosters, and vacation rosters for the period (insert date range). Note: If the site has
not yet developed a duty roster/schedule for the requested period, provide the most current one
available.
27) 10 CFR 1046 implementation plan, including the status of its overall execution.
28) Use-of-force policy.
29) Rules of engagement.
30) Range safety procedures.
Protective Force Memoranda of Understanding or Agreement
31) Current MOU/MOA, showing all required signatures, with law enforcement, emergency
services, and any other support organizations.
Protective Force Training, Testing, and Assessment Plans
32) Protective force annual training plan.
33) Protective force training approval program certification.
34) Protective force performance testing plan.
35) Four to eight performance test plans for protective force essential elements:
a) SNM detection.
b) Entry/exit portal detection.
c) Explosive detection.
Office of Security Assessments Appendix B Appraisal Process Protocols
B-6 January 2017
d) Active shooter.
e) Response to unmanned aerial system.
36) Protective force issues management protocol. (Note: Once on site, EA-22 will request test plans
for specific assessment activities.)
Equipment
37) Protective force inventory of equipment, including vehicles, weapons, radios, and non-lethal
weapons and identifying the locations and types of individual post/patrol assigned equipment.
The following protective force documents should be available upon request for review.
38) Individual security police officer (SPO) qualification records (weapons, physical fitness).
39) Certification records for instructors (classroom and range), armorer, and CAS operators.
40) Individual SPO training records.
41) SPO lesson plans.
42) Sample of supervisor and post logs.
43) Target folders.
44) Description of canine program and training records, schedules, and results associated with
performance testing.
45) Full color photographs (8.5"x11") of SNM that is currently being protected, stored, and/or
processed on site.
46) Vehicle maintenance service records.
Material Control and Accountability (MC&A)
The following items are normally provided to Headquarters to support planning meeting activities.
1) The current approved MC&A Plan with the approval letter from the site office.
2) A draft MC&A Plan if the site anticipates submitting a revised version before EA’s onsite
assessment.
3) All approved material balance area (MBA)-specific MC&A Plans as applicable.
4) MC&A procedures manual.
5) List of approved MC&A deviations with supporting documentation.
6) MC&A documentation used to support the Category of irradiated fuel, including procedures for
monitoring rollup.
7) Site office MC&A survey reports for surveys conducted in (insert date range).
8) List of approved tamper-indicating devices (TIDs) and TID custodians, applicators, and/or TID
verifiers, along with TID procedures (if not included in MC&A procedures manual).
9) Current shipper/receiver agreements.
10) Emergency response plans pertinent to the loss of control of SNM.
11) If temporary material access areas (MAAs) are used, the procedures in effect when temporary
MAAs are established.
12) All facility-reported incidents of security concern (IOSC) involving MC&A, security, and
operations for the past two years with backup documentation.
13) MC&A Training Plans, MC&A Performance Testing Plan, and Measurement Qualification Plans
(these are distinct from procedures but may be included in other documentation).
14) Training records (electronic) for all personnel performing MC&A functions.
15) Complete list of findings and corrective actions from DOE surveys and internal reviews for the
past two years.
16) Internal Review and Assessment Plan and procedures, list of internal reviews and assessments
completed since (insert date).
Office of Security Assessments Appraisal Process Protocols Appendix B
January 2017 B-7
17) A current list of MBAs and MBA custodians and alternates.
18) Plans and procedures for measuring holdup (if not included in the MC&A procedures manual).
19) Safeguards termination documentation on nuclear materials terminated in (insert date range).
20) Dates of Nuclear Materials Management and Safeguards System annual submissions for (insert
date range).
21) Summary of shipper/receiver differences identified since (insert date).
22) List of current measurement procedures for approved accountability systems.
23) List of current measurement control procedures for approved accountability systems.
24) Inventory procedures and schedules (if not included in the MC&A procedures manual).
25) If Comprehensive Analyses of Safeguards Strategies (COMPASS) software is used, results of the
two most recent COMPASS analyses.
26) VA demonstrating MC&A analyses for Category I SNM locations and other locations where
rollup to Category II is credible.
27) List of any MC&A performance incentives/award fee items in the MC&A topical area.
The following information should be readily available from site MC&A personnel.
28) Summary inventory list for MBAs, showing total quantities for each material type (may be kept
with accounting organization and will be reviewed during the assessment).
29) Material balance reports (M-742) since (insert date).
30) Inventory differences, limit of error of the inventory difference, and key measurement error
contributors by material type, by MBA, for each inventory period for the last 24 months.
31) Qualification reports for all instruments currently in use for accountability measurements.
32) Verification and accountability measurement results, including resolution paths for items that
failed verification.
33) List of replicate data used in the determination of measurement uncertainties for the last six
inventories.
34) The results of process and item monitoring (for all MBAs that employ these activities) for the past
24 months.
35) Summary information on key holdup points in the facility and holdup measurements conducted
since (insert date).
36) Status of SNM considered unmeasured and/or not amenable to measurement, by MBA.
37) Sampling plans (if not included in the MC&A procedures manual) and the results of the
verification/confirmation program for the previous two physical inventories.
In addition to the data listed above, please prepare a briefing (to be given on the first day of the assessment,
after inbriefing and initial orientations) that provides the following information:
A description of the current MC&A organizational structure (Federal and contractor, including the
names of individuals and funding lists).
Any changes to the MC&A system and the operational status of the facility (including any process
activities that may have changed the characteristics of existing material types or produced new
material types) since (insert date).
An overview of planned activities/production schedule for MC&A during the next two years (e.g.,
mixed oxides, processing of Category I and II quantities), along with any staffing/funding concerns.
Summary/status of any ongoing and proposed termination of safeguards activities (e.g., plutonium
downblending).
Office of Security Assessments Appendix B Appraisal Process Protocols
B-8 January 2017
It is very important that the briefing also include the results of recent assessments and key issues currently
being addressed by the MC&A program. Copies of the briefing notes should be provided to the assessors.
Classified Matter Protection and Control (CMPC)
The following items are normally provided to Headquarters to support planning meeting activities.
1) A list of vaults, VTRs, and Closed Areas (by building and room number) that currently store any
level (Confidential, Secret, Top Secret) and category (e.g., National Security Information,
Restricted Data, Weapons Data, North Atlantic Treaty Organization, Foreign Government
Information) of classified matter, regardless of form (documents, electronic media, parts, etc.).
2) A list of classified holdings that are stored in non-standard storage configurations, and the
compensatory measures for each location.
3) Copies of the security plans for non-conforming or non-standard storage areas, including the
Officially Designated Federal Security Authority’s approvals for the areas.
4) A list, by organization, of the custodians or persons responsible for the classified repositories,
vaults, VTRs, and Closed Areas listed above that have accountable items (Top Secret, United
Kingdom, North Atlantic Treaty Organization, Sigma 14, Sigma 20 media, and Secret/Restricted
Data stored outside of a Limited Area, or as non-accountable), including phone numbers and
locations (building and room number).
5) A current inventory listing of all accountable classified matter, regardless of form (documents,
electronic media, parts, etc.) and their storage locations or by account.
6) Copies of the last two annual inventories.
7) Copies of local procedures, self-assessments, and equivalencies/exemptions (or deviations
pending conversion to equivalences/exemptions) pertaining to the following programs/areas:
a) Operations security (OPSEC)
b) CMPC, including training material
c) Hand-carry, including briefing material
d) Classified matter destruction, reproduction, and transmission
e) Mailrooms
f) Technical surveillance countermeasures (TSCM).
8) List of points of contacts with phone numbers for the programs listed above, including both
Federal and contractor personnel.
9) Copies of the corrective action plans for the last EA assessment of CMPC.
10) Copy of the classified matter custodian training program.
11) Copies of the OPSEC Plan and site critical information.
12) Copy of the most recent (within the past 36 months) OPSEC assessments or reviews.
For items 13 and 14 below, include these only if the items are unclassified. If they are classified, EA-22
will review them on site.
13) Copy of local policy and procedures for the TSCM program and team.
14) Copies of training and briefing materials used to train TSCM and site personnel.
The following items should be made available for review at the site during the week of (insert date). Since
these items are normally classified, the assessors expect to conduct this portion of the assessment in the
space where these items are normally stored/used.
15) Copies of the last five years of reports of TSCM services.
16) Information on the scheduling of TSCM services.
17) Last self-assessment of the TSCM program at the site.
Office of Security Assessments Appraisal Process Protocols Appendix B
January 2017 B-9
18) Information concerning lifecycle replacement of TSCM equipment.
Incidents of Security Concern
19) Copy of the IOSC program plan and documentation of approval.
20) IOSC-related procedures.
21) List of all Federal and contractor IOSC incidents in FY (insert date range), sorted by category and
incident type and including local tracking numbers and resolution actions.
22) Copy of IOSC damage assessments for FY (insert date range).
23) Copy/access to IOSC awareness training materials.
24) Copy of the inquiry official’s qualification summary and associated training records.
25) Copy of the inquiry official’s appointment/delegation letter.
26) Copy of foreign ownership, control or influence (FOCI)/facility clearance (FCL) procedures,
including the process/procedure for reporting changes that may affect the facility clearance level.
27) Copy of documentation for FCL approvals on current contractors and tier parents for which the
site has cognizance.
28) A list of all “Possessing” facilities for which the site has cognizance, and their facility codes.
29) A list of cleared personnel within each of the facilities:
a) Identify any contractors under FOCI mitigation and provide documentation for the trustees,
proxy holders, and/or outside directors in connection with the mitigation plan.
b) For contractors under mitigation, provide certification that no changes have occurred in their
ability to protect classified information.
c) Copy of documentation of annual reviews and, for contractors under mitigation, certification
that no changes have occurred in their ability to protect classified information.
d) Copy of documentation pertaining to FCL suspensions related to FOCI mitigation.
Human Reliability Program, Personnel Security, and Unclassified Foreign Visits and Assignments
30) An organization chart(s) or other means of describing the structure supporting the overall
personnel security program, sufficient to show where all key program officials and support staff
reside organizationally and the chain of command to each key program official and support staff.
Human Reliability Program (HRP)
31) Copy of the HRP implementation plan(s) and documentation of review and approval.
32) List of positions identified for HRP certification and a description of the process for evaluating
these positions for certification, including those positions applicable to 10 CFR 712 Category 4.
33) List of all Federal and contractor employees enrolled/certified in the HRP program as of (insert
date).
34) A separate, alphabetized (last name first) list for each of the following for the period of (insert
date range), as follows:
a) All HRP individuals who have been temporarily removed, with the date and the reason for
temporary removal (security, safety, medical, or change of position/employment).
b) All HRP individuals who have had their HRP certification revoked, with the date and the reason
for revocation (security, safety, medical, or change of position/ employment).
c) All individuals denied HRP certification by the site Certifying Official.
d) All HRP individuals who have had any disciplinary action(s), including the reason for the
disciplinary action, the date the disciplinary action was taken, and whether the individual was
temporarily removed from HRP because of disciplinary action.
e) All HRP individuals who have been involved in an accident or incident, including IOSCs
reported to DOE as an occurrence.
Office of Security Assessments Appendix B Appraisal Process Protocols
B-10 January 2017
f) All non-HRP escorted individuals, indicating the dates of entry, grouped for each escorted
visitor and including all dates (most recent first) of entry for each MAA accessed; the
individual’s employer; and authorized clearance level at the time of entry (Q or L).
35) A list of job titles for each organization where job task analyses have been developed and an
example (blank form) of the format used by each organization.
36) Access to HRP files and HRP-associated medical and psychologist files (make available during
onsite visits).
37) Copy of HRP training and instructional materials (computer-based training, classroom, etc.).
38) Copy of designation letters for the following positions (as applicable):
a) HRP Certifying Official.
b) HRP Management Official(s).
c) Designated Physician.
d) Designated Psychologist.
Personnel Security
39) Separate, alphabetized (last name first) lists of the following for the site contractor:
a) A list of personnel with an access authorization. The timeframe for all lists is (insert date
range).
b) A list of all completed pre-employment checks.
c) A list of all individuals who had absences of 90 days or more. (Payroll records should be used
to develop this list.)
d) A list of all contractor/subcontractor employees for whom the site has notified or reported to
the DOE personnel security organization information of personnel security interest as a result
of a disciplinary action (omit security infraction reports of the sort requested by the CMPC
topic team), the reason for the disciplinary action, and the date reported to the DOE personnel
security organization.
40) A separate, alphabetized (last name first) list of the following for contractor employees with a Q
access authorization for the period of (insert date range):
a) A list of access authorization terminations and the date (omit transfers or anything that is not a
termination of the access authorization).
b) A list of all employment terminations and the date (omit individuals who were re-employed by
the site or a subcontractor within six months).
c) A list of all individuals who have been granted an initial access authorization, the date DOE
took the action to grant, and the date a DOE security badge was issued.
d) A list of all individuals who have transferred from another DOE facility, the date DOE took
the action to transfer, and the date a DOE security badge was issued.
e) A list of all individuals who have had their access authorization reinstated, the date DOE took
the action to reinstate, and the date a DOE security badge was issued.
41) Procedure/description of pre-employment and annual random drug testing for Federal personnel,
contractors, and subcontractors with access authorizations or those requesting an access
authorization. Testing of these individuals is required by Secretarial memorandum dated
September 14, 2007.
42) List of cleared contractor employees (including subcontractors) whose workspace as of (insert
date) is located outside of a Limited, security/exclusion, or Protected Area. This list should
include offsite buildings/facilities only if there are more than ten cleared employees in the
workspaces.
43) Copy of the current initial, comprehensive, and termination briefings and the most recent annual
security refresher briefing.
Office of Security Assessments Appraisal Process Protocols Appendix B
January 2017 B-11
Unclassified Foreign Visits and Assignments (UFV&A)
44) The total number of unclassified foreign national visitors and assignees who visited the site from
(insert date range), broken out in the following categories: non-sensitive country foreign nationals,
sensitive country foreign nationals, sensitive subjects, and access to security areas (Limited Areas,
Protected Areas, MAAs).
45) A separate, alphabetized (last name first) list of UFV&As from (insert date range) for items a
through j below. Each of these lists should provide the following information: name and
nationality of visitor/assignee, date of visit/assignment, name of host/escorts, facilities included in
the scope of the visit/assignment, and, when applicable, approval for remote or onsite access to
computing systems.
a) Foreign nationals from sensitive countries.
b) Sensitive subjects.
c) Access to an MAA, Protected Area, Limited Area, or security/exclusion area.
d) Foreign nationals from terrorist countries.
e) Unescorted access to any site security area (Limited Area, Protected Area, or MAA).
f) Access to site computing assets, including a termination date for access to the computing assets.
g) Remote access to site computing assets, including a termination date for remote access.
h) After-duty-hours access to any site facility.
i) Incidents and inquiries of security concern.
j) The most frequently visited site facilities (building or areas) and program organizations.
46) A diagram showing all site security areas (MAAs, Protected Areas, Limited Areas), including
building numbers. (This diagram may be a duplicate of those provided to other topic teams.)
47) Procedures and/or protocols used to process and approve all UFV&As.
48) An example of a generic security plan for a foreign visit or assignment and a copy of a specific
security plan.
49) Copy of host/escort guidance, or training materials.
50) Procedure or description of how lessons learned are shared with other hosts/escorts.
Office of Security Assessments Appendix C Appraisal Process Protocols
C-1 January 2017
Appendix C – Example of Multi-Topic Assessment Plan
OFFICE OF ENTERPRISE ASSESSMENTS PLAN
FOR CONDUCTING SAFEGUARDS AND SECURITY ASSESSMENT AT THE
(Facility Name)
(DATE)
Office of Enterprise Assessments
U.S. Department of Energy
Office of Security Assessments Appraisal Process Protocols Appendix C
January 2017 C-2
OFFICE OF ENTERPRISE ASSESSMENTS PLAN
FOR CONDUCTING A SAFEGUARDS AND SECURITY ASSESSMENT AT THE
(Facility Name)
(DATE)
Approved by: ______________________________________ Date: ________________
(Name) Director
Office of Security Assessments
Office of Cyber and Security Assessments
Approved by: ______________________________________ Date: ________________
(Name) Director
Office of Cyber and Security Assessments
Office of Enterprise Assessments
Approved by: ______________________________________ Date: ________________
(Name) Deputy Director
Office of Enterprise Assessments
Office of Security Assessments Appendix C Appraisal Process Protocols
C-3 January 2017
OFFICE OF ENTERPRISE ASSESSMENTS PLAN
FOR CONDUCTING A SAFEGUARDS AND SECURITY ASSESSMENT AT THE
(Facility Name)
(DATE)
PLAN CONTENTS
Assessment Plan and Approach ........................................................................................................ C-4
Attachment A: Assessment Schedule .............................................................................................. C-9
Attachment B: Assessment Team Composition .............................................................................. C-10
Attachment C: Protection Program Management Lines of Inquiry ................................................. C-11
Attachment D: Physical Security Systems Lines of Inquiry ............................................................ C-12
Attachment E: Protective Force Lines of Inquiry ............................................................................ C-14
Attachment F: Material Control and Accountability Lines of Inquiry ............................................. C-16
Attachment G: Classified Matter Protection and Control Lines of Inquiry ..................................... C-17
Office of Security Assessments Appraisal Process Protocols Appendix C
January 2017 C-4
OFFICE OF ENTERPRISE ASSESSMENTS PLAN
FOR CONDUCTING A SAFEGUARDS AND SECURITY ASSESSMENT AT THE
(Facility Name)
(DATE)
ASSESSMENT PLAN AND APPROACH
I. INTRODUCTION
The Office of Enterprise Assessments (EA) will conduct a comprehensive safeguards and security
(S&S) assessment at the (insert site/facility), (insert dates). Assessment activities will assess the
effectiveness of management processes and mission operations associated with the protection of
special nuclear material (SNM) and classified matter. The objectives of this assessment are to
evaluate the effectiveness of the (insert site/facility) protection program by examining specific
security topical areas and to provide senior managers within the U.S. Department of Energy (DOE),
or the National Nuclear Security Administration (NNSA), if applicable, with an evaluation of the
adequacy of Departmental policy requirements and their implementation.
EA conducts assessments in accordance with Departmental directives, EA protocols, and
assessment guides. Although this plan outlines projected assessment activities, EA may adjust
specific focus areas, performance testing, and/or documentation requests during the assessment in
response to emerging concerns, requests from senior DOE managers, or performance trends
observed during data collection. If observations indicate a need to deviate substantially from the
documented scope of this plan, such changes will be coordinated with the (insert field office, site,
facility, program office, contractor management, etc., as applicable).
II. APPRAISAL SCOPE
EA will use a variety of performance testing and data collection methods to assess the readiness
of protection systems to counter a broad spectrum of threats and adversary capabilities. To
accomplish the objectives of this assessment, EA will evaluate the following topics and subtopics:
Protection Program Management (PPM); Physical Security Systems (PSS); Protective Force (in
coordination with the Office of Emergency Management, if applicable); Material Control and
Accountability (MC&A); and Classified Matter Protection and Control (CMPC), which will also
include elements of program management and personnel security.
PPM activities will concentrate on the effectiveness and validity of S&S management and
administration; S&S planning and procedures, including security plans, Security Conditions,
performance assurance (i.e., performance of essential elements), surveys, and self-assessment
programs. The S&S training program for non-protective force personnel and the implementation
of the DOE Tactical Doctrine will also be included in this assessment.
PSS assessment activities will include a review of security areas and secure storage facilities,
including the effectiveness of associated intrusion detection and assessment systems, along with
procedures and equipment associated with access control systems and entry/exit inspections. The
comprehensiveness and adequacy of the PSS-related performance assurance program (PAP) and
testing and maintenance activities will be evaluated, including actions associated with false and
nuisance alarms. Assessment activities will incorporate protective force communications, the
lock and key program, lifecycle management of security equipment, and other related topics.
Office of Security Assessments Appendix C Appraisal Process Protocols
C-5 January 2017
Protective Force assessment activities will focus on protective force management, training,
equipment, facilities, and routine and emergency duty performance. EA will emphasize
performance-based testing, including force-on-force (FOF) exercises that incorporate recapture
and recovery, limited-scope performance tests (LSPTs), firearm proficiency testing, and training
and readiness. EA will evaluate the protective force’s knowledge of and training on Departmental
use-of-force policies and arrest authorities, familiarity with and training on approved rules of
engagement, and the consistency of protective force operations and plans as set out in the site
security plan (SSP) documentation. The integration of protective force and Office of Emergency
Management operations will be assessed through the review of applicable plans and procedures,
personnel interviews, and applicable protective force performance test activities. Areas of
evaluation include event categorization and classification, initial notifications and
communications, and coordinated actions to protect the health and safety of onsite personnel and
the public.
MC&A assessment activities will include a review of MC&A program planning documentation.
Observations, interviews, and LSPTs will determine the adequacy of program administration,
materials accountability, and materials control. Performance testing activities will address
measurements, tamper-indicating devices, inventory and accounting, and front and back
accountability checks of selected items. EA will interview personnel responsible for these
subtopical areas.
CMPC assessment activities will address site-specific procedures and training of personnel,
operations security, technical surveillance countermeasures, and physical protection and control,
which includes accountability of classified matter, aspects of personnel security, unclassified
foreign visits and assignments, the S&S awareness program, the human reliability program (HRP),
and the incidents of security concern (IOSC) program. Areas for performance testing include
transmission, reproduction, destruction, and front and back checks of accountable matter.
At present, EA assessments include an evaluation of the site’s actions to address the insider threat
in each topical area. In general, EA examines the comprehensiveness of the insider analyses
supporting current SSPs, administrative and engineered controls for access control, deterrence
actions that aid in preventing unauthorized access by either U.S. citizens or foreign nationals, and
removal or misuse of site security interests. EA will identify specific concerns related to the
insider threat in the relevant sections of the final report; the overall assessment of this area will
appear in the executive summary of the final report.
This Assessment Plan includes separate attachments containing lines of inquiry for each topic.
III. ASSESSMENT SCHEDULE
The Appraisal Team Chief and his Deputy will conduct a leadership-scoping visit (insert dates).
The protective force team will conduct an onsite walkdown (insert date) and scenario
development (insert date). The EA-22 PPM, PSS, MC&A, protective force, and CMPC teams
will conduct planning and data review at DOE Headquarters in Germantown, Maryland (insert
date). All topic teams will conduct performance testing and onsite data collection activities
(insert dates). EA will conduct additional LSPTs, analyze and validate overall assessment
results, develop a draft report (insert dates), and present a closeout briefing at (insert office, site,
facility) on (insert date). EA will provide a draft report to (insert office, site, facility, and program
stakeholders) management for factual accuracy review before the closeout briefing. After the
Office of Security Assessments Appraisal Process Protocols Appendix C
January 2017 C-6
closeout briefing, EA will provide the draft report to the site and the program office points of
contact the following week. All parties will have ten calendar days to review the draft report.
Each topic team will develop individual schedules (in coordination with their Federal and
contractor counterparts) for more specific onsite activities to implement this overall plan.
IV. ASSESSMENT TEAM RESPONSIBILITIES AND ASSIGNMENTS
Mr. (insert EA official), Assessment Team Chief, will be the senior DOE official and EA point
of contact managing the conduct of onsite assessment activities. Mr. (insert EA official), Deputy
Team Chief, will support Mr. (insert EA official). The Assessment Team Chief and his staff will
ensure that evaluation activities are consistent with this plan, documented protocols, and approved
procedures. The appropriate team management, listed below, will address any questions
regarding the planned lines of inquiry (see Attachments C through G).
ASSESSMENT TEAM CONTACT INFORMATION
Name – Team Chief Telephone Email Address
Name – Deputy Team Chief Telephone Email Address
Name – PPM Topic Team Leader Telephone Email Address
Name – PSS Topic Team Leader Telephone Email Address
Name – Protective Force Topic Team Leader Telephone Email Address
Name – MC&A Team Leader Telephone Email Address
Name – CMPC Topic Team Leader Telephone Email Address
Professional Conduct and Conflict Resolution
All EA team members and (insert office, site, facility, as applicable) representatives will represent
their respective organizations with the highest standards of professional conduct, and will interact
in a manner defined by reciprocal professional courtesy, mutual respect, and objectivity. These
standards of conduct will further define the professional relationship between EA and (insert office,
site, facility, as applicable) in supporting effective and efficient execution of the assessment
process.
Potential concerns related to professional conduct will be resolved in a manner that adheres to the
following tenets:
Validate the complaint.
Report and discuss the concern with management of both organizations on site.
Make every attempt to resolve the concern at the lowest organizational level.
Office of Security Assessments Appendix C Appraisal Process Protocols
C-7 January 2017
V. ASSESSMENT PROCESS
The appraisal process includes:
Scoping, planning, and reviewing the data call.
Collecting data and performance testing.
Analyzing data and determining the causal factors for any identified program performance
weaknesses.
Preparing a report, identifying findings and deficiencies, and conducting management
validation briefings.
Scoping, Planning, and Data Call Review
Scoping, planning, and data call review are instrumental in developing topic-specific schedules to
augment this plan, thus helping to ensure that (insert office, site, facility, as applicable)
representatives are informed of ongoing data collection efforts. Supplemental to these activities,
EA will make all representatives aware of any significant changes in proposed activities and focus
areas.
Collecting Data and Performance Testing
Accurate data collection is a central feature of the appraisal process. Methodologies for collecting
data, as part of this assessment, will entail the conduct of document reviews, facility tours,
interviews, observations, LSPTs, and FOF performance testing. Evaluative criteria established for
data collection and performance tests will be consistent with Departmental requirements, approved
procedures, (insert office, site, facility, as applicable) training, and/or DOE National Training
Center (NTC) standards.
Analyzing Data
Careful examination of the collected data highlights topical program trends and provides evidence
of progress or decline in overall protection program performance. A common emphasis during
analysis is the effectiveness of performance feedback and improvement mechanisms, such as
survey programs, the PAP, self-assessment activities, and associated corrective action mechanisms.
Report Preparation, Findings/Deficiencies Identification, and Management Briefings
The assessment report will identify observed site protection program strengths and weaknesses, as
well as document gaps or shortfalls in DOE policy that adversely affect protection program
performance. (Insert office, site, and facility, as applicable) line management, and policy
organizations as necessary, will have the opportunity to verify the factual accuracy of results before
issuance of the final report. For major appraisals, the cognizant Program Secretarial Officer will
have the opportunity to submit a brief (approximately two pages) written management response to
the conclusions and any recommendations included in the final draft report. If such a response is
submitted, EA will include it as an appendix in the final report.
Findings are performance-based and/or systemic deficiencies that warrant a high level of
management attention and that, if left uncorrected, could adversely affect the DOE mission, worker
safety and health, the public, or national security. In these cases, (insert office, site, facility, as
applicable) will develop appropriate corrective action plans and track them to completion in
Office of Security Assessments Appraisal Process Protocols Appendix C
January 2017 C-8
accordance with DOE Order 226.1B, Implementation of Department of Energy Oversight Policy,
and DOE Order 227.1A, Independent Oversight Program. Deficiencies may serve as the basis for
one or more findings. The applicable program office will determine whether such deficiencies
warrant entry into an issues management system and/or whether they require formal corrective
actions.
A best practice is a technique, process, or program attribute observed during an appraisal that may
merit consideration by other DOE and contractor organizations for implementation because it: (1)
has been demonstrated to substantially improve the safety or security performance of a DOE
operation; (2) represents or contributes to superior performance (beyond compliance); (3) solves a
problem or reduces the risk of a condition or practice that affects multiple DOE sites or programs;
or (4) provides an innovative approach or method to improve effectiveness or efficiency.
Opportunities for improvement are suggestions offered that may assist cognizant managers in
improving programs and operations. These suggestions are not prescriptive.
EA will provide briefings to the appropriate (insert office, site, and facility, as applicable) managers,
highlighting the analyzed results of EA’s data collection efforts to aid in management resource
decisions.
Office of Security Assessments Appendix C Appraisal Process Protocols
C-9 January 2017
ATTACHMENT A
ASSESSMENT SCHEDULE
Scoping, Data Collection, Performance Testing, Report Writing, and Closeout Activities
Assessment Phase Date Location Activity
Leadership Scoping [insert date] [insert site] Coordinate assessment activities with site
and DOE/NNSA leadership.
FOF Scoping [insert date] [insert site] Coordinate assessment activities, scenario
development scoping, and logistical
support requirements.
EA Headquarters
Planning Meeting
[insert date] Germantown, MD All topic teams review documents and
plan data collection activities.
Scenario
Development
[insert date] [insert site] Protective Force Performance Testing
Scenario development.
Data Collection/
Performance Testing
[insert date] [insert site] All topic teams continue data collection
and performance testing activities.
Data Collection and
Analysis
[insert date] [insert site] All topic teams continue data collection,
performance testing activities; teams
analyze data and assist in the preparation
of the draft assessment report.
Analysis and Report
Writing
[insert date] [insert site] All topic teams analyze data and prepare
draft assessment report.
Closeout Activities [insert date] [insert site] Assessment team leadership conducts
closeout briefing.
Factual Accuracy
Review Comments
[insert date] [insert site] Comments from the program office are
due to EA.
Office of Security Assessments Appraisal Process Protocols Appendix C
January 2017 C-10
ATTACHMENT B
ASSESSMENT TEAM COMPOSITION
Team Management and Support
Name - Team Chief
Name - Deputy Team Chief
Name - Technical Advisor
Name - Lead Writer
Name - Field Administrative Coordinator(s)
Protection Program Management
Name - Topic Team Lead
Name - Writer
Names of PPM Team Members
Physical Security Systems
Name - Topic Team Lead
Name - Writer
Names of PSS Team Members
Protective Force
Name - Topic Team Lead
Name - Writer
Names of Protective Force Team Members
Material Control and Accountability
Name - Topic Team Lead
Name - Writer
Names of MC&A Team Members
Classified Matter Protection and Control
Name - Topic Team Lead
Name - Writer
Names of CMPC Team Members
Office of Security Assessments Appendix C Appraisal Process Protocols
C-11 January 2017
ATTACHMENT C
PROTECTION PROGRAM MANAGEMENT
LINES OF INQUIRY
S&S Management and Administration
1) Has the program office and site management provided adequate direction in the areas of Delegations
of Authority, Directives Implementation, and Deviations from Departmental requirements?
2) Has management implemented a personnel development and training program?
S&S Planning
1) Does the site have an SSP/facility security plans, and do they reflect site operations and describe all
aspects of S&S operations?
2) Are security plans supported by a sufficient and documented analytical basis to ensure that protection
requirements can be met?
3) Is the DOE Tactical Doctrine adequately applied?
4) Has an immediately implementable Security Conditions Response Plan been developed?
S&S Program Assurance
1) Are surveys and self-assessments conducted to ensure that S&S systems and processes at facilities/sites
are operating in compliance with Departmental-, national-, and site-level requirements?
2) Do the site’s issues management programs provide the means for timely identification, correction of
deficiencies, and validation of the effectiveness of corrective actions implemented to prevent
recurrence?
3) Has the site established a PAP that identifies the essential elements of the protection program and
establishes monitoring and testing activities of sufficient rigor to ensure that the program elements are
at all times operational and functioning as intended?
Office of Security Assessments Appraisal Process Protocols Appendix C
January 2017 C-12
ATTACHMENT D
PHYSICAL SECURITY SYSTEMS
LINES OF INQUIRY
Systems Management
1) Does the physical protection system protect Departmental elements from malevolent acts, and are
security areas established?
2) Is there an effective means of assessing alarms?
3) Does management provide adequate resources, including personnel (plus training), equipment, and
facilities to meet the requirements contained in the procedures and policies?
Access Control
1) Are personnel and vehicle access control mechanisms in place and effective?
Vaults and Vault-type rooms
1) Are vaults and vault-type rooms used for open storage of classified mater and/or is SNM effectively
alarmed and controlled?
Locks and Keys
1) Has a lock and key program been established in a graded fashion?
2) Does the program include a use-and-protection strategy for grand master, master, and sub-master, and
have the control keys been evaluated and documented in the SSP?
3) Has an inventory system been implemented to ensure the accountability of Level I, II, and III security
locks, keys, key rings, key ways, and pinned cores?
Testing and Maintenance
1) Are physical protection systems, including components, part of a scheduled testing and maintenance
program?
2) Is screening equipment capable of detecting prohibited and controlled articles before being permitted
into DOE facilities?
Barriers
1) Are barrier delay mechanisms used to deter and delay access to, and removal or unauthorized use of,
Category I/II SNM, and do these mechanisms reduce reliance on recapture/recovery operations?
Communications
1) Do systems remain operable during the loss and recovery of primary electrical power?
2) Is the radio system designed to resist eavesdropping and transmission of deceptive messages and to
protect against jamming?
Closed-Circuit Television and Lighting
1) Does the lighting system allow for detection and assessment of unauthorized persons?
Intrusion Detection Systems
1) When protecting Category I/II SNM, is intrusion detection and assessment immediate?
2) Has the intrusion detection system been designed, installed, operated, and maintained to ensure that the
number of false and nuisance alarms do not reduce system effectiveness and to deter adversaries from
circumventing the detection systems? Have gaps in detection coverage been eliminated?
3) If an early warning intrusion detection system is used to supplement the perimeter intrusion detection
and assessment system, have false and nuisance alarm rates been established that do not degrade the
Office of Security Assessments Appendix C Appraisal Process Protocols
C-13 January 2017
overall effectiveness of the system, including personnel monitoring the system’s ability to assess and
manage the alarms?
4) Does the intrusion detection system have primary electrical power from onsite power?
5) Can a single person, such as a system administrator or maintenance technician, modify settings to
obscure or prevent alarm signals from being properly identified by the alarm monitoring system
operator?
Entry/Exit Screening
1) Is an inspection program documented in the SSP that ensures prohibited and controlled articles are
detected and DOE assets are not removed when entering/exiting the Protected Area or material access
area?
2) Has the facility analyzed the potential for an adversary to use explosives to affect such consequences
as sabotage or theft? If warranted, have protection measures been identified, approved by the Officially
Designated Security Authority, and documented in the SSP?
Badge Office Operations
1) If local site-specific only badges are used, has DOE line management prescribed or approved
procedures for their design issuance, use, accountability, and return?
2) Does the site maintain records of issued badges, including disposition, description and badge number,
date of issuance, name, organization, and date of destruction and destruction certificate?
3) Are records maintained showing disposition of credentials and shields to include those that are lost? If
lost, is it reported immediately?
Office of Security Assessments Appraisal Process Protocols Appendix C
January 2017 C-14
ATTACHMENT E
PROTECTIVE FORCE
LINES OF INQUIRY
Management
1) Do directives, plans, and general and special orders meet DOE requirements for currency, clarity, and
applicability to site-specific standards?
2) Do protective force programs, functions, or activities incorporate basic planning principles to ensure
that they accomplish their intended purpose, in accordance with DOE S&S requirements?
3) Is sufficient operational guidance provided through the establishment and maintenance of a formalized
written directives system to ensure that protective force missions/functions are accomplished as
intended?
4) Do protective force personnel comply with the Departmental medical, physical readiness, and firearms
qualifications and training requirements outlined in 10 CFR Part 1046?
5) Are protective force self-assessment and corrective action programs adequately implemented?
6) Are adequate numbers of supervisors assigned to all shifts, and do the supervisors provide the necessary
level of supervision required by local and DOE orders and policies?
Training
1) Is training conducted in accordance with a DOE NTC-approved program in accordance with the
training approval program and provided to ensure performance of assigned functions and tasks under
both normal and emergency conditions?
2) Is the formal training and qualification program based on a valid and complete set of job tasks, with
identified levels of skills and knowledge needed to perform the essential functions outlined in 10 CFR
Part 1046?
3) Are the knowledge, skills, and abilities necessary to competently perform the tasks associated with
assigned protective force duties identified based upon the job analysis/Mission Essential Task List
applicable to each job assignment?
4) Do protective force personnel who are assigned instructor duties have current certification to the level
of training delivered?
5) Have all firearms instructors successfully completed the DOE Firearms Instructor course offered at
NTC (instructors for Security Police Officer IIs and IIIs must complete the Advanced Weapons Systems
Instructor Certification course)?
Equipment and Facilities
1) Is the protective force equipped and provided with the necessary resources to effectively, efficiently,
and safely perform both routine and emergency duties in daylight or under reduced visibility
conditions?
2) Is equipment, specifically weapons and communications systems, tailored to effectively combat and
defeat adversaries identified in the Department’s threat policy (i.e., DOE Order 470.3B, Graded
Security Protection Policy, and DOE Order 470.3C, Design Basis Threat Policy, as applicable) and
site-specific threat guidance or as specified in the SSP under all environmental and tactical conditions?
3) Does the site have an armorer with the knowledge, capability, and responsibility for inspecting,
maintaining, and repairing all firearms available for use?
4) Are permanent (routine and emergency duty) posts that control access constructed consistent with the
vulnerability analysis as documented in the SSP?
5) Are suitable training facilities, to support applicable protective force activities, provided and maintained
based on mission-specific needs?
Office of Security Assessments Appendix C Appraisal Process Protocols
C-15 January 2017
Duties
1) Do protective force personnel demonstrate familiarity with, and knowledge of, the responsibilities
identified in the job analysis/Mission Essential Task List for their assignment and demonstrate
proficiency in the skills and abilities necessary to perform required and assigned job tasks?
Emergency Response/Emergency Management
1) Has the site developed and implemented an integrated and comprehensive Emergency Management
Plan that identifies protective force requirements?
2) Can the protective force respond effectively and efficiently to all operational emergencies and provide
emergency assistance so that appropriate response measures are taken to protect the worker, the public,
the environment, and national security?
Performance Testing
1) Does the PAP identify the essential elements of the protection program and establish monitoring and
testing activities with sufficient rigor to ensure that the program elements are at all times operational,
functioning as intended, and interacting in such a way as to identify and preclude the occurrence of
adverse activity before security is irreversibly compromised?
2) Does the performance testing program include operability tests to confirm, without any indication of
effectiveness, that a system element or total system is operating as expected?
3) Does the performance testing program include the conduct of effectiveness tests to provide assurance
that essential elements of the system are working as expected, separately or in coordination, to meet
protection program objectives?
Performance Testing Safety
1) Are tests documented in a test report that includes a narrative description of the testing activity and an
analysis of test results?
2) Are issues requiring corrective action documented and tracked until resolved?
3) When unsatisfactory results of a test indicate that national security and/or the health and safety of
facility/site employees or the public is jeopardized, are immediate compensatory measures taken until
the issue is resolved and are normal reporting procedures followed?
4) Does the S&S program incorporate a risk-based approach to protect the health and safety of DOE
Federal and contractor employees or the public?
5) Does the site conduct timely collection, reporting, analysis, and dissemination of information on
environment, safety, and health issues as required by law or regulations or as needed to ensure that
DOE and NNSA are kept fully informed on a timely basis about events that could adversely affect the
health and safety of the public or the workers?
Demonstrator Protestor
1) Has the site developed and implemented plans and procedures to identify demonstrators or protestors
and execute the appropriate response in accordance with DOE policy?
Workplace Violence Active Shooter
1) Has the site developed, in accordance with the DOE Tactical Doctrine, a site-specific plan for managing
site personnel and procedures for managing the disposition of workers in the event of a workplace
violence incident or active shooter attack scenario?
Office of Security Assessments Appraisal Process Protocols Appendix C
January 2017 C-16
ATTACHMENT F
MATERIAL CONTROL AND ACCOUNTABILITY
LINES OF INQUIRY
Program Management
1) Does site management provide sufficient focus and resources to adequately safeguard nuclear material?
2) Are SNM quantities and categories accurately characterized and appropriate levels of protection
provided?
3) Is the MC&A program adequately documented and approved by DOE?
4) Are personnel trained and qualified to perform MC&A-related activities?
5) Is the effectiveness of the MC&A system periodically verified?
6) Are systems established to maintain accountability of SNM?
7) Are detection systems in place to respond to and report potential losses of SNM?
Material Control
1) Are material balance areas and personnel responsibilities defined, and are internal controls
implemented?
2) Are systems in place to detect the loss or diversion of SNM in a timely manner?
3) Are category limits observed?
4) Is a graded approach in place for SNM use, storage, and protection?
5) Is termination of safeguards properly applied to designated material?
6) Is a comprehensive tamper-indicating device program established and implemented?
7) Do transfers and shipments of SNM include appropriate controls and documentation?
8) Are controls in place to prevent unauthorized removal of SNM in waste streams?
Measurements
1) Is the technical basis for measurements and measurement uncertainty qualified, validated, documented,
and approved by DOE?
2) Are sources of significant measurement uncertainty, including sampling errors, identified and used to
estimate random and systematic errors?
3) Is a statistically-based measurement control system implemented?
4) Are key measurement points established and included in operations procedures?
5) Are uncertainties appropriate and routinely verified to be consistent with approved target values?
6) Are measurement systems, including traceable standards, calibrated and re-calibrated on a defined
frequency?
Material Accounting
1) Are accurate records of nuclear materials maintained, and are transactions and adjustments made in the
system?
2) Are account structures established and maintained?
3) Are the quality, integrity, and capability of the accountability system maintained?
4) Is reporting to Nuclear Materials Management and Safeguards System appropriate?
5) Is a physical inventory regularly conducted?
6) Are physical inventories conducted at defined frequencies?
7) Are sampling approaches used to conduct physical inventories statistically valid?
8) Is the physical inventory process timely and comprehensive?
9) Is the physical inventory based on measured values?
10) Does the site have the capability to conduct emergency physical inventories?
11) Are inventory differences evaluated and investigated?
Office of Security Assessments Appendix C Appraisal Process Protocols
C-17 January 2017
ATTACHMENT G
CLASSIFIED MATTER PROTECTION AND CONTROL
LINES OF INQUIRY
Classified Matter Protection and Control
1) Has the information security program or CMPC program developed adequate procedures to ensure that
classified information in all forms is protected in accordance with all applicable laws, regulations,
policies, directives, and other requirements?
2) Does the information security program or CMPC program ensure that all individuals with authorized
access to classified information receive instruction with respect to their specific security duties as
necessary to ensure that they are knowledgeable about their responsibilities and applicable?
3) Has the information security program or CMPC program established handling and protection
procedures for classified information throughout its lifecycle?
4) Does the information security program or CMPC program ensure that access to classified information
requires appropriate clearance, relevant access approval, and need to know?
Incidents of Security Concern
1) Has site management established an IOSC process that includes timely identification, notification,
inquiry, reporting, and closure of IOSC?
2) Has a site IOSC program plan been developed that addresses the components of an IOSC program as
outlined in DOE Order 470.4B, Safeguards and Security Program?
3) Does the site’s IOSC program integrate with the site’s PPM function for the purpose of influencing
other programs (i.e., performance assurance, self-assessment, and oversight) and enhancing site-
specific implementation of security policies?
4) Does the site’s issues management programs provide the means for timely identification and correction
of deficiencies in non-compliant conditions to prevent adverse events and validate the effectiveness of
corrective actions implemented to correct identified deficiencies?
Foreign Ownership and Control or Influence (FOCI) and Facility Clearance (FCL)
1) Does the site meet the requirements to possess and secure classified matter or SNM (and, as applicable,
to protect other assets and conduct other security activities on behalf of DOE)?
2) Have the prime contractors implemented provisions pertaining to subcontractors and that all
subcontractors are processed for facility clearances when required and terminated or transferred as
appropriate?
3) Are FOCI and FCL procedures documented to ensure changes to key management personnel are
verified as they occur and that access authorizations are immediately processed for a new key
management personnel?
4) Does the responsible program ensure that the Safeguards and Security Information Management
System database accurately reflects established facilities, security assets, and activities under the local
Federal office’s jurisdiction; ensure that updates and changes to such information are recorded in this
database immediately; and ensure that accurate forms are submitted for this purpose?
Personnel Security, Workplace Substance Abuse, HRP, S&S Awareness, and Unclassified Foreign
Visits and Assignments
1) Does the site’s access authorization process ensure that pre-employment screening, clearance requests,
terminations, and reporting requirements are performed properly and efficiently?
2) Does the site’s workplace drug and alcohol program ensure a drug free workplace?
3) Does the site’s HRP ensure that individuals who occupy designated positions meet the highest standards
of reliability and mental and physical suitability?
Office of Security Assessments Appraisal Process Protocols Appendix C
January 2017 C-18
4) Does the security awareness program communicate security responsibilities to all individuals and
provide a means to continuously instruct individuals with access authorizations in their duties and
responsibilities?
5) Does the unclassified foreign visits and assignments program provide a process to ensure that
unauthorized access is denied and approved visits and assignments are documented and tracked?
Office of Security Assessments Appendix D Appraisal Process Protocols
D-1 January 2017
Appendix D – Appraisal Scheduling Process
The U.S. Department of Energy (DOE) Office of Security Assessments (EA-22), within the Office of
Enterprise Assessments (EA), adheres to strategic planning processes outlined in the Independent Oversight
Program Appraisal Process Protocols. However, EA-22 has developed additional guidance to ensure
adequate coverage and efficiency in the scheduling process, given the complexity of scheduling appraisal
activities across physical and information security programs throughout the Department. This appendix
provides EA-22 staff with detailed expectations and milestones for each phase of the appraisal scheduling
process. The key elements affecting EA-22’s scheduling process are the comprehensiveness of data
considered, site and program office involvement, and transparency.
EA-22 generally conducts appraisal activities every two to three years at sites that protect Category I
quantities of special nuclear material or highly sensitive classified information. To ensure appropriate
prioritization and scheduling of assessments for the sites and facilities with the highest potential risk, EA-
22 uses a formal process for collecting and analyzing a broad range of protection program performance
data, and for coordinating with the applicable program office and site security officials to identify potential
areas of management interest and any potential schedule conflicts.
The analysis process begins approximately six months before each new calendar year starts. This analysis
is based on specific factors (see Appendix E, right column in Table E-1), which EA-22 evaluates using
available empirical data and professional judgment. One product of the analysis is a priority designation
(based on the level of perceived risk) to guide the determination of each site’s priority and how often it will
be subject to scheduled assessment activities. EA-22 uses this information in the context of available time
and resources to develop an assessment schedule for the upcoming calendar year and an initial
determination of the type and scope of each proposed assessment. The assessment schedule includes
proposed dates and estimated resources to support physical security assessments as well as information
security assessments, such as sensitive compartmented information (SCI) facility assessments and classified
matter protection and control (CMPC) special reviews.
After the development of the proposed schedule and resources, EA-22 management convenes internal
meetings with physical and information security subject matter experts and senior EA management to
further evaluate and refine the schedule. Follow-on meetings with the appropriate Headquarters program
offices and site security officials, generally held between June and August of each calendar year, provide
these senior management stakeholders with the proposed EA-22 schedule and an opportunity to
communicate input on desired focus areas or relay feedback on the proposed appraisal dates, type, and
scope. Before discussing plans for limited-notice performance test (LNPT) activities with any personnel
outside of EA, EA-22 requires completion of a trusted agent form to help protect the confidentiality of the
proposed test subject areas and dates.
After the scheduling meetings, EA-22 finalizes the appraisal schedule and formally coordinates with the
program offices and within EA to reconcile any schedule conflicts. The Director, EA-22, then forwards
the final schedule to senior EA management for approval, along with recommendations and the rationale
for any adjustments to the prioritization and/or deferral of assessment activities. After the Director of EA
approves the schedule, EA-22 distributes it to the various DOE program offices, usually by mid-November
of each calendar year.
The milestones and responsible individuals for the actions associated with the phases of the EA-22 appraisal
scheduling process are described below.
Office of Security Assessments Appraisal Process Protocols Appendix D
January 2017 D-2
Milestones and Responsible Individuals
May: Director, EA-22, hosts a meeting at Headquarters to develop an initial schedule and resources:
Receive EA management scheduling guidance.
Review assessment and survey reports.
Review incident reports and enforcement records.
Conduct informal discussions with program offices and site security managers regarding potential
appraisal activities.
Develop an initial schedule of physical/information security assessments and reviews, SCI facility
assessments, and CMPC special reviews, and identify necessary resources.
Brief/provide initial schedule to EA management.
June – August: EA-22 hosts formal scheduling conferences or workshops with the National Nuclear
Security Administration and the DOE Offices of Environmental Management, Science, Nuclear Energy,
and Intelligence and Counterintelligence:
Receive program offices’ input and requests for support.
Ensure that program office personnel sign trusted agent forms where necessary.
Share initial schedule (minus LNPT activities).
Review and analyze recent performance data.
Brief EA management and receive additional guidance.
September: Director, EA-22, integrates and finalizes the appraisal schedule:
Address program office and EA management comments.
October: Director, EA-22, forwards the schedule to EA management for approval.
November: Director, EA-22, transmits the approved annual schedule to program offices and affected field
sites.
Office of Security Assessments Appendix E Appraisal Process Protocols
E-1 January 2017
Appendix E – Factors for Determining a Site’s Appraisal Priority, Type, and Scope
Table E-1. Factors for Determining a Site’s Appraisal Priority, Frequency, Type, and Scope
Priority and Frequency Factors for Determining Appraisal Type and Scope
Priority I:
Sites with high-value assets or with
higher potential risk; includes all sites
that store and use Category I special
nuclear material (SNM) and/or high
consequence classified information
(e.g., Top Secret and sensitive
compartmented information,
Secret/Restricted Data, and/or Sigma
14, 15, 18, and 20).
Under non-emergency conditions,
multi-topic assessments at Priority I
sites are typically conducted every 30
months.
Asset characterization and site mission: The types and
attractiveness of security interests located at a U.S.
Department of Energy (DOE) site and the associated
risks (or perceived risks) are significant considerations in
determining the scope and type of appraisal activity to be
conducted.
Changes in site mission and operations: Significant
changes in site mission/operations, site geography, or the
site security program contract structure are factors that
the Office of Security Assessments (EA-22) evaluates in
the appraisal scoping process.
Changes in policy: DOE policy and other applicable
national standards represent the fundamental measure by
which EA-22 conducts its appraisal activities. Changes
in policy can significantly impact site operations and are,
therefore, important considerations in determining
appraisal scope.
Previous evaluation results: Previous Office of
Enterprise Assessments (EA) assessment results and
regulatory enforcement actions, DOE Inspector General
inspections, site/operations office surveys, and
contractor assurance activities, as well as evaluation
results from external agencies, provide valuable insights
that EA-22 considers during the appraisal scoping
process.
Incidents of security concern: Recent/ongoing major
vulnerabilities, significant incidents, or chronic
recurrence of lesser incidents influence EA-22’s scoping
and prioritization process.
Line management input: EA-22 is committed to support
Headquarters and field line managers in accomplishing
their safeguards and security responsibilities and to assist
them in achieving DOE mission objectives. Therefore,
input regarding requested focus areas from Headquarters
and field line management is an especially important
factor in determining appraisal priority, type, and scope.
Priority II:
Sites with medium-value assets or with
lower potential risk; includes sites with
Category II or III SNM in accessible
and transportable form and/or Secret
(and lesser) classified matter.
Under non-emergency conditions,
multi-topic assessments at Priority II
sites may be conducted every 36 to 48
months.
Priority III:
Sites primarily with non-national
defense missions, or possessing less
than Category III quantities of SNM
and/or very limited classified matter.
Assessment type and frequency at
Priority III sites are typically directed
by EA management on an “as needed”
basis consistent with insights gained in
reviewing the factors for determining
appraisal type and scope.
Office of Security Assessments Appraisal Process Protocols Appendix F
January 2017 F-1
Appendix F – Data Analysis, Deficiencies, Findings, Opportunities for Improvement, Recommendations, Best Practices, and Ratings
Data Analysis
The U.S. Department of Energy (DOE) Office of Security Assessments (EA-22), within the Office of
Enterprise Assessments (EA), conducts appraisals to assess the extent to which DOE sites comply with
current program requirements and how well site protection programs are performing. While DOE security
policy requires sites to perform certain functions and achieve certain levels of protection, policy does not
always contain specific information on what measures must be taken or how to achieve an appropriate level
of protection. Therefore, EA-22 appraisal activities emphasize an approach that builds on both compliance
and performance measures to make the most accurate evaluation of the effectiveness of the protection
program.
While analysis is an ongoing process during all phases of an appraisal, it culminates during the reporting
phase. Analysis involves a critical review of all data collection results, particularly any identified program
strengths or deficiencies (inadequacies), and leads to logical, supportable conclusions regarding how well
the program functions and satisfies the intent of DOE and national security policy.
In accordance with DOE Order 227.1A, EA-22 documents appraisal results that clearly identify findings,
opportunities for improvement, and ratings. When applicable, EA-22 has chosen to also document program
shortfalls that do not impact risk and performance (such as the failure to meet a compliance requirement)
as deficiencies. A detailed discussion of deficiencies, findings, and opportunities for improvement is
provided below.
Deficiencies
A deficiency is an inadequacy (e.g., failure to implement a requirement or meet a performance standard)
identified during an appraisal. Deficiencies may serve as the basis for one or more findings.
When the assessment team identifies deficiencies, the analysis must consider these weaknesses individually
and collectively, and the team must then evaluate/analyze them in the context of identified strengths or
mitigating factors to determine their overall impact on the program’s effectiveness. Once a deficiency has
been identified, the team must determine its significance and how it will be documented in the appraisal
report. Factors considered during analysis include:
The magnitude and significance of the actual or potential vulnerability to DOE security interests
resulting from the deficiency.
The deficiency’s actual or potential effect on mission performance.
Site knowledge of the deficiency and corrective actions. For example, findings may be warranted if
management self-identified a weakness but did not implement timely or appropriate corrective actions.
Whether the observed deficiency is isolated or systemic. Systemic deficiencies – those that are
widespread, chronic, or recurring – are more serious.
Other effective programs or program elements that could mitigate the impact of a deficiency.
Figure F-1 at the end of this appendix presents a tool to help EA-22 assessors determine the significance of
a given deficiency.
Office of Security Assessments Appendix F Appraisal Process Protocols
F-2 January 2017
Examples of deficiencies that do not rise to the level of a finding:
(Note: There is no correlation between the example deficiencies; they are separate, standalone examples.)
Scenario: The assessment determined that Federal personnel conduct periodic safeguards and security
surveys and identify appropriate, meaningful findings. Further, the contractor conducts adequate causal
analyses of the findings, develops appropriate milestones for their correction, closely monitors corrective
actions to ensure that they are properly accomplished in a timely manner, and verifies the efficacy of the
corrective actions. Even though the attendant records of these activities are properly entered into a
contractor corrective actions management system, the findings are not being entered into the Safeguards
and Security Information Management System.
Deficiency: The field office has not entered and tracked survey findings in the Safeguards and
Security Information Management System as required. [DOE Order 470.4B, Appendix A, Section 2,
paragraph 9.b]
Scenario: EA’s review of contractor’s training approval program (TAP) certificate indicated that it was
last approved in 2009. Further review determined that the contractor had taken no action to submit an
updated TAP request packet to the National Training Center (NTC) for review and
approval/certification. No other significant deficiencies in training or training performance were noted.
Deficiency: The contractor has not ensured that the TAP certification is current or was submitted
to the NTC for approval. [DOE Order 473.3A, Attachment 2, Section B, paragraph 1.a]
Scenario: Classified matter is stored in approximately 150 General Services Administration-approved
security containers and vault-type rooms throughout the site. EA’s review determined that all repositories
contained the proper documentation, and adequate physical protection for classified matter. Due to a recent
voluntary reduction in force at the site, many repository custodians have terminated employment, requiring
a large number of combination changes. Except for a single contractor organization, most combinations
have been changed in a timely manner. At the time of the assessment, there were six repository access lists
containing names of 12 personnel no longer employed by the contractor. Further review determined that
the combinations for these six repositories had not been changed for over two months. As partial mitigation
of this condition, contractor management had previously recovered the terminated employees badges and
canceled their access privileges in the automated entry control system to help prevent unauthorized access
to the repositories.
Deficiency: The contractor did not change classified repository combinations as soon as practical to
preclude access by individuals who no longer have a need to know. [DOE Order 471.6, Change 2,
paragraph 4.b.(5)(d)4]
Findings
Findings are deficiencies that warrant a high level of attention on the part of management. If left
uncorrected, findings could adversely affect the DOE mission, the environment, worker safety or health,
the public, or national security. Findings define the specific nature of the deficiency and whether it is
localized or indicative of a systemic problem, and identify which organization is responsible for corrective
actions. Consequently, findings provide for a high level of management attention and focus on improving
protection program performance. Team members are responsible for determining which appraisal results
are designated as findings. Minor and non-systemic deficiencies must be appropriately identified so that
they can be corrected, but they are normally not designated as findings.
Office of Security Assessments Appraisal Process Protocols Appendix F
January 2017 F-3
EA-22 documents findings by discussing the specific problem, identifying all relevant factors, describing
the impact on protection, and referencing the appropriate requirement. Assessment team members are
reminded that with the publication of revised DOE safeguards and security directives and policies, many
of the specific requirements (regarding how to meet program objectives) contained in prior directives were
not carried over into the current directives. Therefore, findings may have to be based on overall program
goals, objectives, and requirements; local implementing procedures; or relevant Federal standards.
If multiple identified deficiencies address specific aspects of a single standard, they should be “rolled up”
and reported as a single finding if the single finding statement can clearly and completely convey the
problems observed. Finding narratives are formulated to express the specific nature of the deficiency,
clearly indicate whether the deficiency is localized or indicative of a systemic problem, and identify which
organization (DOE Headquarters or field element, facility contractor, etc.) is responsible for corrective
actions. In the appraisal report, each finding typically appears after a discussion of the impact of the
condition described, including any mitigating factors and compensatory measures.
Examples of findings:
(Note: As in the examples of deficiencies, these are standalone examples that are unrelated to one another.)
Scenario: This assessment determined that several findings from the last EA appraisal either were not
corrected or had recurred. Such repeat weaknesses included incomplete vulnerability assessment reports
(which continued to omit the same necessary items); continued excessive false and nuisance alarms
prompted by inadequate preventive or corrective maintenance; and the protective force’s inappropriate use
of force. EA found that the contractor does not monitor and track issues to ensure that causal factors,
corrective action milestones, timeliness of actions, or effectiveness reviews are appropriately analyzed so
that issues are corrected in a manner that prevents recurrence. As a result, the organizations continue to
experience systemic problems that impact multiple protection programs.
Finding: The contractor has not implemented timely, effective corrective actions for issues identified
in self-assessments and surveys and has not validated the effectiveness of corrective actions to prevent
recurrence of the issues. [DOE Order 474.B, Attachment 2, Section 2, paragraph 5.f ]
Scenario: This assessment determined that protective force personnel could not demonstrate the requisite
skills in using multiple firearms systems. While on the range, several protective force officers failed to
keep their M-4 rifle muzzles pointed downrange at all times when on the firing line, while others could not
(or did not) take immediate actions to correct rifle malfunctions. Further, only 70 percent of the shooters
were able to qualify with the M-203 40mm grenade launcher on the first attempt. These examples represent
a systemic problem in weapons training that significantly impacts protective force readiness.
Finding: The contractor protective force did not demonstrate the requisite firearms proficiency with
the M-4 rifle and the M-203 40mm grenade launcher. [DOE Order 473.3A, Attachment 2, Section
B, paragraphs 2.a and 5.a]
Scenario: EA determined that the site has developed the required human reliability program (HRP)
implementation plan covering most persons identified in 10 CFR 712, and is implementing the HRP in
accordance with that plan. However, the protective force armorers were not enrolled in the program.
Because these individuals have unescorted access to the weapons that the protective force relies on to
perform its mission, they represent a significant potential insider threat capable of rendering the protective
force ineffective.
Office of Security Assessments Appendix F Appraisal Process Protocols
F-4 January 2017
Finding: The contractor has not ensured that armorers who have unescorted access to firearms are
enrolled in the HRP. [DOE Order 473.3A, Attachment 2, Section A, paragraph 3.b]
Opportunities for Improvement
EA assessors have a broad range of knowledge in their individual topical areas of expertise, and also have
the advantage of observing methods of program implementation across the entire DOE complex. When
they identify deficiencies or inefficiencies in program implementation during an appraisal activity, they can
provide insights on approaches that line managers could adopt to improve program performance. Often,
these suggestions are based on successful approaches observed at other DOE sites. EA-22 identifies such
opportunities for improvement in appraisal reports; however, they are provided only in the context of
recommendations for consideration by line managers, not as directed actions. Opportunities for
improvement that correlate to findings or deficiencies are normally provided as suggested approaches that
line management may consider in their corrective action plans.
Examples of opportunities for improvement:
Protective force management should consider enhancing overall firearms proficiency for Site X
security police officers (SPOs). Specific actions to consider include:
Temporarily increasing the frequency of SPO training and qualifications for the M-4 rifle, 40mm
grenade launcher, and M-249 machinegun until negative performance trends no longer exist.
Developing and implementing practical shooting courses of fire (in a simulated combat course
environment) for the M-4 rifle, 40mm grenade launcher, and M-249 machinegun to enhance SPOs’
proficiency with these firearms and to prepare SPOs to use these firearms more effectively in the
event of an actual adversary attack.
Site information technology managers should consider modifying the vulnerability scanning process
to provide more timely information on actual protection status. Technical actions to consider include:
Conducting network scanning more often than quarterly.
If possible, conducting vulnerability scans monthly as soon as possible after the regular software
patches are applied and virus definitions are updated, to minimize the number of false positives
created by slightly outdated patches or virus definition files.
Recommendations, Best Practices, and Ratings
The EA Independent Oversight Program Appraisal Process Protocols provides guidance on
recommendations, best practices, and ratings.
Office of Security Assessments Appraisal Process Protocols Appendix F
January 2017 F-5
RD
= R
estr
icte
d D
ata
SA
P =
Sp
ecia
l A
cces
s P
rogra
m
SC
I =
Sen
siti
ve
Com
par
tmen
ted
In
form
atio
n
SN
M =
Sp
ecia
l N
ucl
ear
Mat
eria
l
Fig
ure
F-1
. D
efic
ien
cy S
ign
ific
ance
Det
erm
inat
ion
Pro
cess