earned value management system (evms) surveillance ...€¦ · evms surveillance sop september 26,...

EARNED VALUE MANAGEMENT SYSTEM (EVMS)

SURVEILLANCE STANDARD OPERATING PROCEDURE

(ESSOP)

DEPARTMENT OF ENERGY (DOE)

OFFICE OF ENGINEERING AND CONSTRUCTION

MANAGEMENT (OECM)

SEPTEMBER 26, 2011

EVMS SURVEILLANCE SOP September 26, 2011 OECM i

TABLE OF CONTENTS

TABLE OF CONTENTS .............................................................................................................................. i

1. PURPOSE................................................................................................................... 1

2. REFERENCES .......................................................................................................... 1

3. DOE ORDER 413.3B EVMS SURVEILLANCE REQUIREMENTS ................. 2

4. EVMS SURVEILLANCE ROLES AND RESPONSIBILITIES .......................... 2

4.1. OECM .................................................................................................................................. 2

4.1.1. OECM EVM Specialist .................................................................................................... 3

4.1.2. OECM Project Analyst..................................................................................................... 3

4.2. Project Management Support Office (PMSO) ..................................................................... 4

4.3. Federal Project Director (FPD) ............................................................................................ 4

4.4. Contractor ............................................................................................................................ 5

4.5. Contracting Officer .............................................................................................................. 6

5. EVMS SURVEILLANCE PLANNING AND BUDGETING PROCESS ............ 6

6. EVMS SURVEILLANCE PROCESS ..................................................................... 6

6.1. Surveillance Scope ............................................................................................................... 9

6.2. Surveillance Schedule .......................................................................................................... 9

6.3. Team Composition ............................................................................................................... 9

6.4. Artifacts and Information ..................................................................................................... 9

6.5. Conduct of Surveillance ..................................................................................................... 10

6.6. Stage 1 Surveillance – Risk Assessment and On-going Monthly Analysis....................... 13

6.6.1. Data Analysis ................................................................................................................. 13

6.6.2. Assess Project and Project Risk and Develop Risk Profile ............................................ 16

6.6.3. Develop Stage 2 and 3 Surveillance Schedule ............................................................... 16

6.7. Stage 2 Desk Top Surveillance .......................................................................................... 17

6.7.1. Surveillance Scoping Meeting ....................................................................................... 17

6.7.2. Team Selection ............................................................................................................... 18

6.7.3. Roles and Responsibilities ............................................................................................. 18

6.7.4. Project Selection ............................................................................................................. 19

6.7.5. Contractor Documentation and Artifact Review ............................................................ 19

6.7.6. Conduct Data Traceability ............................................................................................. 20

EVMS SURVEILLANCE SOP September 26, 2011 OECM ii

6.7.7. CAM, Program/Project Staff Interviews ........................................................................ 20

6.8. Stage 3 On-Site Surveillance ............................................................................................. 20

7. CORRECTIVE ACTION REQUESTS (CAR) AND CONTINUOUS IMPROVEMENT OPPORTUNITIES (CIO) .............................................................. 21

7.1. Types of CARs ................................................................................................................... 21

7.2. Documenting CARs ........................................................................................................... 21

7.3. Corrective Action Plan (CAP) Review .............................................................................. 22

7.4. CAR Verification and Closure ........................................................................................... 22

7.5. CAR/CIO Database ............................................................................................................ 23

8. EVMS SURVEILLANCE RESULTS DOCUMENTATION .............................. 23

8.1. EVMS Standard Surveillance Report (SSR) Sections ....................................................... 23

8.2. EVMS Standard Surveillance Report (SSR) Distribution ................................................. 25

8.3. EVMS Standard Surveillance Report (SSR) Retention ..................................................... 26

8.4. EVMS Report Transmittal ................................................................................................. 26

9. EVMS EVALUATION AND FEEDBACK ........................................................... 26

APPENDIX A —ACRONYMS ....................................................................................... 1

APPENDIX B ---ANSI/EIA-748 GUIDELINES & ORGANIZATION PROCESSES ALIGNMENT ................................................................................................................... 2

APPENDIX C --- CONDUCTING AN EVMS DATA TRACE.................................... 3

Table 1. System Compliance Issues Indicator ............................................................................. 14 Table 2. Desk Top Surveillance Team Roles/Responsibilities ..................................................... 18 Figure 1. Fiscal Year Budget Planning .......................................................................................... 6 Figure 2. Surveillance Process Overview ...................................................................................... 8 Figure 3. DOE Surveillance Process ............................................................................................. 12 Figure 4. Surveillance Annual Planning and Budget Schedule ................................................... 17

EVMS SURVEILLANCE SOP September 26, 2011 OECM 1

1. PURPOSE DOE O 413.3B, the Office of Management and Budget (OMB) Circular A-11, and the Federal Acquisition Regulation (FAR) require implementation of an EVMS on all DOE capital asset projects. This SOP contains many of the best practices that are accepted throughout government and industry in implementing and using EVM. This intent of this EVMS Surveillance Standard Operating Procedure (SOP) is to provide a common understanding of EVMS Industry and Government best practices relating to surveillance activities recommended for use by the Department of Energy (DOE) oversight, program, and project teams, as well as contractors doing business with DOE. Surveillance activities cover the areas of surveillance objectives, suggested processes, roles, and responsibilities. All information contained herein clarifies requirements in DOE O 413.3B. In its simplest form, Earned Value Management (EVM) is the discipline of managing projects successfully. It is the planning and controlling of authorized work to achieve cost, schedule, and technical performance objectives. Special emphasis is placed on efficiency and effectiveness in the execution of work through the development and operation of an EVMS that integrates the application of people, systematic processes, and innovative tools and techniques. EVM helps project managers and their management teams operate more effectively in the execution of risky, high dollar, and complex projects. Successful project management requires well-qualified and highly skilled project managers along with integrated teams backed by management systems that provide immediate access to reliable and accurate data on project costs, schedule, and technical performance. Project managers and their management teams perform best when they are well informed. The surveillance of a contractor’s EVMS is intended to provide all stakeholders, including the Acquisition Executive (AE) and senior leadership within the DOE, as well as Congress with confidence that the EVMS produces reliable, accurate, and timely information that is used to effectively manage cost, schedule, and technical performance and in making informed decisions. The purpose of performing surveillance of a contractor’s certified EVMS is four fold:

(1) Verify the EVM data is useful, timely, and effective (2) Assess whether the data is used to make informed decisions (3) Ensure company processes are being followed (4) Demonstrate continued compliance to the ANSI/EIA-748 guidelines

2. REFERENCES

• American National Standards Institute/Electronic Industries Alliance (ANSI/EIA) 748-B • OMB Circular A-11, Part 7, Capital Programming Guide • Federal Acquisition Regulations 34.2 and 52.234, Earned Value Management Systems • DOE Order 413.3B, Program and Project Management for the Acquisition of Capital

Assets • DOE Guide 413.3-10, Earned Value Management Systems • DOE Guide 413.3-20, Change Control Management


• GAO-09-3SP, GAO Cost Estimating and Assessment Guide – Best Practices for Developing and Managing Capital Program Costs, March 2009

• Department of Defense Earned Value Management Implementation Guide, latest version. • NDIA PMSC, Earned Value Management Systems Intent Guide, 2006 • NDIA PMSC, Surveillance Guide, 2011

Additional guidance and templates referred to in this SOP are available at http://energy.gov/management/office-management/operational-management/project-management/earned-value-management. 3. DOE ORDER 413.3B EVMS SURVEILLANCE REQUIREMENTS DOE Order 413.3B, Project Management for the Acquisition of Capital Assets, requires EVMS surveillance as follows:

• Annual surveillance conducted by the contractor with documentation provided to the DOE contracting officer, Project Management Support Office (PMSO), and OECM.

• During the tenure of a multi-year contract (at the contract midpoint or every 2 years, during contract extensions, or as requested by the AE), surveillance will be accomplished by the PMSO or OECM according to established thresholds with documentation provided to the DOE contracting officer.

• For contracts where the portfolio includes projects with a TPC equal to or greater than $50M but less than $100M, surveillance can be accomplished by the PMSO. If the PMSO does not conduct surveillance, OECM will perform the necessary surveillance.

• For contracts where the portfolio includes projects with a TPC equal to or greater than $100M, or as requested by the AE, OECM will conduct the surveillance.

• OECM will conduct surveillance prior to Critical Decision-3 (CD-3) of a Major System project where the contractor’s EVMS was previously self-certified or PMSO-certified.

4. EVMS SURVEILLANCE ROLES AND RESPONSIBILITIES Effective EVMS surveillance involves all stakeholders to include OECM, PMSO, Federal Project Director (FPD), Contracting Officer (CO), and the contractor, working in an integrated, transparent manner.

4.1. OECM OECM, in cooperation with DOE stakeholders, is responsible for the development and implementation of policies and uniform procedures defining the certification and surveillance process. OECM encourages the full participation and cooperation of all stakeholders. Therefore, all stakeholders may be asked to function as surveillance team members. Stakeholder participation is an essential ingredient to an effective and successful surveillance program. OECM will lead surveillance efforts in accordance with dollar thresholds provided in DOE O 413.3B. Since the scope of EVMS surveillance focuses on continued compliance and implementation, the full contractor portfolio of capital asset projects at a single site where EVMS is applicable will be

http://energy.gov/management/office-management/operational-management/project-management/earned-value-management



included in OECM led surveillances. Coordination of surveillance activities with all stakeholders is vital to avoid duplication of effort, to minimize costs, to minimize disruption to the projects, and to increase communication. Where OECM is the designated surveillance lead either per DOE O 413.3B or per request of the PMSO, OECM will be responsible for the entire surveillance effort to include organizing, coordinating, and leading the surveillance team, defining the scope of the surveillance, closing any CARs, documenting the results, and informing the CO. OECM will provide support to the PMSO led surveillance on contracts with a portfolio of projects with a TPC equal to or greater than $50M but less than $100M. Upon PMSO request or if the PMSO does not conduct the surveillance, OECM will lead the surveillance. For contractors that have previously been certified, continued compliance assessments will be accomplished through the performance of surveillance. For multi-year contracts, PMSO or OECM surveillance activities will take place at the approximate contract midpoint or at a minimum, every two years. In the case of a Major System Project where the contractor has been previously certified, surveillance activities must be conducted prior to CD-3 in accordance with DOE O 413.3B.

4.1.1. OECM EVM Specialist The OECM EVM Specialist is assigned overall responsibility for surveillance of the EVMS and serves as the OECM lead for surveillance team activities. Specific responsibilities include:

• Coordinating surveillance processes with all stakeholders to avoid duplication of effort, minimize cost, and increase communication.

• Planning and conducting EVMS surveillances in accordance with DOE O 413.3B and as further defined in this SOP.

• Ensuring contracts contain the required EVMS FAR clauses. • Determining subcontract surveillance requirements and assessing contractor management

of subcontractors in accordance with EVMS guidelines. • Evaluating contractor proposed alterations to the system, including changes to

documented processes, and procedures. • Monitoring successful closeout of surveillance activities. • Preparing the continued compliance letter for OECM Director’s signature to the CO. • Uploading all surveillance documents to OECM’s repository.

4.1.2. OECM Project Analyst

The OECM Project Analyst is responsible for conducting ongoing project level surveillance and analysis activities, which includes some or all of the following:

• Review and analysis of the contractor’s Contract Performance Report • Review and analysis of the contractor’s Integrated Master Schedule • Review of contract modifications, and correct application of EVM • Management Reserve, Undistributed Budget usage analysis • Baseline stability analysis and verification


• Independent Estimate at Completion • Independent Risk Assessments • Continuous monitoring of correct usage of the organizational structure and work

breakdown structure • Identification of any deficiencies and trends • Identifications of data integrity issues • Integration into the CAR process, including follow up and resolution • Assessments of current and future impacts of non-mitigated risks or unsuccessful

corrective actions • Communication of results and system health to the customer • Coordinating with OECM EVM Specialist regarding EVMS issues which are potentially

compliance related • Serving as a member of EVMS Surveillance Teams

In conducting project level analysis, the OECM Project Analyst plays a key role in providing an early warning of and assessing issues that may involve EVMS processes and implementation. The analyst is responsible for coordinating with the EVM Specialist and working together to resolve data integrity issues and participating in system surveillance activities, which may include site visits, and evaluation of Corrective Action Plan (CAP) approval and closeout.

4.2. Project Management Support Office (PMSO) In accordance with DOE O 413.3B, the PMSO conducts surveillances of contractor EVMS where the contractor’s portfolio includes capital asset projects with a TPC equal to or greater than $50 but less than $100M. The PMSO will provide OECM copies of all surveillance reports. The PMSO may request OECM to conduct the surveillance. The PMSO will be requested to participate as a team member in surveillance activities conducted by OECM on contracts with a portfolio of projects with a TPC greater than $100M.

4.3. Federal Project Director (FPD) While the contractor has the primary responsibility for annual EVMS surveillance, DOE line management (i.e., the FPD/Site Office up through the PMSO) also shares in the responsibility. The FPD/Site Office is encouraged to conduct annual surveillances of the contractor EVMS in a manner that can verify (a) continued compliance with the certified EVMS, (b) that the certified EVMS has been properly implemented, and (c) that the data is timely, accurate, and being used to manage the project. Such surveillances may be conducted jointly with the contractor. If the DOE FPD/Site Office does not conduct joint surveillance, then they should assess the results of the contractor surveillance program to determine if additional DOE surveillances are warranted. In reviewing the results of the surveillances, DOE FPD/Site Office may decide to initiate surveillances or it may request a PMSO led surveillance, or an OECM led surveillance (through its program office).


To confirm data accuracy, the FPD/Site Office conducts periodic physical verifications to ensure that the progress being reported is commensurate with actual progress being incurred, and that the actual costs are being reported. The DOE FPD/Site Office also verifies on a monthly basis that the data from the certified EVMS is accurately uploaded into PARS II. The FPD/Site Office’s surveillance should also closely monitor areas previously identified by CARs to assess effectiveness of actions to prevent reoccurrence. Repeat findings are of particular concern as they may demonstrate an inherent weakness in the management processes and thus warrant more concentrated surveillance. When the PMSO or OECM leads a surveillance review, FPD/Site Office support in accomplishing surveillance is essential. This support includes:

• Keeping the PMSO and OECM informed of actions and matters that could affect system surveillance;

• Assisting in the resolution of problems cited in surveillance reports; • Reviewing, evaluating, and analyzing performance reports and schedules and bringing

system and implementation concerns, and data integrity issues to the attention of PMSO and OECM; and

• Participating as members of the surveillance team as requested.

4.4. Contractor The contractor is responsible for developing, implementing, and maintaining a surveillance program to ensure continued compliance with ANSI/EIA 748. The contractor is also responsible for ensuring that its EVMS is implemented on a consistent basis, is used effectively on all applicable projects, and EVMS clauses are flowed down to subcontractors in accordance with the rules applied to the prime. Pursuant to Appendix C of DOE O 413.3B, the contractor is required to conduct a self-surveillance and provide documentation of the self-surveillance to the contracting officer, PMSO, and OECM confirming the continued compliance of their EVMS with the ANSI/EIA-748. In conformance with industry best practices as detailed in NDIA’S PMSC Surveillance Guide, the contractor surveillance is expected to cover all 32 Guidelines annually. This comprehensive review can be spread throughout the year at the contractor’s discretion. The surveillance should be conducted by a team independent of the contractor’s project team, such as an internal audit group, to avoid potential conflicts of interest. In accordance with the system certification letter from OECM, the contractor is also responsible for notifying OECM in writing of any changes, to their certified EVMS. An acceptable approach to surveillance planning could begin with the establishment of a comprehensive surveillance plan prepared by the contractor and provided for information and comment to the FPD/Site Office and PMSO. The surveillance plan should include a clear scope of surveillance, responsibilities, methods for performance, and schedule. The plan typically spans multiple years, is supplemented by an annual schedule with additional detail regarding the planned surveillances, and project(s) selected for review. EVMS guideline assessment templates, based on industry and government best practices, are available at http://energy.gov/management/office-management/operational-management/project-management/earned-value-management.




4.5. Contracting Officer The DOE CO is responsible for ensuring all applicable EVMS regulatory and contractual requirements, FAR clauses, data item and contract deliverables, and language relating to EVMS is included in the contract. The contracting officer also ensures that contractor performance is integrated with the contract award fee determinations and other mechanisms to ensure pay for performance including the assessment of EVMS implementation, certification, reporting, and project performance. It is incumbent upon the contracting officer, OECM, PMSO, and the FPD(s) to work together to ensure project needs are met and understood. Following successful closeout of HQ surveillance activities, OECM or the PMSO will provide a letter to the CO affirming continue compliance of the contractor’s system. Should a contractor fail to maintain their system, the CO, in consultation with the Program and OECM, may withdraw the EVMS certification and elect to invoke contractual remedies. 5. EVMS SURVEILLANCE PLANNING AND BUDGETING PROCESS On-site visits to support EVMS Surveillance and program analysis activities will be conducted based on risk and data analysis rather than be event driven. Funding estimates to support travel will be based on historical data and lessons learned. Semi-annually by March 30 and September 30, OECM will provide surveillance support funding estimates for the current fiscal year, as well as a two-year projection for succeeding budget years. OECM EVMS analysts will coordinate with the OECM program analysts, PMSO, and FPD, to identify contractors that have one or more projects greater than or equal to $100M. EVMS Surveillance for projects less than $100M will be funded by the Program requesting the review (Fig. 1).

Figure 1. Fiscal Year Budget Planning

6. EVMS SURVEILLANCE PROCESS The surveillance process selected for use by OECM- led reviews as described in this SOP is modeled after the risk based, data driven EVMS surveillance approach widely endorsed by


industry groups and Governmental agencies, such as National Defense Industry Association (NDIA), General Accounting Office (GAO), Department of Defense (DoD), and Energy Facility Contractor’s Group (EFCOG). OECM’s surveillance process is recommended for use for PMSO-led reviews as well. The surveillance process focuses on areas identified as a result of risk and data analysis. Surveillance may be routine, i.e. continual, or situational as in an Implementation Review or Review for Cause (RFC). Regardless of the purpose of review, i.e. routine surveillance, implementation, or RFC, the basic process of risk evaluation and data analysis is the same. The difference is that the scope, depth, and rigor may be tailored to suit the situation prompting the review. The EVMS surveillance process differs from the EVMS certification process in that the certification review includes all guidelines from both a procedural and implementation perspective, and a portion of the review is conducted on site to validate integration of tools, conduct interviews, and interact with users. The EVMS surveillance process begins with reviewing and analyzing EV data and other artifacts, including reports from recent project reviews. An EVMS risk assessment is also conducted to identify areas of EVMS risk in each project. If the data and/or risk warrant a deeper look, then a desk audit may be conducted to gain more insight to include phone/VTC interviews with contractor and FPD staff, and review of additional supporting data requested from the contractor. Should areas of concern arise that cannot be sufficiently addressed off site, then an on-site review team may be assembled to focus on those remaining areas of concern. This tailored, focused review with subject matter experts based on the particular needs based on risk and analysis indicators is similar to the Project Peer Review process used throughout DOE. Figure 2 provides a graphical representation of the overall surveillance process.


Figure 2. Surveillance Process Overview


6.1. Surveillance Scope The EVMS risk assessment is employed to determine the scope and nature of the surveillance. An industry standard EVMS risk assessment tool modified for DOE will be used. The surveillance schedule should include all processes, with more intense evaluations on those programs/contracts with high or medium risk since they are most likely to cause unfavorable cost, schedule, and technical performance impacts. The scope may consist of just a few control accounts where the risk assessment identified the greatest cost and schedule risks or the greatest to go cost. The risk assessment may have also identified multiple projects where project performance indicates that the system may not have been institutionalized with the project management organization. The surveillance may also indicate that an on-site review is necessary to consult directly with CAMs and other project team members to determine whether the system continues to comply with ANSI/EIA-748. The minimum surveillance scope will be an off-site review of available project and project performance information. Other artifacts may be requested depending on the nature and scope of the surveillance.

6.2. Surveillance Schedule An annual surveillance schedule will be prepared by OECM and coordinated with the PMSOs. Surveillance review scheduling is influenced by the length of time since the last surveillance or certification review and the risk factors determined during the risk assessment process. Schedules will be adjusted to align with project events and other reviews to ensure that project staffs are available to support the surveillance.

6.3. Team Composition The size and composition of the team conducting the surveillance is governed by the size and complexity of the contract and project portfolio, the focus of the review based on analysis and risk indicators, and whether the surveillance is being conducted on-site or remotely. In all cases, the surveillance team should include EVMS subject matter experts as well as project analysts who are responsible for conducting oversight of the project and the contractor. The surveillance team includes individuals from OECM, PMSO, and other DOE Site Offices, and may include contracted support personnel. All participants are expected to be experienced in the surveillance process, knowledgeable in the application of the ANSI/EIA-748 EVMS guidelines, and familiar with the contractor’s work scope and EVMS processes. Similar to the Project Peer Review process, the EVMS surveillance team provides consistency among DOE sites and contractors by maintaining a cadre of experienced team members. In an effort to strengthen the EVMS core competencies in DOE, employees from other DOE Site Offices are encouraged to participate in these reviews as a training opportunity.

6.4. Artifacts and Information


While the artifacts and information employed in surveillance are identical to those used for a certification review, the depth and scope of a surveillance review is typically not as extensive as a certification review. Artifacts are dynamic information outputs that results from the operation of the EVMS. These include logs, change requests, reports, and other information containing pertinent information. Artifacts can be contrasted with information that is primarily static, such as process and system descriptions. Artifacts for surveillance contain more information than those provided for certification, such as performance data, logs, and other data that were produced as a result of the system operation and project progress. A list of typical artifacts and information used in surveillance reviews may be found at http://energy.gov/management/office-management/operational-management/project-management/earned-value-management. The specific artifacts and information that are necessary for surveillance depend on the scope of the surveillance, which is an outcome of the risk assessment process. The availability and accessibility of the artifacts and information is an integral part of the surveillance. For example, the inability to readily produce a Management Reserve Utilization Log or similar artifact is an indicator that the system is not functioning as originally reviewed and certified.

6.5. Conduct of Surveillance The EVMS surveillance is performed in a manner that facilitates answering four fundamental questions:

• Is the data accurate, timely and reliable? • Is the system being used to manage the project? • Does the data represent the entire scope? • Does the system comply with ANSI/EIA-748?

The surveillance involves examining artifacts such as logs, change requests, and budget reconciliation documents to ensure that the system is operating as designed and described in the system description. System utilization is primarily evidenced through the explanation provided for the data and the subsequent managerial decisions in response to the data. Determination of accuracy, timeliness and reliability requires examination of the data to ensure that it represents true project performance. This is evidenced through the examination of trends, which correspond to project events. The validity of the PMB is a key objective of surveillance and is primarily achieved through the examination of budgets, baseline change proposals (BCPs), reconciliations, and other data to ensure that all costs are reflected in the PMB and that all costs are related to defined scope as identified in the Integrated Master Schedule. The overall process for conducting the surveillance is focused on the entire portfolio of projects for each contract and/or site using a risk-based approach that factors in the project performance trends, process indicators, and project events such as baseline changes, time since last review, and others. The risk-based approach ensures that the surveillance program is focused where the need and risk to the agency are the greatest and is incorporated throughout the process to




determine the scope and content of the review. The surveillance process is conducted in three stages:

• Stage 1 – Risk assessment and on-going monthly analysis • Stage 2 – Desk top surveillance review. • Stage 3 - On site surveillance review.

For most contracts, the surveillance will be conducted through Stage 2. For some projects within the contractual portfolio, Stage 1 data analysis may be sufficient and determined as low risk. For high risk projects where data analysis indicates an increased risk in data stability, a more in-depth on-site approach, i.e. Stage 3, involving a team of subject matter experts will be used to ensure identification and resolution of issues. OECM’s risk-based surveillance program is coordinated with the PMSOs and FPDs. The overall process flow depicted in Figure 2 is further detailed in Figure 3 to identify responsibility and stages associated with the surveillance process. The following sections describe the process in detail.


Figure 3. DOE Surveillance Process


Collaboration is an essential part of EVM system surveillance and

project analysis.

6.6. Stage 1 Surveillance – Risk Assessment and On-going Monthly Analysis Risk-based surveillance translates to increased surveillance on processes and guidelines that have the greatest risk of unfavorably affecting system integrity. The nine EVMS process areas included in the risk assessment are:

• Organizing • Scheduling • Work authorization • Accounting • Indirect management • Management & Analysis • Change management • Material management • Subcontract management

These contractor business and management processes are cross-referenced to the ANSI/EIA-748 guidelines resulting in a comprehensive definition of an acceptable EVMS. This cross reference can be found in Appendix B.

6.6.1. Data Analysis Data analysis is conducted in collaboration with OECM Project Analysts and EVM Specialist, as well as PMSO, FPD, and project personnel. PARS II is considered the authoritative source of information on contractor project phases, and earned value data. Other data sources include, but are not necessarily limited to, the contractor’s EVMS self-surveillance documentation, and any assessments conducted by the FPD, PMSO, and/or OECM that evaluates project performance. These sources are analyzed to identify data disconnects, negative trends, and significant changes that may, upon further review, identify systemic issues between the earned value data and the physical progress as compared to the approved baseline. The OECM Project Analyst will coordinate with the OECM EVM Specialist when potential non-compliances or systemic concerns are identified or suspected. Conversely, the OECM EVM Specialist will interact with the OECM Project Analyst, PMSO, and the FPD who have project level knowledge of the technical baseline, progress, and cost, schedule, and technical risks. This interaction provides the OECM EVM Specialist with valuable insight into the identification of disconnects and system issues among other projects. The EVM Specialists will support Project Analysts as requested in conducting monthly assessments to identify and investigate EVM-related non-compliance issues. This process ensures that corrective actions are completed swiftly thus minimizing the impact to the performance indices.


The following table provides some of the more common indicators of systemic EVMS issues that should be examined in more detail and identified in the risk profile as a routine function of the monthly performance analysis process.

Table 1. System Compliance Issues Indicator

Indicator Explanation EVMS Compliance Indicator

No variances at summary level

Variances are normal. No variances are abnormal.

May be excessive LOE or inadequate planning of Control Accounts resulting in consistent underruns offsetting negative variances in other Control Accounts.

Abrupt disappearance of variances

Recovery from unfavorable variances by management action requires accomplishment of work that was not accomplished according to plan.

Cost overruns cannot disappear by work effort unless scope is decreased or budget is increased (application of MR). Both of which are violations of the standard.

Control Account budgets, MR, and UB do not reconcile to TPC

All budgets must be accounted for in an account. Budgets are not funds. Budgets must always reconcile to the TPC.

The inability to reconcile budgets indicates that changes have been made which were not accounted for by the system processes.

Milestones not met, but no cost or schedule variances

Milestones are dependent upon task/activities, which are the work packages that comprise Control Accounts. Missed milestones therefore, result from incomplete work, which create variances in Control Accounts. Missed milestones must have corresponding variances.

Schedule contains work not in PMB. PMB does not accurately identify the entire scope of work.


Excessive MR utilization rate

Excessive use of MR indicates that Control Accounts and Work Packages are not being sufficiently defined prior to start of work

MR cannot be used to eliminate overruns. MR can only be used for uncompleted (future) work. MR cannot be applied to Control Accounts where the BCWP equals BCWS

Unfavorable performance indices with no schedule impact

Cost variances do not automatically result in a schedule variance. Schedule variances indicate that the planned work was not completed in the period in which it was planned and scheduled. The variance analysis should always address the schedule impact. Not all schedule variances will have a critical path impact (and consequently, a milestone impact). However, all schedule variances should be evident in the schedule status.

A valid PMB depends on cost and schedule integration. Variances that have no corresponding schedule impact and schedule status that has no corresponding variance indicates an issue with cost and schedule integration.

Identical variance analysis and mitigation plans across time periods

Variance analysis should be directly associated with explaining and mitigating specific variances

Generic and obvious explanations with no specific proposed or planned management actions indicates that the performance information is misunderstood or not being used.

Continuing unfavorable performance trend

A key objective of EVM is to provide information to manage performance. Continuing unfavorable trends with no direction change indicates that there is no action taking place to mitigate performance

Analysis of variances and management action as a result of the analysis is a key tenet of EVM. Either the management processes do not provide the appropriate level of insight into the significance of the performance information or the information is not being used.


Excessive Baseline Change Proposal (BCP) rate

All projects undergo change but the change rate should decrease as the project matures. Excessive BCP rates indicate inadequate planning or conversion of summary level planning packages into Control Accounts and Work Packages. This does not include BCPs that are driven by fact of life changes such as directed changes in scope or budget.

Control Accounts must be planned in detail before work is authorized. Translation of the SOW into plan must account for uncertainty in later work.

Excessive favorable variances

Significant favorable variances can only be explained by increased productivity, significant labor or material cost reductions, or inadequate planning which resulted in a pessimistic plan

Reduced scope must have a corresponding budget reduction. Rate changes must also be documented. Scope or budget changes must be documented using the change management process.

6.6.2. Assess Project and Project Risk and Develop Risk Profile

Semi-annually, the OECM EVM Specialist will conduct a risk assessment to generate a risk profile for the entire portfolio of projects for each contract and/or site for which earned value is required and is subject to surveillance. The risk assessment provides a risk rating for applicable projects and EVMS processes to support planning and preparation of the EVMS Surveillance Schedule. Surveillances need to be conducted on the EVMS process when the risk assessment indicates possible systemic risks (multiple projects or multiple control accounts) or significant cost or schedule risks that are related to one or more processes. The EVM Specialist will identify and select projects for additional surveillance based on the risk associated with each project using an algorithm contained in the DOE EVMS Risk Matrix used to assign relative weights to each risk. The risk matrix template is available at http://energy.gov/management/office-management/operational-management/project-management/earned-value-management. Once the project(s) in the portfolio have been rated and ranked, the EVM Specialist will generate process risk ratings that are used as an aid in identifying system risk. Although several characteristics in rating process risk may allow for a subjective call, the EVM Specialist should be as objective as possible, using sound reasoning for determining the risk level for processes and selecting guidelines. The purpose of the risk ratings is to assist the EVM Specialist in prioritizing the EVM surveillance schedule, and to determine depth and scope should Stage 2 surveillance be warranted.

6.6.3. Develop Stage 2 and 3 Surveillance Schedule




Once the risk-based selection approach has been completed, a surveillance schedule is developed. The surveillance schedule is prioritized based on high and medium risk areas on high impact contractors/projects and DOE Order 413.3B requirements. The schedule will identify the contractor’s EVMS processes to be reviewed, the selected projects, and the anticipated timeframe. Using a continuous, data-driven approach, the surveillance may be conducted over several months or during a single review. Most surveillance will be off-site desk top reviews of individual projects. Site visits for surveillance of a contractor’s EVMS including the entire project portfolio will be scheduled based on the results of the desk top reviews and risk assessments. Surveillance schedule revisions may be necessary due to potential impact of identified concerns, resource issues, or other events. The surveillance schedule will be prepared and updated annually in accordance with the following milestones. This schedule ensures that the planning and budgeting for surveillance is executed in time for Department budget activities. The EVM Specialist will periodically review the schedule with OECM management.

Figure 4. Surveillance Annual Planning and Budget Schedule

6.7. Stage 2 Desk Top Surveillance

Stage 2 of the surveillance process is focused on specific procedures, contractor business, and management processes. The input to Stage 2 is one or more high risk areas identified during the Stage 1 surveillance. Typically these would be specific processes or procedures that do not appear to comply with ANSI/EIA-748. The objective of the Stage 2 surveillance is to validate the concerns from the Stage 1 surveillance, and issue CARs as required. Stage 2 surveillance requires the review of the EVMS documentation and artifacts and may include consultation with CAMs and other project team members. Such consultations will be accomplished using emails and audio and/or video teleconferences. The outcomes of the Stage 2 surveillance are CARs and CIOs.

6.7.1. Surveillance Scoping Meeting


Program/FPD and OECM representatives will conduct a surveillance scoping meeting to collaboratively define the scope of the surveillance based on the risk matrix and data analysis. An OECM EVMS representative will chair the scoping meeting, and attendance should include appropriate Program/FPD and project personnel. The results of the scoping meeting will include a determination of the procedures and processes to be examined; the documentation and artifacts necessary for the surveillance; the team composition; and the timeline for the surveillance.

6.7.2. Team Selection

Based on the surveillance scope, the scoping meeting attendees will outline the required knowledge and skills required for the review team members. The OECM Lead will:

1) Develop the surveillance plan, 2) Assign areas of responsibility to team members including peer review members, 3) Execute the surveillance, 4) Develop the out-brief, and 5) Issue the EVMS surveillance report.

The EVMS Surveillance team members are expected to provide independent input in the development of the out-brief and draft report. OECM will approve the final EVMS Surveillance team membership through approval of the surveillance plan. For many projects, the Stage 2 Surveillance Team will be much smaller than that which is necessary for an on-site review. The team composition for surveillances is a function of the number of projects, control accounts, and procedures that must be reviewed as well as the number of CAMs and other personnel to be interviewed. By its nature, a Stage 2 surveillance team will consist of an OECM lead, the OECM/PMSO analyst, and any necessary contract support. Program project staff may be necessary for some parts of the surveillance.

6.7.3. Roles and Responsibilities During the EVMS surveillance process, roles and responsibilities should be clearly understood by all participants (Table 3). For continuous ongoing surveillance triggered by projects equal to or greater than $100M, or as requested by the PMSO, the OECM EVM Specialist will monitor monthly earned value data from PARS II, surveillance activities conducted by contractors and FPD staff, and any other relevant data and reports. Table 2. Desk Top Surveillance Team Roles/Responsibilities

Role Responsibility OECM EVM Specialist Federal lead; facilitate the process; resolve issues; document results OECM/PMSO Project Analyst Support process with technical project specific insight Program/FPD Support process with resources, time, data, and personnel Contract Support EVMS team; assist in preparing review documentation and reports While not always possible, every effort should be made to clarify and resolve differing opinions of team members. The OECM EVM Specialist will be responsible for facilitating resolution. The


OECM EVM Specialist is also responsible for documenting all interviews, CARs, CIOs, data traces, and the final report. Those responsibilities may be delegated to contract support.

6.7.4. Project Selection In order to determine if any systemic issues exist, the entire contractor portfolio of all projects requiring EVMS will be considered for EVMS surveillance. A risk-based approach is then used to determine which projects should be reviewed in which areas based on the risk matrix. The risk profile is the primary means for determining the scope of the surveillance, including examination of multiple projects and control accounts within those projects. Factors determining risk include:

• Capital Asset Project TPC • Current or cumulative cost or schedule variance • Frequency of baseline resets • Nature of remaining work • Volume of remaining work • Critical Decision Stage of project • Type of project • Experience of contractor • Time since last review • Concerns since last review

These and other factors in the risk profile are intended to identify the areas where there is greatest cost and/or schedule risk. The result of the risk assessment is the identification of possible systemic or significant issues for projects where these issues could result in degraded performance.

6.7.5. Contractor Documentation and Artifact Review

Reviewing contractor’s documentation encompasses the review of static information such as the system description document and artifacts, which are dynamic outputs of the contractor’s business and management processes, such as change control logs. The following list provides the documentation and artifacts that are to be reviewed as part of the surveillance. Ideally, these documents and artifacts will be reviewed continually, over time, and as questions arise. OECM will identify a need for additional surveillances and determine if corrective action is necessary.

• At least two month of EVMS monthly reports • EVM variance analysis and correction action • Program schedules • Risk management plans • System Description Document and other pertinent procedures, directions • WBS/OBS and WBS dictionary • EAC supporting documentation • Contract budget log(s) including MR/UB/Contingency • Responsibility Assignment Matrix (RAM) (Dollarized)


• Work authorization documentation • Staffing plan

In preparation for a CAM interview, allow time for a complete review of the documentation and artifacts, to discuss any concerns among the interview team, and to ensure all areas of concern are addressed.

6.7.6. Conduct Data Traceability Because EVMS is an integrated set of processes, these processes must work in concert as a total system. Each sub-process of the system requires input from another process and provides outputs to still other processes. Tracing the data flow between processes is a critical element of the review process for the review team. Appendix C provides information for conducting data traces. Disconnects between the EVMS processes indicates that the system is not functioning as intended and that the processes and procedures must be examined in detail. This in-depth examination includes discussions with affected CAMs and other project staff. CAM discussions should be accomplished using audio, web-based, and/or video teleconferences to provide the insight necessary to determine if and what type of corrective action is necessary.

6.7.7. CAM, Program/Project Staff Interviews When necessary, the surveillance team should conduct interviews with appropriate program/project staff to gain additional insight or knowledge that would help them determine compliance with EVMS requirements. These interviews may be less than full interviews with CAMs and may be conducted by audio and/or video teleconferences, as well as email and WebEx. Some of the interview areas to consider are:

• Work authorization • Organization • EVM methodologies • Cost and schedule integration • Cost accumulation • Scheduling and budgeting • Material management • Subcontract management and integration of data • Risk assessment and mitigation • Variance analysis • Use of the information • Change control and maintenance • EAC process • EVMS program training

6.8. Stage 3 On-Site Surveillance

The primary difference between Stage 2 and Stage 3 surveillances is that the Stage 3 surveillance includes an on-site segment consisting of interviews with CAMs, management, and other project


staff, the observation of demonstrations of tools and traces that could not be conducted remotely, and physical verification of progress to assess reported work performed is accurately reflected. The on-site surveillance is a focused review, specifically to assess concerns raised in Stages 1 and 2 that could not be completely evaluated via the desk top surveillance. This surveillance is conducted over a consecutive period and requires the availability of program/project staff to support the review. Consequently, OECM will notify the PMSO and FPD at least eight weeks prior to an on-site visit to a contractor. 7. CORRECTIVE ACTION REQUESTS (CAR) AND CONTINUOUS IMPROVEMENT

OPPORTUNITIES (CIO) During the course of conducting EVMS surveillance, non-compliances will be documented via a CAR. The purpose of a CAR is to formally notify the contractor of deficiencies and non-compliances. All CARs require a documented course of action (i.e., corrective action plan) to bring the EVMS into compliance with ANSI/EIA-748 (latest version or as specified in the contract). If deficiencies are identified during the course of the surveillance, it is the surveillance team’s responsibility to prepare and issue a CAR. Deficiencies must be documented on a CAR and address the consequences if the non-compliance is not corrected. Additionally, the surveillance team should evaluate the appropriateness of all contractor corrective actions. All CARs will be documented and tracked to closure. Verbal CARs are not acceptable. The CAR process applies whenever a discrepancy is identified, even if discovered outside of a formal surveillance. CIOs may be issued to identify areas for process improvement. These may include suggested best practices, lessons learned, or other efficiency or effectiveness measures to streamline processes. CIOs do not require a written response from the contractor and approval by the team; however, they are encouraged to share their thoughts and plans pertaining to the ideas provided.

7.1. Types of CARs Two types of EVMS non-compliance are documented in CARs. The first is a process non-compliance, i.e., the contractor’s EVM System Description and/or supporting procedures (the ‘written word’) do not comply with one of the 32 ANSI/EIA-748 Guidelines. Corrective actions associated with instances of process non-compliance will require changes to the contractor’s EVM System Description. The second is an implementation non-compliance, i.e., the contractor’s EVM System Description and/or implementing procedures are not being followed in practice.

7.2. Documenting CARs CARs must contain a succinct description of the non-compliance, relevant guideline number, and a quote from the Intent Guide providing the verbiage not being adhered to (paraphrasing is not allowed). For process issues, a quote from the EVM System Description containing the non-compliant verbiage for each guideline impacted must be included. For implementation issues, a quote will be included from the EVM System Description describing the process not being


properly implemented. The EVMS CAR Template used by OECM is available at http://energy.gov/management/office-management/operational-management/project-management/earned-value-management. Sufficient back up data shall be submitted with each CAR as exhibits, i.e. screenshots, examples, to substantiate the non-compliance. Exhibits should provide easy to understand “pictures” of the problem, and should include a title describing the exhibit, and annotation of the area of interest by circles, arrows, or any other indicator to assure clear understanding of the non-compliance. Process CARs will include an exhibit containing the completed EVM guideline assessment identifying which guidelines are not being adhered to within the EVM System Description. In the case of process CARs, each non-compliant guideline may be grouped and listed on one CAR or they can be grouped by process area into multiple CARs. The CAR will not simply state “all guidelines” unless supported in the CAR description and the EVM guideline assessment. CARs are issued to the contractor typically within 5 days after an on-site surveillance. The OECM EVM Specialist makes the distribution of the CARs and CIOs to the contractor, FPD, PMSO, and surveillance team.

7.3. Corrective Action Plan (CAP) Review The contractor’s response to the CAR shall include a CAP that provides the following attributes:

• Identification of the root cause(s) of the problem • Actions to be taken or planned to resolve the identified non-compliance • Internal controls established to prevent a recurrence of the non-compliance • Verification that the proposed CAP will result in compliance with the affected ANSI

Guideline. Upon receipt of the contractor’s response, the surveillance team, working through OECM, shall perform a review of the CAP to ensure the items above have been satisfactorily addressed. The CAP Analysis report will be issued by the OECM EVM Specialist to the contractor, FPD, PMSO, and surveillance team.

7.4. CAR Verification and Closure All corrective actions must be verified through follow up actions. The OECM EVM Specialist shall plan, schedule, and approve all verification follow up actions and closure of CARs. Verification of corrective action shall be based on the following:

• Inspection of supporting documentation and/or on-site visual inspection of corrective actions;

• Effectiveness of the corrective action in satisfying the guideline(s); and • Previous CAPs to verify compliance maintenance with guideline(s).

As part of the CAP verification, the surveillance team shall document the CAR closure criteria. The closure criteria should contain what actions are required to be successfully accomplished before the CAR can be closed out. The surveillance team is responsible for ensuring that the




closure criteria are verified and a mutual understanding has been reached. As part of closure criteria verification, the team should consider the following:

• Is the guideline being met? • How is this different from when the guideline was not being met? • Are internal controls in place to prevent guideline non-compliance from recurring? • Does this CAR affect the contractor being compliant with other guidelines? • Are other projects affected by the CAR? If so, will they be compliant with the

guidelines? If the OECM EVM Specialist determines that verification is not necessary, then the surveillance team documents the status of the closure verification. If the verification follow-up results in continued non-compliance or new deficiencies, then the EVM Specialist will decide if escalation is warranted. Before a decision to escalate is made, the surveillance team may be asked to determine if more verification is required, if a different type of verification is necessary, and if a new or revised CAP is required.

7.5. CAR/CIO Database When the CAR/CIO Database is completed, the OECM Specialist will ensure CARs are documented and their status updated in the database. Information tracked will include affected guideline, area, problem, author, reviewer, contractor, project, date, type of surveillance, and whether it is a CAR or CIO. The CAR/CIO database will generate reports to show distribution of Guidelines/Areas, or by contractor, or by project, or by author. 8. EVMS SURVEILLANCE RESULTS DOCUMENTATION

8.1. EVMS Standard Surveillance Report (SSR) Sections For OECM led reviews, the surveillance team will document the results of system surveillance in a written report that is issued by the OECM Director. The report will include an overall assessment of the contractor’s implementation of the EVMS, scope of the review, personnel interviewed, and findings of deficiency or non-compliance that resulted in Corrective Action Requests (CAR). This report format is recommended for use on PMSO, Site Office/FPD, and contractor led surveillances. Standard Surveillance Reports should include at a minimum:

• Surveillance Selection Risk Matrix(s); • Guidelines or Processes reviewed; • Program Managers (PM) and Control Account Managers (CAM) interviewed and Control

Accounts examined; • System deficiencies identified; • Information on issued CARs and CIOs

For contractor self-surveillances, it is recommended that the report include the completed EVM guideline assessments as all must be reviewed annually.


Sections of the report include:

• Executive Summary • Scope • Surveillance Summaries

o Interview Summaries o Independent Data Traces & Document Research (sometimes attached to the EV

Templates) • Surveillance Results

o CARs and CIOs Issued o CAR Status

• Guideline Compliance Summaries

The Executive Summary - The SSR executive summary should, at a minimum, refer to the established SSP, the contractor, the members of the surveillance team, and the compliance review CAP, if applicable. It should include the dates of the surveillance and indicate if any deficiencies were found in the implementation of the evaluated EVMS guidelines. The executive summary should also include:

• An impact statement noting significant concerns raised by the surveillance and the potential effects of those concern areas.

• A reference to follow on actions or any open items.

The Scope of the Surveillance -- The surveillance scope section should include the following: • The start and finish dates of the surveillance, • All guidelines and processes reviewed • The Project(s) being reviewed and the respective contract number(s), • The Customer or purchasing organization, • An explanation of how the risk assessment results support the scope of this review, • An explanation of any revisions or deviations from the approved annual EVMS

surveillance schedule, • The DOE contact information, • The contractor EVM focal point’s contact information • Identification of the compliance review CAP evaluated, if applicable.

Surveillance Summaries -- The surveillance summary section aligns with the scope of the review. It describes the CAM discussions, the areas identified for independent data traces, and any follow up research that was conducted. Interview Summaries -- Each interview summary should contain:

• CAM name and his/her area of responsibility. • Guideline or Process area focus. • Summary of interview discussion points. • Description of findings. • An assessment of the interview session tone, i.e., size, length, complexity, extent of

cooperation, effectiveness, etc.


• Common themes or relationships to other interviews. Examples: WBS elements, technical area, component, etc. New issues or areas of interest identified as a direct result of the CAM interview.

Independent Data Traces & Document Research -- This section should identify any independent data traces or document research accomplished in support of the surveillance. It should discuss the research and data trace objectives in the context of surveillance review scope. Summarize and/or attach selected exhibits that support findings or other items in the SSR as necessary. Traces and research material must be dated and sources must be noted to ensure the long term credibility of supporting material. These may be attached to or associated directly with the EV templates for easy reference. Attached exhibits should be limited to keep the overall document size under 5MB. Further documentation can be included as separate referenced files. Surveillance Results -- This section should document all surveillance findings and their status as of the report date. It should also provide a status of all outstanding CARs on the program and a trend analysis.

• CARs Issued -- This section should quantify the CARs discussed in the surveillance summaries. It should contain succinct statements that communicate EVMS issues across guidelines and processes. Discuss any pending actions, outstanding unresolved surveillance issues, and systemic issue trends. Discuss progress made towards compliance review CAP implementation, if applicable.

• CAR Status - List newly issued CAR(s), ongoing/existing CAR(s), and any CAR(s) that were resolved during the reporting period. Briefly summarize the new CAR(s) and note the affected guidelines or processes. Discuss the status of open CAR(s) and the status of their respective corrective action plans (adequacy of CAP, estimated completion, delays, contractor concerns, etc.)

Guideline Compliance Summary -- This section should identify how many guidelines were found to be non-compliant during the surveillance. Reference and attach completed EVM guideline assessment for the included in this review.

8.2. EVMS Standard Surveillance Report (SSR) Distribution Surveillance Data Distribution – The following sections identify distribution lists for each surveillance product.

• Risk Matrices/Data Analysis Working Papers – • OECM • PMSO • FPD

• CAR Distribution, CAP, Surveillance Report, Compliance Letter • OECM • Contractor • PMSO • FPD


• Contracting Officer

8.3. EVMS Standard Surveillance Report (SSR) Retention Surveillance files shall be established and maintained indefinitely by the OECM EVM Specialist to hold all pertinent surveillance data and information.

8.4. EVMS Report Transmittal OECM will use the final EVMS Surveillance Report, in combination with any corrective actions identified in the approved CAP, to assess whether the contractor EVMS remains in compliance with ANSI/EIA-748 (latest version or as specified in the contract). OECM may also use information from EIRs, contractor surveillance reports, PARSII data, and other information in assessing whether the EVMS is compliant. OECM will transmit the final SSR and document its recommendation with respect to continued compliance of the contractor’s EVMS in a memorandum from the OECM Director to the appropriate CO.

9. EVMS EVALUATION AND FEEDBACK EVMS evaluation and feedback is highly encouraged and valued in an effort to continuously improve and add value to surveillance reviews. Program offices, project teams, and PMSOs are encouraged to provide OECM with feedback on the conduct of the EVMS Surveillance, including any comments related to:

• Application of the Risk Matrix • Surveillance Plan development • Knowledge and professionalism of the EVMS team members • Preparation and support of the EVMS team • Resolution conference • Timeliness and responsiveness of OECM and the EVMS team • Quality of the review and findings • CAP review process

Feedback forms are available at http://energy.gov/management/office-management/operational-management/project-management/earned-value-management. The EVMS surveillance lead will distribute forms, and the forms will be requested to be transmitted to the OECM POC. The OECM POC will maintain the confidentiality of the submitter(s) of the forms, and ensure that feedback is only communicated through compilations.




APPENDIX A—ACRONYMS

AC actual cost ACWP actual cost of work performed ANSI American National Standards

Institute AUW authorized unpriced work BAC budget at completion BCWP budgeted cost for work performed BCWS budgeted cost for work scheduled BR budget remaining CA control account CAM control account manager CAP corrective action plan CAR corrective action request CBB contract budget base CD critical decision CIO continuous improvement

opportunity CO contracting officer CPI cost performance index CPR contract performance report cum cumulative CV cost variance CPR contract performance report DCAA Defense Contract Audit Agency DoD Department of Defense DOE Department of Energy EAC estimate at completion ETC estimate to complete EIA Electronic Industries Alliance EIR external independent review EV earned value EVMS earned value management system FAR Federal Acquisition Regulation FPD federal project director G guide GAO Government Accountability Office

IG Inspector General IPR independent project review IPT integrated project team LRE latest revised estimate (at completion) M million or milestone MR management reserve NDIA National Defense Industrial

Association O order OBS organizational breakdown structure OECM Office of Engineering and

Construction Management OMB Office of Management and Budget PARS II Project Assessment and Reporting

System II PB performance baseline PEP project execution plan PMB performance measurement baseline PMSC Program Management Systems

Committee POC point of contact PP planning package PSO Program Secretarial Officer PV planned value RA readiness assessment RAM responsibility assignment matrix SLPP summary level planning package SPI schedule performance index SV schedule variance TCPI to complete performance index TPC total project cost UB undistributed budget VAC variance at completion WBS work breakdown structure WP work package WR work remaining


APPENDIX B---ANSI/EIA-748 GUIDELINES & ORGANIZATION PROCESSES ALIGNMENT


APPENDIX C--- CONDUCTING AN EVMS DATA TRACE One of the objectives of a surveillance review is to ensure traceability throughout the system. If any inconsistencies or anomalies are apparent, they are to be addressed in Corrective Action Requests as appropriate. When conducting traces, you should document your evidence and attach examples where possible. The guideline(s) that may relate to the trace are provided in parentheses. This list is intended as a guide only and is not all-inclusive. For the selected work packages or activities associated with work scope, authorization and responsibilities, the team should accomplish the following:

ORGANIZATION

Guideline Data Trace Method

1

Determine which control account contains the trace item by reviewing the CWBS and CWBS dictionary. Ensure that the CWBS and CWBS dictionary adequately define the contractual effort to be accomplished within this control account. Annotate the CWBS and CWBS dictionary pages to indicate the contract line item and end item elements that relate to this control account.

2, 4

Review the RAM to locate the control account that contains the trace item. Ensure that this control account is assigned to a responsible organization element that is consistent with the effort to be accomplished. Annotate the RAM to indicate that the control account was developed at the intersection of the CWBS to the organizational structure and that the CWBS was extended down to the control account level.

3, 22, 26, 27

Review the work authorization documents for the control account that contain the trace item. Verify that the organization assigned in the RAM, is the responsible organization in the work authorization documents. Ensure that the work authorization documents are approved and signed by the responsible functional managers designated in the RAM. Ensure that the work authorization and CWBS definitions of the effort to be accomplished within the control account are consistent. Provide the control account work authorization documents as exhibits.

1 Select sample from Statement of Work (SOW) and verify its inclusion in the WBS dictionary and vice versa.


SCHEDULING


6 Review control account/work package schedules. Ensure that the scheduled dates on the authorization document for the control account are the same as the dates on the detailed plans.

6, 7 Confirm that the schedule contains all contractual activities.

6 Accomplish a vertical schedule trace which shows the flow from these schedules through the intermediate schedules to the master schedules.

6 Accomplish a horizontal trace which shows that the appropriate control accounts and work packages are logically linked (use network schedules if available).

7, 23 If appropriate, confirm the identification of work progress and forecast of completion dates. Check that the CAM’s status (as shown on the status turn-around document) has been reflected on the revised schedule.

MANAGEMENT and ANALYSIS Guideline Data Trace Method

16, 22 Ensure that earned value is being claimed in the same manner in which it was planned. For example, if an earned value technique of 0-100% is used, there should be no interim BCWP claimed.

27 Ensure that any EAC reported reflects information to date. Check that cumulative variances are either explained and a corrective action plan is in place or the variance is reflected in the EAC.

27 Check EAC amounts for completed control accounts or work packages and ensure that the ACWP does not exceed the EAC (should be equal).

23, 26

Review variance analysis reports to ensure the following • Reasons are adequately explained (i.e. it does not simply say that there was a

variance) • Impact is identified, how it affects other control accounts and whether it

affects the program overall • Corrective action or recovery plan is identified and implemented • Analysis is approved at a higher level than it is prepared


BUDGETING


8, 10

Review the Control Account Planning sheets for the control account that contains the items. Confirm that these plans reflect the way in which work is to be done, that there is an appropriate number of work packages verses planning packages, and that the planning packages are neither too general nor too large in scope, value, and duration.

9, 10, 11

Review control account documentation and internal reports as they pertain to the trace items. Ensure that the sum of the planning package budgets plus the work package budgets equals the control-account budget. Ensure that the planning packages have their own budget values and that there are adequate procedures for converting a planning package into a work package.

8

Review control account planning sheets and other performance measurement reports for the control account that contains the trace item. Determine how BCWS was time-phased and established. Determine if these budgets were established in a manner which is consistent with the method used for material accounting (if applicable).

15, 29

Review the budget information in the Work Authorization documents, the RAM, and the internal performance measurement reports to ensure that they are reconcilable. Then check that the amounts on internal Cost Performance Report are consistent with the external report being forwarded to the Government.

9 Select a sample of control account plans and ensure that budget is broken down by significant cost elements (labor, material, ODC etcetera) as appropriate.

12 Review LOE content of control account budgets to ensure it is only applied where appropriate. If possible obtain a summary of LOE accounts from the contractor.

14 Obtain MR and UB logs and trace from entry in logs to location of transfer. Also reconcile with CPR amounts.

CHANGE MANAGEMENT


14, 28, 29, 32 Review change request documents to ensure that traceability exists between the control account(s), change requests, MR, UB as appropriate (including current budget trace to original budget).

28, 30

Approval dates on change request documentation should be in advance of the period of the proposed change. This needs to be in accordance with whatever the system description says about “freeze periods” for changes. Review the number of changes in the current period

28 Check to see the cycle time to incorporate changes into control account plans from submittal, approval to incorporation.


MATERIAL MANAGEMENT Guideline Data Trace Method

16, 22

Select a material item for each type of material and trace its flow through the procurement cycle. This should include the bill of materials, purchase orders, billing, issuing from inventory types of documentation. Ensure that material items are being tracked from control account authorization to completion.

9 Review how budgets including scrap and attrition values were established. Check to see that BCWP is being claimed in the same manner in which it was planned.

27 Review how the material budgets are time-phased to ensure it is consistent with the requirements of the system description and how the work is being performed

22, 23

Review internal reports that identify initial material quantities and then review documents provided to CAM to assess actual usage etc. Check variance analysis reports to determine whether price and usage variances are separated for managerial analysis.

9 If applicable, locate the trace item in the bill of material (BOM) and/or purchase order. Check for consistency and determine how total budget values were established.

12 Establish the value of the material and how much is being claimed as LOE. Generally only low-value material should be claimed as LOE.

27 Ensure that commitment values for material and actual material costs are incorporated into the EAC in a timely manner.

SUBCONTRACT MANAGEMENT


2 Ensure that the responsibility for subcontract management is identified

9, 10, 12 BCWS should be based upon identifiable milestones where possible and the use of LOE is minimized. Check to see how the subcontracted effort is planned and what earned value technique at attributed to measure performance

6, 23 Ensure subcontractor schedules are vertically and horizontally integrated with prime’s schedules.

9, 10 Check the process for tracking material issued from the prime to the subcontractor for work.

16, 22 Check for proper incorporation of subcontractor’s data into the prime’s system.

23 Verify the subcontractor’s baseline and ensure that contract changes are incorporated in a timely manner.

27 Ensure that EAC includes subcontractor updates for actual costs, material values etc.

earned value management system (evms) surveillance ...€¦ · evms surveillance sop september 26,...

Documents