self-assessment surveillance review report of the fermi … evms dec 2017... · 2018-03-22 · the...

1

Self-Assessment Surveillance Review

Report

of the

Fermi Research Alliance, LLC (FRA) Earned Value Management System

(EVMS)

Fermi National Accelerator Laboratory (FNAL)

December 2017

Final 03-21-2018

2

TABLE OF CONTENTS

1.0 Self-Assessment Review Summary ...................................................................................................... 3

1.1 Corrective Action Requests (CARs) ............................................................................................. 3

1.2 Continuous Improvement Opportunity (CIOs) ............................................................................. 4

2.0 Individual Guideline Review, Analysis, and Comments ...................................................................... 5

2.1 Organization .................................................................................................................................. 6

2.2 Planning, Scheduling and Budget ................................................................................................. 9

2.3 Accounting Considerations ......................................................................................................... 17

2.4 Analysis and Management Reports ............................................................................................. 21

2.5 Revisions and Data Maintenance ................................................................................................ 26

Appendixes ................................................................................................................................................. 30

A. Surveillance Plan ............................................................................................................................ 30

B. Review Agenda ............................................................................................................................... 42

3

FRA-FNAL Earned Value Management System (EVMS) Self-Assessment Surveillance Review Report

December 6 - 7, 2017

1.0 Self-Assessment Review Summary

A self-assessment surveillance review was conducted on the certified FRA Earned Value Management System (EVMS) on December 6 – 7, 2017. The two-day surveillance review took place at the Fermi National Accelerator Laboratory (FNAL) facility located in Batavia, Illinois. The objective of the self-assessment surveillance review is to ensure that the FNAL EVMS continues to meet the EIA-748-C Standard and that implementation of the FNAL EVMS remains effective. The surveillance review team membership contained participants from other Office of Science National Laboratories to ensure the independence of the review process. A list of the reviewers in attendance can be found within the Surveillance Plan in Appendix A. The surveillance review was based on an examination of the previously certified FNAL EVMS description and procedures, previous surveillance reviews, and the current application of the FNAL EVMS on the Muon to Electron Conversion (Mu2e) and the Long Lead Procurement (LLP) portion of the Long Baseline Neutrino Facility/Deep Underground Neutrino Experiment (LBNF/DUNE) projects. These projects are required to implement the FNAL EVMS in accordance with DOE-O-413.3B and were 60% and less than 1% complete, respectively, at the time of the review (as of October 2017 baseline data). The Mu2e project has an early completion of May 2021 and the LBNF/DUNE project’s LLP scope completes in January 2023. The surveillance review team determined that the FNAL EVMS is a beneficial management tool and the implementation of their certified Earned Value Management System is in compliance with the criteria of the EIA-748-C standard. The surveillance review team identified three corrective action requests (CARs) and seven continuous improvement opportunities (CIOs). A summary of the CARs and CIOs are listed below with further details and supporting information included in Section 2.0 of this report. 1.1 CARs (guidelines listed with the corrective action)

• CAR-01 Work Breakdown Structure (WBS) Dictionary Updates (GL-01)

• The Mu2e WBS Dictionary has missing WBS elements, incorrect Control Account Manager (CAM) listings, and would benefit from a second look at the scope statements.

• DOE is utilizing the WBS Dictionary to closeout CD-4, so additional attention should be given to an accurate WBS Dictionary.

• CAR-02 Missing Work Authorization Documents (WADs) (GL-03) • The Mu2e project is missing WADs for Control Accounts (CAs) listed within the

RAM. All of these are Other Project Costs (OPC) CAs so they should be documented as closed if they are no longer in progress.

4

• CAR-03 Use of Steps and Quantifiable Back-up Data (GL-07)

• On Mu2e, a significant number of instances where activities using the % Complete Performance Measurement Technique (PMT), spanned more than two accounting periods, but did not possess some objective measurement of work completed or remaining – this was also a CAR in the last review.

• One instance where Level Of Effort (LOE) is used, should have been % Complete.

• Instances of partial credit on listed Steps that should be only 0/100. • A few instances where % Complete PMT was used with Percent Type in P6 as

“duration.” • One instance where PMT was not coded.

1.2 CIOs (guidelines listed with the continuous improvement initiative) There are a total of seven CIOs; two with an asterisk and five without.

• CIO-01 Number of Start-to-Start (SS) and Finish-to-Finish (FF) Relationships (GL-06) • 17% of activities have SS relationships and 10% have FF relationships on Mu2e.

Some of these are leading to Open Starts and Open Ends – should be reviewed and used cautiously. This is also present on LBNF but to a lesser degree.

• CIO-02 Resource Leveling Within Activities to Reflect Required Time-phasing (GL-08) • Resource analysis should be performed on Mu2e and the availability of resources

to the project should be translated to the schedule.

• CIO-03 New CAMs Should Review and Update Inherited CAs (GL-06) • When a CAM inherits a CA that is in-progress, there should be a detailed review

of schedule and costs and update as required.

• CIO*-04 Capitalization Threshold Stated in the EVM System Description is Incorrect (GL-21)

• The Accounting Considerations section of the EVM System Description indicates a capitalization threshold of $50K, whereas the Cost Accounting Standards Disclosure Statement and financial procedures set the capitalization threshold at $500K in accordance with DOE guidance.

• The capitalization threshold in the EVM System Description should be updated to $500K.

• CIO-05 Receipt of Material Costs (GL-16, 21) • The process for receiving goods in remote locations, like South Dakota, appears

to be undefined and may not comply with the CAS Disclosure Statement. • The CAS Disclosure Statement states that materials are reported as cost when

they are received. • The process for receiving goods remotely for LBNF appears to be undefined and

receipt is recorded when the CAM is contacted about the receipt, which may be at some interval (e.g. monthly) or not until the invoice is received.

5

• There should be a process in place for CAMs to notify Receiving immediately when materials are received.

• CIO*-06 Variance Analysis Report (VAR) Quality and Use of Corrective Actions is Not

Consistently Applied (GL-23) • Some VAR explanations and impacts differed from the explanations the PM and

CAMs gave in presentations and interviews on Mu2e. • Inconsistent use of the Corrective Action portion of the VAR. It was pointed out

that Corrective Actions should have been used on some high dollar value variances or variances where the negative trending continued and no mitigation action was being taken.

• Process would benefit from a more thorough understanding of variances before approval, which would also aid in understanding when a Corrective Action should be implemented.

• CIO-07 Traceability of Changes (GL-32)

• The Mu2e Project Baseline Change Requests (BCRs) were challenging to assess compliance. Consider preparing a guide for reviewers to understand what the package contains and how to drilldown into the information to verify compliance.

• Consider adding a column either within the BCR, BCR summary, and/or change log summary sheet that reflects the BCR’s implementation period for ease of understanding the reporting month of implementation.

Further details on the surveillance review plan, team participants, and the guideline areas for which they were responsible are identified in Appendix A. Individuals interviewed during the surveillance are identified in the agenda shown in Appendix B. 2.0 Individual Guideline Review, Analysis, and Comments

This section covers the review and analysis of the five major EVMS categories: 1) Organization; 2) Planning, Scheduling, and Budgeting; 3) Accounting Considerations; 4) Analysis and Management Reports; and 5) Revisions and Data Maintenance. Each category has the associated guidelines identified, an evaluation of FNAL EVMS compliance, and related observations from the responsible surveillance review team member.

Compliance is indicated along with any corresponding CAR or CIO with each of the 32 guidelines. Some of the CARs/CIO*s/CIOs may affect more than one guideline. If this is the case, the full CAR/CIO*/CIO write-up will appear in the most impacted guideline area but will address the specific violation/corrective action for all guideline areas impacted. However, reference to all CAR/CIO*/CIOs that impact a guideline will be made within the observations and finding areas. More details regarding the definition of CAR, CIO*, and CIO can be found in the Surveillance Review Plan in Appendix A.

6

2.1 Organization

Guideline 1: Define the authorized work elements for the program. A work breakdown structure (WBS), tailored for effective internal management control, is commonly used in this process.

Reviewer Name: Jennifer Fortner, ANL Compliant with EIA-748: No, see CAR below

Observations and Findings: Each project had a hierarchical Work Breakdown Structure (WBS) which digests the project’s scope of work into logical deliverables. Each of the projects presented a WBS extending to multiple levels below level 1 with Control Accounts (CAs) assigned at appropriate levels to allow for appropriate monitoring and planning. The WBS dictionaries were reviewed and were found to contain varied levels of details associated with the scope description of the WBS elements. Improvements could be made regarding the amount of detail in the WBS dictionaries. There were errors related to scope not being represented within the dictionary, missing entire WBS elements. See CAR-01 for more details. The project WBS elements were found to be utilized for organizing project documents such as the Responsibility Assignment Matrix (RAM), work authorization documents, and project schedule.

CAR-01 Subject (Issue): WBS Dictionary Updates Referenced Guideline(s): 1 Referenced Data Trace: WBS Dictionaries, RAM, and CAM Interviews Description of Issue: The Mu2e WBS Dictionary has missing WBS elements, incorrect CAM listings, and would benefit from a second look at the scope statements. A sample of the issues identified are provided below:

o CAMs missing from WBS Dictionary: Alber, Biery, Lombardo. The elements are assigned to them on the RAM but not within the WBS Dictionary, the Dictionary has different CAMs assigned.

o Two CAMs listed within WBS Dictionary: Still and Kasper are listed as the owner of 475.02.08 and the sub elements within this area has Still listed, that should all be Kasper and those sub elements should not be listed as CAs since the CA level is 475.02.08 according to the RAM. It creates two owners of the same CA area.

o Use of the term Cost Account Manager vs Control Account Manager. o Areas missing from WBS Dictionary: 475.02.10, 475.04.11, 475.05.13, 475.06.07, 475.08.09 o Level missing or clarification needed within WBS Dictionary on 475.07.03 – RAM shows

CA 475.07.03.02 as the only level within this WBS. The WBS Dictionary shows

7

475.07.03.01 and 475.07.03.02.01-02 as sub elements. 475.07.03.01 is missing the title description and then it skips down to 475.07.03.02.01-02.

o No update from March 2016 to November 2017

Given that DOE is utilizing the WBS Dictionary to closeout CD-4; additional attention should be given to an accurate WBS Dictionary. Recommendation: Removing the CAMs names from the WBS Dictionary and allowing the RAM and WBS to create the crosswalk might be worth considering. This would eliminate many of the issues identified. Additionally, DOE is utilizing the WBS Dictionary to perform closeout procedures for CD-4, so additional attention should be given to an accurate WBS Dictionary concerning the scope statement.

Guideline 2: Identify the program organizational structure including the major subcontractors responsible for accomplishing the authorized work, and define the organizational elements in which work will be planned and controlled.

Reviewer Name: Jennifer Fortner, ANL Compliant with EIA-748: Yes

Observations and Findings: Each project’s organization chart defined its Organizational Breakdown Structure (OBS) showing the organizational elements in which the work was planned and being controlled. The OBS facilitated the assignment of responsibility, accountability, and authority by WADs for all the work to be performed by each of the projects. The CAMs interviewed were aware of their roles within the project’s organization and to whom they reported. Major subcontractors were also present on the organizational chart.

Guideline 3: Provide for the integration of the company’s planning, scheduling, budgeting, work authorization and cost accumulation processes with each other, and as appropriate, the program work breakdown structure and the program organizational structure.

Reviewer Name: Jennifer Fortner, ANL Compliant with EIA-748: No, see CAR below

Observations and Findings: The integration of the planning, scheduling, budgeting, work authorization and cost accumulation processes is accomplished at the intersection of the project’s work breakdown structure and the project’s organizational structure. This is represented in the dollarized Responsibility Assignment Matrix (RAM) that exists and further documented with the Work Authorization Documents (WADs). The CAMs interviewed were aware of the Control Accounts (CAs) assigned to them through the work authorization process. Data traces were performed between the WAD, RAM, Contract Performance Reports (CPR), and Change Requests (CRs). For Mu2e there were missing WADs and no closed WADs for accounts that were stated as complete. See CAR-02 for more information.

8

Additionally, during interviews it was revealed that the some CAMs were unsure of how to close a Control Account once all work was completed in that account. This was a CIO in prior review and the Corrective Action Plan says they have Desktop Instructions, but the CAM was not aware of it.

CAR-02 Subject (Issue): Missing WADs Referenced Guideline(s): 3 Referenced Data Trace: WADs, RAM, and CAM Interviews Description of Issue: The Mu2e project is missing WADs for CAs listed within the RAM. All of these are OPC CAs so they should be documented as closed if they are no longer in progress. When the CAMs were asked about these CAs they knew they were R&D related but were not able to discuss the closeout procedure that should have been used to create the final WADs. Recommendation: Document the OPC CAs as closed and determined a method for capturing these if a WAD is not going to be created for them. Additionally, make sure the CAMs are aware of the process.

Guideline 4: Identify the company organization or function responsible for controlling overhead (indirect costs).

Reviewer Name: Jennifer Fortner & Betsy O’Connor, ANL Compliant with EIA-748: Yes

Observations and Findings: The CAMs were aware that their portion of the project included both direct and indirect costs. The Budget Office is responsible for establishing and controlling the Lab’s indirect rates and the Accounting Department establishes and monitors the labor-based indirect rates, like Fringe and Vacation. The Accounting Department prepares a monthly report that shows the performance of the indirect rates. The spending and recovery are monitored closely and adjustments may be made mid-year to account for significant over or under recoveries. Any adjustment to the rates is made in the current month, retroactively to October 1st and projects are not required to prepare a BCR for changes in the indirect rates. In FY2020, in accordance with DOE requirements, Fermilab will eliminate the cap on the indirect expense assessment on procurements, where only the first $500K of a contract is assessed indirect

9

expense. The Budget Office is preparing scenarios and working to develop a new rate structure that complies with DOE requirements with consideration of the impact on projects.

Guideline 5: Provide for integration of the program work breakdown structure and the program organizational structure in a manner that permits cost and schedule performance measurement by elements of either or both structures as needed.

Reviewer Name: Jennifer Fortner, ANL Compliant with EIA-748: Yes

Observations and Findings: Each project had a RAM that identifies the established control accounts with their associated budgets and the responsible manager. Work/planning packages have been generated under the control accounts. Both of the projects reviewed use planning packages appropriately. Actual costs and performance data are collected at the CA level, which provides essential information for management control. Most CAs appeared to be at an appropriate level for proper management. The chargeable task codes are located at the CA level and are where actual costs are collected for the work performed. An assessment of producing an organizational breakdown of costs for measurement was not performed during this review and production of a Format 2 was not required.

2.2 Planning, Scheduling, and Budgeting

Guideline 6: Schedule the authorized work in a manner which describes the sequence of work and identifies significant task interdependencies required to meet the requirements of the program.

Reviewer Name(s): Raj Gutta, BNL Compliant with EIA-748: Yes, with CIOs below

Observations and Findings: The Mu2e and LBNF CD3a project schedules are in Oracle Primavera and are composed of well decomposed, distinct activities and milestones. The project contains logic ties between the activities and milestones, is logically driven, and can generate a critical path. In most cases, the level of activity decomposition seems very appropriate and reasonable. There were cases of questionable logic relationships that has resulted in open starts/ends and high float numbers and some activities were missing logic. See CIO-01 for more information. The schedules are organized by the project WBS and reflect the scope of work described within the WBS Dictionary and discussed during CAM interviews. Some CAMs inherited a Control Account without properly reviewing and updating the attributes such as PMT and steps. See CIO-03 below for further details.

10

Both projects have established and are maintaining baseline schedules, and are conducting monthly status updates of their working schedules; and performance is monitored against the latest approved baseline.

CIO-01 Subject (Issue): Number of Start-to-Start and Finish-to-Finish Relationships Referenced Guideline(s): 6 Referenced Data Trace: Detailed Project Schedule Review, Acumen FUSE Analysis, and CAM interviews Description of Issue: Based on the review of statements in a previous CAR for this area, the schedule has significantly improved on the Mu2e project and appears well planned for LBNF CD3a project. However, some missing logic and very high float numbers still exists on some activities. This still causes questions related to the schedule integrity. A number of activities have only SS or FF relationships. This results in open ends and open starts, further contributing to the questions of schedule integrity. Based on Fuse analysis, 17% of activities have SS relationships and 10% have FF relationships, which are high but not unreasonable. The concern is more that the use of these relationships only on a series of activities could lead to float calculation issues if they are not appropriately anchored in the activity chain. Recommendation: Review the missing logic and high float numbers. Use SS and/or FF relationships very cautiously to make sure there are no open starts or open ends.

CIO-03 Subject (Issue): New CAMs Should Review and Update Inherited CAs Referenced Guideline(s): 6 Referenced Data Trace: CAM interviews, WBS Dictionary, Work Authorization Documents Description of Issue: During CAM interviews there were multiple CAMs that did not start on the project and originally develop the CAs that they are currently managing. It was clear that these CAMs still needed to take ownership of their inherited CAs and perform a forward looking review that would allow them to modify the CA to better suit their management needs. Anytime a CAM inherits a CA, especially one that is in-progress, there should be a detailed review of schedule and costs with an allowed update, as required. For example, activity 47504.10.002145 was discussed during an interview and the CAM thinks the PMT should be % complete, but in P6 it is LOE. There were also instances where the steps were not revised based

11

on scope changes that the CAM perceived within the CA. Upon further discussion the review team was informed that the CAM inherited this CA which led to some of the above problems. Recommendation: Whenever a CAM inherits a CA, an opportunity should be given to review and update attributes, as required. CAMs should be encouraged and trained to do this as part of their duties when taking ownership of the new CA. It may be best that Mu2e have all CAMs review their CAs that were inherited and update, as needed, for proper ownership going forward.

Guideline 7: Identify physical products, milestones, technical performance goals, or other indicators that will be used to measure progress.

Reviewer Name(s): Raj Gutta, BNL Compliant with EIA-748: No, see CAR below

Observations and Findings: The CAMs for both projects understood their scope of work, deliverables, and clearly discussed/supported their assessment of the Control Account status. The Mu2e and LBNF CD3a projects use activities and milestones to identify tasks and deliverables, and these are tracked monthly against the baseline schedule. The CAMs identified Performance Measurement Techniques (PMT) for use within their Control Accounts, with most of the methods being LOE or % Complete. To facilitate reliable performance measurement, the baseline schedules are generally decomposed to a level that progress is determined by task completion. For some longer duration tasks with % Complete PMT, the projects are using “Steps” within Primavera. While the use of Steps was observed in the CAM interviews, some of them were questionable and the CAM agrees that those steps need revision based on revised scope. There were a considerable number of instances where activities using the % Complete PMT spanned more than two accounting periods, but did not possess some objective measurement of work completed or remaining. It was observed during CAM interviews and data traces that most CAMs are not using a documented, objective basis for assessing EV. Some CAMs indicated that they walk around site to assess progress or take photographs. These need to be transferred or translated to quantifiable backup data that can be produced for proper assessment. Some CAMs even mentioned that they don’t control the work and they use whatever % complete the field person provides. The CAMs must develop a method for longer duration activities/work packages when the activity is delayed. See CAR-03 for further information.

12

CAR-03 Subject (Issue): Use of Steps and Quantifiable Back-up Data Referenced Guideline(s): 7 Referenced Data Trace: Detailed Project Schedule, CAM Interviews Description of Issue: 1. Upon reviewing the Mu2e detailed schedule and discussion in CAM interviews, the review team noticed that there are scheduled activities with the PMT of % Complete where the planned durations are greater than two accounting periods. These activities did not have objective measures for establishing status provided during the review. (47508.6.1.1080, 47508.5.1.001310, 47508.8.4.001005) 2. For activity 47504.10.002145, the CAM thinks the PMT should be % complete, but in P6 it is LOE. Upon further discussion, the review team was informed that the CAM inherited this CA and was either not given an opportunity to revise attributes as required or did not know that they could. Further referenced in CIO-03. 3. There were instances where partial credit was taken on steps and where the steps were not revised based on scope change. Partial credit on Steps lends itself to creating a standard percent complete application of the earned value. This practice should be reviewed. 4. Activity 47504.9.1.001320 does not have PMT code. This was identified as an error during the review and the review team was told that a change request was initiated to correct the issue. Recommendation: Establish objective measures for any activity that is spanning more than 2 accounting periods (or as identified in the FRA system description). This can be in the form of backup document maintained by CAMs or within the schedule/cost system. Avoid taking partial credit on steps. If it is required, consider break down the steps further. Review the PMT codes for inherited CAs. Process the change request to add the missing PMT.

Guideline 8: Establish and maintain a time-phased budget baseline, at the control account level, against which program performance can be measured. Initial budgets established for performance measurement will be based on either internal management goals or the external customer negotiated target cost including estimates for authorized but undefinitized work. Budget for far-term efforts may be held in higher level accounts until an appropriate time for allocation at the control account level. On government contracts, if an over-target baseline is used for performance measurement reporting purposes, prior notification must be provided to the customer.

Reviewer Name(s): Raj Gutta, BNL Compliant with EIA-748: Yes, with CIO

Observations and Findings: Both Mu2e and LBNF CD3a presented a time-phased budget baseline at the Control Account level, along with Control Account Plan (CAP) reports. Some BOE documents were reviewed, and they matched to the baseline data. Both projects utilize work packages (WPs) and planning packages

13

(PPs). The CAMs are very aware about the timeframe for decomposing PPs to measurable WPs. The budget values and overall baseline totals were validated against the dollarized RAM for each project, as well as the CPRs and WADs posted for this review. During CAM interviews, it was revealed that a few CAMs were unsure of how to close a Control Account once all work was completed in the account. This was a CIO in prior review and the Corrective Action Plan states that Desktop Instructions exist, but some CAMs are not aware of it. Training all CAMs on this process would be beneficial. The schedules are resource loaded with detailed estimates of elements of cost and the total costs. The resources selected seem appropriate for the scope of work. However, slips that have been identified in the schedule are often attributed to a lack of resources to perform the work. This would lead one to conclude that a resource analysis or resource leveling has not been done on the project. See CIO-02 below for more information.

CIO-02: Subject (Issue): Resource Leveling Within Activities to Reflect Required Timephasing Referenced Guideline(s): 8 Reference Data Trace: Detailed Schedule Review and CAM interviews Description of Issue: During CAM interviews, it was noted that slips identified in the schedule are due to a lack of resources to perform the work. From discussions, it did not appear that a detailed resource analysis nor an exercise in resource leveling has occurred for the remaining work. Recommendation: Resource analysis should be performed and the availability of resources to the project should be translated to the schedule. Leveling the resources for those that are actually available will provide a more accurate picture of how long completion of tasks will take.

Guideline 9: Establish budgets for authorized work with identification of significant cost elements (labor, material, etc.) as needed for internal management and for control of subcontractors.

Reviewer Name(s): Raj Gutta, BNL Compliant with EIA-748: Yes

Observations and Findings:

14

The budgets for both Mu2e and LBNF CD3a projects were established using a BOE process that entailed the identification of resources needed to complete the scope of work. Each project utilizes detailed WADs at the Control Account Level, which include budget for authorized work with identification of significant cost elements such as labor and materials. These WADs are updated after each baseline change. Budget values presented were consistent with the budget values shown on the posted RAM, CAP reports, CPR, and WADs with the exception of those already noted in CAR-02. There were instances where negative budget values were found on some activities. These were done as single point adjustments but FNAL Project Management should continue to refine the process for establishing the baseline in a way that it eliminates the use of negative budget values. The CAM’s were knowledgeable about the process to plan and authorize work, and appeared compliant with the process. They also have a clear understanding of the change control process. Each project’s resource loaded schedule contained detailed cost estimates for each control account that were coded multiple ways, to include element of cost. The budgets were traceable to the WADs and the RAMs. The CAMs were able to explain the resources needed for their scopes of work and were knowledgeable of resource constraints.

Guideline 10: To the extent it is practicable to identify the authorized work in discrete work packages, establish budgets for this work in terms of dollars, hours, or other measurable units. Where the entire control account is not subdivided into work packages, identify the far term effort in larger planning packages for budget and scheduling purposes.


Observations and Findings: Both projects breakdown their budget into discrete work packages and planning packages that summarize to Control Accounts in the RAM. The budgets are loaded into P6 at the activity level by hours and direct material dollars as measurable units. These units are priced in Cobra to establish the time-phased budget using rates established by the Budget Office. The work packages have a Performance Measurement Technique (PMT) assigned. CAMs interviewed are aware of PP and WP concepts. Actual costs are collected at the control account level for both projects and variance analysis is performed at the Control Account level.

Guideline 11: Provide that the sum of all work package budgets plus planning package budgets within a control account equals the control account budget.


15

Observations and Findings: The sum of all the work package and planning package budgets equals the Control Account budget for Mu2e and LBNF CD3a. The RAM was used to verify the budget in the project baseline and budgets reported in the CPR and it reconciles through a data trace conducted for both projects with the exceptions as noted in CAR-02.

Guideline 12: Identify and control level of effort activity by time-phased budgets established for this purpose. Only that effort which is unmeasurable or for which measurement is impracticable may be classified as level of effort.


Observations and Findings: Both projects have work packages that contain LOE work scope. The LOE activities were typically project management or oversight support activities and distinctly separate from discrete work. During the CAM interviews and discussions, there was an understanding of LOE and its proper use. Time-phased budgets are established on both projects and the use of LOE activities within the schedule appeared identifiable and coded as LOE. There were few questionable instances (see below) which might be across the projects. There should be a review of the schedule for these kinds of errors and update as required. CAR-03 contains similar information.

1. CAM says he wants activity 47504.10.002145 to be a % complete PMT. But in P6 it is coded as LOE.

2. CAM says he doesn’t control the activity 47504.10.000748 and the percent complete type in P6 is noted as duration which makes it an LOE though the PMT says it should be % complete.

Guideline 13: Establish overhead budgets for each significant organizational component of the company for expenses which will become indirect costs. Reflect in the program budgets, at the appropriate level, the amounts in overhead pools that are planned to be allocated to the program as indirect costs.

Reviewer Name(s): Betsy O’Connor, ANL Compliant with EIA-748: Yes

Observations and Findings: The Budget Office, reporting to the Chief Financial Officer, is responsible for establishing the Lab’s indirect rates. The process begins in late spring or early summer with the issuance of guidance to the direct and indirect areas of the Lab. This guidance includes assumptions, like salary increases, and incorporates any new initiatives or other anticipated changes that may impact the rate calculation.

16

The direct program spending base is difficult to predict and funding may be more at risk for Fermilab, a single program Lab, especially as the DOE High Energy Physics Program Office shifts funding from research to special projects that may not involve Fermilab. When indirect rates increase, generally the perception is that the pool (indirect budget) increased when typically the spending base decreased. The pools are kept flat from year to year if possible, but the addition of Laboratory Directed Research and Development (LDRD) did result in an increase in the indirect rates for the last several years. In addition, indirect expense is assessed once on the first $500K of a contract, so work that carries over from year to year recovers no indirect expense after the initial year, further eroding the spending base. The labor-based indirect rates, like the fringe and vacation rates, are computed by the Accounting Department. The indirect and labor rates are reviewed and approved by the Chief Financial Officer and communicated to Laboratory Management. Upon approval, the Accounting Department issues a memo outlining all of the rates for the current fiscal year at the beginning of the fiscal year.

Guideline 14: Identify management reserves and undistributed budget.


Observations and Findings: Mu2e maintains a Management Reserve (MR) amount that is reported in the CPR each month. LBNF doesn’t have MR. Contingency for both projects are tracked in the change control log. MR for Mu2e is authorized through the Baseline Change process and is “replenished” from contingency. The Project Manager has authority over use of the MR and it is an allocation of funding through the change control process. Undistributed budget (UB) is not used on either project.

Guideline 15: Provide that the program target cost goal is reconciled with the sum of all internal program budgets and management reserves.


Observations and Findings: The Control Account budgets plus MR and contingency for Mu2e and LBNF CD3a reconcile to the total program budgets, as evidenced by reviewing and reconciling the RAM, CPR, and monthly reports and change control logs against the total project cost (TPC) for both projects.

17

2.3 Accounting Considerations

Guideline 16: Record direct costs in a manner consistent with the budgets in a formal system controlled by the general books of account.

Reviewer Name: Betsy O’Connor, ANL Compliant with EIA-748: Yes, with CIO

Observations and Findings: The financial information from the Oracle financial reports was traced to the contract performance reports for the Mu2e and LBNF projects and balanced without exception. In discussions with CAMs, most were involved in submitting accrual information and clearly understood the process. However, the materials receipt process for LBNF could benefit from a more defined receiving process that insures material costs are recognized when materials are received. See CIO-05 for more information.

CIO-05: Subject (Issue): Receipt of Material Costs Referenced Guideline(s): 16 and 21 Reference Data Trace: CAM Interviews, FRA EVM System Description Description of Issue: The LBNF CAM works in South Dakota and was very knowledgeable about the accrual process. When asked about receipts for materials delivered to the remote South Dakota location, the CAM stated that typically he is contacted by someone from Fermilab who asks if the materials ordered were received. It was not clear if he was contacted by the Receiving Department, Procurement, Accounts Payable or some other employee, or what prompted the individual to contact the CAM, e.g. the expected receiving date, the invoice, month end, etc. The Cost Accounting Standards Disclosure Statement indicates that the cost for materials is recorded when the materials are received, which follows Generally Accepted Accounting Principles and DOE guidelines. However, since the CAM is unaware of the receipt process for items received at the remote LBNF location, material costs might not be recorded in a timely manner or in accordance with Disclosure Statement. The EVM System Description (Section 4.1.5 “Material Cost and Accounting”) states, “…Fermilab accounting procedures are in place to record receipt of materials not physically received at Fermilab.” If procedures do exist, they should be communicated and included in the CAM training. Recommendation: Fermi should consider developing a formal process for receiving materials at remote locations if needed. The procedure can be very simple, such as directing recipients to email the Lab’s centralized Receiving Department when goods are received to insure that materials costs are recorded in a

18

timely manner. If a process is in place, it should be communicated to all off-site employees who may receive materials.

Guideline 17: When a work breakdown structure is used, summarize direct costs from control accounts into the work breakdown structure without allocation of a single control account to two or more work breakdown structure elements.

Reviewer Name: Betsy O’Connor, ANL Compliant with EIA-748: Yes

Observations and Findings: Both the Mu2e and LBNF projects provided crosswalks from the WBS to the Chargeable Task Codes (CTC) used in the Oracle financial system, indicating that the relationship between the WBS and the financial system codes was one to one at the lowest level. All traces between the financial statements and the contract performance reports confirmed the relationship. There were a few Variance Reports that mentioned that the cost variance was due to the unavailability of CTCs. In other words, time had not been charged to a project because the CTC was not opened yet. In discussing the process with the Accounting Department, they explained that the request for a new CTC is sent to Accounting first. Then, Accounting verifies the need for the CTC with the Budget Office who verifies that funding is available for the new CTC. After Budget approval, Accounting establishes the new CTC in the financial system. That process is relatively quick and efficient. However, some Project Managers prefer to control who can charge their projects at the labor entry point. Before labor can be charged to the new CTC in Kronos, the Fermilab Time and Labor system, the CTC has to be made available to the appropriate employees for charging. This seems to be the step that takes time and can delay the process.

Guideline 18: Summarize direct costs from the control accounts into the contractor’s organizational elements without allocation of a single control account to two or more organizational elements.


Observations and Findings: Costs are reported in the Oracle general ledger or subsystems to the general ledger which are then imported for reporting. Effort is reported weekly via the Kronos System, known as Fermi Time and Labor. The effort is reported in hours and priced out using standard rates. Managers can review the information reported for accuracy every week and request adjustments if errors are found.

19

Employee labor costs were traced from the official financial statement for the Mu2e project for September 2017. Three employees charged effort to 475.09.05 or Chargeable Task Code 475.685. The labor cost and hours were displayed in the effort reporting system by employee and the dollars were traced successfully to the financial statements, which balanced to the EVMS. The Variance Reports for the LBNF project identified an accrual in September and October as the cause of the variances in Control Account 131.01.02.02.04.02 Pre-Evacuation. In discussions, the CAM explained that the design deliverable was expected and received before the end of September and therefore the accrual was recorded. However, the deliverable was found to be unacceptable so performance for this activity was not claimed, thus the variance between cost and performance. The September accrual and October reversal were traced to the official financial reports. The cost was partially re-accrued in October. The CAM was very knowledgeable about this situation and appropriately did not claim performance even though the cost was reported.

Guideline 19: Record all indirect costs which will be allocated to the project.


Observations and Findings: A Mu2e project was selected randomly to trace the October 2017 overhead application, WBS 475.09.05 or Chargeable Task Code 475.685. The labor cost was assessed the Fringe rate, the Other Paid Time Off (OPTO) rate, and the Vacation Allocation. The labor base plus the labor overheads were assessed the Program Support, Common Site Support and General & Administrative overheads in accordance with the Lab’s Cost Accounting Standards Disclosure Statement. The indirect assessment for the materials cost for the same WBS was also reviewed. In accordance with the Disclosure Statement, the materials cost was assessed the Materials & Subcontracts overhead and the General & Administrative overhead. The calculation of each individual assessment was correct and traced to the official financial statements and the October 2017 Contract Performance Report for the Mu2e project.

Guideline 20: Identify unit costs, equivalent unit costs, or lot costs when needed.

Reviewer Name: Betsy O’Connor, ANL Compliant with EIA-748: Not Applicable

Observations and Findings: This guideline refers to manufacturing organizations only. Therefore, this guideline is not applicable to Fermilab. Guideline 21: For EVMS, the material accounting system will provide for:

20

• Accurate cost accumulation and assignment of costs to control accounts in a manner consistent with the budgets using recognized, acceptable, costing techniques.

• Cost performance measurement at the point in time most suitable for the category of material involved, but no earlier than the time of progress payments or actual receipt of material.

• Full accountability of all material purchased for the project including the residual inventory.

Reviewer Name: Betsy O’Connor, ANL Compliant with EIA-748: Yes, with CIO*

Observations and Findings: In discussions with the CAMs, the cost of materials is recorded when the materials are received. However, the materials receipt process for LBNF could benefit from a more defined receiving process that insures material costs are recognized when materials are received. See CIO-05, in GL-16, for more information. The Cost Accounting Standards Disclosure Statement also states that the cost of materials is recorded upon receipt, as does the EVM System Description. However, the capitalization criteria in the EVM System Description did not reflect the criteria stated in the Disclosure Statement. See CIO*-04 below, for further information.

CIO*-04: Subject (Issue): Capitalization Criteria Referenced Guideline(s): 21 Referenced Data Trace: EVM System Description, Section 4.4, “Burdened Costs” Description of Issue: The EVM System Description states, “Fermilab follows DOE policy of capitalizing assets with a useful life of 2 years or more and a cost of $50,000 or greater.” DOE changed the capitalization threshold from $50,000 to $500,000 beginning in FY2012. The Fermilab Cost Accounting Standards Disclosure Statement indicates the correct threshold of $500,000 and in practice, only assets with a cost of $500,000 or greater are capitalized. Recommendation: Section 4.4 of the EVM System Description should be modified to reflect the correct capitalization threshold of $500,000.

21

2.4 Analysis and Management Reports

Guideline 22: At least on a monthly basis, generate the following information at the control account and other levels as necessary for management control using actual cost data from, or reconcilable with, the accounting system:

• Comparison of the amount of planned budget and the amount of budget earned for work accomplished. This comparison provides the schedule variance.

• Comparison of the amount of the budget earned and the actual (applied where appropriate) direct costs for the same work. This comparison provides the cost variance.

Reviewer Name: Mike Barry, LBNL Compliant with EIA-748: Yes

Observations and Findings: Each month, Project Controls provides performance reports to CAMs including a CPR/Format 1 at the Control Account Level (WBS Level 3 for Mu2e), and a Stoplight Report at Control Account Level for both projects and at the work package level for Mu2e. Both reports include current month and cumulative-to-date values for planned budget, earned and actual costs, and accurately calculate Schedule Variance and Cost Variance. The Schedule and Cost Performance Indices (SPI/CPI) are calculated on the Stoplight Summary report for current and cumulative-to-date values. Project Controls also provide SPA and SPI/CPI graphs that display the trends for the variances and indexes. CAMs frequently use the CPR, Stoplight Reports, and graphs to perform analysis and they find them very useful.

Guideline 23: Identify, at least monthly, the significant differences between both planned and actual schedule performance and planned and actual cost performance, and provide the reasons for the variances in the detail needed by program management.

Reviewer Name: Mike Barry, LBNL Compliant with EIA-748: Yes, with CIO*

Observations and Findings: Each month Project Controls provides performance reports to CAMs on both LBNF and Mu2e. The reports include a CPR/Format 1 at the Control Account Level (WBS Level 3), as well as a Stoplight Report at summary values at the Control Account Level for both projects and the work package level for Mu2e. Both reports include current month and cumulative-to-date values for planned budget, earned and actual costs and accurately calculate Schedule Variance and Cost Variance. The SPI/CPI are calculated on the CPR and Stoplight Report for both current and cumulative-to-date values. Both reports are color coded to coincide with the variance thresholds as defined for each project.

22

For Mu2e and LBNF, thresholds are defined in the Monthly Status Reporting procedure (Revision 6.1), section 4.3, Management Variance Analysis Report and Thresholds. LBNF and Mu2e Variance Thresholds

According to the Monthly Status Reporting procedure (Revision 6.1), section 4.3, CAM Variance Review and Analysis, variances exceeding red thresholds, require the responsible CAM to prepare a VAR and have the report reviewed and accepted by the Project Manager. However, it is currently expected that both projects write VARs when the yellow thresholds are exceeded. A selection of VARs written for Mu2e (August 2017 – October 2017) and LBNF (September 2017 and October 2017) were reviewed for completeness, quality, and proper management approval. Each VAR for LBNF was found to include appropriate root cause explanations, impact, and corrective action (if needed). Each report had the proper management approval as well. The Mu2e VAR quality and consistency is discussed in CIO*-06 below.

CIO*-06 Subject (Issue): Variance Analysis Report (VAR) Quality and Use of Corrective Actions is Not Consistently Applied Referenced Guideline(s): 23 Referenced Data Trace: Mu2e VARs from August 2017 through October 2017, the following 4 Control Accounts specifically:

• Control Account 475.03.05 • Control Account 475.04.03 • Control Account 475.04.05

23

• Control Account 475.04.10 Description of Issue: GL23 states that each VAR must provide the reasons for the variances in the detail needed by program management. It further states that a VAR should be analyzed at a level of detail required to manage the effort; i.e., to enable management decision-making and corrective action. The Mu2e Variance Analysis Reporting (VAR) quality and use of Corrective Actions is not being consistently applied based on an evaluation of the VARs outlined below. CA 475.03.05: VAR is showing that both budget +$316K and earned +$314K or 99% complete, have occurred and that there are no associated actuals. The explanation for the variance is that FESS hasn’t opened "project/task codes" so the project team hasn’t charged the CA. This variance has been an issue since at least August 2017. BAC for this is $316K, so the work is behind them. No Corrective Action and no entry in the Corrective Action log occurred due to this VAR. We were told by the PM in interviews that the appropriate scope resides on another Control Account and that the costs reside with that Control Account. The PM wants to accept the underrun and close the Control Account. The lack of sufficient detail in the explanation hasn’t enabled management to get to the root cause and enact a Corrective Action. CA 475.04.03: VAR is showing a -$4.6M CV. The Impact states that $3.6M is attributed to historic variances and is not recoverable. Most of the variance is associated with the setup of the test facilities (CHL and HAB) use to test the TS prototype magnet and the production TS test units. However, it should be noted that this CA is still generating current period negative variances. The PM indicates that there is only a month of work remaining and that the VAR will be edited to reflect an acceptance of the overrun. We were also told this should have had a corrective action, but now they are a month away from completing the work and will change the narrative in the VAR. CA 475.04.05: VAR is showing a -$758K CV. The explanation is that the tasks were underestimated and also variances were being masked by underruns on the feedbox design. The CA is only 51% complete. We were told again that this work is nearing completion and that they will accept the overrun in costs. This was not supported by the current details in the VAR. CA 475.04.10: The narrative in the VAR is very different from the information, provided in the CAM’s presentation, regarding these same variances. The CAM also stated that the work is on the critical path. This comment, per the Project Manager, was not accurate because the cable tray work is far from the Critical Path. This resulted in an unclear representation of a -$414K SV. Recommendation: Together, the PMs and CAMs should review the VARs for accuracy before approval. This would alleviate any confusion regarding the explanation and impact of the VAR, and it would aid in deciding whether or not a Corrective Action should be generated and tracked in the log.

24

Guideline 24: Identify budgeted and applied (or actual) indirect costs at the level and frequency needed by management for effective control, along with the reasons for any significant variances.


Observations and Findings: A Mu2e project was selected randomly to trace the October 2017 overhead application, WBS 475.09.05 or Chargeable Task Code 475.685. The indirect expense charged to the labor cost and the materials cost was correct and traced to the official financial statement and the October 2017 Contract Performance Report for the Mu2e project. The Budget Office and Accounting Department monitor the performance of indirect rates monthly and may recommend a change to the indirect rates mid-year. Any change is made retroactively to October 1st and the change impacts the cost in the current month only. Project Managers are not required to submit a BCR for changes to the indirect rates. CAMs were well aware of the impact of indirect rate changes to their projects but recognized that such changes are out of their control.

Guideline 25: Summarize the data elements and associated variances through the program organization and/or work breakdown structure to support management needs and any customer reporting specified in the project. Reviewer Name: Mike Barry, LBNL Compliant with EIA-748: Yes

Observations and Findings: Each month Project Controls provides performance reports to CAMs including a CPR/Format 1 at the Control Account Level (WBS Level 3), as well as a Stoplight Report at the Control Account Level and Mu2e also provides the report at the work package level. Both reports include current month and cumulative-to-date values for planned budget, earned and actual costs and accurately calculate Schedule Variance and Cost Variance. SPI/CPI are calculated and are included in the CPR and Stoplight Reports for both current and cumulative-to-date values. The Stoplight Reports are color coded to coincide with the variance thresholds as defined for each project. Both projects utilize the standard variance thresholds defined in the Monthly Status Reporting procedure (Revision 6.1), section 4.3, Management Variance Analysis Report and Thresholds. CAMs stated they frequently use the CPR and Stoplight Reports to perform analysis and find them very useful. While CAMs also receive Control Account Level EV performance charts and SPI/CPI and SPA trend charts, the LBNF CAM did not speak to them. And this could be due to the CA and CAM just beginning the EV reporting. PM and CAMs on Mu2e, both showed the charts in their presentation and spoke to them accurately.

25

Guideline 26: Implement managerial action taken as the result of earned value information. Reviewer Name: Mike Barry, LBNL Compliant with EIA-748: Yes

Observations and Findings: The Monthly Status Reporting procedure (Revision 6.1), section 4.2, CAM Variance Review and Analysis, states, “After accepting the analysis, the Project Manager (or designee) may note any required corrective action in the Corrective Action Log. These corrective actions must be tracked to completion.” Both projects maintain Corrective Action Logs and review the logs during the monthly PMG and POG meetings. Improvements could be made in this area, see CIO*-06 for further information. However, it is evident that managerial action is being taken as a result of the monthly EV variance analysis:

• For example, the September 2017 VAR for LBNF, WBS 131.01.02.02.04.02, indicated a negative CV associated with an erroneous accrual in the Pipe Conveyer Preliminary Design CA. To correct this the accrual, the CAM took action by requesting that finance reverse the accrual, as evidenced by the correction in CV in the October VAR.

• In another example, in a Mu2e VAR previous to the 3 months of VAR reviewed, on the Cosmic Ray Veto Electronic, WBS/CA 475.08.06, the VARs helped the CAM to realize that the CA was incurring costs, but there was not any associated earned value being recorded. To address this situation of underperformance, the scope was reassigned to the University of Virginia via BCR #37.

Guideline 27: Develop revised estimates of cost at completion based on performance to date, commitment values for material, and estimates of future conditions. Compare this information with the performance measurement baseline to identify variances at completion important to company management and any applicable customer reporting requirements including statements of funding requirements. Reviewer Name: Mike Barry, LBNL Compliant with EIA-748: Yes

Observations and Findings: The FRA-FNAL EVM-SD (version 8.1), section 5.2.6, Project Performance Analysis, states, “Control Account Managers periodically develop a comprehensive Estimate at Completion (EAC) at the Control Account Level using all available information to arrive at the best possible estimate.” Also in section 5.2.6, “On a monthly basis, CAMs review the status of expended effort and the

26

achievability of the remaining forecasted work using all available information to arrive at the best possible EAC.” CAMs for both projects, Mu2e and LBNF, indicated that they owned and reviewed their EAC values monthly. Each month, per the Monthly Status Reporting procedure (Revision 6.1), section 4.5, at the Activity level of the P6 forecast schedule, the CAMs enter their estimated remaining units in hours and direct dollars. Through the monthly integration process with Cobra, this information is used to generate an Estimate to Complete (ETC) value which is then used to derive the EAC. Each month, CAMs are provided with calculated EAC values down to the work package level on both the CPR and the Stoplight Report. These two reports also include a Variance at Completion (VAC) column that is calculated by taking the difference between BAC and the newly calculated EAC value.

2.5 Revisions and Data Maintenance

Guideline 28: Incorporate authorized changes in a timely manner, recording the effects of such changes in the budgets and schedules. In the directed effort prior to negotiation of a change, base such revisions on the amount estimated and budgeted to the program organizations.

Reviewer Name: D. Sherese Humphrey, ANL Compliant with EIA-748: Yes

Observations and Findings: The CAMs interviewed were able to discuss the FNAL change control process, their involvement in initiating, reviewing, and implementing the change. The LBNF/DUNE and Mu2e Projects have processed changes in accordance with the Fermi Research Alliance, LLC Earned Value Management (EVM) System Description. All changes implemented were processed in accordance with the prescribed guidelines.

Guideline 29: Reconcile current budgets to prior budgets in terms of changes to the authorized work and internal replanning in the detail needed by management for effective control.


Observations and Findings: The LBNF/DUNE and Mu2e Projects have processed changes in accordance with the Fermi Research Alliance, LLC Earned Value Management (EVM) System Description. Each project maintains a change control log that details the management of management reserve and its

27

application within the PMB. All changes to the PMBs were recorded on the project’s change control log. The baseline integrity of each project is being maintained to ensure effective management of the cost and schedule. Budgets appeared to reconcile and changes reflected in the before-after schedule report and the before-after baseline budget compare reports. Where changes were bundled, detail was provided for management control and traceability, however, it was difficult to navigate without the help of the project controls group. See CIO-07 for more information.

Guideline 30: Control retroactive changes to records pertaining to work performed that would change previously reported amounts for actual costs, earned value, or budgets. Adjustments should be made only for correction of errors, routine accounting adjustments, effects of customer or management directed changes, or to improve the baseline integrity and accuracy of performance measurement data.


Observations and Findings: The LBNF/DUNE and Mu2e Projects have processed changes in accordance with the Fermi Research Alliance, LLC Earned Value Management (EVM) System Description. There was no evidence found on either project that retroactive changes were being performed to mask variances or estimates related to schedule and cost completion. Instances where current period changes had to occur, the FPD signed the change request.

Guideline 31: Prevent revisions to the program budget except for authorized changes.


Observations and Findings: Data traces of the documents provided revealed no apparent evidence that the projects’ scopes, cost, or schedule baselines had been revised without properly authorized change requests. The LBNF/DUNE and Mu2e Projects have processed changes in accordance with the Fermi Research Alliance, LLC Earned Value Management (EVM) System Description. All changes were implemented were approved and implemented within the respective PMBs. Neither project is being managed as an “over-the-target” baseline.

Guideline 32: Document changes to the performance measurement baseline.

Reviewer Name: D. Sherese Humphrey, ANL

28

Compliant with EIA-748: Yes, with CIO

Observations and Findings: Both LBNF and Mu2e provided documentation that supported approved change requests. LBNF processes individual changes; and Mu2e processes bundled changes. The documentation provided by LBNF was easily traceable by the review team. There were some challenges trying to review and reconcile the Mu2e project documents, see CIO-07 for more information.

CIO-07 Subject (Issue): Traceability of Changes Referenced Guideline(s): 32 Referenced Data Trace: Review of Fermi System Description Section 6 Change Control Process and Change Control Procedure 12.PM-007 Section 4.6.4 Documents and Records. Data traces were performed on the following Mu2e Control Accounts: 475.02.07 External Beamline; 475.04.09 Solenoids Ancillary Equipment and 475.04.10 Solenoids System Integration, Installation and Commissioning; and 475.08.04 Cosmic Ray Veto Fibers. Description of Issue: Independently, the review team found it quite challenging reviewing BCR documentation to ensure compliance with the Fermi Research Alliance, LLC Earned Value Management (EVM) System Description. Fundamentally, review teams should be able to review baseline change request by control and trace the approved change to the associated project documents. This was not easily done by the reviewer, which resulted in their review being dependent on the Mu2e project controls staff leading the trace to confirm compliance. The review team also found it challenging to understand when the approved changes would be implemented within the project baseline. The project’s change control log only reflected the approval date. This made it hard to determine the implementation period when the approval date reflected approval during the next period. Lastly, there is a CAR (CAR-01 WBS Dictionary Updates) associated with this Guideline. The WBS Dictionary was not being updated in a timely fashion to reflect scope changes. Further information on this noncompliance can be found under Area 1 Organization, CAR-01. Recommendation: It is recommended that the Mu2e Project consider preparing a guide for reviewers to understand what the package contains and how to drilldown into the information to verify compliance. At the very

29

least, the team should have a pre-meeting with future review teams to assist in outlining the BCR documentation process. It is also recommended that the Mu2e Project consider adding a column either within the BCR, BCR summary, and/or change log summary sheet that reflects the BCR’s implementation period for ease of understanding the reporting month of implementation.

30

Appendix A – Surveillance Plan

Fermi Research Alliance, LLC Earned Value Management System

Annual Surveillance Review Plan

29-Nov-2017

of 11

Earned Value Management System (EVMS) Surveillance Plan Overview

The Fermi Research Alliance, LLC (FRA) EVMS was certified by the DOE Office of Engineering and Construction Management (OECM) in January 2010. FRA management maintains an effective and efficient EVMS system though surveillances, which includes an annual assessment of compliance with the FRA EVMS requirements. FRA continuously improves its EVMS processes by examining the most current techniques and processes to manage projects. The FRA EVMS Surveillance and Maintenance procedure (12.PM-008) establishes the methodology for FRA EVMS surveillances. This specific Surveillance Plan summarizes the approach to be used to complete the surveillance of the certified FRA EVMS.

1. SURVEILLANCE OVERVIEW A Surveillance is defined as the process of reviewing the implementation of the EVMS process on one or more programs or projects. The purpose of an EVMS surveillance is to ensure the effectiveness of the EVMS to monitor and manage cost, schedule, and technical performance for FRA and its customers. An effective surveillance process provides an assessment of the current implemtation as well as encourages continuous improvement of the FRA EVMS process for use in current and future projects.

2. OBJECTIVES OF REVIEW Maintaining an EVMS is a requirement within the DOE FRA contract, (DE-AC02-07CH11359). The specific goals of the annual FRA EVMS surveillance is to confirm that the FRA processes and procedures continue to satisfy the guidelines in the American National Standards Institute/Electronic Industry Alliance’s (ANSI/EIA) 748-C Standard for Earned Value Management Systems as well as to ensure that FRA EVMS processes and procedures are being implemented appropriately. An overview of the surveillance process includes a review of each of the ANSI guideline categories: 1 Organization 2 Planning, Scheduling, and Budgeting 3 Accounting Considerations 4 Analysis and Management Reports 5 Revisions and Data Maintenance

http://www.fnal.gov/directorate/OPMO/PolProc/12.PM-008/Procedure_12.PM-008_EVMS_Surveillance_and_Maintenance.pdf

of 11

3. SCOPE OF REVIEW The surveillance review scope is limited to the review of the FRA EVMS System Description (SD) and procedures as they pertain to ANSI Standard 748-C. The surveillance is not a review of the status or performance of the individual projects included in the review, because project status and performance issues are addressed separately by other reviews. This review focuses on two projects: 1. The muon-to-electron (Mu2e) conversion experiment and the LBNF/DUNE project. The Mu2e project has been officially baselined and is required to comply with EVMS criteria for surveillance on the FRA certified system. The LBNF/DUNE project has not been baselined, but has been authorized to perform initial construction activities at the far site via a CD-3A. The activities authorized under the CD-3A are expected to comply with the FRA EVMS. LBNF/DUNE activities outside of the CD-3A scope are not expected to comply with FRA EVMS, and are not included in this review.

4. SURVEILLANCE MEMBERSHIP Surveillance membership for this review consists of participants from other labs to ensure independence of the surveillance process. Individuals participating in the surveillance review include the following:

FRA EVMS Representative

Richard Marcum – Fermilab National Accelerator Laboratory Project Controls Manager, Office of Project Support Services (OPSS)

Surveillance Team Reviewers

Jennifer Fortner (team lead) – Argonne National Laboratory, Project Controls Program Manager

Michael Barry – Lawrence Berkeley National Lab, Project Controls Group Lead

Rajendra Rao Gutta – Brookhaven National Laboratory, Project Management Specialist

Sherese Humphrey – Argonne National Laboratory, Project Controls Analyst

Betsy O’Connor – Argonne National Laboratory, PMO Financial Manager

Observers

Denise Hoomes – Oak Ridge National Laboratory, Project Controls Specialist

of 11

EVMS Surveillance Team Assignments Team Member Responsible Area Guidelines

Jennifer Fortner Organization 1-3,5

Rajendra Rao Gutta Planning, Scheduling, and Budgeting 6-12,14,15

Betsy O’Connor Accounting Considerations 4,13,16-21, 24

Michael Barry Analysis and Management Reports 22,23,25-27

Sherese Humphrey Revisions and Data Maintenance 28-31

5. PROCESS AND GUIDELINE SELECTION All aspects of the FRA EVMS will be considered during this comprehensive system surveillance. The surveillance will address the full content of the FRA EVMS SD and will also consider the results of other DOE related reviews, as appropriate. As discussed above, this FRA EVMS surveillance will be based upon the remaining work and content that is specific to the LBNF/DUNE and Mu2e projects being reviewed. The selection of EVMS guidelines and processes to be reviewed will be relevant to the projects’ phases. This surveillance is organized to provide a structured setting to assess the LBNF/DUNE and Mu2e projects’ approach to the FRA EVMS process implementation and their consistent use across the projects’ Control Account Managers (CAMs). This is facilitated by:

• A clear code of conduct; • Understanding of how results will be used; • Including contractor and customer project office personnel as observers on the

surveillance team; • Obtaining out-briefings and discussions of potential Corrective Actions and Continuous

Improvements before a report is generated; • A clearly defined format for reporting Corrective Actions and Continuous Improvements.

6. CODE OF CONDUCT

Responsibilities The surveillance team will provide adequate advanced notification of specific control accounts and processes that will be reviewed based on data that will be provided by the project team prior to the initiation of the on-site surveillance. It is the intent of this surveillance to minimize any impacts with on-going project work to the extent possible. The surveillance team will not

of 11

require extensive presentations or preparations, but rather focus on the review and interpretation of data provided in the selected projects’ native formats. The review will be conducted in a professional manner and in a spirit of constructive assessment and discovery. The surveillance team leader is solely responsible for the final determination of Corrective Actions and Continuous Improvements and ensuring that the results are communicated to the project and Laboratory management. Project personnel should be prepared to demonstrate through objective project information that they are complying with applicable FRA EVMS policies and procedures. The project team members should coordinate with the surveillance team to ensure that CAMs responsible for areas of specific interest are available while minimizing the impact on ongoing project activities. The project personnel should also ensure that adequate data and project policies are available to the surveillance team sufficiently in advance of the review to allow for meaningful analysis. For this review, “sufficient data” is defined as three consecutive months of recent project data provided at least two weeks prior to review. In this case, the three most recent and available months are August, September, and October 2017. The surveillance team leader will ensure that the review focuses on system compliance rather than non-system-related issues or project specific performance issues. Additionally, the surveillance team leader will make certain corrective actions identified during any previous review were addressed appropriately. Observer Participation Observers are guests approved by the team leader to accompany the team, and observe the review process to ensure that the system under review is compliant. The non-DOE Observers may be assigned to assist team member(s) with their assigned responsible area. The purpose of having non-DOE observers participate in this process is for them to learn what the requirements are for a compliant EVMS and how surveillances are performed to validate that compliance. This experience is to enhance their EVMS knowledge and to aid them with their current and future job responsibilities. Project Information Successful surveillance is predicated upon demonstration of compliance with the FRA EVMS System Description and procedures through explanations and illustrations using objective project information consisting of documents, computer files, working papers, notes, or other forms of data and communication which demonstrate compliance/non-compliance with a policy, procedure, or process. Objective project information is created in the normal conduct of business and is not prepared solely for the review of a surveillance team. The surveillance team will be located in a central location that facilitates access to project information within the Laboratory. Examples of objective project information include work authorizations, critical decision documentation, cost and schedule status databases, variance analysis reports, and estimate-to-complete rationale. A complete list of required documents will be provided to the project team prior to the review date but additional documents may be requested during or

of 11

prior to the actual review. It is the intent that required documents will be available via web access to the surveillance team at least two weeks prior to the review. Orientation An orientation will be included at the start of the on-site EVMS review to introduce members of the surveillance and project teams and to discuss key EVMS-related forms and procedures. The surveillance team will use the orientation period to explain the goals and scope of the review, the code of conduct, the disposition of Corrective Actions/Continuous Improvement, and the resolution process. A brief overview of the nature of the selected projects will be provided by FRA to the surveillance team to ensure they understand the goals of the projects, unique language usage, and any unusual organizational relationships. Data Gathering The surveillance review will be conducted both through interviewing FRA management, CAMs, and project staff and by verifying the integrity of objective project information. The initial number and scope of interviews will be defined after the project team has provided a dollarized Responsibility Assignment Matrix (RAM), with the intent that it be provided no later than three weeks prior to the review date. At least two weeks prior to the review, the surveillance team will provide a list of CAM and project staff interviews to the project teams. The surveillance team request is to be a balance between obtaining sufficient data, meeting the time limitations for the review, and minimizing the impacts to ongoing project activities. The project teams will coordinate the scheduling of these interviews and provide an agenda to the surveillance team prior to the on-site review. Based on surveillance results, additional interviews may be conducted. Interviews will generally be conducted in a location close to the CAM’s office, which will facilitate ease of access to objective project information. During each interview, the surveillance team will assess the level of understanding and compliance with FRA EVM policies, procedures, and processes, while monitoring local practices to assess how well they comply with the intent of the EVM guidelines. The surveillance review will be thorough and structured. This involves developing a list of subject areas to facilitate scheduled interviews, ensuring that discussions address the complete EVMS process. CAM interviews are a key component of EVMS surveillance because CAMs are the source of much of the EVMS information. CAM interviews are supplemented with data integrity tests performed independently. The ultimate objective is to determine the CAMs’ use of the information derived from the EVMS as an effective management tool. All interviews will incorporate the common attributes based on the National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC) Intent Guide, May 2011 edition. The purpose of the interview is to assess the CAMs’ understanding and implementation of the following subjects:

of 11

1. Organization

a. Verify that the Work Breakdown Structure (WBS) contains (Guideline 1 Intent Guide)

i. All project work, including revisions for authorized changes. ii. All contract line items and end items.

iii. All external reporting elements. iv. Elements extended to the control account level. v. Maps to the WBS dictionary.

b. Verify that a Work Authorization with scope, schedule, and budget exists at control account level (Guideline 2 Intent Guide). Verify that external Work Authorization with the identified Customer exists, at least, at the project level.

c. Verify that the Organizational Breakdown Structure (OBS) is documented (Guideline 3 Intent Guide).

d. Verify that the same WBS is linked between schedules, work authorization, and control account plans (Guideline 3 Intent Guide).

e. Verify that the there is a documented process and organizations established to specifically manage and control indirect costs (Guideline 4 Intent Guide).

f. Verify that Responsibility Assignment Matrix or equivalent documents control accounts at appropriate level (Guideline 3 & 5 Intent Guide).

2. Planning, Scheduling and Budgeting

a. Ensure Project Schedule specifics (Guideline 6 Intent Guide) i. WBS/OBS identifiers exist in the project schedule at activity level for

summarization. ii. Project schedule reflects entire WBS Dictionary.

iii. Critical target/contractual dates are identified in the project schedule. iv. The project schedule identifies significant interdependencies. v. Task durations are meaningful and relatively short.

vi. Longer tasks use objective earned value techniques. vii. Resource estimates are reasonable and consistent with the schedule.

viii. The baseline is reasonable to achieve project requirements as demonstrated through schedule analysis techniques.

ix. The project schedule baseline is established. x. The schedule provides current status and forecasts of completion dates

for all discrete work. xi. The project has a critical path.

b. Verify that objective completion criteria are used as basis to determine achievement (Guideline 7 Intent Guide).

c. Verify that CAM updates schedule status (Guideline 7 Intent Guide). d. Verify that the integration of scope, schedule and budget at the control account

level (Guideline 8/9 Intent Guide). e. Verify that the time-phased Performance Measurement Baseline (PMB) equals

the work authorization and summarizes above the control account to the contract value (Guideline 8/9 Intent Guide).

of 11

f. Verify that control account budgets identify elements of cost including subcontractor (Guideline 9 Intent Guide).

g. Verify that management reserve and undistributed budget, if any, track to logs (Guideline 9/14 Intent Guide).

h. Verify that schedule and cost variances are collected at control accounts (Guideline 10 Intent Guide).

i. Verify the work packages are uniquely identified, have a budget, and have an earned value technique (Guideline 10 Intent Guide).

j. Verify that planning packages are not in the current month and reflect the manner in which the work will be performed (Guideline 10 Intent Guide).

k. Verify that the control account work packages and planning packages (if any) add to the control account total budget (Guideline 11 Intent Guide).

l. Identify level of effort designated work is appropriately categorized and identifiable (Guideline 12 Intent Guide).

m. Verify there is a documented process for managing indirect costs with an organizational structure identifying ownership, responsibility, authority levels, and distribution of indirect costs reflected in the program budgets at the appropriate level. (Guideline 13 Intent Guide). .

n. Verify that management reserve and undistributed budget logs reconcile with last two months of Cost Performance Reports (CPR) (Guideline 14 Intent Guide).

o. Verify that baseline control logs reconcile with performance measurement baseline (Guideline 15 Intent Guide).

3. Accounting Considerations

a. Verify that Actual Cost of Work Performed (ACWP) in the CPR reconcile with books of record (Guideline 16 Intent Guide).

b. Verify that WBS and OBS summarize direct costs from one control account (Guideline 17/18 Intent Guide).

c. Verify that indirect costs are applied to the direct costs per Laboratory Policy (Guideline 19 Intent Guide).

d. Verify that unit cost are identified when needed (Guideline 20 Intent Guide). e. Verify that effective performance measurement is assessed on material no

earlier than point of receipt and consistent with the method budgeted (Guideline 21 Intent Guide).

f. Verify that an established process exists for reporting subcontractor costs and material actual costs (Guideline 21 Intent Guide).

4. Analysis and Management Reports

a. Verify that variance analysis is performed to the project thresholds as required (Guideline 22 Intent Guide).

b. Verify that variance analysis contains cause, impacts, and corrective action as appropriate (Guideline 22/23 Intent Guide).

c. Verify that corrective actions are assessed and closed in a timely manner (Guideline 23/26 Intent Guide).

of 11

d. Identify budgeted and applied (or actual) indirect costs at the level and frequency needed by management for effective control, along with the reasons for any significant variances (Guideline 24 Intent Guide).

e. Verify that variance analysis as reported to the customer reconciles with the analysis at the control account level (Guideline 25 Intent Guide).

f. Verify Estimate to Complete (ETC)/Estimate at Complete (EAC) (Guideline 27 Intent Guide)

i. Verify that Comprehensive EACs are updated per requirements and take into account efficiencies.

ii. Verify that CAMs review achievability of control account EAC monthly. iii. Verify that time-phased ETC reconciles with the EAC as reported

externally. iv. Verify that risks and opportunities are integrated into summary schedule

and ETC resource plans.

5. Revisions and Data Maintenance a. Verify that work authorization plus any baseline change documentation equal

current control account budget (Guideline 28/29 Intent Guide). b. Trace last change proposal authorized. Verify schedule and cost integration at

control account level and that the WBS is updated as appropriate (Guideline 28/29 Intent Guide).

c. Verify that change logs reconcile and contain justification (Guideline 28/29 Intent Guide).

d. Verify that retroactive changes are made only for correction of errors, accounting adjustments, effects of customer management directed changes to improve accuracy of data. If any have been made, verify that they are consistent with disclosed EVMS policy (Guideline 30 Intent Guide).

e. Verify, in at least one control account, that last month’s changes as reported to the customer and this month’s PMB reconcile to entries in the contractual baseline log (Guideline 30 Intent Guide).

f. Verify that negative earned value status, if any, has been adequately explained (Guideline 31/32 Intent Guide).

g. Verify that all baseline changes within a month reconcile to baseline control requests or the equivalent (Guideline 31/32 Intent Guide).

Additional interviewees may include the project manager, the project controls manager, and line management.

of 11

7. SURVEILLANCE RESULTS Concerns Identified During the Surveillance The surveillance team will gather data through reviewing documentation and conducting interviews. A key component of the surveillance is communicating timely, pertinent, and candid feedback. Surveillance team members and project personnel should seek clarification to fully understand questions asked, the data sought, and the responses provided. If, after fully understanding the information provided, a surveillance team member believes that there may be a question of compliance; the surveillance team will discuss the observation. If the surveillance team determines that the observation is still a question of compliance; the FRA project lead representative and pertinent members will be notified by the surveillance team of the concern no later than during the out-brief discussions at the end of each day. This gives the project the opportunity to supply the surveillance team additional information to clarify the observation. This may result in the “concern of the observation” being resolved, or may result in a recommendation for Continuous Improvement, or an observation of non-compliance requiring Corrective Actions. Corrective Actions/Continuous Improvement are defined as:

Corrective Actions - Corrective Actions fall into two broad categories: 1) non-compliance with the accepted EVMS description or procedures and 2) non-compliance with the ANSI/EIA 748-C EVMS guidelines. Failure to resolve Corrective Actions reduces confidence in the ability of project management to effectively use the EVMS process to achieve project goals and objectives of the stakeholders. A Corrective Action Plan is required for each finding. Continuous Improvements - The team members may recommend EVM implementation enhancements such as sharing of successful practices, tools, or other items that come to their attention. Continuous Improvements, however, are not the same as Corrective Actions and, therefore, need not be tracked for closure. However, should a recommendation have an asterisk (*), the team members have elected that this practice is critical enough to require tracking to closure.

Surveillance Final Out-Brief The surveillance team will evaluate what they have observed and the information received during the interviews from the project team and come to a consensus if any Corrective Actions/Continuous Improvement items should be issued. Also, the surveillance team should identify if the observations are systemic rather than isolated issues. Any Corrective Actions/Continuous Improvement items are to be presented by the surveillance team to the project team at the Final Out-Brief. It is possible that the project teams may disagree with the final surveillance results. When a finding is not due to a team’s misunderstanding, the FRA EVMS Representative must be able to explain the impact of deviating from FRA policy and the benefits to the project and management team of non-compliance with the intent of the EVMS guidelines.

of 11

Final Report The surveillance team will develop a preliminary report and give the FRA EVMS Representative the opportunity to provide any additional feedback in a reasonable timeframe. The surveillance team will take into consideration any feedback received when developing the final report. The final report will be issued by the surveillance team leader to the FRA EVMS Representative, Head of OPSS, and the Chief Project Officer (CPO). Dates for report delivery will be agreed to at the Final Out-brief. Corrective Action Plan The FRA EVMS Representative will develop a Corrective Action Plan (CAP) to address any Corrective Actions or Continuous Improvements identified in the Final Report from the surveillance team. The CAP should include a schedule with realistic dates for when the corrective actions are to be completed. The project personnel will provide input regarding corrective actions, including estimated completion dates. The surveillance team will receive a copy of the CAP for information only; unless it is determined at the Final Out-brief that further actions are required by the surveillance team – such as a follow-on review. Corrective Action Plan Processing and Tracking Problem areas identified during the assessment that are determined to be non-compliant with management system requirements or the organization’s implementing requirements will be reported as Corrective Actions , documented on Corrective Action Plans (CAPs), and processed in accordance with the Fermilab Corrective & Preventive Action Procedure, 1004.1001. Surveillance Review Close-out The FRA EVMS Representative is to ensure that CAP has been acceptably completed. The close-out of the CAP and any follow-up verification performed should be documented and retained for future EVMS surveillances. Table of Revisions Author Description Revision Date M. Kaducak Initial Draft Draft V0 11/16/2017 M. Kaducak Updated EVMS Assignments Table Guidelines V1 11/27/2017 M. Kaducak Updated ANSI 748B to 748C V2 11/29/2017

42

Appendix B – Review Agenda

Agenda Earned Value Management System Surveillance Review

Fermi Research Alliance, LLC (FRA) Fermi National Accelerator Laboratory (FNAL) – Bldg. Wilson Hall/Directors’ Conference Room (2nd floor east side)

December 6-7, 2017 PURPOSE: This will be a formal review of the Fermi Research Alliance, LLC (FRA) Earned Value Management System (EVMS) implemented at Fermi National Accelerator Laboratory (FNAL). Discussions will occur and interviews will be performed to determine how the current EVMS works and its implementation across all of FNAL (including such information as accounting, procurement, software tools and interfaces, scheduling, cost estimating, reporting, Control Account Managers (CAMs), work authorization, etc). The outcome of this meeting will be a formal assessment of the FRA EVMS by an independent review team, including corrective actions and continuous improvements. AGENDA:

Wednesday - Dec.6 Time Subject Session Leader/Speaker Location

8:00 - 8:30 Review Team In-room Setup and Discussions Closed Review Team

Director’s Conference Room, Wilson Hall 2nd floor East

8:30 - 9:00 Welcome / Safety Review / Review Team In-Briefing Closed Review Team, Marc Kaducak

9:00-10:30

EVM System Overview, Corrective Actions (30 min) Open Richard Marcum Mu2e Project Overview (20 min) Open Ron Ray LBNF EVMS Implementation Update (20 min) Open McCluskey/O’Sullivan Break (20 min)

10:30 - 12:00 CAM Interview #1 – Josh Willhite (LBNF) Closed Review Team

12:00 - 1:00 Lunch, Review Team Discussions Closed Review Team

1:00 - 2:30 CAM Interview #2 – Dean Still (Mu2e) Closed Review Team

2:30 – 4:00 CAM Interview #3 – Jeff Brandt (Mu2e) Closed Review Team 4:00 - 5:00 Review Team Meeting Closed Review Team 5:00 – 5:30 Informal Daily Out-Brief Open Review Team, FRA

Thursday – Dec.7 Time Subject Session Leader Location

8:00 - 9:00 Review Team Preparation Closed Review Team

Director’s Conference Room, Wilson Hall 2nd floor East

9:00 – 12:00 Accounting Interview – Jim Wollwert Closed Interview Team – Accounting

9:00 - 10:30 CAM Interview #4 – Craig Dukes (Mu2e) Closed Review Team

10:30 - 12:00 Review Team Work Session / Team Discussions / Data Traces Closed Review Team

12:00 - 1:00 Lunch, Review Team Discussions Closed Review Team

1:00 - 2:00 Management Interview [If Needed] – Marc Kaducak and/or Mike Lindgren, others as requested Closed Interview Team (Select

Members)

2:00 - 5:00 Resolve Any Remaining Issues & Interview Needs / Review Team Consensus / Writing of any CARs, CIOs Closed Review Team

5:00 - 5:30 Formal Out-Brief and Review of Next Steps Open Review Team, FRA