SNS Instrument Projects: Lessons Learned

John HainesORNL

Dec 7, 2007

2

Outline• Project approaches• Level of Technical Risk• Lessons learned

– Project Management– Technical Requirements– Cost– Schedule– Risk Management– Engineering– Procurement– Installation– Project Completion

3

What models have we used?

• Five SNS project instruments (BLs 2, 4A, 4B, 6, 11A)

• DOE grant funded instruments (ARCS and CNCS)

• DOE MIE funded instruments (SING, SING II, FNPB)

• WFO funded instruments (VULCAN, part of NSE)

Project Approaches:MIE vs. Grant Funding

5

Elements Needed to Conduct a Successful Science Project

Involved scientific

community

Experienced technical team with full array of

required expertise

Excellent management

team

6

What is the best way to conduct these types of projects?

• A project can be successful (or a failure) within almost any framework

• Some environments are more conducive to success

• The key is getting the right people and expertise

7

Model 1: Conducting the Project through the Facility, e.g. SNS instruments being designed and built by the SNS team

• Large engineering team working on multiple instruments share lessons learned and common designs

• Expert groups on detectors, neutron optics, DAS, choppers, vacuum, and sample environment are part of the team

• Facilitates standardization of components – Installation benefits– Maintenance and spares inventory

• Professional and experienced management team available

• “Bundled” procurements – economy of scale• Integration with science community achieved through

IDT leader as External PI and SNS Instrument Scientist as Sub-Project Leader

• National lab has higher cost structure compared to university

8

• Potential for more direct integration with science community

• Usually a lower cost structure• Scientists don’t always make the best project managers

– Requirements and timely decisions– Usually research focused as opposed to deliverable focused– May not rigorously use PM tools and processes

• More effort required to achieve integration with SNS facility– Physical interfaces– Compatibility with ORNL ES&H and labor standards

requirements– Less opportunity for sharing ideas and designs

Model 2: Conducting the project through an external science-based team, e.g. SNS instrument design and

construction being managed by a university

9

Our Solution: MIE Structure with Multiple Instruments in a Single Project and Strong Involvement from IDT for Each Instrument

• Accountability inherent in MIE structure and process improves likelihood of successful project completion– Disciplined process for establishing baseline and controlling

changes– Monthly EVMS reports are a useful management and

communication tool– Regularly scheduled DOE-SC Reviews/IPRs add value

• Helps incorporate lessons learned from other projects• Peer pressure is strong motivator

• Multiple instruments facilitates sharing of engineering designs and lessons learned

• Peer pressure from within the project can motivate individual instrument (subproject) teams

• Lowers overall management cost – economy of scale• Pooling of contingency lowers overall risk

Selecting the “Right” Level of Technical Risk

11

Technical Risk• Setting the appropriate level of technical risk is difficult• For an MIE, must aim for “world-class”, but must have

reasonable likelihood of success• Our approach seems to be working:

– Get approval/buy-in from IDTs in the form of external PIs to ensure that the instrument meets users needs/desires

• And also makes appropriate compromises– Aim as high as possible

• Set high goals in DCD, less aggressive goals in the PEP• Having IS lead subprojects (future owner and user) helps ensure

that we aim as high as possible• Using the MIE process (i.e. strict use of 413.3A) may not

be wise for all instrument projects• High risk/high benefit ventures should have a place too

Best Practices/Lessons Learned

13

Lessons Learned

• We have many detailed lessons we have learned in conducting instrument projects at SNS– Some are ORNL/SNS specific– Examples of more generic ones are provided

in following slides

14

Most of the Documents and Processes Required by DOE Order 413.3A and EIR Process Are Useful

• Detailed Resource Loaded Schedule • Detailed Cost Estimate • System Functions and Requirements Document (also referred to as the

"Design to“ requirements or Design Criteria)• Results of and Responses to Site Preliminary Design Review• Preliminary Design Drawings• Project Execution Plan• Start-up Test Plan (as appropriate)• Hazards Analysis• Risk Management Plan/Assessment • Acquisition Strategy • Value Management/Engineering Report

√√

√√

√√

√ √√

X

X

15

Value Management and Engineering Report

• Value Management and Engineering is an important process not a report– Requiring such a document adds little or no value,

and conveys a lack of understanding or appreciation for the process

– This document also leads to the perception that VE is something you think about one time during a project

– If a project team is actually using VE processes and principles, they will not have enough time/resources to document the items that have been considered

• Requiring or even attempting to provide such documentation would be an impediment to the process

16

Preliminary Design Drawings• Requiring Preliminary Design Drawings for a project is

an antiquated idea• Since the mid-1980’s high tech organizations have

designed, and in some cases, built hardware without producing any detailed drawings– Use 3-D CAD models to define parts– In modern, high tech organizations drawings are only required

for fabrication shops, i.e. a complete set of detailed drawings are only made at the very end, and never in some cases

• If an EIR team requires this item, it wastes time and resources and leaves the project team with the impression that the review team is not qualified, i.e. not in touch with modern practices

17

Design Criteria Document• Fundamental building block in NScD project

management• Defines technical scope, interface requirements, codes

and standards, … for a particular instrument• Used to satisfy EIR/CD-2 requirement - “EIR Item 6.

System Functions and Requirements document”• Essential for cost and schedule management/control• This is a living document that is updated as needed by

the subproject team – it provides a way of documenting the initial baseline and subsequent changes and evolutions during project execution– Change control is defined in the PEP– Used to define/confirm project completion

18

Management Principles• Strong project management team

– Technically qualified for the project– Highly motivated since they belong to the same organization as

the future users– Strong institutional support– Partnership with DOE

• Shared vision between DOE and ORNL– What is to be built and why– Realistic cost and schedule baselines

• Trust at all levels, open communications, and constructive criticism

• Emphasis on a rigorous safety culture• Active identification and resolution of issues before they

become significant; without waiting for reviews and written reports

19

Management Principles (cont’d)• Regular, disciplined peer review process on all

aspects of project, i.e. SC (Lehman) reviews and Advisory Committees

• Other reviews as needed (project management systems, design reviews, etc.)

• Constant, unrelenting control of cost / schedule using disciplined management systems

20

The need for highly trained and experienced personnel was recognized

early as a key to project success

• SNS successfully recruited and retained key staff by implementing the Office of Science-approved compensation and benefit tools – SNS HR Toolkit

– SNS identified as DOE pilot program to promote recruiting and the assignment of staff from other DOE national laboratories

– Provided project-level authority to act– Introduced or enhanced variable pay options for key personnel at ORNL or

partner laboratories– Included service recognition applied to specified benefits for inter-laboratory

transfers– Attracted key personnel quickly– Retained key personnel– Now approved by DOE for other projects, including US ITER

• Partner Labs were integrated into the Project– Project was managed as though it were contained within a single institution

• SNS staff were encouraged to obtain certification as Project Management Professionals (PMP)

– TWELVE key federal and contractor staff achieved PMP certification during the project

21

PM Tools Can Help• PM tools, e.g. EVMS, Risk Assessment, …, have proven

to be useful, so they should be judiciously used to maximize the probability of having a successful project

• On the other hand, formal PM systems are just tools– Administering the tools should not drive the project– These tools can be misused or overrated

• For example, existence of a formal risk assessment process does not prove that risks are being well managed, and conversely, lack of a formal risk assessment process does not mean that risks are not being well managed

• Using EVMS data to determine whether a project is “green, yellow, or red” is overly simplistic, a misuse of an otherwise excellent PM tool, and leads people to waste effort making useless adjustments to the project

22

Technical Requirements

• Document all significant technical requirements with the baseline– DCD serves this purpose– Use disciplined change control process to

make changes• Editorial comment: Use of this process is

probably the biggest factor in whether an instrument will meet its cost and schedule goals or not

23

Schedule• Sub-project leaders (Instrument Scientist and

Lead Engineer) must own the schedule– Define logic and durations and assess monthly status– Project Controls Staff advise and implement– Drives team to make timely decisions

• Purchase high risk items earlier than needed– This will obviously give more time for items with

schedule uncertainty– For cost risk items, it gives the project more time to

deal with the problem if an item comes in with a significantly higher cost

• Negotiate with bidders – cost drivers, reduced performance• Reduce non-essential scope elsewhere

24

Cost Estimating

• Realistic escalation (not required to use OECM guideline)– Use ORNL salary planning information for internal

labor– Use appropriate escalation rates for purchases

• Systematic process for contingency evaluation• Feedback from recent history

– Compare estimate to real costs from similar instruments (design, installation, procurement of similar equipment)

25

Risk Management

• Identify high risks and pay attention to their mitigation strategies– Concentrate on the critical few; don’t get

caught-up in the process and dwell on low risk items

• Bottoms-up input from sub-project leaders and the rest of the team

• Update regularly (quarterly seems to work)

26

Engineering

• Grouping instrument engineers together increases efficiency, uniformity, and helps share good ideas and practices– Shared designs– Common equipment minimizes spares

inventory and improves maintenance and installation

– Must create an atmosphere where mistakes and their lessons learned are shared

27

Procurements• Economy of scale• Foreign procurements/exchange rates – try to

get in fixed dollars• Phase funding to optimize BA usage• Involve all relevant technical staff before

awarding contract• Limited vendors with limited capacities

– Work closely with vendors– Regular weekly conferences from the beginning

• Dedicated procurement staff• Large vendor base

28

Engineering/Installation Interface

• Have installation drawings and documents complete before starting installation– Complete scope description before starting– Avoid verbal orders – RFI process

• Realistic alignment tolerances• Set liberal tolerances on concrete shield blocks

or in equipment that interfaces with concrete structures – If needed, fill-in gaps (grout or steel shot in socks)

and/or allow for alignment adjustment

29

Installation• Local management of installation is essential

– Allows leveling of resources– Prioritization of schedule– More efficient use of materials– Allows use of construction professionals familiar with

local labor rules and market– Improves safety – consistent rules and cadre of

trained staff and labor force– Better engagement by labor force– Facilitates feedback to design team

30

Project Completion• SING I and II Model gives clear definition of project

completion– Verify by equipment acceptance tests plus measurement and

calculations of integrated performance without beam– Includes passing Instrument Readiness Review, i.e. ready to

open the shutter and take beam– No operation with beam– Set critical requirements in PEP

• More aggressive goals in DCD• Using the Instrument Scientist as the sub-project leader (future

owner/user) and requiring instrument acceptance by a separate division (NFDD to NSSD hand-off) helps ensure integrity of process

– Formal process requiring sign-offs

31

Project Completion (cont’d)

• Another alternative requires that all measurements be made with beam– Significant completion delay may be caused

by SNS operations outage – beyond the control of the project team

– Delay may also be caused by instrument operations learning/tuning process

• Ready for users is the extreme case