fleet data diode implementation (final...

32
Fleet Implementation of Data Diodes Experiences / Lessons from Exelon Data Diode Implementation Project 2012 R*TIME User’s Conference Clearwater, FL Steve Cafrelli – Exelon BSC-IT

Upload: dangmien

Post on 07-Apr-2018

213 views

Category:

Documents


0 download

TRANSCRIPT

Fleet Implementation of Data Diodes

Experiences / Lessons from Exelon Data Diode Implementation Project

2012 R*TIME User’s Conference

Clearwater, FL

Steve Cafrelli – Exelon BSC-IT

2December 21, 2011

Project Goals• Replace PPC firewalls with unidirectional device• Implement at all 10 sites (17 units) in 15 months• Ensure commonality across the fleet

Challenges• Guidance (NEI-08-09) still being finalized• Limited dedicated resources• Aggressive schedule (10 sites in 15 months)• Maintaining remote access to plant data

Introduction / Overview

3December 21, 2011

Installation Timeline – Early VersionClinton

Peach Bottom

Limerick

Oyster Creek

Braidwood

Byron

TMI

Dresden

Quad Cities

LaSalle

2/1/2011 3/1/2011 4/1/2011 5/1/2011 6/1/2011 7/1/2011 8/1/2011 9/1/2011 10/1/2011

Legend:

WEST – Single Site

EAST – Single Site

WEST – Dual Unit Site

EAST – Dual Unit Site

4December 21, 2011

Actual Installation TimelineClinton

Peach Bottom

Limerick

Oyster Creek

Braidwood

Byron

TMI

Dresden

Quad Cities

LaSalle

2/1/2011 3/1/2011 4/1/2011 5/1/2011 6/1/2011 7/1/2011 8/1/2011 9/1/2011 10/1/2011

Note 1

Note 1: First Oyster Creek installation failed due to a PVCS anomaly. Second installation in September was successful.

Legend:

WEST Team – Single Site

EAST Team – Single Site

WEST Team – Dual Unit Site

EAST Team – Dual Unit Site

5December 21, 2011

Design Concepts Common Design (Design Once, Install Many)

• Two diode servers Data Acquisition (PPC) Network (L3) Corporate LAN / WAN network (L2)

• Network Switch (L3)• Backup / Anti-Virus Application Server (L3)

Disseminate anti-virus definition files Perform backups of plant network servers (where possible)

• Data Diode Pairs PPC / CMS / Historian / etc.

• Time Servers (where needed)

6December 21, 2011

Design Concepts (cont.)

To PPC / CMS Network

Diode Server

Diode Server

TSM / Anti-Virus Server

Time Server

Time Server

Diode Diode Diode Diode

To Corporate LAN / WAN

Level 3

Level 2

7December 21, 2011

Design Concepts (cont.) Common Design (Design Once, Install Many) (cont.)

• Advantages Support made easier due to common design / equipment Spare parts could be shared across sites if needed Share resources for plant installations Multiple sites were able to be designed concurrently

o Important due to schedule tightness

8December 21, 2011

Design Concepts (cont.) Site-Specific Design

• Each site had unique requirements River Screen House @ LaSalle SCADA @ Limerick Met Tower @ Oyster Creek

• Had to account for older systems VAX / VMS (PBAPS, Limerick, CMS) Windows NT 4.0 / 2000 (TMI / OC)

Common design and recognition of site specifics allowed for timely creation of design documentation!

9December 21, 2011

Design Concepts (cont.) Computer Laboratories

• Built in East (Kennett Square) and West (Cantera) Provide platform for “proof-of-concept” Early identification and remediation of issues with common

design

Equipment Procurement• Single purchase of servers / switches / diodes

Equipment “warehoused” until needed by each site Reduced overall procurement time Ensure standardization of critical components of common design

10December 21, 2011

Test Philosophy / Program Two Parts

• Factory Acceptance Tests (FAT) Performed on lab equipment at Cantera and Kennett Square Comprehensive test of functionality (to the extent possible) Core set of tests / Individual FAT tests (more later) Core FAT tests executed once for entire project

• Site Acceptance Tests (SAT) Performed at the site during installation Verify operability / configuration of each site’s equipment Common and site-specific tests

11December 21, 2011

Test Philosophy / Program (cont.) Core Factory Acceptance Tests (FAT)

• Diode Servers, Data Diodes, Network Switch Includes confirmation / verification of configuration

• Historians Includes eDNA and PI

• Time Servers

• Robustness Tests Power cycle equipment Loss of connectivity (to the extent possible)

12December 21, 2011

Test Philosophy / Program (cont.) Other Factory Acceptance Tests (FAT)

• Core Monitoring Systems Includes 3-D Monicore, WCMS, Beacon, and PowerPlex

• Plant Process Computer Unique FAT for each site / installation

13December 21, 2011

Test Philosophy / Program (cont.) Core Site Acceptance Tests (SAT)

• Data Diodes• Anti-Virus• Network & Time Synchronization• Backups• Safe State• Plant Process Computer• Core Monitoring System

Other Site Acceptance Tests (SAT)• Dependent on site-specific equipment

River Screen House, Met Tower, TSC, state agencies, etc.

14December 21, 2011

Site Installation Process Preparation started 12 weeks before scheduled

installation date:• Week T-12

Deliver hardware to site Complete IT design input to Engineering

• Week T-11 SAT procedures and installation instructions / sequence

submitted to site for review• Week T-10

Hardware built, configured, and installed in racks• Week T-9

Begin site work order planning and scheduling

15December 21, 2011

Site Installation Process (cont.)Level 2 – Level 3 Data Diode Installation

Work Order Activities Roadmap / Overview (Page 1 of 4)

1. Load Windows Server 2008 operating system as per the Server Build instructions and configuration in the appropriate NRT section for the following servers:

Level 3 Data Diode Server ***Level 3 TSM Backup / AV Server

2. Configure the Level 3 Data Diode Switch with the site configuration as per the appropriate NRT section 3. Install the TSM and SEP software on the Level 3 TSM Backup / AV Server as per the installation instructions and configuration noted in the appropriate NRT section4. Install the TSM and SEP software on the Level 3 Data Diode Server as per the installation instructions and configuration noted in the appropriate NRT section.

Activity 01 – Level 3 IT Server Builds (Will need a PCS)

Staging / Preparation(IT Server Build / Pre-Stage New L2 Systems)

1. Install rails as needed in cabinets in the designated Level 2 Site IT Computer Room2. Install Level 2 eDNA Server and connect to Level 2 Switch but DO NOT power on3. Install Level 2 Core Monitoring Replicant Server and connect to Level 2 Switch but DO NOT power on

Activity 03 – Rack and Stack Level 2 Equipment (Will need a PCS)

*** This build loads the Waterfall Management software but does NOT configure the data diodes at this time

Week T-10 to T-4

Week T-9 to T-3

A

Work Order #1(Cabinet / Systems Prep / Initial Test)

T-0 or earlier (pre-T-0)

1. Install Level 3 Data Diode Switch in cabinet – connect to power but power off 2. Install new KVM Device in cabinet – connect to power but power off3. Install Level 3 Data Diode Server in cabinet – connect to power but power off4. Install Data Diode Pairs in cabinet – connect to power but power off5. Install Level 3 TSM / AV Server in cabinet – connect to power but power off6. Install Level 2 Data Diode Server in cabinet – connect to power but power off

Activity 02 – Install Data Diode Hardware in Cabinet (Requires a plant WO)

T-0 or earlier (pre-T-0)

1. Connect Level 3 Data Diode Server to Level 3 Data Diode Switch and KVM Device2. Connect Level 3 TSM / AV Server to Level 3 Data Diode Switch and KVM Device3. Connect Data Diode Pairs to Level 3 Data Diode Server

Activity 03 – Inter-Cabinet Wiring (Requires a plant WO)

1. Power on Level 3 Data Diode Switch2. Power on Level 3 Data Diode Server3. Power on Level 3 TSM / AV Server4. Verify (SAT Test) TSM Client for Level 3 Data Diode Server5. Verify (SAT Test) TSM Client for Level 3 TSM / AV Server6. Verify (SAT Test) AV Client for Level 3 Data Diode Server7. Verify (SAT Test) AV Client for TSM / AV Server

Activity 04 – Initial Power Up / Check-Out (Requires a plant WO)

T-0 or earlier (pre-T-0)

T-0 or earlier (pre-T-0)

1. Load Windows Server 2008 operating system as per the Server Build instructions and configuration in the appropriate NRT section for the following servers:

Level 2 Data Diode Server Level 2 eDNA Server

2. Build the Level 2 Core Monitoring System Replicant Server (if applicable) with the current core monitoring software in accordance with appropriate NRT section3. Install the TSM and SEP software on the Level 2 eDNA Server as per the installation instructions

Activity 02 – Level 2 IT Server Builds (Will need a PCS)

Week T-10 tp T-4

T-0 or earlier (pre-T-0)

1. Remove equipment as needed from existing cabinet to make space for new equipment

Activity 01A – Remove Equipment (Requires a plant WO)

1. Install new cabinet for Data Diode equipment

Activity 01B – Install New Cabinet (Requires a plant WO)OR

NOTEActivity 01A is applicable to sites where an existing cabinet may is used for the new equipment. Activity 01B is applicable to sites that are installing a new cabinet for the new equipment.

16December 21, 2011

Site Installation Process (cont.)Level 2 – Level 3 Data Diode Installation

Work Order Activities Roadmap / Overview (Page 2 of 4)

1. Make required PPC (PMS) / PSS database modifications as per appropriate NRT section2. Make required PPC (PMS) / PSS display modifications as per appropriate NRT section3. Shutdown application software on PPC (PMS) Standby and PSS servers4. Build and install modified software on Standby PPC (PMS) and PSS servers 5. Remove all obsolete software applications from Standby PPC (PMS) and PSS servers6. If feasible, configure SYSLOG for propagation through diode for PPC (PMS) Standby server7. If feasible, configure TSM Client for Level 3 Backup of PPC (PMS) Standby Server8. Restart PPC (PMS) Standby Server – perform a failover to PPC (PMS) Standby server9. Shut down PPC (PMS) on new Standby server and install modified software on Standby Server10. Remove all obsolete software applications from Standby PPC (PMS) Server11. If feasible, configure SYSLOG for propagation through diode for PPC (PMS) Standby server12. If feasible, configure TSM Client for Level 3 Backup of PPC (PMS) Standby Server13. Restart PPC (PMS) on Standby server

NOTE: PPC (PMS) vendor and NRT support may be required for this activity

Activity 04 – Install / Update PPC & PSS Software (Requires a plant WO)

Week T+1

1. If PSS is being relocated, power off PSS and all associated PSS equipment2. If PSS is being relocated, remove PSS from current location and relocate to new Level 2 location3 If PSS is being relocated. power on PSS and associated PSS equipment4. Configure IP addresses for PSS and associated equipment to Level 2 address range5. Connect PSS to Level 2 network at designated location6. Configure Level 2 PSS Server to Level 2 Time Source 7. If feasible, configure SYSLOG on PSS for propagation to central monitoring8. If feasible, configure TSM Client for Level 2 Backup of PSS Server9. If necessary, restart application software on PSS

Activity 05 – Relocate PSS / PSS-RAID (Requires a plant WO)

Week T+1

1. If needed, update appropriate PPC network switch configuration for diode operation2. Connect / move PPC (PMS) servers / PPC (PMS) network to Level 3 Data Diode Switch3. Verify (SAT Test) PPC to PSS Functionality / Data Transmission From Level 3 to Level 2

NOTE: Waterfall and PPC (PMS) vendor support may be required for this activity

Activity 06 – Cutover PPC / Initial Check (Requires a plant WO)

Week T+1

B

1. Configure Remote View Client on Level 3 eDNA Server per NRT Section 4.1.3

A

1. Configure eDNA Transmit and Receive Data Diode per appropriate NRT section (if present)2. Configure CMS Transmit and Receive Data Diode per appropriate NRT section 3. Configure PPC/PMS Transmit and Receive Data Diode per appropriate NRT section 4. Configure Remote View Transmit and Receive Data Diode per appropriate NRT section 5. Configure PI Transmit and Receive Data Diode per appropriate NRT section (if present)

NOTE: Waterfall vendor support required for this activity

Activity 01 – Configure Data Diodes (Requires a plant WO)

Work Order #2(Data Diode Configuration / Systems Cutover)

Week T+1

1. Stop Level 3 eDNA Services – Make Service Updates for Diode Operation2. Configure L3 eDNA Server to L3 Time Source3, Remove Level 2 DNS entries for Level 3 eDNA server4. Restart Level 3 eDNA Services5. Power on Level 2 eDNA Server 6. Configure Level 2 eDNA Server to Level 2 Time Source 7. Connect / move Level 3 eDNA Server connection to Level 3 Data Diode Switch8. Verify (SAT Test) eDNA Data Transmission from Level 3 to Level 2

NOTE: Waterfall vendor support may be required for this activity

Activity 02A – Cutover eDNA / Initial Check (Requires a plant WO)

Week T+1

1. Power on Level 2 Core Monitoring System Replicant Server2. Configure Core Monitoring System Replicant Server to Level 2 Time Source2. Shut down Level 3 Core Monitoring System Servers3. Re-configure IP address as needed for first Core Monitoring System Server 4. Configure first Core Monitoring System Server to Time Synch with Level 3 Time Source5. Install first Core Monitoring System Server connection to Data Diode Switch 6. Restart first Core Monitoring System Server A7. Verify (SAT Test) first Core Monitoring System Server operation from Level 3 to Level 2 Replicant8. Re-configure IP address as needed for second Core Monitoring System Server 9. Configure second Core Monitoring System Server to Time Synch with Level 3 Time Source10. Install second Core Monitoring System Server connection to Data Diode Switch 11. Restart second Core Monitoring System Server – Designate as Primary System12. Verify (SAT Test) second Core Monitoring System Server operation from Level 3 to Level 2 Replicant13. Install / configure CMS interface software as needed on appropriate workstations 14. Verify (SAT Test) CMS interface software as need on appropriate workstations

NOTE: Waterfall vendor support and NRT support may be required for this activity

Activity 03 – Cutover CMS / Initial Check (Requires a plant WO)

Week T+1 NOTE:Need to be concerned with how long Core Monitoring System will be out of service

NOTE:Activity 2A is performed only if eDNA is configured for data transmission between Level 3 and Level 2 at the site.

1. Stop Level 3 PI services – make any required service updates for diode operation2. Configure L3 PI Server to L3 Time Source3, Remove Level 2 DNS entries for Level 3 PI server as appropriate4. Restart Level 3 PI services5. Power on Level 2 PI Server 6. Configure Level 2 PI Server to Level 2 Time Source 7. Connect / move Level 3 PI Server connection to Level 3 Data Diode Switch8. Verify (SAT Test) PI Data Transmission from Level 3 to Level 2

NOTE: Waterfall vendor support may be required for this activity

Activity 02B – Cutover PI / Initial Check (Requires a plant WO)

Week T+1 NOTE:Activity 2B is performed only if PI is configured for data transmission between Level 3 and Level 2 at the site.

17December 21, 2011

Site Installation Process (cont.)Level 2 – Level 3 Data Diode Installation

Work Order Activities Roadmap / Overview (Page 3 of 4)

B

1. Install / activate the Remote View Client on the Level 3 eDNA Server2. Install / activate the Remote View Client on the Level 3 TSM Backup / AV Server3, Install / activate the Remote View Client on the Level 3 Data Diode Server4. Install / activate the Remote View Client on site designated Level 3 servers5. Install / activate the Remote View Client on site designated Level 3 workstations6. Verify (SAT Test) Remote View capability for the Level 3 eDNA Server7. Verify (SAT Test) Remote View capability for the Level 3 TSM Backup / AV Server8. Verify (SAT Test) Remote View capability for the Level 3 Data Diode Server9. Verify (SAT Test) Remote View capability for site designated Level 3 servers10. Verify (SAT Test) Remote View capability for site designated Level 3 workstations

NOTE: Waterfall vendor support may be required for this activity

Activity 07 – Configure Remote View / Initial Check (Requires a plant WO)

Week T+1

1. Connect / move any other required site systems / workstations to Level 3 Data Diode switch2. Verify (SAT Test) connection of connected site systems

Activity 08 – Cutover Other Systems / Initial Check (Requires a plant WO)

Week T+1

1. Install / activate the TSM Backup Client on the Level 3 eDNA Server (if present)2. Install / activate the TSM Backup Client on the Level 3 PI Server (if present)3. Install / activate the TSM Backup Client on the Level 3 site designated Level 3 servers4. Install / activate the TSM Backup Client on the Level 3 site designated Level 3 workstations5. Verify (SAT Test) Level 3 eDNA Server TSM Backup (if present)6. Verify (SAT Test) Level 3 PI Server TSM Backup (if present)7. Verify (SAT Test) Level 3 TSM Backup of site designated servers8. Verify (SAT Test) Level 3 TSM Backup of site designated workstations

Activity 01 – Install TSM Clients / Initial Check (Requires a plant WO)

Work Order #3(TSM / AV Client Workstation Configuration)

Week T+2

1. Install / activate the SEP-11 AV Client on the Level 3 eDNA Server (if present)2. Install / activate the SEP-11 AV Client on the Level 3 PI Server (if present)3. Install / activate the SEP-11 AV Client on site designated servers4. Install / activate the SEP-11 AV Client onsite designated workstations5. Verify (SAT Test) Level 3 SEP-11 AV Server updates to Level 3 eDNA Server (if present)6. Verify (SAT Test) Level 3 SEP-11 AV Server updates to Level 3 PI Server (if present)7. Verify (SAT Test) Level 3 SEP-11 AV Server updates to site designated Level 3 servers8. Verify (SAT Test) Level 3 SEP-11 AV Server updates to site designated Level 3 workstations

Activity 02 – Install SEP-11 Clients / Initial Check (Requires a plant WO)

Week T+2

NOTE:Up to a total of 10 Windows-based workstations can be designated for remote view.

NOTE:Activity 08 is performed only if additional systems are required to be connected to the Level 3 Data Diode switch due to unique plant circumstances.

18December 21, 2011

Site Installation Process (cont.)Level 2 – Level 3 Data Diode Installation

Work Order Activities Roadmap / Overview (Page 4 of 4)

1. Configure existing Level 2 systems to interface with Level 2 eDNA Server (as needed)2. Configure existing Level 2 systems to interface with Level 2 PI Server (as needed)3. Configure existing Level 2 PI server(s) to interface with Level 2 eDNA Server (as needed)4. Configure existing Level 2 PI server(s) to interface with Level 2 PI Server (as needed)5. Verify (SAT Test) Level 2 system(s) interface with Level 2 eDNA Server6. Verify (SAT Test) Level 2 system(s) interface with Level 2 PI Server7. Verify (SAT Test) existing Level 2 PI server(s) to Level 2 eDNA Server8. Verify (SAT Test) existing Level 2 PI server(s) to Level 2 PI Server

Activity 01 – Configure Level 2 eDNA Interfaces (Will need a PCS)

Work Order #4(Miscellaneous Configurations / Final Test)

Week T+2

1. Update LAN users to interface with relocated PSS Server (as needed)2. Verify (SAT Test) Level 2 LAN Users interface to relocated Level 2 PSS Server

Activity 02 – Configure PSS User Interface (Will need a PCS)

Week T+2

1. Update Level 2 DNS Server with Level 2 Core Monitoring System Replicant Server IP address2. Update Level 2 DNS Server with Level 2 PSS Server IP address (as needed)3. Update Level 2 DNS Server with Level 2 eDNA Server IP address (if present)4. Update Level 2 DNS Server with Level 2 PI Server IP address (if present)

Activity 03 – Level 2 DNS Updates (Will need a PCS)

Week T+2

1. Update Spectrum Monitoring to remove site firewall definition2. Update Firewall Manager to remove site firewall definition

Activity 04 – Firewall Configuration Updates (Will need a PCS)

Week T+2

1. Power down site firewall 2. Complete Final SAT Testing

Activity 06 – Final Site Acceptance Test (Requires a plant WO & PCS)

Week T+2

1. Remove all obsolete / abandoned / unneeded cables as a result of Data Diode implementation

Activity 01 – Remove Obsolete / Abandoned Cables (Requires a plant WO)

Work Order #5(Equipment Removal)

Week T+3

1. Remove any abandoned / obsolete equipment associated with relocation of PSS to Level 2

Activity 02 – Remove PPC / PSS Peripherals (Requires a plant WO)

Week T+3

1. Remove any abandoned / obsolete KVM equipment associated with Data Diode implementation

Activity 03 – Remove KVM Equipment (Requires a plant WO)

Week T+3

1. Remove any abandoned / obsolete Core Monitoring System equipment due to Data Diode install

Activity 04 – Remove Abandoned CMS Equipment (Requires a plant WO)

Week T+3

1. Remove any abandoned network equipment (for example, switches no longer needed)2. Remove any abandoned / obsolete Level 3 server equipment 3. Remove any abandoned / obsolete Level 3 workstation equipment

Activity 05 – Remove Miscellaneous Equipment (Requires a plant WO)

Week T+3

1. Remove site firewall

Activity 06 – Remove Firewall (Requires a plant WO)

Week T+3

1. Move / re-connect cables in support of data diode implementation (as needed)

Activity 05 – Miscellaneous Cable Moves (May require a plant WO)

Week T+2

NOTE:Activity 02 is performed only if equipment related to the PPC / PSS is no longer needed as a result of the data diode implementation.

NOTE:Activity 03 is performed only if old KVM equipment is replaced and no longer needed as a result of the data diode implementation.

NOTE:Activity 04 is performed only if old CMS equipment is no longer needed as a result of the data diode implementation.

NOTE:Activity 05 is performed only if Level 3 equipment is no longer needed as a result of the data diode implementation.

NOTE:Activity 01 is performed only if the designated cables are no longer needed as a result of the data diode implementation.

19December 21, 2011

Site Installation Process (cont.) Preparation started 12 weeks before scheduled

installation date (cont.):• Week T-8

Updates made to lab as needed for additional testing Begin revising FATs / SATs as needed

• Week T-7 Complete site work order planning

• Week T-6 Build replicant servers as needed

• Week T-5 SATs and installation instructions completed Verify work orders ready and properly sequenced

20December 21, 2011

Site Installation Process (cont.) Preparation started 12 weeks before scheduled

installation date (cont.):• Week T-4

Perform walk downs as needed Begin site communication with shift crews FAT tests completed

• Week T-3 Arrange visitor badging Arrange site communications

• Week T-2 All server / network pre-installation complete

• Week T-1 OPEN – address any remaining preparation / open items

21December 21, 2011

Site Installation Process (cont.) Installation Window – 4 weeks allotted

• Week T+0 – Preparation Week Stage hardware / software without any physical connectivity to

production environment• Week T+1 – Cutover Week

Diode cutover week Usually, 1 day required for CMS, PPC, and Historian cutovers

• Week T+2 – Test Week Execute SAT tests

• Week T+3 – Closeout Week Resolve any open issues Complete SAT testing Close out plant paperwork as needed

22December 21, 2011

Project Team Composition Project Start

• Minimal dedicated resources (1 permanent, 2 part-time)

Core Team Formed July 2010• Server Builds / Configuration (1 Person)• Tivoli Storage Manager / Anti-Virus (1 Person)• Network Switch Configuration (1 Person)• Core Monitoring Systems / Replication (2 Persons)• Plant Process Computer (2 Persons)• Historian (eDNA / PI) (2 Persons)• IT Designers (4 Persons)

23December 21, 2011

Project Team Composition (cont.) Installation Teams

• East Team Site IT from Limerick, Oyster Creek, Peach Bottom, and TMI Core team designers, PPC, historian, and applicable CMS

• West Team Site IT from Braidwood, Byron, Clinton, Dresden, LaSalle, and

Quad Cities Core team designers, PPC, historian, and applicable CMS

24December 21, 2011

Project Team Composition (cont.) Core Team Shared Responsibilities

• Data Diode Configuration (Designers / Server Build)• Installation Instructions (All core team members)• Write Test Procedures (All core team members)• Execute Factory Acceptance Tests (FAT)

East / West Team Shared Responsibilities• Identification / Confirmation of Data Sources• Site-Specific Design Review• Interface with Site Planning and Scheduling Organizations• Production Installation of Equipment / Software• Execute Site Acceptance Tests (SAT)

25December 21, 2011

Project Team Composition (cont.)Figure 2

Data Diode Project Team Composition

3

9 10 1114 14 16 16

3740 42

35 3734

2722

05

101520253035404550

Jun-

10

Jul-1

0

Aug

-10

Sep

-10

Oct

-10

Nov

-10

Dec

-10

Jan-

11

Feb-

11

Mar

-11

Apr

-11

May

-11

Jun-

11

Jul-1

1

Aug

-11

Sep

-11

# Pe

rson

nel

Core Team East Team West Team Total

26December 21, 2011

Meetings Core Team Members

• Twice per week throughout project

Site-Specific Meetings• Started six weeks before site installation• Twice per week until one week before start of site installation

window

27December 21, 2011

Meetings (cont.) Site Installation Meetings

• Daily starting at one week before site installation window Involved all East / West team members involved in installation

Installation Status Meetings• Daily for installation window

Involved only a single representative

28December 21, 2011

Detailed Project TimelineSee White Paper for month-by-month account of:

Activities / Accomplishments

Issues / Concerns

Installations

29December 21, 2011

Lessons Learned / Status Key Lessons Learned

• Pre-staging of equipment allowed for early detection of issues and allowed for timely resolution prior to placing equipment into the production environment.

• Initial port configuration at each site was streamlined by completing a “Data Port Spreadsheet” detailing what type and details for each connector.

• The use of parallel runs of new servers (where possible) with existing servers saved cutover time and allowed for the data to “settle down” prior to the actual cutover.

• Have adequate support personnel available to address non-project issues in order to limit distractions of the installation team.

30December 21, 2011

Lessons Learned / Status Status / Follow-Up Items

• Conduct data diode training for site / support personnel (scheduled for completion January 2012)

• Update earlier installations with improvements made / discovered at later installations

• Address minor software issues discovered after implementation in the production environment

31December 21, 2011

Summary / Conclusions Implementation of data diodes was a fast-track

modification with an aggressive implementation schedule

Through careful coordination of project resources and schedule, the objective was met

Each successive installation was performed more efficiently by applying lessons learned and leveraging the gained and shared experiences of the project team members.

Strong, committed oversight and support by management was instrumental in the success of the project.

32December 21, 2011

Comments / Questions