control center virtualization for the cisco connected ... · pdf filecisco connected pipeline...
TRANSCRIPT
Control Center Virtualization for the Cisco Connected PipelineDesign GuideApril 2016
Connected Refineries 1.0 Cisco Reference Documentii
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY,
"DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS,WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM
ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR
A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR
TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL,
CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR
DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUP-
PLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR
APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFES-
SIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL
ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT
TESTED BY CISCO.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California,
Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981,
Regents of the University of California.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To
view a list of Cisco trademarks, go to this URL: http://www.cisco.com/go/trademarks. Third-party trademarks mentioned are the
property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any
other company. (1110R).
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone
numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are
shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and
coincidental.
Control Center Virtualization for the Cisco Connected Pipeline Design Guide
© 2016 Cisco Systems, Inc. All rights reserved.
Design Guide
C O N T E N T S
Document Objective and Scope vii
Contributors vii
Connected Pipeline Overview 1-1
Executive Summary 1-1
The Oil and Gas Value Chain 1-2
Pipeline Management Systems 1-5
SCADA System Design Principles 1-5
Availability 1-9
Security 1-9
Management 1-11
Baseline Integrated SCADA System 1-12
Connected Pipeline Architecture 2-1
Control Center Overview 2-2
Schneider Electric Pipeline Management Solutions 2-2
Production Environment 2-3
Test and Development 2-4
Test System 2-4
Development System 2-4
Decision Support System/Industrial Demilitarized Zone 2-4
Control Center Availability and Security 2-5
Availability 2-5
Security 2-5
Multi-Service Traffic (Non-Operational Applications) 2-6
Operational Communications Network Overview 2-7
Use Case: SCADA Real-time /Near Real-time Operations 2-10
SCADA to Pipeline 2-11
Control Center-to-Control Center Communications 2-11
Connected Pipeline System Design 3-1
BLISS Design 3-1
Functional Data Center Design 3-2
Network 3-2
iiiControl Center Virtualization for the Connected Pipeline
Contents
Compute 3-3
Storage 3-3
Security 3-3
Security Design 3-3
Segmentation Design 3-3
Control Center Segmentation and Zones 3-3
Compute Segmentation 3-6
Virtual Switch (vSwitch) 3-6
Storage Segmentation 3-7
File System Management 3-7
Fiber Channel Zoning and VSANs 3-7
LUN Masking 3-8
Summary 3-8
Firewall and Industrial DMZ Design 3-8
Overview 3-9
Firewall Deployment Options 3-9
Industrial DMZ Concepts 3-10
IDMZ and DSS Design 3-11
Remote Access Overview 3-12
Inter-Control Center Firewall Considerations 3-14
Pipeline Communication Firewall Considerations 3-14
Intra-Control Center Firewall Considerations 3-15
Summary 3-15
Network Infrastructure Security 3-15
Shutdown Unused Ports 3-15
Trunk Ports 3-15
DHCP Snooping 3-16
Dynamic ARP Inspection (DAI) 3-16
Port Security 3-16
Traffic Storm Control 3-17
Infrastructure Management 3-17
Control Plane Policing (CoPP) 3-17
Availability 3-17
Network Redundancy 3-18
Port Channel 3-19
Virtual Port Channel 3-19
ASA Redundancy 3-20
Summary 3-20
Compute Redundancy 3-20
Fabric Redundancy 3-20
ivControl Center Virtualization for the Connected Pipeline
Design Guide
Contents
End Host Mode 3-21
SCADA Server Connectivity and Redundancy (Virtual Machines) 3-21
Storage Redundancy 3-22
Hardware Redundancy 3-22
Redundant SAN 3-23
Multipath 3-23
Summary 3-23
Schneider Electric OASyS Application Design Application Redundancy 3-24
Virtual Machine Redundancy 3-24
SQL SAN Storage Hardware Mirroring 3-24
Virtual Guest Storage 3-24
Microsoft SQL Server-Based Historian 3-24
Operator Workstation Connectivity 3-25
Network Management 3-25
Cisco UCS Manager 3-25
Cisco Adaptive Security Device Manager 3-26
Out of Band Management 3-26
Role-Based Access Control 3-27
Summary 3-27
Time Synchronization 3-27
System Implementation Considerations 4-1
Server Sizing and Storage Sizing 4-1
Scalability Considerations 4-1
System Components 5-1
A P P E N D I X A Glossary A-1
vControl Center Virtualization for the Connected Pipeline
Design Guide
Preface
This Cisco Control Center Virtualization for the Connected Pipeline System Cisco Validated Design (CVD) documents best practice design and implementation of safe, highly available, and secure oil and gas pipeline infrastructure and applications. This CVD identifies customer use cases, maps those use cases to relevant architectures, and describes how Cisco and partner technology are leveraged to deliver unprecedented value for our customers. This CVD:
• Describes a Low Level Design (LLD) detailing Control Center Virtualization for the Connected Pipeline System. It will provide guidance supporting Supervisory Control and Data Acquisition (SCADA) principles, unified networking and powerful compute for the Control Center.
• Documents best practices from real world implementations, detailing the designs and architectures that are mapped back to the customer use cases.
• Addresses real-life customer deployment scenarios by providing a solution that supports implementation of a scalable, secure, and redundant operational network supporting both industrial and multi-service applications.
• Details support for implementing Control Center application virtualization, secure remote access, the Industrial Demilitarized Zone (IDMZ), and cyber-security.
• Specifies topology, Quality of Service (QoS), high availability, security services, network management services, and Control Center virtualization implementations.
• Provides information about enforcing cyber-security best practices that follow the recognized Industrial Control System (ICS) security standards and guidelines including International Society of Automation 99(ISA99)/International Electrotechnical Commission (IEC) 62443, the National Institute of Standards and Technology (NIST) Cyber Security Framework, and the Purdue Model of Control.
• Documents suggested equipment and technologies, system level configurations, and recommendations. It also describes caveats and considerations that pipeline operators should understand as they implement best practices.
Although this CVD focuses on midstream transport pipelines, the technologies, use cases, and principles are applicable for gathering and distribution pipelines.
viControl Center Virtualization for the Connected Pipeline
Design Guide
REVIEW DRAFT—CISCO CONF IDENT IAL
Document Objective and ScopeIn this initial release, Cisco has partnered with Schneider Electric to provide architecture, design, and technologies for the Control Centers, Operational telecoms network, and the pipeline stations. Cisco provides infrastructure expertise with its unified compute and networking security platforms while Schneider Electric provides the PMS (PMS) leadership with its OASyS Dynamic Network of Applications (DNA) SCADA system hardware and software.
The release will focus on the Control Center environment and security architecture to support pipeline operators. It is recommended that the reader become familiar with the following joint Cisco/Schneider Electric white papers:
• Integrated Enterprise SCADA System Architectures for Safe and Efficient Pipeline Operations at the following URL:
– http://www.cisco.com/c/dam/en/us/solutions/collateral/industry-solutions/dlfe-683318406.pdf
• Converged Telecommunication Architectures for Effective Integrated Pipeline Operations at the following URL:
– http://www.cisco.com/c/dam/en/us/solutions/collateral/industry-solutions/dlfe-683318407.pdf
As with any architecture and design program, functional requirements, use cases, and architectures evolve. Therefore, this CVD will evolve and will be updated in future phases.
ContributorsJason Greengrass, Solutions Architect, IVSG, Cisco Systems
Rik Irons-McLean, Lead Architect Oil and Gas, IoE Vertical Solutions Group, Cisco Systems
Brian Malkinson, Global Solution Architect, Pipeline Management Systems, Schneider Electric
viiControl Center Virtualization for the Connected Pipeline
Design Guide
ContDesign Guide
C H A P T E R 1
Connected Pipeline OverviewThis chapter includes the following major topics:
• Executive Summary, page 1-1
• The Oil and Gas Value Chain, page 1-2
• Pipeline Management Systems, page 1-5
• SCADA System Design Principles, page 1-5
Executive SummaryThis chapter provides a high level overview of the end-to-end Oil and Gas value chain and where pipeline solutions fit into this chain. It also provides an overview of the emergence of virtualization technologies into these environments. This document is written for an industry with a number of key trends:
• Health and Safety—The health and safety of employees continues to be of major importance for organizations. The industry looks to improve overall worker safety while specifically providing a safe working environment for remote or unaccompanied workers.
• Environmental Safety and Compliance—Solutions must meet or exceed industry standards or regulations such as the Pipeline and Hazardous Materials Safety Administration (PHMSA), with increased attention to safety and compliance in regulations a major design factor for telemetry and SCADA systems today.
• An Aging Workforce—Worker age and skill sets have changed. As younger workers with more of an IT-based skill set join the workforce, being able to train and provide remote expertise and consultation to new workers is essential.
• Predictive Automation and Process—Through Big Data, fog or edge compute, and analytics and cloud-based services, sensors are able to provide real-time information on such measures as temperature, vibration, pressure, flow, and current. Combining this with statistical models provides predictive methods for maintenance of equipment and streamlining of processes. The Internet of Things (IoT) has focused on connecting the unconnected through wireless and wired networks, and previously inaccessible data is now available for use.
• Security—As technology evolves, more devices are connected to the network, attackers use increasingly sophisticated methods, and OT and IT technologies begin to converge, protecting assets, people, and intellectual property from cyber and physical threats becomes ever more important.
1-1rol Center Virtualization for the Connected Pipeline
Chapter 1 Connected Pipeline Overview The Oil and Gas Value Chain
It is essential to understand that a single technology cannot enable the industry to meet these requirements. Only a properly architected, secure integration of a number of technologies and applications will keep workers safe, improve efficiencies, reduce cost, and continue to drive innovation.
The Oil and Gas Value ChainAt a high level, the Oil and Gas value chain starts with discovering resources through exploration, and then the development, production, processing, transportation/storage, refining, and marketing/retail of hydrocarbons. This value chain is normally grouped into the upstream, midstream, and downstream areas, as shown in Figure 1-1.
Figure 1-1 Oil and Gas Value Chain
• Upstream—Upstream includes the initial exploration, evaluation and appraisal, development, and production of sites. This is referred to as Exploration and Production (E&P). These activities take place onshore and in the ocean. Upstream focuses on finding wells, determining how best and how deeply to drill, and determining how to construct and operate wells to achieve the best return on investment.
• Midstream—Midstream primarily focuses on the transport and storage of hydrocarbons via transmission pipelines, tankers, tank farms, and terminals, providing links between production and processing facilities, and processing and the end customer. Crude oil is transported downstream to the refinery for processing into the final product.
Midstream also includes the processing of natural gas. Although some of the needed processing occurs as field processing near the source, the complete processing of gas takes place at a processing plant or facility, reaching there typically from the gathering pipeline network. For the wholesale markets, natural gas must first be purified by removal of Natural Gas Liquids (NGLs) such as butane, propane, ethane, and pentanes, before being transported via pipeline, or turned into Liquid Natural Gas (LNG) and shipped. The gas can be used real-time or stored. The NGLs will be leveraged downstream for petrochemical or liquid fuels, or turned into final products at the refinery.
• Downstream—Downstream is concerned with the final processing and delivery of product to wholesale, retail, or direct industrial customers. The refinery treats crude oil and NGL and then converts them into consumer and industrial products through separation, conversion, and
Upstream Midstream Downstream Upstream Midstream Downstream
Explore Develop Produce Storage & Transportation Refine Marketing
Marketing Process Transport
Offshore Shipping Trading
Research & Development, Engineering, High Performance Compute
Office Facilities, Call Center, Data Center
Key Oil Gas 3
76
49
8
1-2Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview The Oil and Gas Value Chain
purification. Modern refinery and petrochemical technology can transform crude materials into thousands of useful products including gasoline, kerosene, diesel, lubricants, coke, and asphalt. Downstream also includes gas distribution pipeline networks.
A visual overview of the value chain is shown in Figure 1-2.
Figure 1-2 Oil and Gas System
Transmission pipelines are the key transport mechanism for the oil and gas industry and operate continuously outside of scheduled maintenance windows. Pipelines provide an efficient, safe, and cost-effective way to transport processed or unprocessed oil, gas, and raw materials and products both on- and offshore. It is essential that they operate as safely and efficiently as possible, and, where problems occur, they must be able to rapidly restore normal operation to meet environmental, safety, and quality requirements.
Oil and Gas pipelines (Figure 1-3) comprise operating process, safety, and energy management functions geographically spread along the pipeline for a set of stations. Stations vary in size and function, but typically include large compressor or pump stations, mid-size metering stations, Pipeline Inspection Gauge (PIG) terminal stations, and smaller block valve stations. Each process and application must be linked with the applications and processes at other stations, and at the Control Centers (main and backup) through an operational field telecoms infrastructure. The process must be done in a reliable and efficient way, avoiding communications outages and data losses. The Control Centers should also be securely connected to the enterprise through a Wide Area Network (WAN) to allow users to improve operational processes, streamline business planning, and optimize energy consumption.
1-3Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview The Oil and Gas Value Chain
Figure 1-3 Example Pipeline Station Distribution
Oil and Gas pipeline management is challenging, with pipelines often running over large geographical distances, through harsh environments, and with limited communications and power infrastructure available. In addition, pipelines must comply with stringent environmental regulations and operate as safely as possible, and address growing cyber and physical security threats.
Key pipeline requirements, however, have not changed. Pipeline integrity, safety, security, and reliability are essential elements that help operators meet demanding delivery schedules and optimize operational costs.
At the same time, new operational and multi-service applications are enhancing the way assets and personnel operate. Modern cathodic detection, distributed acoustic leak detection, landslip/earthquake detection, intrusion detection, and physical security applications allow operators to reduce downtime, optimize production, and decrease energy and maintenance costs. Real-time operational data access allows incidents to be identified and addressed quickly, or prevented from occurring in the first place.
Challenges must be addressed through a secure communications strategy to ensure operators can confidently rely on remote data, video, and collaboration solutions for safety and security in addition to operations.
Communications architectures, technologies, solutions, and management for process, energy, security, and multi-service applications (Figure 1-4) must be robust, flexible, and scalable. They should be based on open standards, allowing operations from field device to Control Center, and Control Center to enterprise by combining real-time process and business control automation, information management, energy management, and security with global supervision.
MCC CP BCC M C
P T CP M B B B B B B B B B B B B B
Main/Backup Control Center
Metering/PIG Sta�on
Compressor / Pump Sta�on
Block Valve Sta�on
Pipeline Length
B
B
B
Terminal Sta�on
Component Function
Control Center Monitoring and control of the pipeline system
Compressor station Provides pressure for gas pipelines to keep flow moving
Pump station Provides pressure for oil pipelines to keep flow moving
Metering station Simultaneous, con�nuous analysis of quality and quan�ty being transferred in a pipeline
PIG station Cleaning and inspec�ng the pipeline and flowlines
Terminal station Where product will be delivered to end customer
Block valve station Isolate a segment of the line for leaks or maintenance
MCC BCC
Control Centers may be part of pipeline or in different geographic loca�ons
37
64
99
1-4Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview Pipeline Management Systems
Figure 1-4 High Level Pipeline Architecture
Pipeline Management SystemsReal-time monitoring and control through sharing and collection of data to a centralized PMS is critical for ensuring that the product is transported safely and efficiently. A PMS combines operational SCADA with real-time applications specific to the oil and gas industry, host-based leak detection, and historical flow measurement.
A well-designed PMS uses a hardware and software architecture that allows functions to be mobile, scalable, flexible, and robust. It also permits distribution of processing among different SCADA system components to optimize overall performance of the PMS.
These integrated applications provide pipeline operators:
• Real-time/near real-time control and supervision of operations along the pipeline through a SCADA system based in one or more Control Centers
• Accurate measurement of flow, volume, and levels to ensure correct product accounting
• Ability to detect and locate pipeline leakage, including time, volumes, and location distances
• Integrated security systems for personnel, the environment, and infrastructure using video surveillance, access control, and Intrusion Detection Systems (IDS)
• Ensured safe operations through instrumentation and safety systems
• Energy management system to visualize, manage, and optimize energy consumption within the main stations
SCADA System Design Principles Pipeline requirements will vary depending on customer preference and project, so it is essential that any communications solution be scalable and modular; elements must be able to be interchanged without affecting fundamental architectural and operational functions.
Pipeline
Main Sta�on (xN) (Compressor /
Pump)
Main Sta�on (xN) (Metering / PIG /
Terminal)
Block Valve Sta�on (xN)
Converged IT and OT Opera�onal Field Telecoms
WAN
Ope
ra�o
nal
Appl
ica�
ons
Mul
�ser
vice
Ap
plic
a�on
s
Ope
ra�o
nal
Appl
ica�
ons
Mul
�ser
vice
Ap
plic
a�on
s
Ope
ra�o
nal
Appl
ica�
ons
Mul
�ser
vice
Ap
plic
a�on
s
Corporate Office / Business Domain
Main Control Center Backup Control Center
Ope
ra�o
nal
Appl
ica�
ons
Mul
�ser
vice
Ap
plic
a�on
s
Virt
ualiz
ed
Data
Cen
ter
Ope
ra�o
nal
Appl
ica�
ons
Mul
�ser
vice
Ap
plic
a�on
s
Virt
ualiz
ed
Data
Cen
ter
37
65
00
1-5Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview SCADA System Design Principles
The Control Center consists of operational and non-operational domains (Figure 1-5), which are divided into secure zones based on IEC 62443-3-3, Functional Requirement (FR) 5, Restricted data flow. This segments the control system architecturally via zones and conduits to limit the unnecessary flow of data.
Figure 1-5 Control Center Segmentation Domains
SCADA system environments within an architecture include operational zones for:
• Production—Domain controllers, real-time, historical and leak detection servers, operator workstations
• Test—Non-production replica of the operational SCADA system, allowing software and system changes to be validated prior to production without disrupting production
• Development—Development area for reports, displays and database changes
• Training—Training area built upon the environment that a pipeline controller lives in every day, via a fully functional SCADA control system together with accurate simulation of the pipelines they control
• Decision Support/Industrial DMZ (L3.5):
– Decision support domain controllers and real-time historical and remote access services
– Isolates the operational system from any external systems or users
– Synchronized with real-time and historical data from the Production system to provide a secure method of providing real-time data to external users
• With firewalls creating a DMZ, secure services are enabled to provide the external environments exposure to the data.
• Functions may include secure File Transfer Protocol (FTP) connections and long term historians
And non-operational zones:
• Multi-service—Non-production applications which support the operation of the pipeline such as physical security, voice, Public Address and General Alarm (PAGA) safety announcements, video, and wireless
These building blocks are core components of a SCADA architecture design that can be deployed in one or more Control Centers. Customer philosophy and individual requirements will dictate how these zones are created and into which sites they are deployed. These are typically based on an internal risk assessment to ensure safe and reliable operations. Risk assessment guidelines to help determine philosophy and architectural design are available in the IEC 62443-3-2 standard Security for Industrial Automation and Control Systems - Security Risk Assessment and System Design.
RealTime (Core)ProductionSCADA
Primary
SCADA
Backup
Decision SupportIDMZ
Training
Test
Decision SupportIDMZ
Training
Non-RealTime (Core)
DevelopmentReplica of Production
Location may vary
Mai
n C
ontr
ol
Cen
tre
Bac
kup
Con
trol
C
entr
e
RealTime (Core)ProductionSCADA
Primary
SCADA
Backup
Decision SupportIDMZ
MultiservicePhysical Security Voice Services
Emergency
AnnouncementsWireless
Remote Expert Data Access
Ope
ratio
nal D
omai
ns
Non
-Ope
ratio
nal
Dom
ains
37
65
01
1-6Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview SCADA System Design Principles
To allow appropriate data flows to take place between zones for system operation, a communications and security infrastructure is implemented (see Figure 1-6), which provides conduits to transport only permitted data.
Figure 1-6 Control Center Communication Building Blocks
This network segmentation defined in IEC 62443-3-3, System Requirement (SR) 5.1, Network Segmentation, focuses on separating control system networks from non-control system networks, and also critical control systems from non-critical control systems. Options exist for logical segmentation, physical segmentation, or both. Again, the choice is based on customer philosophy. Guidance is available for the risk assessment to determine the philosophy in IEC 62433-3-2.
In addition to segmentation, boundary control needs to be established between each of these areas. Guidance on boundary control is found in IEC 62443-3-3, SL 5.2.
To ensure the required high availability and security, the SCADA PMS has traditionally been deployed on physically separate stacks (compute, storage, and network) with boundaries at the firewall level (see Figure 1-7). This ensures strict policy enforcement between zones, and leverages logical segmentation inside zones to differentiate between traffic types.
Figure 1-7 SCADA Deployment Physical Separation
37
65
02
WAN to Pipeline
SCADA A VPN
Multiservice VPN
Production
Test
IDMZ• Decision Support
• Remote Access
SCADA B VPN
Multiservice
Non-Operational Domain
Operational Field Telecoms
Training
Development
RealTime (Core)
Non-RealTime (Core)
Communications and Security Infrastructure
FC Switch
L3 SwitchSync
Storage
ASA F/W
L3 WAN Routers
L3 Switch
Dual FC SAN
Dual FC SAN
Dual FC SAN
Enterprise Enterprise
Test A Server
Test B Server
Development
Remote Access/DMZ
Production Server A
Production Server B
DSS Server
Dual FC SAN
Test/DevelopmentDSS/IDMZ
Production
WAN
Layer 2Modular
Training
Positioning undecided. Could fit in either environment
Routers
L2 Switch
UCS C Server
Virtual
IP
10.1.1.1
10.1.1.2 10.1.1.3
VM VM
37
65
03
1-7Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview SCADA System Design Principles
Virtualization has transformed the data center environment over the past decade, allowing consolidation of multiple standalone bare-metal servers and applications onto smaller and more powerful nodes. The primary business push for virtualization was the aim to achieve better server usage from powerful and smaller physical servers, thus leading to an improved Total Cost of Ownership (TCO) and additional operational efficiencies (such as power, cooling, and space).
The deployment of full or partial virtualized solutions has increased in recent years, and industrial security standards and guidelines such as IEC 62443, NERC-CIP, IEC 62351, and the NIST Cybersecurity Framework have an increasing focus on the deployment of virtualization technologies.
This means that the core SCADA Control Center architectural building blocks are implemented, with areas of shared infrastructure as shown in Figure 1-8.
Figure 1-8 SCADA Deployment Leveraging the Consolidated Architecture
The choice of deployment depends on end user requirements and philosophy. However, any deployment must focus on high availability, redundancy, and security. A holistic security strategy should include risk assessment, change management, recovery processes, training, information protection, and incident response—these will determine choice.
For the purposes of this CVD, the consolidated deployment will be used. This will allow the core architectural principles to be leveraged, and provides best practice design guidance and validation, including virtualization. This also ensures that high availability, reliability, and security requirements are met as per the following design guidance sections.
Security concepts and enforcement are drawn directly from IEC 62443 to support high availability and reliability. When any potential impacts may occur (such as scope for misconfiguration), they will be highlighted in the document.
37
65
14
VM
VM
Fabric Ports
Ethernet Ports
FC Links
Virtual
IP
10.1.1.1
10.1.1.3
10.1.1.2
Production A
Test A
DSS/IDMZ B
DSS/IDMZ A
Test B
Storage B
WAN
Production B
Dev B
Dev A
L2 Switch
L3 WAN Routers
Storage A
UCS Chassis A
UCS Chassis B
ASA Firewalls
Dual FC SAN
Dual FC SAN
L2 Switch
Fabric Interconnect
Storage
ASA F/W
Routers
UCS Chassis
UCS Blade Server
Enterprise
Fabric Interconnection
1-8Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview SCADA System Design Principles
The jointly architected and validated approach to pipeline management and telecommunications offers many realizable benefits. Solution integration quality and interoperability are maximized, while design and testing time are minimized. End users have a single point of reference accountable for integration and operational success from hardware, software, security, and management perspectives throughout a project life cycle. The jointly architected design will provide maximum benefit for current operations, and be a platform for future application enablement and integration.
The key elements of this jointly architected and validated design will be discussed in detail in the following sections.
AvailabilityThe system design must encompass a highly available architecture. The pipeline operator must have control of the pipeline 24 hours a day and 365 days of the year. Any loss of visibility or communications will either enforce a shutdown of the process resulting in loss of revenue, or in a worst case scenario, not provide the ability to shut down the pipeline under a catastrophic safety incident such as a major leak.
No Single Point of Failure (SPOF) should occur on any critical system component of the SCADA system design. A critical component is any component whose failure directly and adversely affects the overall performance of the SCADA system or its ability to continue performing the critical SCADA functions of monitoring and control. The SCADA system uses modular components so that the failure of a single component does not render other components inoperative.
Within this design, redundancy is provided for all critical SCADA functions for monitoring and control. Components comprising the standby capability continuously receive updated data, as appropriate, to provide a hot-standby capability in case of a hardware- or software-initiated failover. As an example, a hot server or critical SCADA application will have a standby equivalent within a Control Center and updates will be passed from this server/application to a backup Control Center if deployed.
The SCADA system connects to the telecommunication networks in such a way that a failure of these networks does not affect the ability of the SCADA system to perform its critical functions for monitoring and control. The logic within controllers and the safety systems along the pipeline will still operate if the Control Center loses connectivity to the pipeline stations; however, the ability to control and monitor the pipeline would be lost. Therefore, it is critical that communications to a Control Center are maintained.
SecuritySecurity, safety, and availability are tightly aligned within an industrial security framework. When discussing industrial network security, customers are concerned with how to keep the environment safe and operational.
Historically, industrial control systems were seen as isolated from the outside world and utilized proprietary technologies and communications. Security was seen as more of a security-by-obscurity approach. Security outside of physical security wasn't a primary concern. With the modernization of control systems moving towards Consumer-Off-the-Shelf (COTS) products leveraging standardized protocols and connecting to public networks, the process domain now, more than ever, depends on a security framework and architecture. By using more IT-centric products and technologies, and providing connectivity to the enterprise and outside world, new cyber-attacks from both inside and outside the operational environment can potentially occur.
Security incidents can be categorized as either malicious or accidental:
1-9Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview SCADA System Design Principles
• Malicious acts are deliberate attempts to impact a service or cause malfunction or harm. An example is a disgruntled employee planning to intentionally affect a process by loading a virus onto a server used within the operational control domain or taking control of a process by spoofing a Human Machine Interface (HMI).
• Accidental incidents are probably more prevalent in these environments. Someone may accidentally incorrectly configure a command on a piece of networking equipment, or connect a network cable to an incorrect port. These may be accidental, such as human error, but could be malicious as well while compromising the safety of people, processes, and the environment.
It is recommended to follow an architectural approach to securing the control system and process domain. Recommended models would be the Purdue Model of Control Hierarchy, International Society of Automation 95 (ISA95) and ISA99/IEC 62443. To help adhere to the requirements of IEC 62443 and achieve a robust solution for security and compliance, it is essential to use an end-to-end approach with technologies designed to operate together, while minimizing risk and operational complexities, as shown in Figure 1-9.
Figure 1-9 IEC Foundational Requirements
Figure 1-9 highlights the seven Foundational Requirements (FRs) defined in the ISA-62443 series of documentation:
• Identification, Authentication & Control (IAC) (ISA-62443-3-3 FR 1)—Identify and authenticate all users (humans, software processes and devices) before allowing them to access to the control system.
• Use Control (UC) (ISA-62443-3-3 FR 2)—Enforce the assigned privileges of an authenticated user to perform the requested action on the IACS and monitor the use of these privileges.
• System Integrity (SI) (ISA-62443-3-3 FR 3)—Ensure the integrity of the IACS to prevent unauthorized manipulation.
• Data Confidentiality (DC) (ISA-62443-3-3 FR 4)—Ensure the confidentiality of information on communication channels and in data repositories to prevent unauthorized disclosure.
• Restricted Data Flow (RDF) (ISA-62443-3-3 FR 5)—Segment the control system via zones and conduits to limit the unnecessary flow of the data
• Timely Response to Events (TRE) (ISA-62443-3-3 FR 6)—Respond to security violations by notifying the proper authority, reporting needed evidence of the violation and taking timely corrective action when incidents are discovered.
Defense in depth Detec�on in depth
Data Confiden�ality
Restricted Data Flow
Timely Response to Events
Resource Availability
Iden�fica�on & Authen�ca�on
Control
Use Control System Integrity IEC 62443-3-3 Founda�onal Requirements
ICS
“It’s more than just a bunch of boxes, it’s solu�ons that work together”
Cisco Solu�ons
Network security
Content security
Access security
37
65
04
1-10Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview SCADA System Design Principles
• Resource Availability (RA) (ISA-62443-3-3 FR 7)—Ensure the availability of the control system against the degradation or denial of essential services.
Defense in Depth is a common term that denotes the multiple layers required to incorporate a security framework. Security isn't just about the technologies to prevent incidents, but also incorporates people, processes, training, and continual assessment. In addition to Defense in Depth, it is also essential to Detect in Depth to ensure malicious activity can be discovered by more than one medium.
Security is not a one-off incident and response; it must be treated in a life cycle manner. This involves everything from awareness through to response. Figure 1-10 highlights the security life cycle approach in the Cisco Risk Control Framework. These concepts are built into the proposed architecture.
Figure 1-10 Cisco Risk Control Framework
ManagementManagement and diagnostics involves tools, applications, and devices used to monitor and maintain the Control Center architecture. Although a typical pipeline network does not drastically change after deployment, the network needs to be maintained and managed. Historically, these functions have not been incorporated into the automation and control systems, but this paradigm is changing because infrequent updates are more prevalent in today’s pipeline.
Business practices and levels of expertise may dictate the management practice and roles and responsibilities of the system. Companies are merging the boundaries between Information Technology (IT) and Operational Technology (OT) where these have historically remained separate domains. Staff may have differing levels of knowledge to support the applications and infrastructure of the Control Center environment. As an example, if more IT-aware personnel are maintaining the Control Center environment, they must be aware of the business requirements that dictate the security and availability aspects of the operating environment.
Pipeline architectures have differing availability requirements (24 hour per day continuous operations), Service Level Agreements (SLA), and processes compared to the typical enterprise domain. If personnel managing the system are more operationally focused, then it is important to understand the networking technologies and tools used to administer the system. It is therefore critical that staff training that is in conjunction with validated and documented processes is established so that any misconfiguration incidents are restricted.
The management systems follow the Fault, Configuration, Accounting, Performance and Security (FCAPS) model to help provide and maintain the availability and security of the Control Center architecture. FCAPS management all help to facilitate and monitor the health of the system.
37
65
05
1-11Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 1 Connected Pipeline Overview SCADA System Design Principles
• Fault Management—Detect, isolate, and notify administrators of any issues or faults within the Control Center to help aid with a timely fault resolution.
• Configuration Management—Configure, manage, and maintain the infrastructure and applications for the Control Center. This includes inventory management, software management (upgrades and patch management), configuration management (backups, archiving, and comparisons), and well documented, validated procedures to support change and replacement management.
• Accounting Management—Account management is usually defined for billed networks in order to provide accounting information. Within the context of this system, the A will stand for accounting as it relates to user access and permissions to provide Role-Based Access and Control (RBAC).
• Performance Management—Collect, analyze, and report the application and infrastructure performance of the Control Center and report on any thresholds that may be exceeded.
• Security Management—Manage, monitor, log, and control the security of the infrastructure to identify, defend, and prevent any security threats or breaches.
Although not explicitly defined in each of these areas, logging and audit trails need to be maintained to understand all aspects of FCAPS for the Pipeline Control Center.
Baseline Integrated SCADA System The design guide and validation effort follows Schneider Electric's best practice deployment model for achieving a holistic integrated solution for the Control Center SCADA.
Baseline Integrated SCADA System (BLISS) combines Schneider Electric's Enterprise OASyS SCADA, and its advanced applications with Cisco compute and networking, to provide a fully integrated holistic PMS for the Control Center. BLISS offers a validated solution to provide fast, secure, and efficient deployment either as part of a pipeline Greenfield deployment or as an upgrade to an existing pipeline.
1-12Control Center Virtualization for the Connected Pipeline
Design Guide
ContDesign Guide
C H A P T E R 2
Connected Pipeline ArchitectureThis chapter includes the following major topic:
• Control Center Overview, page 2-2
The Connected Pipeline System delivers a forward-looking, flexible, and modular architecture that enables customers to build the components into an existing system or for a Greenfield deployment. Throughout the architecture, high availability and security are key deliverables. The end-to-end infrastructure provides the following:
• High Availability—Redundancy and reliability mechanisms at the physical, data, and network layer, including robust differentiated QoS and device level redundancy
• Multi-Service Support—Operational and non-operational applications coexisting on a communications network, with mechanisms to ensure the right applications operate in the right way at the right time
• Multi-Level Security—Protect against both physical and cyber-attacks, and non-intentional security threats
• Integrated Management—Network, security, and administration management, from the instrumentation or sensor to the Control Center application
• Open Standards—Based on IP, with the ability to transparently integrate and transport traditional or older serial protocols, and ensure interoperability between current and future applications
The Operational Telecom Network (OTN) and pipeline station architecture are outside the scope of this document, although a high level description of the architecture is included. This document will prioritize the virtualization and operations of the OASyS DNA SCADA software and supporting applications into an operational Control Center architecture based on BLISS.
2-1rol Center Virtualization for the Connected Pipeline
Chapter 2 Connected Pipeline Architecture Control Center Overview
Figure 2-1 Cisco/Schneider Electric Connected Pipeline Reference Architecture
The architecture is based on a three-tier building block approach as defined in the joint Cisco Schneider Electric reference architecture (see Figure 2-1).
• Control Center—Virtualized, geographically separate, redundant Control Centers
• Operational Telecoms Network—End-to-end communication from field device to Control Center application, for operational and multi-service applications
• Pipeline Stations—Local area networks inside the stations for operational and multi-service applications
Control Center OverviewThe SCADA system provides operators who are located in a control room the ability to monitor, operate, and control the operations along the pipeline. The SCADA system consists of multiple environments within the architecture.
Schneider Electric Pipeline Management SolutionsSchneider Electric's Enterprise Pipeline Management System (EPLMS) consists of multiple services and applications to facilitate safe and efficient operations, as shown in Figure 2-2:
L1 Basic
Control
L2 Supervisory
Control
L3 Opera�onal
Control
L3.5 Industrial
DMZ
L4-5 Office & External
Domain
L2.5 Protec�on
IEC62443 ISA99
Horizontal Inter-Zone, Intra-Zone, Inter-System Security
Process Control Power Safety Systems
Compressor / Pump Sta�on
Mul�service
Sta�on WAN, Aggrega�on & Security
Process Domain
Metering / PIG / Terminal Sta�on M
eter
ing
PIG
Syst
ems
Gas
Qua
lity
Mul�service Sta�on WANAggrega�on & Security
Process Domain
SCADA & Opera�onal Business Systems
Engineer Worksta�ons
Applica�on Servers
Domain Controller
Instrumenta�on / Sensors Instrumenta�on Instrumenta�on Instrumenta�on
Quantum Quantum
MiCom c264
SIL3 Controller
SIL3 Controller
GTW RI/O GTW RI/O
Historian Operator Sta�on Historian PACIS
Operator Historian Operator Sta�on
Wireless
Mobile Worker
IP Voice
Access Control
CCTV
RFID
Controller Controller Controller
Historian Historian Historian
SCADA Primary
Leak Detec�on
Physical Security
Operator Worksta�ons
SCADA Backup
Historian Repor�ng
Metering Systems
Main Control Center
Video Opera�ons
Access Opera�ons
Video Storage
Incident Response
Engineering
(virt
uali
-virt
ualiz
ed)
Mul�service Process Domain
Block Valve Sta�on
Quantum
Instrumenta�on
CeCCe Internet Edge 3rd Party Support
Voice
WLAN Controller
Call Manager
PAGA
Magelis
ION Metering
SEPAM Protec�on
TeSys T Motor Mgt
Al�var Drive
MiCOM Feeder
Protec�on
Magelis
RI/O
ScadaPack
SIL3 Op�on No SIL Op�on
Wireless op�on
Crew Welfare / Infotainment
Domain Controller
Engineering
Leak Detec�on
Database SCADA
Real-�me
SCADA Historical
Leak Detec�on
Applica�on Test
Real-�me SCADA Zone Development Test
Decision Support
Remote Access
Domain Controllers
(Indu
stria
l DM
Z)
Backup Control Center
Mobile Worker
IP Voice
Access Control
CCTV
RFID
WAN Connec�on & Security WAN Connec�on & Security
Mobile Worker
IP Voice
Access Control
CCTV
RFID
Sta�on WAN, Aggrega�on & Security
,
WAN Connec�on & Security
Phys
ical
Sec
urity
SCAD
A &
Bus
ines
s Sys
tem
s
Voic
e &
PAG
A
Wire
less
Deci
sion
Sup
port
& ID
MZ
Internet
WANNetworks
3G/LTE, WiMax, 900Mhz RF Mesh, Satellite, MicrowaveConverged Opera�onal Field Telecoms WirelessConverged Opera�onal Field Telecoms Wired
DWDM, Ethernet, IP/MPLS, MPLS - TP
37
65
10
2-2Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
Figure 2-2 Schneider Electrics Pipeline Management Systems
• RealTime SCADA—Schneider Electric's OASyS DNA transcends the traditional SCADA environment by incorporating the workflow needs of customers in real-time. OASyS DNA is an infrastructure product that adapts to the diverse and changing needs of an enterprise. From the field to the enterprise, OASyS DNA allows access to operational and historical data securely at anytime from anywhere.
• Oil and Gas Application Suite—Schneider Electric's RealTime Oil and Gas Suite works with the proven Schneider Electric OASyS DNA SCADA system to centralize delivery of key oil and gas pipeline information, enhancing a company's operational environment. Critical data is received for improving pipeline operations and meeting business goals. Schneider Electric offers up-to-the-minute metering and flow totaling; and calculates and monitors line pack, tank storage, hydraulic profiles, and compressor and pump performance in real-time.
• Leak Detection—The main strength of Schneider Electric's SimSuite Pipeline lies in its ability to accurately model the pipeline more completely than other available solutions. The leak-detection application uses a combination of methods to detect and locate leaks. Leaks can occur anywhere on the pipeline; they can vary in size; and they can be caused by fatigue, corrosion, equipment failure, or theft. Large and small leaks can be detected using multiple mass-balance calculations. Pressure-drop calculations can be used to locate the leak.
• Measurement Data—The Schneider Electric Measurement Advisor, empowered with Schneider Electric's advanced measurement user interface, provides the efficient and accurate means to configure devices and collect, validate, modify, and reconcile oil and gas measurement data. Part of the Schneider Electric suite of oil and gas solutions, Schneider Electric Measurement Advisor is the high-mileage solution that gathers measurements for multiple pipelines that interface with various Ethernet in the First Mile (EFM) polling engines, SCADA systems, chart integrators, third-parties, and manual input. Schneider Electric Measurement Advisor allows the precision required at every step to achieve process-wide accuracy.
Production EnvironmentThe Production Environment enables real-time operations of the pipeline network. It consists of real-time servers, operator workstations, historical servers, support servers, leak detection servers, and any ancillary services such as domain controllers. The following is a brief description of the server and its function within the production environment:
• Real Time Servers—Real-Time Service shall provide the monitoring and control function for the SCADA system.
• Domain Controller—Dedicated for SCADA, areas of responsibility, and user roles
37
65
07
2-3Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
• Logging Server—OASyS Logs, data playback, and windows events
• Deployment Server—Display and database commissioning system
• Historical Servers—Provide long-term storage of Real-Time measurement, event, alarming and other relevant data monitored or generated by the SCADA system
• Operator Work Stations—Interface for operating, controlling, and monitoring the pipeline
• Leak Detection Server
Test and DevelopmentThis section describes the test and development system.
Test System
Test systems are a non-production replica of the Production SCADA system. This provides an environment for the following functions:
• Non-production replica of the Production SCADA system
• Operating system patch, application change validation prior to production without affecting the Production SCADA system
Development System
The development system, which provides at least one development server per OASyS DNA system, has the following functions:
• Baseline application binary deployment
• Source code repository for customer developed programs, reports and scripts
• Database change control and auditing
• Storage and editing of baseline and customer-specific displays
Decision Support System/Industrial Demilitarized ZoneThe Decision Support System (DSS) isolates the operational system from any external systems or users. It is updated with real-time and historical data from the live system to provide a secure method of providing real-time data to external users. With firewalls creating an Industrial Demilitarized Zone (IDMZ), secure services are enabled to provide the external environments exposure to the data. The following describes the servers within the DSS and their functions:
• IDMZ Domain Controller
• DSS Historical Server
• Remote Access Server
The IDMZ will facilitate extra services to assist with the day-to-day operations in maintaining the integrity of the system. Such functions as patch management servers, secure FTP servers, and anti-virus servers will be housed within the IDMZ environment.
2-4Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
Control Center Availability and SecurityThis section describes control center availability and security.
Availability
Within a SCADA system design, it is recommended that no SPOF should exist. To provide the level of availability required, multiple layers of resiliency are built into the Control Center and communications architecture so that a failure of a single component does not disable the ability to control the pipeline. Redundancy is provided for all critical components and functions of the SCADA system. The Control Center will ensure availability of the IACS system against degradation or denial of essential services. High availability is recommended for the network, server, storage and application components. Depending on the risk assessment, customer philosophy, and cost of infrastructure, the chosen architecture may or may not adhere to all of the following guidelines recommended within a Control Center:
• Dual servers (physical or virtual) for all critical components
• Dual application instances in a hot/standby state machine configuration
• Dual networking platforms, including routers, switches, firewalls, and Fabric Interconnects
• Storage redundancy incorporating Redundant Array Of Independent Disks (RAID), controller, and chassis redundancy where appropriate
• Dual Storage Area Networks (SANs)
• Multiple data communication paths across the network
• Enhanced resiliency technologies provide fast failover from active to standby components
• Constant update of information between hot/standby application instances
• QoS to provide prioritization of key network traffic and data
• Process and procedures to guide quick replacement of failed devices in situations where redundancy may have not been provided
Because of the critical nature of supporting a pipeline infrastructure, at least one Backup Control Center is recommended, but situations may dictate more or less. This backup provides redundancy for events that may take down a primary Control Center. Communications may be lost with the remote pipeline or a natural disaster may occur that renders a Control Center unavailable to the pipeline stations. This Backup Control Center should be geographically separate from the Primary Control Center. The real-time and historical data stored on the components located at the backup system are kept synchronized with the primary system located in the Primary Control Center, at all times. In the event of a failure, the backup system is capable of running the entire operations for the pipeline.
Security
After availability, the integrity and confidentiality of the control system are the next most important. Integrity protects data and systems from intentional or accidental alteration. Confidentiality helps ensure that data cannot be accessed by unauthorized users. The following key security concepts, which are described in Chapter 3, “Connected Pipeline System Design,” will help provide availability of the system and support the networking infrastructure:
• Data Confidentiality and Privacy. Ensure the confidentially of information on the communications network and in storage. This may include methods such as segmentation, protecting against unauthorized access, and encryption of the data.
2-5Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
• Provide Network Segmentation and Isolation Techniques using Zones and Conduits. Within the Control Center, isolate each of the environments into zones that share a common set of security requirements or functions. Security issues seen in one zone need to be restricted from affecting or cross-pollinating into other zones. The security zones can be segmented using physical separation such as specific devices and interfaces or logically such as with Virtual Local Area Networks (VLANs).
• Restricted communications between zones using Firewalls, Access Lists, and Intrusion Protection System (IPS) devices where available. VLANs restrict traffic within a Layer 2 networking instance and the Layer 3 gateway can be used to apply policy to allow or restrict traffic between security zones.
• Threat detection and mitigation using the Firewalls, IPS devices, and segmentation.
• Access control to restrict access to resources to which a user or device is not authorized.
• Manage, monitor, log, and control the security of the infrastructure to identify, defend, and prevent any security threats or breaches.
IDMZ/DSS to isolate the operational system from any external systems or users and provide authorized remote access to the process control domain from external sources.
Figure 2-3 highlights the zones and conduits seen within the BLISS Control Center architecture. Each of the colors depicts an environment, its zone, and conduit. Details of the specific implementation are described in Chapter 3, “Connected Pipeline System Design.”
Figure 2-3 BLISS Zones and Conduits
Multi-Service Traffic (Non-Operational Applications)Multiple services will be deployed to support pipeline communications, physical security, and business enabling applications within the pipeline architecture. Segmentation of the multi-service applications from the operational communications is a common requirement. Regulatory demands, security concerns, and confidence of the operator to house the multi-service on the same infrastructure as the SCADA servers will drive the Control Center multi-service architecture. The two deployment models for enabling multiple services within the Control Center architectures for pipelines are as follows:
37
65
02
WAN to Pipeline
SCADA A VPN
Multiservice VPN
Production
Test
IDMZ• Decision Support
• Remote Access
SCADA B VPN
Multiservice
Non-Operational Domain
Operational Field Telecoms
Training
Development
RealTime (Core)
Non-RealTime (Core)
Communications and Security Infrastructure
2-6Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
• The first will house the multiple services on a separate infrastructure from the SCADA environment, as depicted in Figure 2-4.
• The second will house the multiple services on the same physical infrastructure as the SCADA environment, as depicted in Figure 2-5.
Figure 2-4 Segmented Multi-Service from SCADA Option 1
Figure 2-5 Segmented Multi-Service from SCADA Option 2
This version of the architecture will support Option 1 where multiple services will be passed to a totally separate infrastructure, outside of the operational environment.
Operational Communications Network OverviewThe Operational Telecoms architecture provides a multi-service environment that encompasses operational services (such as SCADA and process applications) and non-operational services (such as CCTV and Voice) that enable business efficiency and security along the pipeline. See Figure 2-6.
37
65
08
WAN to Pipeline
SCADA A VPN
Multiservice VPN
Production IDMZ• Decision Support
• Remote Access
SCADA B VPN
Multiservice
Non-Operational Domain
Operational Field Telecoms
RealTime (Core)
37
65
09
WAN to Pipeline
SCADA A VPN
Multiservice VPN
Production IDMZ• Decision Support
• Remote Access
SCADA B VPN
Multiservice
Non Operational Domain
Operational Field Telecoms
RealTime (Core)
2-7Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
Figure 2-6 End-to-End Communications Architecture Overview
Connectivity is required for communications between Control Centers, between the Control Centers and the pipeline, and for any inter-station communication along the pipeline. The key requirements of availability and security for the services, as mentioned in the Control Center overview are primary considerations for the architecture. Multiple secure paths through the network are provisioned to support this communication. Services are segmented, isolated, and prioritized so that SCADA networks and multi-service traffic will not affect each other under normal operations, security incidents, or network congestion.
The communications network can be built using various networking technologies. Factors that influence the communications architecture include power and space availability at the various sites, capital and operational costs, and the customer's preferred technology. Figure 2-7 provides an overview of the various technologies that may contribute to a pipeline communications architecture key requirements.
L1 Basic
Control
L2 Supervisory
Control
L3 Opera�onal
Control
L3.5 Industrial
DMZ
L4-5 Office & External
Domain
L2.5 Protec�on
IEC62443 ISA99
Horizontal Inter-Zone, Intra-Zone, Inter-System Security
Process Control Power Safety Systems
Compressor / Pump Sta�on
Mul�service
Sta�on WAN, Aggrega�on & Security
Process Domain
Metering / PIG / Terminal Sta�on
Met
erin
g
PIG
Sys
tem
s
Gas
Qua
lity
Mul�service Sta�on WANAggrega�on & Security
Process Domain
SCADA & Opera�onal Business Systems
Engineer Worksta�ons
Applica�on Servers
Domain Controller
Instrumenta�on / Sensors Instrumenta�on Instrumenta�on Instrumenta�on
Quantum Quantum
MiCom c264
SIL3 Controller
SIL3 Controller
GTW RI/O GTW RI/O
Historian Operator Sta�on Historian PACIS
Operator Historian Operator Sta�on
Wireless
Mobile Worker
IP Voice
Access Control
CCTV
RFID
Controller Controller Controller
Historian Historian Historian
SCADA Primary
Leak Detec�on
Physical Security
Operator Worksta�ons
SCADA Backup
Historian Repor�ng
Metering Systems
Main Control Center
Video Opera�ons
Access Opera�ons
Video Storage
Incident Response
Engineering
(virt
uali
-virt
ualiz
ed)
Mul�service Process Domain
Block Valve Sta�on
Quantum
Instrumenta�on
CeCCe Internet Edge 3rd Party Support
Voice
WLAN Controller
Call Manager
PAGA
Magelis
ION Metering
SEPAM Protec�on
TeSys T Motor Mgt
Al�var Drive
MiCOM Feeder
Protec�on
Magelis
RI/O
ScadaPack
SIL3 Op�on No SIL Op�on
Wireless op�on
Crew Welfare / Infotainment
Domain Controller
Engineering
Leak Detec�on
Database SCADA
Real-�me
SCADA Historical
Leak Detec�on
Applica�on Test
Real-�me SCADA Zone Development Test
Decision Support
Remote Access
Domain Controllers
(Indu
stria
l DM
Z)
Backup Control Center
Mobile Worker
IP Voice
Access Control
CCTV
RFID
WAN Connec�on & Security WAN Connec�on & Security
Mobile Worker
IP Voice
Access Control
CCTV
RFID
Sta�on WAN, Aggrega�on & Security
,
WAN Connec�on & Security
Phys
ical
Sec
urity
SCAD
A &
Bus
ines
s Sys
tem
s
Voic
e &
PAG
A
Wire
less
Dec
isio
n Su
ppor
t & ID
MZ
Internet
WANNetworks
3G/LTE, WiMax, 900Mhz RF Mesh, Satellite, MicrowaveConverged Opera�onal Field Telecoms WirelessConverged Opera�onal Field Telecoms Wired
DWDM, Ethernet, IP/MPLS, MPLS - TP
37
65
10
2-8Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
Figure 2-7 Pipeline Communications Architecture Technology Overview
The colors in Figure 2-7 signify the following:
• Green—Meets the requirements shown in column headings
• Amber—Meets requirements, but limitations exist
• Red—Does not meet requirements
As detailed earlier, various types of stations will run along the pipeline. A pipeline segment will be bookended with larger stations such as a main terminal or a pumping/compressor station. In between these larger facilities intermediate stations, such as metering stations and block valve stations, will exist. Regardless of the roles, the stations still have very similar requirements, with environmental or physical factors affecting the architecture within a station. Power may be limited at some of the stations, especially those unmanned and in remote locations such as the block valves.
Services at the stations include:
• Process control for the transportation of oil along the pipeline
• Intelligent power and motor control to manage and control the energy supply at larger stations
• Voice over IP (VoIP) for traditional enterprise voice services and a PAGA system
• Video surveillance as a component of a perimeter security system
N/A
Technology Bandwidth Latency Distance Reliability<50ms Re-
convergenceQoS
Skillsets for
Deploy/Operate
Multiservice
Support
Ethernet
MPLS
DWDM
3G
LTE
Satellite
WiMax
37
65
33
2-9Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
Figure 2-8 Compressor/Pump Station
These services lend themselves to providing a multi-service and operational infrastructure within the station environment. Security and availability are key considerations again at the station level. Segmentation of services, as previously defined, will be built into the architecture. Security policies at the network edge will be implemented that protect against unauthorized access, which is particularly important where the stations are unmanned and prevent cross-pollination of traffic between security zones.
QoS classifications, markings, and policies will provide the prioritization of resources for the services at the station and for traffic traversing the operational network to other stations or for Control Center communication.
Use Case: SCADA Real-time /Near Real-time OperationsThis section describes SCADA-to-Pipeline and Control Center-to-Control Center communications.
Figure 2-9 depicts SCADA real-time/near real-time operations.
Process Control Power Safety Systems
Compressor / Pump Sta�on
Mul�service
Sta�on WAN, Aggrega�on & Security
Process Domain
Instrumenta�on / Sensors Instrumenta�on Instrumenta�on
Quantum Quantum
MiCom c264
SIL3 Controller
SIL3 Controller
GTW RI/O GTW RI/O
Historian Operator Sta�on Historian PACIS
Operator Historian Operator Sta�on
Wireless
Mobile Worker
IP Voice
Access Control
CCTV
RFID
Magelis
ION Metering
SEPAM Protec�on
TeSys T Motor Mgt
Al�var Drive
MiCOM Feeder
Protec�on
RI/O Crew Welfare / Infotainment
WAN Connec�on & Security
37
65
11
2-10Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
Figure 2-9 SCADA Real-time/Near Real-time Operations
SCADA to Pipeline
Operators in the control room rely on consistent reliable data from the pipeline in real-time to make critical decisions for the transport of the product. The control system must be able to poll, collect, store, and display information from Intelligent Electronic Devices (IEDs), including Remote Terminal Units (RTUs), Programmable Logic Controllers (PLCs), substation controllers, and measurement instruments. Real-time communications between the pipeline and the control system must be available, reliable and secure.
The operational SCADA traffic is segmented from any other traffic that may be traversing the network and should be prioritized over non-operational less critical traffic. This segmentation must be carried through the architecture from the instrumentation reporting data within the pipeline stations into the Control Center. Prioritization within the SCADA system is also a requirement. In an application such as leak detection, information must be relayed to a decision point within the system as a priority above all other SCADA traffic.
Unsolicited traffic is sent at irregular intervals as events occurs from the RTU/Controllers to the Active Control Center. Solicited traffic is originated from the Control Center, either through an application or initiated by an operator.
Control Center-to-Control Center Communications
A backup Control Center provides disaster recovery within the system. The real-time and historical data stored on the components located in the backup system are kept synchronized with the primary system located in the primary Control Center at all times. Data is continuously updated and availability of communications is a priority. A variety of communication takes place between OASyS DNA Control
RealTime (Core)ProductionSCADA
Primary
SCADA
Backup
Decision SupportIDMZ
Training
Main Control Center
RealTime(Core)ProductionSCADA
Primary
SCADA
Backup
Decision SupportIDMZ
RTU/Controller
Application
Server
Instrumentation
Pressure, Temperature,
Differential Pressure
Ope
ra�o
nal W
AN
Cont
rol C
ente
r St
a�on
Meter Station
RTU/Controller
Application
Server
Instrumentation
Pressure, Temperature,
Differential Pressure
Block Valve Station
RTU/Controller
Application
Server
Instrumentation
Pressure, Temperature,
Differential Pressure
Meter Station
Backup Control Center
Polling unsolicited
Polling standard
RTU to instrument
RTU to server
Server to instrument
Backup polling standard
CC to CC replication
37
65
13
2-11Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 2 Connected Pipeline Architecture Control Center Overview
Centers. For example, the OASyS RealTime Servers pass configuration information and real-time SCADA data. Control Centers will also have one or more domain controllers that need to communicate Active Directory information to each other.
The system provides secure available communications for these components between the Control Centers. Each OASyS DNA system within an installation must have a properly configured firewall or router acting as a boundary between the control network and any external LAN or WAN as per IEC 62443-3-3, Foundational Requirements, SR 5.2, Zone Boundary Protection. This configuration restricts communications to only those ports required for domain controller communications and OASyS DNA functionality. Firewall access controls must be configured to control traffic bi-directionally.
2-12Control Center Virtualization for the Connected Pipeline
Design Guide
ContDesign Guide
C H A P T E R 3
Connected Pipeline System DesignThis chapter includes the following major topics:
• BLISS Design, page 3-1
• Security Design, page 3-3
• Availability, page 3-17
BLISS DesignFigure 3-1 provides an overview of the SCADA system physical architecture. The BLISS system follows a single layer data center design. The Control Center is built around the Cisco Unified Compute System (UCS), Nexus 3524 Ethernet-collapsed Aggregation/Access Networking Platform, Cisco Adaptive Security Appliance (ASA), and the EMC VNX series storage arrays.
Figure 3-1 SCADA System Physical Architecture Overview
Inside, Outside, DMZ(DSS)
BLISS BaseLine Integrated
SCADA System
Enterprise
Fiber Channel
Twinax
Ethernet
Dual ASA 55X5-x Firewalls
Ac�ve/standby
Dual Nexus 3524 vPC Connec�vity with ASA &
UCS
Dual UCS Fabric Interconnects
2 x Direct A�ached Storage arrays to UCS Fabric Interconnects
2 x UCS Chassis with B200 M4
servers
WAN Routers
2x B200 M4 servers per SCADA Zone Each server
housed in a different Chassis
Fabric
Interconnection
37
65
25
3-1rol Center Virtualization for the Connected Pipeline
Chapter 3 Connected Pipeline System Design BLISS Design
In the context of the Schneider Electric BLISS, the UCS B series servers will house each of the production, DSS, test, development, and training environments using virtualized servers. These services and applications will run on Microsoft Hyper-V virtual-enabled machines. Each physical server will house an environment dedicated to it with a backup version of the server housed on another physical server in another UCS chassis. Two storage arrays will support redundancy and will be directly connected to the UCS system at the 6248 Fabric Interconnects. A pair of Nexus 3524 switches will extend the Layer 2 domain to a pair of ASA firewalls where Layer 3 gateway functionality will enable policy management for any traffic that needs to traverse outside of the Layer 2 domain. The operator workstations will have connectivity into the environment through a pair of Catalyst switches that will have connectivity to the Nexus 3524 switches.
Functional Data Center DesignThe Control Center design, as shown in Figure 3-2, can be broken into four key functional blocks:
• Network—LAN design, Layer 2 and Layer 3 Networking
• Compute—UCS B series, Hypervisor, vSwitch
• Storage—Fiber Channel, SANs (directly attached to Fabric Interconnects)
• Network Security—IDMZ, Decision Support System
Figure 3-2 Functional Data Center Design
Network
The Networking Layer connects to the WAN Provider Edge (PE) router terminating the L3VPN. The Nexus 3524 Layer 2 switch is serving as both an access and aggregation node. Although the ASA is stated to be a security device, it is also serving as a Layer 3 gateway within this architecture to enforce networking policy and provide termination of the Layer 2 domain. The 3850 switches will extend the networking architecture to provide network connectivity for the operator workstations.
Netw
ork
Com
pute
Sto
rage
2x ASR 902
2x Nexus 3524
2x Catalyst 3850
2 x EMC VNX 3200
Storage Array
Directly Attached
Storage to Fabric
Interconnects
2 x UCS 6248UP Fabric
Interconnects
2 x UCS 5108
16 x UCS B200 M4 (8 per chassis)
Security
2 x ASA 55x5-X
37
65
16
3-2Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Compute
The Compute Building block includes several systems. The Unified Compute System features redundant 6248 Fabric Interconnects, UCS 5108 blade chassis, and B series blade servers. The hypervisor in this architecture will be Microsoft Hyper-V. Within this architecture, the Hyper-V vSwitch provides a virtual access switching layer that is applied to the Virtual Machines on the physical servers that can apply QoS, Access Control Lists (ACLs), and enhanced security deeper into the architecture.
Storage
The storage layer includes the 6248 Fabric Interconnects for connectivity to the EMC storage. This is directly connected from the Fabric Interconnects to the storage using Fiber Channel. EMC is used here in this release; however, the solution is storage agnostic.
Security
Finally, the ASA firewalls have the capability to not only provide traditional firewall services such as stateful inspection, access control, and remote access, but with Sourcefire can also provide IPS, Intrusion Detection System (IDS), and anomaly detection. In this phase, IPS and IDS will not be enabled. Only traditional firewall services will be validated.
Security DesignThis section describes security design, which includes segmentation design, compute segmentation, and storage segmentation.
Segmentation DesignIn Oil and Gas Pipeline architectures, the ability to securely restrict and isolate services to preserve the integrity of the traffic is paramount. Intentional and accidental cross-pollination of traffic between untrusted entities must be restricted. Securing a perimeter for a SCADA service can be achieved using isolation techniques at the various levels of the infrastructure. The security zones can be segmented using physical separation, such as specific devices and interfaces, or logically, such as with VLANs. VLANs restrict traffic within a Layer 2 networking instance and an ASA firewall can be used to apply policy to allow or restrict traffic between security zones. With logical association of virtual firewalls, VLANs, and L3VPN instances, physically-separated interfaces of equipment and security policies, perimeters and zones can be defined. These techniques are not just restricted to traditional networking platforms, but also must be followed throughout the architecture into the compute and storage areas. These techniques may be physical or logical: physical in the segmentation of physical equipment or logical where comfort and familiarity with virtualized techniques exists or where it makes sense from a business perspective.
Control Center Segmentation and Zones
To address the requirement to segment services to promote a dedicated infrastructure per service, the Connected Pipeline 1.0 architecture will promote physical and logical path isolation techniques. The BLISS architecture will take ownership of the security and segmentation from the firewalls. Multi-service traffic will be terminated on the WAN routers outside of the BLISS domain, placed onto
3-3Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
a separate interface, and passed to the relevant domain. Figure 3-3 represents the network segmentation for a Control Center. The WAN routers are not incorporated as components of the BLISS architecture. The WAN/Communications network should be agnostic to the BLISS.
Figure 3-3 BLISS Segmentation
Layer 2 Network Segmentation
VLANs will provide the segmentation on a per-zone or system level within the Control Center. The production environment seen here will have its own dedicated VLAN as will the test, development and IDMZ/DSS, training, and DSS. The production environment will incorporate all of the servers required to operate the real-time environment and include the operator workstations.
802.1q trunking and Virtual Port Channel (vPC) will be enabled to extend VLAN segmentation on the shared links between the servers, Nexus 3524, and firewalls. 802.1q trunking and vPC provide a method to allow the VLANs that are logically separated to share a set of physical links and connect to two separate Nexus switches (Figure 3-4, option 1). The data path for communications is still contained within the VLAN and segmentation is still maintained.
37
65
17
WAN to Pipeline
SCADA A VPN
Multiservice VPN
Production
Test
IDMZ• Decision Support
• Remote Access
SCADA B VPN
Multiservice
Non-Operational Domain
Operational Field Telecoms
Training
Development
RealTime (Core)
Non-RealTime (Core)
Communications and Security Infrastructure
3-4Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Figure 3-4 Layer 2 Network Segmentation Options
The Connected Pipeline architecture recognizes that further segmentation above pure logical segmentation may be a business- or risk-associated requirement. Therefore, a second option for the segmentation may include physical segmentation where the business warrants it.
An example is the production environment that has requirements for an extra level of segmentation in the networking architecture. Physical isolation using dedicated non-shared ports may be deployed for this zone. This may also be a requirement for the DSS environment. Therefore, dedicated physical ports from the UCS system through to the 3524 switches and the ASA firewalls could be deployed for the production environment.
802.1q trunking of VLANs will be configured on shared ports for the less essential services. For example, the test, development, and training environments will use this method.
Figure 3-4, Option 2 provides a view of the physical/logical segmentation seen in the design at the networking level. The production servers (Gray) housed on separate UCS chassis hit the Fabric Interconnects via a unified fabric. The VLAN is then extended through the switch and to the ASA firewall on its own dedicated physical cabling and port. The test, training, and development zones are transported using logically separate VLANs, but over a shared physical medium using 802.1q trunking.
When looking to use segmentation with more of a physical rather than logical approach, be aware of the port availability on the infrastructure components, specifically the firewalls.
Layer 3 Network Segmentation
Any inter-VLAN routing will require explicit configuration on the firewall. The firewall is acting as the Layer 3 gateway for the servers and will therefore enforce policies for any inter-zone or inter-system traffic. Access lists, rules, and policies will be configured at the firewall to allow or deny communication
The firewalls will have routing enabled to the WAN-facing routers and will provide access to the Enterprise through an Industrial DMZ.
Summary
• VLANs provide the logical segmentation for the network.
• 802.1q trunks and vPC provide shared infrastructure for the logical segmentation.
• Further segmentation above logical segmentation can be provided with dedicated physical ports as described.
x8 x8
x8x8
Fabric Ports
All Zones Shared, dot1q
37
65
18
x8 x8
x8x8
Fabric Ports
Dedicated Physical ports Production ZoneDot1q VLANs for Test, Development and Training Zones
Production
Domain
Test, Development
DomainProduction
Domain
Test, Development
Domain
3-5Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
• Policy enforcement is at the ASA firewall where the Layer 2 is terminated and the ASA provides the Layer 3 gateway functionality for the Control Center zones.
Compute SegmentationFigure 3-5 provides an overview of the compute and storage setup. Each environment will have its own set of dedicated servers or server. In the production environment, it is recommended to have a pair of physical servers to provide redundancy. To enhance redundancy, each of these physical servers will be housed in a separate B series chassis. The virtual machines in these environments/zones all belong to the same VLAN and no requirement exists to house a virtual machine/server outside of its dedicated VLAN.
Figure 3-5 Compute and Storage Setup Overview
Virtual Switch (vSwitch)
From a networking perspective, each of the machines in a zone has the capability to see each other as they are in a single VLAN instance. A vSwitch is an existence of a switch housed on the hypervisor to provide typical network switching functions for the Virtual Machines. This is very similar to a standard access switch that would house regular physical compute. A Hyper-V virtual switch has the capabilities to tie a VLAN to a Virtual Machine providing segmentation at the hypervisor. If intra-VLAN traffic requires policies, then ACLs can be applied at this layer to enforce policy within a zone.
Virtual Network Interface Cards (vNICS)
In conjunction with a vSwitch, the Cisco Virtual Interface Cards (VICs) have the capability to create virtual adapters or vNICs. These allow multiple vNICs to be created rather than physical interfaces. This allows separation of traffic where logical segmentation of data traffic from the Virtual Machines can be separated from the infrastructure management traffic.
37
65
19
Fiber Channel
Unified Fabric FI
Connection
Ethernet
UCS B series Physical Server
Production Domain Active
• VM Real Time SCADA
• VM Historical Cluster• VM Logging, Eng Server
• VM Domain Control
• VM LDS
• VM RAS
UCS B series Physical Server
Production Domain Standby
• VM Real Time SCADA
• VM Historical Cluster• VM Logging, Eng Server
• VM Domain Control
• VM LDS
• VM RAS
UCS B series Physical Server
Decision Support System Active
• VM Real Time SCADA
• VM Historical Cluster• Vm Domain Control
• VM RAS
UCS B series Physical Server
Decision Support Standby
• VM Real Time SCADA
• VM Historical Cluster• VM Domain Control
• VM RAS
Boot Storage for
Standby Servers
Boot Storage for
Active Servers
SAN A FC Link x1
to each FI-A
SAN B FC Link to
FI-B
x2
x2
x2
x2
Production/Historical
SQL DBs
Production
DomainDecision
Support Sys
DSS Historical SQL
DBs
x8
x8 x8
x8
3-6Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Summary
• Each environment will have a set of dedicated servers or server.
• In the Production Environment, it is recommended to have a pair of physical servers to provide redundancy.
• Physical pairs of servers will be spread between two UCS Chassis. Chassis redundancy provides an extra level of comfort over pure fabric redundancy.
• Virtual Machines are connected to a vSwitch where VLAN, QoS, and security policies are applied.
• Segmentation is carried through the architecture from the Virtual Machine through the vSwitch, UCS system, Nexus 3524 to the firewalls using VLANs.
Storage SegmentationThis section describes file system management, fiber channel zoning and VSANs, and LUN masking.
File System Management
Each Virtual Machine will have its own dedicated storage for its running environment housed on the storage array. The Microsoft Hyper-V hypervisor's cluster file system management creates a unique Virtual Hard Disk (VHD) per Virtual Machine, ensuring that multiple Virtual Machines cannot access the same VHD sub-directory within the New Technology File System (NTFS) volume, and thus isolating one tenant's VHD from another.
Fiber Channel Zoning and VSANs
Segmentation is enhanced with the configuration of Fiber Channel zoning and Virtual Storage Area Networks (VSAN) in a Fiber Channel storage network. These technologies allow for the physical SAN fabric to be segmented into smaller logical domains.
The architecture will deploy a pair of VSANs. The VSANs will be tied to fabric paths across the infrastructure. This enables the fiber channel redundancy through the use of a Fabric A and Fabric B, which are physically and logically separate.
This partitioning of fabric services greatly reduces network instability by containing fabric reconfigurations and error conditions within an individual VSAN. The strict traffic segregation provided by VSANs helps ensure that the control and data traffic of a specified VSAN are confined within the VSAN's own domain, increasing SAN security and availability. Fiber Channel zoning is deployed to take security to the next layer by providing access control between devices in a VSAN; for example, in the architecture, it allows control and storage policy of storage communications between a host and a storage controller/chassis.
Depending on the security policy and risk assessment, multiple VSANs can be deployed per SCADA zone. Similar to the networking architecture where the networking traffic had its own physical links, the production environment may require its own set of VSANs tied to separate physical SANs. Care must be taken to accommodate the correct equipment to ensure port availability for this architectural approach. This, though, is outside the scope of this CVD.
Figure 3-6 highlights the design within scope. This is the traditional deployment where a VSAN A will be tied to Fabric A, and the VSAN B will be tied to Fabric B. Both storage arrays will be connected using these VSANs/Fabrics.
3-7Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Figure 3-6 Fiber Channel Zoning Design
LUN Masking
Logical Unit Number (LUN) masking should be configured as an extra level of security at the storage level over VSANs and FC zoning. LUN masking creates an authorization process that restricts storage LUN access to specific hosts on the shared SAN. The storage controllers usually create LUN masking to enforce access policies between hosts and particular LUNS on the storage system.
Summary
• Dual Storage Arrays, each serving different hosts and physical servers on each UCS Chassis
• Segmentation on the storage system using VSANs, FC Zoning, and LUNs
• Standard VSAN A/B mapping to UCS Fabric A and B
• The strict traffic segregation provided by VSANs helps ensure that the control and data traffic of a specified VSAN are confined within the VSAN's own domain, increasing SAN security and availability
• An option to enhance segmentation with extra VSANs to isolate the production storage network from the non-production environments is outside the scope of this document, but is acknowledged as a consideration
Firewall and Industrial DMZ DesignThis section describes firewall deployment options, IDMZ concepts, IDMZ, and DSS design.
37
65
20
Fiber Channel
Unified Fabric FI
Connection
Ethernet
UCS B series Physical Server
Production Domain Active
• VM Real Time SCADA
• VM Historical Cluster• VM Logging, Eng Server
• VM Domain Control
• VM LDS
• VM RAS
UCS B series Physical Server
Production Domain Standby
• VM Real Time SCADA
• VM Historical Cluster• VM Logging, Eng Server
• VM Domain Control
• VM LDS
• VM RAS
UCS B series Physical Server
Decision Support System Active
• VM Real Time SCADA
• VM Historical Cluster• Vm Domain Control
• VM RAS
UCS B series Physical Server
Decision Support Standby
• VM Real Time SCADA
• VM Historical Cluster• VM Domain Control
• VM RAS
Boot Storage for
Standby Servers
Boot Storage for
Active Servers
SAN A FC Link x1
to each FI-A
SAN B FC Link to
FI-B
x2
x2
x2
x2
Production/Historical
SQL DBs
Production
DomainDecision
Support Sys
DSS Historical SQL
DBs
x8
x8 x8
x8
3-8Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Overview
The firewalls, as shown in Figure 3-7, provide the following foundational security functions within the pipeline architecture:
• A security boundary between the WAN for connection to the pipeline and the Backup Control Center within the process control network
• Architecture for communicating with the external business and enterprise domain using an IDMZ
• An Intra-Control Center policy between zones and segments as explained in the Network segmentation section earlier
• An IDMZ for operational data that can be accessed by the enterprise, and a secure staging area for patching and anti-virus services
Figure 3-7 Firewall and IDMZ Design
Firewall Deployment Options
The choice of firewall deployment will depend on customer philosophy and user requirements, as detailed earlier. The section highlights three deployment options; this CVD, however, will focus on the consolidated firewall deployment.
Single Consolidated Firewall
The Single Consolidated Firewall provides policy for all communications between zones in the Control Center, including the policies for the DSS/IDMZ, WAN, and any inter-zonal communications between environments. Remote access policies are enabled on these firewalls to provide the access into the process domain. All services are deployed as a single pair of ASA devices, providing a single point of administration.
WAN to Pipeline
RealTime/ Historian Deployment Servers
Production
IDMZDecision Support System
RAS
SCADA VPN
BLISSBaseline Integrated
SCADA System
WAN/Communications Network
Inside
100
DMZ
50
Outside 0
Enterprise 0
Enterprise
Inter CC Replication and SCADA Pipeline traffic
Remote Client Service to RAS
and Secure Remote Access
Replicated RealTime
and Historian tp DSS
for access via RCS
37
65
21
3-9Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Multi-Context Firewalls
Multi-Context Firewalls can be thought of as virtual instances of firewalls on the same physical device. Each context provides an independent device with its own security policy, interfaces, and administration. Within the Control Center architecture, this would allow differing policies and administration for the DSS/IDMZ from the rest of the Control Center, akin to a totally separate firewall. The limitations to the deployment are detailed below:
• Redundancy/failover supports active/active mode only
• Multiple Context mode does not support the following features:
– RIP
– OSPFv3 (OSPFv2 is supported)
– Multicast routing
– Threat Detection
– Unified Communications
– QoS
The ASA family has supported multi-context firewalls since its initial release; however, no virtualization support for Remote Access has existed with multi-context configured until now. From 9.5.2, multi-context-based virtualization has support for VPN Remote Access connections to the ASA. For more information, please refer to the ASA: Multi-Context Mode Remote-Access (AnyConnect) VPN at the following URL:
• http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-firewalls/200353-ASA-Multi-Context-Mode-Remote-Access-A.html
Note If the supported features are not sufficient to support remote access VPN, a separate set of firewalls would need to be deployed. This may lead the customer to physically separate firewalls if administrative separation is required since multiple physical firewalls would need to be deployed (two sets of firewalls).
Physically Separate Firewalls
Multiple physical firewalls may be deployed. A separate pair of firewalls are deployed for the IDMZ/DSS and remote access, and a second pair of firewalls provide the policy for the remaining zones and WAN communications. This could have differing administration and administrators.
Industrial DMZ Concepts
Services and data need to be exchanged between the untrusted Enterprise network and the trusted Process Control Network (PCN). This exposes near real-time and historical information to the Enterprise and allows for better business decisions. Systems located in the Industrial DMZ bring all the data together for company personnel in a near real-time system, allowing the ability to forecast and change daily operations that will generate the most revenue for the company.
In aligning with standards such as IEC 62443/ISA99 and ISA 95, a requirement exists in the Process Control Domain (PCD) to provide strict policy enforcement between the trusted Levels 1-3 of the PCD and the untrusted Levels 4-5 of the Enterprise/business domain. No direct communications are allowed between the Enterprise and PCD. The IDMZ commonly referred to as Level 3.5 provides a point of access and control for the access and exchange of data between these two entities. The IDMZ
3-10Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
architecture provides termination points for the Enterprise and the process domain and then has various servers, applications, and security policies to broker and police communications between the two domains.
The following are key guidelines and concepts for the IDMZ:
• No direct communications should occur between the Enterprise and the PCD.
• The IDMZ needs to provide secure communications between the Enterprise and the PCD using mirrored or replicated servers and applications.
• The IDMZ provides for remote access services from the external networks into the PCD.
• The IDMZ should a security barrier to prevent unauthorized communications into the PCD and, therefore, create security policies to explicitly allow authorized communications.
IDMZ and DSS Design
Within the context of the Connected Pipeline architecture, the IDMZ houses the DSS, which provides proxy or brokering services for communications between the enterprise and PCD.
A historical and real-time server is located within the IDMZ that interacts with servers within the Level 3 PCN production environment. The information from the PCN is replicated to the servers in the IDMZ so that services such as Remote Client Server can facilitate the passing of real-time and historical information to the IDMZ.
Remote access servers such as a remote desktop gateway are also situated in the IDMZ. This allows for a secure method to communicate with the PCD for remote workers or contractors that may require access. Other services would typically include operating system patch server, anti-virus server, and secure FTP server/gateway: in fact, any service that requires information or interaction with Level 4 and above to assist with the day-to-day operations of the Pipeline Control Center.
Policy
The firewalls are recommended to follow the practice of denying any service unless explicitly configured. All services must be identified and rules put in place to allow the communications through the firewall. This requires that any TCP/UDP port numbers are identified so that this explicit configuration can be created.
Security Levels
The interfaces on the ASA require security levels to define the level of trust associated with that particular interface. The security level is set for values of 0-100 where the higher the value, the more trusted the interface. Implicit communications are allowed between higher to lower security levels. However, it is recommended to explicitly configure policies to overwrite the implicit nature of security levels. Therefore, within the Control Center firewall design, the security levels are really nothing other than an identifier of trust for an interface.
VLAN Segmentation
VLAN segmentation, as explained earlier, is a common component in the security framework to assist with isolating services in the Control Center architecture. This can be extended into the IDMZ architecture too. In creating several VLANs within the IDMZ and DSS environment, the servers can be isolated so that, if compromised, the servers within the VLAN container can be restricted from impacting other servers within the IDMZ.
3-11Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Remote Access Overview
The business and security practices of a customer may influence the chosen method for remote access. The methods discussed in this section focus on using remote desktop services to access the environment. This is not an exclusive list and the client-based VPN service using Cisco AnyConnect could be used to offer another method to access the IDMZ or process domain.
Schneider Electric Remote Client Service
The Schneider Electric Remote Client Service (RCS) provides a means for secure remote access to the SCADA applications. The client is required to be a member of the control system domain. See Figure 3-8.
Figure 3-8 Schneider Remote Client Service Design
• The client communicates with the RCS server in the IDMZ through ports explicitly configured on the firewall. The corporate network is prevented from seeing any other server.
• For individual users remotely accessing the Decision Support SCADA system applications, user access management is handled through a secured RCS server after establishing a VPN connection to the SCADA system. The client initiates a connection using an Operating System client and is authenticated for application access via an instance of Active Directory.
• The RCS server queries data from the RealTime and Historical services running within the Decision Support/IDMZ on behalf of the user.
• The Historical and RealTime services in the Decision Support/IDMZ, in turn, receive their data from the operational instances of the Historical and RealTime services running in the Production Environment through replication services.
• All communication takes place between the client and the RCS server over a single port with SSL-encrypted TCP/IP.
ENTERPRISE
Produc�on
DMZ Decision Support and RAS
Remote Access Jump Servers, RCS & RDP
Cisco ASA
Leve
ls 1
-3
Leve
l 3.5
Le
vels
4-5
mote Accessmp Servers, RCS &P
Replica�on to DSS Systems
from RealTime and Historical
RCS Service with Authorized Users
User exXOS Client
37
65
22
RealTime/ Historian Produc�on Servers
3-12Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Clientless VPN Remote Desktop Protocol Plug-in
The ASA has the capabilities to terminate remote VPN connectivity with use of a web browser and SSL encryption. This is termed as a clientless VPN because no forced download or requirement exists for the end user to have a specific client installed for VPN access (Cisco also offers a client-based VPN using Cisco AnyConnect). See Figure 3-9.
Figure 3-9 Clientless VPN RDP Plug-in
• The user, if outside of the corporate network, creates a VPN session to the corporate firewall. (Policy, authentication, and authorization would all have to be worked with the enterprise IT team to allow this connection.)
• The user opens a browser and enters the IP address of the IDMZ Firewall. Once connected, the user is authenticated against the Active Directory in the IDMZ.
• After authentication, the user is presented with a web portal and a preconfigured bookmark based on the user's security profile. In this instance, this is an Remote Desktop Protocol (RDP) bookmark to a server in the IDMZ or could be as seen an RDP session into the Decision Support SCADA system.
• The Remote Desktop session is opened from the firewall to the defined server in the bookmark and the RDP session is now presented back to the user.
• No direct access to the environment exists.
Remote Desktop Gateway Service
An alternative to using the clientless VPN and RDP plug-in is to position a Remote Desktop Gateway in the IDMZ. This server can broker RDP communication between the client outside of the IDMZ to servers in the IDMZ or into the PCD. See Figure 3-10.
Produc�on
ENTERPRISE INTERNET
3rd Party
Remote Worker
Cisco ASA
Leve
ls 1
-3
Leve
l 3.5
Le
vels
4-5
Business Domain
on
Cisi
Woooooooooooooooooooooooooooooooooorrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrkkkkkkkkkkekkkkkekekeekekekekekekekkkkeeekekkkkkkkeeekkkkkkekeeekkkkkkekekkkkkkkkkkkkkkkkkkkkk
innnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
IDMZ Decision Support and RAS
IIINNNTTTEERNET
Outside VPN Session
Clientless VPN Session RDP Session
37
65
23
RealTime/ Historian Produc�on Servers
3-13Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Figure 3-10 Remote Desktop Gateway
The user, if outside of the corporate network, creates a VPN session to the corporate firewall. (Policy, authentication and authorization would all have to be worked with the enterprise IT team to allow this connection.)
The user opens the RDP client from a computer. The user enters the address of the specific host to which they wish to connect and the address of the gateway server. (The firewall will only allow access to the gateway server in the IDMZ so the direct access to the server would be denied.)
The gateway server authenticates the user and, once authenticated, brokers RDP communication between the client and the server in the PCD or IDMZ.
Inter-Control Center Firewall Considerations
An overview of Control Center-to-Control Center communications can be found in Use Case: SCADA Real-time /Near Real-time Operations, page 2-10. Services for replication of data between the primary and backup Control Center and supporting services such as Active Directory need to be explicitly configured. This configuration restricts communications to only those ports required for Domain Controller communications and OASyS DNA functionality. Firewall Access Controls must be configured to control traffic bi-directionally.
Other services such as remote access may be required between Control Centers and should be configured if warranted.
Pipeline Communication Firewall Considerations
To allow communication between the production environment and the pipeline devices, explicit IP addresses and TCP/UDP port numbers should be identified for the protocols, services, and devices that require communication. Depending on the SCADA protocols used, this could be different for every customer.
Produc�on
DMZ Decision Support and RAS
Remote Access Jump Servers, RCS & RDP
ENTERPRISE INTERNET
3rd Party
Remote Worker
Cisco ASA
Leve
ls 1
-3
Leve
l 3.5
Le
vels
4-5
Business Domain
c�on
Jump Servers, RCS &RRDP
IIIIIINNNNNNTTTTTTEEEEEERNRNRNRNRNRNETETETETETET
Outside VPN Session SSL Session
RDP Session
37
65
24
RealTime/ Historian Produc�on Servers
3-14Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
Intra-Control Center Firewall Considerations
The firewall is the policy point for configuration of any intra-Control Center communication. In keeping with the segmentation policy, any inter-VLAN traffic should be explicitly configured on the firewall if communications is required between zones.
Summary
• Firewall policies for the SCADA Zone, IDMZ/DSS (including Remote Access and Enterprise interaction), and Development and Test environments.
• Traffic and route policies applied at the ASA Firewall to prevent inter-zone traffic, unless explicitly configured.
• Operational Telecoms WAN to SCADA and vice versa: ACLs will be explicitly configured to permit traffic between the WAN and the SCADA zone at the ASA.
• Replication between Control Centers will require specific ports to be opened at the firewalls of both Control Centers. The Schneider Electric Oil and Gas OASyS DNA 7.6 Firewall Requirements document contains the specific ports required between Domains (Zones) and physically-separated locations.
• Using the instances of replicating historical and real-time data, the servers in the SCADA zone will be explicitly permitted to allow traffic to the DSS in the IDMZ.
• RCS port will be open to allow corporate users to access the DSS for the purpose of reading the historical and real-time data.
• Clientless VPN and RDP Gateway services are alternatives for remote access to the RCS from Schneider Electric.
• DSS/IDMZ could be deployed with extra VLAN segmentation within the environment to provide further isolation of servers.
Network Infrastructure SecurityTo provide extra levels of security, basic infrastructure mechanisms exist to help protect the switching infrastructure. The guidelines described in this section should be followed.
Shutdown Unused Ports
Place any ports not being used into a shutdown state. For the purpose of a switch, add the switchport VLAN command with an unused VLAN (not VLAN 1) so that if a port is accidentally activated, it will not affect any deployed VLANs.
Trunk Ports
The following best practices should be practiced or order to prevent such attacks as VLAN hopping.
When configuring trunk ports, turn DTP off and explicitly configure trunking on the infrastructure ports. Use a dedicated VLAN ID on all trunk ports. Do not use VLAN 1 for anything and configure all tagged mode for the native VLAN on trunks. Only add the VLANs that are required on the trunk ports.
3-15Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Security Design
DHCP Snooping
If servers or workstations in the architecture are using Dynamic Host Configuration Protocol (DHCP), then DHCP snooping and Dynamic ARP Inspection (DAI) should be considered.
DHCP snooping is a security feature that acts like a firewall between untrusted hosts and trusted DHCP servers. The DHCP snooping feature performs the following activities:
• Validates DHCP messages received from untrusted sources and filters out invalid messages
• Rate-limits DHCP traffic from trusted and untrusted sources
• Builds and maintains the DHCP snooping binding database, which contains information about untrusted hosts with leased IP addresses
• Uses the DHCP snooping binding database to validate subsequent requests from untrusted hosts
Other security features, such as DAI, also use information stored in the DHCP snooping binding database. Refer to the "Configuring DHCP" chapter of the Consolidated Platform Configuration Guide, Cisco IOS XE 3.7E and Later (Catalyst 3850 Switches) at the following URL:
• http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/37e/consolidated_guide/b_37e_consolidated_3850_cg/b_37e_consolidated_3850_cg_chapter_01000001.html
Dynamic ARP Inspection (DAI)
DAI is a security feature that validates ARP packets in a network. DAI intercepts, logs, and discards ARP packets with invalid IP-to-MAC address bindings. This capability protects the network from some man-in-the-middle attacks. Please refer to the "Configuring Dynamic ARP Inspection" chapter of the Consolidated Platform Configuration Guide, Cisco IOS XE 3.7E and Later (Catalyst 3850 Switches) at the following URL:
• http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/37e/consolidated_guide/b_37e_consolidated_3850_cg/b_37e_consolidated_3850_cg_chapter_01000011.html
Port Security
Port security limits the amount of MACs on a particular interface. This helps to prevent threats such as MAC attacks. Port security should be enabled on access ports. Port security with dynamically learned and static MAC addresses can be used to restrict a port's ingress traffic by limiting the MAC addresses that are allowed to send traffic into the port. When secure MAC addresses are assigned to a secure port, the port does not forward ingress traffic that has source addresses outside the group of defined addresses. Port security should be enabled on the 3850 for the workstation connectivity. Port security is currently not available on the Nexus 3000 Series.
Details of the port security feature and configuration guidelines can be found in the "Configuring Port-Based Traffic Control" chapter of the Consolidated Platform Configuration Guide, Cisco IOS XE 3.7E and Later (Catalyst 3850 Switches) at the following URL:
• http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/37e/consolidated_guide/b_37e_consolidated_3850_cg/b_37e_consolidated_3850_cg_chapter_01000110.html
3-16Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
Traffic Storm Control
A traffic storm occurs when packets flood the LAN, creating excessive traffic and degrading network performance. The traffic storm control feature can be used to prevent disruptions on Ethernet interfaces by a broadcast, multicast, or unknown unicast traffic storm.
For guidance and configuration for this feature, refer to “Configuring Traffic Storm Control” of the Cisco Nexus 3000 Series NX-OS Layer 2 Switching Configuration Guide, Release 7.x at the following URL:
• http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/layer2/7x/b_Cisco_Nexus_3000_Layer_2_Switching_Config_7x/b_Cisco_Nexus_3000_Layer_2_Switching_Config_7x_chapter_01100.html
For port-based traffic control on the 3850, refer to the “Configuring Port-Based Traffic Control” chapter of the Consolidated Platform Configuration Guide, Cisco IOS XE 3.7E and Later (Catalyst 3850 Switches) at the following URL:
• http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/37e/consolidated_guide/b_37e_consolidated_3850_cg/b_37e_consolidated_3850_cg_chapter_01000110.html
Infrastructure Management
Most of the networking management protocols that are used are insecure (such as SNMP, Telnet, and FTP). The secure variants of these protocols (SSH in place of Telnet, SNMP V3) should be used.
Use OOB management to manage the infrastructure. The preferred method is to run the management infrastructure separately from the main Control Center infrastructure. If this is not possible, use a dedicated VLAN for management.
Protect the user access and privilege levels of all the networking devices. AAA should be enabled. The preferred method would be to use a Terminal Access Controller Access-Control System Plus (TACACS+).
Control Plane Policing (CoPP)
To protect the control plane from being overwhelmed, the Cisco devices have Control Plane Policing (CoPP) features. In the case of the Nexus 3524 switch, the device segregates different packets destined for the control plane into different classes. Once these classes are identified, the Cisco NX-operating system device polices the packets, which ensures that the supervisor module is not overwhelmed.
For guidance for CoPP on a Nexus 3524 switch, refer to the “Configuring Control Plane Policing” chapter in the Cisco Nexus 3000 Series NX-OS Security Configuration Guide, Release 7.x at the following URL:
• http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/security/7x/b_Cisco_n3k_Security_Config_7x/b_Cisco_n3k_Security_Config_7x_chapter_01011.html
AvailabilityA foundational requirement for the Control Center implementation is to provide a highly available infrastructure. This section discusses some of the technologies required to implement this requirement. The goal of this section is to provide the technologies that will be validated in the design so that those implementing a Control Center architecture can make informed decisions for the best implementations that will address their business requirements and objectives.
3-17Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
Figure 3-11 highlights the key redundancy principles described in Chapter 2, “Connected Pipeline Architecture.”
Figure 3-11 Key Redundancy Principles
• Dual servers for all critical components
• Dual application instances in a hot/standby
• Dual networking platforms, including routers, switches, firewalls, and Fabric Interconnects
• Storage redundancy incorporating RAID, controller, and chassis redundancy where appropriate
• Dual SANs
• Multiple data communication paths across the network
• Using enhanced resiliency technologies provide fast failover from active to standby components
• Constant update of information between hot/standby, mirrored servers, and applications
• QoS to provide prioritization of key network traffic and data
• Process and procedures to guide quick replacement of failed devices in situations where redundancy may have not been provided
Network RedundancyFigure 3-12 depicts network redundancy.
Inside, Outside, DMZ(DSS)
BLISS BaseLine Integrated
SCADA System
Enterprise
Fiber Channel
Twinax
Ethernet
Dual ASA 55X5-x Firewalls
Ac�ve/standby
Dual Nexus 3524 vPC Connec�vity with ASA &
UCS
Dual UCS Fabric Interconnects
2 x Direct A�ached Storage arrays to UCS Fabric Interconnects
2 x UCS Chassis with B200 M4
servers
WAN Routers
2x B200 M4 servers per SCADA Zone Each server
housed in a different Chassis
Fabric
Interconnection
37
65
25
3-18Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
Figure 3-12 Network Redundancy
Port Channel
A port channel bundles individual links into a channel group to create a single logical link that provides the aggregate bandwidth of up to eight physical links. If a member port within a port channel fails, traffic previously carried over the failed link switches to the remaining member ports within the port channel.
Port channels are configured as highlighted in Figure 3-12 at the Firewalls and the Fabric Interconnects.
Virtual Port Channel
vPCs allow multiple links to be used between a port channel-attached device and a pair of participating switches. The two Nexus 3524 switches act as vPC peer endpoints and look like a single logical entity to the port channel-attached devices. vPC should be enabled to provide link redundancy and optimal link bandwidth utilization between the Nexus 3K nodes and the following devices:
• ASA
• 6248 Fabric Interconnects
A vPC provides the following benefits:
• Allows a single device to use a port channel across two upstream devices
• Eliminates Spanning Tree Protocol (STP) blocked ports
• Provides a loop-free topology
• Uses all available uplink bandwidth
• Provides fast convergence if either the link or a switch fails
• Provides link-level resiliency
• Assures high availability
Details for the vPC feature and configuration can be found in "Configuring Virtual Port Channels" in the Cisco Nexus 3548 Switch NX-OS Interfaces Configuration Guide, Release 6.x at the following URL:
• http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3548/sw/interfaces/602_a1_1/b_N3548_Interfaces_Config_602_A1_1/b_N3548_Interfaces_Config_602_A1_1_chapter_0110.html#concept_090B6DECCB594BC7AE8368433FFB3B3B
ASA Firewall (Active)
ASA Firewall ( Standby)
3524 Switch PairvPC Enabled
Dual Fabric InterconnectsPort channel Enabled Uplinks
WAN
ASA Firewalls Port channels Enabled Uplinks
37
65
26
3-19Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
ASA Redundancy
Two ASA firewalls in the consolidated architecture provide the policy point for all communications in the Control Center. Layer 3 interfaces, as previously mentioned, are enabled on the firewall and act as the Layer 3 gateway to police traffic between all of the zones in the Control Center and between the Control Center and the WAN. The firewalls are deployed as an active/standby pair.
Summary
• Link and platform redundancy should exist at all levels
• Link aggregation and resiliency should be enabled with port channels and vPC
• Firewalls should be in active/standby mode
Compute RedundancyThis section includes information about Fabric redundancy, end host node, and SCADA server connectivity and redundancy.
Fabric Redundancy
A fully redundant Cisco Unified Computing System consists of two independent fabric planes: Fabric A and Fabric B. Each plane consists of a central Fabric Interconnect (Cisco UCS 6200 Series Fabric Interconnects) connected to an I/O module (Fabric Extender) in each blade chassis. The two Fabric Interconnects are completely independent from the perspective of the data plane
All network endpoints, such as HBAs, vNICs, and management entities such as Cisco Integrated Management Controllers (IMCs; formerly referred to as baseboard management controllers, or BMCs), are dual-connected to both fabric planes. See Figure 3-13.
Figure 3-13 Fabric Redundancy
37
65
27
Nexus 3K Ethernet
Switches
Ethernet uplinks
Fabric Interconnect
A
Fabric Interconnect
B
Unified Ports 8x10g
per Fabric
Adapter Adapter
vHBA
vNIC
vNIC
vNIC
B200 Server B200 Server
Fabric Extender Fabric Extender
3-20Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
End Host Mode
In end-host mode, Cisco UCS presents an end host to an external Ethernet network. The external LAN sees the Cisco UCS Fabric Interconnect as an end host with multiple adapters. The uplink ports appear as server ports. This mode is the default and recommended configuration if the upstream device is Layer 2 switching.
End-host mode features include the following:
• STP is not run on both the uplink ports and the server ports.
• MAC address learning occurs only on the server ports; MAC address movement is fully supported.
• Links are active/active regardless of the number of uplink switches.
• The system is highly scalable because the control plane is not occupied.
More details related to fabric redundancy and end host mode can be found in Cisco Unified Computing System Ethernet Switching Modes at the following URL:
• http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/whitepaper_c11-701962.html
SCADA Server Connectivity and Redundancy (Virtual Machines)
Within the production environment, pairs of B series servers, one in each of two chassis, will exist. Each of these servers will be running duplicate Virtual Machines for each service: for example, two Virtual Machines for the real-time service and two Virtual Machines for the historical. These Virtual Machine pairs will be configured to run on separate physical servers that will be located on separate UCS Chassis.
These two virtual servers will be configured with a single vNIC, which is not the norm for this application. Dual NICs and NIC teaming are normally applied. This is not required in this virtual environment where the Hyper-V virtual switch is deployed. The virtual switch will provide a single vNIC to each virtual server. The vSwitch will have redundant connectivity to the UCS Fabric Interconnects through the use of Fabric Failover.
From an addressing standpoint, each server will be configured with an IP within the same address space, but will communicate with the other SCADA components through its virtual IP. Figure 3-14 gives a representation of the design.
Figure 3-14 SCADA Server Connectivity and Redundancy Design
Virtual
IP
10.1.1.1
10.1.1.2 10.1.1.3
VM VM
Hyper-V
Vswitch
Hyper-V
Vswitch
Nexus 3000
6248 Fabric
Interconnects
2 x B200 servers in
Separate Chassis
Redundant VMs
37
65
28
3-21Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
Storage RedundancyThe storage redundancy follows the same consistent approach to the other areas of availability within the architecture consisting of hardware and path redundancy. See Figure 3-15.
Figure 3-15 Storage Redundancy Design
Hardware Redundancy
Redundancy is carried throughout the architecture at the hardware level. We will take the example of a production environment in Figure 3-15 to explain the hardware redundancy modeled.
Server Redundancy
Two physical UCS B200 servers will house the hot and standby instances of the production application Virtual Machines. A hot instance will exist in one physical server and its standby instance will exist in a separate physicals server. These two servers are located in different UCS Chassis, as depicted in the figure, promoting chassis redundancy. Both servers are equipped with dual-port adapters.
UCS Fabric Interconnects
Two 6200 series Fabric Interconnects exist in the architecture. These provide the management and communications for the UCS B series and 5108 UCS chassis. The pair of 6200 Fabric Interconnects form a single, highly available management domain. The pair of fabrics interface with the SAN and the Ethernet network and provide active/active access to both the storage and Ethernet networks while acting as an active passive mode for the management of the UCS system.
Storage Chassis and Controllers
Two storage chassis will exist in the architecture. These will be equipped with dual controllers to provide redundancy within a chassis.
37
65
29
Fiber Channel
Unified Fabric FI
Connection
Ethernet
UCS B series Physical Server
Production Domain Active
• VM Real Time SCADA
• VM Historical Cluster• VM Logging, Eng Server
• VM Domain Control
• VM LDS
• VM RAS
UCS B series Physical Server
Production Domain Standby
• VM Real Time SCADA
• VM Historical Cluster• VM Logging, Eng Server
• VM Domain Control
• VM LDS
• VM RAS
UCS B series Physical Server
Decision Support System Active
• VM Real Time SCADA
• VM Historical Cluster• Vm Domain Control
• VM RAS
UCS B series Physical Server
Decision Support Standby
• VM Real Time SCADA
• VM Historical Cluster• VM Domain Control
• VM RAS
Boot Storage for
Standby Servers
Boot Storage for
Active Servers
SAN A FC Link x1
to each FI-A
SAN B FC Link to
FI-B
x2
x2
x2
x2
Production/Historical
SQL DBs
Production
DomainDecision
Support Sys
DSS Historical SQL
DBs
x8
x8 x8
x8
3-22Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
Redundant SAN
Within storage networking, it is a standard best practice to provide two completely separate SAN fabrics, providing two distinct paths between the compute and the storage. Within this design, a minimum of two VSANs will be created on the UCS system that map to the separate fabrics. Each of dual fabrics in this architecture are directly attached to the storage arrays. One fabric will provide connectivity for Fabric A and one for Fabric B. VSANs, as described earlier, extend SAN networking concepts to create logical segmentation on the same physical infrastructure. Figure 3-16 highlights FC port channels. This is the same concept as in Ethernet port channels where multiple physical links are bundles together to provide one logical link.
It is worth noting that both Fabrics and SANs are active and passing traffic between the hosts and storage. This is an active/active, not active/standby, model.
Figure 3-16 Redundant SAN Design
Multipath
Multipath software is loaded on top of an operating system and provides a layer that sits between the operating system and storage adapters. The multipath software closely monitors all physical paths and storage presentations while load-balancing SAN traffic across the available HBAs and fabric paths. This software layer also accomplishes failover should an HBA, path, or upstream device fail.
Summary
• UCS provides storage redundancy starting at the server with dual port adapters per host.
• Two physical servers house the hot and standby instances of the production application Virtual Machines. A hot instance in one physical server and its standby instance will occur in a separate physical server.
• Each server in a server pair is housed in a separate UCS Chassis.
• Connectivity will occur through a redundant fabric to a pair of Fabric Interconnects.
• Two storage chassis will be directly connected to the Fabric Interconnects using a redundant SAN A/B architecture.
• The guest operating systems and storage are pinned to a specific host to ensure physical redundancy and dedicated server resources.
37
65
30
SAN A FC Link x2
FC Port Channel
SAN A FC Link x2
FC Port Channel
SAN B FC Link x2
FC Port Channel
SAN B FC Link x2
FC Port Channel
Fabric A Fabric B
8x 10g 8x 10g
8x 10g8x 10g
Unified Fabric F1
Connection’s
Unified Fabric F1
Connection’s
3-23Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
• A RAID 1 hardware mirroring solution is used to mirror all historical databases onto online disks available to the SCADA system.
• All historical databases on disks are mirrored.
• Two-node clustering is configured for the historical servers' redundancy.
Schneider Electric OASyS Application Design Application RedundancyThis section includes information about Virtual Machine redundancy, SQL SAN storage hardware mirroring, virtual guest storage, Microsoft SQL server-based Historian, and operator workstation connectivity.
Virtual Machine Redundancy
As the required redundancy mechanisms are built into the OASyS software and applications, no requirement exists to configure Virtual Machine redundancy such as Hype-V live migration in the Schneider Electric and Cisco Control Center architecture. The OASyS applications are provided in state machine-aware pairs and replicate required memory segments between the virtual guests. System monitors check critical components for failures and take the least intrusive course of action to recover from any failure
SQL SAN Storage Hardware Mirroring
A RAID 1 hardware mirroring solution is used to mirror all historical databases onto online disks available to the SCADA system. All historical databases on disks are mirrored, so that a failure of a disk does not cause a failure of a particular historical database. When a disk fails, the mirrored disk has the same copy of data as the master disk. This disk makes available historical data for the failed disk to the SCADA system. Once the failed disk is replaced, the SAN software automatically copies the information from the mirror disk back onto the new disk.
Virtual Guest Storage
Similar to historical data storage, the Virtual Hosts and Virtual Guests are stored on a storage array on the SAN. A RAID 5 set of disks is used to boot the host operating system and another RAID 5 group of disks is used to boot the guest operating systems.
Microsoft SQL Server-Based Historian
As the SCADA system historian is not critical to safe operations, a Microsoft Hot/Standby Cluster is used to share SQL-based data between the two OASyS Historical servers at one site. Microsoft SQL Server Cluster provides both automatic and manual failover capability for SQL Server services to another node in case the active node is down. The active node can be down due to an operating system or a hardware failure, in which case the automatic failover to an available node on the same cluster can occur and the users will start using the application through the failover node. SQL Server services can be manually moved to a different node at times when a planned maintenance like an operating system upgrade or patch maintenance is required on the active node. Figure 3-17 illustrates the combined architecture of the OASyS Historian and Microsoft SQL Cluster.
3-24Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
Figure 3-17 OASyS Historian and Microsoft SQL Cluster Combined Architecture
Operator Workstation Connectivity
The operator workstations are connected within the SCADA zone. These stations will be configured with dual NICs and connected to traditional Catalyst 3850 stackable switches. Generally, the workstations will be in a different physical location so it will be required to extend the VLAN from the 3524 Nexus switches to the Catalyst. The NICs on these workstations will operate in an active/standby configuration and each NIC will be connected to a different stackable 3850. See Figure 3-18.
Figure 3-18 Operator Workstation Connectivity
Network ManagementManagement and diagnostics involves tools, applications, and devices used to monitor, configure and maintain the Control Center architecture. Although a typical pipeline network does not drastically change after deployment, the network needs to be maintained and managed. The business practices and levels of expertise, however, may dictate the management practice, roles, and responsibilities of the system. This section describes the platforms, technologies, and practices used.
Cisco UCS Manager
Cisco UCS Manager resides as embedded software on the Cisco UCS Fabric Interconnects, Fabric Extenders, servers, and adapters. No external management server is required, thereby simplifying administration and reducing capital expenses for the management environment. The communication between the manager on the Fabric Interconnect and the subsidiary functions found in the Fabric Extenders, chassis, servers, and adapters are built in and automatic.
37
65
31
Virtual IP Address
Operator Workstation
ACTIVE NODE
Operating
System Drive
Operating
System Drive
PASSIVE NODE
HBA (Host Bus Adapter) HBA (Host Bus Adapter)
SAN (Storage Area Network)
Data Files Log Files Quorum Drives
37
65
32
3-25Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
UCS Manager provides an intuitive GUI, a CLI, and a robust API to manage all system configuration and operations. Cisco UCS Manager enables storage, networking, and server maintenance staff to collaborate easily on defining service profiles for applications while preserving subject matter expertise and roles. Key features include:
• Supports Cisco UCS B-Series blade and C-Series rack servers, UCS Mini, Cisco Composable Infrastructure and Cisco HyperFlex hyperconverged infrastructure
• Unified, embedded, policy-driven management programmatically controls server, network, and storage resources, so that they can be efficiently managed at scale through software
• Works with HTML 5, Java, or CLI GUIs
• Auto Discovery can detect, inventory, manage, and provision system components that are added or changed
• An open XML API facilitates integration with third-party systems management tools
• Role-based administration builds on existing skills and supports collaboration across disciplines. For more information, see “Cisco UCS Manager” at the following URL:
– http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html
Cisco Adaptive Security Device Manager
Cisco Adaptive Security Device Manager (ASDM) can be accessed directly with a Web browser from any Java plug-in-enabled computer on the network, providing security administrators with rapid and secure access to their Cisco ASA Firewalls. ASDM allows the user to configure, monitor, and troubleshoot Cisco firewall appliances with a user-friendly application. Ideal for small or simple deployments, the Cisco ASDM provides the following:
• Setup wizards for configuring and managing Cisco firewall devices
• Powerful real-time log viewer and monitoring dashboards that provide an at-a-glance view of firewall appliance status and health
• Troubleshooting features and powerful debugging tools such as Packet Trace and Packet Capture
Out of Band Management
A separate Out of Band (OOB) network should be deployed to provide a dedicated infrastructure to support the management traffic. The OOB network segment hosts console servers, network management stations, AAA servers, analysis and correlation tools, NTP, FTP, Syslog servers, network compliance management, and any other management and control services. An OOB management network should be deployed using the following best practices:
• Provide network isolation
• Enforce access control
• Prevent data traffic from transiting the management network
• Enforce secure use of network management traffic (SSH, SNMP v3)
If an OOB network is not viable, then a dedicated VLAN should be used for the management network following the same best practices above and classifying/prioritizing management traffic using QoS.
3-26Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter 3 Connected Pipeline System Design Availability
Role-Based Access Control
Role-Based Access Control (RBAC) is a method of restricting or authorizing system access for users based on a user's role. This role consists of privileges that will be assigned to the user. These privileges can be very granular; for example, a user can be restricted from accessing particular commands and strings. This becomes very useful in the Control Center environment, especially in the UCS domain. With the consolidated nature of the UCS system, varying users with differing expertise can access the platform. Using RBAC, we can create server, storage, and network administration privileges and roles. The UCS system can even take this to the next level using Locales. For example, a user with the Server Administrator role for the Test and Development environment could update server configurations in that environment, but could not update server configurations in the Production environment unless the locales assigned to the user include the access privilege.
Summary
• OOB Management should be used where possible. Dedicated VLANs should be used if In-band Management is used.
• Use secure methods to provide extra security for network management traffic. SSH and SNMP v3 are examples.
• Provide visibility of events, faults, and performance of the network to the operators in the Control Center using Syslog and SNMP.
• Enable RBAC for access to the infrastructure components.
Time SynchronizationAll components of the SCADA system architecture must be time synchronized. Time synchronization is critical for event correlation and analysis of events in the pipeline network. Therefore, all infrastructure components must be enabled with Network Time Protocol (NTP) and configured to a central source. A Network Time Server (NTS) is leveraged for Windows time synchronization at each site (for example, one at the Primary Control Center and one at the Secondary Control Center). The NTS can serve accurate time to any system running an NTP or Simple Network Time Protocol (SNTP) client and can support 200,000 network clients (such as workstations, servers, and routers) with an NTP timestamp accuracy of <10 microseconds. Each time the client will require access to the IP address of the network time server.
The NTS derives accurate time from the atomic clocks aboard the GPS satellite system. To obtain the time, it uses the Code Division Multiple Access (CDMA) mobile telecommunications network used by many cellular telephones. This means the antenna can be conveniently located anywhere a cell phone signal is available.
3-27Control Center Virtualization for the Connected Pipeline
Design Guide
ContDesign Guide
C H A P T E R 4
System Implementation ConsiderationsThis chapter includes the following major topics:
• Server Sizing and Storage Sizing, page 4-1
• Scalability Considerations, page 4-1
Server Sizing and Storage SizingTraditionally, SCADA Host software has been the mechanism for viewing graphical displays, alarms, and trends. Control from the SCADA host itself only became available when control elements for remote instruments were developed. These systems, which were isolated from the outside world and were the domain of operators, technicians and engineers, had the responsibility to monitor, maintain, and engineer processes and SCADA elements. With IT advancements, this is no longer the case. Many different stakeholders now take advantage of real-time access to the data that the SCADA host software generates.
Accounting, maintenance management, and material purchasing requirements are performed or partly performed from data derived from the SCADA system. Consequently, the SCADA host is driven to be an enterprise entity providing data to a number of different users and processes. This has encouraged SCADA host software development to adopt standards and mechanisms to support interfacing to these systems. Key to sizing of a SCADA system is the overall quantity of pipeline network assets to be controlled.
• Servers and storage should be considered
• Number of devices, points polled per device, polling interval, and the applications on top of SCADA determine the CPU and memory requirements
• Number of physical domain controllers, virtual hosts, virtual guests, virtual guest backups, and built-in expansion need sizing to determine hard drive requirements
• Microsoft SQL storage requirements for online data and backups
• Memory sizing for performance of all virtual guests
• Number of remote connected users determines resources allocated to the remote access servers
Scalability ConsiderationsInformation retrieval and communications over the network WAN for the SCADA system should not be a major concern; however, how the traffic traverses the Control Center network should be understood. The Primary Production system doesn't just facilitate communications with the pipeline network, but, as
4-1rol Center Virtualization for the Connected Pipeline
Chapter 4 System Implementation Considerations Scalability Considerations
described, also requires communications with the enterprise and the Secondary Production system. The communications with the backup Control Center, from a bandwidth utilization perspective, can be very different based on the applications and the type of pipeline. For example, 20Mb/s on a gas pipeline is contrasted with 100 to 200 Mbps for replication on a LNG pipeline driven mainly by software applications running on top of OASyS DNA such as the Liquids Management System (LMS). The ASA as the security device is in line of these flows of data. Therefore, the following should be considered:
• ASA sizing
• ASA throughput degrades when enabling IPS/IDS and other computation-intensive features
• The traffic traversing the different points in the network
• Gas and liquid pipelines will have differing replication between the Control Center based on the applications
• The customers' traffic profile
• Number of devices, points polled per device, polling interval, and the applications on top of SCADA may drive replication data between Primary and Secondary Sites
• Data replicated to the IDMZ from the Operational environment
To obtain guidance to ASA firewall scalability when comparing models, see the “Compare Models” chapter in Cisco ASA 5500-X Series Firewalls at the following URL:
• http://www.cisco.com/c/en/us/products/security/asa-5500-series-next-generation-firewalls/models-comparison.html
4-2Control Center Virtualization for the Connected Pipeline
Design Guide
ContDesign Guide
C H A P T E R 5
System ComponentsTable 5-1 details Cisco products used in the Cisco Connected Pipeline system.
Table 5-2 details Schneider Electric products used in the Cisco Connected Pipeline system.
Table 5-1 Cisco Products
Cisco Product Software Release Description
ASA 55x5-X 9.2(3)4 Firewall
Nexus 3524 6.0(2)A4(5) Network Switch
Cisco UCS 2.2(5a) Unified Compute Infrastructure, Fabric Interconnects
B200 M4 2.2(5a) Server
ASR 902 15.5(2)S WAN-facing router
Table 5-2 Schneider Electric Products
Component Role SW Version
Windows Server 2012 R2
MS SQL Server 2012 SP1
Visual Studio 2013 Professional
OASyS DNA Elk SP4 ML 7.7.1
OASyS DNA OGP
• LMS: R4.1.1
• Measurement: R5.6,
• Realtime Gas: R5.2
• Gas Day Operations: R5.4
• OGX: CR2
• Liquids clients:
– LibAPI.Installer.1.0.19
– WebClientInstaller2013.1.0.5
– Liquid.Installer.NET45-1.0.28
7.6
EMC VNXe3200 Storage 3.1.1.5395470
5-1rol Center Virtualization for the Connected Pipeline
ContDesign Guide
C H A P T E R A
GlossaryTable A-1 lists acronyms and initialisms used in this document.
Table A-1 Acronyms and Initialisms
Term Definition
AAA Authentication, Authorization, and Accounting
ACL Access Control List
ARP Address Resolution Protocol
ASA Cisco Adaptive Security Appliance
ASDM Cisco Adaptive Security Device Manager
BLISS Baseline Integrated SCADA System
CDMA Code Division Multiple Access
CoPP Control Plane Policing
COTS Consumer off the Shelf
CVD Cisco Validated Design
DAI Dynamic ARP Inspection
DHCP Dynamic Host Configuration Protocol
DNA Dynamic Network of Applications
DSS Decision Support System
E&P Exploration and Production
EFM Ethernet in the First Mile
EPLMS Schneider Electric's Enterprise PipeLine Management System
FCAPS Fault, Configuration, Accounting, Performance and Security
FTP File Transfer Protocol
HBA Host Bus Adaptor
HMI Human Machine Interface
IAC Identification, Authentication & Control
ICS Industrial Control System
IDMZ Industrial Demilitarized Zone
A-1rol Center Virtualization for the Connected Pipeline
Chapter A Glossary
IDS Intrusion Detection System
IEC International Electrotechnical Commission
IED Intelligent Electronic Device
IMC Cisco Integrated Management Controller
IoT Internet of Things
IPS Intrusion Protection System
ISA International Society of Automation
LLD Low Level Design
LNG Liquid Natural Gas
LUN Logical Unit Number
NFTS New Technology File System
NGL Natural Gas Liquid
NIST National Institutes of Standards and Technology
NTS Network Time Server
OOB Out of Band
OTN Operational Telecom Network
PAGA Public Address and General Alarm
PCD Process Control Domain
PCN Process Control Network
PHMSA Pipeline and Hazardous Materials Safety Administration
PIG Pipeline Inspection Gauge
PLC Programmable Logic Controller
PMS Pipeline Management System
QoS Quality of Service
RAID Redundant Array of Independent Disks
RBAC Role-Based Access and Control
RCS Schneider Electric Remote Client Service
RDF Restricted Data Flow
RDP Remote Desktop Protocol
RTU Remote Terminal Unit
SAN Storage Area Network
SCADA Supervisory Control and Data Acquisition
SLA Service Level Agreements
SNTP Simple Network Time Protocol
SPOF Single Point of Failure
STP Spanning Tree Protocol
Table A-1 Acronyms and Initialisms (continued)
Term Definition
A-2Control Center Virtualization for the Connected Pipeline
Design Guide
Chapter A Glossary
TACACS+ Terminal Access Controller Access-Control System Plus
TCO Total Cost of Ownership
TRE Timely Response to Events
VHD Virtual Hard Disk
VIC Virtual Interface Card
VLAN Virtual Local Area Network
vNIC Virtual Network Interface Card
VoIP Voice over IP
vPC Virtual Port Channel
VSAN Virtual Storage Area Network
WAN Wide Area Network
Table A-1 Acronyms and Initialisms (continued)
Term Definition
A-3Control Center Virtualization for the Connected Pipeline
Design Guide