mike robinson cisco
DESCRIPTION
MPLS - WAN - UtilitiesTRANSCRIPT
MPLS Architectural Considerations
Thinking Differently About Utility WANS Michael Robinson Director - Connected Energy Services Cisco Systems
2 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Agenda 1. Our Situation � Drivers � Existing Networks � Likely Choices
2. Key Arguments for MPLS � Virtualization � Performance � Security
3. MPLS Architecture Variations � Public vs Private MPLS � IP/MPLS vs MPLS-TP � MPLS extensiveness (how far?)
3 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Drivers that Wide Area Networks Must Address: How will we respond? ;�:
☐ Changing Client (Operations) Requirements � Advanced Applications – Control, safety, monitoring, analytics, workforce enablement, etc � Increases in reliability, performance, data transmission, security, flexibility � Changes in system/application architecture
☐ Evolving Regulatory Requirements � State objectives for system operation � Critical Infrastructure mandates � Governmental objectives for operations � Adoption of National Frameworks (Interoperability Standards) � Cyber- and Physical Security mandates
☐ Complex Enterprise Requirements � Enterprise Applications - voice, video, collaboration, mobility, security (physical and cyber), customer care,
handheld, workforce management � Inadequate assets - marginal transport capabilities, overlay assets, technology refresh, sparing, evolutionary
changes in the communication system rapidly becoming an operational risk � Operational risks - inability to effectively control the network, administration costs, isolated management
systems, forecast increases in workload
Promote collaboration throughout the organization and enable partnerships
4 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Cascading Effects of an Aging Infrastructure and Architecture: Why they’re Not Ideal
: Inadequate � Multiple physical networks rather than multiple virtual networks � Lacks the means of integrating new systems and requirements � Promotes overbuilding purpose-built networks (spiral effect) � Security risks and cumbersome deployment of trust
boundaries � Not strategic for forthcoming operational applications
: Inefficient � 93% Operational Network is TDM-based. “Channelization”
causes inefficient use of resources and premature exhaust. � Purpose-built networks increase clients’ overall cost structure
(assets & expense)
: Unmanageable � Operational complexity in managing multiple point solutions � Non-modular. One large logical domain. Vulnerable to
catastrophic faults.
Key portions of the WAN infrastructure are insufficient for handling core business initiatives
Efficiency Comparison of Packet vs Channelized Approach
Challenges of Siloed Solutions
5 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Existing Substation
ChannelBank
RS232
FXS
RS232 T1
T1
DACS
DFR
Analog Phone
RTUA
RTUB
Gateway
Considerations: 1. Quantity of devices 2. Technologies 3. Administrative responsibilities 4. Degree of intelligence
6 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Future Substation
Switch
Switch Router
RS232
Ethernet
Ethernet
Ethernet
MPLS
RTU
EthernetSwitches
Gateway
IED
IED
IED
RTU
DistributedCompute
EthernetSwitches
IED Phone
IEDIED
Physical Security
HMI
WiFiAP
AMIToP
FAN Radios
Ethernet
Ethernet
Ethernet
RTU
HMI
IED
Considerations: 1. Quantity of devices 2. Technologies 3. Administrative responsibilities 4. Degree of intelligence
7 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
WAN Technology Considerations
Pros: • (Generic) Implementation and
management simplicity • Low costs • End-to-end pervasiveness • (Almost) De-facto standard • Standards continues to evolve
Cons: • Lacks stringent controls for
substation communications • (Perceived) Security concerns • Legacy protocol challenges • Single network approach
Layer 3 IP Layer 2 Ethernet MPLS
Pros: • (Generic) Implementation and
management simplicity • Low costs • Pervasiveness (almost) • (Almost) De-facto standard
Cons: • Not meant for expansive WANs • Not best QoS and performance
mechanisms • Security concerns • Ethernet only (utilities need more
interfaces and protocols)
Pros: •Superior technology for expansive WANs •Offers TDM-like features while operating as a packet system •Superior QoS, performance and security mechanisms •Retains policy management within the network
Cons: •Relatively ‘new technology’ for most enterprise users •Layer 2.5 protocol that can make security more complex in some environments
Ethernet
Sub
CC
Sub Sub
Virtual SW
MPLS
Sub
CC
Sub Sub
Any SW
Sub
CC
Sub Sub
IP Any SW
B
C A
B C A
?
8 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Agenda 1. Our Situation � Drivers � Existing Networks � Likely Choices
2. Key Arguments for MPLS � Virtualization � Performance � Security
3. MPLS Architecture Variations � Public vs Private MPLS � IP/MPLS vs MPLS-TP � MPLS extensiveness (how far?)
9 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Key Arguments for MPLS-based WAN
�Virtualization �Organizational �Operational �Network Evolution
�Performance
�Security
10 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Virtualization is critical for multiple workgroups to have their own networks and optimize their unique
business operations
Data, Operations, & Control Center Networks
Service Provider Network
Gen
Data, Operations, & Control Center Networks Data, Operations, & Control Center Networks Data, Operations, & Control Center Networks
Service Provider Network Service Provider Network
Service Provider
Metering Network
Trans Dist
Utility’s
Metering Network
Organizational
NW
Zo
ne
SW
Zo
ne
SE
Zo
ne
NE
Zo
ne
Performance & Reliability
Z-1
Z-3
Z-5
Z-7
Z-2
Z-4
Z-6
Z-8
L2-Multicast Scheduling
Operational
Functional
Con
trol Z
one
App
Zo
ne
HM
I Zo
ne
11 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Network Evolution Using MPLS
Frame Relay
Frame Relay
Frame Relay
Channel Bank
Channel Bank
Channel Bank
Router
Router
Router
Switch
Switch
Switch
AToM Any Transport
over MPLS
FRoMPLS TDMoMPLS
L2VPN L3VPN
12 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
MPLS: More Network Performance Controls MPLS TE Fast Re-Route (FRR)
� Subsecond recovery against node/link failures
� Scalable 1:N protection
� Greater protection granularity
� Cost-effective alternative to 1:1 protection
� Bandwidth protection Primary TE LSP
Backup TE LSP
IP/MPLS
R2
R1
R8
13 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
� Massive (44%) packet loss at router B→router E!
The Problem with Shortest-Path
�Changing to A->C->D->E won’t help
Router F
Router C Router D
Router A
Router B
OC-3
OC-3
DS3
DS3
DS3 OC-3
OC-3
� Some links are DS3, some are OC-3
� Router A has 40M of traffic for router F, 40M of traffic for router G
Router E
Router G
Node Next-Hop Cost B B 10 C C 10 D C 20 E B 20 F B 30 G B 30
IP (Mostly) Uses Destination-Based Least-Cost Routing Alternate Path Under Utilized
14 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
How MPLS TE Solves the Problem
� Router A sees all links
� Router A computes paths on properties other than just shortest cost; creation of 2 tunnels
� No link oversubscribed!
Router C Router D
OC-3
OC-3
DS3
DS3
DS3 OC-3
OC-3
Router F
Router C Router D
Router G
Router A
Router B
Router E
Node Next-Hop Cost B B 10 C C 10 D C 20 E B 20 F Tunnel 0 30 G Tunnel 1 30
15 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
MPLS-FRR Link Protection Operation
� Requires next-hop (NHOP) backup tunnel
� Point of Local Repair (PLR) swaps label and pushes backup label
� Backup terminates on Merge Point (MP) where traffic rejoins primary
� Restoration time expected under ~50 ms
Primary TE LSP
Backup TE LSP
IP/MPLS
R1
25 22
16 22
22
R2 R6 R7
R3
R5
16 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
FRR Node Protection Operation � Requires next-next-hop
(NNHOP) backup tunnel
� Point of Local Repair (PLR) swaps next-hop label and pushes backup label
� Backup terminates on Merge Point (MP) where traffic rejoins primary
� Restoration time depends on failure detection time
Primary TE LSP
Backup TE LSP
IP/MPLS
R1
25 36
16 22
36
R2 R6 R7
R3
R4
36
R5
17 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Bandwidth Protection � Backup tunnel with
associated bandwidth capacity
� Backup tunnel may or may not actually signal bandwidth
� PLR will decide best backup to protect primary (nhop/nnhop, backup-bw, class-type, node-protection flag)
Primary TE LSP
Backup TE LSP
IP/MPLS
R1 R2 R6 R7
R3
R4
R5
18 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
1. MPLS (to the substation) can be used to – a. Provide Layer 2 access into the substation b. Isolate Layer 3 networks (L3VPNs) from other traffic (see below) c. Create/support unique security frameworks
2. L3VPN security enablement a. Traffic separated from other VPNs (NERC CIP CCA traffic can be in its own closed user group,
Internet access distribution can be its own closed user group) b. Addressing plans separate from and not apparent to other networks and devices c. Constrains IP-spoofing to the originating VPN d. Routes individualized per VRF (critical traffic wouldn’t appear ‘in the wrong locations’) e. Route updates independent between VRFs (deters against malicious attacks) f. Core topology concealed from users g. Through MPLS TE and due to unique core addressing, the impact of DoS can be minimized* h. DoS can be minimized to the MPLS edge and constrained to the originating VPL i. Route limiting can guard against route flood attacks j. Access Control List (ACL) administration simplification k. Enhanced security policy administration achieved using path design along with strategically placed
security appliances l. Inter-VPN routing controlled by centrally administered policies
Enhancing Security using MPLS
* Note importance of PE security hardening and design. CE peers also require ACLs and MD5 authentication.
19 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
� CIP-002 Requirement 3 directs the Responsible Entity (RE) to develop a list of Critical Cyber Assets (CCA)1. Also provides criteria to help qualify what is a CCA : �R3.1 Cyber Asset uses a routable protocol to communicate outside the Electronic Security Perimeter �R3.2 Cyber Asset uses a routable protocol within a control center �R3.3 Cyber Asset is dial-up accessible.
� Identifying Critical Cyber Assets2 published by NERC in June 2010 to assist identification of CCA as described in CIP-002 R3. �MPLS is specifically listed as a Layer 2 protocol (NOT “routable”) � If devices in the substation are NOT communicating using a routable protocol, and if MPLS is
transporting the information across the WAN, then the CA is NOT considered a CCA
� In compliance with NERC CIP, MPLS networks provide secure, reliable, flexible and cost-effective communication: � Between CA’s and other smart grid elements that use non-routable (Layer 2) communications � Between CCA’s and other smart grid elements that use routable (Layer 3) communications � Security enhanced when MPLS is configured to establish closed user groups (virtual networks)
•IP packets originating from external endpoints, including the Internet, cannot enter these closed user groups, thus preventing many types of external attacks.
NERC CIP Considerations
1. http://www.nerc.com/files/CIP-002-1.pdf 2 http://www.nerc.com/fileUploads/File/Standards/Critcal%20Cyber%20Asset_approved%20by%20CIPCl%20and%20SC%20for%20Posting%20with%20CIP-002-1,%20CIP-002-2,%20CIP-002-3.pdf
20 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
� MPLS-based networks provide secure, reliable, efficient, flexible and cost-effective communication between CCA’s and other smart grid network elements.
� MPLS networks natively support communication with non-routable protocols With the current requirement exemption, Bulk Electric System (BES) endpoints connecting to non-CCA MPLS network (and allowed to be outside the ESP boundary) can communicate over MPLS network with non-routable protocol.
� If/when the non-routable protocol exemption is removed, all communication endpoints providing external access into an ESP, including MPLS endpoints, will be considered CCA (therefore subject to NERC CIP).
� MPLS networks can also be extended for secure communication between CCA’s of the BES and other smart grid network elements in compliance with NERC CIP.
� The traffic isolation capabilities inherent in MPLS provides network-based access control for BES CCA’s and other smart grid network elements.
� Several types of MPLS services can be configured to establish closed user groups.
� IP Packets originating from external endpoints, including the Internet, cannot enter these close user groups, thus preventing many types of external attacks.
NERC CIP Compliant MPLS Design
21 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Agenda 1. Our Situation � Drivers � Existing Networks � Likely Choices
2. Key Arguments for MPLS � Virtualization � Performance � Security
3. MPLS Architecture Variations � Public vs Private MPLS � IP/MPLS vs MPLS-TP � MPLS extensiveness (how far?)
22 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Comparison of Public vs Private MPLS Models Alternative 1 (Public)
� Monthly circuit budget: �Remote site $500 �Hub/large site1 $2000
� Subsequent network modifications drive internal/external costs
� QoS support drives additional costs
� SLA tough to enforce and non-punitive
� All traffic (from diverse business units) carried in single L3 domain!
Alternative 2 (Private) �Monthly circuit budget:
�Remote site $350 �Hub/large site1 $1000
�Subsequent network modifications are internal costs �QoS doesn’t drive additional costs �SLA enforceable (internally) �Traffic easily segmented into isolated L3 domains for each user group.
(1) Note that Hub site figures can also reflect future telecom cost structure for large/high bandwidth endpoints like cell towers and office locations.
23 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Financial Comparisons ; Private MPLS saves enterprise 47% in monthly telco costs
�Conservative estimates indicate annual telco OPEX savings (or minimizing future spend) from SCADA traffic alone can exceed $1.2M �Taking a holistic approach to diverse and forward-looking transport needs at various sites (Function 1,,,,Function 6, etc), it is rational that private MPLS helps cap OPEX growth for the transport department1 �Savings will need to be vetted with the OPEX for private equipment in the core network
; Private MPLS minimizes future moves/adds/changes costs �Evolving business needs will almost certainly drive such activities
; Faster problem identification and resolution enabled by private MPLS helps minimize future telecom department OPEX
; Faster problem identification and restoration on private MPLS helps maintain uptime for enterprise revenue generation
9 Private MPLS minimizes near-term/long-term telecom OPEX
(1) Private MPLS affords new services to be carried over a single T1 access circuit.
24 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
Intrinsic Comparisons ; Private MPLS allows enterprise higher degree of control over end-to-
end service delivery ; Private MPLS offers enterprise business units individual SLAs
�“Pay for what they need” �Telecom processes can be designed around SLAs and KPIs
; Private MPLS supports deterministic traffic �Control traffic and other growing business requirements �T1 and DS3 circuit emulation
; Private MPLS supports multiple Layer 2 services �T1, DS3, Ethernet and switched Ethernet
; Private MPLS utilizes telco transport (circuits) leveraging their workforce/systems at the edge of the network
; Private MPLS can support other enterprise services (Function 1,,,Function 6, etc)
9 Private MPLS enables key service options critical for Enterprise business units
25 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
MPLS-TP
Working LSP
PE PE
Protect LSP
NMS for Network
Management Control *
Client node Client node
MPLS-TP LSP (Static or Dynamic)
Pseudowire
Client Signal
with e2e and segment OAM Section Section
*Or Dynamic Control Plane
Connection Oriented, pre-determined working path and protect path Transport Tunnel 1:1 protection, switching triggered by in-band OAM, Options with NMS for static provisioning, or dynamic control plane for routing and signaling
Note: The cloud represents one MPLS-TP network, e.g., it may be in aggregation or access
26 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.
� Multi-segment pseudowires (MS-PW) enable layer-2/-1 services over a combined MPLS-TP and IP/MPLS infrastructure
� S-PE (switching provided edge router) switches traffic between a static and a dynamic segment
� MPLS-TP domain uses static LSP as PSN tunnel and static PW segment
� IP/MPLS domain uses signaled LSP (LDP or RSVP-TE) as PSN tunnel and signaled PW segment
MPLS-TP MPLS-TP IP/MPLS
Aggregation Access Core Aggregation Access
T-PE S-PE S-PE S-PE
Static PW Static Tunnel
Signaled PW Signaled Tunnel
Static PW Static Tunnel
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
Primary Control / Data Center Access/ Aggregation Network T&D Substations
Core Network T&D Substations
Secondary Control / Data Center
Data Center
Control Center HMI SCADA FEP EMS Historian Analytics
CPAM VSOM ACS CA LDAP
Data Center
Control Center HMI SCADA FEP EMS Historian Analytics
CPAM VSOM ACS CA LDAP
Multiservice MPLS/IP Core
AAA DHCP DNS
OAM Subsystem
Prime Optical
Prime Performance
Prime Provisioning
Prime Network
NOC
MPLS/IP Ring Aggregation
Bridged (REP) Ring Aggregation
MPLS/IP Linear Aggregation
MPLS/IPoDWDM Optical Network Packet / Hybrid Microwave
Fiber
Substation CE
Substation PE
Substation PE
Substation Switch
Substation PE
Substation PE
Substation PE
Substation PE
Control Center PE
Control Center PE
Control Center CE
Control Center CE
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
Thank You