TeraPathsTeraPaths: Establishing End-to-End QoS Paths through L2 and L3
WAN Connections
Presented byPresented by
Dimitrios Katramatos, BNLDimitrios Katramatos, BNL
2
Outline
The TeraPaths projectThe TeraPaths project Motivation
Concept and implementation
View of the world (network)
Interoperating with WAN servicesInteroperating with WAN services
L2 vs. L3L2 vs. L3
What is required from the site LAN?What is required from the site LAN?
Status/futureStatus/future
3
Motivation
The problem: The problem: support efficient/reliable/predictable peta-scale support efficient/reliable/predictable peta-scale
data movement in modern high-speed networksdata movement in modern high-speed networks Capacity is not limitless
Multiple data flows with varying priority
Default “best effort” network behavior can cause performance and
service disruption problems
Solution:Solution: enhance network functionality with QoS features to enhance network functionality with QoS features to
allow prioritization and protection of data flowsallow prioritization and protection of data flows Classify traffic
Schedule network usage
4
Prioritized vs. Best Effort Traffic
5
TheTeraPathsTeraPaths Service: Reserve End-to-End Paths with Guaranteed Bandwidth
WAN
WAN web services
TeraPaths
TeraPaths1
2
3
6
Data Flow Information
Owner info (user)Owner info (user)
Data flow IDData flow ID Source IP and port Destination IP and port IPs and ports can be ranges (multiple flows) Direction (unidirectional/bidirectional) Protocol
Bandwidth (class of service)Bandwidth (class of service) Multiple flows will share (best effort within the class)
Start time and durationStart time and duration Minute resolution
7
Path Setup
Participating end site subnets are controlled by TeraPaths software Participating end site subnets are controlled by TeraPaths software
instances (TeraPaths Domain Controllers or TDCs)instances (TeraPaths Domain Controllers or TDCs) TDCs configure end site LANs to prioritize authorized flows via the DiffServ
framework at the network device level
Source site polices/marks authorized flow packets
Destination site admits/re-polices/re-marks packets
End site LANs hand over/receive marked packets to/from the WAN
WAN provides MPLS tunnels or dynamic circuitsWAN provides MPLS tunnels or dynamic circuits Initiating TDC requests MPLS tunnel or dynamic circuit with matching
bandwidth and lifetime, or…
TDC funnels several flows into MPLS tunnel or dynamic circuit with
aggregate bandwidth and lifetime
WAN preserves packet markings
8
Path Setup (ii)
WAN domains must interoperateWAN domains must interoperate Each end site’s TDC has a single point of contact for WAN services
TDCs have no knowledge of WAN internals other than what is
exposed by the WAN services End sites have no direct control over the WAN
Either tunnel or circuit through WANEither tunnel or circuit through WAN Cannot mix and match
9
Conceptual View of the Network
TeraPaths
TeraPaths
TeraPaths
TeraPaths
Site A
Site B
Site C
Site D
WAN 1
WAN 2
WAN 3
service invocation
data flow
peering
WAN chain
10
TeraPaths Testbed ( )
current
US ATLAS T2 sites
11
TeraPathsTeraPaths Web Services Architecture
Internal Services
Public Services
Web Interface
Admin Module
NDCNDCNDC • • •
Database
protected network
API
remote
local
WAN Services
• • •
WAN Services
• •
•
proxy
proxy
• •
•
CLI s/w client
12
Interoperating with WAN Services
TeraPaths “proxy” serversTeraPaths “proxy” servers Implement interface required by TeraPaths core
Hide WAN service differences
Clients to WAN web services (OSCARS and DRAGON) Close cooperation with ESnet and I2 development teams
Submit reservations for MPLS tunnels or dynamic circuits
Handle security requirements
Handle errors
MPLS tunnels vs. dynamic circuitsMPLS tunnels vs. dynamic circuits Utilization requires drastically different approach
13
L2 vs. L3 (i)
MPLS tunnel starts and ends within WAN domainMPLS tunnel starts and ends within WAN domain Packets are admitted into the tunnel based on flow ID information
(IPsrc, portsrc, IPdst, portdst)
WAN admission performed at the first router of the tunnel (ingress)
WANborder routerborder router
MPLS tunnel ingress/egress
router
MPLS tunnel ingress/egress
router
14
L2 vs. L3 (ii)
Dynamic circuit appears as VLAN connecting end site Dynamic circuit appears as VLAN connecting end site
border routers with single hopborder routers with single hop Cannot use flow ID data directly
Flow must be directed to the proper VLAN
WAN admission performed within end site LAN
Select VLAN with Policy Based Routing (PBR)
WANswitch switch
border routerborder router
15
Site LAN Setup (DiffServ)
16
Site LAN Setup (DiffServ w/pass-thru)
17
3rd Party WAN Segments
Some WAN segments may not be Some WAN segments may not be
automatically configurableautomatically configurable
Static configuration allows DSCP Static configuration allows DSCP
bits to go throughbits to go through Only allow specific interfaces
ACLs and aggregate policers
18
L2-Specific Issues
Limitations with VLANsLimitations with VLANs Tag range - tentatively selected 3550-3599 (50 VLANs)
Tag conflicts - eliminate by synchronizing site databases
Scalability problemsScalability problems Flow grouping
Logistics
PBR overhead Virtual border router
Sensitive/3Sensitive/3rdrd party network segments party network segments VLAN pass-thru
19
Additional Setup for L2
20
Summary
TeraPaths stitches together virtual paths with guaranteed TeraPaths stitches together virtual paths with guaranteed bandwidth…bandwidth… through end-site LANs (direct control)… and end-site interconnecting WANs (indirectly, automatically)… from end host to end host
TeraPaths…TeraPaths… utilizes DiffServ for LAN QoS… makes arrangements for WAN MPLS tunnels or dynamic circuits by
interfacing with WAN (web) services… schedules bandwidth usage with advance reservations… utilizes “pass-thru” techniques for sensitive or 3rd party network
segments
21
Status and Future
Currently: basic software ready, infrastructure testedCurrently: basic software ready, infrastructure tested API and web interface, simple negotiation Statically allocated bandwidth classes L3 paths (MPLS tunnels) through ESnet Elementary AAA BNL UMich
In the works, futureIn the works, future Testbed expansion to US ATLAS Tier 2 sites Utilization of L2 paths (dynamic circuits) through ESnet and Internet2 Dynamic bandwidth allocation within service classes CLI, extended API, configurable negotiation Grid-style AAA (GUMS/VOMS) Admin module to facilitate end site LAN setup
http://www.racf.bnl.gov/terapathshttp://www.racf.bnl.gov/terapaths