lambda station: on-demand flow based routing for data intensive grid applications over multitopology...
TRANSCRIPT
Lambda Station: On-demand flow based
routing for data intensive GRID applications over
multitopology networks
GridNets 2006, San Jose,CA, October 1 – 2, 2006
Fermi National Accelerator Laboratory Don Petravick (PI), Phil Demar, Matt Crawford
California Institute of Technology. Harvey Newman (co-PI)
by Andrey Bobyshev, Fermilab.
Outline of the talk:
● goals and major directions of the project● software architecture, API, SOAP/XML, some details● results of using Lambda Station service, SC05 Demo ● problems and challenges, plans
Basic terms
● Lambda Station – a host with special software to control traffic path across LAN and WAN on-demand of applications● PBR – policy based routing● PBR Client – a system or cluster and applications running on it sourcing traffic flows that can be subject for policy based routing● Flow - a stream of packets with some attributes in common such as endpoint IP addresses (or range of addresses), protocols, protocol's ports if applicable and differentiated services code point (DSCP).
The main goal of Lambda Station project is to design, develop and deploy a network path selection services to interface production storage and computing facilities with advanced research networks.
– selective forwarding on a per flow basis
– alternate network paths for high impact data movement
– access control in site edge routers for those selected flows
– on-demand from applications (authentication & authorization)
– current implementation based on policy-based routing & including the
support of DSCP marking
The goal of the project... deals with the last-mile problem in local area networks
Lambda Station TestBed
● two HEP sites, Fermilab and
Caltech
● 10G & 1G End-To-End paths
via DOE UltraScienceNet and
NFS UltraLight
● components of test and
production
computing/storage/network
infrastructures
● 10Gb and 1Gb servers
Lambda Station Building Blocks
LSInterfaceLS-Management &Reporting Interface
mySQL:requests, history,security
LSDIScovery Service
LSRESource Scheduler LSController
SiSi
local definitions
online updates
SiSi
SOAP/Clarens
Vendor specificmodules
NETWORK CONFIGURATOR
CISCO Force10 WAN
Storage & application spaceManagement
SOAP/Clarens
LSInterface
Remote lambdastation
Data Exchange
Control & Management
Service-based Architecture:● CLARENS is framework for service based architecture, mutual authentication of requests●LSController – synchronizes work of all services, has control functions ●LSI - unified interface for intercommunication between LS and applications, and LS-to-LS● LSDIScovery service – detects new lambdastations, and PBR clients at remote and local sites● LSRESource Scheduler – estimates bandwidth allocation, monitor real-time usage of resources● NETWORK CONFIGURATOR – dynamic reconfiguring of LAN and WAN
SOAP
Lambda Station Software Multiple API-calls or primitives (~ 20) divided into several groups:● informative (whoiam,sayHello,getClientInfo, getKnownLambdastations....)● service or operational (openSvcTicket,cancelTicket,getTicket,getFlowSpec...)● internal (getNextDSCP, assignDSCP,updateMyLambdaDefs)● single interface for client-to-LS and LS-to-LS communication
Scenarios to configure LANs:● dynamically configurable local networks with dynamically or statically assigned DSCP● without DSCP tagging● statically configured networks with predefined matching criterion for flows
LS Software v1.0 – a fully functional prototype but no interoperability across different platforms
Java implementation of LS Software● Service Oriented Architecture● build by using JClarens framework as a grid-aware tool● messages are defined by XML schemas● core authentication is based on gLite library to support standard Grid proxies and KCA-issued certificates● secure document/literal wrapped SOAP messages, Web Services Interoperability Profile (WS-I Basic Profile Version 1.1)
Multi threaded LS implementation (LS Controller, NetConfig
and LS/PBR Config as JClarens background processors)
Flows and DSCP tagging Any combination of flow's attributes can be used by Lambda Station software to identify flows on per ticket's basis.
Typical steps of alternate paths:● generate request for service (application, application's proxy, admin)● LS negotiates service and parameters with remote site● configure local and wide area network (not yet available)● marking traffic (if specified).
Current LS software is capable to complete all these steps within 3 – 5 mins. That is why it is desirable to know flow selection parameters before transferring is started:● endpoint IP addresses ● DSCP
DSCP Tagging
Complexity of using DSCP tagging:● preservation of DSCP is not guaranteed in WAN● DSCP tagging needs to be synchronize between sites for dynamically configurable networks ( asymmetry is bad for high-performance transferring )
LS software does support two different modes of DSCP tagging :● fixed DSCP values to identify site's traffic. ● DSCP value is assigned dynamically on per ticket base.
Interaction between applications and LSMultiple scenarios how applications and Lambda Station may communicate:● modified applications with LS awareness capability to place and manipulate tickets● w/ and w/o DSCP traffic marking ● unmodified applications, a proxy controls paths for their traffic
Operational modes of openSvcTicket request (subject to authentication, authorization and quotas):
● new ticket - creating a new ticket● join - provides ID and required parameters(DSCP) of already opened ticket● extend ticket – allows to extend already created ticket.
Lambda Station Aware Applications lsiperf, lsTraceRoute:● Full support of Lambda Station awareness● Places a ticket for an alternative path, negotiates and synchronizes with remote server's application the parameters (DSCP tagging with IPTables ) of request● watches ticket's status SRM/dCache:● limited Lambda Station awareness● places ticket, join and extend● recognizes absence of Lambda Station● tries to use it only for specific destinations
1. Data transfer started:– 10GE host; 5 tcp streams
– Network path is via ESnet
– OC12 bottleneck…
– Path MTU is 1500B
– LambdaStation service ticket is
opened
2. LambdaStation changes
network path to USN
3. Host path MTUD check
detects a larger path MTU
4. LambdaStation service ticket
expires:– Network path changed back to
ESnet
LSiperf End-to-End Test
1
2 3
4
Flows based switching demo at SC05
Project accomplishments
● Software version 1 (a fully functional prototype supporting whole cycle of LS functionality)● positive results of testing between Fermilab and Caltech● positive results during SC05 via general Internet and SCinet (10G path)● lsiperf, lsTraceroute – wrappers around well known applications to add Lambda Station awareness● SRM/dCache integration – testing, added in production 1.7.0 release
Problems and challenges● Traffic Asymmetry is bad for high performance applications● Network (Lambda Station) awareness is too complex● Definition of PBR Clients is a complex issue, auto definition is not yet available
References: http://www.lambdastation.org/
Plans● release jLS, interoperability across platforms● SRM/dCache with production quality LS support● add real-time monitoring of resources● add WAN control plane module
END
Miscellaneous Slides
● Simulation of multiple Lss● screen shoots of ticket's queue● SC05 Demo
● Single web services ServiceTicket request interface for client2LS and LS2LS message flows
● Multi threaded LS implementation:
– LS Controller, NetConfig and LS/PBR Config as JClarens background
processors
● Apache XmlBeans API for XML data binding
● JClarens utilized to provide User Session management, authorization,
authentication and transparent DB connections pooling
● Interoperability tests with perl, python clients
LS Java API
Netconfig Module● dynamically modifies the configurations of local network devices. ● a vendor dependent components. ● Cisco routers with IOS version supporting sequencing type of named ACLs.
Tasks to configure PBR in Cisco devices:● interface on which PBR is applied needs to be configured with “ip policy route-map” statement● route map needs to be configured as ordered list of match/action statements● match criteria need to be associated with ACLs
openSvcTicket request
● new ticket - creating a new ticket● join - provides ID and required parameters(DSCP) of already opened ticket● extend ticket – allows to extend already created ticket.
Accepts many input parameters, most of them could also be determined automatically:● Dst site's identifiers● Src/Dst PBRClient identifier or/and list of Src/Dst IP in (CIDR)● IP protocols, protocol's ports● localPath, remotePath identifiers● BWout,BWin● boardTime, startTime, endTime
Returns ID of locally assigned ticket
Operational modes (subject authentication, authorization and quoting):
Directions of the project.
● building a wide-are testbed infrastructure● designing, developing Lambda Station software, Lambda Station (network) aware applications● researching effects of flow based switching on applications
Application's behavior in flow based
switching environment
Tuning of end systems for maximum rates is not a subject of the Lambda Station project, however, we need to see an increase of data movement performance when selectively switch flows
● DSCP tagging with IPtables ● switching between two paths with different MTUs
Effect of DSCP tagging with IPTables
Tomcat + Axis container
JClarens
LS Service Request
Interface
LS PBR clients
Interface
LS Controller
Client API
LS NetConfig
LS/PBR Config Manager
LS JAVA API Components
Motivation of the project
● unprecedented demands for data movement in physics experiments such as CDF, D0, BarBar and coming LHC experiments ● massive, globally distributed datasets growing to the 100 petabytes by 2010● collaborative data analysis by global communities of thousands of scientists ● available data communication technology will not be able to satisfy these demands simply by plain increasing bandwidth in LANs and WANs due to technology limitations and high deployment and operation costs.● Advance Research optical networks – greater capacity, no production quality of service, are not universally available for all pairs of communicating endpoints
LS multitopology network model
PBR-client B
Multiple Network Toplogies
Red
Green
BlueAdmission Group of network devices
RT1 RT2 RT3
RT1 RT2 RT3
PBR-client A
H1 H2 H3
NG-A
NG-C
NG-B
NG-ADM
H1 H2 H3
PBR-client
PBR-clients or regular clientsat the remote sites
ClientA rules for Red& GREEN topologies:
RT1 RT2 RT3
BLUE is ProductionPath
RED-ClientA-INGREEN-ClientB-INRED-ClientB-INGREEN-ClientB-IN
RED-B-IN
GREEN-OUT RED-OUT
Client A, RED & GREEN rules for NGC
Cisco IOS, dynamic configuring of PBR, extended sequencing ACLs + access policy ACLs
lambdastation@FNALlambdastation@SC05
Fermilab SC05/Seattle
nws-lab.fnal.gov
charley.fnal.gov
NAA-2-NAA
protocol
netconfig
LS-2-LS
protocol
A122.302.sc05.org
CommodityInternet/SCinet
A126.302.sc05.org
lsiperf
srmcp
ls-request
reply
default path
alternative path forspecific flows only
SC05/HighSpeedLinks
LambdaStation SC05 Demo
Note A
Note A: We believe it is a HW/ASIC problem with SNMP monitoring, a time to time SNMP -get returns the same counters as in previous cycle.
PMTUD