Server Traffic ManagementServer Traffic Management

Jeff Chase

Duke University, Department of Computer Science

CPS 212: Distributed Information Systems

The Server Selection ProblemThe Server Selection Problem

Which network site?

Which server?

“Contact the weather service.”

server array A server farm B

better old solutionsDNS round robin [Brisco, RFC 1794]WebOS “smart clients” etc. [Vahdat97]

today’s buzzwordscontent-aware traffic managementcontent switching (L4-L7)server switchingweb switching

not-so-great solutionsstatic client bindingmanual selectionHTTP forwarding

Traffic Management for ClustersTraffic Management for Clusters

Today we focus on the role of the network infrastructure in routing requests to servers in a cluster.

Ignore the wide-area problem for now (DNS and other tricks later).

Relatively simple switches can support ACLs to filter traffic to specific TCP or UDP ports from given addresses or subnets.

Current-generation server switches incorporate much richer L4 and content-aware switching features.

How much of the front end support can we build into the network elements while preserving “wire speed” performance?

What request routing policies should server arrays use?

Key point: the Web is “the only thing that matters” commercially.

TCP with HTTP+SSL is established as lingua franca, so more TCP/HTTP/SSL functionality migrates into hardware or firmware.

Traffic Management for ClustersTraffic Management for Clusters

server array

Clients

L4: TCPL7: HTTP

SSLetc.

Goalsserver load balancingfailure detectionaccess control filteringpriorities/QoSexternal VIP managementrequest localitytransparent caching

smart switch

virtual IP addresses

(VIPs)

What to switch/filter on?L3 source IP and/or VIPL4 (TCP) ports etc.L7 URLs and/or cookiesL7 SSL session IDs

L4 Server Load Balancing (SLB)L4 Server Load Balancing (SLB)

server array

load balancer

Issuesswitch redundancymechanics of L4 switchinghandling return trafficserver failure detection (health checks)

Key point: the heavy lifting of server selection happens only on connect request (SYN).

Performance metric: connections per second.

Policiesrandomweighted round robin (WRR)lightest loadleast connections

Limitationsconnection-grainedno request localityno session localityfailover?

Mechanics of L4 SwitchingMechanics of L4 Switching

“Client C at TCP port p1 requests connection to TCPserver at port p2 at VIP address x.”

a b c d

Smart switch:1. recognizes connect request (TCP SYN)2. selects specific server (d) for service at p23. replaces x with d in connect request packet4. remembers connection {(C,p1),(d,p2)}5. for incoming packets from (C,p1) for (x,p2) replace virtual IP address x with d forward to d6. for outgoing packets from (d,p2) for (C,p1) replace d with x forward to Can instance of network address translation (NAT)

x

Handling Return TrafficHandling Return Traffic

server array

Clients slow smart switch

incoming traffic routes to smart switch

smart switch changes MAC address

smart switch leaves dest VIP intact

all servers accept traffic for VIPs

server responds to client IP

dumb switch routes outgoing traffic

simply a matter of configuration

fast dumb switch

examplesIBM eNetwork Dispatcher (host-based)Foundry, Alteon, Arrowpoint, etc.

alternativesTCP handoff (e.g., LARD)

URL SwitchingURL Switching

server array

web switch

IssuesHTTP parsing costURL lengthdelayed bindingserver failuresHTTP 1.1 session localityhybrid SLB and URLpopular objects

Example: Foundryhost nameURL prefixURL suffixSubstring patternURL hashing

Idea: switch parses the HTTP request, retrieves the request URL, and uses the URL to guide server selection.

Advantagesseparate static content from dynamicreduce content duplicationimprove server cache performancecascade switches for more complex policies

a,b,c

d,e,f

g,h,i

The Problem of SessionsThe Problem of Sessions

In some cases it is useful for a given client’s requests to “stick” to a given server for the duration of a session.

This is known as session affinity or session persistence.

• session state may be bound to a specific server• SSL negotiation overhead

One approach: remember {source, VIP, port} and map to the same server.• The mega-proxy problem: what if the client’s requests filter through

a proxy farm? Can we recognize the source?

Alternative: recognize sessions by cookie or SSL session ID.• cookie hashing• cookie switching also allows differentiated QoS

Think “frequent flyer miles”.

LARDLARD

server array

LARD front-end(a,b,c: 1)(d,e,f: 2)(g,h,i: 3)

Idea: route requests based on request URL, to maximize locality at back-end servers.

LARD predates commercial URL switches, and was concurrent with URL-hashing proxy cache arrays (CARP).

Policies

1. LB (locality-based) is URL hashing.

2. LARD is locality-aware SLB: route to target’s site if there is one and it is not “overloaded”, else pick a new site for the target.

3. LARD/R augments LARD with replication for popular objects.

a,b,c

d,e,f

g,h,i

LARD front-end maintains an LRU cache of request targets and their

locations, and table of active connections for each server.

LARD Performance StudyLARD Performance Study

LARD paper compares SLB/WRR and LB with LARD approaches:

• simulation study

small Rice and IBM web server logs

jiggle simulation parameters to achieve desired result

• Nodes have small memories with greedy-dual replacement.

• WRR combined with global cache-sharing among servers (GMS).

WRR/GMS is global cache LRU with duplicates and cache-sharing cost.

LB/GC is global cache LRU with duplicate suppression and no cache-sharing cost.

LARD Performance ConclusionsLARD Performance Conclusions

1. WRR has the lowest cache hit ratios and the lowest throughput.

There is much to be gained by improving cache effectiveness.

2. LB* achieve slightly better cache hit ratios than LARD*.

WRR/GMS lags behind...it’s all about duplicates.

3. The caching benefit of LB* is minimal, and LB is almost as good as LB/GC.

Locality-* request distribution induces good cache behavior at the back ends: global cache replacement adds little.

4. Better load balancing in the LARD* strategies dominates the caching benefits of LB*.

LARD/R and LARD achieve the best throughput and scalability; LARD/R yields slightly better throughput.

LARD Performance: Issues and QuestionsLARD Performance: Issues and Questions

1. LB (URL switching) has great cache behavior but lousy throughput.

Why? Underutilized time results show poor load balancing.

2. WRR/GMS has good cache behavior and great load balancing, but not-so-great throughput.

Why? How sensitive is it to CPU speed and network speed?

3. What is the impact of front-end caching?

4. What is the effectivness of bucketed URL hashing policies?

E.g., Foundry: hash URL to a bucket, pick server for bucket based on load.

5. Why don’t L7 switch products support LARD? Should they?

[USENIX 2000]: use L4 front end; back ends do LARD handoff.

Possible ProjectsPossible Projects

1. Study the impact of proxy caching on the behavior of the request distribution policies.

“flatter” popularity distributions

2. Study the behavior of alternative locality-based policies that incorporate better load balancing in the front-end.

How close can we get to the behavior of LARD without putting a URL lookup table in the front-end?

E.g., look at URL switching policies in commercial L7 switches.

3. Implement request switching policies in the FreeBSD kernel, and measure their performance over GigE.

Mods to FreeBSD for recording connection state and forwarding packets are already in place.

4. How to integrate smart switches with protocols for group membership or failure detection?