anycast for any servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · current services using...
TRANSCRIPT
![Page 1: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/1.jpg)
Anycast for Any Service
Michael J. Freedman
Karthik LakshminarayananDavid Mazières
http://oasis.coralcdn.org/
![Page 2: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/2.jpg)
mycdn
?
What’s the replica-selection problem?
� Client needs to choose a “good” replica server
� Performance and cost dependent on replica selection
![Page 3: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/3.jpg)
What do we currently do?
![Page 4: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/4.jpg)
How bad can it get?
![Page 5: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/5.jpg)
Anycast is the solution
� Anycast = automated “good” replica selection
� OASIS is a flexible anycast system for multiple services
?
mycdn
![Page 6: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/6.jpg)
The need for anycast
� Internet systems rely on replicated content and services
� Distributed mirrors: Web servers, FTP servers, …
� Content Distribution Networks: Akamai, CoralCDN, …
� Internet Naming Systems: DNS, SFR, DOA, …
� Distributed File Systems: CFS, Shark, ….
� Routing Overlays: RON, Detour, i3, …
� Distributed Hash Storage Systems: OpenDHT, …
� All could benefit from anycast service
![Page 7: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/7.jpg)
How should oneimplement anycast?
![Page 8: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/8.jpg)
Strawman: probe & find nearestmycdn
ID
B
C
A
E ICMP
![Page 9: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/9.jpg)
Strawman: probe & find nearestmycdn
ID
B
C
A
E ICMP
, ,E I D
���� Result highly accurate
���� Lots of probing
���� Slow to compute
![Page 10: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/10.jpg)
mycdn
ID
B
C
A
E ICMP
, ,E I D
���� Result highly accurate
���� Lots of probing
���� Slow to compute
Strawman: probe & find nearest
![Page 11: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/11.jpg)
mycdn
ID
B
C
A
E ICMP
, ,E I D
���� Result highly accurate
���� Lots of probing
���� Slow to compute
Avoid probing on-demand
![Page 12: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/12.jpg)
mycdn
ID
B
C
A
E ICMP
, ,E I D
Avoid probing on-demand
���� Result highly accurate
���� Lots of probing
���� Slow to compute
18.0.0.0/8
[IMC05] shows IP prefixes often preserve locality( 99% of /24s by stub AS at the same location )
This is the problem Akamai must solve
![Page 13: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/13.jpg)
What about yourcdn?
mycdn E I D yourcdn M N O
18.0.0.0/8
���� Result highly accurate
���� Lots of probing
���� Slow to compute
![Page 14: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/14.jpg)
Idea: Use geographic coordinates
mycdn yourcdn
18.0.0.0/8
(42N,71W)
���� Result highly accurate
���� Lots of probing
���� Slow to compute
���� Stable across services, time, and failures
Assume all replicas know geo-coords
(42N,71W)
���� Amortize costs
![Page 15: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/15.jpg)
OASIS provides…
���� Result highly accurate
���� Stable across time, services, and failures
���� Amortize costs
���� Fast response time
���� Supports flexible anycast policies� Balances tension between:
� Performance: finding nearest replica
� Cost: minimizing 95% bandwidth usage
![Page 16: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/16.jpg)
Outline
� Architecture and design decisions
� Detailed design
� Evaluation
� Deployment and integration lessons
� OASIS deployed since November 2005
� Currently in use by 10 services
![Page 17: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/17.jpg)
Two-tier architecturemycdn
OASIS core
Large set of replicas that assist in measurement
Reliable core of hosts that implement anycast
=DNS OASIS
nodeRPC
mycdnreplicaproxy=
![Page 18: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/18.jpg)
Using OASIS via DNS
1. Client issues DNS request for mycdn.nyuld.net
2. OASIS redirects client to nearby application replica
mycdn
OASIS core
Client Resolver
12
![Page 19: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/19.jpg)
123
OASIS core
mycdn
1. Client issues HTTP request
2. Web cgi-bin issues RPC to OASIS core
3. Client redirected to nearby application replica
Client
Using OASIS via HTTP
![Page 20: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/20.jpg)
servicebucketing
proximity replicas
IP prefix policy
coords
response
requestIP addr name
Using OASIS via HTTPHow does core answer anycast?
![Page 21: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/21.jpg)
How does core answer anycast?
servicebucketing
proximity replicas
IP prefix
coords
requestIP addr name
policy
response
mycdn
18.26.4.9171.66.3.181
216.165.109.81
mycdn .nyuld.net18.71.0.3
18.0.0.0/8
18.26.4.9
![Page 22: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/22.jpg)
How to map IP prefix to coords?
proximityIP prefix ( Lat, Lng, RTT distance )
location accuracy
![Page 23: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/23.jpg)
� Two-pronged approach� Find closest replica proxy
How to map IP prefix to coords?
:18.0.0.0/8
(42N,71W)(42N,71W)
proximityIP prefix ( Lat, Lng, RTT distance )
location accuracy
![Page 24: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/24.jpg)
� Two-pronged approach� Find closest replica proxy
� Use closest replica’s geo-coords + error RTT as location
How to map IP prefix to coords?
:18.0.0.0/8
(42N,71W)
, 6.0 ms
(42N,71W)
proximityIP prefix ( Lat, Lng, RTT distance )
location accuracy
![Page 25: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/25.jpg)
� Two-pronged approach� Find closest replica proxy with less probing
� Use closest replica’s geo-coords + error RTT as location
18.168.0.23
Find replica nearest prefix efficiently
18.0.0.0/8
“Probe 18.0.0.0/8”
![Page 26: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/26.jpg)
� Two-pronged approach� Find closest replica proxy with less probing
� Use closest replica’s geo-coords + error RTT as location
18.168.0.23
Find replica nearest prefix efficiently
[ Meridian 05 ]
![Page 27: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/27.jpg)
18.168.0.23
Find replica nearest prefix efficiently
[ Meridian 05 ]� Two-pronged approach
� Find closest replica proxy with less probing
� Use closest replica’s geo-coords + error RTT as location
:18.0.0.0/8 , 6.0 ms(42N,71W)
![Page 28: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/28.jpg)
Geographic distance vs. RTT
� Strong correlation b/w geographical distance and RTT
![Page 29: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/29.jpg)
Geographic distance vs. RTT
� Strong correlation b/w geographical distance and RTT
� RTT accuracy has real-world meaning
� Check if new coordinates improve accuracy vs. old coords
![Page 30: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/30.jpg)
Geographic distance vs. RTT
� Strong correlation b/w geographical distance and RTT
� RTT accuracy has real-world meaning
� Check if new coordinates improve accuracy vs. old coords
[ Meridian 05 ]:18.0.0.0/8 , 6.0 ms(42N,71W):18.0.0.0/8 , 3.0 ms(42N,72W):18.0.0.0/8 , 3.0 ms(42N,72W)
![Page 31: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/31.jpg)
Geographic distance vs. RTT
� Strong correlation b/w geographical distance and RTT
� RTT accuracy has real-world meaning
� Check if new coordinates improve accuracy vs. old coords
[ Meridian 05 ]:18.0.0.0/8 , 6.0 ms(42N,71W):18.0.0.0/8 , 3.0 ms(42N,72W)
:18.0.0.0/8 , 3.0 ms(42N,72W)
![Page 32: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/32.jpg)
Geographic distance vs. RTT
� Strong correlation b/w geographical distance and RTT
� RTT accuracy has real-world meaning
� Check if new coordinates improve accuracy vs. old coords
� Useful for sanity check for network peculiarities
� Do multiple results satisfy constraints (e.g., speed of light) ?
![Page 33: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/33.jpg)
Outline
� Architecture and design decisions
� Detailed design
� Evaluation
� Deployment and integration lessons
� OASIS deployed since November 2005
� Currently in use by 10 services
![Page 34: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/34.jpg)
mycdn opendht
� OASIS core
� Global membership view
� Epidemic gossiping• Scalable failure detection• Spread policies, prefix coords
� Consistent hashing• Divide up responsibility for prefixes
� Service replicas
� Heartbeats to OASIS node
� Form global Meridian overlay for probing
OASIS core
![Page 35: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/35.jpg)
How to find “nearby” nodes?
replicas
IP prefix
coords
requestIP addr name
servicebucketing
proximity
policy
response
mycdn
mycdn .nyuld.net
18.26.4.9171.66.3.181
216.165.109.81
18.26.4.9
18.26.4.9
18.0.0.0/8
Local info from gossiping(stale data okay)
![Page 36: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/36.jpg)
How to find “nearby” nodes?
replicas
IP prefix
coords
requestIP addr name
servicebucketing
proximity
policy
response
mycdn
18.26.4.9Local info from gossiping(stale data okay)
mycdn .nyuld.net18.26.4.9
18.0.0.0/8
18.26.4.9171.66.3.181
216.165.109.81
Clients react poorly to stale data
![Page 37: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/37.jpg)
Aggregate replica information
OASIS
mycdn
� Define service’s rendezvous node via consistent hashing
� Service replicas send keepalives to nearby OASIS nodes
� Update rendezvous when replicas join, leave, large load change
OASIS
H(srv)
![Page 38: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/38.jpg)
Aggregate replica information
OASIS
mycdn
� Define service’s rendezvous node via consistent hashing
� Service replicas send keepalives to nearby OASIS nodes
� Update rendezvous when replicas join, leave, large load change
OASIS
Bottleneck?H(srv)
![Page 39: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/39.jpg)
Aggregate replica information
OASIS
mycdn
� Aggregate over k nodes for scalability
� Rendezvous gossip liveness state for loose consistency
� k can be dynamic for better scalability
OASIS
H(srv)
![Page 40: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/40.jpg)
A client’s view: Finding a nameserver
OASISOASIS
� Core lookup: Contacts 1 of 13 nameservers for .nyuld.net
� OASIS “uses itself” to discover replica for service dns
H(dns)
Client
![Page 41: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/41.jpg)
A client’s view: Finding a nameserver
OASISOASIS
� Core lookup: Contacts 1 of 13 nameservers for .nyuld.net
� OASIS “uses itself” to discover replica for service dns
� Returns nearby nameservers for subsequent requests
H(dns)
Client
![Page 42: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/42.jpg)
OASISOASIS
H(mycdn)
� Replica lookup: Client contacts nearby nameserver
� OASIS discover replica for service mycdn
� Returns nearby replicas for application
A client’s view: Finding a replica
R
H(dns)
Client
![Page 43: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/43.jpg)
Evaluation
� Deployed on PlanetLab since November 2005
� How much end-to-end benefit from OASIS?
� How accurate is OASIS?
� Effective for load balancing?
� What are OASIS’s bandwidth costs?
![Page 44: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/44.jpg)
E2E download of web page
290% faster than Meridian
500% faster than RR
Cached virtual coordshighly inaccurate
![Page 45: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/45.jpg)
Client RTT to chosen replica
Outperforms Meridian 60% of time
![Page 46: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/46.jpg)
OASIS minimizes bandwidth spikes
Load + Latency
0.00.00.023.3Latency Only
GermanyNYTXCA
95% bandwidth usage per replica (MB)
metricloc
� 8 clients in CA repeatedly request 1 MB file
� Replicas report load as log (95% bandwidth per 1-min slot)
![Page 47: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/47.jpg)
9.29.611.39.0Load + Latency
0.00.00.023.3Latency Only
GermanyNYTXCA
95% bandwidth usage per replica (MB)
metricloc
� 8 clients in CA repeatedly request 1 MB file
� Replicas report load as log (95% bandwidth per 1-min slot)
OASIS minimizes bandwidth spikes
![Page 48: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/48.jpg)
Bandwidth costs: OASIS v. on-demand
1-2 orders of magnitude
# DNS reqsto CoralCDN
![Page 49: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/49.jpg)
Outline
� Architecture and design decisions
� Detailed design
� Evaluation
� Deployment and integration lessons
� OASIS deployed since November 2005
� Currently in use by 10 services
![Page 50: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/50.jpg)
Sanity check for network peculiarities
� Employ measurement redundancy
� Easy visualization significantly helped debugging
![Page 51: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/51.jpg)
� Probing generates abuse complaints
� Your service can get blacklisted!
Keyword frequency on PlanetLab support lists9 months, 1820 threads, 4682 msgs
Netops have low tolerance for probing
![Page 52: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/52.jpg)
Netops have low tolerance for probing
� Be careful what you probe� Probe slowly and rarely
� No random ports or obvious attack vectors (TCP port 22/23)
� Be careful whom you probe� Check blacklist for netblock and target IP (after traceroute)
![Page 53: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/53.jpg)
Make it easy to integrate
replicaproxy
dns
nakikaOASIScorenode
listen(7060)
ServiceName nakikaLocalPort 7060SecretCode 555555
ServiceName nakikaServiceAlias nakika.netSortType latencycapMaxAddrs 2AddrTTLs 120
![Page 54: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/54.jpg)
replicaproxy
dns
nakikaOASIScorenodecode
loadcap
ServiceName nakikaLocalPort 7060SecretCode 555555
ServiceName nakikaServiceAlias nakika.netSortType latencycapMaxAddrs 2AddrTTLs 120
listen(7060)?
Clients immediate use nakika.nyuld.net
Make it easy to integrate
![Page 55: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/55.jpg)
Current services using OASIS…
� Chunkcast block anycast (Berkeley)
� CoralCDN (NYU)
� Na Kika content distribution (NYU)
� OASIS
� RPC, DNS, HTTP interfaces
� OCALA overlay convergence (Berkeley)
� Separate services for client and server IPs gateways
� OpenDHT public DHT service (Berkeley)
� OverCite distributed library (MIT): Deployed on RON
![Page 56: Anycast for Any Servicemfreed/docs/oasis-nsdi06... · 2007-08-08 · Current services using OASIS… Chunkcast block anycast (Berkeley) CoralCDN (NYU) Na Kika content distribution](https://reader035.vdocument.in/reader035/viewer/2022070913/5fb494074aae5c4e663a3060/html5/thumbnails/56.jpg)
Summary
� OASIS is a general, open anycast service
� Supports multiple services: more are better
� Performs accurate server selection
� Removes all on-demand probing
� Provides easy integration
� Use OASIS for your distributed system!
http://oasis.coralcdn.org/