anycast latency - ripe 73 · 2016-10-24 · what rtt is possible? 0 0.2 0.4 0.6 0.8 1 0 50 100 150...
TRANSCRIPT
Anycast LatencyHow many sites are enough?
Presented by
Ricardo de Oliveira Schmidt
October 25, 2016 Madrid, Spain
Presentation copyright © 2016 by Ricardo de Oliveira Schmidt
Reference:
Anycast Latency: How Many Sites Are Enough?
Ricardo de O. Schmidt, John Heidemann and Jan Harm Kuipers
Technical Report ISI-TR-2016-708, USC/Information Sciences Institute, May 2016 • http://www.isi.edu/~johnh/PAPERS/Schmidt16a.pdf
Anycast
Definition Multiple copies of a service at different locations configured to answer with the same IP address
Purpose Distribution, redundancy, resilience, performance
Where Content Delivery Networks (CDNs) DNS at the root and TLD levels
Anycast
Anycast
Anycast
Anycast Anycast uses BGP to map users to sites But BGP only approximates closest match
If BGP only approximates closest in anycast: 1. How good (or bad) is this approximation? 2. What is the impact of location of anycast sites? 3. What is the impact of routing policy?
Can we determine a minimum number of anycast sites as optimal?
To answer these questions we... ... study the relationship between latency and number of anycast sites ... look at the Root DNS as case of study
RQ's and Approach
Our case of study were four letters of the Root DNS: C: 8 sites F: 58 sites K: 33 sites L: 144 sites
We DO NOT focus on: What a letter should or not do
Root DNS (or a part of it)
C-RootF-RootK-RootL-Root
F
F FF
F
FF
F
F
F
F
F
F
F
FF
F
F
F
FF
FF
F FF
FF
FF
F
F
F
F
F
F
F
F
F
F
F
F
F
F
F
F
F
F
FF
F
F
F
FFFF
F
L
L
L
LL L
L
LL
L L
L
L
L
L
L
LL
LL
L
LLL
L
L
L
L
L
LL
L
L
LL
L LL L
LL L
LL
L
L
L
L
L LL
L
L
LL
LL LL
LL
L
L
LL
L
L
LL
L
L
L
L
L
LL
L
L
L
L
L
L
L
L
LL
L
LLLL
LL
L L
L
L
L
LL
LL
LLL
L
L
L
LL
L L
L
L
L
LL
L
L
LL
L
L L
L L
L
LL
L
L
L
L
LLLLL L L
KKK
K
KKKKKK
K
KK
KK
K
KK
KK K
K
K KKKKK
KCC
CC CC
CC
FKL
RIPE Atlas
Our vantage points (VPs) are RIPE Atlas probes 7,900 of them 174 countries 2927 ASes
The bias towards Europe does not impact our qualitative results and conclusions
Step 1: determine catchmentCHAOS queries to hostname.bind to anycast prefix...
... returns a string that identifies a server
Measurements
anycast service
AMS
SYD
LAX
S1
S1
S1
S2
S2
S2
VP
CHAOS query
S1SYD.anycast-service.org
Step 2: determine (optimal) latencyICMP (ping) to all sites of the anycast...
... as given by the identifier from CHAOS
Measurements
anycast service
AMS
SYD
LAX
S1
S1
S1
S2
S2
S2
VP RTT to SYD
RTT to AMS
RTT to LAX
Is RTT to SYD the lowest?
What Performance do we see?Distribution of RTT for all letters:
We do not determine a threshold for good or bad performance... ... but we do believe that 30ms is very good
0
0.2
0.4
0.6
0.8
1
0 50 100 150 200 250 300 350
CD
F
RTT (ms)
C-Root actualF-Root actualK-Root actualL-Root actual
median RTT ~30ms
Comparing actual to optimal possible: C-Root with 8 sites is at its optimal L-Root with 144 sites has plenty of room But their median is roughly the same
Note: Comparing anycast and unicast routing
What RTT is possible?
0
0.2
0.4
0.6
0.8
1
0 50 100 150 200 250 300 350
CD
F
RTT (ms)
C-Root actualC-Root optimal
median RTT = 32ms
0
0.2
0.4
0.6
0.8
1
0 50 100 150 200 250 300 350
CD
F
RTT (ms)
L-Root actualL-Root optimal
median RTT = 30ms
C-Root
L-Root
Location matters!Simulation using C-Root sites from west to east coast
Note: C-Root has sites in Europe Bias from RIPE Atlas is visible
What about location?
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200 250 300 350
LA
X (
only
)
LA
X+O
RD
LA
X+O
RD
+IA
D
LA
X+O
RD
+IA
D+JF
K
C-R
oot optim
al
CD
F
RTT (ms)
Location matters!Simulation using C-Root sites from US and Europe
Note: C-Root has sites in Europe Bias from RIPE Atlas is visible
What about location?
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200 250 300 350
CD
F
RTT (ms)
C-Root optimalCDG
CDG + LAXCDG + LAX + JFK
CDG + LAX + JFK + FRA
Location matters!Simulation using C-Root sites from US and Europe
Note: C-Root has sites in Europe Bias from RIPE Atlas is visible
What about location?
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200 250 300 350
CD
F
RTT (ms)
C-Root optimalCDG
CDG + LAXCDG + LAX + JFK
CDG + LAX + JFK + FRA
long distribution tail
So, at the end many sites do help those at the end of the distribution tail... ... but it depends where and how connected
Sites vs Location
0
100
200
300
MY
(2
0)
SC
(6
)
RE
(5
)
MU
(5
)
UG
(5
)
TZ
(5
)
ID (
33
)
BD
(1
3)
NP
(8
)
AU
(6
9)
CN
(1
4)
ZA
(5
2)
BT
(5
)
IN (
15
)
SG
(4
5)
PH
(2
6)
UY
(1
0)
CL
(1
6)
KE
(1
0)
NC
(5
)
AR
(1
8)
PK
(7
)
HK
(2
2)
SN
(6
)
BR
(4
9)
NZ
(5
3)
GU
(5
)
TW
(5
)
LK
(2
2)
KR
(1
0)
AE
(8
)
JP (
72
)
LB
(5
)
BJ
(6)
IR (
28
)
AD
(9
)
KZ
(2
1)
CR
(5
)
TN
(7
)
IL (
18
)
CY
(1
0)
PR
(6
)
GR
(4
6)
AM
(1
7)
GE
(8
)
TR
(2
0)
IS (
8)
RU
(3
07
)
BY
(1
5)
MK
(9
)
AL
(1
5)
IE (
75
)
BA
(7
)
FI
(59
)
NO
(8
3)
LT
(1
0)
LV
(1
5)
EE
(1
8)
PT
(4
8)
RO
(2
8)
UA
(1
28
)
SE
(9
6)
BG
(6
6)
IT (
15
3)
MD
(8
)
PL
(9
2)
RS
(2
9)
CA
(1
32
)
HR
(2
6)
ES
(9
3)
BE
(1
40
)
CZ
(1
66
)
HU
(3
6)
DE
(6
12
)
US
(6
82
)
DK
(7
8)
GB
(3
47
)
CH
(1
63
)
FR
(4
77
)
SI
(28
)
AT
(1
06
)
NL
(3
30
)
SK
(3
2)
LU
(2
5)
RT
T (
ms)
country code (# of VPs)
Africa (F)Asia (A)
Europe (E)South America (S)
North America (N)Oceania (O)
F F F F F F F F F FA A A A A A A A A A A A A A A A A A A A A A A
E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E ES S S S
N N N N NO O O O
C-R
oot
So, at the end many sites do help those at the end of the distribution tail... ... but it depends where and how connected
Sites vs Location
0
100
200
300
MY
(2
0)
SC
(6
)
RE
(5
)
MU
(5
)
UG
(5
)
TZ
(5
)
ID (
33
)
BD
(1
3)
NP
(8
)
AU
(6
9)
CN
(1
4)
ZA
(5
2)
BT
(5
)
IN (
15
)
SG
(4
5)
PH
(2
6)
UY
(1
0)
CL
(1
6)
KE
(1
0)
NC
(5
)
AR
(1
8)
PK
(7
)
HK
(2
2)
SN
(6
)
BR
(4
9)
NZ
(5
3)
GU
(5
)
TW
(5
)
LK
(2
2)
KR
(1
0)
AE
(8
)
JP (
72
)
LB
(5
)
BJ
(6)
IR (
28
)
AD
(9
)
KZ
(2
1)
CR
(5
)
TN
(7
)
IL (
18
)
CY
(1
0)
PR
(6
)
GR
(4
6)
AM
(1
7)
GE
(8
)
TR
(2
0)
IS (
8)
RU
(3
07
)
BY
(1
5)
MK
(9
)
AL
(1
5)
IE (
75
)
BA
(7
)
FI
(59
)
NO
(8
3)
LT
(1
0)
LV
(1
5)
EE
(1
8)
PT
(4
8)
RO
(2
8)
UA
(1
28
)
SE
(9
6)
BG
(6
6)
IT (
15
3)
MD
(8
)
PL
(9
2)
RS
(2
9)
CA
(1
32
)
HR
(2
6)
ES
(9
3)
BE
(1
40
)
CZ
(1
66
)
HU
(3
6)
DE
(6
12
)
US
(6
82
)
DK
(7
8)
GB
(3
47
)
CH
(1
63
)
FR
(4
77
)
SI
(28
)
AT
(1
06
)
NL
(3
30
)
SK
(3
2)
LU
(2
5)
RT
T (
ms)
country code (# of VPs)
Africa (F)Asia (A)
Europe (E)South America (S)
North America (N)Oceania (O)
F F F F F F F F F FA A A A A A A A A A A A A A A A A A A A A A A
E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E ES S S S
N N N N NO O O O
0
100
200
300
BD
(14)
NP
(9)
SC
(6)
IR (
26)
PH
(20)
PR
(5)
BT
(6)
SG
(48)
PT
(52)
SN
(6)
GE
(11)
JP (
88)
HK
(25)
LK
(20)
BY
(13)
IL (
20)
HR
(37)
BA
(6)
IQ (
7)
BJ
(6)
ES
(112)
AL (
17)
IS (
10)
MY
(20)
MT
(5)
KE
(15)
CY
(13)
DO
(6)
IT (
189)
MK
(13)
EE
(24)
LT
(15)
MD
(9)
RE
(5)
MU
(5)
LB
(6)
PE
(5)
BG
(70)
NL (
407)
HU
(56)
CA
(150)
IN (
13)
SI (3
4)
PL (
120)
SK
(36)
DE
(730)
AT
(147)
RO
(34)
US
(812)
AD
(8)
MX
(5)
CZ
(214)
FI (8
4)
IE (
88)
LU
(37)
GB
(464)
LV
(20)
RU
(371)
ZA
(64)
PK
(7)
AU
(81)
KZ
(21)
BE
(171)
GR
(62)
UA
(172)
FR
(572)
CH
(191)
TH
(5)
AR
(20)
MZ
(5)
DK
(98)
ID (
30)
CL (
11)
NZ
(58)
BR
(42)
GU
(5)
NO
(107)
AE
(6)
SE
(135)
KR
(14)
UY
(13)
RS
(34)
TR
(24)
RT
T (
ms)
country code (# of VPs)
Africa (F)Asia (A)
Europe (E)South America (S)
North America (N)Oceania (O)
F F F F F F F FA A A A A A A A A A A A A A A A A A A A A A
E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E ES S S S S
N N N N N NO O O
C-R
oot
L-R
oot
So, at the end many sites do help those at the end of the distribution tail... ... but it depends where and how connected
Sites vs Location
0
100
200
300
MY
(2
0)
SC
(6
)
RE
(5
)
MU
(5
)
UG
(5
)
TZ
(5
)
ID (
33
)
BD
(1
3)
NP
(8
)
AU
(6
9)
CN
(1
4)
ZA
(5
2)
BT
(5
)
IN (
15
)
SG
(4
5)
PH
(2
6)
UY
(1
0)
CL
(1
6)
KE
(1
0)
NC
(5
)
AR
(1
8)
PK
(7
)
HK
(2
2)
SN
(6
)
BR
(4
9)
NZ
(5
3)
GU
(5
)
TW
(5
)
LK
(2
2)
KR
(1
0)
AE
(8
)
JP (
72
)
LB
(5
)
BJ
(6)
IR (
28
)
AD
(9
)
KZ
(2
1)
CR
(5
)
TN
(7
)
IL (
18
)
CY
(1
0)
PR
(6
)
GR
(4
6)
AM
(1
7)
GE
(8
)
TR
(2
0)
IS (
8)
RU
(3
07
)
BY
(1
5)
MK
(9
)
AL
(1
5)
IE (
75
)
BA
(7
)
FI
(59
)
NO
(8
3)
LT
(1
0)
LV
(1
5)
EE
(1
8)
PT
(4
8)
RO
(2
8)
UA
(1
28
)
SE
(9
6)
BG
(6
6)
IT (
15
3)
MD
(8
)
PL
(9
2)
RS
(2
9)
CA
(1
32
)
HR
(2
6)
ES
(9
3)
BE
(1
40
)
CZ
(1
66
)
HU
(3
6)
DE
(6
12
)
US
(6
82
)
DK
(7
8)
GB
(3
47
)
CH
(1
63
)
FR
(4
77
)
SI
(28
)
AT
(1
06
)
NL
(3
30
)
SK
(3
2)
LU
(2
5)
RT
T (
ms)
country code (# of VPs)
Africa (F)Asia (A)
Europe (E)South America (S)
North America (N)Oceania (O)
F F F F F F F F F FA A A A A A A A A A A A A A A A A A A A A A A
E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E ES S S S
N N N N NO O O O
0
100
200
300
BD
(14)
NP
(9)
SC
(6)
IR (
26)
PH
(20)
PR
(5)
BT
(6)
SG
(48)
PT
(52)
SN
(6)
GE
(11)
JP (
88)
HK
(25)
LK
(20)
BY
(13)
IL (
20)
HR
(37)
BA
(6)
IQ (
7)
BJ
(6)
ES
(112)
AL (
17)
IS (
10)
MY
(20)
MT
(5)
KE
(15)
CY
(13)
DO
(6)
IT (
189)
MK
(13)
EE
(24)
LT
(15)
MD
(9)
RE
(5)
MU
(5)
LB
(6)
PE
(5)
BG
(70)
NL (
407)
HU
(56)
CA
(150)
IN (
13)
SI (3
4)
PL (
120)
SK
(36)
DE
(730)
AT
(147)
RO
(34)
US
(812)
AD
(8)
MX
(5)
CZ
(214)
FI (8
4)
IE (
88)
LU
(37)
GB
(464)
LV
(20)
RU
(371)
ZA
(64)
PK
(7)
AU
(81)
KZ
(21)
BE
(171)
GR
(62)
UA
(172)
FR
(572)
CH
(191)
TH
(5)
AR
(20)
MZ
(5)
DK
(98)
ID (
30)
CL (
11)
NZ
(58)
BR
(42)
GU
(5)
NO
(107)
AE
(6)
SE
(135)
KR
(14)
UY
(13)
RS
(34)
TR
(24)
RT
T (
ms)
country code (# of VPs)
Africa (F)Asia (A)
Europe (E)South America (S)
North America (N)Oceania (O)
F F F F F F F FA A A A A A A A A A A A A A A A A A A A A A
E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E ES S S S S
N N N N N NO O O
dist
ribut
ion
tail C
-Roo
tL-
Roo
t
Looking at real-world anycast deployments we learned that... ... number of sites can make a difference on performance, but... ... location and connectivity of sites have higher impact... we believe that ~12 well-connected sites is "enough" for performance
For the future, we will focus on other purposes of anycast: Resilience to Denial of Service attacks Load balancing
Final Considerations
Looking at real-world anycast deployments we learned that... ... number of sites can make a difference on performance, but... ... location and connectivity of sites have higher impact... we believe that ~12 well-connected sites is "enough" for performance
For the future, we will focus on other purposes of anycast: Resilience to Denial of Service attacks Load balancing
Final Considerations
in about 20 minutes (depending on Wouter)
Acknowledgements:
Geoff Huston, George Michaelson, Ray Bellis, Cristian Hesselman, Benno Overeinder, Jaap Akkerhuis, Duane Wessels, Paul Vixie, Romeo Zwart, Anand Buddhdev, and C-Root operators.
SIDN Labs, NLnet Labs and SURFnet
Self-managing Anycast Networks for the DNS (SAND) project | http://www.sand-project.nl/ DNS Anycast Security (DAS) project | http://www.das-project.nl/
[email protected]://www.ricardoschmidt.com/