beyond counting: new perspectives on the active ipv4
TRANSCRIPT
Beyond Counting:New Perspectives on the Active IPv4 Address Space
Philipp Richter Georgios Smaragdakis David Plonka Arthur BergerTU Berlin MIT Akamai Akamai/MIT
@IETF 96 Berlin (maprg) July 2016
work under submission comments highly appreciated!
preprint: http://arxiv.org/abs/1606.00360
Philipp Richter | INET / TU Berlin 1
IPv4 Address Space Exhaustion
IPv4 Standard
1981
RIR Framework
InitiationFirst RIR (RIPE)
founded
1992
APNICexhausted
2011
RIPEexhausted
2012
ARINexhausted
2015
Early Registration Needs-Based Provision Depletion & Exhaustion
2005
Last RIR(AFRINIC)founded
LACNICexhausted
2014 /8
equ
ivale
nts
● ●●
●●
●●
●
●
●
●
●
●
●●
●●
0
50
100
150
200
250
Nov 1997 Jan 2002 Jan 2006 Jan 2010 Jan 2014
routable address space limit (220.7 /8 equivalents)
total address space limit (256 /8 equivalents)
● allocated address blocksrouted address blocks
Figures: P. Richter, M. Allman, R. Bush, V. Paxson: A Primer on IPv4 Scarcity, ACM CCR 45(2), 2015.
• IPv4 has been around for ~35 years • Theoretically routable IP addresses: 3.7B, ~2.8B routed • IANA exhausted its address pool in 2011 • Today: Less than 2% of the IPv4 address space “free”
http://arxiv.org/abs/1606.00360
Operators' Community Efforts
Efforts in the IETF community: • IPv6 transition mechanisms • IPv4 multiplexing/sharing mechanisms (e.g., EnIP, A+P) • Efforts to conserve IPv4 address space
IANA/Regional Internet Registries: • Establishment of address transfer policies • Incentives for increasing address space utilization
e.g., draft-fleischhauer-ipv4-addr-saving-05, RFC6346, draft-chimiak-enhanced-ipv4-03
Philipp Richter | INET / TU Berlin 2http://arxiv.org/abs/1606.00360
Academic Community Efforts
• Measurements to understand “where we are” right now • Internet-wide: Number of actively used IPv4 addresses:
“1.2B IP addresses in use in 2014”, statistical estimation
“5.3M /24 address blocks in use in 2013”, passive+active measurement
• Challenge: No single vantage point captures all activity
Philipp Richter | INET / TU Berlin 3http://arxiv.org/abs/1606.00360
Zander et al., IMC ‘14
Dainotti et al., JSAC ‘16
Academic Community Efforts
• Measurements to understand “where we are” right now • Internet-wide: Number of actively used IPv4 addresses:
“1.2B IP addresses in use in 2014”, statistical estimation
“5.3M /24 address blocks in use in 2013”, passive+active measurement
• Challenge: No single vantage point captures all activity
Philipp Richter | INET / TU Berlin 3http://arxiv.org/abs/1606.00360
Zander et al., IMC ‘14
Dainotti et al., JSAC ‘16
What can we say from our CDN’s perspective?Can we do more than counting active IP addresses?
The CDN as an Observatory
CDN front-end servers
HTTP(S) requests
• 200,000+ servers • 3 trillion requests per day • CDN logs: number of requests per IP per day
Totals for the entirety of 2015: • 1.2B active IPv4 addresses (42% of routed) • 6.5M active /24 address blocks (59% of routed)
Philipp Richter | INET / TU Berlin 4http://arxiv.org/abs/1606.00360
Visibility: CDN logs vs. ICMP scan (ZMap project, 8 snapshots)
% IPv4 addresses visibly active (N=950M, Oct. 2015)
CD
N o
nly
CD
N &
ICM
P
ICM
P on
ly
0 20 40 60 80 100
Peak IPv4?
●●●●●●●
●●●●●●●
●●●●●
●●●●●●●
●●●●●
●●●●●●●
●●●●●●●●●●●●●
●●●●
●●●●●●●
●●●●●●●●●●●●●●●●
●●●●●●●●●●●●
●●●●●●●●●
date [ticks: January of each year]
uniq
ue IP
v4 a
ddre
sses
200M
400M
600M
800M
1B
2008 2009 2010 2011 2012 2013 2014 2015 2016
● unique active IPv4 addresses per monthlinear regression until 2014−01
●
IANA exhaustion●
RIPE exhaustion
●
ARIN exhaustion
●
APNIC exhaustion●
LACNIC exhaustion
Active IPv4 address counts have stagnated since 2014
Philipp Richter | INET / TU Berlin 5http://arxiv.org/abs/1606.00360
Daily IPv4 Activity and Churn
●●●●●●●●●●●●●●●●●●●
●●●●●●●
●●●●●●●
●●●●●●●
●●●●●●●
●●●●●●●
●●
●●●●●●●●●●●●
●●●●●●●
●●●●●●●●●
●●●●●●●●●●●●
●●●●●●●●●
●●●●
●
●●
days from 2015−08−17 to 2015−12−06
uniq
ue IP
v4 a
ddre
sses
020
0M40
0M60
0M
0 14 28 42 56 70 84 98 112
● active IPv4 addressesup eventsdown events
Philipp Richter | INET / TU Berlin 6http://arxiv.org/abs/1606.00360
Churn on all Timescales
day-to-day: ~7% come, ~7% go
week-to-week: ~5% come, ~5% go
month-to-month: ~5% come, ~5% go
The number of active IPv4 addresses stays constantthe set of active addresses varies on all timescales
Philipp Richter | INET / TU Berlin 7http://arxiv.org/abs/1606.00360
Long-term Effect of Address Churn
time lag from 2015−01−01
chan
ge in
act
ive IP
v4 a
ddre
sses
1 week 26 weeks 52 weeks−200
M−1
00M
010
0M20
0M
appear
−25%
−12.
5%0
12.5
%25
%
disappear
Over the course of one year, 25% of the active IP address pool changed
Philipp Richter | INET / TU Berlin 8http://arxiv.org/abs/1606.00360
Address Activity Matrix
130.149.0.6 130.149.0.5 130.149.0.4 130.149.0.3 130.149.0.2 130.149.0.1
…
…addr
ess
spac
e
days
for each day on which an IP address was active (requested content), we draw a red dot.
Philipp Richter | INET / TU Berlin 9http://arxiv.org/abs/1606.00360
Patterns: “In situ” Address Activity
time [months]
IP a
ddre
ss a
ctiv
ity w
ithin
/24
0 1 2 3 4
.0.1
27.2
55
time [months]IP
add
ress
act
ivity
with
in /2
40 1 2 3 4
.0.1
27.2
55
time [months]
IP a
ddre
ss a
ctiv
ity w
ithin
/24
0 1 2 3 4
.0.1
27.2
55
time [months]
IP a
ddre
ss a
ctiv
ity w
ithin
/24
0 1 2 3 4
.0.1
27.2
55static block DE University DHCP pool US University
residential users US ISP residential users DE ISP
“in situ” activity:address assignment practice
+user behavior
(no visible modification of address assignment practice)
Philipp Richter | INET / TU Berlin 10http://arxiv.org/abs/1606.00360
Patterns: Operational Change
time [months]
IP a
ddre
ss a
ctiv
ity w
ithin
/24
0 1 2 3 4
.0.1
27.2
55
time [months]
IP a
ddre
ss a
ctiv
ity w
ithin
/24
0 1 2 3 4
.0.1
27.2
55
DE University DE University
Philipp Richter | INET / TU Berlin 11http://arxiv.org/abs/1606.00360
Activity Matrix at Scale
Philipp Richter | INET / TU Berlin 12http://arxiv.org/abs/1606.00360
20k adjacent IP addresses (in active /24s), University Network
Metric 1: Filling Degree per /24
Number of active IP addresses per /24 [1…256]
time [months]
IP a
ddre
ss a
ctiv
ity w
ithin
/24
0 1 2 3 4
.0.1
27.2
55
rather low (degree = 29)
time [months]
IP a
ddre
ss a
ctiv
ity w
ithin
/24
0 1 2 3 4
.0.1
27.2
55
Philipp Richter | INET / TU Berlin 13http://arxiv.org/abs/1606.00360
high (degree = 254)
Metric 1: Filling Degree per /24
Philipp Richter | INET / TU Berlin 14http://arxiv.org/abs/1606.00360
active IP addresses within /24
CD
F: a
ctive
/24
bloc
ks
1 64 128 192 256
0.0
0.2
0.4
0.6
0.8
1.0
Metric 1: Filling Degree per /24
Philipp Richter | INET / TU Berlin 14http://arxiv.org/abs/1606.00360
active IP addresses within /24
CD
F: a
ctive
/24
bloc
ks
1 64 128 192 256
0.0
0.2
0.4
0.6
0.8
1.0
only less than 50% of all active /24 blocks have
filling degree > 250
Addressing: Static vs. Dynamic
Philipp Richter | INET / TU Berlin 14http://arxiv.org/abs/1606.00360
active IP addresses within /24
CD
F: a
ctive
/24
bloc
ks
1 64 128 192 256
0.0
0.2
0.4
0.6
0.8
1.0
static all dynamic
• We tagged likely static/dynamic blocks using PTR records • We identified 262K static blocks and 456K dynamic blocks
Addressing: Static vs. Dynamic
Philipp Richter | INET / TU Berlin 14http://arxiv.org/abs/1606.00360
active IP addresses within /24
CD
F: a
ctive
/24
bloc
ks
1 64 128 192 256
0.0
0.2
0.4
0.6
0.8
1.0
static all dynamic
• We tagged likely static/dynamic blocks using PTR records • We identified 262K static blocks and 456K dynamic blocks
more than 70%of “static”-tagged blocks have filling degree < 64
Addressing: Static vs. Dynamic
Philipp Richter | INET / TU Berlin 14http://arxiv.org/abs/1606.00360
active IP addresses within /24
CD
F: a
ctive
/24
bloc
ks
1 64 128 192 256
0.0
0.2
0.4
0.6
0.8
1.0
static all dynamic
• We tagged likely static/dynamic blocks using PTR records • We identified 262K static blocks and 456K dynamic blocks
more than 70%of “static”-tagged blocks have filling degree < 64
more than 80% of “dynamic”-tagged blocks have filling degree > 250
Metric 2: Spatio-temporal Utilization
low utilization (18%)time [months]
IP a
ddre
ss a
ctiv
ity w
ithin
/24
0 1 2 3 4
.0.1
27.2
55
time [months]
IP a
ddre
ss a
ctiv
ity w
ithin
/24
0 1 2 3 4
.0.1
27.2
55
Philipp Richter | INET / TU Berlin 15http://arxiv.org/abs/1606.00360
sum(<active IP, day>)sum(all possible <active IP, day>)
= redred + grey
rather high (75%)
Dynamic addressing: Configuration/Pool sizes matter
Utilization: Blocks w/ > 250 active IPs
% of max possible spatio−temporal utilization
activ
e /2
4 bl
ocks
0 20 40 60 80 100
040
K80
K12
0K
Philipp Richter | INET / TU Berlin 16http://arxiv.org/abs/1606.00360
Utilization: Blocks w/ > 250 active IPs
% of max possible spatio−temporal utilization
activ
e /2
4 bl
ocks
0 20 40 60 80 100
040
K80
K12
0K
Philipp Richter | INET / TU Berlin 16http://arxiv.org/abs/1606.00360
majority of - likely dynamic - blocks show high utilization
Utilization: Blocks w/ > 250 active IPs
% of max possible spatio−temporal utilization
activ
e /2
4 bl
ocks
0 20 40 60 80 100
040
K80
K12
0K
Philipp Richter | INET / TU Berlin 16http://arxiv.org/abs/1606.00360
a third of - likely dynamic - blocks show low utilization
spatio-temporal utilizationtraffi
c contribution
rela
tive
host
cou
nt
Summary
Philipp Richter | INET / TU Berlin 17http://arxiv.org/abs/1606.00360
• Comprehensive study of IPv4 address activity • Metrics “beyond” binary notion of IPv4 activity • Can inform: Network operations, address [re]assigment • Can inform: Network security and host reputation
Figure: active /24 address blocks • Spatio-temporal utilization • Traffic contribution • Relative host count
Backup: IPv4 Traffic Consolidation
Philipp Richter | INET / TU Berlin http://arxiv.org/abs/1606.00360
months [2015]
% tr
affic
sha
re o
f top
10%
IPs
4950
5152
53
01 06 12
weeklymoving average (4 weeks)
Backup: Churn Visibility in BGP
Philipp Richter | INET / TU Berlin http://arxiv.org/abs/1606.00360
1 day 7 days 28 days
aggregation window size
% e
vent
s co
rrela
ted
with
BG
P ch
ange
0.0
0.5
1.0
1.5
2.0
2.5
up eventsdown eventsactive (no change)
Backup: Classification ICMP-only IPs
Philipp Richter | INET / TU Berlin http://arxiv.org/abs/1606.00360
ASes(N=2k)
BGP prefixes(N=55k)
/24s(N=495k)
IPs(N=77m)
0.0
0.2
0.4
0.6
0.8
1.0
server server/router router unknown
server identification: ZMap scans HTTP(S), POP3(S), IMAP(S)router identification: Ark, TTL exceeded received
ASes(N=51k)
BGP prefixes(N=460k)
/24s(N=6m)
IPs(N=950m)
0.0
0.2
0.4
0.6
0.8
1.0
CDN only CDN & ICMP ICMP only
Visibility CDN/ICMP ICMP-only hosts
Backup: IPv6 /64 Growth
Philipp Richter | INET / TU Berlin http://arxiv.org/abs/1606.00360
Mar-20
14
Apr-20
14
May-20
14
Jun-20
14
Jul-20
14
Aug-20
14
Sep-20
14
Oct-20
14
Nov-20
14
Dec-20
14
Jan-20
15
Feb-20
15
Mar-20
15
Apr-20
15
May-20
15
Jun-20
15
Jul-20
15
Aug-20
15
Sep-20
15
Oct-20
15
Nov-20
15
Dec-20
15
Jan-20
16
Feb-20
16
Mar-20
16
200 M
400 M
Active WWW client IPv6 /64 count