design and experience with supporting nat and firewall in
TRANSCRIPT
Design and Experience with Supporting NAT and Firewall in End System Multicast
by
Aditya Ganjam
Computer Science Department
School Of Computer Science
CARNEGIE MELLON UNIVERSITY
Submitted in partial fulfillment of the requirements
for the degree Master of Science
May, 2003
Contents
Acknowledgments 3
Abstract 4
1 Introduction 5
2 Background on NAT and Firewall 8
3 Related Work and Motivation 10
4 Solution Framework and Design 13
4.1 General Solution Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2 ESM Specific Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.3 Other Proposed NAT/Firewall Solutions in this Framework . . . . . . . . . . . . . . 15
5 NAT/Firewall Support in ESM 16
5.1 Overview of End System Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2 Solution for End System Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.2.1 OID-IP Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2.2 Achieving Bi-directional Communication . . . . . . . . . . . . . . . . . . . . . 19
5.3 Performance Implication of Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.3.1 Connectivity Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.3.2 Reduced Parent Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.3.3 Transport Layer Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1
6 Experience and Evaluation 23
6.1 Emulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.2 Experience from Real Internet Broadcasts . . . . . . . . . . . . . . . . . . . . . . . . 25
6.2.1 Distribution Of Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.2.2 Host Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
6.2.3 NAT/Firewall Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
6.2.4 Analysis of NAT/Firewall Support . . . . . . . . . . . . . . . . . . . . . . . . 30
7 Lessons Learned 34
7.1 Supporting NAT is Important and Challenging . . . . . . . . . . . . . . . . . . . . . 34
7.2 Key NAT/Firewall Factors For Application-Endpoint Systems . . . . . . . . . . . . . 35
8 Conclusion 37
Reference List 37
2
Acknowledgments
First, I would like to thank Srini for introducing me to the ESM group. I would like to thank
Hui for his guidance and also for allowing me to work on this very interesting project, Sanjay and
Yang-hua for working closely with me and guiding me on this project and Kay and Eugene for their
valuable support and feedback.
3
Abstract
End System Multicast(ESM)[2] is an overlay based architecture to enable multicast in the In-
ternet, and the application-endpoint model of deploying ESM shows tremendous promise because
there is no dependence on dedicated infrastructure. However, the presence of Network Address
Translators(NAT)[8] and firewalls[4] challenges the deployment of ESM and other such systems
because NATs and firewalls impose fundamental connectivity constraints. Although several net-
work level solutions have been proposed, none have been well deployed, therefore there is a need
for an application level solution to support NAT and firewall that is immediately deployable by
application-endpoint based systems. To achieve this goal in ESM, we have developed a two level
solution that first resolves the intricate addressing issues and increases the degree of connectivity
between pairs of hosts, and second optimizes performance of the overlay by incorporating NAT-
aware enhancements. With this solution, NAT and firewall hosts are able to receive content from
and provide content to public hosts.
Through the course of building and evaluating this solution using both controlled testbed
emulations and real Internet broadcasts, we have gained valuable experience with supporting NAT
and firewall hosts in ESM. The key lesson we have learned is that supporting NAT and firewall
is a more important and challenging problem for application-endpoint based systems than we had
initially considered because: (i) there exist realistic environments where these hosts consist of
a large portion of the participating hosts, close to 70% in our experience, (ii) they may lead to
unexpected performance degradation of even non-NAT/firewall hosts and (iii) a solution to support
them will likely result in a wide range of changes from the transport layer to the overlay protocol’s
key functionalities such as group membership in ESM. With regard to our goal, we have built a
complete solution in ESM, where we can sustain a reasonable fraction of NAT/firewall hosts with
good performance and have discovered and started incorporating promising mechanisms to improve
the solution for more extreme environments.
4
Chapter 1
Introduction
End System Multicast(ESM)[2] is a new architecture to enable multicast in the Internet. Its
strength lies in the fact that only the end systems are involved in the functionality to support
multicast, allowing it to be deployed without dependence on the network for support. Further, this
architecture provides a unique deployment model, where all entities involved in a multicast group
are in fact users of the application, meaning there is no dependence on dedicated infrastructure.
For example, consider a video broadcasting application using ESM, where the only entities that
are required to perform multicast functionality are the hosts that desire to view the broadcast.
This model, called application-endpoint, has become a promising model for deploying many overlay
based applications in the Internet and many systems are currently being developed for deployment
using this model. A clear example of another application is the Gnutella [5] file-sharing system.
Some other applications that may benefit from this model are distributed file systems and games.
Using the application-endpoint model, however, presents a new challenge to these systems:
constrained connectivity with the presence of Network Address Translators(NAT) and firewalls.
The key difficulty is due to the fact that these systems assume the underlying network can directly
connect any two participating hosts, an assumption that NATs and firewalls break. NATs and
firewalls fundamentally constrain communication between end hosts. For example, hosts behind
different NATs may not be able to communicate with each other. A second issue that NATs present
is due to the functionality of various kinds of NATs, where there will likely not exist a static one-to-
one mapping between a NAT host and its address. This stems from two sub-problems: non-unique
host addressing, where multiple hosts share the same address and address inconsistency, where a
host is identified using different addresses by different peers.
There is a necessity to address this problem in the application because no general, easy to deploy
network layer solution currently exists. Many people have advocated IPv6[3] as the solution to all
problems with NAT. This will only be true with a universal deployment of IPv6, which does not
5
seem feasible in the foreseeable future. Any partial deployment will still require NATs to translate
between IPv4 and IPv6 addresses as well as private IPv4 addresses. In addition, some connectivity
constraints are enforced for security reasons through the use of firewalls. Since security will always
be an issue, firewalls will continue to be used in some form.
Many may believe that the problems presented by NATs and firewalls can be easily “patched up”
in an auxiliary component during the deployment phase after other aspects of a system are working.
To the contrary, NAT and firewall issues may have impacts in the protocol and in certain cases
may deeply impact the architecture of the system itself. Many may also believe that the percent
of NATs and firewalls that will join the group will be low enough that either supporting them does
not justify the cost or that their participation can be minimized and the solution simplified. I will
show that a large percent of the hosts can be behind NAT or firewall and as a result minimizing
their participation may adversely effect the rest of the group.
In this thesis I propose an architecture for supporting NAT and firewall hosts in application-
endpoint systems and particularly in End System Multicast[2]. My solution has two components:
(i) a Connectivity Service that strives to directly connect end hosts and (ii) NAT/firewall aware en-
hancements for the application-endpoint system, which may be required for correctness or improved
performance. This split of functionality into two components has several advantages. First, the
Connectivity Service component can be made independent of the application, although currently
it is specific to ESM, while the NAT/firewall aware enhancements are specific to each application.
This allows for a general Connectivity Service component to be built that can potentially be utilized
in any system. Second, in most cases either component can be independently optimized for im-
provement in performance and depending on the application, different tradeoffs may exist between
the performance gain of an optimization and its cost.
Through the course of building and evaluating this solution using both controlled testbed
emulations and real Internet broadcasts, we have gained valuable experience with supporting NAT
and firewall hosts in ESM. The key lesson we have learned is that supporting NAT and firewall
is a more important and challenging problem for application-endpoint based systems than we had
initially considered because: (i) there exist realistic environments where these hosts consist of
a large portion of the participating hosts, close to 70% in our experience, (ii) they may lead to
unexpected performance degradation of even non-NAT/firewall hosts and (iii) a solution to support
them will likely result in a wide range of changes from the transport layer to the overlay protocol’s
key functionality such as group membership in ESM. I will also show results from a simulation
study using a trace taken from an Internet broadcast that analyzes the effectiveness of different
enhancements in both components of the solution and shows that realizing and taking advantage
of NAT specific details can greatly improve connectivity and, as a result, performance.
6
Our solution in ESM can sustain a reasonable fraction of NAT/firewall hosts with good per-
formance and we have discovered and started incorporating promising mechanisms to improve the
solution for more extreme environments. In the rest of this thesis, I will discuss the motivation
to support NAT and firewall hosts in the application along with related work in the area, my
solution to support NAT and firewall in End System Multicast, experience and evaluation results
from real Internet use of this solution, and ways of extending this solution to be used in other
application-endpoint systems.
7
Chapter 2
Background on NAT and Firewall
The fundamental challenge that NATs and firewalls present is constrained communication between
end hosts. These constraints depend on the properties of the NAT or firewall, the transport
layer used, and whether the hosts are behind the same NAT or firewall. Before describing these
constraints in detail, I first describe some properties of NATs. NATs can be classified into two
categories. Full Cone NATs can receive incoming packets to a port from an arbitrary host once
it sends a packet on that port to some destination. In contrast, Symmetric NATs allow incoming
packets only from the host that the NAT host sent a packet to1. To illustrate, consider figure 2.1.
In the case that the NAT is a Full Cone NAT, if host A sends a packet to host C resulting in
the translated IP and port (IPpub,Ppub), then host D could use (IPpub,Ppub) to communicate with
host A without host A ever directly sending a packet to D. In the other case, where the NAT is
a Symmetric NAT host D would not be able to use (IPpub,Ppub) to communicate with A, in fact
it would have to receive a new IP and port directly from A to communicate with it. I also view
firewalls in two categories: firewalls that filter UDP in either direction and firewalls that allow UDP
in both directions (possibly maintaining state for outgoing UDP packets). In addition I assume
firewalls only allow TCP connections to be initiated from within the private network. I currently
only consider the second category of firewalls, however the first category can be incorporated into
this solution in the future.
Given the above definitions the connectivity matrix in figure 2.2 characterizes the constraints
for the different NATs, firewalls and transport layers. Notice the significant impact of the transport
layer used, where communication between Full Cone NATs and other NATs and firewalls is possible
through UDP, but not TCP. The reason has to do with the connection initiation implementation of
TCP and the fact that it resides in the kernel and the requirement of Full Cone NAT described in
1The STUN[6] draft classifies NATs into four categories, but I only require these two currently. I classify the othertwo categories Restricted Cone NAT and Port Restricted Cone NAT as simple cases of Symmetric NAT. The richerclassification that STUN provides can be used in the future to further optimize connectivity.
8
����� ,QWHUQHW����� � �����������������
$
%
&
'
����� �������� ��������������
IPA: Apriv IPA: Apub1
IPA: Apub2IPB: Bpriv
Figure 2.1: Addressing issues with NAT
??UDPTCPFirewall
?XUDPTCPSymmetric NAT
UDPUDPUDPTCPFull Cone NAT
TCPTCPTCPTCPPublic
FirewallSymmetric NATFull Cone NAT
Public
ParentChild
TCP : can have connectivity through TCP and UDP
UDP : can only have connectivity through UDP
Figure 2.2: Connectivity Matrix
the definition above. In addition, because of the functionality of Symmetric NAT, communication
between two Symmetric NATs is not possible using either transport protocol unless host D could
somehow predict the translated port that the NAT would assign for a packet from A to D. The ?
in figure 2.2 indicates that communication is sometimes possible between such hosts and depends
on the implementation of the firewall.
In addition to maximizing connectivity, dealing with NATs and firewalls presents an important
addressing issue: there may not exist a one-to-one mapping between a NAT host and its address.
For example, consider again figure 2.1 where hosts A and B are behind the same NAT, and C and D
are public hosts. A must use Bs private address to communicate with it, but C must use Bs public
address to communicate with it. In addition, hosts C and D may use different public addresses to
communicate with host B (if B were behind a symmetric NAT). The implication here is that hosts
B,C and D must learn the address and port they must each use to communicate with host A and
may not be able to share these values with each other.
9
Chapter 3
Related Work and Motivation
The first and most important question in supporting NAT and firewall is where should this func-
tionality exist? Should it exist in the network alone and be transparent to the application, should
it exist as a network level solution that aids the application or should it exist in the application
alone? There have in fact been solutions developed in all three of these categories as illustrated in
figure 3.1.
Address Virtualization Enabling Service (AVES)[12] is a purely network level solution that uses
a waypoint architecture to address hosts behind NAT. It requires the addition of AVES-waypoints
and AVES-DNS servers, changes to the NATs and optionally changes to current DNS servers. In
AVES, hosts behind NAT are given a globally unique hostname, which is dynamically mapped
to an AVES-waypoint IP address when an external host initiates communication with the NAT
host by performing a DNS lookup. Communication from the remote host then goes through the
mapped AVES-waypoint to the NAT host and the reverse may go directly to the remote host
without going through the waypoint. The key feature that AVES provides is it allows any host to
initiate connections to a host behind NAT and does not require the application on either end to be
changed. However, since AVES is not currently deployed and it requires deployment of waypoint
infrastructure and changes to the existing NAT infrastructure, relying on it to support NAT and
firewall in a application-endpoint system would hinder deployment since the application-endpoint
system does not require any infrastructure deployment of its own.
Universal Plug and Play (UPnP)[1] is a solution proposed in the second category. It requires
changes to the NAT, the operating system and the application, but does not require any third-
party infrastructure to be deployed. UPnP gives the application control of mappings in the NAT. In
particular, the application can add and delete mappings and learn the public address of a mapping.
4+4[13] is another solution in the second category. 4+4 requires changes to the NAT, DNS and
the operating system. In 4+4, IP packets contain both the private and public addresses of a host
10
�����
��������� ������ ���
������������
�����
�������� �
IPv64+4UPnPSTUN
IPv64+4STUN
IPv6 IPv6*4+4UPnPAVES
IPv64+4AVES (optional)
AVESAVES
Required Changes
* IPv6 requires NATs to be removed
Figure 3.1: Required changes for other proposed solutions
behind NAT. When the packet is in the private realm, the private address is in the IP address
field of the packet and the public address is in an option field. When the packet is in the public
realm, the address are reversed. It is the goal of the NAT and the operation system to make sure
these invariants are held true. The drawback of this solution compared to UPnP is that it requires
DNS to be changed to return the private address of a NAT host. The advantage of this solution is
the NAT host does not have to actively manage mappings in the NAT for itself and multiple host
behind the same NAT can use the same public ports since they are no longer translated.
Both of these solutions may be slow to deploy in the Internet because the current NAT infras-
tructure must be changed and in the case of 4+4 the DNS infrastructure must be changed. Relying
on their deployment will hinder the deployment of an application-endpoint system for many years
to come. In addition, none of these solution address the connectivity constraints that firewalls
impose. As a result, solutions that work at the application level are required and should take
advantage of such network level solutions as they become available.
Gnutella[5] uses a solution in the application alone to transfer files between a public host and
a NAT or firewall host. If a public host needs to request a file from a NAT or firewall host, it sends
a message through the overlay requesting the NAT or firewall host to send it the file. This solution
however, does not allow file transfers between NAT or firewall hosts. In addition, the solution
is very specific to Gnutella and cannot be generalized to other peer-to-peer systems because: (i)
they do not address issues with NAT to NAT communication, which can become a key problem
11
in other applications and more structured systems and (ii) in some systems there may not exist
a routing mechanism that allows a host to request a NAT host to initiate communication with it.
Simple Traversal of UDP Through NATs (STUN)[6] is an application level protocol that allows
applications to discover the presence and types of NATs and firewalls they are behind and also the
public IP address allocated to them by the NAT. In addition, it provides several mechanisms to
enable UDP communication between certain types of NATs. Several mechanisms that the STUN
protocol uses for UDP connectivity are similar to ones that I use, however they do not explore
them in the context of aplication-endpoint systems.
Clearly there is a need for an application level solution for NAT and firewall that is immediately
usable by peer-to-peer systems. The following sections describe such a solution for End System
Multicast.
12
Chapter 4
Solution Framework and Design
A solution to support NAT and firewall hosts in any application-endpoint system can have a wide
range of complexity, where the tradeoff is connectivity and therefore the extent of NAT and firewall
host participation. In this section I will: (i) provide a framework that describes the solution space
for solving NAT and firewall issues in peer-to-peer systems, (ii) show how it applies to End System
Multicast and (iii) show how other proposed solutions fit into this framework.
4.1 General Solution Framework
We can view a solution that supports NAT and firewall hosts in peer-to-peer systems as a point in
a two-dimensional space as shown in figure 4.1. On the x-axis the solution addresses issues with
connecting pairs of hosts. As shown in the figure, there are several degrees to which connectivity can
be achieved, for example: connectivity can be achieved between public and NAT/firewall hosts only,
improved to connect Full Cone NATs to any NAT/firewall hosts or improved further to connect
hosts that are more restrictive in their connections, etc. Each improvement has different amounts
of complexity and connectivity gain. On the y-axis the solution strives to optimize performance
of the overlay using NAT/firewall aware modifications given a certain degree of connectivity on
the x-axis. It is important to note that the y-axis is only needed because there are fundamental
restrictions to the degree of connectivity that the solution can achieve on the x-axis. If the solution
could achieve 100% connectivity, then no optimizations on the overlay would be required.
Viewing the solution space in this framework allows for a two level architecture, where in
the bottom level a general Connectivity Service optimizes over the x-axis and application specific
NAT/firewall aware modifications to the overlay optimize over the y-axis. A simple interface is
provided between the two levels where the Connectivity Service categorizes hosts into two groups:
(i) connectable hosts who can communicate with any other hosts and (ii) constrained hosts who
13
Connectivity
Overlay Features
0% 100%Public-N/F Full NAT-N/F ...
:
NAT aware heuristics
N/F as Parent
N/F as Children
Better Performance
Better Performance
Figure 4.1: Solution Space in the context of ESM
can only communicate with the connectable hosts. Using this partition of hosts the overlay can
implement features that may be required for correctness of the system or optimizations that provide
increased performance.
4.2 ESM Specific Design
The solutions for connectivity on the x-axis are in most cases independent of the application of the
peer-to-peer system, however, the optimizations on the y-axis are specific to each application. I now
discuss some of the tradeoffs with NAT/firewall aware modifications and optimizations specific to
ESM. The simplest modification for ESM is to make NATs and firewalls simple clients. They do not
participate in the overlay protocol so they cannot be included in group membership or any overlay
optimizations, therefore their information (IP address) does not need to be distributed to others. In
addition they cannot be parents, so no other host will want to initiate communication with them.
This simple case requires the NAT or firewall host to initiate communication to some existing host
to receive data. Since the NAT or firewall host is not participating in the overlay protocol it cannot
recover from any congestion or parent leave events and cannot contribute bandwidth to the group.
An improvement from the the above simple case requires the NAT or firewall host to participate
in the overlay protocol. To accomplish this, the addressing issues mentioned in section 2 must be
resolved. In this solution, the NAT and firewall hosts still cannot be parents so no other hosts
will be interested in initiating communication with them. The advantage of this solution is that
the NAT and firewall hosts can optimize their performance along with the other members and
can recover gracefully from congestion or parent leaves. The drawback is that the system is not
utilizing the bandwidth resources available at the NAT and firewall hosts. In a bandwidth intensive
application-endpoint system such resources are very critical and should be optimized for. Therefore
a third option is to give the NAT and firewall hosts the ability to become parents. This raises many
14
complications because now public hosts are interested in initiating communication with NAT and
firewall hosts because they are potential parents. In 5.2 I discuss the design of the solution for ESM
and how some of these issues are resolved.
4.3 Other Proposed NAT/Firewall Solutions in this Framework
An advantage of this framework is that it encompasses other proposed solutions and allows us
to see how they can be utilized. UPnP and 4+4, for example, are network layer solutions that
optimize over the connectivity axis and whose presence can help improve performance. AVES
is an infrastructure based solution that actually provides 100% connectivity between hosts who
subscribe to the service, which can also improve performance. In contrast to these, STUN is an
application level protocol that provides mechanisms to optimize over the connectivity axis. The
Gnutella protocol uses an application level solution that provides connectivity between public and
NAT/firewall hosts and uses NAT/firewall aware modifications to provide file transfer capability
between them.
15
Chapter 5
NAT/Firewall Support in ESM
5.1 Overview of End System Multicast
This section introducecs End System Multicast and some of the protocol details that are directly
impacted by the existence of NATs and firewalls
End System Multicast (ESM) is an overlay based multicast architecture where end systems
perform all multicast funtionallity such as group management and data forwarding. Our imple-
mentation of ESM is a distributed self-organizing protocol that builds a source-based multicast tree
and incrementally improves it using learned measurements. A new host joins the group by contact-
ing the source and retrieving a random list of currently participating members. Group membership
information is maintained by each host and is propagated using a gossip-like protocol[]. In the
gossip protocol, each host periodically sends a subset of its membership list to a randomly chosen
host. To gather performance information about other memebrs in the group such as latency to that
memeber and bandwidth the memeber is receiving, each host will periodically probe a subset of its
membership list. After receiving responses to its probes, a host may decide to change its parent if
it beleives it will receive better performance at the new host. The parent change procedure is as
follows: (i) the child picks a potential parent that it believes will give it better performance and is
not already saturated with too many children, (ii) the child requests the potential parent to become
its parent, (iii) the parent either accepts or reject based on its current number of children and total
potential children and (iv) if the parent accepted the child, it initiates a TCP connection to the
child for data forwarding. A new host uses the same procedure as parent change when picking its
initial parent. It is important to note that this system uses UDP for all control traffic and TCP
for data traffic.
16
Connectivity
Overlay Features
Public-N/F Full NAT-N/F ...
:
NAT aware heuristics
N/F as Parent
N/F as Children0% 100%
High
Cost
Low Performance
Current Solution for ESM
Figure 5.1: Design choices for End System Multicast
5.2 Solution for End System Multicast
To solve the issues related to NAT and firewall in ESM, the framework described in section 4 is
used. The primary design choices were to pick the degree of connectivity as shown in on the x-axis
and the degree of overlay optimizations on the y-axis in figure 5.1. The design goals that lead
to these decisions were to keep the solution simple and implement features that will with high
probability result in significant performance improvement and maintain high transparency to the
overlay.
In terms of connectivity our choice was to achieve Public to NAT/firewall connectivity, which is
of course required. We chose not to implement Full Cone-NAT to NAT/firewall connectivity because
we did not have any data on the distribution of Full Cone and Symmetric NATs so we did not know
the performance gain. In addition, the cost of implementing Full Cone NAT connectivity was high
since it required a complete change of the transport protocol from TCP to a UDP based congestion
control and loss resilient protocol. In terms of the overlay features we choose to implement the
solution for NAT as parent because the cost was not high, although it did raise many issues related
to bi-directional connectivity such as: (i) public hosts may want to initiate UDP communication
to NAT/firewall hosts to pick them as parent and (ii) when a parent choice is made one of them
(the host behind NAT) must initiate a TCP connection to the other to start the data forwarding.
In order to achieve the desired connectivity, the addressing issues first need to be resolved. To
achieve unique addressing the solution uses a unique overlay identifier(OID) for each host and maps
it to a tuple containing the public and private IP addresses and ports. A necessary feature of the
OID is that it is decoupled from the IP address and port, meaning an OID may map to different IP
addresses and ports on different hosts. Once addressing is resolved, connectivity must be achieved
to the desired degree. In the case of ESM, we required public to NAT/firewall connectivity for both
UDP and TCP.
17
The following sections describe the details of both achieve unique addressing and bi-directional
connectivity.
5.2.1 OID-IP Binding
When a host A intends to join the group, it contacts the source, and is allocated an OID as part
of the protocol. The source creates a binding that maps the OID of A to its public and private
addresses and ports:
OIDA <=>< pubIPSA, privIPSA, pubPortSA, privPortSA >
This binding is distributed as part of the normal group membership propagation operations of the
protocol.
Each host B maintains the OID and associated binding for every other member A that it knows,
which includes the public address and port of host A that B must use while communicating with
A. We note that this is in general different than the public address and port of host A that the
source S uses to communicate with A. In particular, this is different if A is behind a Symmetric
NAT , but the same if A is behind a Full Cone NAT .
This binding is maintained during normal protocol operations and is referred to for translation
from an OID to IP address and port when B wishes to communicate with A. All functionality for
maintaining the binding, and for translation between an OID and < IPAddr, port > is handled in
the Connectivity Service Layer between the overlay layer and the transport layer as shown in figure
5.2. A clean API is provided between the overlay and connectivity layers.
The Connectivity Layer performs the following functions:
• Learns and maintains bindings: There are two ways for a host B to learn bindings for host
A. First, it can learn the binding as part of the group membership operations when it first learns
about A’s existence. Second, it may receive packets directly from A. While the bindings learnt
in the two ways is identical if A is behind a Full Cone NAT , they are different if A is behind a
Symmetric NAT . Thus, bindings learnt by the second method are prioritized. An implication of
this is while B can initiate communication to hosts behind Full Cone NATs once it learns about
them, it cannot initiate communication to hosts behind Symmetric NATs until it explicitly receives
a packet from such hosts. Typically, such packets are received as part of the group membership
and probing algorithms of the protocol.
• Performs translations: When B wishes to send a packet to host A, the packet is sent to the
public IP address and port of A as present in the binding, in general. However, if A and B are
hosts behind the same NAT, then, the private address and port of A are used. To determine if A
18
�����������¢ �������
������¢ ��������
������������ �����
��������� ��������
� ������� ��������������� ������� �� !#"���$ $ % ����$ &'%�( ") ��* $ * �$ � !�% �+���,!�$ * %�� * -����!���.#.# �(
/
01����2�3 ��456���72������ �7�
8:9 �,( ; / <( % =��
>�>@?
�,( . A�* � - �����? !#"
,��� ; %�%�"�=+�
B ( ���. ; $ � 8 ) C $ % ) DE �F�� �$ ��GH ,�+��* � <B ( ,��. ; $ � ) D $ % 8 ) C
I � 56�J�72���,,� ���
GroupMembership
K %�%�"�=+��GL ,�+��* � <
Figure 5.2: Connectivity Service for End System Multicast
is behind the same NAT or not, we currently use a simple heuristic based on prefix matching the
public addresses of A and B. However, this heuristic could wrongly label A as belonging to the
same NAT as B. This situation is determined by B not receiving packets from A for a while, when
it marks A as belonging to a different NAT.
5.2.2 Achieving Bi-directional Communication
In the case of ESM, bi-directional communication means communication can be initiated by: public
to public, NAT to public, firewall to public, public to NAT, or public to firewall. The first three
cases are trivial and need no extra support since public hosts do not pose any constraints. Public
to NAT communication has two cases: when the NAT is a Full Cone NAT and when the NAT
is a Symmetric NAT. In the first case, as long as all hosts use the same address and port to
communicate with the Full Cone NAT host, communication will proceed. However, in the second
case, the Symmetric NAT host must send a message to each host that wishes to communicate with
it, but of course the Symmetric NAT host will not know this list of public hosts. The solution
we use here is probabilistic probing, where the Symmetric NAT host probes a random subset of
hosts every T seconds. Every host that it has recently probed (within a certain timeout specific to
the NAT), will be able to communicate with it. In ESM, this probing is already done as part of
the measurement probing algorithm and no extra probes are required. In section 5.3 I analyze the
effectiveness of this algorithm.
The final case of connectivity is public to firewall. Depending on the implementation of the
firewall this case will be equivalent to public to public or public to NAT communication and
therefore no additional support is required. As mentioned before, this solution does not handle
firewalls that block UDP traffic in either direction.
19
Once two hosts agree to create a parent child link, they will need to open a TCP connection for
data. We accomplish this by using bi-directional connection initiation, meaning both parent and
child attempt to open a connection to the other, regardless of NAT or firewall. If one is public and
one is NAT or firewall, then only one of the connections will be successful. If both are public, then
both connections will be successful and the connectivity service arbitrates over the two by picking
the connection that was initiated by the host with lower IP address.
5.3 Performance Implication of Solution
Since total connectivity cannot be guaranteed by the connectivity service, there are several limita-
tions. The most significant limitation is that a certain subset (the constrained hosts) of the hosts
will not be able to communicate with each other as shown in the connectivity matrix in figure
2.2. There are two implications of this: (i) the system can get saturated without any bandwidth
constraints, where new constrained hosts cannot join, and (ii) constrained hosts have a reduced set
of choices for their parent, specifically they can only pick from connectable hosts. A limitation is
inflicted on the set of connectable hosts by the existence of hosts behind Symmetric NAT (which
are part of the constrained hosts), where a connectable host can communicate with a Symmetric
NAT host only if it previously received a message from that Symmetric NAT host. The implication
here is that connectable hosts have a reduced set of choices for their parents also, although not as
limited as constrained hosts.
5.3.1 Connectivity Limit
As the percent of constrained hosts in the group increases, there is a limit beyond which a connected
tree cannot be built. This limit depends mainly on the average out degree of the connectable hosts.
Fromally, to be able to construct a connected tree the ratio of constrained hosts to connectable
hosts must be less than or equal to the out degree of the connectable hosts, which is equivalent to
the following inequality:
n
1 − n<= d (5.1)
where n is the fraction of constrained hosts and d is the average out degree of connectable hosts. If
n/1-n equals d then, in the optimal case, each connectable host will be the parent of only constrained
hosts and will be the child of a constrained host. For example, to support a group where 80% of
the hosts are constrained hosts, the average out degree of connectable hosts must be at least 4.
20
0
0.2
0.4
0.6
0.8
1
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
Fra
ctio
n of
Sym
met
ric N
AT
s
Group Size
Expected Fraction of Symmetric NATs Visible Vs Group Size
Expected Fraction of Symmetric NATs Visible
Figure 5.3: Expected fraction of Symmetric NAT hosts reachable by a public host
5.3.2 Reduced Parent Selection
The second implication is reduced parent selection, where constrained hosts can only pick their
parents from the set of connectable hosts in the group. As a result they may not be able to choose
an optimal parent in terms of bandwidth and latency and may suffer in performance. An added
complication here is if the constrained host is itself a parent then all of its descendents will suffer
at least the same performance degredation.
If there are hosts behind Symmetric NAT, then a connectable host will be able to communicate
with one of them only if the connectable host previously received a packet from that Symmetric
NAT host. In ESM, these packets are generally received through the probing mechanism, however
in other protocols these packets may need to be sent in addition. To analyze the effectiveness
of a random probing mechanism, let Q equal the number of Symmetric NAT hosts a public host
receives a packet from in a period of time T in seconds. I assume the Symmetric NAT will timeout
translations created by packets greater than T seconds old. The expected value of Q, or the expected
number of Symmetric NAT hosts reachable to a public hosts is:
E[Q] =N∑
k=0
k
(
N
k
)
(
r
P
)k (
1 −
r
P
)N−k
(5.2)
where N is the number of Symmetric NAT hosts, P is the number of connectable hosts and r is
the number of random probes sent by each Symmetric NAT host in a time T. It is very desirable
to keep the probe rate constant to be able to scale to large group sizes, however the implication of
this is that the expected fraction of Symmetric NAT hosts reachable by a public host will decrease
21
with increased group size. Figure 5.3 shows this expected fraction where the percent of NAT hosts
is held constant at 50% and the probe rate is held constant close to 5 per second, which is what we
use in practice in the broadcast I will analyze in the evaluation, and the group size is varied from
10 to 5000. The figure shows that the random probing algorithm, in this case, scales up to a group
size of 1500. This value is directly related to the probe rate, which if held constant will always show
a similar trend. The important result is that the random probing mechanism does not scale to very
large group sizes with a constant probe rate. The key to solve this problem is to notice that a public
host will not need to communicate with all the Symmetric NAT hosts, but only a limited subset
of them. To determine the relavent subset we can take advantage of any clustering mechanisms or
neighborhood information in the overlay. In ESM, locality based clustering or hierarchies may be
used to scale to larger group sizes, which can be used to more intelligently probe those members in
the same or nearby clusters.
5.3.3 Transport Layer Implications
An important issue comes into play because of our choice of TCP for data packets and UDP for
control packets. As shown in Table 2.2, communication between certain pairs of NATs (for example,
a pair of Full Cone NATs) and firewalls is possible using UDP, but not using TCP. This could lead
to the following sequence of events: (i) a NAT host A learns about another NAT host B (through
the group membership protocol); (ii) A sends a control message requesting B to be a parent; and
(iii) B sends a control message to A accepting it as a child. However, since actual data flow is setup
using TCP, neither A nor B are able to setup a connection to each other, and A receives no data
for a while. To handle this, our protocol currently has explicit mechanisms that try and prevent
control messages between pairs of NATs, however it does not have mechanisms that prevent such
a problem with firewalls allowing the possibility of increased loss for firewall hosts.
22
Chapter 6
Experience and Evaluation
To evaluate the solution for NAT and firewall in ESM we used two separate evaluation method-
ologies: controlled testbed emulations and real Internet broadcasts. The emulations were run on
the planetlab[7] distributed testbed with 15 individual machines and several virtual hosts on each
machine with a total of 37 hosts. The testbed is located mainly in the US, but also in Europe
and Canada. Each host ran a NAT/firewall emulator that emulates the fundamental connectivity
constraints between NAT/firewall hosts assuming all hosts use TCP for data transfer. This means
NATs and firewalls cannot communicate with other NATs and firewalls but can freely communicate
with public hosts. The emulation however did not consider any issues with Symmetric NAT hosts.
The second and more realistic evaluation methodology was based on real Internet broadcasts.
Using the End System Multicast architecture, we have developed a complete Internet video broad-
casting system, which a producer connected to the Internet can use to broadcast content to any
receiver on the Internet. Using this system we have conducted several broadcasts of conferences
and lectures and I focus on one large broadcast to the Slashdot[10] community. This broadcast
consisted of a 24-hour program of archived video of potential interest to the Slashdot community
and reached over 1300 individual participants, 908 of which were behind NAT, and peaked at 140
hosts. Table 6.1 summarizes the extent of the deployment and characteristics of the participating
hosts. In addition to the hosts shown, we had provisioned about 20 way-point hosts to increase the
bandwidth of the system.
6.1 Emulation Results
In the emulation studies there were effectively no bandwidth constraints since all hosts were con-
nected through Internet2. Also, there were no group dynamics (all hosts joined at the beginning)
and the group size was held constant. The variable that we experimented with was the percent
23
Slashdot broadcast 12/2002 2pm-10:30pm (total 1316 hosts)Region North America (967) Europe (185) Oceania (48) Asia (8) Unknown (107)Background Home (825) University (127) Industry (85) Government (80) Unknown (199)Connectivity Cable Modem (490) 10+ Mbps (258) DSL (389) T1 (46) Unknown (133)NAT NAT (908) Public (316) Firewall (92)
Table 6.1: Host distributions for two broadcast events (shown only for a certain period of time)
200
220
240
260
280
300
320
340
360
380
0 5 10 15 20 25 30 35 40
Ban
dwid
th (
Kbp
s)
Host Rank
Ranked Average Bandwidth Per NAT Percent
0 % NAT10 % NAT30 % NAT50 % NAT70 % NATNAT Hosts
Public Hosts
Figure 6.1: Average bandwidth of hosts in the emulated experiments
of emulated NAT/firewall hosts in the group. Figure 6.1 shows the average bandwidth of hosts
averaged over three 10 minute runs at different NAT percent. The figure shows a gradual reduction
of received bandwidth with increased NAT/firewall percent that becomes clearly noticeable beyond
30% NAT/firewall hosts. This means that with fairly moderate NAT/firewall percent this solution
is able to support NAT and firewall and maintain acceptable bandwidth to all hosts, however when
the NAT/firewall percent increases there are severe performance issues. Further, the types of hosts
receiving reduced bandwidth is mixed among NAT and non-NAT hosts as shown by the + and x
marks representing individual runs.
The average bandwidth metric shows the overall performance of the system. I now look at
the join time, which is defined as the time the first data packet is received. This metric gives
an idea of the time to initially stabilize the system. Figure 6.2 is the average join time for the
same runs and shows a significant increase in join time for the 70% case. Similar to the average
bandwidth, join time also seems to perform well with a moderate percent of NAT/firewall hosts,
however deteriorates when the percent is increased.
24
0
10
20
30
40
50
60
70
80
90
100
0 5 10 15 20 25 30 35 40
Join
Tim
e (s
ec)
Host Rank
Ranked Average Join Time Per NAT Percent
0 % NAT10 % NAT30 % NAT50 % NAT70 % NATNAT Hosts
Public Hosts
Figure 6.2: Average join time of hosts in the emulated experiments
6.2 Experience from Real Internet Broadcasts
The emulated evaluations show that the solution does work and performance is acceptable with
reasonable NAT/firewall percent in certain constrained cases. However to gain experience on how
the solution works in a realistic setting with group dynamics, heterogeneous bandwidth and real
NAT/firewall distributions, I now analyze the results from the Slashdot broadcast described above.
6.2.1 Distribution Of Hosts
One question that comes up when supporting NAT and firewall hosts is: in a peer-to-peer appli-
cation, are there environments where a large percent of hosts are behind NAT or firewall? In fact,
the Slastdot broadcast had close to 70% NAT and firewall hosts as shown in table 6.1. Figure 6.3
gives a better idea of how the number of NAT hosts changed over time for a period of 8 hours
during the beginning of the broadcast. The top curve shows the total number of hosts present in
the group and the curve below it shows the number of NAT/firewall hosts in the group. Clearly
the trend is maintained throughout the duration shown. The next question to ask is how much
bandwidth do NAT/firewall hosts have? Our system differentiates hosts by the amount of outgoing
bandwidth they have, specifically hosts that have enough bandwidth to support several children
(DSL/CABLE/T1 versus T3 and above). The bottom two curves in figure 6.3, which show the
25
0
20
40
60
80
100
120
140
160
10000 15000 20000 25000 30000 35000 40000
Cou
nt
Time (sec)
Group Distribution: Number of NAT
Total HostsNAT/Firewall Hosts
Total HighBW HostsNAT/Firewall HighBW Hosts
Figure 6.3: Distribution of NAT/firewall hosts and their bandwidths
total number of hosts that can support children and the number of these hosts that are behind
NAT or firewall, shows that about half of the high bandwidth hosts were behind NAT or firewall.
6.2.2 Host Performance
Given the distribution of hosts regarding connectivity and bandwidth resources, I now look at
performance of hosts in the system using two different metrics: average received bandwidth and
recovery time after a large tree event, such as a host that has a large number of descendants leaving.
While the first metric shows the overall performance of the system, the recovery time shows the
transient performance. Using both of these metrics I will compare performance between public
hosts and NAT/firewall hosts.
Average Bandwidth In the Slashdot broadcast, we used three video streams summing up to 420Kbps,
however we are using a variable bitrate encoder and in our experience we have noticed an average
bandwidth close to 400Kbps. There are two methods to consider average bandwidth of an end host
since each host may join multiple times: (i) treat each join as an individual entity or (ii) combine
multiple joins of each host and treat them as one entity. After analyzing both methods, there was no
significant difference in the overall result so I only report on the first method. Figure 6.4 (a) depicts
the cumulative distribution of the average received bandwidth for public hosts and NAT/firewall
hosts, showing a slight, but not very significant, reduction of average bandwidth received by the
26
0
50
100
150
200
250
300
350
400
450
500
0 10 20 30 40 50 60 70 80 90 100
Ban
dwid
th (
Kbp
s)
Cumulative Percentage of Hosts
Average Bandwidth Public vs NAT/Firewal
Public HostsNAT/Firewall Hosts
0
50
100
150
200
250
300
350
400
450
500
0 10 20 30 40 50 60 70 80 90 100
Ban
dwid
th (
Kbp
s)
Cumulative Percentage of Hosts
Average Bandwidth High vs Low Outgoing Bandwidth Hosts
NAT/Firewall HostsNAT/Firewall Low Bandwidth Hosts
NAT/Firewall High Highbandwidth Hosts
(a) (b)
Figure 6.4: Cumulative distribution of average bandwidth received by end hosts
NAT/firewall hosts as compared to the public hosts. In addition, the average bandwidth received
by the bottom 25% of the public hosts is significantly degraded.
Recovery Time I now compare the performance using recovery time from an event, such as congestion
or parent leave, that caused a host to experience packet loss. The recovery time is defined as the
period of time that a host experiences continuous loss. Figure 6.5 shows the average recovery time
for public hosts and NAT/firewall hosts on a logarithmic scale. The recovery time for NAT/firewall
hosts is clearly higher than for public hosts, especially at the tail, and similar to average bandwidth,
the recovery time for public hosts is not very good for the top 10% of the hosts.
There are two question that arises from both of these results:
• What caused the extra degradation of performance for the NAT/firewall hosts ?
• Why is the performance not very good for public hosts ?
To address the first question I look at two key differences between the set of public hosts and
NAT/firewall hosts that can effect performance: (i) there exists a significantly larger population
of low bandwidth hosts (DSL,Cable Modem) in the NAT/firewall set versus the public set and
(ii) the NAT/firewall hosts have constrained connectivity due to the fact that they are behind
a NAT or firewall. To analyze the first case I split the two sets further into hosts with high
bandwidth connections and hosts with low bandwidth connections. To split each set I considered
two techniques: the user’s initial choice of high or low bandwidth connection and measurements we
27
10
100
1000
10000
75 80 85 90 95 100
Rec
ovar
y T
ime
(sec
)
Cumulative Percentage of Hosts
Average Recovary Time Public vs NAT/Firewal
Public HostsNAT/Firewall Hosts
Figure 6.5: Cumulative distribution of average recovery time received by end hosts
had taken during the broadcast to infer the connection speed of the hosts. I chose to use the second
method because it gives a slightly more accurate differentiation between the hosts, however the
end result does not change with either method. Figure 6.4 (b) shows the cumulative distribution
of average received bandwidth after splitting the NAT/firewall hosts. Clearly, the performance of
the low bandwidth hosts is very close to the overall NAT/firewall average, and in fact, most of the
high bandwidth hosts receive equivalent or degraded performance. The bandwidth constraint on
each host does not seem to cause the reduction in bandwidth for NAT/firewall hosts. I analyze the
effect of the connectivity constraint in section 6.2.3.
One hypothesis as to why the performance (reduced bandwidth/increased recovery time) of
public hosts was low is: since the NAT/firewall hosts can be parents, any reduction in bandwidth
they suffer will be transferred to all their children, who must be public hosts. To get an idea of how
many children NAT/firewall hosts had, figure 6.6 shows the cumulative distribution of the average
number of children for NAT/firewall hosts. This shows that the top ten percent of NAT/firewall
hosts have an average of 2-3 children. In addition, any descendants of these NAT/firewall hosts
arbitrarily below them in the tree will be effected.
The main conclusion from the results in the section is bandwidth was not the major factor in the
degradation of performance. The next section analyzes the impact of the connectivity constraint.
28
0
0.5
1
1.5
2
2.5
3
3.5
0 10 20 30 40 50 60 70 80 90 100
Ave
rage
Num
ber
of C
hidr
en (
max
6)
Cumulative Percentage of Hosts
Cumulative Distribution of Average Number of Chidren
NAT Hosts
Figure 6.6: Cumulative distribution of the average number of children for NAT/firewall hosts
6.2.3 NAT/Firewall Interactions
How do the connectivity constraints degrade performance for NAT and firewall hosts? To analyze
this I recall from section 5.3 that in the current solution NAT/firewall hosts cannot choose other
NAT/firewall hosts as parent and therefore they have a reduced set of potential parents. This
reduction is very critical because during a broadcast the set of potential parents gets dynamically
reduced further due to parents being saturated with their maximum number of children (currently
set statically to 4 or 6) or potential parents experiencing poor performance such as low bandwidth.
Therefore, an informative metric is the number of good hosts over time, where a good host is one
that fulfills connectivity requirements due to NAT/firewall, is not saturated, and received acceptable
bandwidth. Figure 6.7 (a) shows the number of good hosts over time during a peak of hosts. The
top curve shows the number of hosts in the group, the Deg 4-6 curve shows the total number of
hosts that can be parents regardless of connectivity constraints and the lower four curves shows
the number of good hosts in the system for NAT/firewall hosts taking different thresholds for
acceptable bandwidth. Surprisingly, the number of good hosts dips very low during parts of the
duration shown. An effect of this is longer time to recover from parent leaves/deaths or congestion
on the path to the source and longer time to join the group initially. All of these will contribute
to degrading the average bandwidth received. To compare this to the number of good hosts for
public hosts consider figure 6.7 (b), which shows results in the same format. Although the number
of good hosts dips for public hosts, it is considerably higher than the number of good hosts for
29
0.8 1 1.2 1.4 1.6 1.8
x 104
0
20
40
60
80
100
120
140
160NAT
Time (sec)
Num
ber
of H
osts
All hostsUnsatGood > 300Good > 350Good > 400Deg 0Deg 4−6
0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6
x 104
0
20
40
60
80
100
120
140
160
Time
Num
ber
of H
osts
All hostsUnsatGood > 300Good > 350Good > 400Deg 0Deg 4−6
(a) (b)
Figure 6.7: Good Hosts for NAT and public
NAT hosts. The main conclusion from this is because of the connectivity constraints of NAT hosts,
they had very few parents to choose from, especially at busy periods and this could contribute to
degrading their performance. A question that remains is how much was the system constrained
due the available bandwidth and constrained connectivity? This is addressed in the next section.
6.2.4 Analysis of NAT/Firewall Support
In this section I investigate the fundamental ability of the system to function in environments con-
strained by bandwidth and connectivity and analyze the effectiveness of optimizations in supporting
NAT and firewall hosts discussed in section 4. In order to do this I use the Feasibility Index, or
the ratio of the number of children the members currently in the group can sustain to the number
of members in the group for a particular source rate. A feasibility index of 1 indicates that the
system is saturated and a ratio less than 1 indicates that some of the participating hosts in the
broadcast would either be rejected or required to receive degraded bandwidth. In this analysis I
compare the feasibility index for an optimal overlay running with the Slashdot host distribution,
where an optimal overlay is one that builds the best possible tree to support NATs and firewalls,
the users always choose their correct bandwidth, and there are no added waypoints.
First, I compare in the top and bottom curves in figure 6.8 (a) the feasibility index of the
optimal system without and with the constraint of connectivity, where I assume that NAT and
firewall hosts cannot be parents of any hosts. Interestingly, the curve, which was above 1.5 allways
without connectivity constraints, drops below 1 at several instances with connectivity constraints
30
0
0.5
1
1.5
2
2.5
10000 15000 20000 25000 30000 35000
Fea
sibi
lity
Inde
x
Time (seconds)
Optimal - No Connectivity ConstraintNAT/Firewall is Child / Public-NAT/Firewall
NAT/Firewall is Parent / Public-NAT/Firewall
0
0.5
1
1.5
2
2.5
10000 15000 20000 25000 30000 35000
Fea
sibi
lity
Inde
x
Time (seconds)
Optimal - No Connectivity ConstraintNAT/Firewall is Parent / Public-NAT/Firewall
NAT/Firewall is Parent / Full NAT-NAT/Firewall
(a) (b)
Figure 6.8: Feasibility index of the system (a) comparing support for NATs and firewalls as parentsand not-parents keeping constant support for connection only between NAT/firewall and publichosts and (b) comparing support for Full NAT to NAT connectivity keeping constant the optimiza-tion of supporting NATs and firewalls as parents
and is close to 1 most of the time. Therefore, a system that uses a naive solution of supporting NAT
and firewall hosts as just children cannot function well in this environment even with an optimal
overlay. Next, I compare in the bottom two curves in figure 6.8 (a) the feasibility index of this naive
solution to that of a solution where NAT and firewall hosts can be parents of public hosts. These
to techniques come from the overlay optimization axis from figure 4.1 and clearly, the feasibility
index improves significantly with this optimization and approaches the fundamental limit which is
represented by the curve assuming no connectivity constraints.
To compare techniques from the connectivity axis in figure 4.1, I consider the improvement from
Public to NAT/firewall connectivity to Full Cone NAT to NAT/firewall connectivity assuming that
NAT and firewall hosts can be parents. Figure 6.8 (b) shows that this improvement essentially
eliminates all the connectivity constraints in the Slashdot distribution. Intuitively, this means that
the number of public hosts is greater than the number of children that NAT and firewall hosts can
support allowing an optimal overlay to fill up all children of NAT and firewall hosts. As a result,
any new host, public or NAT/firewall, can choose any unsaturated parent and the connectivity
constraint does not come into effect (since all children of NAT/firewall hosts have been filled up).
This may seem to be contradictory to the distribution of hosts in the Slashdot broadcast, however
it is clear when we consider the number of Full Cone NAT hosts in the system, shown in figure
6.9, because with the improvement their connectivity restrictions are equivalent to those of public
31
0
20
40
60
80
100
120
140
160
10000 15000 20000 25000 30000 35000 40000
Num
ber
of H
osts
Time (sec)
Group Distribution Full NAT
Total MembersNAT Hosts
Full NAT Hosts
Figure 6.9: Distribution of Full NATs
hosts. In addition, it is also useful to know the residual capacity of the Full Cone NAT hosts.
Figure 6.10 shows the cumulative distribution of public and Full Cone NAT hosts as a function
of the average number of children. It clearly shows that the Full Cone NAT hosts have a large
amount of residual capacity. This is due to the fact that most of the hosts are NAT/firewall hosts
and cannot pick Full Cone NAT hosts as parent, however, by implementing the solution for Full
Cone NAT to NAT/firewall connectivity we can greatly increase the utilization of these hosts.
32
0
1
2
3
4
5
6
0 10 20 30 40 50 60 70 80 90 100
Ave
rage
Num
ber
of C
hidr
en (
max
6)
Cumulative Percentage of Hosts
Cumulative Distribution of Average Number of Chidren
Public HostsFull Cone NAT Hosts
Figure 6.10: Cumulative distribution of the average number of children for public and Full ConeNAT hosts
33
Chapter 7
Lessons Learned
In this section I will discuss several lessons we have learned through experience supporting NAT
and firewall in ESM. These include lessons from the results of our broadcasts relating to the issues
and challenges we faced and lessons about the key factors when supporting NAT and firewall, which
we learned through a better understanding of the details of NAT and firewall.
7.1 Supporting NAT is Important and Challenging
There are several reasons why supporting NAT/firewall is important and challenging: (i) there
exist environments with a large population of NAT/firewall hosts, (ii) performance issues may arise
that effect public hosts, as well as, NAT/firewall hosts and (iii) supporting NAT/firewall well will
likely require changes at several places in an application-endpoint system.
• Harsh Environments From the emulation results we see that supporting a moderate number of
NAT/firewall hosts can be done with minimal performance issues, however with more extreme
populations the performance may be severely impacted. The Slashdot broadcast showed a
real-world environment with a large population of NATs and showed the performance issues
that arise. Therefore, for an application-endpoint system to function well in a real Internet
environments, it must be able to handle large NAT/firewall populations well
• Performance Impact The Slashdot broadcast clearly showed the performance issues faced by
NAT/firewall hosts, but more importantly, it showed that public hosts may also have degraded
performance with the presence of NAT/firewall hosts.
• Complexity of Supporting NAT and firewall As solution framework described in section 4, there
are tradeoffs in the design choices when supporting NAT/firewall. One choice we had made
was to not implement support for Full NAT-NAT/firewall connectivity because it required a
34
Connectivity
Overlay Features
Public-N/F Full NAT-N/F ...
0% 100%
Necessary For ESMCurrent Solution
for ESM
Low Performance
:
NAT aware heuristics
N/F as Parent
N/F as Children
Figure 7.1: Solution Space
significant change of the transport protocol. After the Slashdot broadcast we learned that
the cost of this is justified by the necessary performance gain it will result in. In addition,
we have implemented NAT-aware enhancements for the membership and probing protocols.
Therefore, a good solution to support NAT and firewall will likely require a wide range of
changes. Figure 7.1 shows the planned solution for ESM.
7.2 Key NAT/Firewall Factors For Application-Endpoint Systems
We have identified three key factors to consider when supporting NAT and firewall hosts in an
application-endpoint system: system saturation, constrained neighbor selection, and impact of
neighbor leaves. An example of saturation in ESM was shown in section 5.3. For another example
consider the Chord[11] lookup service. Without NAT or firewall hosts, Chord can always accept a
new host into the group, however with the existence of NAT and firewall hosts, it is possible for
NAT or firewall hosts to be rejected. This occurs when every position that a new host can take
requires it to have a NAT or firewall host as a neighbor, which of course is not possible if the new
host is a NAT or a firewall host itself. This point of saturation will exist in all peer-to-peer systems
that support NAT or firewall, however it can be vastly different in different applications because a
system that imposes very little structure on the hosts usually has a high saturation point, such as
in Gnutella where saturation directly depends on the maximum number of neighbors of a host.
The second factor is constrained neighbor selection. When a NAT or firewall host wishes to
choose a neighbor, it can only do so from a subset of the hosts present. For example, when a new
host wishes to join a CAN[9] group, it must choose a point in the d dimensional space of the group.
When a NAT or firewall host wishes to join, a necessary condition is that it must choose a point
in the d dimensional space where at least one of its neighbors will be a public host. In addition,
the more NAT/firewall hosts it has as neighbors, the longer the number of hops it will need to
send and receive packets as shown in figure 7.2. In this example of a 2-dimensional CAN, NAT (A)
35
PublicNATNATPublic
NATPublicNATNAT
NATNAT (A)NATPublic
NATNAT (B)PublicNAT
Figure 7.2: CAN with NAT
wishes to send a message to its neighbor NAT (B), however, this messages must travel significantly
more hopes to reach NAT (B) because of the presence of NATs throughout the network. Therefore,
the number of NAT/firewall hosts in a CAN group will significantly influence the average number
of hops to route a message. In Chord this factor is extremely important because hosts are not
given a choice of their neighbors, but in fact are assigned neighbors based on a hash of a unique
identifier, such as public+private IP address in the case of NAT. If the result of the hash places a
NAT/firewall host where both its neighbors are behind NAT or firewall, then the Chord ring will
be broken.
A third factor that becomes very important when supporting NAT and firewall hosts is the
impact of leaves and deaths of hosts present in the system. Consider a public host in Chord whose
neighbors are both NAT or firewall. If this host leaves, then its neighbors are forced to become
neighbors of each other, which is not possible in this case.
36
Chapter 8
Conclusion
The application-endpoint model for deploying overlay based systems is very promising, however it
introduces new challenges for these systems. In this thesis I address the challenge of constrained
connectivity imposed by the presence of NAT and firewall in the context of End System Multicast.
We have built a complete solution to support NAT and firewall hosts in ESM, where they are able
to receive content from and provide content to public hosts. The solution has two components:
a Connectivity Service that resolves the intricate addressing issues and increases the degree of
connectivity between pairs of hosts and NAT-aware enhancements to the overlay for correctness or
improved performance.
Through the course of building and evaluating this solution using both controlled testbed emu-
lations and real Internet broadcasts, we have gained valuable experience with supporting NAT and
firewall hosts in ESM. Through our evaluation we have seen that the solution can sustain a reason-
able fraction of NAT and firewall hosts with good performance. Further, through our deployment
of ESM with this solution we have learned that supporting NAT and firewall is an important and
challenging problem for application-endpoint systems because there exist harsh environments that
impose unexpected performance impacts and may require increased complexity of the solution.
Also, from our deployment we have discovered and started incorporating promising mechanisms to
improve the solution for these more extreme environments. Finally, through developing the solution
we have determined several key factors that many application-endpoint systems will face with the
presence of NAT and firewall.
37
Reference List
[1] Understanding Universal Plug and Play. Microsoft White Paper.
[2] Y. Chu, S.G. Rao, and H. Zhang. A Case for End System Multicast. In Proceedings of ACM
Sigmetrics, June 2000.
[3] S. Deering and R. Hinden. Internet protocol, Version 6 (IPv6) Specification, rfc-2460, Decem-
ber 1998.
[4] N. Freed. Behavior of and Requirements for Internet Firewalls. RFC-2979, October 2000.
[5] Gnutella. http://www.gnutella.com/.
[6] C. Huitema J. Rosenberg, J. Weinberger and R. Mahy. STUN - Simple Traversal of UDP
Through Network Address Translators. IERF-Draft, December 2002.
[7] Planetlab. http://www.planet-lab.org/.
[8] P.Srisuresh and K.Egevang. Traditional IP Network Address Translator (Traditional NAT).
RFC-3002, January 2001.
[9] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. A scalable content-addressable
network. In Proc. of ACM SIGCOMM, 2001.
[10] Slashdot. http://www.slashdot.org/.
[11] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan. Chord: A scalable
peer-to-peer lookup service for Internet applications. In Proc. of ACM SIGCOMM, 2001.
[12] I. Stoica T.S. Eugene Ng and H.Zhang. A Waypoint Service Approach to Connect Hetero-
geneous Internet Address Spaces. In Proceedings of USENIX Annual Technical Conference,
June 2001.
[13] Z. Turanyi and A Valko. IPv4+4. In Proceedings of 10th International Conference on Network
Protocols, November 2002.
38