agnès pouélé ([email protected]), jan novak ([email protected]) inter-domain multicast in...
TRANSCRIPT
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Inter-domain Multicast in European Research Networking
TEN-155 Operational Experience and Deployment on GÉANT
NANOG24
Miami 12th February 2002
Agnes Pouélé, DANTE Network Engineer
Jan Novak, Cisco Systems Inc. Network Consulting Engineer
1
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Agenda• DANTE
• TEN-155 Operational Experience– Evolution of the TEN-155 multicast topology from
1998 to 2000– Operational Experience
• GÉANT deployment– Overview of GÉANT Network– GÉANT’s Multicast design– GÉANT Multicast and Unicast Coverage– GÉANT Multicast Service and Monitoring
• Conclusion2
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
DANTE, TEN-155, GÉANT• DANTE
• DANTE is a not-for-profit company set up in 1993 by European National Research Network organizations.
• TEN-155 • was an ATM based network built initially on OC3
links and then upgraded in 2000 • GÉANT successors of TEN-155
• launched in December 2001• 10 Gbps Pan-European Network• Is partially funded by the EC
3
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Part I TEN-155 Operational Experience
4
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Typical mess of DVMRP tunnels on Sun WSs with usual tunnel routing problems.
5
Starting pointMBONE TUNNELED TOPOLOGY
Agnès Pouélé ([email protected]), Jan Novak ([email protected]) 6
OSLO IETF transmitted over both native STM-1 ATM based network and DVMRP tunnelled infrastructure
1999: PIM-SM and DVMRP set-up
PIM-SM domain
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Multicast deployment in TEN-155
• Based on this first set-up and successful operation, we concluded to enable multicast on all production routers in TEN-155.
• Multicast code stable, CPU usage ok, M-BGP ok• PIM-SM “only” for production service
• PIM-SM to DVMRP border works, but not possible to operate routinely (NOC)
• From 1999 to 2000 each country was migrated from the DVMRP cloud to a interconnection with TEN-155 using PIM-SM/MBGP/MSDP
7
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
2000: TEN-155 final topology
NLUnited Kingdom
France
Italy
Greece
Belgium
Spain
CHFR
UK
SE
IT
DE
Slovenia
Czech Rep
Germany
AT
Switzerland
Portugal
AS8933OSPF + internal MBGP
Poland
Israel
NetherlandsNordics
e x t e r n a l
e x
t e
r n
a l e x t e r n
a l
B G P
AT & T
UUNET
Luxembourg
Hungary
US
ABILENE
8
tunnel
unicast/mcast linededicated mcast
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
TEN-155 Operational ExperiencePerformance Impact
• Parameters to be considered for the exploitation of Multicast– CPU - parameters
• about 1600 forwarding (mroute) entries• max 8 outgoing interfaces (average 2-3)• max 20 Mbit/s of data forwarded by one router resulting in 5-10% of CPU increase (mainly PIM)
9
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
TEN-155 Operational ExperiencePerformance Impact
• Parameters to be considered for the exploitation of Multicast– Memory
• about 3000 SA messages in the cache• about 1600 forwarding entries (mroute)• about 10 000 routes in the MBGP table
• These parameters didn’t represent a significant memory usage
10
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Usage of MSDP – new component – new storms
• Default peer in redundant topology
• “Redundant” mesh-groups
MSDP peers default peer
Group A
Group C Group B
Impact of a MSDP storm on the CPU load
11
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Usage of MSDP – early implementation problems
• Cisco IOS 12.0.6S and lower– ghost SA entries in the MSDP cache
• SA messages recreated by the incoming (S,G) joins
• FIXED– Origination of SA messages only when source registered
to the originating RP
– “ip msdp redistribute” command – without arguments• re-originated all known SAs
• caused huge increase of the SA counts worldwide• FIXED
12
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
TEN-155 MSDP monitoring
•MSDP usage monitoring •number of RPs, average 90 (40 EU)•number of groups•number of pairs source, group (S,G)•number of SA messages per minute
•Monitoring set-up
WS - DANTE DE.TEN-155router
•Software – adapted C++ MSDP •implementation of Steve Rubin
13
MSDP peering
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
TEN-155 MSDP monitoring
Number of RPs originators announced to TEN-155 with and without multicast US connectivity
number of RPs
14
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
TEN-155 MBGP monitoring
•MBGP monitoring •stability/updates•number routes, max about 9000 (760 EU)•number of ASNs, average 240 (80 EU)
•Monitoring set-up
WS - DE PoP DE.TEN-155router
•Software - Merit’s MRTD - •modified SAFI definition for MBGP
15
MBGP peering
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
TEN-155 MBGP monitoring
•In blue, •the number of routes originated from one AS
•In green, •the number of updates originated by the same AS
stability/updates
16
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
TEN-155 Multicast Monitoring tools
• http://www.dante.net/pubs/dip/40/40.html
• http://www.dante.net/pubs/dip/41/41.html
• http://www.dante.net/mbone/
• http://sigma.dante.org.uk/stats/mrtg/msdp/data/
• http://www.dante.net/mbone/mbgp
The graph values from mbgp and msdp monitoring are historical values.
17
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
TEN-155 Operational ExperienceConclusion
• The adoption of PIM-SM domain (without DVMRP) encouraged the NRENs to do the same.
• Unicast and multicast non congruent• MSDP peer doesn’t need to be RP.• Concept of two BGP tables and “multi-protocol” RPF
check often still misunderstood.• Inter-domain Multicast debugging
– Almost impossible to fix problems in just one week
• TEN-155 work provided valuable inputs for GEANT multicast services and design
18
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
PART IINEXT GENERATION
GÉANT DEPLOYMENT
19
Agnès Pouélé ([email protected]), Jan Novak ([email protected]) 20
www.dante.net/nep/GEANT-MULTICAST/map.html
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANT Services • GÉANT Standard IP Service
– IP traffic from NREN to NREN and Research peerings.
• Multicast Service (rolling out now)
• Replacement of TEN-155 Managed Bandwidth Service– GÉANT Premium IP Service – Layer-2 VPNs (forthcoming)
• Upcoming
• Security and Dos attack detection, IPV621
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANT’s Customers and Other Peerings
• GÉANT Unicast Customers– 27 countries in Europe
• GÉANT Multicast Customers – 24 countries in Europe
• GÉANT Unicast and Multicast research and commercial peerings– Abilene, Canarie and ESnet via GTREN– Infonet, UUNET– KPNQwest, Global Crossing
22
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANT Multicast Customers
NREN ACCESS MCAST Type
Austria POS STM-4 NO Mcast
Slovenia POS STM-4* NATIVE
Belgium POS STM-16 NATIVE
Croatia ATM E3 NATIVE
Czech Rep. POS STM-16 NATIVE
Cyprus E3 TBA
Germany POS STM-16 NATIVE
ESTONIA POS STM-1 NATIVE
Portugal POS STM-4 NATIVE
Greece POS STM-16* NATIVE
Ireland POS STM-1 NATIVE
Hungary POS STM-16 NATIVE
Italy POS STM-16 NATIVE
Israel T3 NATIVE
United Kingdom POS STM-16 NATIVE
Latvia E3 NATIVE
Lithuania T3NATIVE
Norway, Sweden, Finland,Island
POS STM-16 NATIVE
Poland POS STM-16 NATIVE
Spain POS STM-16 NATIVE
France POS STM-16 NATIVE
Luxembourg POS STM-1 NATIVE
Roumania E3 NATIVE
Slovak Rep. POS STM-1 NATIVE
Netherlands POS STM-16 NATIVE
Switzerland POS STM-16 NATIVE/TBA
Bulgaria E3 TBA
23
www.dante.net/nep/GEANT-MULTICAST/map.html
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANT Multicast Implementation
• Current design built on– The experience gained from TEN-155– Guidelines from multicast experts– Juniper Laboratory tests – GÉANT Multicast services
–Multicast transit domain NRENs to NRENs–Multicast transit domain NRENs to other
PEERS–Beacon monitored backbone
• GÉANT Multicast routing policy at:–
http://www.dante.net/nep/GEANT-MULTICAST/routing-policy.html
24
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
HostX
Manganese
Iridium
HeliumGallium
Sulfur
HostYXenon
Network 10.2/16 –AS 102
Network 10.0/16-AS 100
Network 10.1/16-AS 101
HostZ
Tin
Network 10.3/16 –AS 103
Group 233.1.10.1Source 10.1.10.2
Source 10.2.20.2Group 233.2.20.1 Group 233.3.30.1
Source 10.3.30.2
Group 233.0.1.1Source 10.0.1.2
Group 233.0.3.1Source 10.3.30.2
TEST-BED LAB 26/27th Nov 2001
M-BGP peering
MSDP peering
I-MBGP full mesh
•Junos 5.0R3.3
• mesh group
25
I-MSDP full mesh
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
PIM SMv2 GÉANT domain
• GÉANT single PIM-SMv2– Version 2 , enabled on all interfaces.
• Three Rendez-vous Points with private anycast address. – backup for internal sources and receivers.– Private Anycast address (filtered out)– closest RP based on the OSPF cost
• All other interconnected administrative domain have to be PIM-SM v2 enabled with their dedicated RP.
26
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
PHYSICAL TOPOLOGY: RP
uk
fr
at
ch
cz
pl
hu
sk
gr
ie
10
10
9
7 640
20
40
20
40
10
35
40
160
159
77
7
40
35
630
10si
160
170
be
nl
40
40
40
lu
640
640
5
it
es
de
se
IUCCJanet NY4-1 Nordunet
Eenet
Latnet
Litnet
Posnan
DFNNY4-2
Infonet
Multicast access
Rendez-vous Point
Cesnet
Sanet
HungarnetRoEduNet
Arnes
Grnet
Switch&Cern
Renater
Rediris
FCCN
Surfnet
Belnet
Restena
Heanet
Carnet
INFN
Aconet
Unicom-bCynet
STM64/OC192STM16/OC48STM4/OC12STM1/OC3
27
PoP
www.dante.net/nep/GEANT-MULTICAST/map.html
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANT Design: MBGP and MSDP• MBGP
• Separate multicast routing table (inet.2) • Currently congruent BGP and MBGP topology in
Europe
• iMSDP • MSDP is fully meshed between 19 PoPs• Use of mesh group• i-MSDP Peering with loopback addresses (Not
the anycast address !!)
• eMSDP• NREN <---> GÉANT Access Router
28
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
MSDP peering Logical view
iMSDP Mesh Group
iMBGP Full Mesh
ukse
de1
cz
huat
es
fr
Iucc Janet Ny4-1
Renater
Switch&Cern
Rediris
FCCN
INFN
Arnes
RoEdunet
Sanet Posnan
Cesnet
DFN
NY4-2
Infonet
GRnet
Nordunet
Latnet
Litnet
Eenet
Carnet
gr
CynetUnicom-b
ch
lu
Restena
beBelnet
it
si
Hungarnet
sk
de2
plie
Heanet
pt
Surfnetnl
Aconet
External MSDP peering NRN ----- GÉANT router
29
RP
GTREN
Abilene
www.dante.net/nep/GEANT-MULTICAST/map.html
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
MSDP SA Filtering
• A list of filtered SA is defined at:– http://www.dante.net/nep/GEANT-MULTICAST/
deployment-msdp.html
• We filter the recommended list.
• We authorise 239.194.0.0/16 from the IPV4 Organisation Local Scope through GÉANT.
30
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANTAS20965
CAnet
Third party providerEuropean Distributed Access
Commodity Internet Access
DFN
NREN2
JANET
NREN4
GTRENRESEARCH Peerings
STM-16
UKDE
Esnet
... ...
Abilene
31
STM-4
www.dante.net/nep/gtren.html
GÉANT Multicast and Unicast Coverage
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANT Multicast Service
• Access to the service– Via the primary access to GÉANT– Via a GRE tunnel (currently nobody)– Support of PIM-SM v2 only
• Operational procedures (rolling out now)– Goal: fully supported service as Unicast.
• Troubleshooting – Extension of the trouble ticket systems to multicast
incidents
32
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANT multicast monitoring
• Beacon– Tool initially developed by Kai Chen from
NLANR• dast.nlanr.net/projects/Beacon/
• Relies on a number of Agents spread over the network which simultaneously send and receive multicast packets carrying a packet sequence number and a timestamp.
• Communicates with a central server which displays matrices of Agents via web pages.
33
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANT multicast monitoring• Recommended by TF-NGN group
– (www.dante.net/tf-ngn)
• To use from day 1– monitoring of multicast inside and outside of GÉANT.
• Server code enhanced with historical functionality http://noc.man.poznan.pl/noc/index/strony (Menu item
“Applications”)
Multicast Beacon Agent written in C http://www.cesnet.cz/tf-ngn/multicast/
34
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
GÉANT multicast monitoring• Beacon’s matrices
– One for the internal sources of GÉANT• Each GÉANT POP has a beacon agent installed
– One for the external sources of GÉANT• http://beaconserver.geant.net:19999/
– We have assigned two multicast groups from GLOP range [RFC2770] for each matrix
• Parameters monitored• Loss• Delay• Jitter
35
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Beacon internal/external matrix
36
www.dante.net/nep/GÉANT-MULTICAST/deployment-beacon.html
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Other Monitoring Tools
• Per group monitoring from TEN-155– Based on the IETF IP-MROUTE MIB – Shows traffic per multicast group per interface– Under installation on ws1.se.geant.net
• MSDP and MBGP monitoring tools– Not yet available
37
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
CONCLUSION
• Deployment status– beaconserver.geant.net:19999/
• Conclusion– From TEN-155 to GÉANT Unicast and multicast
moves to a congruent topologyacross Europe and towards research peerings.
• Links– www.dante.net/nep/GEANT-MULTICAST/– www.dante.net/mbone/– www.dante.net/tf-ngn/
38
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
We would like to thank and acknowledge the help of the people who worked and are working with
us on these projects, mainly from all EU and US research networks
39
THANKS
Agnès Pouélé ([email protected]), Jan Novak ([email protected])
Peering relationship Address DescriptionAll 224.0.1.2/32 SGI "Dogfight" gameAll 224.0.1.3/32 RWHODAll 224.0.1.8/32 Sun's NIS+All 224.0.1.22/32 SVRLOCAll 224.0.1.24/32 MICROSOFT-DSAll 224.0.1.25/32 NBC-proAll 224.0.1.35/32 SVRLOC-DAAll 224.0.1.39/32 Cisco's Rendezvous Point Announcement ProtocolAll 224.0.1.40/32 Cisco's Rendezvous Point Discovery ProtocolAll 224.0.1.60/32 HP's Device Discovery ProtocolAll 224.0.2.1/32 rwho group (BSD)All 224.0.2.2/32 Sun's Remote Procedure Call ProtocolAll 229.55.150.208/32 Norton "Ghost" disk duplication softwareAll 234.42.42.42/32 ImageCast disk duplication softwareAll 234.142.142.142/32 ImageCast disk duplication softwareAll 232.0.0.0/8 Default SSM-range. Do not do MSDP in this range.Out of GÉANT 239.0.0.0/8 Administratively Scoped IPv4 Group Addresses
In GÉANT
239.0.0.0/8 with exception of 239.194.0.0/16
Administratively Scoped IPv4 Group Addresses. Th 239.194.0.0/16 address address range is reserved for GÉANT only multicast traffic.
All 10.0.0.0/8 Private addresses (RFC 1918)All 172.16.0.0/12 Private addresses (RFC 1918)All 192.168.0.0/16 Private addresses (RFC 1918)All 127.0.0.1/8 Loopback address
SA Filter list
41