the abilene observatory
TRANSCRIPT
Internet2 Presentation TemplateUniversity of Minnesota 23 October
2003
11/10/2003 2
Internet2 Infrastructure • Abilene, HOPI, RONs • Observatory will evolve to new infrastructures
Measurements • History • Operational / Research aspects • Abilene Upgrade
The Observatory • Data Collections • Data Views • Collocation • Input from research community
11/10/2003 3
Abilene – Design
11/10/2003 4
Abilene – Current
11/10/2003 5
Abilene Scale IP-over-DWDM (OC-192c) and IP-over-SONET
OC-48c Backbone, Juniper T-640 Routers 48 Connectors (OC-3c → 10 GigE) 222 participants – research universities & labs
• All 50 states, District of Columbia, & Puerto Rico • Aggregation on the rise!
Expanded access • 92 sponsored participants, expected to increase • 30 State networks, expected to increase
Performance • Early IPv4/IPv6 tests – 8 gbps cross country (packet blasts) • Recent tests – 5.6 gbps TCP flows from CalTech to CERN!
11/10/2003 6
Abilene International Peering
Abilene Federal/Research Peering
Internet2/NYSERnet partnership Ethernet Switch
• Cisco 6513 (with new fabric/backplane soon) • Located in same building as Abilene New York City Router Node • CA*net bringing in OC-192 to same building
Abilene Tyco/SurfNet connection • OC-192 from Amsterdam
GLIF (Global Lambda Integration Facility) Installing Cisco 15454
• Located in MANLAN rack • Will bring CA*net and SurfNet OC-192s to that switch • Potential for experimental TDM paths • Ethernets to MANLAN • SONET to Abilene New York City Router Node
11/10/2003 9
• Abilene Packet Based Infrastructure • Regional Optical Networks – RONs • National LambaRail Wave
Major Questions • Vision for the Network • Packet switching / circuit switching futures • How do we go forward in the near future
– Design team – Initial design February 2004
• How do we measure!! – Research community involvement
11/10/2003 10
California (CALREN) Colorado (FRGP/BRAN) Connecticut (Connecticut Education Network) Florida (Florida LambdaRail) Indiana (I-LIGHT) Illinois (I-WIRE) Maryland, D.C. & northern Virginia (MAX) Michigan New York + New England region (NEREN) North Carolina (NC LambdaRail) Ohio (Third Frontier Network) Oregon Rhode Island (OSHEAN) SURA Crossroads (southeastern U.S.) Texas Utah
11/10/2003 11
11/10/2003 12
Measurement Infrastructure
Abilene Measurement Infrastructure History
• Original Abilene racks included measurement devices – Included a single PC – Early OWAMP, surveyor measurements – Optical splitters at some locations
Data collection motivation • Operational data – collected by the NOC
– How is the network performing? – All users, other network operators to understand network
• Some research data collected from the beginning
11/10/2003 13
Measurement Infrastructure
Upgrade to Juniper T-640 routers and OC-192c
• Important decision about rack space • Two racks with one dedicated to measurement platform
• Potential for research community to collocate equipment
11/10/2003 14
Houston Router Node
nms1: throughput tester (iperf) • gigE direct to T-640, MTU 9000
nms2: ad-hoc on-demand (+ routing?) • gigE to switch to T-640, MTU 1500
nms3: statistics collection (flow, snmp) • 100bT to switch to T-640 • Local data collection to capture during network instability
nms4: latency tester (owamp, troute) • 100bT to switch to T-640 • CDMA “GPS” timing source from endruntechnologies.com
11/10/2003 17
Hardware
Intel SR2200 from ioncomputer.com Intel SCB2 motherboard Dual 1.26 GHz Pentium III, 512K L2 1GB PC-133 DRAM in two banks ServerWorks ServerSet HE-SL chipset 64 bit 64 MHz PCI Syskonnect GigE SK-9843 SX Redundant 48VDC
11/10/2003 18
OS
nms1’s ( in transition to: ) • Linux 2.4.20, SMP, no changes • All traffic default routed through gigE • Buffers tuned a-la LBL document • 990 Mb/s TCP between any two • (better TCP platform)
nms2-4: • FreeBSD 4.6-STABLE • Buffers tuned • (better UDP platform)
11/10/2003 19
Measurement Capabilities
One way latency, jitter, loss • IPv4 and IPv6
Regular TCP/UDP throughput tests – ~1 Gbps • IPv4 and IPv6; On-demand available (see “pipes”)
SNMP (NOC) [octets, packets, errors; collected frequently]
• NOC working on SNMP proxy
“Netflow” (ITEC Ohio) [anon. by 0-ing last 11 bits] Multicast beacon with historical data Routing data (BGP & IGP) [IGP under development]
• Looking at Zebra + mods, Japanese routing research is driver
(Optical splitter taps on backbone links at select location(s))
11/10/2003 20
Throughput Tester Evolution
Original design had all four servers with the same operating system
• Ease of administration • Lots of experience using FreeBSD for TCP testing • FreeBSD features for capturing UDP packets (router flow capture)
• FreeBSD had more respected NTP implementation
• Linux has Web100 development and large high- energy physics installed base
FreeBSD 4.6-Release deployed
Throughput Tester Evolution
Over the last year, more TCP experience in wide area (9000 byte MTU, gigE direct to T-640)
• DC to LA: 980 Mb/s UDP, but ~380 Mb/s TCP • NC-ITEC to LA: 990 Mb/s TCP, if sender has tuned, but otherwise stock 2.4.20 Linux kernel
• Replicated in testing from Indianapolis (same location, same connectivity, same hardware)
• The FreeBSD sender’s CPU is pegged, TCP stack apparently starved (PUSH set on all frames)
11/10/2003 22
OWAMP
One Way Latency •Requires NTP on endpoints •Control connection used to broker test request based upon policy restrictions and available resources. (Bandwidth/disk limits)
•Enables the combination of regularly scheduled tests with on-demand tests.
•http://owamp.internet2.edu/ •Reference implementation of Draft:
http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-06.txt
11/10/2003 23
Abilene Observatory
A program to provide enhanced support of computer science research over Abilene
• Create network data archive – Consists of many separate databases on a variety of servers – Forms a large correlated database – Create tools to access the database – Support from and for graduate programs
• Create Data views – A snapshot of Abilene • Collocation Component - provision for direct network
measurement and experimentation –Resources reserved for additional servers
• Power (DC), rack space (2RU), router uplink ports (GigE) –Initial deployment is PlanetLab –Additional requests from AMP project and research
team from Japan
• Usage Data • Netflow Data • Routing Data
–BGP now, ISIS in the future
• Latency Data • Throughput Data • Router Data • Syslog Data
11/10/2003 25
Databases
Variety of Interfaces to data • Simple web based for usages data • Rsync for netflow • Simple web based for routing • Soap interface for latency data • Throughput data undecided as yet • Router data – Soap interface • Syslog data undecided as yet
Should this be standardized?
Large, correlated, distributed databases? Other types of data? More importantly, other types of projects? Including networks closer to the edge?
Gigapops and universities? Involving the international community? Future infrastructures? Involving graduate programs – graduate
students?
http://abilene.internet2.edu/observatory/data-views.html
Single page to see ‘10 worst’ measurements • Throughput data • Losses • Variation in latency (95th – min) for links • Maybe top utilization measurements too for comparison
Expect to iterate presentation to find “best” presentation for us
11/10/2003 28
http://netflow.internet2.edu/weekly/ • Summarized flow data
11/10/2003 29
Outline
Measurement Infrastructure
Measurement Infrastructure
Hardware
OS
11/10/2003 2
Internet2 Infrastructure • Abilene, HOPI, RONs • Observatory will evolve to new infrastructures
Measurements • History • Operational / Research aspects • Abilene Upgrade
The Observatory • Data Collections • Data Views • Collocation • Input from research community
11/10/2003 3
Abilene – Design
11/10/2003 4
Abilene – Current
11/10/2003 5
Abilene Scale IP-over-DWDM (OC-192c) and IP-over-SONET
OC-48c Backbone, Juniper T-640 Routers 48 Connectors (OC-3c → 10 GigE) 222 participants – research universities & labs
• All 50 states, District of Columbia, & Puerto Rico • Aggregation on the rise!
Expanded access • 92 sponsored participants, expected to increase • 30 State networks, expected to increase
Performance • Early IPv4/IPv6 tests – 8 gbps cross country (packet blasts) • Recent tests – 5.6 gbps TCP flows from CalTech to CERN!
11/10/2003 6
Abilene International Peering
Abilene Federal/Research Peering
Internet2/NYSERnet partnership Ethernet Switch
• Cisco 6513 (with new fabric/backplane soon) • Located in same building as Abilene New York City Router Node • CA*net bringing in OC-192 to same building
Abilene Tyco/SurfNet connection • OC-192 from Amsterdam
GLIF (Global Lambda Integration Facility) Installing Cisco 15454
• Located in MANLAN rack • Will bring CA*net and SurfNet OC-192s to that switch • Potential for experimental TDM paths • Ethernets to MANLAN • SONET to Abilene New York City Router Node
11/10/2003 9
• Abilene Packet Based Infrastructure • Regional Optical Networks – RONs • National LambaRail Wave
Major Questions • Vision for the Network • Packet switching / circuit switching futures • How do we go forward in the near future
– Design team – Initial design February 2004
• How do we measure!! – Research community involvement
11/10/2003 10
California (CALREN) Colorado (FRGP/BRAN) Connecticut (Connecticut Education Network) Florida (Florida LambdaRail) Indiana (I-LIGHT) Illinois (I-WIRE) Maryland, D.C. & northern Virginia (MAX) Michigan New York + New England region (NEREN) North Carolina (NC LambdaRail) Ohio (Third Frontier Network) Oregon Rhode Island (OSHEAN) SURA Crossroads (southeastern U.S.) Texas Utah
11/10/2003 11
11/10/2003 12
Measurement Infrastructure
Abilene Measurement Infrastructure History
• Original Abilene racks included measurement devices – Included a single PC – Early OWAMP, surveyor measurements – Optical splitters at some locations
Data collection motivation • Operational data – collected by the NOC
– How is the network performing? – All users, other network operators to understand network
• Some research data collected from the beginning
11/10/2003 13
Measurement Infrastructure
Upgrade to Juniper T-640 routers and OC-192c
• Important decision about rack space • Two racks with one dedicated to measurement platform
• Potential for research community to collocate equipment
11/10/2003 14
Houston Router Node
nms1: throughput tester (iperf) • gigE direct to T-640, MTU 9000
nms2: ad-hoc on-demand (+ routing?) • gigE to switch to T-640, MTU 1500
nms3: statistics collection (flow, snmp) • 100bT to switch to T-640 • Local data collection to capture during network instability
nms4: latency tester (owamp, troute) • 100bT to switch to T-640 • CDMA “GPS” timing source from endruntechnologies.com
11/10/2003 17
Hardware
Intel SR2200 from ioncomputer.com Intel SCB2 motherboard Dual 1.26 GHz Pentium III, 512K L2 1GB PC-133 DRAM in two banks ServerWorks ServerSet HE-SL chipset 64 bit 64 MHz PCI Syskonnect GigE SK-9843 SX Redundant 48VDC
11/10/2003 18
OS
nms1’s ( in transition to: ) • Linux 2.4.20, SMP, no changes • All traffic default routed through gigE • Buffers tuned a-la LBL document • 990 Mb/s TCP between any two • (better TCP platform)
nms2-4: • FreeBSD 4.6-STABLE • Buffers tuned • (better UDP platform)
11/10/2003 19
Measurement Capabilities
One way latency, jitter, loss • IPv4 and IPv6
Regular TCP/UDP throughput tests – ~1 Gbps • IPv4 and IPv6; On-demand available (see “pipes”)
SNMP (NOC) [octets, packets, errors; collected frequently]
• NOC working on SNMP proxy
“Netflow” (ITEC Ohio) [anon. by 0-ing last 11 bits] Multicast beacon with historical data Routing data (BGP & IGP) [IGP under development]
• Looking at Zebra + mods, Japanese routing research is driver
(Optical splitter taps on backbone links at select location(s))
11/10/2003 20
Throughput Tester Evolution
Original design had all four servers with the same operating system
• Ease of administration • Lots of experience using FreeBSD for TCP testing • FreeBSD features for capturing UDP packets (router flow capture)
• FreeBSD had more respected NTP implementation
• Linux has Web100 development and large high- energy physics installed base
FreeBSD 4.6-Release deployed
Throughput Tester Evolution
Over the last year, more TCP experience in wide area (9000 byte MTU, gigE direct to T-640)
• DC to LA: 980 Mb/s UDP, but ~380 Mb/s TCP • NC-ITEC to LA: 990 Mb/s TCP, if sender has tuned, but otherwise stock 2.4.20 Linux kernel
• Replicated in testing from Indianapolis (same location, same connectivity, same hardware)
• The FreeBSD sender’s CPU is pegged, TCP stack apparently starved (PUSH set on all frames)
11/10/2003 22
OWAMP
One Way Latency •Requires NTP on endpoints •Control connection used to broker test request based upon policy restrictions and available resources. (Bandwidth/disk limits)
•Enables the combination of regularly scheduled tests with on-demand tests.
•http://owamp.internet2.edu/ •Reference implementation of Draft:
http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-06.txt
11/10/2003 23
Abilene Observatory
A program to provide enhanced support of computer science research over Abilene
• Create network data archive – Consists of many separate databases on a variety of servers – Forms a large correlated database – Create tools to access the database – Support from and for graduate programs
• Create Data views – A snapshot of Abilene • Collocation Component - provision for direct network
measurement and experimentation –Resources reserved for additional servers
• Power (DC), rack space (2RU), router uplink ports (GigE) –Initial deployment is PlanetLab –Additional requests from AMP project and research
team from Japan
• Usage Data • Netflow Data • Routing Data
–BGP now, ISIS in the future
• Latency Data • Throughput Data • Router Data • Syslog Data
11/10/2003 25
Databases
Variety of Interfaces to data • Simple web based for usages data • Rsync for netflow • Simple web based for routing • Soap interface for latency data • Throughput data undecided as yet • Router data – Soap interface • Syslog data undecided as yet
Should this be standardized?
Large, correlated, distributed databases? Other types of data? More importantly, other types of projects? Including networks closer to the edge?
Gigapops and universities? Involving the international community? Future infrastructures? Involving graduate programs – graduate
students?
http://abilene.internet2.edu/observatory/data-views.html
Single page to see ‘10 worst’ measurements • Throughput data • Losses • Variation in latency (95th – min) for links • Maybe top utilization measurements too for comparison
Expect to iterate presentation to find “best” presentation for us
11/10/2003 28
http://netflow.internet2.edu/weekly/ • Summarized flow data
11/10/2003 29
Outline
Measurement Infrastructure
Measurement Infrastructure
Hardware
OS