CPR and GAMMONDeployment Experiences.

Warren Matthews Georgia Institute of

Technology.

Office of Information Technologyhttp://www.oit.gatech.edu

2

Georgia TechGeorgia Tech

• Campus– Backbone group maintain 180 buildings,

1700 switches, 55000 ports.

• Southern Crossroads gigapop (SOX)– Provides connectivity for 20+ Universities

throughout the South East– 10Gbps link to Abilene backbone.

Office of Information Technologyhttp://www.oit.gatech.edu

3

MotivationMotivation

• Measurement Infrastructure typically means WAN monitoring– But problems are LAN and host based

• Network Operations– Single point of view– Catastrophic failure is easier to detect– Little quantitative data to troubleshoot

performance problems

Office of Information Technologyhttp://www.oit.gatech.edu

4

CPRCPR

• Campus-wide Network Performance Monitoring and Recovery

• 50+ hosts on Campus– Collocated with switches in data closets– Multiple views of the network– Especially the users view

• Regular tests across campus network

Office of Information Technologyhttp://www.oit.gatech.edu

5

Gateway Routers

NI SI

Rich

DMSmith

Lyman

Skiles

ET

Servernet

845

811

EDI GLC

SSC

Ajax

Mason

FrenchOHR

Core Routers

FAB OKeefe KingGCATT

SOX

LAWN

IBB

Howey

Admin

SavannahGTL

MRDC

Rich2

OHR

SEB

Neely

MiRC

Cherry-Emerson

Couch

ES&T

MARC

Habersham

Lib-class

French-class

Sc-class

Arch

Daniel

Savant44

Savant

Weber

Boggs

GTRI

YamacrawClassTechSq Classroom

Lyman

505

Rich133

DeploymentDeployment

Office of Information Technologyhttp://www.oit.gatech.edu

6

GAMMONGAMMON

• Georgia Measurement and Monitoring– State-wide initiative

• Distance Learning and Professional Education (DLPE)

• Valdosta State University, Armstrong Atlantic State University, Barrow County School System.

Office of Information Technologyhttp://www.oit.gatech.edu

7

Other DeploymentsOther Deployments

• Local ISPs– Major providers (Level3, Qwest, Charter)– Residential (SpeedFactory, BellSouth,

Charter, Cox)

• Global collaborations– International focus in strategic plan – Metz, Shanghai– Leverage Global PMP Infrastructure and

communicate using emerging standards

Office of Information Technologyhttp://www.oit.gatech.edu

8

Deployment Deployment ExperienceExperience

• On the whole, very smooth• Control of Network

– Firewall– Physical access– Control of satellite server, DNS

• Warmly welcomed– Help is on the way– Pass the buck

Office of Information Technologyhttp://www.oit.gatech.edu

9

Deployment Deployment ExperienceExperience

• Donated hardware– Cheap– Flakey– Scavenging of parts before donation

• GAMMON/International deployment faces more of the traditional challenges

Office of Information Technologyhttp://www.oit.gatech.edu

10

ToolsetToolset

• No in-house development of measurement tools.

• Original plan also didn’t include much visualization.– Inconvenient to click through

numerous graphs

Office of Information Technologyhttp://www.oit.gatech.edu

11

MeasurementsMeasurements• Currently

– Smokeping - roundtrip time and graphs.– Nagios - Services.– Security - nessus and nmap.

• Also available– Iperf (bwctl) - TCP throughput only.– Pathchar, traceroute

• Coming soon– NDT (central, distributed)– Test bed for tools under development

Office of Information Technologyhttp://www.oit.gatech.edu

12

Measurement Measurement ExperiencesExperiences

• Passive Monitoring of border traffic– Bwctl.sox is outside GT.

• OWAMP– NTP

• Alleged network problems are often issues with applications– Added application monitoring using Nagios

• Information Security– CPR as a platform for other groups, research

Office of Information Technologyhttp://www.oit.gatech.edu

13

AnalysisAnalysis

• Analysis– Create base-lines for historical comparison– Use multiple view to detect location– Middleware

• Alarm system– Plateau detector (AMP), RIPE-TT– How should we react to alarms?– Troubleshooting guide

Office of Information Technologyhttp://www.oit.gatech.edu

14

Analysis ExperienceAnalysis Experience

• Initial design was active monitoring– Integrate with passive tools– Wireless tools, NETI– Provide a platform for localized netflow

reports

• Initial design was to avoid looking at graphs– Auto detect and alarms– MyCPR

Office of Information Technologyhttp://www.oit.gatech.edu

15

MyCPRMyCPR

• Configurable alarms and graphs

Office of Information Technologyhttp://www.oit.gatech.edu

16

Case StudiesCase Studies

• CPR has helped solve numerous issues– Firewall– Network slowness for file sharing– Dropped sessions

• Not everything is a network issue

Office of Information Technologyhttp://www.oit.gatech.edu

17

ExperiencesExperiences

• Don’t assume anything– If you don’t measure, you don’t know.

• Don’t expect people to do what they should

• When one door closes, another one opens.

Office of Information Technologyhttp://www.oit.gatech.edu

18

This is the endThis is the end

• Contact– Warren.matthews@oit.gatech.edu

• Project WebSite– http://www.rnoc.gatech.edu/cpr