Internet2 QoS: Overview and Early Experiences Russ Hobby December 4, 2000Campus Focused Workshop on Advanced Networks

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

How We Got Here (short version)

Began chanting: enable advanced applications, Assessed requirementsRecommended DiffServSelected Premium service to meet demands of loss/jitter sensitive appsCharted QBone initiative Specified QBone architectureNow proceeding to implement it and tweak the architecturearchitecturedeployment

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Internet2 QoS Milestones1/2Bay Workshop QoS WG DiffServ RecommendationQBone initiative launchedEvanston QIGRTP QIGQBone architecture last callLas Cruces QIGQPS demonstrated (nM demo)QConBBopQUALITQIG opened Pittsburgh QBone BOFQBone architecture draft (v1.0)QBone CFPHouston I2/DOE QBone Workshop Early Internet2 QoS requirements gatheringAPS Test Program launched QBone Forum launched

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Internet2 QoS Milestones2/2You are here!SC2K QBone demoToronto QoS BOFWG re-chartering

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

IntServ/RSVP vs DiffServIntServ/RSVPBBBBPer-flow service state at every hopAbstract/manage each clouds resources (BBs) Packets colored to indicate forwarding behaviorFocus on aggregates not individual flowsPolicing at edge to get servicesScalability problemsDiffServFocus on multipoint multicast

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

DiffServ OverviewApplications contract for specific QoS profiles Policing at network peripheryColor packets with a few simple, differentiated per-hop forwarding behaviors (PHBs) Indicated in packet headerApplied to PHB traffic aggregatesPHBs + policing rules = range of servicesDS domains contract with each other for aggregate QoS traffic profilesPolicing at cloud-cloud boundarySupports simple, bilateral business agreementsExploits edge/core distinction for scalability

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Example Service #1: PremiumAssurance: like a leased line PHB: Expedited Forwarding (RFC 2598)EF in separate queue configured with minimum departure rateExample mechanisms: strict priority, MDRR, WFQPolicing: police to a specified peak rate and drop out-of-profile packets; effectively a leaky bucket with depth 1 MTU

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Example Service #2: Controlled LoadAssurance: network looks lightly-loaded for conforming traffic PHB: Assured Forwarding (RFC 2597)4 independent AF classes3 drop preference levels within each classExample mechanisms: WRED, WFQPolicing: police to specified rate and burst profile, remarking out-of-profile packets to have higher drop probability

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Example Service #3: CoSAssurance: better than JoePHB: drop the lower classes first (AF or class selector PHBs) Policing: could be based on anything (e.g. higher priority for the CEO)A.K.A.Olympic classes of BE service (e.g. Gold, Silver, Bronze)

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

QBone ArchitectureA Service: QBone Premium ServiceBuilt on Expedited Forwarding (EF) (RFC 2598)Assurance: near-zero loss & low, bounded jitter for marked traffic conforming to a specified peak rate a.k.a. virtual leased line, virtual wireReservation Setup ProtocolNow: long-lived, manual setupProposed: SIBBS protocol between QBone domains; RSVP end-to-end between hostsQBone Measurement ArchitectureUniform collection of QoS metricsUniform dissemination interface

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

QBone E2E PictureKey

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Abilene Premium Service (APS)Goal:Make APS a reference implementation of the QBone architectureWhy?Catalyst: Enable wide-area QoS experimentation and propagate Abilenes experiences to larger communitySafety Belt: Congestion is only one success catastrophe away!The Team:UCAIDIndiana UniversityI-TECs (NCNI, OARnet)CiscoQwestNortelNLANR/NCNE

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

APS ParticipationParticipation CriteriaAbilene connector status necessary but not sufficientQBone participationCapability of access router / Abilene edge cardCurrent ParticipantsMAGPI (U. Penn)iCAIRPSC (Penn State)OARNet (Ohio State)Others in the wingsNASA EOSVarious internationalANLUIUCDOE Science GRID (peering transit network)

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Initial Engineering Plan (obsolete)Automated Setup(GTS)(Surveyor + SNMP + HTTP)(Firehose CAR)(Whiteboard + CLI)(MDRR)

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

APS Phase 1: SweetwaterFirst, crawlIngress policing Measurement feedbackLimited technical supportNo priority queuing yet, but still get benefit of Abilenes over-provisioning!GoalsBegin to establish operational practices for responding to admissions requests Provide participants with the experience of sitting behind a policer

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

APS MeasurementsGoal: implement QBone measurement architecture (collection + dissemination) for a single QBone domainHoped-for Side Effects: Tools to ease implementation for othersAdjustments to QMA based on implementation experiences

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Committed Access Rate (CAR)Classifies traffic based on certain matching criteria and meters it to a leaky bucket traffic profileDepending on metering result, different actions applied (drop, transmit, set DSCP,) Syntax:rate-limit {input | output} [access-group [rate-limit] acl-index] bps burst-normal burst-max conform-action action exceed-action action

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

CAR Experience

For the most part, CAR is exactly what the DiffServ doctor orderedHowever, there are some limitationsPerformanceToken bucket depthClassification of DiffServ aggregates

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

CAR Limitation 1: Performance On E0 edge cards, no ASIC support for CAR there is a performance hit

pps @ 256 bytes* Preliminary E0 CAR performance figures under retest by ITEC-NC

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Addressing CAR PerformanceOut of our handsThe Plan: Load on access interfaces is still light, so performance not really an issue for nowWait for E3 edge cards, which will have CAR in hardware

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Virtual Trunk vs.Firehose ClassificationThe good news: CAR can also classify by:qos-group (Cisco proprietary concept)With packets assigned to QoS groups through QoS Policy Propagation via BGP (QPPB)

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Abilene Architecture Limitation: Porous Edge ProblemQoS theft problem with current architectureMDRR (EF forwarding) on interior interfaces easily subverted by unpoliced connectors

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Current Engineering Plan (GTS)(Surveyor + SNMP + HTTP + WHOIS + traffic matrices)(CAR + QPPB + PIRC filtering?)(Whiteboard + CLI)(MDRR)

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

SC2000 Interdomain QoS DemoNovember 6-9, 2000Premium service over two wide-area pathsLBNL-ESnet-Abilene-SCinet-Internet2 boothStanford-CalREN2-Abilene-SCinet-Internet2 boothCongestion induced at multiple pointsCD-quality interactive audio application shown with/without QoSESnet and Abilene QoS capabilities formed nucleus of QBoneSC2000 Network Challenge Winner: "Most Captivating and Best Tuned Award"

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

AbileneInternet2 SC2000 Boothcore-conf Cisco 12008ESNet/DOE Science Grid10 MbpsOC-3OC-48CalREN2OC-12OC-12OC-12LBNLDevice controlGEGETraffic GeneratorsStanford UniversityI2-GW Cisco 12008OC-3OC-3* All links 100Mbps fast Ethernet (FE) unless otherwise notedInterdomain Quality of Service DemoSCinetOC-12= congestion + QoS control

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

DeploymentGood News:DiffServ functionality in most modern routersMany hosts support QoS signaling Lots of isolated testbed trialsSome partial backbone implementationsBad News:Weak demand Turnkey solutions are a long way offVendor implementations dont always live up to hypeHR scarcityBottom line:Progress can be made, but it takes a lot of people, time, and effort

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Going ForwardNeed increased focus on low-hanging fruitLess than Best-Efforts (LBE)Ad-hoc CoS at congestion pointsBasic LBE ideaLow-priority traffic classAgree on an Internet2 LBE code point BenefitsTraffic managementLicense to hog (some users are self-policing)New advanced applications (e.g. large scale distributed computation)Its easy!

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Starvation: Making LBE StrongerDefine starvation class selector (SCS)An SCS packet MUST never consume a network resource desired by a non-SCS packetCorollary 1: there must not be a configured minimum departure rate for SCS trafficCorollary 2: all queued SCS packets must be evicted before a forwarding element can drop a non-SCS packet (probably difficult to implement)Result: its the dual of EF!

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Any Questions?

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

For more information...Internet2 Home:http://www.internet2.edu/Internet2 QoS Working Group Home:http://www.internet2.edu/wg/qos/QBone Home:http://qbone.internet2.edu/Abilene Premium Service Home:http://www.internet2.edu/abilene/qos/

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Internet2 QoS: Overview and ExperiencesCampus Focused Workshop on Advanced Networks (December 4, 2000)

Architecture (SIBBS)Communication (Workshop, QBone forum)Implementation (APS)OWDP coloring DSCPATM cards in NYCM and IPLS (one port on qOC3 ATM) - results will be more realistic - addresses advertized outside of Abilene - throughput source/sink; bursts up "OC3 line rate"

nycm-pc.abilene.ucaid.edu

Plan:

Washington DC (within 1 week); Deploy at all nodes with next two; could repond to priority needs if they arise

QPS quite similar to IntServ GSOC-12 POSOC-3 POSOC-12 ATMqOC-3 ATM23500072000235500588251787007200015340044675