north america wan rearchitecture for peer- to-peer ... · nyc kanata atlanta. ... ip addressing sj...

33
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Public CoC_IT_Case_Study 1 Cisco on Cisco Technology Seminar North America WAN Rearchitecture for Peer- to-Peer Applications Cisco Information Technology Don Layton & Rich Gore April 2007

Upload: others

Post on 15-Jul-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 1

Cisco on CiscoTechnology Seminar

North America WAN Rearchitecture for Peer-to-Peer Applications

Cisco Information TechnologyDon Layton & Rich GoreApril 2007

Page 2: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 2

Agenda

WAN Rearchitecture Drivers

WAN Design

Hub Site Design

Migration Steps

Results

Page 3: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 3

Americas WAN Rearchitecture – Drivers

• Previous Americas spoke & hub WAN was optimized for client / server apps in SJ; new WAN is optimized for both peer-to-peer and client-server (minimal latency, QoS enabled)• Many newer apps are peer-to-peer

• IP Telephony & voicemail & contact center applications, IP Video, and MeetingPlace application sharing

• Many newer apps should have regional presence (or presence at EVERY site) for acceptable performance:

• e.g. Content Distribution, Unity Voicemail, E-mail and Calendaring

• Rapid growth in bandwidth demand requires a more cost-effective solution

Page 4: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 4

Cisco North America WAN: Client / Server (2001)

Average site-to-hub circuit length = 1800 miles

US WAN Bandwidths

384 K19%

768 K22% T-1

6%

IMA ATM (6-12M)

10%

ECS (2-5 T1)12%

256 K29%

512 K 2%

San Jose

Raleigh

Kanata

`

256 Kbps384 Kbps512 Kpbs768 KpbsT-1 FrameT-1 ECSATM IMADS-3OC-3

Page 5: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 5

Cisco North America WAN: Peer-to-Peer (2003)Sum of US WAN Bandwidth: X 3.5 ( 264 Mbps 972 Mbps )

Cost: up 4% (average circuit length = 271 miles)

79%

13%

2%

3% 3%

2 - T1s4 - T1s6 - T1sDS3 + 4T1s2 DS3s

Frame / VPN2 T1sMulti T1sDS-3sOC-3OC-12

RedwoodCity / SanJose

Denver

Dallas

Chicago

Raleigh_

NYC

Kanata

Atlanta

Page 6: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 6

WAN Rearchitecture Design Summary

• Optimize US WAN by building regional hub architecture following existing fiber paths

•San Jose, RTP, Chelmsford, Kanata, Chicago, Dallas, Atlanta, Denver•Possibly Irvine, Miami, Seattle in the future•Telco hotelling at non-Cisco-owned sites

• WAN Backbone becomes 12 inter-hub OC-3s / OC-12s• WAN becomes 120+ T-1s / multiple-T-1s / DS-3s to nearest

CAPNet hub• Provide WAN QoS for multiple classes of service (per current

WAN QoS project)• Add bandwidth to support new applications• Support regional hub IT locations for eventual application server

sites

Page 7: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 7

Comparing WAN Architecture Advantages2001 (Hub in SJ) 2003 (Regional Hubs)

Optimized for both Peer to Peer and Client / Server (anywhere)Supports regional distribution of data, processingCost effective for larger networksGood performance for real time apps

Optimized for Client / Server (in SJ)

Easy to manage

Easy capacity planning

Cost effective for smaller networks

Good performance for near-real-time apps

Flexible, scalable

Page 8: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 8

WAN Design

Page 9: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 9

WAN Route: Hub Locations on Underlying Fiber Path

OC-3

OC-12

DS-3

Page 10: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 10

WAN Route: Hub Site Populations / Sites

San Jose

20K / 26

Denver

850 / 10

Dallas

2100 / 15

Atlanta

650 / 16

RTP

3500 / 15

New York

2400 / 20

Kanata

900 / 14Chicago

1300 / 26

Total hubbed Population / # Sites as of: Jan 2002

Page 11: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 11

Branch Office Site Connections

2001:– 27% links @ 256K; – 24% links > T-1– Avg link length 1800 miles

Re-architecture (2002-2003):– All Sub-T-1 frame replaced with dual T-1s (94 / 120)– All above T-1 replaced with 4 or 6 T-1s (load balanced); above 6-9 Mbps, replace with DS-3 + n x T-1 or dual DS-3s– Average link length 260 miles

256 K

84 K

512 K

768 K T-1

Multiple T-1

IMA

ECS

Other

256 K384 K512 K768 KT-1Multiple T-1IMAECS

Page 12: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 12

Branch Office Backup Connections2001:

– BRI backup for all FSO links– T-1 backup for some ECS links– Average link length 1800 miles

Re-architecture (2002):– Dual T-1s (primary plus backup) at all sub-T-1 to T-1 sites.– Multiple (load balanced T-1s) will not need backup.– N x T-1 or DS-3 backup for DS-3 links (depends on load, distance, cost)– Average link length 260 miles

Page 13: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 13

QoS on the WANQoS Class

Description IP Precedence

DSCP

6 Network control / routing traffic 6 and 7 48–63

5 Voice bearer traffic 5 46

4 Videoconferencing bearer traffic 4 32–39

3 Signaling 3 24–31

2 High-priority data 2 16–23

1 Batch / scavenger traffic 1 8–15

0 Default / best-effort 0 0

Page 14: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 14

LAN and WAN QoS

Video Port

Data Port –voice (Auxiliary

VLAN) and data packets

Phone marks voice –both data and signaling.Switch trusts:•Voice CoS = 5•Voice Signal CoS = 3

Switch marks:•Video CoS = 4•Video Signal CoS = 4•High Priority CoS = 2•Regular data CoS = unmarked (0)•Low priority CoS = 1

Trusted Edge Switch

Video Camera

CBWFQ Router (using ACLs)

Data Center

Router performs congestion management based on per-class behaviors.LLQ for:• IP Precedence 5 for voice traffic (RTP)CBWFQ for:• IP Precedence 4 for production video traffic• IP Precedence 3 for voice control traffic (SIP, H323, and MGCP)• IP Preference 2 for high priority dataWFQ for:• IP Precedence 0 (regular traffic)Scavenger for:• IP Precedence 1 (low priority traffic)

LLQ and CBWFQDiffServ

FC

Network

Desktop Network

Page 15: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 15

Vendor Strategy

Selected single vendor with single project team

Diversity through paths, equipment

Work with 3rd party vendors where necessary

Collected and consolidated contracts

Business partner relationship

Page 16: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 16

Hub Site Design

Page 17: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 17

Hub Sites – Equipment

Cisco Hub Office or Carrier CO

7606 7606

7200 7200

Carrier OC-3 / OC-12 Carrier OC-3 / OC-12

6509 6509

Gig E to Metro Rings

DS-3

OC-3

OC-12

T-1 Cisco OfficesOC-3 Port

DS-3 PortCisco Offices

Page 18: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 18

Hub Sites – Telco Co-locationWithout Telco co-location With Telco co-location

Carrier COCarrier CO

Cisco Hub Office

Advantages:Advantages:

•Eliminates need for:

On-site IT personnel

Hardened site closet (power, security, etc.)

•Reduces access line costs significantly

Disadvantages:Disadvantages:

•Reduced control

• Increased management overhead

•Slight added vendor cost for hosting

Cisco OfficesCisco Offices

Cisco Hub Office

All sites except RTP and some SJ circuits use vendor Data Center co-location: Chicago, Denver,

Redwood City, New York, Atlanta, Dallas.

Page 19: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 19

Cisco Remote Site

WAN POP BWAN POP B

IDCIDC

IDC to Remote Design

Access Provider AAccess Provider B

IOC A

WAN POP AWAN POP A

IOC B

Vendor POP AVendor POP A Vendor POP BVendor POP BDS3 MUX

To WAN POP 4To WAN POP 2

DS1DS3

OC3

OC12

diverse

To WAN POP 1 To WAN POP 3

Access Lines Vendor A

Access Lines Vendor B

Page 20: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 20

IP AddressingSJ (40%) and RTP (10%) kept current addressing; others required renumbering*

Each hub received an Internet-routing capable /19 (or more), plus RFC1918 block

Addressing plan left room for growth, and took into account possible future topologies, to simplify routing plans.

* Large engineering sites have recently been renumbered. Clear communication with client base of reasons, dates, and transition plans was required

Page 21: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 21

Migration Steps

Page 22: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 22

Americas US / Canada WAN Backbone –August 2001 (before Rearchitecture)

Raliegh

Boston

Kanata

San Jose

Note: Average site circuit length = 1800 miles

Page 23: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 23

Phase 1 Rearchitecture - Sep ’01 – Nov ‘01

San Jose

Raleigh

Boston

Kanata

DS-3

OC-3

OC-12

Page 24: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 24

Phase 2 Rearchitecture: Nov ’01 – Apr ’02

San Jose

Raleigh

New York City

Kanata

DS-3

OC-3

OC-12

Denver

Chicago

Page 25: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 25

Phase 3 Rearchitecture: April ’02 - Sep ‘02

San Jose

Denver

Dallas

Chicago

Raliegh

New York Cicy

Kanata

Atlnata`

DS-3

OC-3

OC-12

Note: Average site-to-hub circuit length = 260 miles

Last site installed

Sept 30 ‘02

Page 26: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 26

Results

Page 27: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 27

Round-trip Performance Improvementin 15 Longest Peer-to-Peer Connections

Site at Hub A Site at Hub B Current (ms) Proposed (ms)

Dif Abs dif (ms)

Chelmsford Chelmsford 227 12 1725% 214RTP RTP 222 13 1649% 209Chelmsford Kanata 249 53 366% 195Chelmsford RTP 225 37 505% 187Atlanta Atlanta 206 24 752% 181Atlanta RTP 214 32 558% 181Atlanta Chelmsford 217 57 280% 160RTP Kanata 246 87 183% 159Chicago RTP 194 41 370% 153Chelmsford Chicago 197 49 299% 147Chicago Chicago 166 19 795% 147Chicago Kanata 218 71 208% 147Atlanta Kanata 238 98 144% 141Atlanta Chicago 186 61 204% 125Atlanta Dallas 174 50 249% 124

Page 28: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 28

Bandwidth Increase: 368%

5563396345 408

565

0100200300400500600700800900

1,000

Circ

uit

Ban

dwid

th S

um

($M

bps)

2001 2002Architecture

Site Bandwidth Comparison

DS-3ATMT-1 TDMT-1 Port Frame768K port Frame

2001 Architecture

768 K portsT-1 portsT-1 TDM ATMDS-3

Total = 264 Mbps

Total = 972 Mbps 708 Mbps increase

2002 Architecture

768 K portsT-1 portsT-1 TDM ATMDS-3

Page 29: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 29

Cost Increase: 4%

$236

$462

$500

$773

$267$206

$0

$200

$400

$600$800

$1,000

$1,200

$1,400

Mon

thly

Circ

uit

Cos

ts ($

K)

2001 2003Architecture

Architecture Cost Comparison

Eng OfficeAccessSales OfficeAccessWAN Backbone

2001 Architecture

CAPNetFSOECS

2002 Architecture

CAPNetFSOECS

Total = $1,198K Total = $1,246K $48K/mo increase

Page 30: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 30

Cost Fundamentals

(1200 mile leased line $ > Frame $ > 800 mile leased line $)

Increased backbone costs offset by reduced tail circuit costs

–More sites, closer to regional hubs reduces prices

–Cisco’s 120 sites, avg 270 miles from hubs, broke even at dual T-1 access

–Increase site #, decrease site distance, reduce tail circuit bandwidth, all would result in lower costs

–Fiber glut brought down all leased line prices – from OC-12 to T-1 (vendors needed to learn how to price OC-12 in 2001)

Homing OC-12 into vendor IDC saved $50K+ per hub site / mo in 2002

Page 31: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 31

Cisco Global WAN Backbone - 2007

Gigabet EthernetOC12OC3DS3

Cisco Backbone WAN 2007Legend

Sydney

India

Singapore

San Jose

Redwood City

Denver

DallasAtlanta

RTPNew York

Kanata

Chicago

Amsterdam

London

Hong Kong

Tokyo

MPLS VPN

Page 32: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 32

Global Peer to Peer Services in placeIP Telephony: All IP Telephony

–262 PBXs and Key Systems 13 CallManager Clusters in Data Centers–70,000 IP Hardware phones, 30,000 IP Communicator (software phone) images

Unity Voicemail: All IP Voicemail –160 Voicemail servers 12 Unity Voicemal server clusters in Data Centers–59,000 Unity Voicemail boxes

Contact Centers: All IPCC –17 Contact Centers supported by 2 Intelligent Contact Managers (ICMs) and 5 IPCC Clusters.– 10 Million calls per year handled by 1,400 agents

Video: Cisco Unified Video Advantage–17,000 desktop / laptop Video cameras in use within Cisco.

MeetingPlace Collaboration–12+ million minutes per month voice / data collaboration

Telepresence–120 sites planned for Cisco worldwide

Page 33: North America WAN Rearchitecture for Peer- to-Peer ... · NYC Kanata Atlanta. ... IP Addressing SJ (40%) and RTP (10%) kept current ... Cost Fundamentals (1200 mile leased line $

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicCoC_IT_Case_Study 33

For additional Cisco IT Case Studies on a variety of business solutions, go to

Cisco on Cisco: Inside Cisco ITwww.cisco.com/go/ciscoit