sky core network positioning for massive video demand ... · sky core network , uknof, 17jan 2018 -...

Sky Core Network

Positioning for Massive Video Demand

UKNOF - 17 Jan 2018

Tim Rossiter – Core Network Architect, Sky Network Services

Sky Core Network , UKNOF, 17Jan 2018 - 2

Positioning Sky Network for Massive Video Demand

• UK Broadband Usage and Pricing

• Sky Network Structure and Evolution

• 2017/18 Capacity Upgrade

• Cost Reduction by Content Delivery (CDN) Localisation


Uk & International Historic Traffic Trends

UK 58%

90% in 2016

40% in 2017

YoY Growth

USA 20%

Source: Ofcom International & UK Communication Market Reports


Volume - Sky Traffic Growth (10 Year History)

• Sky Experience Similar to UK

• Total Traffic x100 over 10 years => 58% YoY

• Per Sub traffic x25 => 38% YoY


• Internet connectivity cost ~ £30 – £50 p.m (also inflation ~50% )

• Access Data rates increased x 100 in 12 years

• Cost per bit Reduces x 67% YoY, or 10 ->1 in 6 years.

• ISP per sub revenue unchanged, Access rate increased by x100

UK Broadband Price Trend

Since 2000


P&T

P&T

Sky Network Structure (Simplified)

Super

CoreSuper

Core

P Router

NCS6k

PoP PE

Rem

CDN

CDN

Core

CDN

PoP PE

PoP

CDN

Exch

Exch

Exch

Exch

CPE

Exch Exchange

DSL or GEA


Resilience and Provisioning Policy

Resilience – 1 + 1 or 1 : N

• Dual Rings, Dual feed to each PoP

• Dual P&T sites

• Multiple CDN (1:N )

• 2 or 3 Entry points to each Super-Core site

Dimensioning

• Serve peak traffic in the event of a single major failure

• Major failure: Loss of one entire site or fibre (up to 80 layer 3 paths)

Capacity Consequences

• Largest Router Capacity ≈ Traffic Served

( In+Out ) x Protection + Interconnect + Extras – Partial Fill

• 8 Tb Traffic Requires 8 Tb Router


Core Router Evolution

2005 Initial Cisco CRS1

• Traffic < 50G, 40Gb per slot

=> 640 Gb chassis

2011 Upgrade to CRS3. Fabric, Line cards

• Traffic 350G, 140Gb per slot

=> 2240 Gb chassis (x3.5)

2014 - Replace with NCS6008

-> New Chassis, Fork Lift

• Traffic 2 Tb, 8Tb per chassis (x12.5)

2017/18 - Add NCS5516

-> New Chassis

• Largest NCS6008 almost full

• Traffic ~ 8 Tb, 55 Tb per chassis (x 90)

CRS1

CRS3

NCS6k

NCS6k B2B

NCS5k

• Retain NCS6008 in Back to Back Config, 16 Tb per chassis


NCS6008 and NCS5516NCS6008

• 8 Slots, 1Tb per slot => 8 Tb node

• 100G or ( 10x10G) ports

• High Functionality

• Roadmap: B2B, 2Tb per slot => 16-32 Tb

NCS5516

• 16 Slots, 3.6Tb per slot => 55 Tb node

• 36 x ( 100G or (4 x 10G) ) ports/slot

• Low touch – MPLS P-Router, No B2B


NCS6008 Back to Back

• 96 x 100G CXP SR10 Connectors => ~1000 Fibre Pairs!

• 2 x 40G Ethernet for processor links

• Shelves can be ~ 100m apart


• CFP Original 100G

Bulky, Expensive

• CPAK,

Cisco Proprietary

• CXP SR10 (10x10G)

• QSFP-28 (4x25G)

– Standard SR4, LR4 + Cost Effective Variants

– Large Quantities (Data Centre Usage)

– SR-4 very low cost

CPAK

100G Optics – CFP, CPAK, QSFP


Matching Broadband Cost Trend ?

• Port savings: 10G -> 100G, Router silicon, Moore’s Law, Reduced Functionality, LR->SR

• NCS5516 + low cost QSFP SR4 nearly matches BB price erosion, 10 : 1 in 6 years


Migration Steps for NCS5516 in Super-core

• Different approach from previous NCS6008 ‘whole ring’ introduction

– Growth slower => Longer timescale for transitions

– No ‘10G -> 100G’ transition issues

– Super-core transmission fully 100G

– Not a ‘big bang’ migration

• Upgrade one node at a time (in multiple stages)

• Birmingham first

• Staged migration modelled to verify capacity for all failures


P&T

P&T

Migration 1: Insert NCS65516

Super

Core

Birmingham

P Router

NCS6k

PoP PE

CDN

Core

CDN

PoP PE

Exch

Exch

Exch

Exch

CPE

NCS5k


P&T

P&T

Migration 2a: Migrate some feeds and consumers

Super

Core

Super

Core

P Router

NCS6k

PoP PE

CDN

Core

CDN

PoP PE

Exch

Exch

Exch

Exch

CPE

NCS5k


P&T

P&T

Migration 2b, 2c … : Migrate remaining feeds and consumers

Super

Core

Super

Core

P Router

NCS6k

PoP PE

CDN

Core

CDN

PoP PE

Exch

Exch

Exch

Exch

CPE

NCS5k


P&T

P&T

Migration 3: Isolate Original NCS6008 & Decom

Super

Core

Super

Core

P Router

NCS6k

PoP PE

CDN

Core

CDN

PoP PE

Exch

Exch

Exch

Exch

CPE

NCS5k


P&T

P&T

Migration 5: ‘Old’ NCS6k Makes B2B NC6k on Other Ring

Super

Core

P Router

NCS6k

NCS6008 B2B

• 16 Tb with 1Tb

cards

• SR4 CPAK

• Accepts 2Tb cards

(-> 32 Tb)

• QSFP/CPAK

Core

CDN

PoP PE

Exch

Exch

Exch

Exch

CPE

NCS5k

Migration ‘Waterfall’

pr2 Lon1 N6k B2B

Leeds N6k B2B

Birmingham N6k B2B

Lon2 N6k B2B

pr3 Lon1 NCS5500

Leeds NCS5500

Birmingham NCS5500

Lon2 NCS5500

Jul 17 Oct 17 Jan 18 Apr 18 Jul 18

• Red triangles: Slot exhaust time, Green cells: Quarter for upgrade

• NCS6k replaced by new NCS5k

• Released NCS6k chassis incorporated into NCS6k B2B configuration

• Staged operation, One upgrade per quarter


Capacity Issue Sorted

• New NCS5516 Ring sufficient for 5+ years

• Remaining NCS6008 B2B exhaust earlier, Replace with NCS5516 ?

• New Cost point with NCS5516 silicon, low-touch and SR4 QSFP

• Router port price trend close to Broadband price Trends

• Any problems ?


Transponder Price Trend

• Transponder price (per Gb) falling at half rate of router port cost ( 10:1 in 12 years)

• 10 Gb Easy (in retrospect) => On-Off Modulation, FEC

• 100G Very Complex => QAM Modulation, Coherent Detection ( 2 Detectors I&Q )

Dual Polarisation ( 2 lasers, 2 x 2 Detectors: H&V x I&Q ),

Ultra High Speed A/D Converters

Dispersion Compensation DSP, Stronger FEC

• Distance is becoming (relatively) more expensive

• Localise Traffic Sources


How to achieve further cost reductions ?

• Most traffic originates from CDN (On-Net or Off-Net)

1. CDN Private Peering ( v.s. Transit and Public Peering)

2. On-Net CDN - 65% of traffic sourced On-Net

3. Localise CDN Source

• How much can be Localised to minimise carriage across network ?

• Practicalities Matter – Diminishing Returns, Long Tails


P&TP&T

CDN Deployment Options – Off-Net via P&T

Super

CoreSuper

Core

P Router

PoP PE

OffNet

CDN

CDN Cache

PoP PE

Exch

Exch

Exch

CPE

OffNet

CDN

• At least 4 transmission links

(Including Protection)

• Peering Interconnects

• Multiple Router Transitions


P&T

CDN Deployment – On-Net, Remote CDN

Super

Core

Rem

CDN

PoP PE

Exch

Exch

Exch

CPE

• Multiple transmission links

• Fewer Router Transitions


CDN Deployment – On-Net, Super Core

Super

CoreSuper

Core

Core

CDN

PoP PE

Exch

Exch

Exch

CPE

• Minimal Core Links

• Traffic still on PoP Links

• SuperCore Constraints

– Space Limited

– Expensive

– Power restrictions


CDN Deployment Options – PoP CDN

Super

CoreSuper

Core

PoP PE

PoP

CDN

Exch

Exch

Exch

CPE

Core

CDN

• No Primary Traffic on Core

or PoP links

• If local CDN fails, sufficient

capacity on PoP links for

central fall-back

• Requires per PoP mapping

to localise traffic source


CDN Deployment – PoP CDN feeding remote PoP

Super

CoreSuper

Core

PoP PE

PoP

CDN

Exch

Exch

Exch

CPE

• Local CDN serves other PoPs

• Upstream ‘for free’


CDN Deployment – Exchange, Set Top Box

Super CoreSuper Core

PoP PE

PoP

CDN

Exch

Exch

Exch

CPE

Exchange CDN

• Problematic

• Insufficient traffic

• Power/Space constraint

• BNG architecture

STB

‘Home Pre-Caching’ ?

• Pre-load by satellite side

channel

• Limited application

- For very popular material


PoP CDN Early Deployment – Low Traffic

• 5 larger providers

(+ many more off-net)

• Representative PoP sizing ( actually 60 PoPs)

PoPs, A few large + Long tail

• Need > 10Gb to warrant PoP CDN => Green Cells

• Early Deployment: 9 Locations, 31% CDN Localised

• Other Locations ( < 10G localised) serve Local+Other PoPs.

Provider Fraction 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Prov1 21% 26 17 13 11 9 8 7 7 6 6 5 5 5 4 4 4 4 4 4 3

Prov2 17% 21 13 10 9 7 7 6 5 5 5 4 4 4 4 3 3 3 3 3 3

Prov3 13% 15 10 8 6 6 5 4 4 4 3 3 3 3 3 3 2 2 2 2 2

Prov4 6% 8 5 4 3 3 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1

Prov5 5% 6 4 3 3 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1


CDN PoP – Higher Traffic Levels

• Further Growth: 4 x Traffic => 63 Deployments, 88% Localised

• 130% More Deployments, 50% More traffic localised

• Diminishing Returns


Prov1 21% 103 67 52 43 37 33 29 27 24 23 21 20 19 18 17 16 15 15 14 14

Prov2 17% 82 54 41 34 30 26 23 21 20 18 17 16 15 14 13 13 12 12 11 11

Prov3 13% 62 40 31 26 22 20 18 16 15 14 13 12 11 11 10 10 9 9 8 8

Prov4 6% 31 20 16 13 11 10 9 8 7 7 6 6 6 5 5 5 5 4 4 4

Prov5 5% 24 16 12 10 9 8 7 6 6 5 5 5 4 4 4 4 4 3 3 3


Prov1 21% 51 34 26 21 18 16 15 13 12 11 11 10 9 9 8 8 8 7 7 7

Prov2 17% 41 27 21 17 15 13 12 11 10 9 8 8 8 7 7 6 6 6 6 5

Prov3 13% 31 20 16 13 11 10 9 8 7 7 6 6 6 5 5 5 5 4 4 4

Prov4 6% 15 10 8 6 6 5 4 4 4 3 3 3 3 3 3 2 2 2 2 2

Prov5 5% 12 8 6 5 4 4 3 3 3 3 2 2 2 2 2 2 2 2 2 2

• Double Traffic => 27 CDN deployments, 58% of CDN traffic


CDN Localisation History

• Easy to see Benefits of PoP CDN

• Not so easy to realise them

• 5 x CDN providers

• Each with own approach

DNS, HTTP Redirect, + Secret Sauce

• All or nothing

– Specify Local and Central Mapping

– Allow headroom for overflow

• BBPE Software Upgrades

– Rolled out to 100 BBPEs

– Separate BBPE features for IPv6

Provider

Localisation

Sky Specifies

Localisation

BBPE & IPv6

Enhancements


IPv6 Deployment

• Dual Stack alongside IPv4 (eg. Not DS-Lite)

• Both LLU (PPP) and “Fibre” (IPoE) services eligible

• > 5 Million customers eligible and enabled (>95%)

• Support all Sky Hub models (1,2,3) and legacy Sky-supplied Sagem routers.

• > 90% uptake. (Remaining assumed to be user-disabled and/or 3rd party an or very

old routers without compatible IPv6 support)

• New customer default since April 2016

• /56 Prefix Delegation

• 2 year exercise 20% -> 90%


Acknowlegements…

Rapid, well executed program through 2017

Made Possible by Great Team within Sky Networks

• Network Architecture & Planning - Forecast, Modelling, Migration Planning

• Network Development – Testing, Verification, Product Advice

• Network Implementation - Configuration, Installation

• Network Operations - Many overnight migrations

• Project Mgt - Complex Project, Juggling Resources

• Cisco - Early testing support, Expedited delivery

Storm Surfers

Sky Core Network

Positionined for Massive Video Demand

Thank you!

sky core network positioning for massive video demand ... · sky core network , uknof, 17jan 2018 -...

Documents