architecting data centre networks in the era of big data ...€¦ · global marketing architecting...

Global Marketing

Architecting Data Center Networks in the era of Big Data and Cloud

Brad Hedlund Spring Interop—May 2012

VIDEO of this session: http://bradhedlund.com/?p=3912

http://bradhedlund.com/?p=3912

Global Marketing

•Centralized, Scale-up Layer 2 networks •Monstrous chassis switches

2

Distributed, Scale-out Layer 3 fabrics Efficient fixed switches Open, industry standard protocols

TRILL OpenFlow VEPA SPB

THE SAME OLD

Or a Different Approach

Two approaches to DC Networking

Brad Hedlund

Global Marketing

Networks that suck for Cloud & Big Data

3

PARTITIONED CAPACITY

Core

Dist

Access

“Data center networks are in my way” -James Hamilton, AWS

VM

Network Topology

Capacity Topology

Brad Hedlund

Global Marketing

Networks that Don’t suck for Cloud & Big Data

4

UNIFORM CAPACITY

Spine

Leaf

All points equidistant

VM

Network Topology

Capacity Topology

Brad Hedlund

Global Marketing

Rack 2

Node

Node

Node

Node

Job Tracker

Rack 1

Node

Node

Node

Node

Name Node

Rack N

Node

Node

Node

Node

Node

switch

Big Data

• Inverse Virtualization •Workloads orchestrated like cattle • L2 or L3 network. Does it matter?

5

Rack 3

Node

Node

Node

Node

Secondary NN

Rack 4

Node

Node

Node

Node

Client

switch switch switch

switch switch

switch

TCP

TCP

Client

Brad Hedlund

TCP TCP World

Global Marketing

Basic requirements of Cloud (IaaS)

• Secure, Scalable Multi Tenancy

• Location independence

•On Demand virtual networks

6

VM VM

FW

VM VM

LB

switch switch

switch switch

switch switch

Physical Network

Virtual Network

World

Brad Hedlund

Global Marketing

Blend the Virtual and Physical Networks

•Tenant subnet = Network VLAN

7

VM VM VM VM

switch

switch switch

VM VM

VLAN 10

VLAN 20

Host Host

vSwitch vSwitch

VM VM

Brad Hedlund

Global Marketing

Abstract the Virtual Network from Physical

•Network Virtualization Overlay •Tenant subnet = Software VLAN

8

VM VM VM VM

switch

switch switch

VM VM

Host Host

vSwitch vSwitch

VM VM

Segment ID 20

Segment ID 10

Brad Hedlund

Global Marketing

Scale-up centralized Layer 2

• 2-post Rooted Architecture

• Centralized L2/L3

• L2/L3/ARP table scale?

• Scale w/ Bigger Boxes

• Precious Pets

• VLAN Provisioning?

• Broadcasts

9

VM VM VM VM

vSwitch vSwitch

L3

L2

Brad Hedlund

Global Marketing

(16)

(2) (8)

(64)

1980 Server ports

Scale-out Layer 3 Leaf/Spine Fabric

• Mesh from Leaf to Spine

• OSPF, ISIS, BGP, TRILL

• ToR w/ 16 uplinks (ECMP)

10

768 Server ports 3072 Server ports 6144 Server ports

(16)

(128)

• Non-blocking Spine

• 3:1 @ ToR

• 128 port 2RU Spine

L3

L2

Brad Hedlund

Global Marketing Brad Hedlund

6144 Server ports

(16)

(2) (8)

(64)

Uniform fabric for Cloud & Big Data

11

L3

L2

(16)

(128)

VM VM VM VM

vSwitch vSwitch

Rack 3 Rack 1

Name Node

Rack 2

Job Tracker

Rack N

Secondary NN Node

Node

Node

Node

Node

Client

Node

Node

Node

Node

Client

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Block I/O NAS Object

Storage Access

Hadoop

Database


(16)

(2) (8)

(64)

Attaching Services & North/South

12

(16)

(128)

Firewall Firewall

World

LB LB

vswitch VM VM VM

vswitch VM VM VM

vswitch VM VM VM

vswitch VM VM VM

vswitch VM VM VM

vswitch VM VM VM Rack 1 Rack N

Node

Node

Node

Node

Node

Client

Node

Node

Node

Node

Client

Name Node Job Tracker

Secondary NN

L3

L2

x86 Gateways

Global Marketing

Generic Logical Architecture 1

13

World

Brad Hedlund

FW

LB

FW

LB

VM VM VM

Green Co. Orange Co.

L3 NAT

L3 NAT

L2

L3

L2

L2

Fabric DC router • Overlay based L2 • Physical/Static FW

VM VM

Big Data

L2

Global Marketing


14

World

Brad Hedlund

FW

LB

FW

LB

VM VM VM


L3 NAT

L3 NAT

L2

L3

L2

L2

Fabric DC router • Overlay based L2 • Virtual/Mobile FW • Overlay Gateway

Pub DMZ

Big Data

VM VM

L2

Global Marketing


15

World

Brad Hedlund

FW

LB

FW

LB

VM VM VM


L3 NAT

L3 NAT

L2

L3

L2

L2

Fabric DC router • No Overlays • TRILL based L2 • Virtual/Mobile FW

Pub DMZ

TRILL

Big Data

VM VM

L2

Global Marketing

Density: Fixed vs. Chassis

0

20

40

60

80

100

120

140

2008 2010 2012 2014

Chassis

Fixed

16

10G per RU @ Line Rate (L3)

Brad Hedlund

Global Marketing

Power: Fixed vs. Chassis

0

2

4

6

8

10

12

14

16

18

2010 2012 2014

Chassis

Fixed

17

Max Watts / Line Rate 10G (L3)

Brad Hedlund


(16)

(2) (8)

What are the Challenges?

(16)

(128)

• Deployment & Cabling

• Configuration & Policy

• Monitor & Troubleshoot

• Layer 2 (TRILL?)

• Design Best Practices

Dell Fabric Manager

L3

L2

18

Fabric Manager Dell

Design Templates Validate deployment Automate fabric configuration Monitoring & Operations

Global Marketing

Webinar: CLOS Fabrics Explained

19 Brad Hedlund

http://closfabric.eventbrite.com/ Wednesday, June 20, 2012 from 10:00 AM to 1:00 PM (ET)

HOST CO-HOST

Ivan Pepelnjak

Brad Hedlund

http://closfabric.eventbrite.com/

Global Marketing

(2) (8)

Three Stage Layer 3 Leaf/Spine Fabric

20

(64)

(128)

L3

L2 24,576 Server ports

(512)

• Non-blocking @ top tiers

• Default route @ ToR & Leaf

• Leaf+ToR mesh groups

• ~8usec worst case

0/0

0/0

/26 /26

/26

Brad Hedlund

Global Marketing

The case for 40G QSFP switch ports

21

QSFP SFP+ SFP+ SFP+ SFP+

10G 10G 10G 10G 10G 10G 10G 10G

VS

$1,800 $1K $1K $1K $1K

Brad Hedlund

32 ToR

$512K $230K

Global Marketing

Comparing Fabric efficiencies of Fixed vs. Chassis designs

Brad Hedlund May 2012

• Supplemental Section

Global Marketing

Total Fabric Power: Chassis vs. Fixed

0

50

100

150

200

250

384 2048 4096 8192

Chassis

Fixed

23

non-blocking KW

Fabric size

Brad Hedlund

Global Marketing

Total Fabric RU: Chassis vs. Fixed

0

100

200

300

400

500

600

700

384 2048 4096 8192

Chassis

Fixed

24

RU

Fabric size

non-blocking

Brad Hedlund

Global Marketing

(2) (8)

8192 non-blocking Fabric

25

(64)

(128)

8192 non-blocking

Brad Hedlund

• 384RU

• 153.6KW

• 8192 ISL

• 192 switches

Global Marketing


26

8192 non-blocking

Brad Hedlund

• 608RU

• 216.3KW

• 8192 ISL

• 288 switches

(32)

(256)

Global Marketing

(2) (32)


27

(64)

4096 non-blocking

Brad Hedlund

• 192RU

• 76.8KW

• 4096 ISL

• 96 switches

Global Marketing


28

4096 non-blocking

Brad Hedlund

• 304RU

• 108.1KW

• 4096 ISL

• 144 switches

(16)

(128)

Global Marketing

(2) (16)


29

(32)

2048 non-blocking

Brad Hedlund

• 96RU

• 38.4KW

• 512 ISL

• 48 switches

Global Marketing


30

2048 non-blocking

Brad Hedlund

• 152RU

• 54KW

• 2048 ISL

• 72 switches

(8)

(64)

Global Marketing

(2) (4)


31

(6)

384 non-blocking

Brad Hedlund

• 20RU

• 8KW

• 96 ISL

• 10 switches

Global Marketing


32

384 non-blocking

Brad Hedlund

• 26RU

• 18KW

• 384 ISL

• 14 switches

(2)

(12)

Global Marketing

(2)

(4)


33

256 non-blocking

Brad Hedlund

• 12RU

• 4.8KW

• 64 ISL

• 6 switches

Global Marketing


34

256 non-blocking

Brad Hedlund

• 22RU

• 6.7KW

• 256 ISL

• 10 switches

(2)

(8)

Global Marketing


0

10

20

30

40

50

60

70

80

768 1536 3072 6144

Chassis

Fixed

35

3:1 oversubscribed KW

Fabric size

Brad Hedlund

Global Marketing


0

50

100

150

200

250

768 1536 3072 6144

Chassis

Fixed

36

RU

Fabric size

Brad Hedlund

3:1 oversubscribed

Global Marketing

(2)

(16)

768 @ 3:1 oversubscribed Fabric

37

768 @ 3:1

Brad Hedlund

• 20RU

• 7.2KW

• 64 ISL

• 18 switches

Global Marketing

768 @ 3:1 non-blocking Fabric

38

256 non-blocking

Brad Hedlund

• 30RU

• 7.5KW

• 256 ISL

• 18 switches

(2)

(16)

Global Marketing

(2) (4)


39

(32)

1536 @ 3:1

Brad Hedlund

• 40RU

• 14.4KW

• 128 ISL

• 36 switches

Global Marketing


40

1536 @ 3:1

Brad Hedlund

• 54RU

• 17KW

• 512 ISL

• 34 switches

(2)

(32)

Global Marketing

(2) (8)


41

(64)

3072 @ 3:1

Brad Hedlund

• 80RU

• 28.8KW

• 1024 ISL

• 72 switches

Global Marketing


42

3072 @ 3:1

Brad Hedlund

• 108RU

• 34KW

• 1024 ISL

• 68 switches

(4)

(64)

Global Marketing

(2) (8)


43

(16)

(128)

6144 @ 3:1

Brad Hedlund

• 160RU

• 57.6KW

• 2048 ISL

• 144 switches

Global Marketing


44

6144 @ 3:1

Brad Hedlund

• 216RU

• 68KW

• 2048 ISL

• 136 switches

(8)

(128)

Global Marketing


0

200

400

600

800

1000

1200

1400

12288 24576 49152 98304

Chassis

Fixed

45

3:1 oversubscribed - Massive KW

Fabric size

Brad Hedlund

Global Marketing


0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

12288 24576 49152 98304

Chassis

Fixed

46

RU

Fabric size

Brad Hedlund

3:1 oversubscribed - Massive

Global Marketing

(2) (8)

12,288 @ 3:1 oversubscribed Fabric

47

(64)

(256)

12,288 @ 3:1

Brad Hedlund

• 448RU

• 166.4KW

• 5120 ISL

• 352 switches

(32)

Global Marketing


48

12,288 @ 3:1

Brad Hedlund

• 432RU

• 136.3KW

• 4096 ISL

• 272 switches

(16)

(256)

Global Marketing

(2) (8)


49

(128)

(512)

24,576 @ 3:1

Brad Hedlund

• 896RU

• 332.8KW

• 10,240 ISL

• 704 switches

(64)

Global Marketing


50

24,576 @ 3:1

Brad Hedlund

• 1120RU

• 328.9KW

• 16,384 ISL

• 800 switches

(32)

(256)

(512)

Global Marketing


51

(256)

(1024)

49,152 @ 3:1

Brad Hedlund

• 1792RU

• 665.6KW

• 20480 ISL

• 1408 switches

(128)

Global Marketing


52

49,152 @ 3:1

Brad Hedlund

• 2240RU

• 657.9KW

• 32,768 ISL

• 1600 switches

(64)

(1024)

(512)

Global Marketing


53

(512)

(2048)

98,304 @ 3:1

Brad Hedlund

• 3584RU

• 1331.2KW

• 40960 ISL

• 2816 switches

(256)

Global Marketing


54

98,304 @ 3:1

Brad Hedlund

• 4480RU

• 1315.8KW

• 65,536 ISL

• 3200 switches

(128)

(2048)

(1024)

The power to do more

55

architecting data centre networks in the era of big data ...€¦ · global marketing architecting...

Documents