VERIZON CLOUD COMPUTE AND THE AMD SEAMICRO 15000

Paul Curtis Chief Architect Verizon Cloud Compute APU13

2

Collaboration

•  Unbelievable amount of support from Seamicro

3

Verizon cloud development goals

•  Very few different hardware components

•  Consistent predictable performance

•  Secure

•  High performance

•  Highly available

•  No modification to customer applications

•  No special purpose hardware

4

Verizon Cloud Differentiation

•  Value for Performance –  User defined availability and performance

–  User defined resources

•  Reserved Performance –  Network, Storage and Compute

•  Workload Simplicity –  Seamless integration with other deployments

–  Single point of control

•  Security ‒  Market leading security capabilities

‒  Embedded into every aspect of platform

•  Continuum of Services ‒  Bridging private, public and hybrid clouds

‒  Allow the blending with colocation, managed services, networking

5

SM15000 SYSTEM 10 Rack Units, draws 3-3.5 KW

!  Compute –  Up to 512 Opteron, Xeon or Atom cores in 10 RU –  2,048 cores in a rack –  Up to 64GB DRAM/socket = 4 terabytes/system

!  Networking –  10 Gbps half duplex bandwidth to each CPU socket –  16 x 10GbE Line Rate uplinks to the network

!  Storage –  Up to 1,408 disks: HDD or SSD –  Up to 128 Terabytes of internal SSD storage –  Up to 5.3 Petabytes of storage

!  Fabric –  1.28 Tbps Freedom Supercompute Fabric

!  Software –  Off the shelf OS, Hypervisors

6

Hardware architecture

•  There are only three hardware component types. This simplifies maintenance

–  Arista 7508 a 384 port x 10Ge non-blocking L2 switch

–  AMD Seamicro SM15000

–  SSDs

•  Network connections

7

Hardware diagram

Juniper  MX960  for  external  connectivity

Arista  7508

Up  to  90  AMD  Seamicro  SM15000s

4x10Gb  links  from  arista  to  each  chassis

8

Verizon’s use of seamicro chassis

•  160 GB of external bandwidth (network and storage)

•  54 Server cards for customer loads

•  2 Server cards for Verizon orchestration

•  8 Server cards for storage services

•  ~1000000 IOPs

•  96 T usable SSD storage

9

Combine Hardware and Verizon software to get

•  A flat layer 2 ethernet switch –  ~12000 port 1 Gb/sec –  ~1500000 vlans –  8.5M mac address table entries –  11.5M traffic flows. –  Software configurable

•  A storage array –  90M IOPs –  8.6 PB of SSD storage

•  Scalable router firewall 1Gb- 400Gb/sec •  Scalable load balancers 1Gb-400Gb/sec •  Configurable IO performance

10

Network Packet flow VM

Queue

Nic

text

Queue

Nic

Hypervisor

NPU

10  G  NIC

Queue

Hypervisor  presents  nic  of  specified  speed  to  VM.  Back  pressure  applied  by  hypervisor  

Hypervisor  fairly  mixes  flows  from  

different  VMs

Prioritized  queue.  

Arista  Switch

Queue

Queue

10  G  NIC

Shaped  to  max  speed  

of  receiving  

nic

Shaped  to  max  speed  of  receiving  

Nic

Policer

VM

Queue

Nic

text

Queue

Nic

Hypervisor

Random  packet  

drop  back  pressure  form  

destination  queue

Limited  to  max  Nic  

speed

NPU

Layer  2  Switching

Layer  2  Switching

11

Networking Layer 2

•  Hypervisor –  Shapes egress traffic

•  NPU –  Provides true layer 2 ethernet switching –  Polices ingress flows –  Shapes egress flows

•  Arista 7508 –  Lots of bandwidth

•  Remote congestion control –  Switch learns speeds of remote flows –  Switch performs remote drop if destination is congested

•  Hardware based security –  Each customer network is on its own vlan

•  Software configurable

12

o

o

VM

Paravirt (xvdb) Storage VLAN AoE Initiator

HV  

Ethernet NPU

NPU

Ethernet AoE Target

AoE Initiator Block ZFS AIO

Ethernet

Ethernet

Block Device(s) AoE Target

SCARD  

SS  N  

Arista

Data I/O Stack

13

Storage

•  Hypervisor –  Shapes disk traffic (IOPs and Bandwidth) –  Participates in disk replication

•  AoE –  Storage over layer 2 ethernet –  Allows storage targets to be any where in world –  Shared volumes

•  Replication •  NPU

–  Shapes read and write bandwidth •  Storage Service

–  Snap shots –  Raid

•  Storage card –  AoE target

14

Networking Layer 3+

Layer 3 and above network services just work since they are all based on layer 2 networking.

•  Soft routers •  Load balancers •  Public IP (No Nat) •  Tunnels •  Wan optimizers

15

Inter-data center features

•  Single user interface

•  Networks can span multiple data centers

•  Replicated disks can span multiple data centers

•  Taking advantage of being part of a network company

16

Availability

•  No single point of failure for network traffic –  “Bonded nics” –  “Bonded NPUs” –  Fabric reroutes itself –  Multiple paths through arista switches

•  No single point of failure for replicated storage –  Raid 1 on SSDs –  Multiple storage servers –  Option to have replicated volumes span data centers

17

Security

•  Physical security

•  DDOS

•  Network security –  Customer traffic on independent VLANs –  Untrusted entities (Hypervisors) firewalled from rest of system

•  Storage security –  Each volume on a separate vlan –  Storage vlans firewalled (only AoE traffic, no target to target traffic)

•  Management software –  Audit logs –  Security alerts

18

Possible Applications

•  Move a current three tier app with your choice of soft router/firewall/load balancer into the cloud

•  Bridge a network from your data center to one in the cloud •  Move XEN and VMWare VMs into the cloud without modification •  Write a clustered app using shared storage •  Configure an applications performance so that you know it won’t fall over

when it is 3:00 in the afternoon and the cloud gets busy •  Write and test a new L3 protocol •  Voice •  Storage arrays •  Network devices