next wave of data centers & interconnects

Next Wave of Data Centers & Interconnects

10GBase-T, DAC,

40G/100G transceivers,

AOCs, Embedded Modules

9/18/2012 1

Brad Smith Sr. VP LightCounting

Brad@LightCounting.com

Agenda

• What is going on in the data centers that is driving the interconnect space

• 10GBase-T, Direct Attach Copper

• 40G & 100G transceivers

• Active optical cables

• Embedded Optical modules

9/18/2012 2

About LightCounting

9/18/2012 3

http://www.lightcounting.com/reports.cfm

Transceiver Research Databases

High speed interconnect Research LH telecom & FTTx -to- Data centers Ethernet, Fibre Channel, InfiniBand Vertical areas: 40/100G, AOCs, EOMs

Problem with Optical Interconnects All Predictions are 35%-90% Traffic Growth

9/18/2012 4

IEEE BWA, Cisco VNI, LightCounting Model

Virtually all the Internet traffic – goes through an optical

transceiver somewhere in the telecom and datacom network!

9/18/2012 5

Virtually all the Internet traffic – goes through an optical

transceiver somewhere in the telecom and datacom network!

Supplied by transceiver companies… About one quarter away for bankruptcy

9/18/2012 6

Virtually all the Internet traffic – goes through an optical transceiver somewhere in the telecom and

datacom network!

Supplied by transceiver companies… About one quarter away from

bankruptcy!

Problem with Optical Interconnects What’s Wrong with this Picture?

9/18/2012 7

IEEE BWA, Cisco VNI, LightCounting Model Optical Stocks

Traffic going through roof Profits going through the floor

Big Changes in Data Centers Changes in Data Centers Drives Interconnects

9/18/2012 8

M&As Restructure the Industry ~24-months

Systems Cabling

Transceivers Semiconductors

9/18/2012 9

Cabling Companies Buy AOC Startups + BUs

• TE bought Zarlink BU • FCI Bought MergeOptics • Molex bought Luxtera BU • Samtec bought AlpenIO

Result: • $ xx Billion, international connector, cabling, companies • Offering transceiver and AOCs, EOM, VCSELs

– New set of competitors for transceiver companies – Full line of copper & optical products – Power cables, every protocol, every MSA, connector, etc

9/18/2012 10

Cisco Introduced Blade Servers Competes against former server partners

• HP bought 3Com/3Par

• IBM bought Blade Network Technologies

• Dell bought Force10

• Oracle bought Sun Microsystems

• Mellanox bought Voltaire (InfiniBand)

• Intel bought QLogic and Fulcrum (InfiniBand) & Crays HPC Intereconnect BU

Now integrated Server and Switching companies

9/18/2012 11

Cisco Buys Transceiver Companies CoreOptics & LightWire

• CoreOptics: 100G Long haul in-house

• LightWire: Silicon photonics in-house

– 4x25G (100G) -- WDM

– High speed modulation (40G) & integrated electronics

– Data center (500 meters) & Metro (10Km)

Result:

• Servers, switches, and interconnects – all in-house

9/18/2012 12

Software Defined Networking

• SDN Virtualizes the networking hardware

• VMWare bought Nicira

• Citrix bought Nutanix

• Oracle bought Xsigo

• Cisco Systems’ investment in Insieme

• Bottom line: – Big technology companies are buying their way into

the virtualized storage and virtualized networking markets.

– Virtualizations will likely increase all the DC traffic

9/18/2012 13

9/18/2012 14

Same thing that happened with PCs in 1980s with Taiwanese clone PCs and big name brands ……is happening to servers and switches! This will change everything!

Enter the White Box Clones

Servers

• Big Internet related companies (Google, Facebook, Amazon, etc.) buying servers directly from ODMs

• Making designs available for anyone to buy

• Server business already at 22% GM & heading lower

• NET: White box commodity servers!

Switches

• White box ToR and EoR switches

• Buy switch chips from Broadcom, Marvel, Mellanox, …Intel

• Run OpenFlow (open source network switching software)

• No longer special switching software play

• No longer a Microsoft-Intel instruction set lock

• NET: White box commodity switches!

9/18/2012 15

Reactions Reactions likely: • M&As: Buy transceiver companies Cisco/Lightwire • Advanced technology: Use technology as a weapon against

low cost manufacturing white boxes • More highly integrated Blade server-switch boxes • Rack-as-a-Sku” • Non-Intel Alternatives: ARM cores, Power7, SPARC

microprocessors competing with Intel • HPC technologies hit high-end data centers • Software Defined Networking

9/18/2012 16

“Rack as a Sku”

Integrated Blade Server/Switches

All these changes are going to rewrite the Server/Switch and Interconnect landscape going forward

Mez board Copper KR

Server blade

Switch blade

Server-Switch Architectures

9/18/2012 17

2 Sockets 4 Sockets

1RU Servers with Top-of-rack Switches Blade Servers with Integrated Switches

No single solution!

Server & Switch Industry Transitions

9/18/2012

Brad Smith Brad@LightCounting.com

“Rack as a Sku”

Traditional Rack Servers

& Top-of-rack switches DAC, QSFR SR & AOCs

SMF 10Km LR4 transceivers

Internally connected with EOMs

and AOCs

SMF 2Km nR4 transceivers

Multiple Blade Server Bays per Rack Custom (e.g Google)

& White Box Clones DAC, QSFR SR & AOCs

SMF 2Km nR4 transceivers

Today 2013-2014

From Major System Suppliers

Cisco, HP, IBM, etc.

Focus on advanced

architectures & long term

Focus on low prices &

3 yr life cycles

Waiting for New Servers Waiting for 10GBASE-T

Waiting for Europe’s economy to stop crashing

Preparing for the Exa Flood of Data

9/18/2012 19

State of the Industry Mid 2012

State of the Industry • Data centers are largely very cost sensitive;

• Cost governs all decision making (over technology) – Decision algorithm (Cost of new product) / Server cost

– Buy 100G LR4 transceiver OR 8 servers?

• Average and large data center needs 4-5G, not 10G yet

• Getting ready for 10G and 40G QSFP

• Most popular optical transceiver = 1G SFP+ & 10G SFP+

9/18/2012 20

State of the Industry

Servers & Switches

• Intel’s next gen microprocessor “Romley” server architecture – delayed ~6-9 mo’s

– 4 cores 10 cores (14 cores AMD) -- PCI Express Gen 2 Gen 3 with twice I/O

– QuickPath to bypass bus bottlenecks -- 32-nm very energy efficient

• Top-of-rack & End-of-row switches are 10G SFP+ + 40G QSFP ready but ….

• Entire industry was “Waiting for new Romley servers”

• And 10GBASE-T low-power 28-nm parts

9/18/2012 21

Intel’s Tick-Tock Model Sets the Clock for Data Center Upgrades

Tick= Architecture upgrade ; Tock= Silicon Shrink

9/18/2012 22

Data center spending delayed by 8 months

Microprocessors

9/18/2012 23

2-4 Cores 10-16 Cores 40-50 Cores

2 MPUs/Server 128GBytes DRAM PCIe Gen2 (8 lanes @5G/lane

? MPUs/Server 1T+ Bytes DRAM PCIe Gen3 (8G/lane) ARM, INTEL, AMD

4 MPUs/Server 768G Bytes DRAM PCIe Gen3 (8/16 lanes @8G/lane

? Data center in a server

I/O for Interconnects Virtualization software • Sends jobs to non busy MPUs

– Increases utilization from 20% to 90%

• I/O increases: Data Direct I/0 - transfer of data directly to from the network interface and cache, bypassing main memory

PCI Express • PCI Express Gen 2 5G 8b/10b and 8 lanes

• PCI Express Gen 3 8G 128/132 and 8-16 lanes

9/18/2012 24

PCIe Bus

20% 90%

Bottom line: ~200-%300 % faster I/O bandwidth

Servers per Rack

9/18/2012 25

48 - Severs / Rack 60-80 Severs / Rack

Traditional Data center Computer & Switch History Museum

9/18/2012 26

Modern Data center

9/18/2012 27

bestwebhostinggeek.com

keyitec.com

9/18/2012 28

.primecaretech.com

Mega Data Center

SuperComputer Site (HPC)

9/18/2012 29

18M Watts of power

Apple South Carolina Data Center

9/18/2012 30

Interconnect Technologies

9/18/2012 31

“Waiting for 10GBASE-T • Will be Reeaaaaly big!

• “When it gets small ….….. it will be big”

• Lan On Motherboard (LOM) is key to market

- PHY chip needs 28-nm (now at 40-nm)

- Late 2013/2014

- 10GBASE-T likely the last LOM implementation

• VCs spent $850M on 10+ startups

• Vendor community still shrinking

– Broadcom, Marvel are the 2 main players left

– Also Applied Micro, Aquantia in game

– PLX/Teranetics

• Sold Phy division to Aquantia

– Aquantia: Last remaining VC funded startup

9/18/2012 32

130-nm - This will be the year! 90-nm - This will be the year ! 65-nm - This will be the year ! 40-nm - This will be the year! 28-nm – This will be the year!

Main advantage: No huge markup like with Optical transceivers Will be big with LOM 2014

IEEE Update 40GBASE-T

• New CFI at the IEEE (July meeting) – everything is indeterminate at this point

• No vendor is anywhere close to payback 10GBASE-T investments so real support is likely to be slow – 2017-2018 event

• Designed for 2016 when Romley+ moves the server to 40G uplinks

• 85-90% use will be Server uplinks to top-of-rack switch

• 4 twisted pairs

• Encoding(?): 10G using differential signaling across pairs

• Scheme: 4x10G configuration

• Reach(?): 50 m (Not likely 100m anymore)

• Connector(?): Likely new RJ-4x; Likely not field terminated

• Cable(?): Cat8? Cat7a

– In any case – it’s a NEW cable! Meaning the “existing installed cable is not usable”

• Biggest advantage is = no optical price mark ups by suppliers

• Not likely to be supported by any startup so its Broadcom/Marvel game

Direct Attach copper • Has become very popular <5-7meters • Filling the 10GBASE-T void • Low-cost, low-power, Plug & Play • Cheaper than optical SFP+ • Active & Passive DAC • Outselling 10GBase-T 4:1

• Numerous problems surfacing

– Electrically linking different vendors equipment together directly – different than linking optically

– “Limiting” and “non-limiting” ports – no warning labels

• Net: “DAC in the Rack” • ---until 10GBASE-T LOM shows up

9/18/2012 34

10G = 7 meters 14G = 3meters 28G = ????

100G SR & LR4

• So called 2 Km solution really 4-6 dB issue

• IEEE is still working the technical issues on Reach 400m-1,200m

• 25G VCSELS maybe reach 25 meters!

– 125 meters with FEC chips added (more cost, more power)

• 50G maybe 12 meters without FEC

9/18/2012 35

Why 100G Needed in Data Centers 100G Driving Forces

• 48 – 80 Servers/ rack

• 10Gbps server uplinks

• 480G -800G problem!

• Will demand 100G uplinks

• Timing: Romley Q2/2012 : Volume 2H/2012

• ~ 2H/ 2013 need for 100G transceivers climb rapidly

• Then it pops to 40G server uplinks in 2015/16!

9/18/2012 36

9/18/2012 37

9/18/2012 38

.primecaretech.com

Mega Data Center

Link this rack

To this rack

400-600metes Not 2Km! 4-6dB link budget

Active Optical Cables

9/18/2012 39

What is an Active Optical Cable? • Parallel optic transceiver in a standard MSA shell but… integrates

the fibers directly into the MSA

• No optical connectors

• Changes the game!

9/18/2012 40

Finisar

Active Optical Cables

• Very popular in High Performance computers (HPCs) TOP25

• Technology is proven out and is rippling down to Top 25-100 & High-end data centers

• Used to connect top-of-rack switches to end-of-row switches (4x10G QSFP)

• Large router companies are starting to use 12x10G CXP

• Advantages: – Plug & Play; green light;

– No optical connector cleaning, familiar copper connector;

– low power (700mW per end)

• InfiniBand today but moving to Ethernet, SAS, PCI Express

9/18/2012 41

1 Core AOC Technology Creates Multiple Products

• Multiple types – 1x10G SFP+ - 4x5G CX-4

– 4x10G QSFP - 4x14G QSFP

– 12x10G CXP - 12x12.5G CXP

– 8 x 8G iPass

• Breakout AOCs: 4-SFP/QSFP and 3-QSFP/CXP

• Embedded Optical Module (EOM) replacing SNAP12

– For inside chassis applications (no MSAs yet)

• Parallel Optical Transceivers

• AOC/Transceiver: AOC with connectorized fibers

• Multiple applications: HPC to consumer

9/18/2012 42

GigaLight

AlpenIO Photonera

9/18/2012 43

CES AOCs

LED lights To USB and HDMI cables To Supercomputers AOCs

Consumer Embedded AOCs 4x12.5G

9/18/2012 44

Embedded AOCs 4x12.5G

9/18/2012 45

Parallel

9/18/2012 46

Embedded Optical Modules The End of Pluggable Optics?

Embedded Optical Modules “Avago MicroPOD-Like Thingies”

• Inside chassis applications

• Mid-board optics

• Start out at 10G but the big market will be at 25G

• Start out in large routers; later move to servers

9/18/2012 47

Finisar

Three Generations of EOMs

9/18/2012 48

9/18/2012 49

16 MicroPods in 4x4 array

Try cleaning this optical mess!

9/18/2012 50

Its all about achieving high density interconnects

IBM HPCs & Avago MicroPODs

9/18/2012 51

9/18/2012 52

9/18/2012 53

9/18/2012 54

Massively complex mechanical assembly!

8-10 Companies Showing Prototypes 2-3 With Announced products

9/18/2012 55

Where Are EOMs Used

9/18/2012 56

Demo board from US Conec and Avago

Sony AOC/EOM CXN2006, CXN2007 and CXN2300 optical interconnection modules

• Designed for 4K x2K video for HDTVs and digital cameras

• 4x10G based on VCSELs with a lens-less internal design

• 10mm x 16mm

9/18/2012 57

http://www.sony.net/Products/SC- P/cx_news/vol68/pdf/featuring_vol68_interconnection.pdf

9/18/2012 58

Fujitsu

SNAP12 vs MiniPOD

9/18/2012 59

Clearly EOMs have an advantage over SNAP12s

NEC 12x25G EOM at OFC2011

9/18/2012 60

Where Are EOMs Used?

9/18/2012 61

9/18/2012 62

Where do you put an EOM?

9/18/2012 63

Avago EOMs

9/18/2012 64

Offers extremely high density Optical interconnects to board edge

125mW/ch @ 10G 12 channels simplex = 1.45W 8.8 W Duplex link would use 12 EOMs = 17W

Optically Enabled Hybrid Chip Packages

• FTTChip

– Mez cards

– Chip/Optical

9/18/2012 65

Reflex/Fulcrum

Embedded Optical Modules The End of Pluggable Optics?

• Answer=NO! But it is one more opportunity. – No standard electrical or optical connectors? – No interoperability requirements? – No standard configurations 4x, 12 channel? – Every supplier has unique products (no second source)

• The industry will be able to achieve enough density in the data center with pluggable optics for at least next 4-5 years – 8 CFP/2s running 100G = 800G – 36 QSFP28 running 100G = 3,600G

1. Significant concerns about mixing 70C optics in 100C ASICs 2. Biggest limiter is repairs on failure

– Pluggable = changing fan belt on car – EOM = changing the piston

3. Rigorous vendor qualification and selection 3. Few major players

9/18/2012 66

Biggest inhibitor of market adoption

Conclusion

• Data center are in tremendous change

– Impact or clone Servers/switches

– M&As re-writing the landscape

• Low cost AOCs and Embedded Optical Modules

• Transition to 25G signaling for SR and LR4

• 25G is likely to “thin the heard” in transceivers

9/18/2012 67

Thank You

9/18/2012 68

Our Website: www.lightcounting.com

Headquarters Location:

858 West Park Street Eugene, OR 97401 Tel: 408.962.4851 Email: info@lightCounting.com

Brad@LightCounting.com

next wave of data centers & interconnects

Documents

optical interconnects: trend and applications

ace™ automated circuit extraction - armms · ace™...

lect09b interconnects topology routing

amphenol other board level and rectangular interconnects ·...

vlsi interconnects

scaling silicon nanophotonic interconnects : silicon

data centers & storage area networks area network (san)...

power interconnects

5 interconnects - pub.ro

ultra micro interconnects - samtec

modelling interconnects

a survey on optical interconnects for data centers

a survey on optical interconnects for data centers speaker:...

optical networking for data center interconnects across...

optical-interconnects-for-data-centers-chapter14 · 2018....

scalable reconfigurable interconnects

basic interconnects

data center interconnects - lightcounting center...

400g optical transceivers: market shares, strategies,...

optical interconnects