cc-4009, "optimizing hadoop deployments with seamicro sm15000" by satheesh nanniyur and...

OPTIMIZED HADOOP DEPLOYMENTS WITH SEAMICRO SM15000

PRESENTED AT AMD DEVELOPER SUMMIT, NOV 2013 SATHEESH NANNIYUR

| RIGHT SIZING HADOOP DEPLOYMENTS 2

Devices Apps Cloud

BIG DATA IS A STRATEGIC DECISION: CAPEX AND OPEX


HIGHER PERFORMANCE, LESS POWER AND SPACE

Data Storage

Data Processing

Data Access

Management

Data Warehouse Data AnalyRcs

Hadoop Technology Stack


SEAMICRO SM15000™ ACCELERATES APACHE™ HADOOP™ DEPLOYMENTS

!  Superior high availability ‒ AcRve/standby NameNode ‒ AcRve/standby JobTracker ‒ Highly resilient fabric for inter-‐node east-‐west traffic

!  Reduced down Rme ‒ Remap or rezone disks to recover data

‒ Hot-‐swappable upgrades or component replacements

!  Hardware redundancy ‒ Power supplies ‒ Network I/O


SM15000 OVERVIEW

64 Industry standard x86 servers •  AMD Opteron™, Intel Xeon®, Atom™ •  Energy efficient processor •  20 Gbps per socket, 16X tradiRonal servers

160 Gbps Network I/O •  Share network I/O across all servers •  Eliminate TOR switch •  Minimize cabling •  In service upgrades as needed

64 HDDs/SDDs •  Share drives across all servers •  Assign one server to one or more drives as needed •  In service upgrades as needed

960 terabytes Fabric Storage •  Extends supercompute fabric to external storage •  Up to 3.84 PB storage capacity; up to 960 3.5” SAS/

SATA drives •  Map to any CPU—same as internal drives


SEAMICRO FREEDOM™ FABRIC ASIC PROVIDES MASSIVE PERFORMANCE, REDUCES POWER AND SPACE

Freedom™

Provides massive bandwidth while eliminating power hungry

top of rack switches

Reduces the power used by any CPU by consolidating and

shutting off unused functionality

Eliminates 90% of the components on a motherboard shrinking power used, cost and space

Freedom Supercompute Fabric

SeaMicro IOVT

TIO

B E N E F I T S


FS 4060-‐L FABRIC STORAGE ENCLOSURE WITH ZONING CAPABILITY

!  High density, power opRmized 4U enclosure with 60 3.5” drives

!  Up to 16 enclosures per SM15000, 960 drives, and 3.84 PB storage capacity

!  Redundant controllers, ports, fans, and PSUs !  Support cost opRmized 24x7 operaRons SATA HDD for high

density Big Data and Object Storage deployments

!  OpRonal configuraRon to logically parRRon an enclosure into two 30 3.5” drive enclosures

!  Balanced disk to core raRo (1:1) for opRmizing Hadoop performance

!  Field configurable to provide utmost flexibility to balance density and performance


FREEDOM FABRIC DISAGGREGATES SERVER RESOURCES

!  SM15000 provides independent scaling of Compute, Storage, and Network

!  Centrally managed provisioning of storage and network resources to compute nodes enabled by CLI and API interfaces

PROVIDES FLEXIBILITY FOR EXPANSION AND INFRASTRUCTURE OPTIMIZATION

SeaMicro SM15000 Server

Fabric Interconnect

CPU CPU

CPU CPU

CPU CPU

Compute and Memory Pool

Shared Storage Pool

Network Pool

Deploy

Run and Analyze

Tune and OpWmize

Cost and Performance

Hadoop OpRmizaRon


SM15000 FLEXIBLE STORAGE ALLOWS ITERATIVELY OPTIMIZING APACHE® HADOOP™ DEPLOYMENT

!  Flexible shared storage with commodity hardware enabled by SeaMicro fabric technology

!  Decoupled from Compute and Network to grow storage independently

!  IteraRvely opRmize Hadoop disk to core raRo as applicaRon needs evolve

Traditional Rackmount

Freedom Fabric

Captive DAS with Rigid Storage to Compute Ratio

Flexible scale-out Fabric Storage up to 5PB

Intel /AMD X86 servers


APACHE® HADOOP™ DEPLOYMENT ON THE SEAMICRO SM15000™

HDFS NameNode

HDFS NameNode

MapReduce JobTracker

MapReduce JobTracker

DataNode/ TaskTracker








ZooK

eepe

r ZooK

eepe

r

60 DataNode/ TaskTracker Nodes

Redundant NameNode and JobTracker

10Gb/s network bandwidth/node

Up to 160 Gb/s bandwidth for data

ingesWon •  8GB/s storage bandwidth •  Flexible storage capacity

SM15000


SEAMICRO REFERENCE ARCHITECTURE FOR APACHE® HADOOP™

!  Hadoop HA with NameNode and JobTracker AcRve/Standby ConfiguraRon

!  Up to 60 DataNode/TaskTracker nodes, 512 x86 cores, 960 TB raw capacity, 10 Gb/s Internode bandwidth and 160 Gb/s uplink bandwidth in 28 RU and 5.8 kW

Highly Available AcRve/Standby NameNode

Highly Available AcRve/Standby JobTracker

DataNode/TaskTracker

SM15000 Internal Drives for NameNode and JobTracker

SoluWon Components •  SeaMicro SM15000 AMD Opteron or

Intel Xeon (Ivy Bridge) CPU chassis •  32 GB memory per node •  2 10GbE Network cards •  8 Storage Controller cards •  4 FS 4060-‐L enclosures with SAS

zoning enabled •  60 4TB 3.5” SAS or SATA drives per

enclosure •  Any Hadoop distribuRon (CDH, HDP,

MapR, Apache Hadoop etc.) •  ZooKeeper for NameNode and

JobTracker HA


SM15000 HADOOP DEPLOYMENT

SoluWon Components SM15000 Intel Xeon

SM15000 AMD Opteron

Use case Performance OpRmized

Cost OpRmized

Racks <1 <1

Servers 64 64

Cores 256 512

DRAM 2TB 4 TB

Hard Drives 240/960 TB 240/960 TB

Cable Management 0 RU 0 RU

ToR SwRches None None

Downlink Network Cables None None

Data Nodes 60 60

Bandwidth per node 10 Gb/s 10 Gb/s

Uplink bandwidth Up to 160 Gb/s Up to 160 Gb/s

Storage bandwidth 8 GigaBytes/s 8 GigaBytes/s


SM15000 – INDUSTRY’S ONLY FLEXIBLE SYSTEM FOR OPTIMIZING HADOOP CLUSTERS

Map

Map

Map

Reduce

Reduce

Storage Intensive

Compute Intensive

Network Intensive

Compute Intensive

Storage Intensive

HDFS Input

Shuffle HDFS Output

Map and Intermediate Data Write

Reduce

Flexible scale-‐out storage with over 1400 spindles and 5 Petabytes of capacity

Up to 512 x86 cores with 4TB DRAM per Fabric Server in 10RU

10 Gpbs Inter-‐Node Bandwidth per server

160 Gbps shared uplink for Inter-‐Rack traffic


SM15000 HADOOP PERFORMANCE BETTER THAN COMPETITIVE OFFERINGS

!  77% less power per node !  30% less power per core !  63% more data sorted per second per Wat than Large Vendor

SeaMicro SM15000 with 64 Nodes based on Intel Xeon® (Ivy Bridge) 1265 L-‐v2 CPU, 32 GB memory, and 2 3.5” 3TB SAS drives per node

SM15000 Large Vendor

Terasort CompleRon

7 min 13 seconds

8 min 33 seconds

Nodes 62 incl. HA 18

CPU Cores 248 216

Power 5800 W 7200 W

MB/s per Wat 0.4 0.24

MB/s per CPU core 9.3 9.0

Wats/Node 94 W 400 W

Wats/Core 23 W 33 W

CompeRRve soluRon consists of dual socket 2U rack-‐mount servers with Intel Xeon E5-‐2667 2.9 GHz octal core CPUs with 64 GB memory, 16 disks, and 4 GbE network links per node

0 0.05 0.1

0.15 0.2 0.25 0.3 0.35 0.4

SM15000 Large Vendor

MB/s p

er W

a`

Terasort -‐ Sort rate per Wa`


WAYFAIR.COM: PERSONALIZED SHOPPING EXPERIENCE FOR “A ZILLION THINGS”

Applica'ons: Apache® Hadoop™, SQL server, PHP

!  Challenge ‒  Space and power constraints hindered availability of “shared

nothing” servers for development ‒  Too costly in space and power to use tradiRonal servers for the

number of servers required and accurately test applicaRon performance

!  SoluRon ‒  SeaMicro SM high density server ‒  256 Intel® Xeon® cores in 10 RU system ‒  64 servers, 1.28 Tbps SeaMicro Freedom™ Supercompute Fabric

!  Results ‒  Reduced development cycles and shortened Rme to market for

new products ‒  Increased producRvity of development engineers by providing

abundant access to “shared nothing” servers versus developing on virtualized server farms

‒  Eliminated unnecessary equipment such as top of rack switches and terminal servers; simplified network and power cabling

“The SeaMicro SM server is helping us operate at a large scale and fast pace. The key benefits are reduced operaRng costs and increased efficiency for our big data development infrastructure. It provides the highest density and flexibility while slashing energy consumpRon: 256 Intel Xeon cores, 64 hosts. It consumes 50 percent less power and doubled our compuRng capacity...” Ben Clark, Director of So@ware Engineering


EHARMONY: INCREASE COMPUTING WHILE REDUCING TOTAL COST OF OWNERSHIP

Applica'ons: Apache® Hadoop™

!  Challenge ‒ Provide cost effecRve compuRng plaxorm for Apache Hadoop

‒ Reduce costs incurred from external cloud compuRng

!  SoluRon ‒  SeaMicro SM10000-‐64 high density server ‒ 512 Intel® Atom™ cores in 10 RU system

!  Results ‒ Reduce TCO by more than 74 percent ‒  Save thousands per month spent on cloud compuRng service

‒ URlize compuRng resources 7 x 24

“We purchased SeaMicro servers and immediately reduced our operaRng expenses…The system has been in place for over two years, and we have had zero down Rme.” Cormac Twomey, Data Center Opera'ons


AMD SEAMICRO PARTNERS WITH INDUSTRY LEADERS

Partner

Hadoop OS/Hypervisor OpenStack

Discussion


Zone 1

Zone 2

FS 4060-‐L SAS ZONING

!  SAS zoning allows the logical parRRoning of FS 4060-‐L enclosure into two 30 disk enclosures in 4U ‒ Provides fully independent end to end path to all 30 drives in each zone ‒ Up to 2 S-‐cards connected to a storage enclosure

FS 4060-‐L

SM15000 S-‐card SM15000 S-‐card

FS 4060-‐L with SAS zoning


DISCLAIMER & ATTRIBUTION

The informaRon presented in this document is for informaRonal purposes only and may contain technical inaccuracies, omissions and typographical errors.

The informaRon contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, so{ware changes, BIOS flashes, firmware upgrades, or the like. AMD assumes no obligaRon to update or otherwise correct or revise this informaRon. However, AMD reserves the right to revise this informaRon and to make changes from Rme to Rme to the content hereof without obligaRon of AMD to noRfy any person of such revisions or changes.

AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION.

AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY DIRECT, INDIRECT, SPECIAL OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

ATTRIBUTION

© 2013 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, AMD Opteron, Freedom and combinaRons thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or other jurisdicRons.. Other names are for informaRonal purposes only and may be trademarks of their respecRve owners.

cc-4009, "optimizing hadoop deployments with seamicro sm15000" by satheesh nanniyur and...

Technology

rigid storage

fabric storage

power hungry

foroprmizinghadoop performance

atom energyecientprocessor

compute ratioflexible

nov2013 satheeshnanniyur

pb storagecapacity