akhilesh kumar, skylake-sp cpu architect malay … · akhilesh kumar, skylake-sp cpu architect....

72
Akhilesh Kumar, Skylake-SP CPU Architect Malay Trivedi, Lewisburg PCH Architect June 12th, 2017

Upload: truongdien

Post on 25-Aug-2018

215 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Akhilesh Kumar, Skylake-SP CPU Architect

Malay Trivedi, Lewisburg PCH Architect

June 12th, 2017

Page 2: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017 2

Notices and Disclaimers

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure.

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.

Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.

Intel, the Intel logo, Intel Optane and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the united states and other countries.

* Other names and brands may be claimed as the property of others. © 2017 Intel Corporation.

Page 3: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Agenda

• Intel® Xeon® Scalable Processor Overview

• Skylake-SP CPU Architecture

• Lewisburg PCH Architecture

3

Page 4: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Xeon® Processor Roadmap

Intel® Xeon® Processor E5Targeted at a wide variety of applications that value a balanced system with leadership performance/watt/$

18 cores

Intel® Xeon® Processor E7Targeted at mission critical applications that value a scale-up system with leadership memory capacity and advanced RAS

Grantley-EP Platform

E5 v3 E5-2600 v4

Brickland Platform

E7 v3 E7 v4

Purley Platform

Skylake

E5 v3 E5-4600 v4 (4S)

Cascade Lake

2016 2017 2018

Intel Xeon GOLD

Intel® Xeon® PLATINUM

Intel Xeon SILVER

Intel Xeon BRONZE

Converged platform with innovative Skylake-SP microarchitecture

This slide under embargo until 9:15 AM PDT July 11, 2017

4

Page 5: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Xeon® Scalable Processor Feature Overview

Skylake-SP CPU

Skylake-SP CPU

2 or 3 Intel® UPI3x16 PCIe Gen3

3x16 PCIe* Gen3

DDR42666

Lewisburg PCH

4x10GbE NIC

Intel®QAT MEIE

High Speed IO

USB3PCIe3SATA3

GPIOBMC

eSPI/LPCFirmware

FirmwareTPM

SPI10GbE

CPU VRs

OPA VRs

Mem VRs

OPA

DMI

OPA1x 100Gb OPA Fabric

1x 100Gb OPA Fabric

BMC: Baseboard Management Controller PCH: Intel® Platform Controller Hub IE: Innovation Engine

Intel® OPA: Intel® Omni-Path Architecture Intel QAT: Intel® QuickAssist Technology ME: Manageability Engine

NIC: Network Interface Controller VMD: Volume Management Device NTB: Non-Transparent Bridge

Feature Details

Socket Socket P

Scalability 2S, 4S, 8S, and >8S (with node controller support)

CPU TDP 70W – 205W

Chipset Intel® C620 Series (code name Lewisburg)

Networking Intel® Omni-Path Fabric (integrated or discrete)4x10GbE (integrated w/ chipset)100G/40G/25G discrete options

Compression and Crypto Acceleration

Intel® QuickAssist Technology to support 100Gb/s comp/decomp/crypto 100K RSA2K public key

Storage Integrated QuickData Technology, VMD, and NTBIntel® Optane™ SSD, Intel® 3D-NAND NVMe &SATA SSD

Security CPU enhancements (MBE, PPK, MPX)Manageability EngineIntel® Platform Trust TechnologyIntel® Key Protection Technology

Manageability Innovation Engine (IE)Intel® Node ManagerIntel® Datacenter Manager

This slide under embargo until 9:15 AM PDT July 11, 2017

5

Page 6: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

DMI x4**

Platform Topologies8S Configuration

SKLSKL

LBG

LBG

LBG

DMILBG

SKLSKL

SKLSKL

SKLSKL

3x16 PCIe*

4S Configurations

SKLSKL

SKLSKL

2S Configurations

SKLSKL

(4S-2UPI & 4S-3UPI shown)

(2S-2UPI & 2S-3UPI shown)

Intel®UPI

LBG 3x16 PCIe* 1x100G

Intel® OP Fabric

3x16 PCIe* 1x100G

Intel® OP Fabric

LBGLBG

LBGDMI

3x16 PCIe*

This slide under embargo until 9:15 AM PDT July 11, 2017

Intel® Xeon® Scalable Processor supports configurations ranging from 2S-2UPI to 8S

6

Page 7: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

• Skylake core microarchitecture, with data center specific enhancements

• Intel® AVX-512 with 32 DP flops per core

• Data center optimized cache hierarchy –1MB L2 per core, non-inclusive L3

• New mesh interconnect architecture

• Enhanced memory subsystem

• Modular IO with integrated devices

• New Intel® Ultra Path Interconnect (Intel® UPI)

• Intel® Speed Shift Technology

• Security & Virtualization enhancements (MBE, PPK, MPX)

• Optional Integrated Intel® Omni-Path Fabric (Intel® OPA)

Intel® Xeon® Scalable ProcessorRe-architected from the Ground Up

This slide under embargo until 9:15 AM PDT July 11, 2017

Features Intel® Xeon® Processor E5-2600 v4 Intel® Xeon® Scalable Processor

Cores Per Socket Up to 22 Up to 28

Threads Per Socket Up to 44 threads Up to 56 threads

Last-level Cache (LLC) Up to 55 MB Up to 38.5 MB (non-inclusive)

QPI/UPI Speed (GT/s) 2x QPI channels @ 9.6 GT/s Up to 3x UPI @ 10.4 GT/s

PCIe* Lanes/ Controllers/Speed(GT/s) 40 / 10 / PCIe* 3.0 (2.5, 5, 8 GT/s) 48 / 12 / PCIe 3.0 (2.5, 5, 8 GT/s)

Memory Population 4 channels of up to 3 RDIMMs, LRDIMMs, or 3DS LRDIMMs

6 channels of up to 2 RDIMMs, LRDIMMs, or 3DS LRDIMMs

Max Memory Speed Up to 2400 Up to 2666

TDP (W) 55W-145W 70W-205W

Core Core

Core Core

Core Core

Shared L3

UPI

UPI

2 or 3 UPI

6 Channels DDR4

48 Lanes PCIe* 3.0

DMI3

DDR4

DDR4

DDR4

DDR4

DDR4

DDR4

UPI

Omni-Path HFIOmni-Path

7

Page 8: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017
Page 9: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Core Microarchitecture EnhancementsThis slide under embargo until 9:15 AM PDT July 11, 2017

Broadwell uArch

Skylake uArch

Out-of-order Window 192 224

In-flight Loads + Stores 72 + 42 72 + 56

Scheduler Entries 60 97Registers –Integer + FP 168 + 168 180 + 168

Allocation Queue 56 64/thread

L1D BW (B/Cyc) –Load + Store 64 + 32 128 + 64

L2 Unified TLB 4K+2M: 1024 4K+2M: 1536 1G: 16

Load Buffer

Store Buffer

Reorder Buffer

5

6

Scheduler

Allocate/Rename/RetireIn order

OOO

INT

VEC

Port 0 Port 1

MUL

ALU

FMA

ShiftALU

LEA

Port 5

ALU

ShuffleALU

LEA

Port 6

JMP 1

ALUShift

JMP 2

ALU

ALU

DIVShift

Shift

FMA

Port 4

32KB L1 D$

Port 2

Load/STAStore Data

Port 3

Load/STA

Port 7

STA

Load Data 2

Load Data 3 Memory Control

Fill Buffers

Fill Buffers

μop Cache

32KB L1 I$ Pre decode Inst QDecodersDecodersDecodersDecoders

Branch Prediction Unitμop

Queue

Memory

Front End

1MB L2$

FMA

About 10% performance improvement per core on integer applications at same frequency

• Larger and improved branch predictor, higher throughput decoder, larger window to extract ILP• Improved scheduler and execution engine, improved throughput and latency of divide/sqrt • More load/store bandwidth, deeper load/store buffers, improved prefetcher• Data center specific enhancements: Intel® AVX-512 with 2 FMAs per core, larger 1MB MLC

9

Page 10: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Key Instruction Set Architecture EnhancementsThis slide under embargo until 9:15 AM PDT July 11, 2017

Page Protection Keys (PPK)Extends paging architecture to provide a page-granular, thread-private user-level

memory protection

Mode Based Execution (MBE)Protects against malicious kernel updates in a virtualized system

MPX (Memory Protection Extension)

Enables bounds checking on data accesses to prevent buffer

overflow attacks

Improved Time Stamp Counter Virtualization

Reduced overhead for VMs moving across CPUs with different base frequency

Intel® AVX-512 2x compute density per core for vector

operations

Cache Management InstructionsCLFLUSHOPT – Lower latency cache line

flushCLWB – Cache line writeback to memory

without invalidation

Compute Virtualization Security

Instruction Set Enhancement across Compute, Virtualization, and Security 10

Page 11: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Advanced Vector Extensions 512 (Intel® AVX-512)

• 512-bit wide vectors• 32 operand registers• 8 64b mask registers• Embedded broadcast• Embedded rounding

This slide under embargo until 9:15 AM PDT July 11, 2017

Microarchitecture Instruction Set SP FLOPs / cycle DP FLOPs / cycle

Skylake Intel® AVX-512 & FMA 64 32

Haswell / Broadwell Intel AVX2 & FMA 32 16

Sandybridge Intel AVX (256b) 16 8

Nehalem SSE (128b) 8 4

Intel AVX-512 Instruction Types

AVX-512-F AVX-512 Foundation Instructions

AVX-512-VL Vector Length Orthogonality : ability to operate on sub-512 vector sizes

AVX-512-BW 512-bit Byte/Word support

AVX-512-DQ Additional D/Q/SP/DP instructions (converts, transcendental support, etc.)

AVX-512-CD Conflict Detect : used in vectorizing loops with potential address conflicts

Powerful instruction set for data-parallel computation11

Page 12: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017 12

Skylake-SP Core Skylake-SP core builds on Skylake core with features architected for data center usage

• Intel® AVX-512 implemented with Port 0/1 fused to a single 512b execution unit

• Port 5 is extended to full 512b to add second FMA outside of Skylake core

• L1-D load and store bandwidth doubled to allow up to 2x64B load and 1x64B store

• Additional 768KB of L2 cache added outside of Skylake core

This slide under embargo until 9:15 AM PDT July 11, 2017

Skylake-SP Core: Optimized for Data center Workloads

INT

VEC

Port 0 Port 1

MUL

ALU

FMA

ShiftALU

LEA

Port 5

ALU

ShuffleALU

LEA

Port 6

JMP 1

ALUShift

JMP 2

ALU

ALU

DIVShift

Shift

FMA FMA

Extended L2 Cache

Extended AVX

Skylake Core

Page 13: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Frequency Behavior While Running Intel® AVX Code

• Cores running non-AVX, Intel® AVX2 light/heavy, and Intel® AVX-512 light/heavy code have different turbo frequency limits

• Frequency of each core is determined independently based on workload demand

This slide under embargo until 9:15 AM PDT July 11, 2017

AVX2

Non-AVX

Cores using AVX-512Cores using AVX2Cores not using AVX

Code Type All Core Frequency Limit

SSEAVX2-Light (without FP & int-mul) Non-AVX All Core Turbo

AVX2-Heavy (FP & int-mul)AVX512-Light (without FP & int-mul) AVX2 All Core Turbo

AVX512-Heavy (FP & int-mul) AVX512 All Core Turbo

Non-AVX_TurboAVX2_Turbo

Freq

uenc

y

Cores

Non-AVX_BaseAVX2_Base

AVX512_Turbo

AVX512_Base

AVX512

Mixed Workloads

Non

-AVX

AVX5

12 Non

-AVX

AVX2

AVX2

13

Page 14: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Performance and Efficiency with Intel® AVX-512

Source as of June 2017: Intel internal measurements on platform with Xeon Platinum 8180, Turbo enabled, UPI=10.4, SNC1, 6x32GB DDR4-2666 per CPU, 1 DPC. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

This slide under embargo until 9:15 AM PDT July 11, 2017

669

1178 2034

3259

760 768 791 767

3.12.8

2.52.1

0

0.5

1

1.5

2

2.5

3

3.5

0

500

1000

1500

2000

2500

3000

3500

SSE4.2 AVX AVX2 AVX512

Core

Fre

quen

cy

GFL

OPs

, Sys

tem

Pow

er

LINPACK Performance

GFLOPs Power (W) Frequency (GHz)

1.001.74

2.92

4.83

0.00

2.00

4.00

6.00

SSE4.2 AVX AVX2 AVX512Nor

mal

ized

to S

SE4.

2 G

FLO

Ps/W

att

GFLOPs / Watt

1.001.95

3.77

7.19

0.00

2.00

4.00

6.00

8.00

SSE4.2 AVX AVX2 AVX512Nor

mal

ized

to S

SE4.

2 G

FLO

Ps/G

Hz

GFLOPs / GHz

Intel® AVX-512 delivers significant performance and efficiency gains

14

Page 15: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017
Page 16: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

CBO

IDI/QPIIIDICore CoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/Q

PII IDI CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

DN

IDI/QPIIIDI UPCore Core

BoCache

BoSAD

LLC2.5MB

CBO

DNIDI

/QPII IDIU

P CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/QPIIIDICore CoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/Q

PII IDI CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

DN

IDI/QPIIIDI UPCore Core

BoCache

BoSAD

LLC2.5MB

CBO

DNIDI

/QPII IDIU

P CoreCoreBo

CacheBo

SAD

LLC2.5MB

DNUP

DN

UP

DN

UP

CBO

IDI/QPIIIDICore CoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/QPIIIDICore CoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/QPIIIDICore CoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/QPIIIDICore CoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/Q

PII IDI CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/Q

PII IDI CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/Q

PII IDI CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

IDI/Q

PII IDI CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

DN

IDI/QPIIIDI UPCore Core

BoCache

BoSAD

LLC2.5MB

CBO

DN

IDI/QPIIIDI UPCore Core

BoCache

BoSAD

LLC2.5MB

CBO

DN

IDI/QPIIIDI UPCore Core

BoCache

BoSAD

LLC2.5MB

CBO

DN

IDI/QPIIIDI UPCore Core

BoCache

BoSAD

LLC2.5MB

CBO

DNIDI

/QPII IDIU

P CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

DNIDI

/QPII IDIU

P CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

DNIDI

/QPII IDIU

P CoreCoreBo

CacheBo

SAD

LLC2.5MB

CBO

DNIDI

/QPII IDIU

P CoreCoreBo

CacheBo

SAD

LLC2.5MB

QPI Agent

QPI Link

R3QPI

QPI Link

IIO

R2PCI

PCI-EX16

IOAPIC

CB DMA

PCI-EX16

PCI-EX8

PCI-EX4 (ESI)

UBox PCU

Home AgentDDR

Mem CtlrDDR

Home AgentDDR

Mem CtlrDDR

This slide under embargo until 1:00 PM PDT June 15, 2017

16Content Under Embargo Until 1:00 PM PST June 15, 2017

Broadwell EX 24-core die Skylake-SP 28-core die

*2x UPI x20 PCIe* x16 PCIe x16DMI x 4CBDMA

On PkgPCIe x16

1x UPI x20 PCIe x16

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

CHA/SF/LLC

SKX Core

MCDDR4

DDR4

DDR4

MC DDR4

DDR4

DDR4

CHA – Caching and Home Agent ; SF – Snoop Filter; LLC – Last Level Cache ; SKX Core – Skylake Server Core ; UPI – Intel® UltraPath Interconnect

New Mesh Interconnect Architecture

Mesh Improves Scalability with Higher Bandwidth and Reduced Latencies

Page 17: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Distributed Caching and Home Agent (CHA)

17

This slide under embargo until 9:15 AM PDT July 11, 2017

• Intel® UPI caching and home agents are distributed with each LLC bank• Prior generation had a small number of QPI

home agents• Distributed CHA benefits

• Eliminates large tracker structures at memory controllers, allowing more requests in flight and processes them concurrently

• Reduces traffic on mesh by eliminating home agent to LLC interaction

• Reduces latency by launching snoops earlier and obviates need for different snoop modes

2x UPI x20 PCIe* x16 PCIe x16DMI x4CBDMA

PCIe x16

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

MCDDR 4

DDR 4

DDR 4

MC DDR 4

DDR 4

DDR 4

2x UPI x20 @ 10.4GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

x4 DMI

3x D

DR4

266

7

3x D

DR4

266

7

Distributed CHA architecture sustains higher bandwidth and lowers latency

Page 18: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Re-Architected L2 & L3 Cache HierarchyThis slide under embargo until 9:15 AM PDT July 11, 2017

Shared L32.5MB/core(inclusive)

Core

L2(256KB private)

Core

L2(256KB private)

Core

L2(256KB private)

Shared L31.375MB/core(non-inclusive)

Core

L2(1MB private)

Core

L2(1MB private)

Core

L2(1MB private)

Previous Architectures Skylake-SP Architecture

• On-chip cache balance shifted from shared-distributed (prior architectures) to private-local (Skylake architecture):• Shared-distributed shared-distributed L3 is primary cache• Private-local private L2 becomes primary cache with shared L3 used as overflow cache

• Shared L3 changed from inclusive to non-inclusive:• Inclusive (prior architectures) L3 has copies of all lines in L2• Non-inclusive (Skylake architecture) lines in L2 may not exist in L3

Skylake-SP cache hierarchy architected specifically for Data center use case 18

Page 19: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Inclusive vs Non-Inclusive L3This slide under embargo until 9:15 AM PDT July 11, 2017

1.375 MBL3

L21MB

1

23

Non-Inclusive L3(Skylake-SP architecture)

Memory

L2256kB

2.5 MBL3

1

23

Inclusive L3(prior architectures)

Memory

1. Memory reads fill directly to the L2, no longer to both the L2 and L3

2. When a L2 line needs to be removed, both modified and unmodified lines are written back

3. Data shared across cores are copied into the L3 for servicing future L2 misses

Cache hierarchy architected and optimized for data center use cases:

• Virtualized use cases get larger private L2 cache free from interference

• Multithreaded workloads can operate on larger data per thread (due to increased L2 size) and reduce uncore activity

19

Page 20: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Cache Performance

Source as of June 2017: Intel internal measurements on platform with Xeon Platinum 8180, Turbo enabled, UPI=10.4, SNC1, 6x32GB DDR4-2666 per CPU, 1 DPC, and platform with E5-2699 v4, Turbo enabled, 4x32GB DDR4-2400, RHEL 7.0. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance. Copyright © 2017 Intel Corporation.

This slide under embargo until 9:15 AM PDT July 11, 2017

Skylake-SP cache hierarchy

significantly reduces L2

misses without increasing L3

misses compared to Broadwell-EP

Low

er is

bet

ter

20

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Relative Change in L2 and L3 Misses Per Instruction for SPECint*_rate 2006 from Broadwell-EP to Skylake-SP

Relative L2 MPI Relative L3 MPI

Page 21: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Cache Performance

Source as of June 2017: Intel internal measurements on platform with Xeon Platinum 8180, Turbo enabled, UPI=10.4, SNC1, 6x32GB DDR4-2666 per CPU, 1 DPC, and platform with E5-2699 v4, Turbo enabled, 4x32GB DDR4-2400, RHEL 7.0. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance. Copyright © 2017, Intel Corporation.

This slide under embargo until 9:15 AM PDT July 11, 2017

Low

er is

bet

ter

Skylake-SP cache hierarchy

significantly reduces L2

misses without increasing L3

misses compared to Broadwell-EP

21

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Relative Change in L2 and L3 Misses Per Instruction for SPECfp*_rate 2006 from Broadwell-EP to Skylake-SP

Relative L2 MPI Relative L3 MPI

Page 22: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Cache Performance

Source as of June 2017: Intel internal measurements on platform with Xeon Platinum 8180, Turbo enabled, SNC1, 6x32GB DDR4-2666 per CPU, 1 DPC, and platform with Intel® Xeon® E5-2699 v4, Turbo enabled, without COD, 4x32GB DDR4-2400, RHEL 7.0. Cache latency measurements were done using Intel® Memory Latency Checker (MLC) tool.Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance. Copyright © 2017, Intel Corporation.

This slide under embargo until 9:15 AM PDT July 11, 2017

Low

er is

bet

ter

Skylake-SP L2 cache latency has

increased by 2 cycles for a 4x

larger L2

Skylake-SP achieves good L3

cache latency even with larger

core count

22

1.1

3.3

18

1.1

3.9

19.5

L1 C AC H E L2 C AC H E L3 C AC H E ( AVG)

LATE

NCY

(NS)

CPU CACHE LATENCYBroadwell-EP Skylake-SP

Page 23: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Memory SubsystemThis slide under embargo until 9:15 AM PDT July 11, 2017

2x UPI x20 PCIe* x16 PCIe x16DMI x4CBDMA

PCIe x16

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

MCDDR 4

DDR 4

DDR 4

MC DDR 4

DDR 4

DDR 4

2x UPI x20 @ 10.4GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

x4 DMI

3X D

DR4

-266

6

3x D

DR4

-266

6

2 Memory Controllers, 3 channels each total of 6 memory channels

• DDR4 up to 2666, 2 DIMMs per channel

• Support for RDIMM, LRDIMM, and 3DS-LRDIMM

• 1.5TB Max Memory Capacity per Socket (2 DPC with 128GB DIMMs)

• >60% increase in Memory BW per Socket compared to Intel® Xeon® processor E5 v4

Supports XPT prefetch and D2C/D2K to reduce LLC miss latencyIntroduces a new memory device failure detection and recovery scheme with Adaptive Double Device Data Correction (ADDDC)

Significant memory bandwidth and capacity improvements23

Page 24: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Sub-NUMA Cluster (SNC)

Prior generation supported Cluster-On-Die (COD)

SNC provides similar localization benefits as COD, without some of its downsides

• Only one UPI caching agent required even in 2-SNC mode

• Latency for memory accesses in remote cluster is smaller, no UPI flow

• LLC capacity is utilized more efficiently in 2-cluster mode, no duplication of lines in LLC

This slide under embargo until 9:15 AM PDT July 11, 2017

24

2x UPI x 20 PCIe* x16 PCIe x16DMI x 4CBDMA

On PkgPCIe x16

1x UPI x 20 PCIe x16

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

MCDDR4

DDR4

DDR4

MC DDR4

DDR4

DDR4

3xD

DR4

266

7

3xD

DR4

266

7

SNC Domain 0 SNC Domain 1

Page 25: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Memory PerformanceBandwidth-Latency Profile

Source as of June 2017: Intel internal measurements on platform with Xeon Platinum 8180, Turbo enabled, UPI=10.4, SNC1/SNC2, 6x32GB DDR4-2400/2666 per CPU, 1 DPC, and platform with E5-2699 v4, Turbo enabled, 4x32GB DDR4-2400, RHEL 7.0. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmarkand MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance

25

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 26: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

60

80

100

120

140

160

180

200

220

Intel® Xeon® E5-2699 v4, DDR4-2400, Dir+OSB

Intel® Xeon® E5-2699 v4, DDR4-2400, Home Snp

Intel® Xeon® E5-2699 v4, DDR4-2400, COD

Intel® Xeon® Platinum 8180, DDR4-2666

Intel® Xeon® Platinum 8180, DDR4-2666, SNC2

LATE

NC

Y (N

S)

NUMA - Local NUMA - Min Remote NUMA - Max Remote UMA - Min UMA - Max

Memory PerformanceCore to Memory Latency

Source as of June 2017: Intel internal measurements on platform with Xeon Platinum 8180, Turbo enabled, UPI=10.4, 6x32GB DDR4-2666, 1 DPC, and platform with E5-2699 v4, Turbo enabled, 4x32GB DDR4-2400, RHEL 7.0. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors.Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance.

This slide under embargo until 9:15 AM PDT July 11, 2017

26

Low

er is

bet

ter

Page 27: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Ultra Path Interconnect (Intel® UPI)This slide under embargo until 9:15 AM PDT July 11, 2017

• Intel® Ultra Path Interconnect (Intel® UPI), replacing Intel® QPI• Faster link with improved bandwidth for a balanced system design

• Improved messaging efficiency per packet• 3 UPI option for 2 socket – additional inter-socket bandwidth for non-NUMA optimized use-cases

Intel® UPI enables system scalability with higher inter-socket bandwidth

27

75%50%

L0 L0pQPI

L0pUPI

Idle PowerData Rate

9.6 GT/s

10.4 GT/s

QPI UPI

Data Efficiency

4% to 21%

(per wire)

Source as of June 2017: Intel internal measurements on platform with Xeon Platinum 8180, Turbo enabled, UPI=10.4, 6x32GB DDR4-2666, 1 DPC, and platform with E5-2699 v4, Turbo enabled, 4x32GB DDR4-2400, RHEL 7.0. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors.Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance.

Page 28: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Processor Integrated I/O

3 independent pipelines of x16 PCIe* Gen3• Each x16 can be bifurcated into 2x8, 1x8+2x4, or 4x4 root ports

• New traffic controller pipeline improves over prior design

• Additional x16 PCIe for Intel® Omni-Path integration

Non-Transparent Bridging (NTB)• One NTB per x16 PCIe, which can be configured as 1x8 or 1x4

NTB

Intel® QuickData Technology (CBDMA)• 2x bandwidth on Mem-Mem copy

• Supports MMIO-Mem copy

Intel® Volume Management Device (VMD)• One VMD domain per x16 PCIe

This slide under embargo until 9:15 AM PDT July 11, 2017

Mesh Stop Mesh Stop Mesh Stop

Write Cache

Traffic Controller

NT

B

VTd

PCIe

x16

Write Cache

Traffic Controller

NT

B

VTd

PCIe

x16

Write Cache

Traffic Controller

NT

B

VTd

PCIe

x16

Write Cache

Traffic Controller

VTd

CBDMADMI

x4

Modular IO design with improved feature set for converged data center28

Page 29: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

>50% aggregate IO bandwidth

improvement in line with memory

bandwidth increase for a

balanced system performance

IO Performance

Source as of June 2017: Intel internal measurements on platform with Xeon Platinum 8180, Turbo enabled, UPI=10.4, SNC1, 6x32GB DDR4-2400/2666 per CPU, 1 DPC, and platform with E5-2699 v4, Turbo enabled, 4x32GB DDR4-2400, RHEL 7.0. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance.

29

This slide under embargo until 9:15 AM PDT July 11, 2017

1.63

1.59

2.12

1.51

2.00

M E M O R Y B W ( R E A D ) A G G . L O C A L P C I E R D W R ( L A R G E P K T )

D M A M E M - M E M D M A M E M - I O D M I

RELA

TIVE

IO B

AND

WID

TH

IO BANDWIDTH CHANGE OVER XEON E5-2699V4

Page 30: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017
Page 31: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Volume Management Device (Intel® VMD)This slide under embargo until 9:15 AM PDT July 11, 2017

Intel® VMD is a CPU-integrated device to aggregate NVMe SSDs into a storage volume and enables other storage services such as RAID • Intel® VMD is an “integrated end point” that

stops OS enumeration of devices under it• Intel® VMD maps entire PCIe* trees into its

own address space (a domain)• Intel® VMD driver sets up and manages the

domain (enumerate, event/error handling), but out of fast IO path

Eliminates additional components to provide a full-feature storage solution 31

IntelVMD

IntelVMD

IntelVMD

Page 32: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Energy Efficiency and Power Management Enhancements

Energy EfficiencyOptimized intermediate core turbo profile

Dynamic power sharing between core, uncore, and fabric HFI

Larger L2 cache for reduced interconnect and coherency activity

Power delivery through integrated VR for core and uncore

Power ManagementIntel® Speed Shift Technology for

autonomous P-state control Improved core and uncore frequency

scaling heuristicsOn-die Pmax detector for rapid response

to power excursionsIndependent per core and CLM (CHA, LLC, and mesh) voltage and frequency domains

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 33: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Optimized Turbo Profiles

*Picture is an illustration only. Not intended to represent any specific SKU or imply any frequency commitments.

This slide under embargo until 9:15 AM PDT July 11, 2017

Prior generation data center CPUs typically decreased turbo by 1 bin for each additional active coreSkylake-SP provides higher intermediate turbo points by stepping down in a more optimal manner• Higher performance dynamically with C-

states• BIOS/OS core disable can be used to

mimic higher frequency SKUs (with some tradeoffs)

Note: there is no guarantee that these frequencies can be achieved for a given workload on all units

P0

P0n

P1Prior Generation CPUs

Skylake-SP

Max

Tur

bo F

requ

ency

Number of Active (C0/C1) Cores

Opportunity for large frequency increase at intermediate active core counts

33

Page 34: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Speed Shift Technology Interface

DVFS – Intel SpeedStep® TechnologyP~V2∙f∙Cdynn+leakage(V) ~ f3

Legacy: OS controls P-state

P1-Pn enumerated via ACPI tables

Explicit P-state selection to P1

Autonomous control in turbo range

New: guided autonomous control

OS provides min, max and preference

Demand based HW control

This slide under embargo until 9:15 AM PDT July 11, 2017

P0 – 1 core

P0 – 2 cores

P1

P2

Pn

T-states

Guaranteed frequency

Turbofrequency

Energy efficientFrequency (min V)

Thermal control

OS control

P0 – 1 core

T-states

LFM

Highest frequency

Lowestfrequency

Maximumfrequency

Minimumfrequency

Optional:Desired

frequency

Cont

rol

Legacy Interface HWP i/F

34

Page 35: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Xeon® Scalable Processor: Intel® Run Sure Technology Features

This slide under embargo until 9:15 AM PDT July 11, 2017

Continued improvement in data center uptime with Intel® Run Sure Technology

Resilient System Technologies - integrate processor, firmware, and software layers that allow the system to diagnose and/or recover from previously fatal errorsResilient Memory Technologies - ensure data integrity & enable systems to keep running reliably over a longer period of time, reducing the frequency of service calls

Intel® Run Sure Technology

Resilient System Technologies Advanced Error Detection and Correction

(AEDC) MCA 2.0 Recovery (as per eMCA gen2

architecture) MCA Recovery-Execution Path MCA Recovery-Non Execution Path Local Machine Check (LMCE) based recovery

Resilient Memory Technologies SDDC + 1, Adaptive DDDC (MR) +1 Addressed Range/Partial Memory

Mirroring

35

Page 36: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Xeon® Scalable Processor: New Virtualization Enhancements

This slide under embargo until 9:15 AM PDT July 11, 2017

Greater consolidation on a Compute node

Improved core performance and larger number of cores

Larger TLBs and per core L2 cache for improved performance and lower

variability on virtualized workloadsImproved memory bandwidth and capacity

to collocate demanding workloads

Secure Creation and movement of VM across systems

Mode Based Execution Control for hardening VM launch vulnerability

Improved timestamp virtualization to reduce overhead of migrating VMs across

different CPU skus

36

Page 37: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Xeon® Scalable Processor: New Security Enhancements

This slide under embargo until 9:15 AM PDT July 11, 2017

Security Performance Intel® AVX-512

Intel® QuickAssist Technology

Hardening against attack surfaces

Mode Based Execution to protect from attacks during VM

creationPage Protection Keys to create

robust applications with differentiated page access

rights managed at user levelIntel® Memory Protection Extension (Intel® MPX) to

prevent buffer overflow attacks

Hardening The Platform

Secure key management within the chipset without a discrete

TPMAuthenticated and measured launch and recovery options

37

Page 38: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

F

CON

NEC

TOR QSFP

module

38

Skylake-SP with Integrated Fabric

Single on-package Omni-Path Host Fabric Interface (HFI)

Fabric component interfaces to CPU using x16 PCIe* lanes

Fabric PCIe lanes are additional to the 48 PCIe lanes on the socket

Single cable from SKL-F package connector to QSFP module

Same socket for Skylake-SP and Skylake-F processors

• Purley platform can be designed to support both processors

• Platform design requires an expanded keep-out zone and additional board components to accommodate both processors

QSFP connector

Skylake-F

Internal Faceplate-to-Processor (IPF) Cable

CON

NEC

TOR

Internal Faceplate Transition (IFT) Connector

and Cage

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 39: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017
Page 40: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Skylake-SP Architecture SummaryNew Architectural Innovations for Data Center

• Up to 60% increase in compute density with Intel® AVX-512

• Improved performance and scalability with Mesh on-chip interconnect

• L2 and L3 cache hierarchy optimized for data center workloads

• Improved memory subsystem with up to 60% higher memory bandwidth

• Faster and more efficient Intel® UPI interconnect for improved scalability

• Improved integrated IO with up to 50% higher aggregate IO bandwidth

• Increased protection against kernel tampering and user data corruption

• Core, cache, memory and IO improvements for increased virtual machine performance

• Enhanced power management and RAS capability for improved utilization of resources

This slide under embargo until 9:15 AM PDT July 11, 2017

40

Page 41: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Skylake-SP Performance

Source as of June 2017: Intel internal measurements with Xeon Platinum 8180 and 8176, Turbo enabled, UPI=10.4, SNC1, 6x32GB DDR4-2666 per CPU, 1 DPC, and platform with E5-2699 v4, Turbo enabled, 4x32GB DDR4-2400, RHEL 7.0. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance.

This slide under embargo until 9:15 AM PDT July 11, 2017

Skylake-SP CPUs provide significant

performance upside compared to prior

generation165W Skylake-SP CPUs provide

more than 40% gain on performance

205W Skylake-SP CPUs provide additional boost to core bound

workloads

41

1.41

1.53 1.

64

1.98

1.53 1.

64

1.64

2.25

S P E C I N T * _ R A T E _ B A S E 2 0 0 6 S P E C F P * _ R A T E _ B A S E 2 0 0 6 S T R E A M - T R I A D L I N P A C K

RELA

TIVE

PER

FORM

ANCE

2 SOCKET SKYLAKE-SP PERFORMANCE OVER INTEL® XEON® E5-2699 V4

Xeon Platinum 8176 (28C, 165W) Xeon Platinum 8180 (28C, 205W)

Page 42: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017 42

Skylake-SP Die Configurations

XCC Die with 28 Cores

• 6x6 Mesh topology

• 5 rows of core and LLC

• 2 memory controllers, one on each side of die

• All IOs at the top

• 3 x16 PCIe Gen3 stacks

• 1 x16 PCIe for MCP use

• Up to 3 Intel® UPI ports

This slide under embargo until 9:15 AM PDT July 11, 2017

2x UPI x20 PCIe* x16 PCIe x16DMI x 4CBDMA

On PkgPCIe x16

1x UPI x20 PCIe x16

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

MCDDR4

DDR4

DDR4

MC DDR4

DDR4

DDR4

CHA – Caching and Home Agent ; SF – Snoop Filter; LLC – Last Level Cache ; Core – Skylake-SP Core; UPI – Intel® UltraPath Interconnect

2x UPI x20 @ 10.4GT/s

1x UPI x20 @ 10.4GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

x4 DMI

3x D

DR4

266

6

3x D

DR4

266

6

Page 43: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

High and Low Core Count Die Configurations

43

This slide under embargo until 9:15 AM PDT July 11, 2017

HCC (up to 18 cores) LCC (up to 10 Cores)

2x UPI x 20 PCIe* x 16 PCIe x 16DMI x 4CBDMA

PCIe x 16

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

CHA/SF/LLC

Core

MCDDR4

DDR4

DDR4

MC DDR4

DDR4

DDR4

CHA – Caching and Home Agent ; SF – Snoop Filter ; LLC – Last Level Cache ; Core – Skylake -SP Core ; UPI – Intel® UltraPath Interconnect

2x UPI x 20 @ 10.4GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

1x16/2x8/4x4 PCIe @ 8GT/s

x4 DMI

3x D

DR4

266

7

3x D

DR4

266

7

2x UPI x 20 PCIe x 16 PCIe x 16DMI x 4CBDMA

PCIe x 16

CHA /SF/LLC

Core

CHA /SF/LLC

Core

CHA /SF/LLC

Core

CHA /SF/LLC

Core

CHA /SF/LLC

Core

CHA /SF/LLC

Core

CHA /SF/LLC

Core

CHA /SF/LLC

Core

CHA /SF/LLC

Core

CHA /SF/LLC

Core

MCDDR4

DDR4

DDR4

MC DDR4

DDR4

DDR4

CHA – Caching and Home Agent ; SF – Snoop Filter ; LLC – Last Level Cache ; Core – Skylake -SP Core ; UPI – Intel® UltraPath Interconnect

2x UPI x 20 @ 10.4GT /s

1x16/2x8/4x4 PCIe @ 8GT /s

1x16/2x8/4x4 PCIe @ 8GT /s

1x16/2x8/4x4 PCIe @ 8GT /s

x4 DMI

3x D

DR4

266

7

3x D

DR4

266

7

Page 44: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017
Page 45: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Server PCHPlatform Control

High Speed IO

Lewisburg: New PCH for the Converged PlatformUbiquity of network security, efficient data storage and packet manipulation in Cloud, Storage, Enterprise and Network appliances.

Lewisburg is a Common Platform Offering in Purley generation to meet the converged requirement

Data Center PCH provides boot, standard legacy and high-speed IO, manageability and clocking solutions

• Intel® Innovation Engine is a manageability sandbox for system builder

Integrated Intel® Ethernet Controller reduces area, power, cost and provides platform LOM capabilities

• Intel® Ethernet Connection X722 with up to 4x10Gbps

• Supports Network Virtualization Offloads and iWARP RDMA

Intel® QuickAssist Technology Accelerators provide security and compression for Communications, Storage, and Cloud deployments

• Crypto / compression up to 100Gbps

Public key exchange up to 100K Ops 45

PKE Public Key EncryptionKPT Key Protection TechnologyRDMA Remote Direct Memory Access

This slide under embargo until 9:15 AM PDT July 11, 2017

Intel® QAT

Mux

DMI I/FPCIe SP PCIe SP

PCH10GbE

PCIe* x16 PCIe x8 DMI x4

Server PCHPlatform Control

High Speed IOInnovation Engine

AcceleratorsBulk Crypto

CompressionPKEKPT

Integrated Ethernet4x10GbE

RDMA

Page 46: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

PCH Generational ComparisonFeatures Intel® C610 series chipset

(Wellsburg)Intel® C620 series chipset

(Lewisburg)

Standard FeaturesIntel® vPro™ technology, AMT, Node Manager 3.0;

Server Trace Lengths; 6 SMBusDiscrete, Integrated and Hybrid Clocking

Intel® vPro™, AMT, Node Manager 4.0;Server Trace Lengths; 6 SMBus

Discrete, Integrated and Hybrid Clocking

SATA Ports Up to 10 SATA 3 (6 Gb/s) Up to 14 SATA 3 (6 Gb/s)

USB Ports Up to 14 USB 2.0; Up to 6 USB 3.0 Up to 14 USB 2.0; Up to 10 USB 3.0

DMI x4, 2.0 speed x4, 3.0 speed

Additional CPU Uplink Options N/A PCIe* 3.0 at x8 and/or x16;

PCI Express* Up to 8 PCIe 2.0 (5 GT/s) Up to 20 ports PCIe 3.0 (8 GT/s)

TPM Support TPM 1.2 TPM 2.0

SATA Express, NVM Express No No / Supported

Intel® RSTe Yes Yes

Innovation Engine No Yes

LAN Integrated 1GbE Integrated Intel® Ethernet Connection X722 with up to 4x10Gb/1Gb ports with iWARP RDMA/ SFI

Intel® QuickAssist Technology - Crypto N/A Up to 100 Gb/s IPSec/SSL

Intel QuickAssist Technology - Public Key N/A Up to RSA2K 100K Ops

Intel QuickAssist Technology – Compression N/A Up to 100Gb/s (Deflate(LZ77))

Enhanced Serial Peripheral Interface (eSPI) N/A up to 60MHz, 1.8V

Green text is new

This slide under embargo until 9:15 AM PDT July 11, 2017

Integrated 10Gb Intel® Ethernet, enhanced I/O and platform security with robust crypto and compression solutions

46

Page 47: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Lewisburg Deployment Configurations

47

This slide under embargo until 9:15 AM PDT July 11, 2017

SKLSKL

SKLSKL

LBGLBG

LBGDMI

Multi-PCHMultiple On-board PCH for failover, flexible partition or IO expansion Optional PCIe uplink for QAT and 4x10GbE

ConventionalOn-board PCH. DualPCIe uplink to single CPU, provisioned for Intel® QAT and 4x10GbE (PCIe Uplink optional)

CPU StraddlingOn-board PCHPCIe uplink to multiple CPU, provisioned for QAT and 4x10GbE

PCIe* Endpoint ModeOn-board or PCIe-card-based acceleration for scalability, provisioned for QAT and 4x10GbE

Wrap-AroundOn-board PCHConfigure low-BW QAT/Ethernet as PCH-attached downstream device

SKX

PCIe

G3x

16

PCIe

G3x

8

DM

I G3x

4

QAT

PCH

4x10G

SKX

PCIe

G3x

8

PCIe

G3x

16

SKX

DM

I G3x

4

QAT

PCH

4x10G

SKX

PCIe

G3x

16

PCIe

G3x

8

QAT

4x10G

SKX

PCIe

G3x

4

DM

I G3x

4

PCH

4x10G

QAT

Page 48: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Secure Key Establishment

Lossless compression for data in flight & rest

Security for data in flight & restBulk Crypto

Intel® QuickAssist TechnologyGeneration 2

48

Public Key Encryption

Compression

This slide under embargo until 9:15 AM PDT July 11, 2017

Intel® QAT

Mux

DMI I/FPCIe* SP PCIe SP

PCH10GbE

Intel® QuickAssist Technology is designed to optimize the use & deployment of crypto and compression hardware accelerators on Intel® Platforms

Gen 1 Gen 2

150Gbs AES256 + HMAC SHA256

300%

100kopsRSA 2048 Decrypt

250%

100+GbsDeflate Compression

400%

Intel®Key

Protection Technology

NewPlatform

Capability

Page 49: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Secure Key Establishment

Lossless compression for data in flight & rest

Security for data in flight & restBulk Crypto

Intel® QuickAssist TechnologyGeneration 2

49

Public Key Encryption

Compression

This slide under embargo until 9:15 AM PDT July 11, 2017

Intel® QAT

Mux

DMI I/FPCIe* SP PCIe SP

PCH10GbE

Gen 1 Gen 2

150Gbs AES256 + HMAC SHA256

300%

100kopsRSA 2048 Decrypt

250%

100+GbsDeflate Compression

400%

Intel®Key

Protection Technology

NewPlatform

Capability

“Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Software and workloads used in performance tests may have beenoptimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. Youshould consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance/datacenter.Configurations: see slide 71

Intel® QuickAssist Technology is designed to optimize the use & deployment of crypto and compression hardware accelerators on Intel® Platforms

Page 50: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® QuickAssist Technology: CryptoUsage Model• Network security (IPsec, SSL/TLS), hashing for data-deduplication,

encrypted storage

Symmetric (Bulk) Cryptography• Ciphers (AES, 3DES/DES, RC4, KASUMI*, Snow 3G)• Message digest/hash (MD5, SHA1, SHA2x, SHA3) and authentication

(HMAC, AES-XCBC)• Algorithm chaining (one cipher and one hash in a single operation)• Authenticated encryption (AES-GCM, AES-CCM)• AES-XTS

Wireless • KASUMI, Snow 3G and ZUC

Asymmetric (Public Key) Cryptography• Modular exponentiation for Diffie-Hellman (DH)• RSA key generation, encryption/decryption and digital signature

generation/verification• DSA parameter generation and digital signature generation/verification• Elliptic Curve Cryptography: ECDSA, ECDHE

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Intel measurements as of May 2017; Intel® Customer Reference Board Neon City with one Intel® Xeon® Skylake-SP processor and one Lewisburg Chipset LBG-T B0 ES2* SKU (DCL: #568586). For more complete information about performance and benchmark results, visit www.intel.com/benchmarks

Performance Coleto Creek Lewisburg

Network Security Protocols

TLS @ 16k records 50 Gbs 150 Gbs1

IPSec @ 1kB 45 Gbs 100 Gbs1

Public Key Encryption

RSA Decrypt 2K 40k Ops 100k Ops

Wireless Ciphers

ZUC/Snow 3G/KASUMI* F82 20 Gbs 50Gbs

Cipher or Hash Only

AES128 CBC @ 4k 50Gbs 150Gbs

SHA1, SHA256, SHA3, MD5 @ 4k 50Gbs 140Gbs

1. Using 16k records using AES-CBC-HMAC SHA1/256, 100Gbs at 1k packet2. KASUMI-F8 (encryption) at 320B packets, 7Gbs for 64B packets

This slide under embargo until 9:15 AM PDT July 11, 2017

50

Page 51: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® QuickAssist Technology: Data Compression

Usage Model• Big data acceleration • WAN acceleration• Http compression• File System• Databases

Compression and Decompression Algorithm• DEFLATE: LZ77 compression followed by Huffman coding,

with a gzip or zlib header

Other Features• Engine can be configured to perform either compression or

decompression • Support for stateful (de)compression• Storage-specific features

Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.Intel measurements as of May 2017; Intel® Customer Reference Board Neon City with one Intel® Xeon® Skylake-SP processor and one Lewisburg Chipset LBG-T B0 ES2* SKU (DCL: #568586). For more complete information about performance and benchmark results, visit www.intel.com/benchmarks

Performance Coleto Creek Lewisburg

Compression 24 Gbs1 100+ Gbs1

Decompression 24 Gbs1 160 Gbs1

Compression + Decompression

24 Gbs1 100 Gbs1

1. Dynamic Deflate Level 1 using 64KB buffer size2. Best case compression ratio with Lewisburg is zlib level 4

(Compression L4 performance 24 Gbs) 3. Measured using Calgary Corpus

This slide under embargo until 9:15 AM PDT July 11, 2017

51

Page 52: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® QAT

Mux

DMI I/FPCIe* SP PCIe SP

PCH10GbE

52

Integrated Intel® Ethernet Connection X722Low-cost solution for up to four ports of 10Gb Ethernet• >50% of server ports will be 10GbE in 20171

• Lower power, less board space: integrated 10GbE saves > 20 cm2 in board area and > 25% in power consumption 2

• Available as integrated Ethernet on motherboard or stand-alone NIC cards

Proven “It Just Works” 10GbE solution• Based on Intel® Ethernet Converged Network Adapter XL710 (Fortville) IP • External PHYs in production today support 1GbE and 10GbE 10GBASE-T

Advanced features to enable Software Defined Infrastructure• Network Virtualization Offloads to support the move to L3 networks• iWARP RDMA for increased bandwidth at lower CPU utilization• Intel® Ethernet Flow Director traffic steering for increased efficiency• Intel® Data Plane Development Kit (DPDK) for advanced packet forwarding

1. Dell’Oro, “Worldwide Server Market Total Network Metrics” January 2017 2. Intel estimates in comparison to Grantley platforms

Dual 10G BaseT Card

Quad-Port SFP Card

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 53: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® 4x10Gb/1Gb Ethernet Controller

53

NC-SISMBus

4x10GbE/1GbEwith RDMA(iWARP)

Integrated IO

PCIe* with SR-IOV

Queue Mgmt & Scheduling

Qm

In-BandMgmt.

PF0 PF1 PFn…

...

4x10G MACs

VF0VF0VF0VF0VF0VF1

VF0VF0… VF0VF0VFm

MCTP

VEB & DCB Traffic Classifiers

Protocol Engine

Q0 Q1

This slide under embargo until 9:15 AM PDT July 11, 2017

Feature Support Customer Value

Quad Port 10GbE MAC/PHY• Based on Intel’s 10GbE solution• Interfaces: 10G (KR, SFI, XFI), 1G (KX)

• Optimized for networking capability in Cloud, Comms, and Storage

• Single network driver on Intel® platform

Remote Direct Memory Access • iWARP

• Routable and scalable RDMA ideal for large segmented networks in private and public clouds

Network Virtualization Offloads• Flexible Filters (ATR, Flow Director)• NVGRE, IPinGRE, VXLAN, MACinUDP

• Abstract the network for cloud flexibility

• Enhanced programmability and application affinity

Standards Based Virtualization • SR-IOV: 4 Physical/128 Virtual Function• VEB (Virtual Ethernet Bridge)

• Broad OS enablement

Power Management • Energy Efficient Ethernet

Manageability• BMC Pass-Through (control & network)• Interfaces: NC-SI, SMBus, and MCTP

• Common transport for LAN-to-BMC

Page 54: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Remote Direct Memory Access (RDMA)RDMA is a network performance optimization• Enables direct app-to-app communication across nodes

• Bypasses the OS stack & kernel

• Provides direct channel for remote memory application access

RDMA offers lower latency, high throughput by:• Avoiding application context switching

• Placing data directly in application buffers (zero-copy DMA)

• Moving protocol processing off the CPU

Intel® Ethernet Connection X722 is featured with iWARP (Internet Wide-area RDMA Protocol) RDMA:• Recommended for easy deployment and configuration, scalability and

congestion control

RDMA-aware applications or messaging layers are required• Windows*/Linux* Drivers supported for Storage, Messaging Middleware

and HPC application categories

Ethernet

iWARPPCIe

Sys Driver

Application

OS Stack

IO Lib

S/WH/W

BasicNetwork

ControllerFlow

TCP/IP

KernelUser

AvoidCopies

& Kernel

iWARPNetworkControllerFlow

WithoutiWARP

WithiWARP

This slide under embargo until 9:15 AM PDT July 11, 2017

54

Page 55: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Introducing the Innovation Engine (IE)

¥BMC is optional if system-builder has implemented a management app in IE that communicates directly with the network

Intel® QAT

Mux

DMI I/FPCIe* SP PCIe SP

PCH10GbE

This slide under embargo until 9:15 AM PDT July 11, 2017

Embedded core in the LBG PCH• Very small Intel® Architecture core • Similar to Management Engine (ME) hardware, with some privilege and IO

differences

Reserved for system-builder’s code, not for Intel firmware• Intel supplies IE hardware only• IE code is cryptographically bound to the system-builder• Code not authenticated by the system-builder will not load

Activation is not required for normal system operation• Optional feature • Requires SPS firmware. ME1x firmware does not support

55

LewisburgPCH

ME IE

BMC¥

SKLCPU

SKLCPU…

Page 56: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Lewisburg ME vs IE Features

Red: delta between ME and IE

LBG ME LBG IE

Processor Intel® Quark™ x86 (412 DMIPS) Intel Quark x86 (412 DMIPS)

Memory 1.7MB SRAM 1.4MB SRAM

Crypto Algorithms Yes Yes

Host Interface Yes Yes

Platform Interface Yes Yes + UART

FW Intel® AMT/vPro, SPS System Builder FW

Security TXT, BtG, PTT, KPT -

Operating States S0/M0, Sx/M3, Sx-Moff S0/I0, Sx/I3, S0/Ioff, Sx/Ioff

This slide under embargo until 9:15 AM PDT July 11, 2017

56

Page 57: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Broad IE Usage ModelsLite, BMC-less Manageability

PCH

IE ME

NIC

BMC/BIOS/ME Assist

PCH

IE ME

BMC

BIOS

SegmentScale-out Cloud & Embedded/Appliance

ValueBasic manageability without cost, space,

power of a BMCCommon manageability solution

across vendors

UsagesSimplified platform management

(IPMI, Redfish, SoftKVM) without a BMCHardware Security ModuleBasic node management

SegmentEnterprise & Scale-up Cloud

ValueImproved system manageability through

tighter platform integrationBetter performance through reduced

BMC/CPU interrupts

UsagesIn-band PECI over DMI link: higher

bandwidth, lower latency internal data paths

Offload tight–loop sensor monitoringPower sequencing & control

Error & reset handlingOOB telemetry for performance and

predictive RASPlatform NVRAM firmware security

(validate BMC/etc)Crash dump & recovery over PECI

Platform OOB key mgmt

57

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 58: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Lewisburg PCH SKU Guidance

†Four ports total: ports 0 & 1 can run up to 10 GbE, while ports 2 & 3 are limited to 1 GbEPackage Size (all SKUs): 34x28mm, Package Pin Count: 1310 Intel recommends two lanes of PCIe3 for each active 10GbE port for networking and x16 if Intel® QuickAssist Technology is activeLBG supports x16, x8, x4 and x1 options, up to the maximum uplink width.¥These Ethernet ports are in addition to the 1Gb port used by ME11.6All LBG skus support 7 year production and 10 year use in line with the Unique Extended Supply Life CPU SKUs (10-year use + NEBS-Friendly Thermal Specification)

Product Name SKU10Gb/1GbEthernet

Ports¥Compression Encryption RSA

Max PCIe*

Uplink

RecommendedMin Uplink

Config

PCIe* Uplink x8 Optional Muxed

Link

Est TDP (W)

Intel® QuickAssist Technology

Intel® C621 Chipset LBG-1G 0/4 N/A N/A N/A x1 x1 n/a ~ 15

Intel® C622Chipset LBG-2 2/4† N/A N/A N/A x8 x4 n/a ~ 17

Intel® C624Chipset LBG-4 4/4 N/A N/A N/A x16 x8 n/a ~ 19

Intel® C625Chipset LBG-E 4/4 20 Gb/s 20 Gbs 20K Ops x16 x16 n/a ~ 21

Intel® C626Chipset LBG-M 4/4 40 Gb/s 40 Gbs 40K Ops x16 x16 enabled ~ 23

Intel® C627Chipset LBG-T 4/4 100 Gb/s 100 Gbs 100K Ops x16 x16 enabled ~ 26

Intel® C628Chipset LBG-L 4/4 100 Gb/s 100 Gbs 100K Ops x16 x16 enabled ~ 21

All SKUs, frequencies and features are PRELIMINARY and can change without notice.

This slide under embargo until 9:15 AM PDT July 11, 2017

58

Page 59: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017 59

Summary of LBG HSIO FeaturesDMI• Dedicated Gen3 x4, 4GB/s raw bidirectional throughput

PCIe* Uplink• Dedicated Gen3 1 x16 controller• Configurable Gen3 1 x8 controller

PCIe* Downlink• Configurable Gen3, 5 x4 controllers, total of 20 lanes

SATA• Configurable Gen3, 14 ports over 2 controllers• Soft strap or GPIO based selection of PCIe or SATA

USB• Configurable 10 USB3.0 lanes, 14 USB2.0 lanes• Closed-Chassis Debug (DCI)

GbE: • 1 MAC port, configurable over 5 lanes

Flex IO Lane MuxingLane # PCIe Up SATA USB GbE

01 USB2 3.03456 PCIe x17 PCIe x2 x18 x4 PCIe x19 x2 x1 GbE

10 PCIe x1 GbE11 PCIe x2 x1 GbE12 x4 PCIe x113 x2 x114 PCIe x1 sSATA GbE15 PCIe x2 x1 6 x116 x4 PCIe x117 x2 x1 GbE18 PCIe x119 PCIe x2 x120 x4 PCIe x1 PCIe21 x2 x1 x8/x4/x122 PCIe x1 SATA23 PCIe x2 x1 8 x124 x4 PCIe x125 x2 x1

PCIe Down

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 60: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017 60

Other FeatureseSPI (Enhanced Serial Peripheral Interface):

• High-speed, low pin count LPC replacement interface

• Simplified routing, low-cost BMC enabling and BOM reduction

• Supports PCH or BMC-attached Flash sharing, GPIO virtualization

Power management:

• ASPM S-states and DeepS5 support. 2b VID control of logic core voltage

• Independent host, CSME, and IE operating states

RAS:

• ADR: Asynchronous DRAM Self-Refresh

• DWR: Demoted Warm Reset

eSPI

(7)

PECI

BMCSIO

LBG

Few

er G

PIO

s

GPIOs

ECSIO

SMLi

nk

LPC(

9-13

)

PECI

15+

GPI

Os

GPIOs

This slide under embargo until 9:15 AM PDT July 11, 2017

WBG

Page 61: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Xeon® Scalable Processor

PURLEYPRESALES30-3-30

The Secure, Agile, Next-Generation Platform for Multi-Cloud Infrastructures

Pervasive performancefor Actionable Insights Agile service deliverySecurity without compromise

Skylake-SP cores Intel® AVX-512

Feeds: UPI, 6x DDR4, 3x16 PCIe, Intel® SSDs

Integration: Intel® Ethernet / Omni-Path / Intel® QuickAssist /

FPGA

Intel® AVX-512 PPK, MPX, MBE

Intel® QAT w/ Secure Key Management

Intel® Trusted InfrastructureIntel® Boot Guard

Intel® Volume Management Device TechnologyIntel® RAS

Open Stack Software Optimizations

61

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 62: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017
Page 63: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017
Page 64: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

New Memory Latency and Bandwidth Optimizations

64

Direct-to-UPIFor UPI requests from remote socket, memory controller directly

sends data to UPI instead of going through CHA

Local and Remote Direct-to-CoreData sent directly from memory controller or UPI to requesting core

UPI PrefetchSimilar to XPT prefetch, but for UPI requests from remote socket

Sub-NUMA Cluster Mode Associates LLC slice with nearest memory controller

Applications use NUMA primitives to achieve lower LLC/memory latency

XPT PrefetchCore miss initiates local memory access in parallel with LLC access

Uses history-based prediction to avoid unnecessary prefetches

Cache & Memory Intel® UPI

UPI OptimizationsHitME cache – directory cache for frequently used lines

IO Directory Cache – directory cache for remote IO writes

Opportunistic Snoop Broadcast (OSB) – Avoids directory lookup and update for local InvItoE

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 65: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Sub-NUMA Clusters – 2 SNC Example

SNC partitions the LLC banks and associates them with memory controller to localize LLC miss traffic

• LLC miss latency to local cluster is smaller

• Mesh traffic is localized, reducing uncore power and sustaining higher BW

Remote SNC Access

65

Core

LLC

Core

LLC

MemCtrl

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

MemCtrl

Core

LLC

Core

LLC

1

2

3

Core

LLC

Core

LLC

MemCtrl

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

Core

LLC

MemCtrl

Core

LLC

Core

LLC

1

2

3

Without SNC

Core

LLCLLCLLCLLC

Core

LLCLLCLLC

MemCtrl

Core

LLC

Core

LLC

Core

LLCLLCLLCLLC

Core

LLCLLCLLC

Core

LLC

Core

LLC

Core

LLCLLCLLC

Core

LLCLLCLLCLLC

Core

LLCLLCLLC

Core

LLC

Core

LLC

Core

LLCLLCLLC

LLCLLCLLCLLC

Core

Core

LLCLLCLLC

Core

LLC

Core

LLC

Core

LLCLLCLLC

Core

LLCLLCLLCLLC

Core

LLCLLCLLCLLC

Core

LLC

Core

LLC

Core

LLCLLCLLC

Core

LLCLLCLLCLLC

Core

LLCLLCLLCLLC

MemCtrl

Core

LLCLLC

Core

LLC

1

2

3

Local SNC Access1

2

3

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 66: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Volume Management Device (Intel® VMD)This slide under embargo until 9:15 AM PDT July 11, 2017

Intel® VMD is a CPU-integrated device to aggregate NVMe SSDs into a storage volume and enables other storage services such as RAID • Intel® VMD is an “integrated end point” that

stops OS enumeration of devices under it• Intel® VMD maps entire PCIe* trees into its

own address space (a domain)• Intel® VMD driver sets up and manages the

domain (enumerate, event/error handling), but out of fast IO path

Intel VMD Driver Interface

Intel NVMe Driver

PCIe Domain Management

OS Storage Interface

I/ O

Block I / O

Configuration Interface Block I / O

I/ O

SSD

SSD S

SD

SSD

Switch

I/ OPCIe Configuration ,

Errors & Events

PCIe Bus

OS Storage Interface

Intel VMD

Eliminates additional components to provide a full-feature storage solution 66

Page 67: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Speed Shift Technology

Hardware P-state (HWP) is a new capability for cooperative hardware + software performance control

• Hardware monitors activity / scalability and selects frequency at much faster time scale

• Node Manager or OS provides guidance and constraints to direct hardware

– Energy Performance Preference: A profile of desired balance between power and performance

– Minimum Performance Level: A performance floor for meeting QoS requirements

– Maximum Performance Level: A ceiling to limit low priority applications and services

Node Manager or OS is no longer required to monitor activity and update frequency requests at regular intervals

• Node Manager or OS can update HWP guidance based on important events

– Change in administrator settings, real-time process started, etc.

This slide under embargo until 9:15 AM PDT July 11, 2017

67

Page 68: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Intel® Speed Shift Technology

68

What is the Energy Performance Preference?

Specifies software preference towards high performance, low power, or some balance between the two

20

30

40

50

60

70

80

90

0 10000 20000 30000

node

pow

er (W

)

performance (throughput)

Energy Performance Preference impact on power

minimum minimum + x maximum - x maximum

0

500

1000

1500

2000

2500

3000

0 10000 20000 30000

freq

uenc

y (M

Hz)

performance (throughput)

Energy Performance Preference impact on performance

minimum minimum + x maximum - x maximum

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 69: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Lewisburg - New Chipset for the Converged Platform

Provides a common footprint from entry level to full configuration with integrated classic server functions

LAN: Intel® Ethernet Connection X722 with up to 4x10GbE

• Supports Network Virtualization Offloads and iWARP RDMA

Comms and Storage accelerator: Intel® QuickAssist Technology (Intel® QAT)

• Crypto / compression up to 100Gbps

• Public key exchange up to 100K Ops

*Other names and brands may be claimed as the property of others.

69

10GbE Intel® QAT

Mux

PCIe* 8/16 uplink

DMI I/F

DMI

SATA sSATA PCIe RootxHCI 1Gbe LAN

High Speed I/O

ME

SPI

eSPI

IE

SMBus

HD Audio

LPC

High Speed I/O

Low Speed I/O

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 70: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017

Lewisburg PCH: Intel® QuickAssist Technology

• Enables standard server platforms to offer ubiquitous compression and security features

• Networking, Storage, Big Data, Cloud, HPC, and Data center applications achieve high performance on:

• Bulk Ciphers, Authentication

• Public Key Cryptography

• Compression

70

0 20 40 60 80 100 120

IPSec

SSL

RSAs 2K

Compression

8920 (Cave Creek) 8955 (Coleto Creek) Lewisburg

Cryptographic Ciphers &

Authentication

Public KeyCryptography

Compression

Symmetric Encryption and Authentication

Secure Key Establishment (Asymmetric Encryption, Digital Signatures, Key Exchange)

Lossless Compression in Flight and at Rest

Certificate Shared Secret

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 71: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017

Content Under Embargo Until 9:15 AM PST July 11, 2017Intel Press Workshops – June 2017 71

Configuration: Intel® QATQAT API Level tests 1-Node, 1 x Intel® Xeon® Platinum 8180 ProcessorProcessor Intel(R) Xeon(R) Gold 6152 H0 (30M Cache, 2.1 GHz)Vendor IntelNodes 1Sockets 1Cores Per Processor 22Logical Processors 44Platform Purley-EP (Lewisburg – B1 stepping)Accelerator Used Intel Lewisburg in x24 link modePlatform Comments Neon CityMemory DIMMs Slots used/Processor 6Total Memory 96 GBMemory DIMM Configuration 16 GB / 2666 MT/s / DDR4 RDIMM

Memory Comments Kingston 9965662-009.A00G, 16GB, 2Rx8

Network Interface Cards 3x XL710 X710 (Quad Fortville Card) , 12 10Gb ports usedOS Ubuntu 16.10OS/Kernel Comments 4.8Primary / Secondary Software QAT1.7.Upstream.L.1.0.0-15

Other Configurations BIOS:PLYDCRB1.86B.0114.R11.16122119, CPU Power&Performance -Perf, P,C and C states disabled, NUMA Enabled

Computer Type ServerBenchmark QAT API Level Sample tests

This slide under embargo until 9:15 AM PDT July 11, 2017

Page 72: Akhilesh Kumar, Skylake-SP CPU Architect Malay … · Akhilesh Kumar, Skylake-SP CPU Architect. Malay Trivedi, Lewisburg PCH Architect. June 12th, 2017