virtex6 pcie gen2 xfest 2009 v1 1 - eetrend.com

Copyright © 2009. Avnet, Inc. All rights reserved.

11

Attention

The material contained in this presentation is the property of Avnet Electronics Marketing. Use of this material in it’s whole or in part is restricted to Avnet’s X-Fest program and Avnet employees. Any use by non-Avnet employees outside of the X-Fest program is prohibited.

For additional information, please contact Jim Beneke at Avnet ([email protected]).

Designing with the Virtex®-6 PCIe® Gen 2 Endpoint Block


2Objectives

Introduce engineers to the Xilinx® Virtex-6 integrated block for PCIe

Provide an overview of the Virtex-6 PCIe Gen 2 Endpoint Block design and verification tools

At the end of the presentation, engineers will learn– The basic architecture of the Virtex-6 integrated block for PCIe– What Xilinx tools to use to design with the Virtex-6 PCIe Gen 2 Endpoint

Block


3Agenda

LXT/SXT family overviewXilinx PCIe solutionsIntroduction to Virtex-6 PCIe Gen 2 Endpoint BlockPCIe Gen 2 clockingDesigning with Virtex-6 PCIe Gen 2 Endpoint BlockPCIe Gen 2 Compliance TestingWrap-Up


4Agenda



5Virtex-6 Platform Family

Virtex-6 is the 3rd generation of Advanced Silicon Modular Block (ASMBL™) architecture– Virtex-4, 1st generation– Virtex-5, 2nd generation

Virtex-6 sub-families– LX : High-performance logic and parallel IO

– LXT: Logic-oriented with serial capabilities

– SXT: Signal processing with serial capabilities

– HXT: Highest channel count and highest performance serial capabilities

Users can choose the best mix of resources to optimize cost and performance

LXPlatformPlatform

LXTPlatformPlatform

SXTPlatformPlatform

HXTPlatformPlatform


6Virtex-6 LXT/SXT Platform Family Embedded Features

High-speed serial GTX transceivers– Up to 6.5Gbps serial data rate (GTH will

support up to 11.18Gbps)– Depending on the package, LXT/SXT

devices support 12-36 GTX transceivers (HXT family will have up to 48 GTH transceivers)

PCI Express® block– Up to 2 PCI Express Gen 1/Gen 2 Blocks

(HXT family will have up to 4 PCIe blocks)– PCI Express Gen 2 at 5.0Gbps– Endpoint and Root Port support

Ethernet MAC– 4 Fully integrated 10/100/1000 Mbps

Ethernet MACs– Designed to the IEEE 802.3-2005

specifications

10/100/100010/100/1000PHYPHY

Virtex-6

TEMAC


7Virtex-6 LX/LXT/SXT Platform Family

LX195T LX240TLX130TLX75T LX365T200K 241K128K74.5K 364K250K 301K160K93.1K 455K3040 365017401045 4130344 416264156 41610 12106 12640 768480288 57620 242012 242 221 24 444 4

240, 8240, 8400, 12 400, 12400, 12360, 12600, 20 600, 20600, 20 600, 20

PCI Express Endpoint Blocks

Logic CellsCLB Flip-Flops

Distributed RAM (Kbits)36-Kbit BRAM Blocks

Mixed Mode Clock ManagersDSP48E1 Slices

GTX Transceivers

10/100/1000 Ethernet MACsPackage SizeFF484 23x23mmFF784 29x29mmFF1156 35x35mm

x,y x = SelectIO, y = GTX Transceivers

LX760 SX315TLX550T SX475T759K 315K550K 476K948K 394K687K 595K8280 50906200 7640720 704632 106418 1218 18864 1344864 20160 2436 360 22 20 44 4

600, 20 600, 20720, 24 720, 24FF1759 42.5x42.5mm

FF1760 42.5x42.5mm720, 24840, 36 840, 36

1200, 01200, 0


8Agenda



9PCI Express Gen 2 Standard

The PCI Express Gen 2 standard is the next-generation evolution of the PCI Express Gen 1 standard– The Gen 2 increase bandwidth equates to 5Gbps per wire pair, in

each direction, double the bandwidth of the current PCI Express Gen 1 specification

– The effective bandwidth is 80% of the raw bandwidth due to 8B/10B encoding

Link

x1x2x4x8

Effective Bandwidth PerDirection

4Gbps8Gbps

16Gbps32Gbps

Raw Bandwidth PerDirection

5Gbps10Gbps20Gbps40Gbps

PCI Express Gen 2 Bandwidth

x12x16

48Gbps64Gbps

60Gbps80Gbps


10PCI Express Use Model

Root Complex– A Root Complex (RC) denotes

the root of an I/O hierarchy that connects the CPU and memory subsystem to the I/O

– Comparable to the PCI North Bridge

Switch– Logical assembly of multiple

virtual PCIe-PCIe bridge devices

– Comparable to the PCI South Bridge

Endpoint– Previously known as the

Peripheral

Root Complex MemoryPCIe to

PCIBridge

CPU

Switch PCIeEndpoint

PCIeEndpoint

LegacyPCIe

Endpoint

PCIeEndpoint

UpstreamPort

DownstreamPort

RootPort

End-Point

At a minimum the Host Bridge, Memory Arbiter, and Root Port are required to create a Root Complex


11PCIe v2.0

PCI Express v2.0 (Gen 2)– 2.5Gbps is required– 5.0Gbps per lane is optional part of the specification

PCI Express specifications– Obtained from PCI-SIG® (PCI Special Interest Group)

http://www.pcisig.com/specifications/pciexpress/specifications– Base Specification for protocol– Card Electromechanical Specification (CEM) for electrical characteristics

PCIe 2.0 specifications defines 68-105 Ohms (85 Ohms nominal) differential trace impedance for 5.0Gbps capable links– PCIe 1.1 specifications defines100 Ohms differential trace impedance


12Xilinx PCIe Solutions

Soft IP (Alliance Partner)Gen 1 / Gen 21, 2, 4, and 8Virtex-5 FXT/TXT

Soft IP (Alliance Partner)Gen 12 and 4Spartan-6

Integrated BlockGen 11, 2, 4, and 8Virtex-5 LXT/SXT/FXT/TXT

Integrated BlockGen 1 / Gen 21, 2, 4, and 8Virtex-6

Gen 1Gen 1

Gen 2

Gen 1Gen 1 / Gen 2

Integrated Block1Spartan®-6

Soft IP (Alliance Partner)4 and 8Virtex-6

Soft IP (DO-DI-PCIE-PIPE)Soft IP (DO-DI-PCIEXP)

IP

1Spartan-31, 4, and 8Virtex-4 FX

LanesFPGA Device

Alliance program members: Northwest Logic® and PLDA®


13Xilinx Virtex-6 PCI Express IP Roadmap

Endpoint Wrapper– x1 - x8 Gen 1/2– Delivered through Coregen™– Simulation and implementation– Source code RTL wrapper– Verilog (VHDL in 11.4)

Root Port Wrapper– Up to x4 Gen 2– Delivered as ZIP file– Verilog only– Simulation only– Simple example design– Qualified customers only

ISE® 11.2 (June 09) ISE 11.3 (Sept 09)

Root Port Wrapper– x1 - x8 Gen 1– x1, x2 and x4 Gen 2– Delivered through Coregen– Verilog (VHDL in 11.4)– Simulation and implementation

Root Complex– x1 Gen 1 only (PLB46™)

– Delivered through EDK®


14PCIe Demonstration Matrix

Performance Demonstration

Basic Functionality/Debug

Custom NWLogic App

Custom Xilinx App

Custom Xilinx App

- MET™- PCI Tree®- HWDirect®- Lecroy CatScan®- Jungo WinDriver®

Software Application

Northwest Logic

XAPP859

XAPP1052

MET – XAPP1022www.pcitree.dewww.eprotek.comwww.lecroy.comwww.jungo.com

Where Do I Get Software?

ML555, ML605

Northwest Logic

Northwest LogicDMA Demo

ML555, ML50XXAPP859PCIe to DDR2 with DMA

ML555™, ML50X™ML605™

XAPP1052Bus Master DMA Reference Design

PIO can easily target any board by modifying the Coregen provided UCF (ML605 board is directly supported by Coregen)

Use Coregen to generate the bitstream

Programmed I/O (PIO)

BoardsWhere Do I Get

Bitstream?

Design Description


15Avnet® Virtex-6 LX130T PCIe x4 Gen 1/Gen 2 Board

Board Features– Virtex-6 LX130T FPGA (Xilinx®)– PCIe x1 and x4 Support– SFP & SMA Connectors (Tyco®)– 128MB DDR3 SDRAM (Micron®)– 32MB Parallel Flash (Numonyx®)– 16MB Platform Flash XL (Xilinx®)– 10/100/1000 Ethernet PHY (NSC®)– USB Bridge (Silicon Labs®)– LVDS Clock Generator (Maxim®)– FMC HPC Slot

~$1395 Target ResaleAvailable ~Q4 2009

FMC HPC Slot

Communication Ports

Miscellaneous I/O

SM/BPI

Memory Interfaces

GTX Interfaces

Clock Sources

PCI-Express x4

SFP Connector

SMA Connectors

10/100/1000 PHY

USB-RS232 Bridge

RS232 Port

Push Switches

DIP Switches

User LEDs

128MB DDR3 SDRAM

16MB Platform Flash XL

32MB Parallel Flash

ProgrammableLVDS Clock Source

LVTTL OSC@ 100 MHz

0.75V, 1.0V, 1.2V,1.5V, 1.8V, 2.5V, and

3.3V Regulators

Power Supply

JTAG Port

Con

nect

or

Virtex-6LX130TFFG784


16Review Questions

What are the sub-families within the Virtex-6 platform family?– LX, LXT, SXT, and HXT

What is the nominal trace impedance for PCIe Gen 2?– 85 Ohms

What is the effective bandwidth of a PCIe x1 Gen 2 link?– 4Gbps


17Agenda



18Xilinx Virtex-6 Integrated PCI Express Block Features

The Virtex-6 PCIe block contains the functionality defined in the specifications maintained by the PCI-SIG– Compliant with the PCI Express Base 2.0 Specification– Configurable for Gen 1 (2.5Gbps) or Gen 2 (5.0Gbps) data rates

• x8, x4, x2, or x1 Gen 1/ Gen 2 lane width– Configurable for Endpoint or Root Port applications

• Endpoint requires minimal fabric logic• Root Port and Root Complex with additional fabric logic• Chip-to-chip communications via PCI Express

– RocketIO™ GTX transceivers implement a fully compliant PCIe PHY– Maximum Payload Size (MPS) of 128/256/512/1024 bytes supported– Up to 6 x 32 bit or 3 x 64 bit BARs


19Xilinx Virtex-6 Integrated PCI Express Block

TransactionLayer

Module

Data LinkLayer

Module

PhysicalLayer

Module

Configuration and Capabilities Module

PCIe Block

Block RAM Interface

TransactionLayer Interface

Clock andReset

Interface

ConfigurationManagement

Interface

RocketIOGTXs

DRP


20Virtex-6 PCIe x8 Gen 2 Endpoint

Virtex-6 x8 Gen 2 Endpoint is fully delivered in ISE 11.3– The Gen 2 wrapper utilizes ~1200 LUT / 2850 FF– 128-bit / 250MHz Local Link Transaction Interface– Virtex-6 x8 Gen 2 Endpoint is only supported in –3 speed grade devices

PCI ExpressIntegrated Block GTX

x8Wrapper(fabric)

8 LanesPCIe

64-bit500MHz

128-bit250MHz

UserInterface

Delivered through Coregen


21Virtex-6 PCIe Block Wrapper Resource Usage

Endpoint / Root Port

15015064x1 Gen1/Gen2

28501200128x8 Gen2 (Endpoint)*65065064x8 Gen135035064x4 Gen1/Gen220020064x2 Gen1/Gen2

Flip-FlopsLUTsUser Interface Bus Width

Product

*Root Port wrapper for PCIe x8 Gen 2 is not currently available.


22Why Alliance Partner IP?

PCI Express Direct Memory Access (DMA) engine– DMA is used in most PCIe applications– Xilinx is not developing a DMA– Alliance partners will deliver a DMA controller for the Virtex-6

PCIe block

Soft PCI Express IP– Enables features not in the integrated block

• End-to-end CRC• Advanced error reporting• Two Virtual Channels

– Additional PCIe Blocks


23V6 Endpoint Example with Alliance Partner DMA

PCIeEndpoint

BlockWrapper

RootComplex

CPU

Virtex-6 FPGA

MemoryController

HostSystemMemory

Memory(DDR2/DDR3)

DMA Controller(PLDA or NWL)

+User Application

Virtex-6 PCIe Endpoint Card


24Virtex-6 PCIe User Interface

250MHz128-Bit8Gen 2250MHz64-Bit4Gen 2

125 (default) or 25064-Bit2Gen 262.5 (default), 125, or 25064-Bit1Gen 2

250MHz64-Bit8Gen 1125 (default) or 25064-Bit4Gen 1

62.5 (default), 125, or 25064-Bit2Gen 162.5 (default), 125, or 25064-Bit1Gen 1

Bus-Speed(MHz)

Bus-Width

LanesGen 1/2

PCIe Block GTX

1-8 Lanes

User Interface

Virtex-6 FPGA

It is possible to select the clock frequency of the core's user interface via Coregen GUI– Each lane width provides multiple frequency choices, a default frequency

and alternative frequencies– Where possible, Xilinx recommends using the default frequency– Non-default frequencies will result in more power and difficulties to close

timing


25

Virtex-6 supports the following new and expanded PCIe features

Virtex-6 New and Expanded PCIe Feature Set

Physical Layer

Data Link

Layer

Transaction Layer

Configuration Module

PCIe Integrated Block

TX BRAM

GTXs

RX BRAM

TransactionInterface

ConfigurationInterface

Phy LayerDRP

– Larger MPS (1024 Bytes)– Dynamic Reconfiguration Port– 1 Virtual Channel 8 Traffic Classes

– Improved buffering– Transmit cut-through– MSI-X


26New Improved Buffering

BRAM usage is a function of Max Payload Size (MPS) and performance requirements

Transmit– Streaming (cut-through) mode

for reduced latency– Buffer Based - Up to 32

Transaction Layer Packets of all types and sizes

Receive– Credit based – efficient use of

BRAM– Buffers enough RX header

credits to allow line rate operation of the core

– Enough completion space so as not to cause RX overflow for Endpoint

Physical Layer

Data Link

Layer

Transaction Layer



TX BRAM

GTXs

RX BRAM



Phy LayerDRP

GUI Block RAM SelectionHigh Performance

vs. Low Resource Usage

GUI Block RAM SelectionHigh Performance

vs. Low Resource Usage


27Transmit Buffers

Each transmit buffer can hold one maximum sized TLP– A maximum sized TLP is a TLP with a 4-DWORD header plus a data

payload equal to the MAX_PAYLOAD_SIZE of the core

For example, if the Capability Max Payload Size selected for theEndpoint core is 256 bytes and the performance level selected is high– There are 29 total transmit buffers– Each of these buffers can hold at a maximum one 64-bit Memory Write

Request (4 DWORD header) plus 256 bytes of data (64 DWORDs) plusTLP Digest (1 DWORD) for a total of 69 DWORDs

Capability MaxPayload Size

(Bytes)

128

Good (Minimize Block RAM Usage)

Performance Level

High (Maximize Performance)

26 32

256 14 29

512 15 30

1024 15 31


28Receiver Credits Available Initial Values

The following table shows the initial space available for the Posted, Non-Posted, and Completion queues– One Header Credit is equal to either a 3 or 4 DWORD TLP header

and one Data Credit is equal to 16 bytes of payload data

CreditCategory

Non-PostedHeader 12

PerformanceLevelGood

128 byteCapability MPS




HighNon-Posted

Data 12GoodHigh

PostedHeader 32

GoodHigh

PostedData

GoodHigh

77154

77154

154308

308616

CompletionHeader 36

GoodHigh

CompletionData

GoodHigh

77154

77154

154308

308616


29Interrupts

INTx Emulation– Assert and de-assert messages– Virtual wire– Supports interrupt pin INTA

MSI– Memory write transactions– 32 interrupt vectors

MSI-X - NEW for Virtex-6– Supports great number of interrupt vectors (2048)– Requires the use of a BAR (specified in Coregen GUI)– Support in Windows Vista® and some builds of Linux®– Some FPGA logic required for the vector table

All interrupt modes are configured via

Coregen GUI


30Dynamic Reconfiguration Port

Virtex-6 Dynamic Reconfiguration Port (DRP) allows users to dynamicallychange PCIe block configuration memory bits

Dynamic Reconfiguration Port

Dynamic Reconfiguration Port

Virtex-6 LXT/SXT FPGA

Physical Layer

Data Link

Layer

Transaction Layer



TX BRAM

GTXs

RX BRAM



Phy LayerDRP

– User can change any attribute of the PCIE_2_0 software primitive (for example, resizing the BARs)

– There are a number of attributes that should not be modified via DRP (refer to the V6 PCIe users guide for more information)

– Care must be taken so that attribute changes match the hardware

– Core must be held in reset when writing to DRP DRP can be enabled via Coregen GUI


31Virtex-6 PCIe Root Port

Root Port GTX EndpointGTX

Hardened IPSoft Logic

PCIe Link

Configuration Setup Logic

Custom UserLogic

Custom UserLogic

Custom UserLogic

Custom UserLogic

Most users will use the Root Port function to enable chip-to-chip communications– No processor or Operating System is required– Configuration support logic is needed to setup the configuration space of the

Root Port and the Endpoint– Root Port must be able to handle Error messages and Interrupts if the

Endpoint generates them (normally handled by an OS)– V6 Root Port can also interface to an ASSP Endpoint


32Conceptual Block Diagram of Root Complex

Root Port(PCI/PCI Bridge)

Root Port(PCI/PCI Bridge)

Memory Arbitration

and Controller

RCE Collector

RCI Endpoint

Host/PCI Bridge

RCRB

Memory

CPU

GTX

PCI B

us 0

EndpointGTX

Hardened IP

Soft Logic

PCIe Link

GTX

RCRB

At a minimum the Host Bridge, Memory Arbiter, and Root Port are required to create a Root Complex– Virtex-6 PCIe block functions as PCI-PCIe bridge– The CPU could be an internal soft processor such as MicroBlaze™


33Review Questions

What are the user interface bus width and frequency for the Virtex-6 PCIe x8 Gen 2?– 128 bits and 250MHz

How does transmit cut-through mode reduce latency?– By streaming the data

Can the Virtex-6 PCIe block support Root Port functionality?– Yes


34Agenda



35Virtex-6 PCIe Clocking

The Integrated Endpoint Block input system clock signal is called sys_clk. The core requires a 100MHz,125 MHz or 250MHz clock input– The clock frequency used must match the clock frequency selection in

the CORE Generator GUI– In a typical PCI Express solution, the PCI Express reference clock is a

Spread Spectrum Clock (SSC), provided at 100MHz

PCIe100MHzClock

Virtex-6PCIe Block

ExternalPLL

125 or 250MHzCLKP

CLKN

PCIe100MHzClock

Virtex-6PCIe Block

CLKP

CLKN

125/250 MHz Clock Input System

100 MHz Clock Input System


36Virtex-6 PCIe Gen 1/Gen 2 Clocking

The Virtex-6 PCIe block operated at Gen 2 data rate will require a 250MHz system clock input – This restriction may be removed in a future version of this core

The Virtex-6 PCIe block operated at Gen 1 data rate can use a 100, 125, or 250MHz system clock input

PCIe100MHzClock

Virtex-6PCIe Block

ExternalPLL

125/250MHzCLKP

CLKN

PCIe100MHzClock

Virtex-6PCIe Block

CLKP

CLKN

Gen 1 System

OR

PCIe100MHzClock

Virtex-6PCIe Block

ExternalPLL

250MHzCLKP

CLKN

Gen 2 System


37Virtex-6 PCIe Clocking Examples (Embedded System)

REFCLK@ 100MHz

PCIe LinkPCIe Switch orRoot Complex

Device

Virtex-6EndpointDevice

PCIe ClockOSC External PLL

CLK @125/250

MHz

100MHz

Embedded System Using 125/250MHzSystem Clock

REFCLK@ 100MHz

PCIe LinkPCIe Switch orRoot Complex

Device


PCIe ClockOSC

100MHz

Embedded System Using 100MHzSystem Clock


38Virtex-6 PCIe Clocking Examples (Open System)

100MHz PCI ExpressClock (SSC)

PCIeLink


PCIe EdgeConnector

CLK @ 125/250MHz

Open System Add-In Card Using125/250MHz System Clock

External PLL

100MHz PCI ExpressClock (SSC)

PCIeLink


PCIe EdgeConnector

Open System Add-In Card Using100MHz System Clock


39PCIe Clocking on Avnet V6LX130T Board

PCIe_RefClk_100_P

PCIe_RefClk_100_N

ICS874003-05

321

ON

3.3V

OFF F_SEL0F_SEL1F_SEL2

nQA0

QA0

PBSwitch MR

CLKnCLK

nQB0

QB0MGTREFCLK0P

MGTREFCLK0N

MGTREFCLK1P

MGTREFCLK1N

The QA0/nQA0 and QB0/nQB0 outputs can be set to 100MHz, 125MHz, or 250MHz using the F_SEL[2:0] inputs of the ICS874003-05 device.


40Agenda



41Designing with Virtex-6 PCIe Endpoint Block

Typical design flow uses the Core Generator – The Core Generator wizard configures the required blocks such as GTX,

BRAM, Clock, and Reset– It outputs a Programmed Input Output (PIO) example design for Endpoint or

a Configurator example design for Root Port

The Core Generator example simulation design consists of the following discrete parts

The example design has been tested and verified with Mentor Graphics®ModelSim™, NC-Sim®, and Synopsys® VCS simulators™– Aldec® simulator can also be used (see the Appendix for more information)– ISIM™ can also be used via a custom script file

Example Design Simulation/Design Files

End-Point - PIODesign Files - PIO example designTestbench - The Root Port Bus Functional Model (BFM), a test benchthat generates, consumes, and checks PCI Express bus traffic

Root Port - Configurator Design Files - Configurator example designTestbench - Slave PIO


42Xilinx Virtex-6 PCI Express Endpoint Configuration

The Transceiver, Memory, Clock and the Reset interfaces are automatically connected in the CORE Generator wrappers– These interfaces are not visible outside of the wrappers– User application must be implemented in the FPGA fabric and

interfaced to the PCIe block using the Transaction Layer Interface

Automatically connected by the CORE Generator

Optional connection

User connection

TransactionLayer

Data LinkLayer

PhysicalLayer

Configuration and Capabilities ModulePCIeBlock

Block RAM Interface


TransceiverInterface

DRP Clock andReset

Interface


Interface


43PCI Express Wizard Endpoint Block Wrapper Output

TransactionLayer

Data LinkLayer

PhysicalLayer

Configuration and Capabilities Module PCIeBlock

UserInterface

SerialInterface

DRP


Interface

BlockRAM

ClockModule

ResetModule

GTXTiles

PCIe Endpoint Block Wrapper


44Core Generator Deliverables

Parameterized hard IP core (RTL wrapper source code)

Programmed Input Output (PIO) or Configurator example design

Customer simulation demonstration test bench– Verilog HDL simulation flow (VHDL is scheduled for 11.4)

Customer implementation demonstration– Example UCF– Complete implementation scripts delivered for PIO or Configurator

example design


45Review Questions

What is the frequency of the system clock input to the Virtex-6 PCIe block?– 100, 125, or 250MHz

In addition to the PCIe hard block, what other blocks are automatically configured by the PCIe Coregen Wizard?– GTX transceivers, BRAM, Clock, and Reset

What is the procedure for assigning GTX pins to the PCIe lanes?– Use Coregen to generate a UCF


46Agenda



47Xilinx Virtex-6 PCIe PCI-SIG Compliance Testing

Virtex-6 PCIe Endpoint block has passed the following tests– PCI-SIG compliance test – 3 SIG Gold suites (Electrical, Configuration and Protocol)– Interoperability– x1, x2, x4, and x8 Endpoint configurations

Reference board used for PCI-SIG compliance– Xilinx ML605 evaluation board x1, x2, x4, and x8 configurations


48Agenda



49Key Takeaways

Virtex-6 integrated PCIe block is fully compliant with the PCI Express Base 2.0 Specification– Configurable for Gen 1 (2.5Gbps) or Gen 2 (5.0Gbps) data rates

• x8, x4, x2, or x1 Gen 1/ Gen 2 lane width– Configurable for Endpoint or Root Port applications– RocketIO™ GTX transceivers implement a fully compliant PCIe PHY

Xilinx Coregen can be used to generate an Endpoint (PIO) or a Root Port (Configurator) example design– Complete implementation scripts and UCF delivered for PIO or

Configurator example design– Verilog HDL simulation support (VHDL is scheduled for 11.4)


50Virtex-6 PCIe Demo

V6 PCIe Demo Description– 8 GTX transceivers running at 2.5Gbps or 5Gbps (PCIe Gen 1/Gen 2) – DDR3 memory interface– Packet based DMA Design – Uses Xilinx ML605 evaluation kit

Other Equipment used – Software Application / GUI / Device Drivers – PC with Windows XP® / Vista OS


51Closing Comments

Please go to the following URL to download the X-Fest 2009 course presentations

http://em.avnet.com/xfsupport2010

Visit the X-Fest 2009 forum for latest discussions on various courseshttp://community.em.avnet.com/t5/XFEST-2009/ct-p/XFEST_2009

Please Visit the Demo AreaThank You

Appendix


53Agenda

Virtex-6 PCIe Block overviewDebugging PCIe DesignsUsing the Aldec Simulator


54PCI Express Communication Methodology

The communication between two PCIe devices is referred to as a transaction– Transactions are packet-based

Each PCIe device can be a Requester and/or a Completer– Requester initiates a transaction– Completer responds to a request– Both the Root Complex and the Endpoint device can function as a

Completer as well as a Requester

Root Complex

Switch PCIeEndpoint

LegacyPCIe

Endpoint

Packet BasedTransactions

Packet BasedTransactions


55PCI Express Protocol Support

The PCI Express protocol supports four types of transactions– Memory (read and write)– I/O (read and write)– Configuration (read and write)– Message (communication information outside of the Memory, I/O, and

Configuration spaces such as interrupt signaling, error signaling, etc.)

Transactions are divided into three categories– Posted transactions– Non-posted transactions– Completion transactions


56Posted and Non-Posted Transactions

Memory writes and message transactions are posted transactions– The requester sends a packet, but the receiver does not return a

completion

Non-posted transactions (memory reads, I/O reads and writes, and configuration reads and writes) require a response and are implemented as split transactions– Completion packets can be directed to the correct originator

because each packet has a unique identifier


57Transaction Layer Module

The Transaction Layer Module (TLM) is the upper layer in the architecture– This module takes Transaction Layer Packets (TLPs) presented by user

logic at the Transaction Layer interface and schedules them for transmission over the link

– The Transaction Layer module also advises the user application when TLPs are received

– TLPs can both make requests and complete requests from another device

Transaction LayerModule


Data Link LayerModule

Block RAM Interface


58Transaction Layer Module - Packet

A Transaction Layer Packet (TLP) is composed of a header, data payload (for most packets), and optional end-to-end CRC (ECRC)– Packets must be formed by the user in accordance with the PCI

Express specification– Packets must be decoded by the user properly

Header

32-bitMachine 3 DWORDS 1 DWORD for address

64-bitMachine 4 DWORDS 2 DWORD for address

Payload

ReadRequest 0 No Payload

Write or ReadCompletion

Max Payload Size(MPS) or less

1024 bytes in Virtex-6 PCIeBlock

ECRC(Digest)

Included 1 DWORD Passed by TL to UserTrimmed 0 Still exists, but trimmed by TL

Criteria Size Notes


59Data Link Layer Module

The Data Link Layer Module (DLLM) resides between the Transaction Layer and the Physical Layer modules– Its primary responsibility is to provide a reliable mechanism for the

exchange of TLPs between two components on a link– Data Link Layer provides data exchange (TLPs), error detection and

recovery, initialization services and the generation and consumption of Data Link Layer Packets (DLLPs)

– The transmission portion of the DLL accepts TLPs from the Transaction Layer and generates the appropriate TLP sequence number and LinkCRC (LCRC), then passes the packet to the Physical Layer

TransactionLayer

Module

Data LinkLayer

Module

PhysicalLayer

Module


Block RAM Interface


60Data Link Layer Module – Data Link Layer Packets

The DLL also generates and consumes special packets called Data Link Layer Packets (DLLPs) that do not pass to the Transaction Layer– Types of DLLPs include acknowledgment (ACK/NAK), flow control, and

power management

The reception portion of the DLL checks the integrity of received TLPs– It also orders retransmission when the received TLP is found to be

corrupted

The transmission portion controls the order of release of the different types of packets– A prioritizer is included to sort the different sources of transmission into

order of priority and schedule them for transmission according to the priority order recommended in the PCIe Base Specification


61Physical Layer Module

The Physical Layer (PL) module carries out the following functions defined for the PL of a PCIe device– Packet framing and de-framing (Start of Frame and End of Frame)– Byte striping and un-striping; that is, distributing Tx packets over the

associated PL lanes and reassembling Rx packets received over the different PL lanes

– Link initialization and training– Scrambling, de-scrambling, and 8B/10B encoding and decoding of data

are also performed by the Physical Layer Module

TransactionLayer

Module

Data LinkLayer

Module

PhysicalLayer

Module


Block RAM Interface

TransceiverInterface


62PCI Express Packet Summary

A TLP is composed of a header, data payload (for most packets), and optional end-to-end CRC (ECRC)

The transmission portion of the DLL accepts TLPs from the Transaction Layer and generates the appropriate TLP sequence number and Link CRC (LCRC)

The Physical Layer appends the Start and End to the packet

Data Payload

Presented to Transaction Layer

HeaderSequenceNumberStart EndLCRCECRC

Appended by Data Link LayerAppended by Physical Layer


63Configuration and Capabilities Module

The Configuration and Capabilities module provides the repository for the different registers within the Configuration Space– It implements the legacy PCI configuration header defined in both the

PCI Express Base Specification and the earlier PCI bus specifications– It also implements the extended configuration space supported by PCI

Express systems that contain • Power management• Message signaled interrupts• Error reporting• Virtual Channel

TransactionLayer

Module

Data LinkLayer

Module

PhysicalLayer

ModuleTransaction

Layer Interface

Configuration and Capabilities ModuleManagementInterface


64Block RAM Interface

The PCIe block buffers are implemented using block RAMs– The sizes of the buffers can vary based on the application’s needs

There are two options for configuring these buffers in PCIe Wizard – Minimize Block RAM usage– Maximize performance

Transaction LayerModule

Data LinkLayer

Module

TransmitBlock RAM

ReceiveBlock RAM


65Agenda



66PCIe Debug Overview

Debug using PCIe configuration space registers– Why are these registers useful– PCI Express capability structure– How to read the capability structure linked-list

De-scrambling– What it is– How it works– Why it might be useful

Debug Ports– Overview of debug options included in Spartan-6 and Virtex-6 to help

customers debug PCIe designs


67PCI Express Configuration Space Review

The PCIe configuration space can be divided into three sections– PCI Express extended capabilities (0x100 – 0xfff)– Legacy extended capabilities list (0x40 – 0xff)– PCI 3.0 compatible configuration space (0x00 – 0x3f)


68Where Does the Capability List Start?

The value contained in configuration space address 0x34 is the starting point and always points to the very first item in the linked-list structure– Note that since PCIe

requires the PCIe capability structure, this register will always contain a value


69Structure of the Extended Capabilities Linked-lists

There are two separate linked lists of capability structures– Legacy PCI – starts with the

capabilities pointer at 0x34– PCIe Capabilities – always

starts at 0x100– Next Ptr = 0 terminates both

lists

Each capability structure must contain two items– ID which identifies the type of

structure– Pointer to next item in the list

Size of the capability structure is implied by the ID Capabilities Pointer0x34

0x40

0x100

ID

ID Next Ptr

ID 0

ID Next Ptr

ID 0PCI Express

Extended

Capabilities

PCI Extended Capabilities

Next Ptr

ID Next Ptr


70PCI Express Capability Structure

Lots of useful debug info in this structure– Device control register

• Max payload, max read request

– Device status register• Errors detected

– Link control register• Read Completion

Boundary (RCB)– Link status register

• Negotiated link width• Current link speed

For exhaustive discussion see Chapter 7.8 of the PCI Express Base Specification Rev 2.0

PCI ExpressCap ID

Next CapPointerPCI Express Capabilities Register

Device Capabilities

Device Status Device Control

Link Capabilities

Link Status Link Control

Slot Capabilities

Slot Status Slot Control

Root Capabilities

Root Status Root Control

Device Capabilities 2

Device Status 2 Device Control 2

Link Capabilities 2

Link Status 2 Link Control 2

Slot Capabilities 2

Slot Status 2 Slot Control 2

071531

00h

04h

08h

0Ch

10h

14h

18h

1Ch

20h

24h

28h

2Ch

30h

34h

38h

ByteOffset


71Customers Common PCIe Issues

Example: Customer is seeing very low bandwidth

Possible reasons– Link partner lane width capability is less than expected

• Many x8 connectors on motherboards only route 4 lanes– Width down-trained due to SI issues on upper lanes– Gen 2 speed change did not occur at link-training due to SI– Customer is not using bus-mastering DMA

Check the link capability and link status registers of both devices on the link– Check the link capability register of both devices– Check “negotiated link width” and “current link speed” fields in the

link status register


72

PCI ExpressCap ID


Device Capabilities


Link Capabilities


Slot Capabilities


Root Capabilities




Link Capabilities 2


Slot Capabilities 2


071531

00h

04h

08h

0Ch

10h

14h

18h

1Ch

20h

24h

28h

2Ch

30h

34h

38h

ByteOffset

Reading the Link Capability/Status Register to Verify Negotiated Link Width and Speed

Link capability register (0Ch)– Where we declare our

capabilities• Supported link speed• Max link width• Etc.

Link status register (12h)– What’s currently happening

on the link• Current link speed• Negotiated link width• Etc.

Also check the same register in the Link Partner to verify Link Partner capabilities


73

PCI ExpressCap ID


Device Capabilities


Link Capabilities


071531

00h

04h

08h

0Ch

10h

ByteOffset

Reading the Link Status Register to Verify Negotiated Link Width and Speed

Also check the same register in the Link Partner to verify Link Partnercapabilities


74Determining the Max Payload Size

Device capability register– Where we declare the

maximum payload size of which we are capable

Device control register– Where the system software

sets all of the devices to a common maximum payload size

PCI ExpressCap ID


Device Capabilities


Link Capabilities


Slot Capabilities


Root Capabilities




Link Capabilities 2


Slot Capabilities 2


071531

00h

04h

08h

0Ch

10h

14h

18h

1Ch

20h

24h

28h

2Ch

30h

34h

38h

ByteOffset


75Debugging Using PCIe Registers

Use PCITREE/HWDirect (NT) or LSPCI (Linux) to read registers in PCIe Endpoint– Most of these register values are outputs to user via buses on the

CFG port– Device control register = cfg_dcommand[15:0]– Device status register = cfg_dstatus[15:0]

Example: Use for ChipScope Triggers– Imagine system is blue screening– Usually, FPGA is transmitting a fatal error message– Triggering on device status bit 2 gives view of situation when

message transmitted• cfg_dstatus<2> = fatal error detected

– At this trigger point, you may want to view status of MGT ports or other PCIe block ports to see if you can determine reason for failure

RsvdZ

015 123456

Transaction PendingAUX Power Detected

Unsupported Request DetectedFatal Error Detected

Non-Fatal Error DetectedCorrectable Error Detected


76Using PCItree

Configuration Registers

Selected PCIe Device


77Using HWDirect

PCItree cannot read extended capability space

HWDirect can read extended space– Low-cost Windows

shareware program (~$38)

– www.eprotek.com

On Linux, use LSPCI with –xxxx switch:e.g.: /sbin/lspci -xxxx


78Debug Using Configuration Space Review

PCI Configuration Space is useful for certain debug situations

– Bandwidth issues – speed and lane width

– Error logging/reporting

– Verifying capabilities of device

– Checking if system software enabled certain capabilities

Knowing how to read the linked-list is helpful

– Especially when checking the link partner capabilities list

Configuration port outputs of the PCIe core can be used as ChipScope or link analyzer triggers


79Debugging Conclusion

Standard PCIe register set is helpful in debugging problems

Virtex-6 and Spartan-6 user guides contain new debug chapters

Debug ports available to assist in analyzing problems


80De-scrambling Design

All TLPs, DLLPs, and Idles are scrambled– Required by the PCIe specification to reduce EMI noise

De-scrambling design places a descrambler on both the TX and RX path– Descrambler taps off TX and RX data path– Produces data in legible format– Packet decoder also included to provide trigger signals for ChipScope

Allows user to view traffic at the MGTs using ChipScope Pro

Also works in simulation

The De-scrambling design will be available as an Answer Record


81De-scrambler Block Diagram

TransactionLayer

Module

GTP/GTXTransceivers

Data LinkLayer

Module

PhysicalLayer

Module

TXDe-scrambler

PCIe Block

Block RAM Interface

ChipScopeILA

ChipScopeILA

RXDe-scrambler

FPGA


82How is De-scrambling Useful?

Many customers do not have link analyzer– This is not a replacement for a link analyzer (better than nothing at all)

Verify ACKs/NAKs being returned as expected– Excessive NAKs or no ACK/NAKs at all could cause problems

Verify flow control is progressing– Sometimes, there seems to be a stall on the link and customers assume

it’s the PCIe block’s fault– Being able to verify flow control credits are being returned and the

ACK/NAK status may be insightful

Verify integrity of packet– Many cases have come up where customer believes block is

“manipulating packets”– Usually, this turns out to be a software issue– Using this design will allow customer to verify packet integrity right before

it goes into MGTs or as it comes out of MGTs


83Agenda



84Table of Contents

What ALDEC® Simulators Can Do?ALDEC® XILINX® Support TimelineALDEC® and XILINX®

Starting Active-HDL™ from ISE™Xilinx Tools in Active-HDL FlowStarting Coregen from Active-HDL


85What ALDEC Simulators Can Do?

Simulate Xilinx ISE designs using any combination of source languages: VHDL, Verilog, System Verilog, SystemC, EDIF, etc.Include pre-compiled Xilinx libraries (including SecureIP)Support latest Xilinx families: Spartan-6 and Virtex-6Support all popular IP Cores (PCIe, RocketIO Transceivers, SERDES, EMAC/TEMAC, etc.)Support SmartModels/SWIFTCan handle DSP co-simulation (MATLAB® and Simulink® interfaces )Simulate structural MPU models from EDKSupport all legacy Xilinx families (all versions of Virtex, Spartan, Coolrunner, XC4000, XC9500)


86Aldec Active-HDL 8.2™

Common-Kernel Mixed Language Simulator

Languages: VHDL, Verilog®, SystemVerilog (Design & Assertions), SystemC & EDIF

HDL Design Tools: Design Entry, Design Creation, Code2Graphics™, Block and State Diagram, Waveform Editor, stimulus generation, code auto-complete and language templates, scripting, legacy design support.

Design Flow Manager: use popular third-party tools throughout the design flow within the same FPGA environment.

Debugging: Code execution tracing, Waveform/Compare, Memory Viewer, Xtrace, Advanced Dataflow and Profiler.

Coverage: Code Coverage, Toggle & Functional Coverage.

Additional Interfaces: MATLAB® and Simulink® co-simulation interfaces, Zuken CADSTAR PCB Design interface

Assertion and Coverage(OPTION) SystemVerilog, PSL & OVA support; dedicated Assertion Viewer, assertion coverage, assertion breakpoints.

Users who need LinuxUsers who need Linux--based simulation based simulation tools or are not interested in graphical tools or are not interested in graphical entry may want to try entry may want to try RivieraRiviera--PROPROline of highline of high--performance simulators.performance simulators.


87

Q4 Q1 Q2 Q3 Q4 Q1 Q2

201020092008Q3

ALDEC® XILINX® Support Timeline

Aldec and Xilinx sign Cooperation Agreement

Over 30 mutual customers confirm working SecureIP

ISE Design Suite 12.1 with SecureIP sourcessupporting Aldec keys and Library Compiler support for Aldec

Second release of Aldec simulators with SecureIP forSpartan-6 and Virtex-6 (Active-HDL 8.2 and Riviera-PRO 2009.06SR1)

First Aldec simulators with SecureIP support(Active-HDL 8.1sp2 and Riviera-PRO 2009.02)


88ALDEC® and XILINX®

Aldec was the creator of design entry, project management and gate-level simulation tools in Xilinx Foundation Classic suites (our tools can import Foundation schematic designs and turn them into HDL designs)

Aldec tools support design flows including all versions of Xilinx tools, starting from Xilinx Alliance/Foundation Classic up to Xilinx ISE 11.2

Xilinx precompiled HDL libraries and SecureIP are currently available directly from Aldec

Xilinx library sources (including SecureIP) will compile for Aldec simulators using ‘compxlib’ in ISE Design Suite 12.1


89Starting Active-HDL from ISE

With very little effort, it is possible to start Active-HDL simulator from Xilinx ISE Project Navigator to run behavioral and timing simulations

The setup procedure is described in the application note “Starting Active-HDL as the Default Simulator in Xilinx ISE” available on ALDEC website (follow the link below or do the search on the title of the document)

The application note provides detailed, easy to follow instructions and the link to multimedia presentation demonstrating the use of theinterface

http://support.aldec.com/KnowledgeBase/Article.aspx?aid=000771http://support.aldec.com/KnowledgeBase/Article.aspx?aid=000771


90Xilinx Tools in Active-HDL Flow

Active-HDL is equipped with Design Flow Manager that, when configured properly, allows quick start of Xilinx or third partysynthesis tools and ISE implementation engine

Options required to start each application can be configured in user-friendly GUI windows

Simulation files are imported back to Active-HDL

Applications like Coregen, Constraints Editor, STA, ChipScope Pro can be stared from the Flow

Detailed instructions are available in the “Using Active-HDL with Xilinx ISE” application note (link listed below)

http://support.aldec.com/KnowledgeBase/Article.aspx?aid=000640http://support.aldec.com/KnowledgeBase/Article.aspx?aid=000640


91

STARTING COREGEN FROM ACTIVE-HDL


92Flow Enabling

To access Coregen in Xilinx ISE 11.2 from Active-HDL, user has to enable Design Flow Manager in Preferences window accessible fromthe Tools menu in Active-HDL


93Flow Tools Configuration

After changes of installed tool versions users should update locations of synthesis and implementation tools in the Tools | Preferences window, Environment | Flows | Integrated Tools category (both version and location of each tool can be modified)


94Tool/Version Selection

When Active-HDL installation is up-to-date, the latest Xilinx tool versions should be visible in flow configuration


95Showing the Flow Manager

Once enabled, Design Flow Manager can be displayed in Active-HDL using main toolbar button:

Displayed Design Flow Manager looks like this:


96Flow Modifications

If quick change of tool version or default Xilinx family is needed, it is possible in Flow Configuration Settings window displayed after clicking Flow Settings button


97Starting Coregen

To start Coregen from the Flow, user should click Tools button in the Design Flow Manager and select CoreGen & Architecture Wizard button in the popup window (see the illustration below)


98Coregen Interface Window

Active-HDL provides interface window that allows starting Coregen and managing generated cores

Users should adjust basic options before clicking Run Core Generatorbutton (see illustration below)


99Selecting GTP in Coregen

Once Xilinx CORE Generator window shows, user should follow the typical steps to generate Spartan-6 GTP core

Be sure to select correct chip and save log before closing this window


100Importing Files to Active-HDL

After Xilinx CORE Generator window closing, generated IP will show up in the Active-HDL interface window

Users should select the IP, then click Add IP Core File to transfer files needed for simulation back to Active-HDL


101Other Options for PCIe Simulation

HDL models of PCIe are accurate, but slow

For increased simulation speed, Bus Functional Models (BFM) of PCIe are available as Verification IP

Aldec simulators are officially supported by many respectable IPvendors, including providers of PCIe cores:

– Northwest Logic with its “PCI Express Verification Suite”

– nSys with its “PCI Express nVS” (golden standard in PCIe verification)

To get more info, visit IP Products page in the Products section of our website

http://www.aldec.com/products/ipcoreshttp://www.aldec.com/products/ipcores


102Simulation

No matter if you are using VHDL, Verilog or both: our simulators will be able to handle your designIf you need to get result quickly – you have speedExpect performance matching leading competitors, but in easier to use packageIf you have to investigate some design issues –you have extensive debugging capabilities Happy Simulating!Happy Simulating!

virtex6 pcie gen2 xfest 2009 v1 1 - eetrend.com

Documents