platform development guide - xilinx€¦ · sdaccel platform development guide 5 ug1164 (v2017.4)...

SDAccel EnvironmentPlatform Development Guide

UG1164 (v2017.4) January 26, 2018

http://www.xilinx.com

Send Feedback

https://www.xilinx.com/about/feedback.html?docType=User_Guides&docId=UG1164&Title=SDAccel%20Environment%20Platform%20Development%20Guide&releaseVersion=2017.4&docPage=2

Revision HistoryThe following table shows the revision history for this document.

Date Version Revision01/26/2018 2017.4 Minor editorial changes.

01/15/2018 2017.4 Complete restructure of documentcontent.Added reference design files forKCU1500 and VCU1525. See thefollowing sections:

• Appendix A: Using the KCU1500Reference Design

• Appendix B: Using the VCU1525Reference Design

SDAccel Platform Development Guide 3UG1164 (v2017.4) January 26, 2018 www.xilinx.com [placeholder text]Send Feedback

https://www.xilinx.com


Revision History




Table of ContentsRevision History...............................................................................................................3

Chapter 1: Overview......................................................................................................9Introduction................................................................................................................................. 9Using Available Xilinx Platforms and Reference Designs.....................................................10Platform Overview.....................................................................................................................11

Chapter 2: Using Dynamically Implemented Platforms.........................15Understanding Partial Reconfiguration................................................................................. 16Understanding the Static and Dynamic Regions.................................................................. 17Populating the Dynamic Region with Kernels....................................................................... 18Example Partial Reconfiguration Platform Design............................................................... 18

Chapter 3: Device Planning..................................................................................... 21Device Selection.........................................................................................................................21Using Platform Board Files...................................................................................................... 22

Chapter 4: Creating the Logic Design using IP Integrator....................25Creating the Top-Level Design................................................................................................ 25Isolating the Dynamic Region for Partial Reconfiguration.................................................. 27Designing the Static Region Logic...........................................................................................27Designing the Dynamic Region............................................................................................... 41Assembling the Vivado PR Project.......................................................................................... 49

Chapter 5: Configuring Platform Properties................................................. 57Setting the Platform Identification Property (PFM_NAME)..................................................57Configuring Platform Interface Properties............................................................................ 58Setting Platform Configuration Properties............................................................................ 63Using the IP Cache.................................................................................................................... 67

Chapter 6: Creating Design Constraints.......................................................... 69I/O and Clock Planning.............................................................................................................69Floorplanning.............................................................................................................................69




Partitioning the Timing Constraints........................................................................................71

Chapter 7: Implementing the Hardware Platform Design....................73Simulating the Design.............................................................................................................. 73Implementation and Timing Validation..................................................................................73

Chapter 8: Generating a DSA..................................................................................75Generating the DSA File........................................................................................................... 75Validating the DSA.....................................................................................................................76Using Older DSA Versions........................................................................................................ 76

Chapter 9: Software Platform Specification.................................................. 77Using the OpenCL Hardware Abstraction Layer (XCL HAL) Driver API...............................77Using the OpenCL Runtime Flow............................................................................................ 85PCIe Kernel Driver for Management Physical Function....................................................... 86User Physical Function..............................................................................................................88Linux Sys FileSystem Nodes..................................................................................................... 91Building Software Stack............................................................................................................93

Chapter 10: Assembling the Platform...............................................................95

Appendix A: Using the KCU1500 Reference Design................................... 97Platform Characteristics for KCU1500.................................................................................... 97Floorplan for KCU1500..............................................................................................................97XDMA IP Configuration for KCU1500...................................................................................... 98SDx Memory Subsystem IP Configuration for KCU1500...................................................... 98

Appendix B: Using the VCU1525 Reference Design..................................101Platform Characteristics for VCU1525.................................................................................. 101Floorplan for VCU1525............................................................................................................102XDMA IP Configuration for VCU1525.................................................................................... 102SDx Memory Subsystem IP Configuration for VCU1525.................................................... 103

Appendix C: New Static Region Features...................................................... 105Embedded Scheduler..............................................................................................................105Management MicroBlaze....................................................................................................... 107Host Interrupt Support...........................................................................................................108Mailbox..................................................................................................................................... 109Frequency Counters for Scalable Clocks.............................................................................. 110




Appendix D: Additional Resources and Legal Notices........................... 111References................................................................................................................................111Please Read: Important Legal Notices................................................................................. 112




Chapter 1

Overview

IntroductionThe Xilinx® SDAccel™ Development Environment lets you compile OpenCL™ C, C, C++, andRTL kernels to run on Xilinx® programmable platforms. The programmable platform is composedof the SDAccel Xilinx Open Code Compiler (XOCC), a Device Support Archive (DSA) whichdescribes the hardware platform, a software platform, an accelerator board, and the SDAccelOpenCL runtime. The XOCC is a command line compiler that takes user source code and runs itthrough the Xilinx implementation tools to generate the bitstream and other files that are neededto program the FPGA-based accelerator boards. It supports kernels expressed in OpenCL C, C++,and RTL (SystemVerilog, Verilog, or VHDL).

The majority of this document describes the steps required to create platform designs for usewith the SDAccel Development Environment. Platform creation involves configuring thenecessary software drivers and runtime software coupled with the hardware platform design andDSA. The hardware platform design provides the physical interface information required for thesoftware kernels to interact with the printed circuit board. These interfaces are captured usingXilinx IP within the Vivado IP integrator environment. Special properties are used to define theboundary of the physical interfaces with the kernel logic coming from the SDx™ Environment.This “shell” is then implemented inside the FPGA with the results stored and reused when eachkernel application is implemented. All of the hardware design components are then packagedinto the Device Support Archive (DSA) for use with the SDx Environment.

This document is written from the perspective of a hardware designer, using terminologycommon to the Vivado® Design Suite. An overview of the OpenCL Design flow written from theperspective of a software developer is provided in the SDAccel Environment User Guide (UG1023).


https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug1023-sdaccel-environment-user-guide.pdf



Using Available Xilinx Platforms andReference DesignsThe process of creating a hardware platform DSA for use with the SDAccel environment can betime consuming, depending on the device selected and the interfaces involved. If you are justlearning the SDAccel solution, or are interested in experimenting with your own OpenCL kernels,you should consider using one of the standard hardware platforms offered by Xilinx. Refer to theSDx Environments Release Notes, Installation, and Licensing Guide (UG1238) for a list of standardplatforms supported by the tool.

In addition, developing your own hardware platform is much simpler when starting with anexisting hardware platform reference design and modifying it as needed to meet your designobjectives.

Refer to the following for information about Xilinx-supplied acceleration reference designs:

• Appendix A: Using the KCU1500 Reference Design

• Appendix B: Using the VCU1525 Reference Design

Chapter 1: Overview


http://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug1238-sdx-rnil.pdf



Platform OverviewHardware PlatformsThe hardware platform consists of a Vivado® Design Suite IP integrator block design with all therequired board interfaces configured and connected to the device I/Os and the dynamicallyprogrammable region. The dynamically programmable region defines the programmable devicelogic partition that accepts the software kernels from the SDAccel™ Development Environment.It contains DDR memeory interfaces and AXI interconnect logic that is dynamically configuredand implemented along with the SDx kernel logic each time an application is run. There is also afixed static logic partition that contains the board interface logic. The static partition is not re-implmented when SDx™ applications are run. Some refer to it as the “shell,” since it describes theboundary of the physical board with the kernel logic.

SDAccel platforms require the device to remain up and running on a remote host whileapplications are downloaded on to it. SDAccel DSAs use partial reconfiguration (PR) technologyto enable compiled binary downloads to the accelerator device while it remains online and linkedto the PCIe® bus of the host. The static logic is intended to keep the device alive and to controlreconfiguration and debugging of the device.

When an SDx Environment kernel application is run, a part of the hardware platform design isalso implemented along with the incoming kernel logic. This helps ensure efficient use of deviceresources and increased performance. This area of the design is referred to as the “dynamicregion.” Each time a kernel application is run, partial reconfiguration technology is applied toreconfigure the dynamic region while the static region remains intact and online.

The Xilinx-supplied reference designs each have four DDR memory interfaces coupled withinterconnect logic. They both utilize the SDx Memory Subsystem (MSS) IP. This IP is unique forSDAccel platforms. It contains multiple DDR IP, coupled with the appropriate interconnect IP. Asthe dynamic region is being implemented when kernels are applied, the IP automatically trimsaway any of the unused DDR interfaces that reside in the dynamic region, allowing for more ofthe device to be available for kernel logic. The reduced logic should reduce runtimes as well.

The basic requirements of an SDAccel hardware platform are as follows:

• The PR design project: Proper configuration of the static and dynamic regions of the platformdesign for use with partial reconfiguration, as described in this document.

• Software compatibility: The memory-mapped IP cores are configured to use address offsetsand ranges that are compatible with the hardware abstraction Layer (HAL) driver providedwith the SDx Environments installation. The kernel mode drivers for the Xilinx DMAsubsystem for PCI Express (PCIe) are likewise provided with the SDx Environmentsinstallation.

Chapter 1: Overview




• Global memory access: DDR4 SDRAM global memory is accessible to both the host and userkernels using AXI4 memory-mapped connectivity provided by AXI SmartConnect IP.

• Platform DSA metadata: The necessary Vivado project properties are applied to identify theplatform, define platform interfaces, and to control platform configuration capabilities throughXOCC commands.

Software PlatformsThe software platform contains the kernel driver and the runtime configuration from the host

processor to the FPGA through the PCIe interface. The Xilinx® SDAccel™ runtime software islayered on top of a common low-level software interface called the Hardware Abstraction Layer(HAL).The HAL provides a unified abstraction for the two physical functions (PFs): User PF andMgmt PF. The xocl kernel driver binds to User PF and the xclmgmt kernel driver binds to MgmtPF.

The following image shows the layers of the software platform:

• The Linux Kernel Mode DMA Driver, xocl

• The Linux Kernel Mode DMA Driver, xclmgmt

• The Xilinx SDAccel Runtime Software

Figure 1: Software Platform Layers

PCIe Configuration

HAL

XOCL XCLMGMT

DSAUSER PF MGMT PF

X20237-012518X20237-012518

The details for the software architecture are defined in Chapter 9: Software PlatformSpecification.

Chapter 1: Overview




Platform Creation Process FlowThe basic process for creating an SDAccel platform includes:

Device and Design Planning

Create Block Designs in IP

Integrator

Define Static Logic

Define Dynamic Region

Assemble the PR Project

Assign Platform Properties

Implement Platform Design

Write DSA and Build Platform

· Plan board interface I/Os and clocks.· Plan SSI strategy – logic partitions, GTs, memory interfaces, clocking, etc.· Examine available device resources against anticipated needs.

· Define the top-level logic static region hierarchy.· Isolate the Dynamic region in a separate block design.

· Instantiate the SDx Memory Subsystem, which is automatically configured by the project’s Board file.

· Configure the Interconnect IP.· Insert debug logic.

· Use the PR capabilities in IP integrator to create the dynamic region PR partition· Complete the logic design and generate the BD

· Define platform identification properties· Define platform interface properties· Define platform configuration properties

· Define and partition the timing constraints.· Floorplan the static logic to maximize device resources for the programmable

region.· Implement the design successfully.

· Configure PCIe, DMA, and board interface IP.· Insert logic required to manage partial reconfiguration and maintain device state.· Insert debug logic.

· Write and validate the DSA using write_dsa and validate_dsa.· Build a platform in the SDx Environment.· Validate platform using a test kernel.

X20190-011118

Terminology

Hardware Platform – Represents the physical board interface and the dynamic region. Thehardware platform consists of a Vivado IP integrator design with a target device and all interfaceIPs configured and connected to the device I/Os.

Chapter 1: Overview




Software Platform – Consists of the runtime, and drivers that are needed to enableinteraction with the hardware platform.

Kernel – Either the OpenCL, C, C++, or RTL modules applied to the platform.

Dynamically Implemented Platform – The type of platform typically used with Xilinx®SDAccel™ where partial reconfiguration techniques are applied. This enables a portion of thedesign to remain static and not be re-implemented allowing the device to remain alive while thedynamic region is implemented.

Dynamic Region – Describes the partition that accepts the hardware functions from the SDxenvironment. The term also describes the physical resources available on the programmabledevice to implement the hardware functions.

Static Region – Represents the fixed logic portion of the programmable device that managesthe design state before, during, and after partial reconfiguration of the dynamic region. This istypically used with SDAccel platforms only, in order to keep the device up and running whilevarious kernels are applied to the platform. This logic is not re-implemented with the dynamicregion. Special parameters and design considerations are required for signals that cross betweenthe static region and the dynamic region.

Device Support Archive (DSA) – Contains all of the design and metadata needed for anSDx hardware function to interact with the physical design. It is the output product of thehardware platform design process described in this guide.

Chapter 1: Overview




Chapter 2

Using Dynamically ImplementedPlatforms

In the following figure, you can see that the PR method for hardware platform design includes astatic logic region and a dynamic region containing board interface logic and DDR memory.Implementing the dynamic region logic along with the kernel logic from the Xilinx® SDAccel™Environment enables the most effective use of the programmable device resources across theentire design and can help maximize the performance of the overall design. A partial bit file iscreated for the entire dynamic region and is used to reconfigure the active device.

The smaller static region contains the logic needed to keep the hardware platform online andconnected while waiting to be reconfigured with the logic in the dynamic region. Typically withthe PR method, the Xilinx DMA Subsystem for PCI Express, basic control interfaces, and clocksources are contained within the “static region”. This region is floorplanned to use as little of thedevice area as possible in order to maximize the available area for kernel resources. The logichierarchy must be designed to separate the static region logic from the dynamic region, in theVivado IP integrator block design. For more information, see Chapter 4: Creating the LogicDesign using IP Integrator.

Figure 2: PR Platform Design Structure

Dynamic region

SDx Memory Subsystem IP

Static region

PCIe XDMA

Clocking & reset

Peripherals

SmartConnectautomatic topology

DDR4 IP

DDR4 IP

DDR4 IP

DDR4 IP

User kernel

User kernel

X20205-011218




Understanding Partial ReconfigurationThe hardware platform design process requires familiarity with the underlying design flow used

in the Xilinx® SDAccel™ Development Environment. The SDAccel programming flow uses thepartial reconfiguration capabilities unique to Xilinx to enable downloading partial bitstreams intothe programmable device, or hardware platform while it remains online and connected to thehost computer's PCIe bus. Refer to the Vivado Design Suite User Guide: Partial Reconfiguration(UG909) for more information about partial reconfiguration.

The partial reconfiguration flow requires the use of a decoupler IP core or compatible logicstructure to hold the design in a safe state while the device is partially reconfigured with thedynamic region logic. Refer to Using the Partial Reconfiguration Decoupler IP.

The partial reconfiguration flow also requires several modifications to the design includingfloorplanning. Refer to Floorplanning.

Programming the Device using Partial Reconfiguration

The FPGA is a peripheral on a PCIe bus, and the system host configures the FPGA through thePCIe connection.

Power-On Reset Programming

After power-on reset of the FPGA, the entire device is configured with a full bitstream (.bit) filefrom flash memory. In this initial state, the host can access DDR4 global memory, but no userkernels are present.

SDAccel Kernel Programming

As each kernel is executed on the FPGA, the SDAccel software controls the loading of partialbitstreams for the dynamic region. Partial bitstreams are fully self-contained. All addressing,header, and footer details are contained within these bitstreams, just as they would be for fullconfiguration bitstreams. Dedicated PR features such as per-frame CRC checks (to ensurebitstream integrity) and automatic initialization (so the region starts in a known state) areavailable, as well as full bitstream features such as encryption and compression.

As part of the static region of the design, the internal Partial Reconfiguration control circuitryoperates without interruption throughout the Partial Reconfiguration process. This allows theportion of the FPGA logic not being reconfigured to continue functioning while thereconfigurable portion is modified. The Decoupler IP or compatible logic structure is used tomaintain the board state while it is partially reconfigured. Refer to Using the PartialReconfiguration Decoupler IP for more information.

Chapter 2: Using Dynamically Implemented Platforms


https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug909-vivado-partial-reconfiguration.pdf



The partial bitstreams are retrieved and then delivered through the PCIe connection and onto theFPGA configuration logic using an Internal Configuration Access Port (ICAP). For moreinformation on the use of ICAP, refer to the UltraScale Architecture Configuration User Guide(UG570).

The DONE pin pulls LOW when reconfiguration begins and pulls HIGH again when partialreconfiguration successfully completes. For this reason, do not use the DONE pin to directlygenerate PCIe reset. See the Vivado Design Suite User Guide: Partial Reconfiguration (UG909) formore information.

Understanding the Static and DynamicRegionsThe static region is the area of the hardware design that is fixed on the hardware platform. Itdoes not get implemented with the dynamic region, and is not affected by the PR flow.Remember the static region logic is used to keep the board in a stable state and to manage thedevice reconfiguration process. As shown in the following figure, it is composed of the mainconnectivity framework IP: the PCIe Endpoint, DMA Controller and supporting logic. There areother smaller interface IP cores that are not shown in the following figure, but that are part of thestatic region, as described in this chapter.

The Dynamic region is reconfigured and implemented along with the user kernels as the SDxdesign is processed. The Dynamic region logic will not contain any user kernels in its initial stateas shown in the following figure. It will be instantiated when the SDx design is processed.

Figure 3: Static Region and Dynamic Region Structure

Dynamic region

SDx Memory Subsystem IP

Static region

PCIe XDMA

Clocking & reset

Peripherals

SmartConnectautomatic topology

DDR4 IP

DDR4 IP

DDR4 IP

DDR4 IP

X20208-011518



https://www.xilinx.com/cgi-bin/docs/ndoc?t=user_guides;d=ug570-ultrascale-configuration.pdf




Populating the Dynamic Region withKernelsThe dynamic region is a container for compiled kernels from the SDx™ environment. It usuallycontains the SDx Memory Subsystem along with its default configuration of AXI SmartConnectand the memory controller interfaces. The SDx kernels are instantiated and automaticallyconfigured within the block when the SDx environment processes the design. The dynamicregion logic is implemented along with the kernel logic each time the design is implemented. The

Xilinx® SDAccel™ XOCC command line input determines the number of kernels in the dynamicregion as well as their connectivity to global memory.

The dynamic region has an AXI slave interface which is used to send commands to the kernels,and an AXI slave interface which connects the XDMA to global memory. One or more clock and areset signals are also wired to the dynamic region.

You can customize the address and data bit width for the AXI master interface, but you shouldutilize the maximum value that the memory controller can support in order to maximizebandwidth and minimize data width converters on the datapath.

Example Partial Reconfiguration PlatformDesignThe Xilinx supplied Acceleration platforms are designed for partial reconfiguration and use adynamic region containing the SDx Memory Subsystem IP which can instantiate or map to theboard’s maximum number of memory controllers, along with the necessary interconnect logic toenable them. This logic structure allocates most of the device’s fabric resources to the kernels.Only the logic necessary to keep the link active and device operational, primarily the Xilinx DMAsubsystem for PCI Express®, basic control interfaces, and clock sources, are contained within astatic “base region” that cannot be used for kernel resources.

To make these additional fabric resources available to implemented kernel logic, the data pathSmartConnect IP and multiple DDR4 memory controller IP are contained within the partiallyreconfigurable “dynamic region” level of hierarchy. When the SDAccel™ System Compilerexecutes, kernel logic is placed and routed among these necessary resources, and a partialbitstream containing them is produced and included in the compiled binary.

IMPORTANT!: An outcome of locating the DDR4 memory controllers in the reconfigurable region isthat DDR4 global memory content is lost when a new partial bitstream is downloaded.





The following figure shows the IP integrator block diagram view of the top-level of the XilinxKCU1500 platform, colored to show the static base logical hierarchy in dark blue, and thedynamic region in yellow.

Figure 4: IP Integrator Top-Level View of Platform, Showing Base and ExpandedRegions

Correspondingly, the following figure shows the Vivado® device view of the implementedKCU1500 platform, where utilized and floorplanned static base region logic is highlighted yellow,and the utilized expanded region logic is highlighted green.





Figure 5: Device View of Implemented Platform, Showing Base and Expanded Regions

Notice the prominent rectangular regions of DDR4 Memory Controller IP instances, which existby default when the DSA is created. When the SDAccel System Compiler executes, only thememory controller instances which are mapped from user kernels are retained.





Chapter 3

Device Planning

Device SelectionCareful planning should be applied when selecting the device for your hardware platform. Typicalconcerns such as device features, board compatibility, cost, power, size, etc. need to beconsidered. When designing a hardware platform design additional thought should be applied tothe following to ensure effective use of the platform.

Device Size

The device needs to accommodate both the hardware platform logic and the kernel logic beingapplied to it. The anticipated logical size of the hardware function or kernels being used mayimpact the feasibility of the device. Consider that the kernels used may not always be optimizedfor logic size, depending on the pragmas and parameters used in SDx. Kernel sizes may end uplarger than anticipated. Therefore, additional room in the device may enable more timelyexperimentation, logic analysis, and optimization.

I/O Interface and Hard Block Locations

To assure the best possible performance of the overall design, you should carefully plan theassignment of the required device interfaces. You should study and understand the devicepackage footprint, the package pin assignments, and the internal die layout of the programmabledevice to determine the best I/Os to use for the various hardware interfaces, especially theexternal processor and memory interfaces.

You should plan the intended logic flow of the hardware platform, while paying attention to clockdistribution, the expected logic size of the interfaces, and the hardened device resources such asPCIe, BRAMs, and DSPs. This is true not only for the static region logic, but also for the dynamicregion logic. You should focus on maximizing the available resources for the dynamic region logic.

This is especially important when using Stacked Silicon Interconnect (SSI) devices.

As you define the hardware platform logic in the Vivado Design Suite, you can use the I/Oplanning features to assign the I/O and clock constraints. You can also use floorplanningtechniques to ensure consistent results and optimal performance of the hardware platformdesign. Refer to the




Vivado Design Suite User Guide: I/O and Clock Planning (UG899) and the Vivado Design Suite UserGuide: Design Analysis and Closure Techniques (UG906) for more information.

Working with SSI Devices

Many of the Xilinx devices take advantage of Stacked Silicon Interconnect (SSI) technology. Thiseffectively creates multiple die areas (SLRs) within the same device. Special care needs to be usedwhen designing hardware platforms using SSI devices. Floorplanning techniques need to beapplied effectively. Clocking and I/O interfaces should be properly planned to ensure optimalconnectivity to the kernel logic. The location of the kernels is important to ensure they will fitwithin the specified SLRs and map to memory resources within them.

Using Platform Board FilesOptionally, a Platform Board (Board) file can be created to support the hardware platform design.A Board file is required if using the SDx Memory Subsystem IP, which is recommended and isbeing used in all new platforms.

The Board file defines the component interfaces on the board, the I/O interfaces between thecomponents, and the configuration options for interface IP in the block design. As the blockdesign is being assembled in the Vivado IP integrator, a Board file facilitates the use ofConnectivity Assistance, which can ease the effort to create a platform design. Also, if variants ofthe hardware platform are planned, consider using a Board file to enable faster configuration ofIP interfaces.

When using a Platform Board file, many of the Platform Properties that are required to be set inorder to create a DSA are defined for you in the file.

The Vivado Design Suite includes board interface files for several Xilinx supplied referenceboards that are delivered as part of the tool. You can also get predefined boards from supportedthird-party providers.

Creating a Platform Board FileThe easiest way to create a Platform Board file is to copy one supplied by Xilinx and modify it foryour needs. For more information on the Board file definition and format and defining your ownboard interface files, refer to this link in the Vivado Design Suite User Guide: System-Level DesignEntry (UG895).

Chapter 3: Device Planning


https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug899-vivado-io-clock-planning.pdf

https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug906-vivado-design-analysis.pdf

http://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug895-vivado-system-level-design-entry.pdf;a=xBoardInterfaceFile

https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug895-vivado-system-level-design-entry.pdf



Enabling Connectivity Assistance in IP IntegratorWhen you define a new project in the Vivado tool you can select a target Xilinx device, or aplatform board. A platform board file facilitates the use of the Board Interface ConfigurationOptions tab and Connectivity Assistance in the IP integrator that enables quick variations of thehardware platform. It simplifies the process of adjusting IP configurations and makingconnections in the Vivado IP integrator block design.

When you use the platform board flow, the Vivado IP integrator shows all the interfaces to theboard in a separate window called the Board window, as shown in the following figure. Whenyou use this window to select the desired components, you can easily connect the IP in your IPintegrator block design to the board components of your choosing. All the interface connectionsand I/O constraints are automatically generated as a part of using the connectivity assistancefeatures in IP integrator.

Figure 6: Board Interface Configuration Options

For more information about using the board view and connectivity assistance, see the VivadoDesign Suite User Guide: Designing IP Subsystems using IP Integrator (UG994).



https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug994-vivado-ip-subsystems.pdf



Chapter 4

Creating the Logic Design using IPIntegrator

The Vivado® IP integrator provides flexibility for capturing the overall structure of your design.Levels of hierarchy can be created for easier design navigation and comprehension. The logichierarchy of the hardware platform design must be carefully planned in order to properly supportthe partial reconfiguration flow (PR). Hierarchy is used to isolate the dynamic region of thedesign, and to define the static base region and other elements of the hardware platform such ascomponents on the platform board or IP cores in the subsystem design. Proper levels ofhierarchy need to be established within the Vivado IP integrator block design.

Creating the Top-Level DesignWhen using a PR hardware platform, it is required to place the static and dynamic regions on thetop-level of the design, as shown in the following figure. This allows you to configure thedynamic region hierarchy for partial reconfiguration as well as define the interfaces. The dynamicregion logic must be isolated in a separate IP integrator block diagram, as described in the Isolating the Dynamic Region for Partial Reconfiguration. The rest of the design is considered thestatic region for the PR hardware platform design. To make the connectivity between the staticand dynamic regions more visible, the hierarchy is created to isolate the static and dynamicregions of the design. To simplify the view, the Show interface connections only option can beused. The following figures show these views for the Xilinx KCU1500 platform.




Figure 7: Top-Level IP Integrator Block Diagram - Default View with Interfaces andWires

Figure 8: Top-Level IP Integrator Block Diagram - Interface Connections Only View

Chapter 4: Creating the Logic Design using IP Integrator




Isolating the Dynamic Region for PartialReconfigurationIn order to support partial reconfiguration, the entire dynamic region logic design must becreated using a separate IP integrator block design separate from the top-level static logic design.The hierarchical level representing the dynamic region as seen above is created automaticallyusing the Vivado® partial reconfiguration capabilities, as described in Designing the DynamicRegion and Assembling the Vivado PR Project.

Designing the Static Region LogicCreating the Top-Level Static DesignThe example described in Chapter 4: Creating the Logic Design using IP Integrator shows the endresult of creating the hardware platform logic design using IP integrator. The process that is usedto create the PR project dictates how the design is assembled. The design needs to start with atop-level BD for the hardware platform design containing only the static region logic andoptionally any I/O buffers that will connect to the dynamic region, as shown in the followingfigure. The I/O buffers can be added later when the dynamic region partition is added to thedesign.





Figure 9: Creating Top-level Static Logic and I/Os

Note: Dynamic region is not contained in the design yet. The hierarchy wil be created automatically whenthe Create Platform command is used, as described in Assembling the Vivado PR Project.

Host ConnectivityThe platform implements connectivity to the host computer using the DMA Subsystem for PCIExpress IP, which contains a scatter-gather DMA and a PCIe 3.x integrated block. For bestperformance, the PCIe integrated block can negotiate a link up to:

• Gen3x8 connectivity, or 8 GT/s, for KCU1500

• Gen3x16 connectivity, or 16 GT/s, for VCU1525





The following figure shows the connectivity from the XDMA IP master, through thedma_pf_demux to produce the management and user paths, to the two AXI Interconnectinstances which connect to the relevant peripherals on each path.

Figure 10: AXI4-Lite Control Path Connectivity

• Host access using XDMA IP is the control path master of the platform. The XDMA IP isconfigured for two physical functions, which are then split into the two distinct low-speed,32-bit AXI-Lite paths using the dma_pf_demux instance in the static base region. Amanagement path corresponds to system peripherals, while a user path controls user kernelsand associated peripherals. Two separate drivers can attach to the 2 physical functions – themanagement driver controls the management physical function (PF1) and the xdma kerneldriver controls the user physical function (PF0). In a datacenter environment, the two driverscould be in different VMs with the management driver in higher privilege environment. Theuser will, therefore, not be able to access the management space in the static region of theplatform that contain critical functions like bitstream download and flash programming. Themanagement interconnect, however, does have access to IPs in user space because it is one ofthe masters of the user space interconnect.

Figure 11: Host and DMA with 2 AXI Interconnects

XDMAManagement Space (PF1) User Space (PF0)

AXI LiteInterconnect

AXI LiteInterconnect

Host

PCIe Gen3x16

MgmtDriver

UserDriver

P

FPGA Static Region FPGA PR Region

Control AXI Lite Interface

Datapath Full

AXI4 InterfacePF_DEMUX

IPs IPs

X20209-011518





• The management path driven by the dma_pf_demux S_AXI_MGNTPF interface provides theXDMA IP master with memory-mapped access to peripherals in both the static base regionsusing a 1x32 AXI Interconnect IP and a downstream 1x4 AXI Interconnect IP in thereconfigurable expanded region. Those two AXI Interconnect instances are isolated from oneanother by partial reconfiguration isolation logic (see Configuring the Dynamic Region forPartial Configuration for details). This is a brief description of the following:

○ AXI Firewall Control User: AXI Firewall IP is designed to protect AXI DMA from hangs andprotocol violations downstream of it that may otherwise lead to host crashes. Thisinstantiation of IP provides protection to the XDMA from downstream peripherals on UserControl AXI-Lite network

○ AXI Firewall Control: AXI Firewall IP provides protection to the Management Control AXILite Interface of XDMA from downstream peripherals on Management AXI network

○ AXI Firewall Data: AXI Firewall IP provides protection to the full AXI4 datapath interface ofXDMA from downstream IPs on datapath.

○ Board Management AXI MB Interrupts: 2 GPIOs through which enables Microblaze tosend interrupts back to host.

○ Board Management AXI GPIO MB Control: GPIO IP controlled by host to reset MB whileloading firmware

○ HWICAP: AXI HWICAP IP that enables write (and read) of the FPGA configurationmemory through the Internal Configuration Access Port (ICAP). This IP is used for PRbitstream download through PCIe link.

○ AXI IIC controller: AXI IIC IP that allows communication with components present on theI2C bus on the board e.g. voltage regulators, fan controller. This IP is not essential forSDAccel functionality, but is essential for board management and is therefore specific tothe board being used.

○ Kernel Clock Wizard 1: Clock Wizard IP that provides the first Kernel clock through anMMCM in the IP

○ Kernel Clock Wizard 2: Clock Wizard IP that provides the first Kernel clock through theMMCM in the IP

○ Embedded Scheduler BRAM controller: AXI BRAM controller that contains firmware thatruns on Microblaze scheduler

○ DDR Calibration Status: AXI GPIO IP that enables monitoring of calibration status of allDDR controller signals ANDed together

○ DDR Memory AXI Lite interface: Provides access to DDR AXI Lite interfaces

○ Flash Programmer: AXI Quad SPI IP that can be used to program the flash chips thatcontain the boot image.

○ Frequency Counter: Dual input frequency counter that measures the outputs clocks fromthe 2 MMCMs





○ Gate PR Reset Controller: AXI GPIO IP that provides isolation control during PR and canalso reset kernels in dynamic region

○ Mailbox: Mailbox IP interface to AXI interconnect in the management space. Mailbox IP ispresent to provide a means of secure communication through hardware FIFOs betweendrivers attached to PF1 and PF0

○ System management Wizard: Provides access to SYSMON IP block that is used fortemperature and voltage readback

○ Scratch Pad RAM: AXI BRAM controller that can be used for storing data

• The user path driven by the dma_pf_demux S_AXI_USERPF interface provides the XDMA IPmaster with memory-mapped access to peripherals in both the static base region using a 2x3AXI Interconnect IP and a downstream 1x3 AXI Interconnect IP in the reconfigurableexpanded region. Those two AXI Interconnect instances are isolated from one another bypartial reconfiguration isolation logic (see Configuring the Dynamic Region for PartialConfiguration for details). The second master interface on the static base region 2x3 AXIInterconnect IP allows the management path to access all peripherals on the user path.

○ Debug Bridge XVC: Debug Bridge XVC IP allows you to use debug cores and debug thedesign over PCIe

○ Mailbox: Mailbox IP interface to AXI interconnect in the user space

○ Feature ROM: AXI BRAM Controller IP that contains platform metadata for driver toretrieve information and perform checks.

○ Dynamic Region: AXI Lite connections to kernels that allows host/embedded scheduler totransfer arguments, schedule tasks and poll for completion





Configuring the PCIe and DMAThe platform design should contain a PCIe and DMA Subsystem master IP, clocking via theClocking Wizard IP, and resets via the Processor System Reset IP.

Use the DMA Subsystem for PCI Express IP, or use a board vendor specific IP for the DMAfunctionality. This IP provides PCIe DMA functionality between the card resident RAM and hostRAM. The XDMA IP requires the addition of two master AXI interfaces:

• One Full AXI interface to master data to the Memory Controller.

• One AXI-Lite interface to control the dynamic region (to configure the kernels) and to controlother peripherals in the static region.

Add the DMA Subsystem for PCI Express I IP to the IP integrator design. The configurationoptions are displayed in the following figure.





Figure 12: DMA Subsystem for PCIe Configuration Options

Configuring the AXI Address SpaceThe device relies on AXI memory mapped bus for addressing the kernel, the device dedicatedRAM, and any other peripherals. There are two AXI masters in the design: the PCIeEndpoint/DMA Controller and the kernel. There are two AXI slave end points in the design: TheAXI-Lite control port of the dynamic region (ocl_block_0) and the MIG controller.

To define the address space:

1. Open the Address Editor tab as shown in the following figure.The Address Editor tab showsthe DMA Controller with one AXI master and 2 slave AXI endpoints. It also shows thedynamic region with one slave AXI endpoint.





Figure 13: AXI Address Spaces

2. Select Auto Assign Address for the DMA Controller and the dynamic region. The Masterinterface of the DMA controller has two address ranges for the two Slave interfaces it isconnected to:

• Slave Port of Memory Controller: Select the full range of the addressable memory which istypically at least 32G with Offset Address 0x000000000 and High Address 0x7FFFFFFFF.The Slave port of the memory controller is called regslice_data_M_AXI.

• There are two control interfaces:

1. regslice_control_M_AXI: This interface is exposed to the dynamic region for control ofManagement AXI Lite peripherals. The Base address of this master interface startsfrom 0x1000000 and has an aperture of 8M. The Address ranges spans from0x1000000 to 0x17FFFFF. This interface is connected to the Slave AXI Lite ports ofthe memory controllers.

2. regslice_control_userpf_M_AXI: This interface is exposed to the dynamic region forcontrol of User AXI Lite peripherals. The base address of this master interface startsfrom 0x1800000 and also has an aperture of 8M.The Address ranges spans from0x1800000 to 0x1FFFFFF. This interface is connected to the Kernel AXI LIte portsand provides the control interface for Xilinx OpenCL Drivers running on the hostmachine.

3. Click Validate Design to validate the design.

Including I/O Buffers for Static I/O CrossingsWith partial reconfiguration, all routing must be contained within the dynamic region. If thefloorplan for the static region includes any I/O ports that are used by signals within the dynamicregion, an IBUF or OBUF needs to be placed in the static region logic design to connect itthrough the static region to the dynamic region, as shown in the following figure.

Currently, this involves preparing a small RTL design containing the I/O buffer logic and adding itto the design using the Vivado® IP integrator Add Module command. An example of such RTLcode is included with the sources supplied with the Xilinx Platform Development Board.





Figure 14: Configuring Dynamic Region I/O Logic in the Static Region Logic Hierarchy

Configuring the Dynamic Region for PartialConfigurationLogic needs to be configured in order to hold the design in a safe state while the device ispartially reconfigured with the hardware function. This can be created manually or by using therecommended Partial Reconfiguration Decoupler IP. A single reset within the IP will put theentire dynamic region into reset. A pair of registers supply back pressure on the AXI interfaces toensure they maintain state. In this way, the PCIe link is not disrupted when a new binary isdownloaded to the reconfigurable expanded region area of the accelerator device. The devicedriver manages this non-disruptive partial bitstream download process with the aid of theplatform hardware.

Using the Partial Reconfiguration Decoupler IP

The PR Decoupler IP can be instantiated and configured into the static region design using the IPintegrator. It is the recommended way to configure your PR decoupling logic. The followingfigures show the Decoupler IP configuration options.

The Global Options are used to set the general AXI interface and Decoupler IP configuration, asshown in the following figure..





Figure 15: Partial Reconfiguration Decoupler IP - Global Options

The Interface Options are used to create custom interfaces and to set the required SignalOptions, as shown in the following figure.

Figure 16: Partial Reconfiguration Decoupler IP – Interface Options





Manually Creating Partial Reconfiguration Decoupler Logic

As seen in the Xilinx reference designs, decoupling interrupt logic IP can be created manually. Asingle reset within the IP will put the entire dynamic region into reset. A pair of registers supplyback pressure on the AXI interfaces to ensure they maintain state. An example of decouplinglogic is shown in the following figure.

Figure 17: Example of PR Decoupling Logic

Configuring Simulation IPAXI Verification IP (VIP) can be inserted into the design to facilitate faster simulation for the PCIeand AXI based logic. It enables bus functional simulation of the AXI interfaces. An example of theIP is shown in the following figure.





Figure 18: AXI Performance Monitor IP for Simulation

The IP can be configured to handle a variety of simulation requirements. The variousconfiguration options are shown below.





Figure 19: AXI Verification IP Configuration Options

For more information, refer to the following resources:

• AXI Stream Verification IP Product Page

• AXI Verification IP Product Guide (PG267)

• AXI4-Stream Verification IP Product Guide (PG277)

Configuring Debug LogicDebug logic can be added to the design to enable hardware validation and debug using theVivado Hardware Manager. The Static region should contain the Debug Bridge IP to sync withthe Debug Bridge and ILA debug cores in the Dynamic Region. The debug bridge IP has aconfiguration option to be used with another Debug Bridge IP and ILAs within the Dynamicregion, as shown in the following figure.



https://www.xilinx.com/products/intellectual-property/axi-stream-vip.html

https://www.xilinx.com/cgi-bin/docs/ipdoc?c=axi_vip;v=latest;d=pg267-axi-vip.pdf

https://www.xilinx.com/cgi-bin/docs/ipdoc?c=axi4stream_vip;v=v1_1;d=pg277-axi4stream-vip.pdf



Figure 20: Static Region Debug Bridge Configuration

Using the Flash ProgrammerThis module allows firmware update via PCIe without JTAG. The AXI Quad SPI IP in the platformprovides a means of programming an image on the flash chip on the board. It provides an elegantway of updating the flash image to contain the new platform bitstream using the XDMA driverwithout needing to use the JTAG pins, since these are not normally available in a datacenter. Thexbsak Software utility shipped with the reference design provides this functionality through theuse of the xbsak flash function.





Figure 21: Enabling Flash Programming

Designing the Dynamic RegionThe dynamic region is represented by a block diagram referenced from the top-level blockdesign. You must configure this module to properly represent the interfaces between thedynamic region and the static logic design. The SDx compiler uses the IP integrator framework toswap out the default dynamic region with an updated dynamic region containing the compiledkernel(s) generated from the SDx device code, and the interconnect topology they require.Vivado uses partial reconfiguration to replace the dynamic region.





Creating a Separate Dynamic Region Logic Design inIP IntegratorThe dynamic region of the hardware platform must be configured in order for the tools toprocess the logic correctly with partial reconfiguration and in order to create a partial bitstreamfile. The dynamic region logic must be a completely separate BD logic module designed apartfrom the static logic BD. This is accomplished by designing a separate BD design in IP integratorrepresenting the dynamic region logic. The remainder of this section describes the process tocreating this isolated block design.

Generate Block Diagram Output Products with OOC Per BD Option

In order for the Vivado® partial reconfiguration capabilities to work properly, the dynamic regionBD output products must be generated using the Out of context per Block Design option, asshown in the following figure.

Figure 22: Generate Output Products with Out of context per Block Design





Working with the SDx Memory SubsystemThe dynamic region block diagram contains very few IP cores, since the complexity of thememory controllers and the AXI SmartConnect network are abstracted by the SDx MemorySubsystem (MSS) IP.

When creating the hardware platform, the MSS instantiates all available DDR4 memorycontrollers and automatically constructs an optimized SmartConnect-based network to connectthe XDMA to the memory controllers for host access to global memory.

When the built DSA is used with the SDAccel™ System Compiler, the MSS is automatically re-customized by the flow to instantiate only the necessary memory controllers as defined by kernelconnectivity to memory banks, and to automatically construct an optimized SmartConnect-basednetwork to connect both the user kernels and the XDMA to those memory controllers. Memorycontrollers corresponding to banks which are unmapped are not instantiated, and theSmartConnect network is optimized accordingly.

The MSS also automatically handles the placement and timing complexities of AXI interfacescrossing SLRs in SSIT devices, in the event that user kernels in a given SLR are mapped to one ormore memory controllers in a different SLR. The user is required only to specify theSLR_ASSIGNMENTS property on the user kernels, which instructs the MSS as to the location ofeach. See the SDx Environments Release Notes, Installation, and Licensing Guide (UG1238) fordetails on this process.

The MSS IP is not customized in the traditional manner, via IP GUI or CONFIG properties.Instead, it derives its configuration from two sources:

1. The Vivado project’s board file. When using the MSS IP, a board file is required. The board filemust fully specify the DDR4 memory controllers, including their pinouts and IPconfigurations. Simply targeting a compatible board file when creating the Vivado projectwhich is used to construct the hardware platform is sufficient for the MSS IP to derive thenecessary DDR4 information.

2. The DSA_INFO environment variable. This environment variable provides the MSS withimportant information associating DDR4 memory controllers with the device’s SLR whichcontains them, pblock names to associate with SLRs, and other device characteristics. Acorrectly-specified dictionary assigned to the DSA_INFO environment variable is required.

Instantiating the SDx Memory Subsystem IPTo instantiate the SDx Memory Subsystem IP:

1. Begin by ensuring that your Vivado project targets a board file which contains fully-specifiedDDR4 memory controller information. When choosing your project’s part, select acompatible board file rather than a part. An example targeting the Xilinx KCU1500 board isshown in the following figure.



https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug1238-sdx-rnil.pdf



Figure 23: Targeting a supported board file

2. The MSS IP should be instantiated in the dynamic region block diagram. But before openingthe block diagram in your Vivado session, set the following parameter via the Tcl console:set_param bd.skipSupportedIPCheck true

This parameter instructs IP integrator to reveal the SDx Memory Subsystem in the IP Catalog.

3. Ensure that the DSA_INFO environment variable is set correctly in your Linux shellenvironment. This globally-scoped environment variable is required to properly supportvalidation and implementation of the MSS IP.

Note: DSA_INFO must be specified whenever the block diagram is opened, both during platform creationand during SDx System Compiler invocation.

The following is an example of the DSA_INFO environment variable value for the KCU1500platform. Refer to the Xilinx hardware platform reference designs (Appendix A: Using theKCU1500 Reference Design and Appendix B: Using the VCU1525 Reference Design) forproperly-constructed DSA_INFO assignments and their automatic application during theflow.

• child_pblock_declarations associates a physical pblock range, parent pblock name,and paths to cells to place in the pblock, for each supported DDR4 controller instance

• slr_pblock_map associates a pblock name with the physical SLR it covers





• board_component_slr_map associates a board file DDR4 instance definition with theSLR that the controller resides in.

{ child_pblock_declarations { pblock_ddr4_mem00 { range CLOCKREGION_X2Y0:CLOCKREGION_X2Y3 parent pblock_dynamic_SLR0 cell_paths memory/ddr4_mem00 } pblock_ddr4_mem01 { range CLOCKREGION_X4Y2:CLOCKREGION_X4Y4 parent pblock_dynamic_SLR0 cell_paths memory/ddr4_mem01 } pblock_ddr4_mem02 { range CLOCKREGION_X2Y6:CLOCKREGION_X2Y9 parent pblock_dynamic_SLR1 cell_paths memory/ddr4_mem02 } pblock_ddr4_mem03 { range CLOCKREGION_X4Y6:CLOCKREGION_X4Y9 parent pblock_dynamic_SLR1 cell_paths memory/ddr4_mem03 } } slr_pblock_map { SLR0 pblock_dynamic_SLR0 SLR1 pblock_dynamic_SLR1 } board_component_slr_map { xilinx.com:kcu1500:ddr4_sdram_c0:1.1 SLR0 xilinx.com:kcu1500:ddr4_sdram_c1:1.1 SLR0 xilinx.com:kcu1500:ddr4_sdram_c2:1.1 SLR1 xilinx.com:kcu1500:ddr4_sdram_c3:1.1 SLR1 }}

Note: If you are targeting a different board, you can adjust the dictionary values as necessary to suit yourmemory and device characteristics.

4. Instantiate the SDx Memory Subsystem IP core in your block diagram, and connect itsinterfaces as necessary, as shown in the following figure illustrating the KCU1500 dynamicregion contents.





Figure 24: KCU1500 Dynamic Region Contents with SDx Memory Subsystem IP

• The S_AXI_CTRL interface is driven by the management PF AXI control path from thestatic region, providing the XDMA access to memory-mapped control resources within theMSS.

• The S00_AXI interface is driven by the AXI data path from the static region, providing theXDMA access to the full range of instantiated DDR4 controllers.

• One differential clock input is present for each DDR4 memory controller instantiatedwithin the MSS.

• One DDR4 interface is present for each memory controller’s top-level IO to memory.

5. It is also necessary to connect associated one or more clocks and a reset signal. A properly-configured MSS IP together with the use of SLR_ASSIGNMENTS during the SDx SystemCompiler flow will result in an efficient and optimized SmartConnect network, and only thenecessary DDR4 channels instantiated in any given partial bitstream.

Other Dynamic Region IPsIt is also necessary to instantiate an AXI Interconnect instance which is driven by the userphysical function AXI data path from the static region. The SDx System Compiler flowautomatically expands this instance and connects it to user kernels for memory-mapped accessto those resources. Set the PFM.AXI_PORT on this instance to instruct SDx to use the resource.Refer to Configuring Platform Interface Properties for more information.

As described in Configuring Simulation IP, AXI VIP instances can be added to the dynamic regionAXI paths to support simulation of the region.

Additionally, a Debug Bridge instance in the dynamic region supports connectivity to the staticregion for debug support of dynamic region resources. See Configuring Debug Logic for moreinformation.





Configuring the AXI Address SpaceThe device relies on AXI memory mapped bus for addressing the SDx™ kernel, the devicededicated RAM, and any other peripherals. Two primary classes AXI masters in the design are thePCIe Endpoint or the DMA Controller and the SDx kernels. Two primary classes of AXI slave endpoints in the design are the AXI-Lite control port of the kernels and the DDR4 memorycontrollers.

To define the address space:

1. Open the Address Editor tab as shown in the following figure.The Address Editor tab showsthe memory-mapped control and data paths of the memory controller IP cores, as accessibleby the DMA Controller resident in the static region.

Figure 25: AXI Address Spaces

2. Most addresses are assigned when configuring the Memory Subsystem IP. Unassignedaddresses can be allocated as needed for your system.

3. Click Validate Design to validate the design.

Adding Custom IPCustom RTL can be added to the design in the IP integrator using one of two ways. RTL onlylogic can be added using the Add Module command. Designs that contain IP or other logicstructures can be packaged as IP and added to the design using the IP Packager.





Using the Add Module Command

Custom RTL can easily be incorporated into the design by using the Add Module command. RTLsources must first be added to the Vivado project using the Add Sources command. It is theavailable in the Add Module dialog box to be added to the design. For more information onadding custom IP, refer to the Vivado Design Suite User Guide: Designing IP Subsystems using IPIntegrator (UG994).

Using the IP Packager

The Vivado IP Packager can be used to define custom IP to add to the hardware platform design.The Vivado IP Packager supports standard interfaces such as AXI, DDR, and PCIe signals. Formore information on packaging custom IP refer to the Vivado Design Suite User Guide: Creatingand Packaging IP (UG1118).

Once packaged, the custom IP can be added to the IP Catalog, and made available in the VivadoIP integrator. For SDAccel™, you would typically use this feature for defining and importing acustom DMA or memory controller, custom RTL, and I/O Buffer logic.

To add a custom IP repository to the IP Catalog:

1. Select Tools > Project Settings to open the Project Settings dialog box.

2. Click the IP tab.

3. Click the Repository Manager tab.

4. Click Add, and select the folders that contain the custom packaged IP.

Configuring Debug LogicDebug logic can be added to the design to enable hardware validation and debug using theVivado Hardware Manager. A Debug Bridge IP and at least one ILA should be added to thedynamic region logic. The Debug Bridge IPs in both the static and dynamic regions are configuredto communicate with each other to coordinate the debug process.

The System ILA IP should be configured to add debug logic to the dynamic region design. It canbe used to monitor and validate the running hardware platform on the Xilinx device, as shown inthe following figure. For more information on debugging IP integrator designs, refer to the VivadoDesign Suite User Guide: Designing IP Subsystems using IP Integrator (UG994).




http://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug1118-vivado-creating-packaging-custom-ip.pdf




Figure 26: System ILA Debug Logic Configuration Options

Assembling the Vivado PR ProjectThe hardware platform design project must be assembled and configured in a very specific way inorder for the partial reconfiguration features to work properly. The following steps describe theprocess and requirements.

Enabling Partial Reconfiguration for Your ProjectThe first step in creating a PR hardware platform design is to create the Vivado project anddefine it as a PR project, which will enable PR functionality to configure the project accordingly.





Figure 27: Enabling Partial Reconfiguration in the Project

Setting the Parameter to Enable PR Features in the IP Integrator

Partial reconfiguration flow support using IP integrator is an early access feature. The followingparameter must be set in the project to enable the PR capabilities in IP integrator.

set_param project.enableprflowIPI true

Adding the Dynamic Region Block Diagram to theProjectOpen the top-level design containing the static and I/O crossing logic as described in Creatingthe Top-Level Static Design

Use the Add Sources command to add the Dynamic region BD design to the Vivado® top-levelproject. It will appear as a separate top-level logic hierarchy tree in the Sources window, asshown in the following figure.





Figure 28: Adding Dynamic Region Block Diagram to Top-Level Project

Configuring the Dynamic Region to beReconfigurableTo configure the dynamic region to be reconfigurable, the dynamic region needs to first beidentified and partitioned within the Sources view.

1. Open the Sources view.

2. Select the Partition Definitions tab to view the design hierarchy.





Figure 29: Defining the Dynamic Region Partition

3. Select the dynamic region module.

4. Right-click and select Edit Partition Definition from the context menu that appears.





Figure 30: Configuring the Dynamic Partition to be Reconfigurable

5. Click OK to define the module as reconfigurable.

For more information on partial reconfiguration, see the Vivado Design Suite User Guide: PartialReconfiguration (UG909)

Adding the Dynamic Region Partition to the Top-Level Block Diagram in IP IntegratorThe reconfigurable module that we just created can now be added to the top-level hardwareplatform design. Open the top-level BD design and use the dropdown menu on the canvas toselect the Add Module command. Select the Module type: Partition Definition and notice thereconfigurable partition is listed, as shown in the following figure. Select it and click OK.






Figure 31: Adding the Dynamic Region Reconfigurable Partition to the Top-Level BDDesign

A hierarchical instance is created on the IP integrator canvas representing the dynamic regionlogic, as shown in the following figure. Notice the large PR logo in the center of the instanceindicating that it is a partially reconfigurable partition.





Figure 32: Creating Dynamic Region Logic Hierarchy Partition

It is not actually imported into the top-level design. It is referenced only.

Connections can now be completed between the static and dynamic regions and I/O ports.

Opening the Dynamic Region Block DiagramReference DesignTo open the lower level dynamic region BD design, click the symbol in the upper left corner ofthe logic symbol. A separate BD is opened.

Notice the chooser in the BD window banner to toggle between the two open BDs, as shown inthe following figure.

Figure 33: Toggling Between Multiple Open BD Designs





Completing the Top-Level DesignUse Connection Assistance and manual techniques to complete connections between the staticand dynamic regions, as shown in the following figure.

Figure 34: Completed Hardware Platform Design





Chapter 5

Configuring Platform PropertiesIn order for the hardware platform to properly communicate with the SDx™ environment, certainproperties need to be configured. These are used to identify the hardware platform and toidentify the signal interfaces between the hardware platform design and the SDx kernel logic.

Setting the Platform IdentificationProperty (PFM_NAME)The PFM_NAME property must be set in each hardware platform design to define the vendor,board lib, platform name, and version. It can be set manually on the Tcl command prompt.

Note: The PFM_NAME API properties replace the previous dsa_vendor, dsa_board_id, dsa_name, anddsa_version properties from previous version platforms. When migrating previous version platforms, theseproperties will override the PFM_NAME properties, if they still exist in the project.

The PFM_NAME property support two formats:

• A full format where four fields are set to define the following:

vendor ; board library ; platform name ; version

Example: set_property PFM_NAME xilinx:kcu1500:kcu1500_base:1.0[get_files $bd_file.bd]

• A simple string format, where you only need to specify the platform name and the other fieldswill get assigned the default values vendor:library,<my_platform_name>:1.0.

Example: set_property PFM_NAME my_platform [get_files $bd_file.bd]

This results in the PFM_NAME property set to vendor:library:my_platform:1.0.




Configuring Platform Interface PropertiesThe Platform Interfaces are defined using the four PFM properties described below. They can bedefined manually in the Tcl Console, or by Tcl script for the design.

The four Platform Interfaces Tcl APIs are:

set_property PFM.AXI_PORT { <port_name> {parameters} \<port2> {parameters} ...} [get_bd_cells <cell_name>]set_property PFM.AXIS_PORT { <port_name> {parameters} \<port2> {parameters} ...} [get_bd_cells <cell_name>]set_property PFM.CLOCK { <port_name> {parameters} \<port2> {parameters} ...} [get_bd_cells <cell_name>]set_property PFM.IRQ { <port_name> {} <port2> {} ...} \[get_bd_cells <cell_name>]

The requirements for the PFM Properties are:

• The value of the PFM interface properties must be specified as a Tcl dictionary, a list ofname/"value" pairs.

IMPORTANT!: The "value" must be quoted, and both the name and value are case sensitive.

• A bd_cell can have multiple PFM interface definitions. However, for each type of PFMinterface, all ports are required to be set in a single set_property Tcl command.

• For each PFM interface property, the name specified for the port object must match the nameof an external port or interface on a bd_cell. Each external port or interface object may onlyhave one PFM interface definition.

• Each different type of PFM interface may have different parameters.

• Setting the PFM property with a NULL ("") string will delete previously defined PFMinterfaces.

Declaring ClocksYou can export any clock source with the platform, but for each clock you must also exportsynchronized reset signals using a Processor System Reset IP block in the platform. ThePFM.CLOCK property can be set on BD cell, external port, or external interface.

The Tcl command for setting the PFM.CLOCK property is:

set_property PFM.CLOCK { <port_name> {parameters} \<port2> {parameters} ...} [get_bd_cells <cell_name>]

Argument Description• Port_name: Clock port name.

Chapter 5: Configuring Platform Properties




• Parameters:

○ id value: Clock ID is a user-defined value that must be a unique non-negative integer.

○ is_default value: Specify "true" if this is the default clock, "false" otherwise. The defaultis "false".

○ proc_sys_reset value: This name/value pair specifies the correspondingproc_sys_reset block instance for synchronized reset signals connected to the clockport.

IMPORTANT!: Every platform must declare one default clock with the is_default parameter setto "true" for the SDSoC environment to use when no explicit clock has been specified.

Examples:

set_property PFM.CLOCK {PL_CLK0 {id "0" is_default "true" proc_sys_reset \"proc_sys_reset_0"} PL_CLK1 {id "1" is_default "false" proc_sys_reset \"proc_sys_reset_1"} PL_CLK2 {id "2" is_default "false" proc_sys_reset \"proc_sys_reset_2"} PL_CLK3 {id "3" is_default "false" proc_sys_reset \"proc_sys_reset_3"} } [get_bd_cells /zynq_ultra_ps_e_0]

To set a CLOCK on an external PORT:

set_property PFM.CLOCK{ACLK_0 {id "4" is_default "false" proc_sys_reset \"proc_sys_reset_4"}} [get_bd_ports /ACLK_0]

Declaring AXI PortsThe Tcl command for setting the PFM.AXI_PORT property is:

set_property PFM.AXI_PORT { <port_name> {parameters} \<port2> {parameters} ...} [get_bd_cells <cell_name>]

Argument Description• Port_name: AXI port name.

• Parameters:

○ memport type: Corresponding memory interface port type. Valid type values include:

- M_AXI_GP: A general-purpose AXI master port

- S_AXI_HP: A high-performance AXI slave port





- S_AXI_ACP: An accelerator coherent slave port

- S_AXI_HPC: A high-performance accelerator coherent slave port

- MIG: An AXI slave connected to a MIG memory controller. The default is MIG.

○ sptag ID: (Optional) A user-defined ID that should start with an alphabetic character. TheID is case-sensitive. The system port tag (sptag) is a symbolic identifier that represents aclass of platform port connections, e.g., S_AXI_HP, S_AXI_ACP, M_AXI_GP. Multiple blockdesign platform ports can share the same sptag.

○ memory: (Optional) Specify the associated MIG IP instance and address_segment. Thememory tag is a unique identifier that combines the Cell name and Base Name columnsin the IP integrator Address Editor. This tag will be associated with connections to theMemory Subsystem HIP, where multiple block design platform ports can share the samememory tag.

Example for an AXI Interconnectset_property PFM.AXI_PORT { \ M_AXI_GP0 {memport "M_AXI_GP"} \ M_AXI_GP1 {memport "M_AXI_GP"} \ S_AXI_ACP {memport "S_AXI_ACP" sptag "ACP" memory \"processing_system7_0 ACP_DDR_LOWOCM"} \ S_AXI_HP0 {memport "S_AXI_HP" sptag "HP0" memory \"processing_system7_0 HP0_DDR_LOWOCM"} \ S_AXI_HP1 {memport "S_AXI_HP" sptag "HP1" memory \"processing_system7_0 HP1_DDR_LOWOCM"} \ S_AXI_HP2 {memport "S_AXI_HP" sptag "HP2" memory \"processing_system7_0 HP2_DDR_LOWOCM"} \ S_AXI_HP3 {memport "S_AXI_HP" sptag "HP3" memory \"processing_system7_0 HP3_DDR_LOWOCM"} \ } [get_bd_cells /processing_system7_0]

Exporting AXI interconnect master and slave ports involves several requirements.

1. All ports on the interconnect used within the platform must precede in index order anydeclared platform interfaces.

2. There can be no gaps in the port indexing.

3. The maximum number of master IDs for the S_AXI_ACP port is eight, so on a connected AXIinterconnect, available ports to declare must be one of {S00_AXI, S01_AXI, ..., S07_AXI}. Donot declare any ports that are used within the platform itself. Declaring as many as possiblewill allow sds++ to avoid cascaded axi_interconnects in generated user systems.

4. The maximum number of master IDs for an S_AXI_HP or MIG port is sixteen, so on anconnected AXI interconnect, available ports to declare must be one of {S00_AXI, S01_AXI, ...,S15_AXI}. Do not declare any ports that are used within the platform itself. Declaring asmany as possible will allow sds++ to avoid cascaded axi_interconnects in generated usersystems.





5. The maximum number of master ports declared on an interconnect connected to anM_AXI_GP port is sixty-four, so on an connected AXI interconnect, available ports to declaremust be one of {M00_AXI, M01_AXI, ..., M63_AXI}. Do not declare any ports that are usewithin the platform itself. Declaring as many as possible will allow sds++ to avoid cascadedaxi_interconnects in generated user systems.

Declaring SDx Memory Subsystem ResourcesTo declare SDx™ Memory Subsystem DDR4 resources, use:

set_property PFM.MEMSS { <resource_name> {sp_name} <resource_name_2} {sp_name} .. }get_bd_cells <cell_name>]

The SDx Memory Subsystem resource_names are advertised in the Base Name field of thememory-mapped DDR4 segments within the IP integrator Address Editor view.

The sp_name is a nickname given to a given memory resource, and is the syntax used on theXOCC command line when mapping a compute unit to a given memory bank.

The following example is for the KCU1500 platform:

set_property PFM.MEMSS {DDR4_MEM00 {bank0} \DDR4_MEM01 {bank1} \DDR4_MEM02 {bank2} \DDR4_MEM03 {bank3} \} [get_bd_cells /memory_subsystem]

Declaring AXI-4 Stream PortsThe Tcl command for setting the PFM.AXIS_PORT property is:

set_property PFM.AXIS_PORT { <port_name> {parameters} \<port_name_2> {parameters} .. } [get_bd_cells <cell_name>]

Argument Description• Port_name: AXI4-Stream port name.

• Parameters:

○ type value: Streaming interface port type. Valid values for type include:

- M_AXIS: A general-purpose AXI master port

- S_AXIS: A high-performance AXI slave port





Examplesset_property PFM.AXIS_PORT {AXIS_P0 {type "S_AXIS"} \[get_bd_cells /zynq_ultra_ps_e_0]

Declaring Interrupt PortsInterrupts must be connected to IP integrator Concat (xlconcat) blocks that are connected to theprocessing system. For Zynq®-7000 family it’s the F2P_irq port. For Zynq® UltraScale+ MPSoCdevices the interrupts are split into two 8-bit ports: pl_ps_irq0[7:1] and pl_ps_irq1[7:1].

IMPORTANT!: If any IP within the platform includes interrupts, these must occupy the leastsignificant bits of the Concat block without gaps.

The Tcl command for setting the PFM.IRQ property is:

set_property PFM.IRQ { <port_name> {} <port2> {} ...} \[get_bd_cells <cell_name>]

Argument Description• Port_name: IRQ port name

• {}: Empty list that serves as a placeholder.

Exampleset irqProp [] for {set i 0} {$i < 8} {incr i} { lappend irqProp In$i {} } set_property PFM.IRQ $irqProp [get_bd_cells /xlconcat_0] set_property PFM.IRQ $irqProp [get_bd_cells /xlconcat_1

TIP: The FOR loop results in a PFM.IRQ property as defined by $irqProp that looks like:In0 {} In1 {} In2 {} In3 {} In4 {} In5 {} In6 {} In7 {}

Declaring IO DevicesIf you use the Linux UIO framework, you must declare the devices. To declare an instance to be aLinux IO platform device use the :

set_property PFM.DEVICE_IO { <port_name> {type} <port2> {type} ...} \[get_bd_cells <cell_name>]

Argument Description• Port_name: AXI port name.





• type value: I/O device type (e.g., UIO, KIO).

Example:

Additional Interface Property ExamplesTo define an AXI_port on interconnect:

set parVal []for {set i 2} {$i < 64} {incr i} { lappend parVal M[format %02d $i]_AXI {memport "M_AXI_GP"}}set_property PFM.AXI_PORT $parVal [get_bd_cells /axi_interconnect_0]

To define an AXI_port on SMC:

set parVal []for {set i 1} {$i < 16} {incr i} { lappend parVal S[format %02d $i]_AXI {memport "MIG" sptag "Bank0"}}set_property PFM.AXI_PORT $parVal [get_bd_cells /smartconnect_0]

To define an AXI_PORT that connects with MIG IP:

set parVal []for {set i 1} {$i < 16} {incr i} { lappend parVal S[format %02d $i]_AXI {memport "MIG" sptag "bank0" memory "ddrmem_0 C0_DDR4_ADDRESS_BLOCK"}}set_property PFM.AXI_PORT $parVal [get_bd_cells /memory_subsystem/interconnect_data/interconnect_aximm_ddrmem0]

To set a CLOCK on an external PORT:

set_property PFM.CLOCK{ACLK_0 {id "4" is_default "false" proc_sys_reset "/proc_sys_reset_4"}}[get_bd_ports /ACLK_0]

Setting Platform Configuration PropertiesThe following chart describes the various platform configuration properties that are used todefine platform behavior, as well as interaction with the SDx™ kernels.





Property Name Description FormatBoard Related

Mandatory

dsa.vendor Sets the vendor attribute in dsa.xml.This is the vendor idassociated with thehardware platform.

set_property dsa.vendor"<your_vendor_name>"[current_project]

dsa.board_id Sets the board attribute in dsa.xmlwhen the project does not have aboard associated with it.

set_property dsa.board_id"<your_board_Lib_name>"[current_project]

dsa.name This is an identifier for the hardwareplatform/DSA

set_property dsa.name"<your_platform_name>"[current_project]

dsa.version Sets the version for the hardwareplatform

set_property dsa.version"<your_version>" [current_project]

Optional

dsa.board_interface_name Backup property to specify boardinterface name when no board isassociated with the project

set_property dsa.board_interface_name"<your_interface_name>"[current_project]

dsa.board_interface_type Backup property to specify boardinterface type when no board isassociated with the project

set_property dsa.board_interface_type"<your_interface_type>"[current_project]

dsa.board_memories Backup property to specify number ofmemories on the board. Each value issplit into 3 attributes. Memory Name,Memory Type and Memory size

set_property dsa.board_memories{{mem0 ddr4 16GB} {mem1 ddr4 16GB}{mem2 ddr4 16GB} {mem3 ddr4 16GB}}[current_project]

dsa.num_compute_units This data is used by SDx runtime codeto determine the maximum number ofcompute units which are instances ofkernels that can be inserted into thedynamic region.

set_property dsa.num_compute_units16 [current_project] 1. Default is 60

dsa.exclude_hw_export This property allows the dsa creator toexclude capturing the HDF content inthe DSA. Default behavior is to captureHDF data

set_property dsa.exclude_hw_exporttrue [current_project] 1. Default is false.This property can be set to excludecapturing the hdf/ hardware handfoffdata from the DSA

Flow Related

dsa.ip_cache_dir Allows user to specify an ip cache thatwill be captured in the DSA and used bySDx Flows(xocc/vpl)

set_property dsa.ip_cache_dir "ip/cache/dir" [current_project]

dsa.ocl_inst_path This is the instance path to the dynamicregion in the ull hierarchy. Empty stringfor SoC platforms as entire design isdynamic region

set_property dsa.ocl_inst_path "ocl/inst/path" [current_project]

dsa.uses_pr This tells xocc that the platform is usingPR and that a partial bitstream will begenerated for the dynamic region

set_property dsa.uses_pr true[current_project] 1. Default is false. 2.Needs to be set to true for PCIe. 3. AllSoC platforms do not use PR. PCIe/SDAccel platforms use PR in most case





Property Name Description Formatdsa.platform_state This is the state of the platform as

captured in the DSA. This allows theSDx Flows code to determine whethersynthesis and implementation need tobe run on full design or if the platformis pre-syntheised or pre-implementedand only the dynamic region needs tobe synthesized/implemented. The SDxflows construct what needs to happenin vpl accordingly.

set_property dsa.platform_state"pre_synth" [current_project] Defaultvalue is pre_synth. No need to set thisfor SoC platforms. Valid values arepre_synth,synth and impl This propertywas introduced with Unified Platforms.It replaces dsa.static_synth_checkpoint.If set to pre_synth, the DSA must havedata to create a project with originalsources. If set to synth, it is soft-flatflow. If set to impl, it could be hard-flatflow or PR flow.

dsa.accelerator_binary_format This is so far specific to AWS platformsonly. This tells the SDx flows the formatin which to generate the xclbin

set_propertydsa.accelerator_binary_format"xclbin2" [current_project] Defaultvalue is xclbin2 in 2017.1_sdx Defaultvalue was xclbin0 in 2016.4_sdx ValidValues are xclbin0 and xclbin2

dsa.accelerator_binary_content This is so far specific to AWS platformsonly. This tells the SDx flows whether tocaptured dcp or bitstream in the xclbin

set_propertydsa.accelerator_binary_content"bitstream" [current_project] Defaultvalue is bitstream. Valid Values arebisttream and dcp

Properties to set xocc xp switches forcompile and link phasesdsa.xocc_compile_xp_switches_defaultdsa.xocc_link_xp_switches_default

These properties allow DSA creator tospecify xocc --xp switch values for thecompile and link phases

set_propertydsa.xocc_compile_xp_switches_default{param:compiler.enableCopyLogFilesXocc=1} [current_project] set_propertydsa.xocc_link_xp_switches_default{param:compiler.enableCopyLogFilesXocc=1vivado_prop:run.impl_1.STEPS.PHYS_OPT_DESIGN.IS_ENABLED=true}[current_project]

Properties to set pre and post systemlinker dsa.pre_sys_link_tcl_hookdsa.post_sys_link_tcl_hook

These properties tell write_dsa tocapture the tcl hooks in the DSA. Ifspecified, the SDx flows code willinvoke the pre hook before callingsystem linker and the post-hook aftersystem linker.

set_property dsa.pre_sys_link_tcl_hookfile.tcl [current_project] set_propertydsa.post_sys_link_tcl_hook file.tcl[current_project]

Properties to set pre and post hooksfor all vivado run steps for run duringvpl in xocc link phasedsa.run.steps.init_design.tcl.predsa.run.steps.init_design.tcl.postdsa.run.steps.opt_design.tcl.predsa.run.steps.opt_design.tcl.postdsa.run.steps.place_design.tcl.predsa.run.steps.place_design.tcl.postdsa.run.steps.route_desgn.tcl.predsa.run.steps.route_design.tcl.post

These properties tell write_dsa tocapture the tcl hooks in the DSA. Ifspecified, the SDx flows code will set allthe standard run step pre and posthook properties in the vivado projectconstructed by vpl.

set_propertydsa.run.steps.init_design.tcl.pre file.tcl[current_project] set_propertydsa.run.steps.init_design.tcl.post file.tcl[current_project] set_propertydsa.run.steps.opt_design.tcl.pre file.tcl[current_project] set_propertydsa.run.steps.opt_design.tcl.post file.tcl[current_project] set_propertydsa.run.steps.place_design.tcl.prefile.tcl [current_project] set_propertydsa.run.steps.place_design.tcl.postfile.tcl [current_project] set_propertydsa.run.steps.route_desgn.tcl.prefile.tcl [current_project] set_propertydsa.run.steps.route_design.tcl.postfile.tcl [current_project]





Property Name Description FormatProperties to set constraint filesdsa.synth_constraint_filesdsa.impl_constraint_file

These properties allow the DSA creatorto package multiple constraint files inthe DSA. Each value is split into 3attributes- file name, processing rderand usedIn values. In the SDx flows,these constraint files are added to thevivado project created by vpl. Theprocessing order and used in valuesare also set as specified.

set_property dsa.synth_constraint_files{{file1.xdc,EARLY, {synthesisimplementation}} {file2.xdc, LATE,implementation}} [current_project]set_property dsa.impl_constraint_files{{file1.xdc,EARLY, implementation}{file2.xdc, LATE, implementation}}[current_project]

Host Related

dsa.host_architecture This is legacy only. It tells the SDx flowscode which host architecture is beingused, i.e. x86, arm etc.

set_property dsa.host_architecture"x86_64" [current_project] Default isx86_64 Possible values are x86_64 andarm7vl, aarch64

dsa.host_interface This is legacy only. It tells the SDx flowscode which ptotocol is bein used forcomunication between host and theFPGA, i.e. PCIe or AXI (zynq)

set_property dsa.host_interface "pcie"[current_project] Default is pciePossible values are pcie and axi

PCIe Related

dsa.pcie_id_vendor Backup property to specify pcie idvendor attribute. This is used by theSDx runtime.

set_property dsa.pcie_id_vendor"0x10ee" [current_project]

dsa.pcie_id_device Backup property to specify pcie iddevice attribute. This is used by the SDxruntime.

set_property dsa.pcie_id_device"0x8038" [current_project]

dsa.pcie_id_subsystem Backup property to specify pcie idsubsystem attribute. This is used by theSDx runtime.

set_property dsa.pcie_id_subsystem"0x0011" [current_project]

dsa.pcie_id_featureid Backup property to specify pcie idfeature Id attribute. This is used by theSDx runtime.

set_property dsa.pcie_id_featureid"0x0011abcd00001010"[current_project]

OCL Related (Legacy Platforms Only)

dsa.ocl_parent_rm_inst_path This is for legacy platforms only.Backup property to specify the parentRM instance that contains the OCLBlock instance.

set_propertydsa.ocl_parent_rm_inst_path"parent/rm/inst/path"[current_project]

dsa.ocl_address_segments This is for legacy platforms only.Hopefully don't have to document it. Itis used by AWS platforms as we cannotdetermine this data without a BD. Thelegacy AWS platforms do not have a BDassociated with them and areconstructed by setting all these backupproperties

set_property dsa.ocl_address_segments{{M00_AXI M00_AXI_addr_seg_00x000100000000 0x100000000expanded_region_memc_ddrmem_0}{M01_AXI seg1 0 0 slave1} {M02_AXIseg2 0 0 slave2} {M03_AXI seg3 0 0slave3}} [current_project] Examplemetadata for M00_AXI:<AddressSegmentName="M00_AXI_addr_seg_0"Base="0x000100000000"Range="0x100000000"Slave="expanded_region_memc_ddrmem_0"/>]





Property Name Description Formatdsa.ocl_ports This is for legacy platforms only.

Hopefully don't have to document it. Itis used by AWS platforms as we cannotdetermine this data without a BD. Thelegacy AWS platforms do not have a BDassociated with them and areconstructed by setting all these backupproperties

set_property dsa.ocl_ports {{NameS_AXI Type addressable Mode slaveProtocol AXI4LITE Base 0x00000000Range 0x00008000 DataWidth 32IdWidth 0 UserWidth 0 AddressWidth17 MaxBurstLength 1 Frequency 5000}}[current_project] This will override thefull S_AXI port information

FeatureROM Related

dsa.rom.aurora_link set_property dsa.rom.aurora_link true[current_project] Default is true

dsa.rom.board_mgmt set_property dsa.rom.board_mgmt true[current_project] Default is false

dsa.rom.scheduler set_property dsa.rom.scheduler true[current_project] Default is false

dsa.rom.aurora_link set_property dsa.rom.aurora_link true[current_project] Default is false

dsa.rom.prom_type set_property dsa.rom.prom_type 0[current_project] Default is 0 Value canrange from 0 to 7

dsa.rom.debug_type set_property dsa.rom.debug_type 0[current_project] Default is 0 Value canrange from 0 to 15

Using the IP CacheSignificant synthesis runtime savings can be achieved by taking advantage of the IP cachingcapabilities in Vivado. IP caching stores the synthesis results for each IP configuration and usesthe cached results in place of re-synthesizing the IP during output generation, and for additionalIP instances that have matching configurations.

In order for the IP to be cached successfully for use in the DSA, the Vivado Settings need to beconfigured so the Cache location is local to the Vivado project prior to generating the IPintegrator block design. This is the default setting, as shown in the following figure.





Figure 35: Vivado Settings - IP Cache

Setting the IP caching repository involves pointing to the IP cache repository. Use the followingTcl command to set the cache prior to creating the DSA.

set_property dsa.ip_cache_dir [get_property ip_output_repo \[current_project]] [current_project]





Chapter 6

Creating Design ConstraintsThis section discusses the various types of physical constraints that are needed to support thereconfigurable hardware platform.

I/O and Clock PlanningOne of the key considerations in the design of a DSA is to identify the I/O interfaces necessaryfor the board requirements. The Static I/Os are expected to operate and remain live during thedevice reprogramming of kernels. The physical I/O locations will influence performance and mustbe considered as part of the floorplanning process, especially when SSI devices are used.

It is recommended that the static I/Os are assigned physically outside of the dynamic region.There may be trade-offs made between the size and shape of the dynamic region toaccommodate an area free of static I/Os and overall system performance.

If the floorplan for the static region includes any I/O ports that are used by signals within thedynamic region, an IBUF or OBUF needs to be placed in the static region logic design to connectit through the static region to the dynamic region. Refer to Including I/O Buffers for Static I/OCrossings.

Refer to the Vivado Design Suite User Guide: I/O and Clock Planning (UG899) for more informationon I/O and clock planning.

FloorplanningWhen using Xilinx Partial Reconfiguration technology, floorplanning is required to separate thestatic and dynamic regions of the design. The floorplanning strategy can dramatically affect theperformance of the design. It can be an iterative process to find the optimal floorplan for yourspecific design and device. Refer to the Vivado Design Suite User Guide: Design Analysis and ClosureTechniques (UG906) for more information on defining floorplan regions.


http://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug899-vivado-io-clock-planning.pdf

http://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug906-vivado-design-analysis.pdf



Floorplanning is performed by assigning logic modules or cells to physical block ranges (Pblocks)on the device canvas. Floorplanning can be performed interactively by opening the Synthesizeddesign in the Vivado IDE or with constraints in the XDC file. Occasionally, in order to optimizeperformance or device resources, multiple Pblocks are used.

Pblocks can be defined as rectangular regions, or by combining multiple rectangles to define anon-rectangular shape. Pblocks can also be nested to enable lower levels of logic to be furtherconstrained to specific regions of the device. If nested Pblocks are desired, ensure the lower levelPblock region is completely within the upper level Pblock region.

TIP: Overlapping Pblocks should be avoided.

Floorplanning for PRFor PR hardware platform designs, the dynamic region must be entirely contained within aPblock to facilitate partial reconfiguration.

When designing a PR hardware platform, only a small static region is needed for the PCIe/DMAbase logic. The rest of the hardware platform logic, including the dynamic region, getsimplemented as part of a single dynamic region hierarchy. You can use Pblocks to constrain thedynamic region hierarchy, leaving the rest of the design for the static logic. In some cases,floorplanning the static logic can also be done to improve performance. Multiple logic modulesfrom the static region can be assigned to a single Pblock, but the dynamic region Pblock can onlybe assigned a single dynamic region module. Multiple Pblock rectangles should be used for thedynamic region logic to reserve and use as much of the device as possible. Ensure as many of theDSP and BRAM device resources are included in the dynamic region Pblock area.

Pblock size is important for the overall system performance. Rather than packing a Pblock astightly as possible, it is often necessary to leave the Pblock loosely utilized to enable the dynamicregion logic to be optimally placed within it for performance reasons. Iterations are often neededto find the optimal Pblock sizes.

Individual logic elements, such as BRAMs and DSPs from the static region, can also be lockedonto specific sites on the device fabric by manually placing these elements onto the device andfixing their location. Refer to the Vivado Design Suite User Guide: Design Analysis and ClosureTechniques (UG906) for more information on manual placement.

See the following appendixes for examples of floorplanning PR platforms with our suppliedreference designs:

• Appendix A: Using the KCU1500 Reference Design

• Appendix B: Using the VCU1525 Reference Design

Chapter 6: Creating Design Constraints


http://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug906-vivado-design-analysis.pdf



Partitioning the Timing ConstraintsWhen using partial reconfiguration, the constraints for the dynamically reconfigurable regionneed to be defined in a separate XDC file within the Vivado constraint set. This way they can beapplied directly to the dynamic region when it is being implemented separately.

The dynamic region constraints file should contain all timing and floorplanning constraintsscoped from the top-level of the design.

Ensuring the Constraint File Ends Up in the DSAIn order to add the XDC constraints file for the dynamic region gets added to the DSA, one of thefollowing properties must be set prior to generating the DSA.

set_property dsa.impl_constraint_files {{file1.xdc,EARLY, implementation} {file1.xdc, LATE, implementation}} [current_project]





Chapter 7

Implementing the HardwarePlatform Design

The hardware platform design should be implemented and validated to ensure it works as

expected in the Xilinx® SDAccel™ flow. The first step in that validation process should be toensure the hardware platform design itself is performing as expected. This can be done using testkernel logic to populate the dynamic region.

Simulating the DesignThe Vivado Design Suite has extensive logic simulation capabilities to enable block or systemlevel validation of the design. Available third party FPGA simulation tools are also supported.Refer to the Vivado Design Suite User Guide: Logic Simulation (UG900) for more information.

The AXI Verification IP, described in Configuring Simulation IP, should help to drastically reducelogic simulation runtime and effectiveness for PCIe based platforms.

Implementation and Timing ValidationThe design should be synthesized and implemented to ensure desired performance is achieved. Itis often required to iterate on floorplanning and implementation strategies to ensure optimalperformance.

It is often important to implement, analyze, and iterate on the hardware platform design toensure that it continues to meet timing during kernel implementation. Using a test kernel,implement the design and then check that the design meets timing by opening the ImplementedDesign.

The floorplan can be examined and modified if need be to optimize implementation results. Referto the Vivado Design Suite User Guide: Design Analysis and Closure Techniques (UG906) for moreinformation.


https://www.xilinx.com/cgi-bin/docs/rdoc?v=latest;d=ug900-vivado-logic-simulation.pdf

https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug906-vivado-design-analysis.pdf



Chapter 7: Implementing the Hardware Platform Design




Chapter 8

Generating a DSAAfter completing your hardware platform design and generating a valid bitstream using theVivado Design Suite, you are ready to create a Device Support Archive (DSA) file for use with theSDAccel Development Environment. A DSA is a single-file capture of the hardware platform, tobe handed off by the platform developer for use with the SDAccel Development Environment.

Generating the DSA FileIMPORTANT!: After creating the DSA file you should retain the source Vivado Design Suite projectfiles so you can recreate or update the DSA file as needed.

Once the required properties have been set, you can generate a DSA file using the write_dsacommand. This creates an archive of the hardware platform that contains all the relevant files anddata needed by the SDSoC Development Environment.

The write_dsa command will also create a bitstream file if one has not yet been created.

After all parameters are properly set up for the DSA, run the following Tcl command in theVivado Tcl Console:

write_dsa <filename>.dsa -include_bit

Where <filename> should be something meaningful that references the board and itscharacteristics. However, the filename is optional. If omitted, a filename is constructed from theDSA properties as:

${DSA.VENDOR}_${DSA.BOARD_ID}_${DSA.NAME}_${DSA.VERSION}

Usage:

Name Description -----------------------------[-unified] (default) Write DSA using new 2017.4 format[-legacy] Write DSA using format from pre-2017.4[-include_bit] Force include bit file(s) in the dsa.




[-force] Overwrite existing device support archive file[-quiet] Ignore command errors[-verbose] Suspend message limits during command execution[<file>] Device Support Archive fileValues: A filename with alphanumeric characters and .dsa extension.

Validating the DSAYou can use the validate_dsa command to validate a custom DSA file to ensure it contains

the proper content and metadata needed to support the hardware platform in the Xilinx®SDAccel™ environment. Use the following command to validate a DSA file:

validate_dsa <dsa file> -verbose

Using Older DSA VersionsPrevious DSA versions are still supported and work as defined. For a complete list of supportedplatforms for any given release, refer to the SDx Environments Release Notes, Installation, andLicensing Guide (UG1238).

Updating an Older Version DSATo update an older format DSA to the new DSA format, open a copy of the original Vivadohardware platform project in the newer version of software, upgrade the IP, implement thedesign, and generate a new DSA. In some cases, additional IP will need to be configured orparameters will need to be set to update to the new DSA format requirements. This cansometimes be a long process for complicated PR platform designs.

Chapter 8: Generating a DSA





Chapter 9

Software Platform SpecificationXilinx software stack for the platform is comprised of a stack of user space libraries and kernelspace drivers. The user space library is comprised of an OpenCL library, libxilinxopencl.so, and alow-level HAL library libxclgemdrv.so which sits on top two kernel mode drivers: xocl andxclmgmt. The following sections describe the HAL API and the kernel driver interfaces.

Using the OpenCL Hardware AbstractionLayer (XCL HAL) Driver APIThe header file xclhal2.h defines data structures and function signatures exported byHardware Abstraction Library (HAL). HAL is part of software stack which is integrated into Xilinxreference platform.

typedef xclDeviceHandle

opaque device handle

A device handle of xclDeviceHandle kind is obtained by opening a device. Clients pass thisdevice handle to refer to the opened device in all future interaction with HAL.




HAL Device Management APIs

API Description Parameters ReturnXCL_DRIVER_DLLESPECunsigned xclProbe()

Enumerate devices found inthe system

count of devices found

XCL_DRIVER_DLLESPEC xclDeviceHandle xclOpen(unsigned deviceIndex, constchar * logFileName,xclVerbosityLevel level)

Open a device and obtain itshandle. • unsigned

deviceIndex: Slotnumber of device 0 forfirst device, 1 for thesecond device...

• const char *logFileName: Log fileto use for optionallogging

• xclVerbosityLevel level: Severity levelof messages to log

Device handle

XCL_DRIVER_DLLESPEC voidxclClose (xclDeviceHandlehandle)

Close an opened device • xclDeviceHandlehandle: Device handle

void

XCL_DRIVER_DLLESPEC intxclResetDevice(xclDeviceHandle handle,xclResetKind kind)

Reset a device or its CL • xclDeviceHandlehandle: Device handle

• xclResetKindkind: Reset kind

0 on success or appropriateerror numberReset the device. All runningkernels will be killed andbuffers in DDR will bepurged. A device may bereset if a user’s applicationdies without waiting forrunning kernel(s) to finish.

XCL_DRIVER_DLLESPEC intxclGetDeviceInfo2(xclDeviceHandle handle,xclDeviceInfo2 * info)

Obtain various bits ofinformation from the device • xclDeviceHandle

handle: Device handle

• xclDeviceInfo2 *info: Informationrecord

0 on success or appropriateerror number

XCL_DRIVER_DLLESPEC intxclGetUsageInfo(xclDeviceHandle handle,xclDeviceUsage * info)

Obtain usage informationfrom the device • xclDeviceHandle


• xclDeviceUsage *info: Informationrecord


XCL_DRIVER_DLLESPEC intxclGetErrorStatus(xclDeviceHandle handle,xclErrorStatus * info)

Obtain error informationfrom the device • xclDeviceHandle


• xclErrorStatus *info: Informationrecord


Chapter 9: Software Platform Specification




API Description Parameters ReturnXCL_DRIVER_DLLESPEC intxclLoadXclBin(xclDeviceHandle handle,const xclBin * buffer

Download FPGA image(xclbin) to the device • xclDeviceHandle


• const xclBin *buffer: Pointer todevice image (xclbin) inmemory

0 on success or appropriateerror numberDownload FPGA image(AXLF) to the device. The PRbitstream is encapsulatedinside xclbin as a section.xclbin may also containsother sections which aresuitably handled by thedriver.

XCL_DRIVER_DLLESPEC intxclReClock2(xclDeviceHandle handle,unsigned short region, constunsigned short *targetFreqMHz

Configure PR regionfrequncies • xclDeviceHandle


• unsigned shortregion: PR region(always 0)

• const unsignedshort *targetFreqMHz:Array of targetfrequencies in order forthe Clock Wizards drivingthe PR region


XCL_DRIVER_DLLESPEC intxclLockDevice(xclDeviceHandle handle)

Get exclusive ownership ofthe device • xclDeviceHandle


0 on success or appropriateerror numberThe lock is necessary beforeperforming buffer migration,register access or bitstreamdownloads.

XCL_DRIVER_DLLESPEC intxclUnlockDevice(xclDeviceHandle handle)

Release exclusive ownershipof the device • xclDeviceHandle



XCL_DRIVER_DLLESPEC intxclBootFPGA(xclDeviceHandle handle)

Boot the FPGA from PROM • xclDeviceHandlehandle: Device handle

0 on success or appropriateerror numberThis should only be calledwhen there are no otherclients. It will cause PCIe busre-enumeration

HAL Buffer Management APIsBuffer management APIs are used for managing device memory and migrating buffers betweenhost and device memory





API Description Parameters ReturnXCL_DRIVER_DLLESPECunsigned intxclAllocBO(xclDeviceHandle handle, size_t size,xclBOKind domain, unsignedflags)

Allocate a BO of requestedsize with appropriate flags • xclDeviceHandle


• size_t size: Size ofbuffer

• xclBOKind domain:Memory domain

• unsigned flags:Specify bankinformation, etc

BO handle

XCL_DRIVER_DLLESPECunsigned intxclAllocUserPtrBO(xclDeviceHandle handle, void *userptr, size_t size, unsignedflags)

Allocate a BO using userptrprovided by the user • xclDeviceHandle


• void * userptr:Pointer to 4K aligneduser memory


• unsigned flags:Specify bankinformation, etc

BO handle

XCL_DRIVER_DLLESPEC voidxclFreeBO(xclDeviceHandle handle, unsigned intboHandle)

Free a previously allocatedBO • xclDeviceHandle


• unsigned intboHandle: BO handle

XCL_DRIVER_DLLESPEC size_txclWriteBO(xclDeviceHandle handle, unsigned intboHandle, const void * src,size_t size, size_t seek)

Copy-in user data to hostbacking storage of BO • xclDeviceHandle



• const void * src:Source data pointer

• size_t size: Size ofdata to copy

• size_t seek: Offsetwithin the BO

0 on success or appropriateerror numberCopy host buffer contents topreviously allocated devicememory. seek specifieshow many bytes to skip atthe beginning of the BObefore copying-in sizebytes of host buffer.





API Description Parameters ReturnXCL_DRIVER_DLLESPEC size_txclReadBO(xclDeviceHandle handle, unsigned intboHandle, void * dst, size_tsize, size_t skip)

Copy-out user data fromhost backing storage of BO • xclDeviceHandle



• void * dst:Destination data pointer


• size_t skip: Offsetwithin the BO

0 on success or appropriateerror numberCopy contents of previouslyallocated device memory tohost buffer. skip specifieshow many bytes to skip fromthe beginning of the BObefore copying-out sizebytes of device buffer.

XCL_DRIVER_DLLESPEC void*xclMapBO(xclDeviceHandle handle, unsigned intboHandle, bool write)

Memory map BO into user’saddress space • xclDeviceHandle



• bool write: READonly or READ/WRITEmapping

Memory mapped bufferMap the contents of thebuffer object into hostmemory To unmap thebuffer call POSIX unmap()on mapped void * pointerreturned from xclMapBO

XCL_DRIVER_DLLESPEC intxclSyncBO(xclDeviceHandle handle, unsigned intboHandle,xclBOSyncDirection dir, size_tsize, size_t offset)

Synchronize buffer contentsin requested direction • xclDeviceHandle



• xclBOSyncDirection dir: To device orfrom device

• size_t size: Size ofdata to synchronize

• size_t offset:Offset within the BO

0 on success or standarderrnoSynchronize the buffercontents between host anddevice. Depending on thememory model this mayrequire DMA to/from deviceor CPU cache flushing/invalidation

XCL_DRIVER_DLLESPEC intxclExportBO(xclDeviceHandle handle, unsigned intboHandle)

Obtain DMA-BUF filedescriptor for a BO • xclDeviceHandle


• unsigned intboHandle: BO handlewhich needs to beexported

File handle to the BO orstandard errnoExport a BO for import intoanother device or Linuxsubsystem which acceptsDMA-BUF fd This operationis backed by Linux DMA-BUFframework





API Description Parameters ReturnXCL_DRIVER_DLLESPECunsigned intxclImportBO(xclDeviceHandle handle, int fd, unsignedflags)

Obtain BO handle for a BOrepresented by DMA-BUF filedescriptor

• xclDeviceHandlehandle: Device handle

• int fd: File handle toforeign BO owned byanother device whichneeds to be imported

• unsigned flags:Unused

BO handle of the importedBOImport a BO exported byanother device.This operation is backed byLinux DMA-BUF framework.

HAL Legacy Buffer Management APIs

IMPORTANT!: Do not develop new features using the following 5 API’s. These are for backwardscompatibility with classic HAL interface and will be deprecated in future. New clients should use BObased APIs defined in HAL Device Management APIs

API Description Parameters ReturnXCL_DRIVER_DLLESPECuint64_txclAllocDeviceBuffer(xclDeviceHandle handle,size_t size)

Allocate a buffer on thedevice • xclDeviceHandle



Physical address of buffer ondevice or0xFFFFFFFFFFFFFFFF in caseof failureAllocate a buffer on thedevice DDR and return itsaddress. This API will bedeprecated in future. UsexclAllocBO() in all newcode.

XCL_DRIVER_DLLESPECuint64_txclAllocDeviceBuffer2 (xclDeviceHandle handle,size_t size,xclMemoryDomains domain,unsigned flags)

Allocate a buffer on thedevice on a specific DDR • xclDeviceHandle



• xclMemoryDomainsdomain: Memorydomain

• unsigned flags:Desired DDR bank as abitmap

Physical address of buffer ondevice or0xFFFFFFFFFFFFFFFF in caseof failureAllocate a buffer on a specificdevice DDR and return itsaddress. This API will bedeprecated in future. UsexclAllocBO() in all newcode.





API Description Parameters ReturnXCL_DRIVER_DLLESPEC voidxclFreeDeviceBuffer(xclDeviceHandle handle,uint64_t buf)

Free a previously buffer onthe device • xclDeviceHandle


• uint64_t buf:Physical address ofbuffer

The physical address shouldhave been previouslyallocated byxclAllocDeviceBuffe() orxclAllocDeviceBuffer2(). The address shouldpoint to the beginning of thebuffer and not at an offset inthe buffer. This API will bedeprecated in future. UsexclFreeBO() togetherwith BO allocation APIs.

XCL_DRIVER_DLLESPEC size_txclCopyBufferHost2Device(xclDeviceHandlehandle, uint64_t dest, constvoid * src, size_t size, size_tseek)

Write to device memory • xclDeviceHandlehandle: Device handle

• uint64_t dest:Physical address in thedevice

• const void * src:Source buffer pointer


• size_t seek: Seekwithin the segmentpointed to physicaladdress

Size of data moved orstandard error numberCopy host buffer contents topreviously allocated devicememory. seek specifieshow many bytes to skip atthe beginning of thedestination before copyingsize bytes of host buffer.This API will be deprecatedin future. UsexclSyncBO() togetherwith other BO APIs.

XCL_DRIVER_DLLESPEC size_txclCopyBufferDevice2Host(xclDeviceHandlehandle, void * dest, uint64_tsrc, size_t size, size_t skip)

Read from device memory • xclDeviceHandlehandle: Device handle

• void * dest:Destination bufferpointer

• uint64_t src:Physical address in thedevice


• size_t skip: Skipwithin the segmentpointed to physicaladdress

Size of data moved orstandard error numberCopy contents of previouslyallocated device memory tohost buffer. skip specifieshow many bytes to skip fromthe beginning of the sourcebefore copying size bytesof device buffer. This API willbe deprecated in future. UsexclSyncBO() togetherwith other BO APIs.





HAL Unmanaged DMA APIsUnmanaged DMA APIs are for exclusive use by the debuggers and tools. The APIs allow clinetsto read/write from/to absolute device address. No checks are performed if a buffer was allocatedbefore at the specified location or if the address is valid. Users who want to take over the fullmemory managemnt of the device may use this API to synchronize their buffers between hostand device.

API Description Parameters ReturnXCL_DRIVER_DLLESPECssize_txclUnmgdPread(xclDeviceHandle handle, unsignedflags, void * buf, size_t size,uint64_t offset)

Perform unmanaged devicememory read operation • xclDeviceHandle



• void * buf:Destination data pointer


• uint64_t offset:Absolute offset insidedevice

size of bytes read orappropriate error numberThis API may be used toperform DMA operationfrom absolute locationspecified. Users may use thisif they want to perform theirown device memorymanagement – not using thebuffer object (BO)framework defined before.

XCL_DRIVER_DLLESPECssize_txclUnmgdPwrite(xclDeviceHandle handle, unsignedflags, const void * buf, size_tsize, uint64_t offset)

Perform unmanaged devicememory read operation • xclDeviceHandle



• const void * buf:Source data pointer


• uint64_t offset:Absolute offset insidedevice

size of bytes written orappropriate error numberThis API may be used toperform DMA operation toan absolute locationspecified. Users may use thisif they want to perform theirown device memorymanagement – not using thebuffer object (BO)framework defined before.

XCL_DRIVER_DLLESPEC size_txclWrite(xclDeviceHandle handle, xclAddressSpacespace, uint64_t offset, constvoid * hostBuf, size_t size)

Perform register writeoperation • xclDeviceHandle


• xclAddressSpacespace: Address space

• uint64_t offset:Offset in the addressspace

• const void *hostBuf: Source datapointer


size of bytes written orappropriate error numberThis API may be used towrite to device registersexposed on PCIe BAR. Offsetis relative to the the addressspace. A device may havemany address spaces.





API Description Parameters ReturnXCL_DRIVER_DLLESPEC size_txclRead(xclDeviceHandlehandle, xclAddressSpacespace, uint64_t offset, void *hostbuf, size_t size)

Perform register readoperation • xclDeviceHandle


• xclAddressSpacespace: Address space

• uint64_t offset:Offset in the addressspace

• void * hostbuf:Destination data pointer


Size of bytes written orappropriate error numberThis API may be used to readfrom device registersexposed on PCIe BAR. Offsetis relative to the the addressspace. A device may havemany address spaces.

HAL Compute Unit Execution Management APIsThese APIs are under development. These functions will be used to start compute units and waitfor them to finish.

API Description Parameters ReturnXCL_DRIVER_DLLESPEC intxclRegisterInterruptNotify(xclDeviceHandlehandle, unsigned intuserInterrupt, int fd)

register eventfd file handlefor a MSIX interrupt • xclDeviceHandle


• unsigned intuserInterrupt:MSIX interrupt number

• int fd: Eventfd handle

0 on success or standarderrnoSupport for non managedinterrupts (interrupts fromcustom IPs). fd should beobtained from eventfdsystem call. Caller should usestandard poll/read eventfdframework in order to waitfor interrupts. The handlesare automaticallyunregistered on process exit.

Using the OpenCL Runtime FlowThe SDAccel OpenCL runtime provides OpenCL 1.2 embedded profile conformant runtime API.The runtime is layered on top of common low-level software interface also called the XilinxOpenCL Hardware Abstraction Layer or XCL HAL.

The SDAccel OpenCL runtime uses a layered hierarchy to stitch together the components whichmake up the OpenCL system. The diagram in the following figure explains the roles of thedifferent layers.





Figure 38: OpenCL DSA Layers

API Layer

Core Runtime (scheduling)

Hardware Abstraction Layer (HAL)

Linux Kernel Driver(kernel space)

XilinxSimulator

Embedded Runtime (ERT)(hardware)

XRT

libxilinxopencl.so

libxclgemdrv.solibcpu_em.solibhw_em.so

xocl.koxclmgmt.ko

sched.bin

X20206-011218

OpenCL Kernel Execution Steps1. Initialize runtime.

2. OpenCL application downloads bitstream to the device.

3. OpenCL application allocates OpenCL buffers in host memory.

4. Runtime effects PCIe DMA transfer to send the buffer from host memory to device memory.

5. Runtime signals the kernel to start via the AXI Slave port.

6. Runtime starts polling the device to monitor for done signal.

7. Kernel executes and loads/stores data from device memory.

8. When kernel is finished running, it changes the status to done.

9. Runtime effects PCIe DMA transfer to read updated buffers from device memory to hostmemory.

10. OpenCL application reads the buffer received from memory.

PCIe Kernel Driver for ManagementPhysical FunctionInterfaces exposed by xclmgmt driver are defined in file, mgmt-ioctl.h. Core functionalityprovided by xclmgmt driver is described in the following table:

# Functionality ioctl request code data format1 FPGA image download XCLMGMT_IOCICAPDOWNLOAD_AXLF xclmgmt_ioc_bitstream_axlf





# Functionality ioctl request code data format2 CL frequency scaling XCLMGMT_IOCFREQSCALE xclmgmt_ioc_freqscaling

3 PCIe hot reset XCLMGMT_IOCHOTRESET NA

4 CL reset XCLMGMT_IOCOCLRESET NA

5 Live boot FPGA from PROM XCLMGMT_IOCREBOOT NA

6 Device sensors (current, voltage andtemperature)

NA hwmon(xclmgmt_microblaze andxclmgmt_sysmon) interface on sysfs

7 Querying device errors XCLMGMT_IOCERRINFO xclErrorStatus

struct xclmgmt_ioc_info

Obtain information from the device used with XCLMGMT_IOCINFO ioctl

Definition Members Descriptionstructxclmgmt_ioc_info{};

Note that this structure will beobsoleted in future and the samefunctionality will be exposed via sysfsnodesstructxclmgmt_ioc_bitstream_axlf

load xclbin (AXLF) device image usedwithXCLMGMT_IOCICAPDOWNLOAD_AXLFioctl

structxclmgmt_ioc_bitstream_axlf{structaxlf*xclbin;};

• xclbin

Pointer to user’s xclbin structure inmemory

• structxclmgmt_ioc_freqscaling

scale frequencies on the boardusing Xilinx clock wizard used withXCLMGMT_IOCFREQSCALE ioctl

structxclmgmt_ioc_freqscaling{unsignedocl_region;unsignedshortocl_target_freq;};

• ocl_region

PR region (currently only 0 issupported)

• ocl_target_freq

Array of requested frequencies, avalue o zero in the array indicatesleave untouched





User Physical Function

This file defines ioctl command codes and associated structures for interacting with xocl PCIdriver for Xilinx FPGA platforms.

Device memory allocation is modeled as buffer objects (bo). For each bo driver tracks the hostpointer backed by scatter gather list – which provides backing storage on host – and thecorresponding device side allocation of contiguous buffer in one of the memory mapped DDRs/BRAMs, etc.

xocl driver functionality is described in the following table. All the APIs are multi-threading andmulti-process safe.

# Functionality ioctl request code data format1 Allocate buffer on device DRM_IOCTL_XOCL_CREATE_BO drm_xocl_create_bo

2 Allocate buffer on device with userptr DRM_IOCTL_XOCL_USERPTR_BO drm_xocl_userptr_bo

3 Prepare bo for mapping into user’saddress space

DRM_IOCTL_XOCL_MAP_BO drm_xocl_map_bo

4 Synchronize (DMA) buffer contents inrequested direction

DRM_IOCTL_XOCL_SYNC_BO drm_xocl_sync_bo

5 Obtain information about bufferobject

DRM_IOCTL_XOCL_INFO_BO drm_xocl_info_bo

6 Update bo backing storage withuser’s data

DRM_IOCTL_XOCL_PWRITE_BO drm_xocl_pwrite_bo

7 Read back data in bo backing storage DRM_IOCTL_XOCL_PREAD_BO drm_xocl_pread_bo

8 Unprotected write to device memory DRM_IOCTL_XOCL_PWRITE_UNMGD drm_xocl_pwrite_unmgd

9 Unprotected read from devicememory

DRM_IOCTL_XOCL_PREAD_UNMGD drm_xocl_pread_unmgd

10 Obtain device usage statistics DRM_IOCTL_XOCL_USAGE_STAT drm_xocl_usage_stat

11 Register eventfd handle for MSIXinterrupt

DRM_IOCTL_XOCL_USER_INTR drm_xocl_user_intr

API Description Definition Membersstructdrm_xocl_create_bo

Create buffer object usedwithDRM_IOCTL_XOCL_CREATE_BO ioctl

structdrm_xocl_create_bo{uint64_tsize;uint32_thandle;uint32_tflags;};

• size: Requested size ofthe buffer object

• handle: bo handlereturned by the driver

• flags:DRM_XOCL_BO_XXX flags





API Description Definition Membersstructdrm_xocl_userptr_bo

Create buffer object withuser’s pointer used withDRM_IOCTL_XOCL_USERPTR_BO ioctl

structdrm_xocl_userptr_bo{uint64_taddr;uint64_tsize;uint32_thandle;uint32_tflags;};

• addr: Address of bufferallocated by user

• size: Requested size ofthe buffer object

• handle: bo handlereturned by the driver

• flags:DRM_XOCL_BO_XXX flags

structdrm_xocl_map_bo

Prepare a buffer object formmap used withDRM_IOCTL_XOCL_MAP_BOioctl

structdrm_xocl_map_bo{uint32_thandle;uint32_tpad;uint64_toffset;};

• handle: bo handle

• pad: Unused

• offset: ‘Fake’ offsetreturned by the driverwhich can be used withPOSIX mmap

structdrm_xocl_sync_bo

Synchronize the buffer in therequested direction betweendevice and host used withDRM_IOCTL_XOCL_SYNC_BOioctl

structdrm_xocl_sync_bo{uint32_thandle;uint32_tflags;uint64_tsize;uint64_toffset;enumdrm_xocl_sync_bo_dirdir;};


• flags: Unused

• size: Number of bytesto synchronize

• offset: Offset into theobject to synchronize

• dir:DRM_XOCL_SYNC_DIR_XXX

structdrm_xocl_info_bo

Obtain information about anallocated buffer obbject usedwithDRM_IOCTL_XOCL_INFO_BOIOCTL

structdrm_xocl_info_bo{uint32_thandle;uint32_tflags;uint64_tsize;uint64_tpaddr;};


• flags: Unused

• size: Size of bufferobject (out)

• paddr: Physical address(out)

struct drm_xocl_axlf load xclbin (AXLF) deviceimage used withDRM_IOCTL_XOCL_READ_AXLF ioctl

structdrm_xocl_axlf{structaxlf*xclbin;};

• xclbin: Pointer touser’s xclbin structure inmemory

Note: This ioctlwill be removedin next release





API Description Definition Membersstructdrm_xocl_pwrite_bo

Update bo with user’s dataused withDRM_IOCTL_XOCL_PWRITE_BO ioctl

structdrm_xocl_pwrite_bo{uint32_thandle;uint32_tpad;uint64_toffset;uint64_tsize;uint64_tdata_ptr;};


• pad: Unused

• offset: Offset into thebuffer object to write to

• size: Length of data towrite

• data_ptr: User’spointer to read the datafrom

structdrm_xocl_pread_bo

Read data from bo used withDRM_IOCTL_XOCL_PREAD_BO ioctl

structdrm_xocl_pread_bo{uint32_thandle;uint32_tpad;uint64_toffset;uint64_tsize;uint64_tdata_ptr;};


• pad: Unused

• offset: Offset into thebuffer object to readfrom

• size: Length of data toread

• data_ptr: User’spointer to write the datainto

structdrm_xocl_pwrite_unmgd

unprotected write to devicememory used withDRM_IOCTL_XOCL_PWRITE_UNMGD ioctl

structdrm_xocl_pwrite_unmgd{uint32_taddress_space;uint32_tpad;uint64_tpaddr;uint64_tsize;uint64_tdata_ptr;};

• address_space:Address space in theDSA; currently only 0 issuported

• pad: Unused

• paddr: Physical addressin the specified addressspace


• data_ptr: User’spointer to read the datafrom





API Description Definition Membersstructdrm_xocl_pread_unmgd

Unprotected read fromdevice memory used withDRM_IOCTL_XOCL_PREAD_UNMGD ioctl

structdrm_xocl_pread_unmgd{uint32_taddress_space;uint32_tpad;uint64_tpaddr;uint64_tsize;uint64_tdata_ptr;};

• address_space:Address space in theDSA; currently only 0 isvalid

• pad: Unused

• paddr: Physical addressin the specified addressspace


• data_ptr: User’spointer to write the datato

structdrm_xocl_usage_stat

obtain device memory usageand DMA statistics used withDRM_IOCTL_XOCL_USAGE_STAT ioctl

structdrm_xocl_usage_stat{unsigneddma_channel_count;unsignedmm_channel_count;uint64_th2c;uint64_tc2h;structdrm_xocl_mm_statmm;};

• dma_channel_count: How many DMAchannels are present

• mm_channel_count:How many storage banks(DDR) are present

• h2c: Total datatransferred from host todevice by a DMA channel

• c2h: Total datatransferred from deviceto host by a DMAchannel

• mm: BO statistics for astorage bank (DDR)

structdrm_xocl_user_intr

Register user’s eventfd forMSIX interrupt used withDRM_IOCTL_XOCL_USER_INTR ioctl

structdrm_xocl_user_intr{uint32_tctx_id;intfd;intmsix;};

• ctx_id: Pass 0

• fd: File descriptorcreated with eventfdsystem call

• msix: User interruptnumber (0 to 15)

• DRM_IOCTL_XOCL_CREATE_BO():

Linux Sys FileSystem Nodesxocl and xclmgmt drivers expose several sysfs nodes under the pci device root node.





xocl

xocl driver exposes various sections of xclbin image which is currently loaded on sysfs includingthe xclbin. The data layout of these sections are defined in xclbin.h and are part of AXLFspecification.

1. ip_layout Exposes IP LAYOUT section of xclbin

2. connectivity Exposes CONNECTIVITY section of xclbin

3. mem_topology Exposes MEM TOPOLOGY section of xclbin

4. xclbinid Exposes xclbin unique identifier

xclmgmt

Device sensors are exposed as standard hwmon file hierarchy. Two hwmon nodes are created:sysmon and microblaze. sysmon exposes device temperature and voltages. microblaze exposesdevice currents on various rails by using an embedded board management firmware. Values with_input suffix represent live values. Thie values are compatible with Linux standard lm-sensors tool.

For example if the bus number of mgmt physical function is 0000:01:00.1 then hwmon wouldshow up under /sys/bus/pci/devices/0000:01:00.1. See session log below:

dx4300:~>tree -L 1 /sys/bus/pci/devices/0000:01:00.1/hwmon/sys/bus/pci/devices/0000:01:00.1/hwmon├├├ hwmon3├├├ hwmon4

2 directories, 0 filesdx4300:~>cat /sys/bus/pci/devices/0000:01:00.1/hwmon/hwmon3/namexclmgmt_sysmondx4300:~>cat /sys/bus/pci/devices/0000:01:00.1/hwmon/hwmon4/namexclmgmt_microblazedx4300:~>sensorsxclmgmt_microblaze-pci-0101Adapter: PCI adaptercurr1: +4.47 A (avg = +0.00 A, highest = +4.63 A)curr2: +4.52 A (avg = +0.00 A, highest = +4.80 A)curr3: +3.12 A (avg = +0.00 A, highest = +3.36 A)curr4: +0.00 A (avg = +0.00 A, highest = +0.00 A)curr5: +1.00 A (avg = +0.00 A, highest = +1.00 A)curr6: +0.50 A (avg = +0.00 A, highest = +0.50 A)

xclmgmt_sysmon-pci-0101Adapter: PCI adapterin0: +0.93 V (lowest = +0.92 V, highest = +0.95 V)in1: +1.79 V (lowest = +1.78 V, highest = +1.80 V)in2: +0.94 V (lowest = +0.92 V, highest = +0.95 V)temp1: +46.6°C (lowest = +36.1°C, highest = +48.8°C)





Building Software StackSystem Requirements• RHEL/CentOS 7.3

• Ubuntu 16.04.3

IMPORTANT!: The key software components making up the software stack are provided as zip files.These files can be found in your SDx installation directory under data/sdaccel/pcie/src.

xoclunzip xocl.zipcd driver/xclng/drm/xoclmakesudo make install

xclmgmtunzip xclmgmt.zipcd driver/xclng/mgmtmakesudo make install

HALunzip xclgemhal.zipcd driver/xclng/user_gemmake

xbsak

Xilinx Board Swiss Army Knife

unzip xclgemhal.zipcd driver/xclng/tools/xbsak_gemmake

Formatsxclbin

xclbin container format is defined in xclbin.h. Since 2017.1 tools use xclbin2 format also known asAXLF format.





Feature ROM

Feature ROM is a BRAM in the device which describes basic properties of the device including itsname and features enabled. The format for the data in Feature ROM is defined inxclfeatures.h.





Chapter 10

Assembling the PlatformAssembling the platform is currently a manual process. Place the kernel driver zip files and HALlibrary binary in the relevant directory for your platform. For example, the following tree viewshows the platform files for xilinx_xil-accel-rd-vu9p_dynamic_5_0.

Once the DSA file has been created for the hardware platform, it needs to be associated with thesoftware platform configuration files and drivers. This is achieved by creating the requireddirectory structure and populating it with the associated files.

The following directory structure reflects the required directory hierarchy and files in order forthe SDx™Environment to recognize the various pieces of the SDAccel™platform.




Figure 39: Directory Hierarchy for SDAccel Platforms

Chapter 10: Assembling the Platform




Appendix A

Using the KCU1500 ReferenceDesign

The KCU1500 platform reference design targets the Kintex® UltraScale™ KCU1500Acceleration Development Board, which uses a Kintex UltraScale KU115 device.

Refer to the README file included with the reference design for instructions on implementingthe Vivado® design to produce the DSA compatible with SDx™. While the majority of theplatform characteristics follow the descriptions and flow in the preceding document, thefollowing are unique to the KCU1500 platform.

Reference Design Files

Design data is in the associated Reference Design File.

Platform Characteristics for KCU1500Connectivity to the host computer uses the Xilinx DMA subsystem for PCI Express® (PCIe),containing a Scatter-Gather DMA and PCI Express 3.x integrated bock. The Kintex® UltraScale™KCU1500 Acceleration development board supports Gen3 x8 connectivity.

Four UltraScale Memory IP instances enable access to available DDR4 SDRAM. The KintexUltraScale KCU1500 Acceleration development board contains four channels of DDR4-2400SDRAM, at 4GB per channel for a total of 16GB global memory.

Floorplan for KCU1500The Kintex® UltraScale™ KU115 device uses SSI Technology and contains two super logicregions (SLRs).


https://www.xilinx.com/member/forms/download/design-license.html?cid=d6cb49e0-6fcf-41b7-8804-baa54353244f&filename=ug1164-sdaccel-platform-development.zip



The static base region of the design is floorplanned to the bottom-right corner of the lower SLR,corresponding to the location of the utilized PCI Express 3.x Integrated Block. Two of the fourDDR4 SDRAM memory controller IP instances are floorplanned to the lower SLR, in regionsaround the High Performance I/O banks they utilize. The remaining two DDR4 SDRAM memorycontroller IP instances are floorplanned to the upper SLR, in regions around the HighPerformance I/O banks they utilize.

XDMA IP Configuration for KCU1500For compatibility with drivers supporting the DSA for the KCU1500 board, the following XDMAsettings are required:

• Vendor ID: 0x10EE

• Device IDs: 0x4B88 and 0x4B87

• Subsystem ID: 0x4350

SDx Memory Subsystem IP Configurationfor KCU1500The SDx Memory Subsystem IP manages connectivity between the XDMA, user kernels, andDDR4 memory controllers. It also manages the IP configurations of the DDR4 memorycontrollers. The DSA_INFO environment variable must be set as follows.

{ child_pblock_declarations { pblock_ddr4_mem00 { range CLOCKREGION_X2Y0:CLOCKREGION_X2Y3 parent pblock_dynamic_SLR0 cell_paths memory/ddr4_mem00 } pblock_ddr4_mem01 { range CLOCKREGION_X4Y2:CLOCKREGION_X4Y4 parent pblock_dynamic_SLR0 cell_paths memory/ddr4_mem01 } pblock_ddr4_mem02 { range CLOCKREGION_X2Y6:CLOCKREGION_X2Y9 parent pblock_dynamic_SLR1 cell_paths memory/ddr4_mem02 } pblock_ddr4_mem03 { range CLOCKREGION_X4Y6:CLOCKREGION_X4Y9 parent pblock_dynamic_SLR1 cell_paths memory/ddr4_mem03 } }

Appendix A: Using the KCU1500 Reference Design




slr_pblock_map { SLR0 pblock_dynamic_SLR0 SLR1 pblock_dynamic_SLR1 } board_component_slr_map { xilinx.com:kcu1500:ddr4_sdram_c0:1.1 SLR0 xilinx.com:kcu1500:ddr4_sdram_c1:1.1 SLR0 xilinx.com:kcu1500:ddr4_sdram_c2:1.1 SLR1 xilinx.com:kcu1500:ddr4_sdram_c3:1.1 SLR1 }}





Appendix B

Using the VCU1525 ReferenceDesign

The VCU1525 platform reference design targets the Virtex UltraScale+ VCU1525 AccelerationDevelopment Board, which uses a Virtex UltraScale+ VU9P device.

Refer to the README file included with the reference design for instructions on implementingthe Vivado® design to produce the DSA compatible with SDx™. While the majority of theplatform characteristics follow the descriptions and flow in the preceding document, thefollowing are unique to the VCU1525 platform.

Reference Design Files

Design data is in the associated Reference Design File.

Platform Characteristics for VCU1525Connectivity to the host computer uses the Xilinx DMA subsystem for PCI Express® (PCIe),containing a Scatter-Gather DMA and PCI Express 3.x integrated bock. The Virtex® UltraScale+™ VCU1525 Acceleration development board supports Gen3 x16 connectivity.

Four UltraScale Memory IP instances enable access to available DDR4 SDRAM. The VirtexUltraScale+ VCU1525 Acceleration Board contains four channels of DDR4-2400 SDRAM, at16GB per channel for a total of 64GB global memory.


https://www.xilinx.com/member/forms/download/design-license.html?cid=d6cb49e0-6fcf-41b7-8804-baa54353244f&filename=ug1164-sdaccel-platform-development.zip



Floorplan for VCU1525The Virtex® UltraScale+™ VU9P device uses SSI Technology and contains three super logicregions (SLRs).

The static base region of the design is floorplanned to the right half of the middle SLR,corresponding to the location of the utilized PCI Express 3.x Integrated Block.

• One of the four DDR4 SDRAM memory controller IP instances is floorplanned to the lowerSLR, in regions around the High Performance I/O banks it utilizes.

• One of the four DDR4 SDRAM memory controller IP instances is floorplanned to the upperSLR, in regions around the High Performance I/O banks it utilizes.

• The remaining two DDR4 SDRAM memory controller IP instances are floorplanned to themiddle SLR, in regions around the High Performance I/O banks they utilize.

○ Of the two DDR4 SDRAM memory controller IP instances in the middle SLR, one instanceresides within the static base region. The SDx Memory Subsystem IP facilitatesconnectivity to this instance by providing an AXI pass-through port to it.

XDMA IP Configuration for VCU1525For compatibility with drivers supporting the DSA for the KCU1500 board, the following XDMAsettings are required:

• Vendor ID: 0x10EE

• Device IDs: 0x6A90 and 0x6A8F

• Subsystem ID: 0x4350

Appendix B: Using the VCU1525 Reference Design




SDx Memory Subsystem IP Configurationfor VCU1525The SDx Memory Subsystem IP manages connectivity between the XDMA, user kernels, andDDR4 memory controllers. It also manages the IP configurations of the DDR4 memorycontrollers. The DSA_INFO environment variable must be set as follows. Theexcluded_board_components and axi_passthrough fields define the handling of the DDR4memory controller instance which resides in the static region, by enabling an AXI pass-throughport to that instance.

{ child_pblock_declarations { pblock_ddr4_mem00 { range CLOCKREGION_X2Y1:CLOCKREGION_X2Y3 parent pblock_dynamic_SLR0 cell_paths memory/ddr4_mem00 } pblock_ddr4_mem01 { range CLOCKREGION_X2Y7:CLOCKREGION_X2Y9 parent pblock_dynamic_SLR1 cell_paths memory/ddr4_mem01 } pblock_ddr4_mem02 { range CLOCKREGION_X4Y11:CLOCKREGION_X4Y13 parent pblock_dynamic_SLR2 cell_paths memory/ddr4_mem02 } } slr_pblock_map { SLR0 pblock_dynamic_SLR0 SLR1 pblock_dynamic_SLR1 SLR2 pblock_dynamic_SLR2 } board_component_slr_map { xilinx.com:vcu1525:ddr4_sdram_c0:1.0 SLR0 xilinx.com:vcu1525:ddr4_sdram_c1:1.0 SLR1 xilinx.com:vcu1525:ddr4_sdram_c2:1.0 SLR1 xilinx.com:vcu1525:ddr4_sdram_c3:1.0 SLR2 } excluded_board_components { xilinx.com:vcu1525:ddr4_sdram_c1:1.0 } axi_passthrough { xilinx.com:vcu1525:ddr4_sdram_c1:1.0 { offset 0x400000000 range 16G slr SLR1 } }}





Appendix C

New Static Region Features

Embedded Scheduler

Host CPU

Command Queue MicroBlaze Embedded Scheduler SW

Embedded Scheduler HW

CU-DMA

CU-ISR

X20187-011018

Functional Description

The Embedded scheduler is a MicroBlaze™ kernel task scheduler that is part of the static region.Previously, scheduling of kernels and polling for their completion was handled by the RunTimesoftware running on the the host. In 2017.4 release, we have added a MicroBlaze in the staticregion that can handle the job of kernel scheduling, copy kernel arguments and identifyingcompletion through polling or interrupts, This is called the Embedded scheduler and it consists ofthe following components :

1. Command Queue

2. MicroBlaze Embedded Scheduler SW

3. Embedded Scheduler HW

4. CU DMA

5. CU ISR




Command Queue

The command Queue is an AXI BRAM memory which is accessible to Host, MicroBlaze SW &CUDMA. The Command Queue is a dynamic queue which is used to store the command packetinformation from the host which indicate a particular workload for a compute unit. Theinformation in the command packet includes various configuration information and the computeunit register map which is required by the compute unit to perform a given task. The CommandQueue is dynamically divided into equivalent slot size based on the type of compute unit presenton the platform and the size of the kernel register-map.

Embedded Scheduler Software

The firmware on the MicroBlaze keeps track of all the activities which occur on the FPGA. TheMicroBlaze gets an interrupt whenever a new slot information is written in the Command Queueby the Host. Once the MicroBlaze gets an interrupt, it notifies the Embedded Scheduler HWmodule by setting a bitmask associated with a particular slot in the register map. It then changesthe status of the particular slot to running, until it receives another interrupt from the EmbeddedScheduler HW module indicating a completion of tasks for a given slot.

If there are multiple compute units on the platform, the scheduler will make sure that all computeunits are operational 100 % of the time. The scheduler also supports out of order execution ofcommand slots incase multiple slots are targeting a CU which is current busy hence improvingthe overall execution.

Embedded Scheduler Hardware

The Embedded Scheduler HW module is responsible for configuring and starting the computeunits. It also keeps track of completion status of each compute unit and notifies the host once aparticular slot is completed via an interrupt. It also masters the CU-DMA and CU-ISR moduleswhich accelerate CU configuration and Interrupt handling.

Compute Unit DMA

The CU-DMA module is an AXILITE DMA module which reads the configuration information ineach slot and copies the relevant payload from the command Queue memory to the computeunit memory. It also starts the compute unit after it has configured it with the relevantparameters from the command Queue.

Compute Unit ISR

The CU-ISR module is an interrupt handler module which is responsible for receiving andservicing interrupts from multiple compute units. When CU-ISR receives a completion interruptfrom the compute unit, it services the interrupt by reading the done bit of the compute unit andalso clearing the ISR bit which is set by the compute unit upon completion. In turn, it notifiesEmbedded Scheduler HW regarding a completion , which then updates the completion statusregister accordingly.

Appendix C: New Static Region Features




Management MicroBlazeFigure 40: Management MicroBlaze

Host(XDMA)

MicroBlazeMemory

ManagementMicroBlaze

InterruptController

Register Map(AXI BRAM)

AXI I2C Controller

SYSMON(Voltage, Temp)

MMCMs(Kernel Clocks)

AXI GPIOx4

AXI UART

GPIOx2(Interrupts)

Board Peripherals(Fan, Regulators,

DIMMs)

BMC(TIMSP432)

AXI_lite

I2C

Host ↔ MicroBlaze Communication

Varies board to board

UART Interrupt

BMC ↔ MicroBlaze Communication

Supported on VCU1525 only.Communication protocol TBD

SYSMON Alerts

PMBUS Alert

MicroBlaze Alerts TBD

X20189-011118

The Management MicroBlaze™ is responsible for board monitoring and control of certainperipherals. Depending on whether the board has an external microcontroller (BMC), it mighthave additional functionality. This feature is specific to the VCU1525 board because it relies onknowledge of the board schematics. It could be easily extended to another board with I2Cconnectivity with a change in firmware.

To enable field updates of the MicroBlaze image, the MicroBlaze device memory can be accessedfrom the host. The host requests the MicroBlaze device to stop from the register map, and whenthe MicroBlaze device acknowledges the new image can then be updated dynamically from thehost. Note that UltraScale+™ platforms have 128KB of memory for the image, and UltraScaledevices have 64KB of memory.

Future Feature Support

1. Monitor Current and Voltage through I2C. (1ms polling)





2. Respond to thermal and voltage alerts from SYSMON.

3. Respond to I2C alerts from board.

4. Control and autoscale MMCMs when power threshold is reached.

5. Store monitoring information into shared register map. Host sends requests to MicroBlazethrough control registers in register map.

6. Support for 2 interrupts back to host.

Host Interrupt SupportIn SDAccel™ v5.0 dynamic platforms for 2017.4, the xDMA IP supports upto 16 individualinterrupt ports for interrupts to host using MSI-X.

These 16 interrupts have been divided into two equal parts:

• 8 interrupt pins are allocated to the userPF → usr_irq_req bit 0 to bit 7.

• 8 interrupt pins are allocated to the MgmtPF → usr_ir_req bit 8 to bit 15.

The UserPF connectivity is described in the following table:

Interrupt PinNumber

Function

0 Embedded Microblaze Command Queue Status Register 0 (0-31 slots)




4 Mailbox interrupt 0

5-7 Reserved for future use

The MgmtPF connectivity is described in the following table :

Interrupt PinNumber

Function

8 AXI4 Datapath Firewall Control and Status

9 UserPF AXI Lite Firewall Control and Status

10 Management PF AXI Lite Firewall Control and Status

11 Mailbox Interrupt 1

12 Board Management interrupt 0

13 Board Management interrupt 1

14-15 Reserved for future use





MailboxIn a multiprocessor environment, the processors need to communicate data with each other. Theeasiest method is to set up inter-processor communication through a mailbox. A similarmechanism is used for secure bidirectional communication between 2 PFs - management PF ofPF1, which is of higher privilege and user PF or PF0. In SDAccel™, the mailbox is configured touse the AXI Lite memory mapped interface.

XDMA

Mailbox

FIFO

FIFO

Management Space (PF1)

User Space(PF0)

AXI Interconnect

AXI Interconnect

Interrupts to Host

AXI Lite Interface AXI Lite Interface

X20207-011218

• Size of mailbox = 8KB (32 bits * 2048 depth)





• Configurable interrupt thresholds and maskable interrupts

• Synchronous operation - both AXI lite interfaces use the same 50 MHz AXI-lite clock

• Bidirectional communication

S0_AXI is connected to AXI-Lite interconnect in user space, S1_AXI is connected to managementspace. Interrupt_0 is connected to XDMA usr_irq_req[4] which is associated with user PF0 andInterrupt_1 is connected to XDMA usr_irq_req[11] which is associated with management PF1.

Frequency Counters for Scalable ClocksIn 2017.4, a shortcoming in SDAccel platforms was that there was a lack of means to confirm thefrequency of scalable clocks in hardware`. We now provide a means to verify clock frequencythrough the use of counters which are accessible through AXI Lite network in the user space. Thefrequency counter is a simple counter used to measure the MMCM output clock to verify correctconfiguration.

The diagram below shows the structure of the frequency counter:

The resolution of the frequency counter is determined by the comparator block in the abovediagram. For example if the reference clock is 100MHz and the stop criteria is set to 1e6 then theresolution will be 1kHz (100MHz/1e6=1kHz). Note that due to the nature of the block there willalways be 0-2 units of uncertainty when stopping the test clock counter. If the precision is 1kHzthen the measuring error can range from 0-2kHz. If the two clocks are not from the same sourcethen there will be additional measurement error based on the accuracy of the original clocksource for both clocks as well as jitter.

Note that the measuring time is proportional to the precision, i.e. 1kHz precision takes 1ms tomeasure.





Appendix D

Additional Resources and LegalNotices

Xilinx Resources

For support resources such as Answers, Documentation, Downloads, and Forums, see XilinxSupport.

Solution Centers

See the Xilinx Solution Centers for support on devices, software tools, and intellectual propertyat all stages of the design cycle. Topics include design assistance, advisories, and troubleshootingtips

References1. SDx Environments Release Notes, Installation, and Licensing Guide (UG1238)

2. SDAccel Environment User Guide (UG1023)

3. SDAccel Environment Optimization Guide (UG1207)

4. SDAccel Environment Tutorial: Introduction (UG1021)

5. SDAccel Environment Platform Development Guide (UG1164)

6. SDAccel Development Environment web page

7. Vivado® Design Suite Documentation

8. Vivado Design Suite User Guide: Designing IP Subsystems using IP Integrator (UG994)

9. Vivado Design Suite User Guide: Creating and Packaging Custom IP (UG1118)

10. Vivado Design Suite User Guide: Partial Reconfiguration (UG909)

11. Vivado Design Suite User Guide: High level Synthesis (UG902)

12. UltraFast Design Methodology Guide for the Vivado Design Suite (UG949)

13. Vivado Design Suite Properties Reference Guide (UG912)

14. Khronos Group web page: Documentation for the OpenCL standard


https://www.xilinx.com/support.html

https://www.xilinx.com/support.html

https://www.xilinx.com/support/solcenters.htm


https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug1023-sdaccel-user-guide.pdf

https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug1207-sdaccel-optimization-guide.pdf

https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug1021-sdaccel-intro-tutorial.pdf

https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug1164-sdaccel-platform-development.pdf

https://xilinx.com/products/design-tools/software-zone/sdaccel.html

https://www.xilinx.com/cgi-bin/docs/rdoc?t=vivado+docs


https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug1118-vivado-creating-packaging-custom-ip.pdf


https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug902-vivado-high-level-synthesis.pdf

https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug949-vivado-design-methodology.pdf

https://www.xilinx.com/cgi-bin/docs/rdoc?v=2017.4;d=ug912-vivado-properties.pdf

http://www.khronos.org



15. Alpha Data web page: Documentation for the ADM-PCIE-7V3 Card

16. Pico Computing web page: Documentation for the M-505-K325T card and the EX400 Card

Please Read: Important Legal NoticesThe information disclosed to you hereunder (the "Materials") is provided solely for the selectionand use of Xilinx products. To the maximum extent permitted by applicable law: (1) Materials aremade available "AS IS" and with all faults, Xilinx hereby DISCLAIMS ALL WARRANTIES ANDCONDITIONS, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TOWARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT, OR FITNESS FOR ANYPARTICULAR PURPOSE; and (2) Xilinx shall not be liable (whether in contract or tort, includingnegligence, or under any other theory of liability) for any loss or damage of any kind or naturerelated to, arising under, or in connection with, the Materials (including your use of theMaterials), including for any direct, indirect, special, incidental, or consequential loss or damage(including loss of data, profits, goodwill, or any type of loss or damage suffered as a result of anyaction brought by a third party) even if such damage or loss was reasonably foreseeable or Xilinxhad been advised of the possibility of the same. Xilinx assumes no obligation to correct anyerrors contained in the Materials or to notify you of updates to the Materials or to productspecifications. You may not reproduce, modify, distribute, or publicly display the Materialswithout prior written consent. Certain products are subject to the terms and conditions ofXilinx's limited warranty, please refer to Xilinx's Terms of Sale which can be viewed at https://www.xilinx.com/legal.htm#tos; IP cores may be subject to warranty and support terms containedin a license issued to you by Xilinx. Xilinx products are not designed or intended to be fail-safe orfor use in any application requiring fail-safe performance; you assume sole risk and liability foruse of Xilinx products in such critical applications, please refer to Xilinx's Terms of Sale which canbe viewed at https://www.xilinx.com/legal.htm#tos.

AUTOMOTIVE APPLICATIONS DISCLAIMER

AUTOMOTIVE PRODUCTS (IDENTIFIED AS "XA" IN THE PART NUMBER) ARE NOTWARRANTED FOR USE IN THE DEPLOYMENT OF AIRBAGS OR FOR USE IN APPLICATIONSTHAT AFFECT CONTROL OF A VEHICLE ("SAFETY APPLICATION") UNLESS THERE IS ASAFETY CONCEPT OR REDUNDANCY FEATURE CONSISTENT WITH THE ISO 26262AUTOMOTIVE SAFETY STANDARD ("SAFETY DESIGN"). CUSTOMER SHALL, PRIOR TO USINGOR DISTRIBUTING ANY SYSTEMS THAT INCORPORATE PRODUCTS, THOROUGHLY TESTSUCH SYSTEMS FOR SAFETY PURPOSES. USE OF PRODUCTS IN A SAFETY APPLICATIONWITHOUT A SAFETY DESIGN IS FULLY AT THE RISK OF CUSTOMER, SUBJECT ONLY TOAPPLICABLE LAWS AND REGULATIONS GOVERNING LIMITATIONS ON PRODUCTLIABILITY.

Appendix D: Additional Resources and Legal Notices


http://www.alpha-data.com/

http://www.picocomputing.com



Copyright

© Copyright 2016-2018 Xilinx®, Inc. Xilinx®, the Xilinx® logo, Artix, ISE, Kintex, Spartan,Virtex, Vivado, Zynq, and other designated brands included herein are trademarks of Xilinx in theUnited States and other countries.OpenCL and the OpenCL logo are trademarks of Apple Inc.used by permission by Khronos.All other trademarks are the property of their respective owners.





platform development guide - xilinx€¦ · sdaccel platform development guide 5 ug1164 (v2017.4)...

Documents