slideshare - pcie
TRANSCRIPT
LINUX PCI EXPRESS DRIVER
Background
PCI Express (Peripheral Component Interconnect Express), officially abbreviated as PCIe, is a high-speed serial computer expansion bus standard designed to replace the older PCI, PCI-X, and AGP bus standards.
At the software level, PCI Express preserves backward compatibility with PCI; legacy PCI system software can detect and configure newer PCI Express devices without explicit support for the PCI Express standard, though new PCI Express features are inaccessible.
Topology
Note: figure from https://en.wikipedia.org/wiki/PCI_Express
Terminology
Lane A lane is composed of two differential signaling pairs, with one pair for receiving data and
the other for transmitting. 1, 2, …, 32 lanes will denote x1, x2, …, x32. Root Complex
the device that connects the CPU and memory subsystem to the PCI Express fabric. It may support one or more PCI Express ports.
Endpoint devices other than root complex and switches that are requesters or completers of PCI
Express transactions. Configuration Space
PCI devices have a set of registers referred to as configuration space and PCI Express introduces extended configuration space for devices.
BAR Base Address Registers (commonly called BARs) to inform the device of its address
mapping by writing configuration commands to the PCI controller. INTx
An in-band messages which emulate the four physical interrupt signals (INTA-INTD) routed between PCI devices and the system interrupt controller
MSI/MIS-X An alternative in-band method of signaling an interrupt, using special in-band messages
to replace traditional out-of-band assertion of dedicated interrupt lines.
Configuration Space Layout
Note: figure from http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2013/20130812_PreConfD_Onufryk.pdf
Configuration Space Header
Status
BIST HeaderType
LatencyTimer
CacheLine Size
Class Code RevID
Device ID Vendor ID
Command
Base Address 0 (BAR0)
Base Address 1 (BAR1)
Base Address 2 (BAR2)
Base Address 3 (BAR3)
Base Address 4 (BAR4)
Base Address 5 (BAR5)
CardBus CIS Pointer
SubsystemDevice ID
SubsystemVendor ID
Expansion ROM Base Address
Reserved CapabilityPointer
Reserved
Max Lat Min Gnt InterruptPin
InterruptLine
00h
04h
08h
0Ch
10h
14h
18h
1Ch
20h
24h
28h
2Ch
30h
38h
34h
3Ch
07152331
Status
BIST HeaderType
LatencyTimer
CacheLine Size
Class Code RevID
Device ID Vendor ID
Command
Base Address 0 (BAR0)
Base Address 1 (BAR1)
SecondaryLat timer
SecondaryStatus
(Non-Prefetchable)Memory LimitPrefetchableMemory Limit
Pretetchable Memory BaseUpper 32 Bits
Prefetchable Memory LimitUpper 32 Bits
IO LimitUpper 16 Bits
Reserved CapabilityPointer
Expansion ROM Base Address
Max Lat Min Gnt InterruptPin
InterruptLine
00h
04h
08h
0Ch
10h
14h
18h
1Ch
20h
24h
28h
2Ch
30h
38h
34h
3Ch
07152331
Type 0 Header Type 1 Header
SubordinateBus #
SecondaryBus #
PrimaryBus #
IOLimit
IOBase
(Non-Prefetchable)Memory BasePrefetchableMemory Base
IO BaseUpper 16 Bits
Capabilities List
Example: http://www.alterawiki.com/wiki/PCI_Configuration_Space
Device Layers and their Associated Packets
Note: figure from http://resources.infosecinstitute.com/system-address-map-initialization-x86x64-architecture-part-2-pci-express-based-systems/
Layer Packets Relationship
Note: figure from http://www.verien.com/pcie_primer.htm
Function of Each PCIE Layer Transaction Layer
Mechanisms for differentiating the ordering and processing requirements of Transaction Layer Packets (TLPs)
Credit-based flow control TLP construction and processing Association of transaction-level mechanisms with device resources
including Flow Control and Virtual Channel management. Data Link Layer
Data Exchange Error Detection and Retry Initialization and power management
Physical Layer Logical Sub-block – symbol encoding, framing, data scrambling, Link
initialization and training, Lane to lane de-skew Electrical Sub-block – defines the physical layer of PCI Express that
consists of a reference clock source, Transmitter, channel, and Receiver.
Packet Routing
Address Routing - Memory and IO requests Address routing is used to transfer data to or from
memory, memory mapped IO, or IO locations. ID Routing - Completions and Configuration
ID routing is based on the logical position (Bus Number, Device Number, Function Number) of a device function within the PCI bus topology.
Implicit Routing - Message requests Implicit routing is based on the intrinsic knowledge
PCI Express devices are required to have concerning upstream and downstream traffic and the existence of a single PCI Express Root Complex at the top of the PCI Express topology.
pciutils – lspci command
Command “lspci” is a utility for displaying information
about PCI buses in the system and devices connected to them
Command “lspci” popular options -t Show bus tree -v Be verbose (-vv for very verbose) -k Show kernel drivers handling each device -x Show hex-dump of the standard part of
the config space
pciutils – lspci example
lspci -vkx02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720
Gigabit Ethernet PCIe Subsystem: Broadcom Corporation Device 2003 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at f0050000 (64-bit, prefetchable) [size=64K] Memory at f0040000 (64-bit, prefetchable) [size=64K] Memory at f0030000 (64-bit, prefetchable) [size=64K] Expansion ROM at f7c40000 [disabled] [size=256K] Capabilities: [48] Power Management version 3 Capabilities: [50] Vital Product Data Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [a0] MSI-X: Enable- Count=17 Masked- Capabilities: [ac] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [13c] Device Serial Number 00-00-00-0a-f7-82-bf-
82 Capabilities: [150] Power Budgeting <?> Capabilities: [160] Virtual Channel Kernel driver in use: tg300: e4 14 5f 16 06 00 10 00 00 00 00 02 10 00 80 0010: 0c 00 05 f0 00 00 00 00 0c 00 04 f0 00 00 00 0020: 0c 00 03 f0 00 00 00 00 00 00 00 00 e4 14 03 2030: 00 00 c4 f7 48 00 00 00 00 00 00 00 0b 01 00 00
Controller Driver
Find and assign resource Controller registers BARs for devices (device tree format -
http://devicetree.org/Device_Tree_Usage#PCI_Address_Translation ) Operation implementation
PCIe configuration access struct pci_ops (defined in include/linux/pci.h)
INTx interrupts struct irq_domain_ops (defined in include/linux/irqdomain.h) struct irq_chip (defined in include/linux/irq.h)
MSI/MSI-X interrupts struct irq_domain_ops struct irq_chip
Example – PCIe host controller driver for Samsung EXYNOS SoCshttps://github.com/torvalds/linux/blob/master/drivers/pci/host/pci-exynos.c + pcie-designware.c
Enumeration Process – Discovering the Topology After a system reset or power up,
configuration software has to scan the PCIe fabric to discover the machine topology and learn how the fabric is populated. Try to read device’s Vendor ID register starting
from bus 0, device 0, function 0 to bus 255. A device is found, update device’s configuration
Primary Bus Number Register Secondary Bus Number Register Subordinate Bus Number Register
Enumeration software must perform a depth-first search.
Single-Root System Enumeration
Note: figure from https://codywu2010.wordpress.com/2015/11/29/how-modern-multi-processor-multi-root-complex-system-assigns-pci-bus-number/
Device Driver - Initialization Steps
Sequence Action Function1 Enable the device pci_enable_device()2 Enable DMA by setting the bus master bit in the
PCI_COMMAND registerpci_set_master()
3 Request MMIO/IOP resources pci_request_region()4 Set the DMA mask size (for both coherent and
streaming DMA)pci_set_dma_mask()
5 Allocate and initialize shared control data pci_ioremap_bar()6 Access device configuration space pci_read_config_xxx()/
pci_write_config_xxx()7 Register IRQ handler request_irq()
pci_enable_msi()pci_enable_msix()
8 Initialize non-PCI (i.e. LAN/SCSI/etc parts of the chip)
9 Enable DMA/processing engines
PCIe + non-PCIe
Note: How To Write Linux PCI Drivers - https://www.kernel.org/doc/Documentation/PCI/pci.txt
Device Driver – Access Methods PCI/PCIe configuration access
pci_read_config_xxx()/pci_write_config_xxx()
Non-PCIe functionality (i.e. LAN…) registers access readx()/writex()
Example – Broadcom Tigon3 ethernet driver https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/broadcom/tg3.c
Common Port Service Drivers
Function Version
Kconfig Port Bus Driver
Files
Native Hotplug (HP)
2.6.11 HOTPLUG_PCI_PCIE
Yes drivers/pci/hotplug/pciehp*
Advanced Error Reporting (AER)
2.6.19 PCIEAER Yes drivers/pci/pcie/aer/*
Active State Power Management (ASPM) /ASPM Policy
2.6.26
3.4
PCIEASPM
PCIEASPM_*
No drivers/pci/pcie/aspm.c
Power Management Event (PME)
2.6.34 PCIE_PME Yes drivers/pci/pcie/pme/*Note:
The PCI Express Port Bus Driver Guide HOWTO - https://www.kernel.org/doc/Documentation/PCI/PCIEBUS-HOWTO.txt,
PCI Express Port Bus Driver Support for Linux - https://www.kernel.org/doc/ols/2005/ols2005v2-pages-9-18.pdf
Reference Resource
• PCI Express Specifications https://www.pcisig.com/specifications/pciexpress/
• PCI/PCIe http://en.wikipedia.org/wiki/PCI_Express http://en.wikipedia.org/wiki/PCI_configuration_space http://en.wikipedia.org/wiki/Conventional_PCI
PCI Vendor(/Device) ID https://pcisig.com/membership/member-companies https://pci-ids.ucw.cz/ http://www.pcidatabase.com/index.php
• pciutils - Linux User Space PCI configuration space access tool https://www.kernel.org/pub/software/utils/pciutils/
BOOK - PCI Express Technology 3.0 https://www.mindshare.com/Books/Titles/PCI_Express_Technology_3.
0