washington washington university in st louis [email protected] overview of the apic pacer

WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

[email protected]

Overviewof the

APIC Pacer

2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

John DeHart- 04/21/23 11:47

APIC Pacing: General Stuff• Pacing is for Transmit Channels only• Cells are NOT Paced out onto the wire

– Not Exactly

• Pacing is done on the PCI bus • Pacing is not a Guarantee, it is just a Restriction• Pacing Calculations include the ATM headers

– But not the APIC header

• More APIC Info:– http://www.arl.wustl.edu/arl/projects/msr/pdf/ApicArch.pdf


John DeHart- 04/21/23 11:47

APIC Pacing: General Stuff• Two pacer controls:

– Global Pacing• APIC Pacing Parameter register (Global, 0x208)

– Per VC Pacing• TX Channel Pacing Parameter Register (TX, 0x500XX68)

– XX is the Channel ID

• Three types of Channels:– Low Delay (Highest Priority)

– Paced

– Best Effort (Lowest Priority)

• All channels are paced by the Global Pacing• Paced Channels also use Per VC Pacing


John DeHart- 04/21/23 11:47

APIC Data Transfers• APIC pulls data from memory across the PCI bus in

Batches of cells.– The number of cells in a Batch is controlled by a register

• The Pacer identifies when it is time to transmit data and which connection should transmit

• Pacer “wakes up” every 14 PCI Bus clock ticks– checks to see if it is time to transmit

• Controlled by the Global APIC Pacing Parameter (APP)

– If it is time to transmit, it takes the first connection off the previously sorted list of keys and transmits its data.

• A lot of gory details about keys and heap storage of connections is not going to be included here. Read Rex’s documentation and/or read the VHDL if you want that level of detail


John DeHart- 04/21/23 11:47

Global Pacing Parameter• Pacing parameters are 24 bits

– 16 bits of Integer– 8 bits of fractional part

• Global Apic Pacing Parameter (APP)

(256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz)

APP = --------------------------------------------------------

(14 * ClockEstimate * LinkRateMbps)

[Items in formula explained on next slide]


John DeHart- 04/21/23 11:47

Explanation of Expression (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz)APP = -------------------------------------------------------- (14 * ClockEstimate * LinkRateMbps)

• 256 : shifts left by 8 bits to set “decimal point”• BatchSz: How many cells per transfer• 53*8: Translate cells/second into bits/second• 8192, InternalClockMhz (85MHz), ClockEstimate

– APIC counts how many of its internal 85MHz clock ticks take place during the time it takes for 8192 PCI bus clock ticks. This value is the ClockEstimate.

– PCI Bus Clock Rate in MHz = (8192 * 85)/ClockEstimate

• 14: # of PCI Bus Ticks in a Pacer Period• LinkRateMbps: Our target rate[Example on next 2 slides]


John DeHart- 04/21/23 11:47

Example: Units in the APP Formula (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz)

APP = --------------------------------------------------------


(256 * Cells * Bytes/Cell * Bits/Byte * 8192 * M/sec)

APP = --------------------------------------------------------

(14 * 1 * MBits/sec)


John DeHart- 04/21/23 11:47

Example: APP for 1Gb/s Link Rate (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz)

APP = --------------------------------------------------------


• BatchSz=8• 53*8: Translate cells/second into bits/second• InternalClockMhz = 85MHz• ClockEstimate = 20954 (typical value)• LinkRateMbps: 1000 (1000 Mb/s == 1Gb/s) (256 * 8 * 53 * 8 * 8192 * 85)

APP = ---------------------------------------- = 2061.15

(14 * 20954 * 1000)

APP = 2061 = 0x80D


John DeHart- 04/21/23 11:47

Example: APP for 1Gb/s Link RateAPP = 2061 = 0x80D

This means that every 14*8 = 112 PCI Bus clock ticks the APIC will be able to pull 8 Cells worth of data across the PCI Bus.

(8 Cells)/(112 * 30ns) = (3392 bits)/(3360ns) ~= 1Gb/s


John DeHart- 04/21/23 11:47

Per VC Pacing• Per VC Pacing Parameter

– What portion of the full link rate can be used– e.g. an integer value of 2 means that this channel can

use half the link rate

• If you change the Pacing Paramter for a VC:– It takes affect the next time the OLD pacing expires!!!

• Conceptually like this:

This TxChannelis Ready

to TransmitBATCH Cells

Countto 14

Countto APP

Countto TX

ChannelPacing

Parameter

33 MHzPCI BusClock


John DeHart- 04/21/23 11:47

Per VC Pacing

XXX XXX

time

Expiredconnections

X

oldExpirationTim + vcPacingParameter newExpirationTime

current pacedTime

vcPacingParameter ~ 10 OneAPICPacingPeriod


John DeHart- 04/21/23 11:47

pacedTime• pacedTime is incremented every global

pacing cycle in which a non-LowDelay connection wins contention

• Example with two connections:– (L) Low Delay at 1/24th of the global rate– (P) Paced at 1/6th of the global rate (.1666667)

L

P

0 6 12 18 24 30 36 42 48 54 60 66 72 78 84

P P P P P P P P P P P P P P

L L L


John DeHart- 04/21/23 11:47

pacedTime (continued)

0 6 12 18 24 30 36 42 48 54 60 66 72 78 84

P P P P P P P P P P P P P P P

L L L L

• We might expect the Paced channel to miss its exact turn and fire on the next global pacing interval but keeps it next expiration on the (0,6,12,18,…) boundaries.

• But…


John DeHart- 04/21/23 11:47

pacedTime (continued)

0 6 12 18 24 30 36 42 48 54 60 66 72 78 84

P P P P P P P P P P P P P P P

L L L L

• Actual rate for Paced connection:– (GlobalRate) * (3*(1/6) + 1*(1/7))/4 – (GlobalRate) * (.1607)

• For a Global Rate of 24Mb/s (DQ test example)– 24 * .1607 = 3.8568

0 5 11 17 22 28 34 40 45 51 57 63 68 74 80

“Real” time

pacedTime

t+

t+


John DeHart- 04/21/23 11:47

Per VC Pacing• If you change the Pacing Paramter for a VC:

– It takes affect the next time the OLD pacing expires!!!

– A Resume of the VC will cause an immediate change to the new Pacing parameter

• This artifact causes some strange behavior for DQ where there are large changes in the Per-VC pacing parameters.– Going from a low rate to a high rate can be delayed by

several DQ periods.

– This can cause the results to appear flawed.

• See the next slide for an illustration.


John DeHart- 04/21/23 11:47

Per VC Pacing

X

time

X

JustTransmitted

NextTransmissionby oldPacing

Paramter

Right after,Want to ChangePacing so it willTransmit every

other PacingInterval

X X X X

These willTransmit

This will not transmit


John DeHart- 04/21/23 11:47

Example of a Pacing Oddity• Suppose we have a channel on which we

are sending single cell packets at a rate of 2 cells every pacing period for that channel and the BATCH size is 1 cell so that the channel should only send 1 cell during each pacing period.

D D D D D DD

• You would expect the connection to build up a backlog, but it doesn’t……


John DeHart- 04/21/23 11:47

Example of a Pacing Oddity (con’t)• Turns out the Driver does a RESUME each time it

puts data in an empty transmit queue to restart it. • A RESUME causes the ExpireTime to be set to

the current PacedTime.• This causes the channel to be expired at the very

next Pacer Period.• Thus the channel transmits at twice its expected

rate

D D D D D DDR R R R R RR

T T T T T TT

washington washington university in st louis [email protected] overview of the apic pacer

Documents