CS510 – Concurrent Systems 1

Performance of memory reclamation for lockless synchronization

By Thomas E. Hart, Paul E. McKenney, Angela Demke Brown, Jonathan Walpole

Handwaved about by Jim Cotillier

The Problem

Why not just stick with classical locks?o Performance issues (blocking)o CAS-class instruction overheado Susceptible to:

• Deadlock• Priority Inversion• Convoying

Lockless synchronization addresses this, but is exposed to Read/Reclaim Races

o Reclamation of shared data elements without coordination with all contenders leads to an inconsistent global state

• Such ex post facto references to deleted data yield unpredictable results

Uncoordinated reclamation…

Some Approaches to Solutions

QSBR –- Quiescent State Based Reclamation EBR/NEBR –- Epoch Based Reclamation HPBR –- Hazard Pointers Based Reclamation LFRC –- Lock-free Reference Counting

Functionality provided by a client/library interface

o But no single, invariant set of interface semantics exists across all schemes

QSBR

Permits the reclamation of data only after a time interval elapses, called a Grace Period

QSBR defines a Grace Period to be the temporal interval (a,b), such that any data element deleted before a can be reclaimed after b

A Quiescent State is a state of a thread, T, in which T holds no references to shared elements, active or deleted (zombie)

Any interval in which each thread passes through a Quiescent State is a QSBR Grace Period

Three-thread QSBR example…

QSBR Fuzzy Barriers

Protect access to “protected” code which no thread should execute before all other threads reach a specified point

Do not absolutely block, a la hard barriers, only prevent execution of “protected” code until barrier opens

Thus, can be used to synchronize reclamation

Using QSBR

Client explicitly declares Quiescent State:

… and thereby enters a fuzzy barrier

Problem: thread failure

A dead thread cannot call quiescent_state() and thus can force QSBR to block…

EBR (Fraser)

Uses Grace Periods, like QSBRo But does not rely upon explicit client Quiescent State

declarations, as QSBR does Encapsulates lockless operations within Critical

Sectionso …which the client explicitly declares, via the functions

critical_enter() and critical_exit() Counts the number of Critical Region

invocations, and then attempts to enter a fuzzy barrier to reclaim memory

Linked list search using EBR

EBR Epochs

Epochs are modeled after [3], the group of equivalence classes modulo 3

Epochs are hierarchical: Global and Local Each epoch has an associated zombie element

list Fuzzy barrier for reclamation is entered upon

entry to each new epoch A thread entering a Critical Region updates its

Local Epoch to match the global epoch After M (magic number) LE updates, a thread will

attempt to increment the GE

EBR Epochs Cont’d.

A GE update attempt only succeeds if the LE of each thread in a CR matches the GE

Since threads update their LE only at the start of a CR, whenever, for a thread T, its LE = GE, then all lockless operations of other threads in progress the last time T was in epoch GE have completed

Thankfully, a grace period has expired!

EBR Epoch Cycle

NEBR – a Modest EBR Improvement

EBR must pay for the expensive fences at the beginning and end of a CS

Modeled a little after QSBR: have the application set/reset a “critical section(s) may be in here” flag

o NEBR then does not “automatically” do this in each CSo “Application independence” dies in favor of performance

Reduces EBR’s overhead modestly--closer to QSBRo NEBR is attractive as the programmer’s responsibilities are

limited to marking sections that might contain lockless operations

HPBR/SMR (Michael)

Each thread T has (magic) K Hazard Pointers used to protect elements from reclamation by other threads

o Thus, for N threads, H = NK HP’s exist in totoo K is small, often 2 (queues and lists); 1 (stacks)

T caches removed elements privately in a list P of size (magic) R

After R removals, T reclaims each element in P that does not have a corresponding HP

If T fails, a maximum of K+R removed elements can be leaked

HPBR Paradigm

HPBR Paradigm Cont’d.

Hazardous References—references to shared elements that may now be zombies or ABA situations

o Algorithms using HPBR must identify a Hazardous Reference, set a Hazard Pointer, then check for element removal

o If an element has not been removed, it continues to be referentially safe

LFRC (Valois, Detlefs, et al.)

Threads track the instantaneous count of references to elements

o When count = 0, element can be reclaimed Many variations on this scheme may or may not

allow element types to change upon reclamationo May require type invariance (Valios); type

independence requires DCAS (Detlefs, et al.)o Zombies may consume unbounded memory

Performance may be worse than lock-basedo CAS, FAA (Intel: LOCK XADD) very expensive

Summary of Schemes

QSBR –- Detects grace periods using application-specified quiescent states

EBR -- Detects grace periods using application-independent epochs

HPBR –- Uses per-thread Hazard Pointers to synchronize reclamation

LFRC – Uses per-element reference counts to synchronize reclamation

Performance Factors…

Depends on a lot of stuffo Memory consistency and constraintso Workload, contention and thread scheduling

Sequentially consistent memory model is still generally assumed by the lock-free literature

o But the hardware trends are toward weaker models• Coder needs to rely on fences (MBarriers), which

artificially add overhead• HPBR, EBR and LFRC require per-operation fences, but not

QSBR—this is shown to be a distinct advantage

Performance Factors Cont’d.

Thread preemptiono Can start when number threads > number CPUs

Descheduled threads are blocked threads, as far as reclamation schemes are concerned

o Anything that prevents a Grace Period from closing is bad

Threads may sometimes need to borrow memory from a locked, global pool

o A thread may be preempted whilst holding such a lock; setting up a thread convoy on memory

o HPBR bounds memory stress and has an advantage here

The μBenchmark

The μBenchmark Cont’d.

Master thread flow logico Create N childreno Start a timero When timer expires, stop children

Average execution time/measured operation = test duration/number of operations

Net CPU time = execution time * number of threads

o If thread count > CPU count, report execution time; otherwise report CPU time.

Driver parameters were selected not to be biased toward any particular reclamation scheme

The μBenchmark Cont’d.

CS implemented on POWER via larx/stcx (LL/SC) Fences implemented via eieio (“Enforce In-order

Execution of I/O”) Spin locks implemented via cas and fences Statically allocated HPBR Hazard Pointers

o Some algorithms may require unbounded HP counts Choice of placement of QSBR QS declarations

may not be obvious in some algorithms

Performance Measurement Guidelines

Measure the base costs firsto Single-threaded execution, small data structures

• No contention, preemption, traversing long lists• Non-blocking queues, single-element linked lists…

Then move toward complexityo Pedagogical approach--try to change only one factor at

a timeo Consider the R/O, the W/O and the R/W cases in each of

the examined reclamation schemes

Base Performance Costs

Scalability with Fractional Workload

Scalability with Traversal Length

Scalability of LFRC

No Preemption; R/O Workload

No Preemption; W/O Workload

Preemption; R/O Workload

Preemption; W/O Workload

Memory Stress Busy Wait

Hash Tables; Update Fraction Workload

No Preemption; R/O Workload with NEBR

Case Study—RCU API in Linux

RCU concepts—”Read/Copy/Update”o Lockless concurrent reads with deferred destruction of

zombie elementso Writers may not prevent readers from accessing shared

datao Writers must coordinate with each other in some way

• RCU does not specify what wayo RCU neither blocks nor fails for readerso Preemptable kernels necessitate the use of

rcu_read_lock() and rcu_read_unlock()to toggle kernel preemption

• …so that context switches do not occur at intolerable times

Case Study—RCU Cont’d.

QSBR is a natural choice for memory reclamation

o EBR could be used as well, but would not offer any advantages over QSBR

RCU is best targeted to read-mostly data structures

o Rare updates imply rare reclamation

Case Study—RCU Cont’d.

SysV IPC subsystem implemented in Linux via CR-QSBR

o Implements semaphores, message queues and shared memory

o Apps use an integer Accessor ID to access in-kernel data structures (essentially a “resource handle”)

o The dynamic, mostly-read (AID/resource) array, formerly spinlocked in stock Linux, was protected here through CR-QSBR instead, and benchmarked

Case Study—RCU Cont’d.

Semopbench, 8-CPU, 700 MHz Intel P-III

Case Study—RCU Cont’d.

DBT1 Database Benchmark Raw Results

Case Study—RCU Cont’d.

DBT1 database benchmark results (TPS)

Conclusions

Reclamation has a huge effect on lockless algorithm performance

o So one must tune to the design of the application Both QSBR and EBR can suffer in the face of

memory exhaustion HPBR and EBR have higher base costs than

QSBR due to fences The NEBR enhancements modestly improve EBR LFRC has the highest overhead due to the per-

element atomic instruction requirement

Conclusons Cont’d.

HPBR scales poorly as the traversal length increases

QSBR is, overall, the best performing reclamation scheme

o …and best suited to an OS kernel environmento Lockless approaches using QSBR can widely

outperform locking approaches by a large margin

Rantings -- STAE

STAE – Specified Thread Abnormal Exito User provides Exit code to be run on condition of thread

error trapo Exit is driven by the etrap interrupt logic; Exit is called

immediately after etrap is detected, e.g., SEGVo Exit has full access to environment of failing thread; may

modify any data, etc.o Exit may:

• Allow failing thread to die (the status quo)• Resuscitate failing thread by telling the dispatcher to restart

the thread at an Exit-specified point in its code• Call a completely new program to run in place of failing

thread (with all of the failing thread’s credentials and context)

Rantings -- PLO

PLO – Perform Locked Operation (IBM z Platform)o Meta instruction that atomically encapsulates all of:

CAL, CAS, DCAS, CASAS, CASADS, CASATS

into single-instruction global atomicityo 32, 64, or 128-bit operands are supportedo Acquires a global hardware interlock unique to PLOo Is very powerful and flexible, but is so complex that it

may require a pre-built parameter list just to “program” it!

o Usually needs to be coded with a zillion operands o Its proprietary μalgorithm has to be huge, but whether

its utility outstrips its cost enough to yield a net gain in performance, has not yet been answered (afaik)

Questions/Musings

Suppose DCAS was “improved” so that it uses an order of magnitude fewer clocks than today.

o To what extent could macroscopically faster hardware atomicity affect the utility of these lockless schemes?

Could the STAE formalism provably solve the failed thread blocking problem in QSBR?

o If you believe the answer is yes, based on the empirical data in this paper, would the paradigm (QSBR+STAE) satisfy Ockham’s Razor and thus become the overall best solution to the lockless reclamation problem?