debugging programs that use atomic blocks and transactional memory

Ferad Zyulkyarov1,2, Tim Harris3, Osman S. Unsal1, Adrián Cristal1, Mateo Valero1,2

1BSC-Microsoft Research Centre2Universitat Politècnica de Catalunya

3Microsoft Research Cambridge

15th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming09-14 January 2010 – Bangalore

Debugging Programs that use Atomic Blocks and Transactional Memory

2

Our Motivation

• It is difficult to debug transactional applications with existing debuggers

• Existing debuggers are not aware of atomic blocks and transactional memory– Lessons learnt from developing complex TM

applications such as Atomic Quake[1] and Quake TM [2].

[1] Zyulkyarov et al. “Atomic Quake: Using Transactional Memory in an Interactive Multiplayer Game Server“, PPoPP'09[2] Gajinov et al. “QuakeTM: Parallelizing a Complex Serial Application Using Transactional Memory “, ICS'09

3

Overview

• Debugging atomic blocks atomically• Examining the TM state• Managing the TM state at debug-time• Conflict point discovery• Design and implementation

4

Debugging at the Level of Atomic Blocks

• Debugger is extended with the semantics of atomic blocks– Atomicity – atomic blocks are treated as

single instruction– Isolation – the user does not observe

intermediate results of concurrently running transactions

• Hides the low level implementation details of atomic blocks (TM or lock inference)

5

Atomicity in Debugging• Step over atomic blocks as if single instruction.• Good for debugging sync errors at granularity of atomic blocks vs. individual statements inside the atomic blocks.

<statement 1><statement 2><statement 3>atomic { <statement 4> <statement 5> <statement 6> <statement 7>}<statement 8><statement 9>

<statement 1><statement 2><statement 3>atomic { <statement 4> <statement 5> <statement 6> <statement 7>}<statement 8><statement 9>

Current Debugger TM Aware Debugger

6

Using Existing DebuggersSV_LinkEdict1 atomic {2 InitializeBoundingBox(ent);3 SV_FindTouchedLeafs(ent, sv.worldmodel->nodes);4 node = FindNewLocation(ent, sv_areanode);5 InsertLinkBefore(&ent->area, &node->trigger_edicts);6 } // end atomic

sv_areanode

Want to move from A to B and take the keys.

Compute the bounding box. Find the leafs the

bounding box maps to.

Find location BAnother player took the keys

before us.

Change the location and take the keys.

Conflict. Rollback.

Re-execute.

7

Using TM Aware DebuggerSV_LinkEdict1 atomic {2 InitializeBoundingBox(ent);3 SV_FindTouchedLeafs(ent, sv.worldmodel->nodes);4 node = FindNewLocation(ent, sv_areanode);5 InsertLinkBefore(&ent->area, &node->trigger_edicts);6 } // end atomic

sv_areanode

Moved from A to B and took the keys.

8

Isolation in Debugging• What if we want to debug wrong code within atomic

block?– Put breakpoint inside atomic block.– Validate the transaction– Step within the transaction.

• The user does not observe intermediate results of concurrently running transactions– Switch transaction to irrevocable mode after validation.

atomic { <statement 1> <statement 2> <statement 3> <statement 4>}

9

Debugging Wrong Code Inside Atomic Blocks

• Why isolation is important?InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; }}

1 5 6

10

Example Importance of Isolation

InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; }}

1 5 64

InsertSorted(4)

11

Example Importance of Isolation

InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; }}

1 5 6

4

12

Example Importance of Isolation

InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; }}

1 3 6

4

Another transaction changed the value to 3. The current TX will be doomed

and operate on invalid data.

13

Example Importance of Isolation

InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; }}

1 3 6

4

Confuse the user because of consuming speculative values and not detecting conflict.

14

Why Isolate Speculative State?

InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; }}

1 3 6

4

Confuse the user because of consuming speculative values and not detecting conflict.

15

Why Isolate Speculative State?

InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; }}

1 3 6

4Conflict.Rollback.

Re-execute.

16

• Debugging atomic blocks atomically• Examining the TM state• Managing the TM state at debug-time• Conflict point discovery• Design and implementation

17

Debugging at the Level of Transactions

• Assumes that atomic blocks are implemented with transactional memory.

• Examine the internal state of the TM– Read/write set, re-executions, status

• TM specific watch points– Break when conflict happens– Filters

• Concurrent work with Herlihy and Lev [PACT’ 09].

18

Querying the TM State

atomic { <statement 1> <statement 2> <statement 3> <statement 4>}

Read Set0x30FA000x30FE160x320B0A0x4A11F8

Write Set0x30FE160x3AEE0D0x4A1124

Status: Active

Re-executions: 1Query TM

StatePriority: 1Nesting: 1

19

TM Specific Watchpoints

atomic { <statement 1> <statement 2> <statement 3> <statement 4>}

Conflict Information

Conflicting Threads: T1, T2Address: 0x84D2F0Symbol: reservation@04Readers: T1Writers: T2

Break when conflict happens

Filter: Break ifAddress = reservation@04Thread = T2

AND

20

• Debugging atomic blocks atomically• Examining the TM state• Managing the TM state at debug-time• Conflict point discovery• Design and implementation

21

Managing Transactions at Debug-Time

• At the level of atomic blocks– Debug time atomic blocks– Splitting atomic blocks

• At the level of transactions– Changing the state of TM system (i.e. adding

and removing entries from read/write set, change the status, abort)

• Analogous to the functionality of existing debuggers to change the CPU state

22

Debug Time Atomic Blocks

• Create and remove atomic blocks while debugging.

• Useful to investigate and patch synchronization errors such as data races, atomicity, and order violation.

• User marks the places that should execute atomically.

23

Example Debug Time Atomic Bblocks<statement 1><statement 2><statement 3><statement 4><statement 5><statement 6><statement 7><statement 8><statement 9><statement 10><statement 11><statement 12><statement 13><statement 14>

24

Example Debug Time Atomic Bblocks<statement 1><statement 2><statement 3>StartDebugAtomic<statement 4><statement 5><statement 6><statement 7><statement 8><statement 9>EndDebugAtomic<statement 10><statement 11><statement 12><statement 13><statement 14>

User marks the startand the end of thetransactions

25

Data Race

Thread 1

atomic { local = counter; local++; counter = local;}

Thread 2

atomic { local = counter; local++;}counter = local;

StartDebugAtomicatomic { local = counter; local++;}counter = local;EndDebugAtomic

26

Split Atomic Block

• Useful to find unnecessarily large atomic blocks or searching for data races

atomic { s1;

s2;}

atomic { s1;<split> s2;}

atomic { s1;}

atomic { s2;}

27

• Debugging atomic blocks atomically• Examining the TM state• Managing the TM state at debug-time• Conflict point discovery• Design and implementation

28

Conflict Point Discovery• TM applications have unanticipated

overheads– Problem raised by Pankratius [talk at ICSE’09]

and Rossbach et al. [PPoPP’10]• Tells the users at which source lines how

many conflicts happen.– Useful to profile and optimize TM applications.

• More comprehensive than Reach Points Gajinov et al. [ICS’ 09]

29

Example OutputFile:Line #Conf. Method LineHashtable.cs:51 152 Add If (_container[hashCode]…

Hashtable.cs:48 62 Add uint hashCode = HashSdbm(…

Hashtable.cs:53 5 Add _container[hashCode] = n …

Hashtable.cs:83 5 Add while (entry != null) …

ArrayList.cs:79 3 Contains for (int i = 0; i < count; i++ )

ArrayList.cs:52 1 Add if (count == capacity – 1) …

30

Optimizing Genome

1 100.0

0.5

1.0

1.5

2.0

2.5

3.084.5

UnoptimizedWithUninitializedBucketsWithInitializedBuckets

#threads

Nor

mal

ized

exe

cutio

n tim

e

31

• Debugging atomic blocks atomically• Examining the TM state• Managing the TM state at debug-time• Conflict point discovery• Design and implementation

32

DesignDebugger Process

WinDbg

TmDbgExt

DbgEng

Target Process

Program

TmTargetDbg

StmLib

ShowRead set

Implements debugger commands

such as setting watchpoints, query

and modify the STM.

Wrapper around the STM to hide its implementation

details.

Function call on the target process.

33

Conclusion

• New principles and approaches for debugging TM applications– Debugging at the level of atomic blocks– Debugging at the level of transactions– Managing transactions at debug-time

• Conflict point discovery– Optimized Genome

• Generic and decoupled design

Край

Slides available at www.bscmsrc.eu

Backup Slides

35

36

Implementing Conflict Point Discovery

atomic { a = 5;}

Addr: 01 : StartAtomic();02 : OpenObjForWrite(a);03 : a = 5;04 : CommitAtomic();

Obj. Addr.

0x40D8E2

…

…

Ret. Addr Counter

0x03 152

… …

… …

CompilerInstrumentation

Conflict

DbgEng

Addr = 0x03

Hashtable.cs:51

Translate tosource line

Return Addr.

0x03

…

…

Log the address of a for conflict

detection.

To know where conflict happens log the return

address of OpenObjForRead.

37

Internal Breakpoint

• Debugger breakpoint used to implement– Atomicity– Watchpoints– Splitting atomic blocks– Debug time atomic blocks

38

Breakpoint-time Callback• Distinguishes ordinary

breakpoints from internal breakpoints

• Overrides the debugger behavior of suspending the target process

• May execute complementary actions

39

Example Use of Internal Breakpoint

atomic{ <statement 1> <statement 2> <statement 3> <statement 4> <statement 5> <statement 6> <statement 7> <statement 8>}

Internal Breakpoint -> StartAtomic()

-> Commit()

Stepping over atomic block as if a single instruction

40

Atomicity Violations

initially a = b = 0; Thread 1 Thread 2

1 atomic{2 a++;3 } atomic {4 a++;5 atomic{ }6 b--;7 assert(a + b == 0);8 }

41

Atomicity Violations

initially a = b = 0; Thread 1 Thread 2

StartDebugAomic1 atomic{2 a++;3 } atomic {4 a++;5 atomic{ }6 b--;7 assert(a + b == 0);8 } EndDebugAtomic

42

Extending the Scope of Atomic Blocks

• We can build over debug time transactions to allow users to extend the scope of existing compile time atomic blocks.

<statement 1><statement 2>atomic { <statement 3> <statement 4>}<statement 5><statement 6><statement 7>

<statement 1><statement 2>atomic { <statement 3> <statement 4> <statement 5> <statement 6>}<statement 7>

<statement 1><statement 2>StartDebugTx <statement 3> <statement 4> <statement 5> <statement 6>EndDebugTx<statement 7>

We want How we do

43

Shrinking the Scope

• The implementation of shrinking the scope of atomic blocks might be tricky for STMs because of the code instrumentation.

<statement 1><statement 2>atomic { <statement 3> <statement 4> <statement 5> <statement 6>}<statement 7>

<statement 1><statement 2><statement 3>atomic { <statement 4> <statement 5>}<statement 6><statement 7>

<statement 1><statement 2>StartTx - Ignore <statement 3>StartDebugTx <statement 4> <statement 5>EndDebugTx <statement 6>EndTx - Ignore<statement 7>

We want How we do

44

Debugging Wrong Code Inside Atomic Blocks?

SV_LinkEdict1 atomic {2 InitializeBoundingBox(ent);3 SV_FindTouchedLeafs(ent, sv.worldmodel->nodes);4 node = FindNewLocation(ent, sv_areanode);5 InsertLinkBefore(&ent->area, &node->trigger_edicts);6 } // end atomic

InsertLinkBefore(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); }}

45

Isolating Speculative State

5

4Insert(3)

5

4

3

Insert(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); }}

3

46

Isolating Speculative State

5

4Insert(3)

5

2

3

Insert(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); }}

Another transaction changed the value to 2.The current TX will be doomed and operate

on invalid data.

47

Insert(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); }}

Isolating Speculative State

5

2

32

3

RotateRight

5

Mislead the user because of consuming speculative values and not detecting

conflict.

48

Isolating Speculative State

5

2

3

FixColors

5

2

3

Insert(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); }}

Conflict.Rollback.

Re-execute.