testing multithreaded java applications for synchronization problems, ista 2011

www.istabg.orgwww.vmware.com www.istabg.orgwww.vmware.com

Testing Multithreaded Java Applications for

Synchronization Problems

Vassil Popovski

www.istabg.orgwww.vmware.com

Why multithreading?

A modern CPU

…core 1

cache

core N

cache

shared memory


Why multithreading?

…Thread 1 Thread N

Java heap

A modern JavaApplication

synchronization

thread 1 thread N

Java heap


• Non-determinism

• New types of defects exist such as deadlocks, livelocks and race conditions

New concept -> new problems


Thread 2Thread 1

Thread interleaving

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2


T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2 T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2


Number of different thread interleavings


Q: What is the number of all interleavings for 3 threads with 2 blocks each?

QUIZ#1


• safety: nothing bad happens

• liveness: something good eventually happens

What to test for?


Image source: http://www.doc.ic.ac.uk/~jnm/book/ppt/ch7.ppt

Example 1: Single Lane Bridge

• safety:– no car crash

• liveness:– every car eventually get an opportunity to cross the bridge

http://www.doc.ic.ac.uk/~jnm/book/ppt/ch7.ppt


Example 2: BoundedBuffer

Producer 1

Producer 2

Producer 3

Producer N

Consumer 1

Consumer 2

Consumer M…

…

• safety:– If empty, must not allow Get(); If full, must not allow Put()

• liveness (quiz#2):– Put() on empty and Get() on full will eventually be allowed


How to test for synchronization issues?Why

• NASA’s Remote Agent

• Therac-25


How to test for synchronization issues?

• Load/Stress testing (blackbox, whitebox)• Specific interleavings testing (whitebox)• All interleavings testing (whitebox)

• Instrumentation (blackbox)


• A lot of threads and operations to exercise different interleavings

Load/Stress testing


Demo!


Deadlock example

Thread 2Thread 1

synchronized ( A ) {

synchronized ( B ) {

synchronized ( B ) {

synchronized ( A ) {

Deadlock !!!


• Tools– ExecutorService -

http://download.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html#invokeAll(java.util.Collection)

– TestNG - http://testng.org/doc/documentation-main.html#annotations

– JUnit - http://www.junit.org/apidocs/junit/extensions/ActiveTestSuite.html

• JUnitPerf - http://testng.org/doc/documentation-main.html#annotations

– Custom Threading– GroboUtils -

http://groboutils.sourceforge.net/testing-junit/using_mtt.html

Load/Stress testing



http://testng.org/doc/documentation-main.html#annotations

http://www.junit.org/apidocs/junit/extensions/ActiveTestSuite.html

http://www.junit.org/apidocs/junit/extensions/ActiveTestSuite.html

http://testng.org/doc/documentation-main.html#annotations

http://groboutils.sourceforge.net/testing-junit/using_mtt.html


• Deterministic and repeatable tests

Specific interleavings testing


• Based on internal clock

– Tick increases when all threads are blocked

– waitForTick(tick) – blocks the thread until tick reaches certain value

– assertTick(tick) – asserts the current tick

MultithreadedTC


MultithreadedTC

Thread 2Thread 1

put 42

get 42

put 17(blocks)

BoundedBuffer with size = 1

get 17


MultithreadedTC

Thread 2Thread 1

put 42

get 42

BoundedBuffer with size = 1

get 17

Tick 0

put 17(blocks)

Tick 1

waitForTick(1)

assertForTick(1)


Demo!


• Events based

– IMUnit.fireEvent(“event1”)

– @Schedule(“event1 -> event2”)

– @Schedule(“[event1] -> event2”)

IMUnit


Demo!


• Two threads only – Main and Secondary

• By default: For each line of Main thread (T1) -> block and execute fully Secondary thread (T2)

ThreadWeaver

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2


• Two threads only – Main and Secondary

• By default: For each line of Main thread (T1) -> block and execute fully Secondary thread (T2)

• Powerful Breakpoints

ThreadWeaver


Demo!


• Tools– MultithreadedTC -

http://code.google.com/p/multithreadedtc/• Enhanced version for Junit 4 -

http://code.google.com/p/multithreadedtc-junit4/

– IMUnit - http://mir.cs.illinois.edu/imunit/– ThreadWeaver -

http://code.google.com/p/thread-weaver/– Awaitility - http://code.google.com/p/awaitility/

Specific interleavings testing

http://code.google.com/p/multithreadedtc/

http://code.google.com/p/multithreadedtc-junit4/

http://mir.cs.illinois.edu/imunit/

http://code.google.com/p/thread-weaver/

http://code.google.com/p/awaitility/


• Exercise all possible interleavings

All interleavings testing


JavaPathFinder

Traditional testing

OK Code

Testing

error Image source: http://javapathfinder.sourceforge.net/events/JPF-workshop-050108/tutorial.ppt


JavaPathFinder

Model Checking with JavaPathFinder

Image source: http://javapathfinder.sourceforge.net/events/JPF-workshop-050108/tutorial.ppt

OKCode

properties

Model Checking

error trace

Line 5: …Line 12: ……Line 41:…Line 47:…


Program testing can be used to show the presence of bugs, but never to show their absence!

--Edsger Dijkstra

JavaPathFinder


Demo!


• Tools– JavaPathFinder (JPF) -

http://babelfish.arc.nasa.gov/trac/jpf

• JavaRaceFinder - http://www.cise.ufl.edu/research/JavaRacefinder/Java_RaceFinder/JRF_Home.html

All interleavings testing


• Instrument the code to catch problems easier

Instrumentation


AspectJ

Original code.class or .jar

AspectJ Compiler

Instrumented .class or .jar

Aspect Definition.aj


Demo!


• Tools– AspectJ - www.eclipse.org/aspectj/

• RacerAJ - http://www.bodden.de/tools/raceraj/

– CalFuzzer - http://srl.cs.berkeley.edu/~ksen/calfuzzer/– (commercial) Flashlight / Jsure -

http://www.surelogic.com/concurrency-tools.html

Instrumentation


• CHESS (native DLLs & managed executables) - http://research.microsoft.com/en-us/projects/chess/

• MoonWalker (.NET)- http://code.google.com/p/moonwalker/

Other tools


Recommended books


Thank you!


Q & A


• org.ista2011.multithreaded.common– Several implementations of AccountManager, Counter, BounderBuffer and

MultipleReadersSingleWriter

• org.ista2011.multithreaded.executorservice– Load/stress tests using ExecutorService

• org.ista2011.multithreaded.testng– Load/stress tests using TestNG

• org.ista2011.multithreaded.mtc– MultithreadedTC tests

• org.ista2011.multithreaded.imunit– IMUnit tests

• org.ista2011.multithreaded.threadweaver– ThreadWeaver tests

Source code packages (1)


• org.ista2011.multithreaded.javapathfinder– JavaPathFinder examples

• ca.mcgill.sable.racer – RacerAJ source code

• org.ista2011.multithreaded.aspectj– Custom aspect that increase the chance of hitting multithreading problem

during testing

• org.ista2011.multithreaded.blockbox.*– Rest based AccountManager client/test and server (in .server package)

Source code packages (2)

www.istabg.org

testing multithreaded java applications for synchronization problems, ista 2011

Software

block b t2

line of main thread

multithreadedtc thread

deadlock example thread

synchronized b

thread n java heap

junitperf http

tools multithreadedtc