Java™PerformanceTestingforEveryone

PresentedBy:

ShelleyLambert(AdoptOpenJDKCommitter,EclipseOpenJ9Committer,IBMRuntimesTestLead)

WhoAmI?

VariousRolesDeveloper/TestLeadDevelopmentManagerYogaTeachertuneupfitness.com/teacher/shelley-lambert

ChiefFoodForesterottawafoodforests.comnanabushfoodforests.com

TheScope

AdoptOpenJDK

•  Projects:EclipseOMR,EclipseOpenJ9,AdoptOpenJDK•  EnsuringFreeandVerifiedJava™fortheCommunity•  6+Jenkinsservers

•  18+platforms•  6+versions,4+implementations•  7+testcategories->100,000’softests

•  18x6x4x100000=~43milliontests…nightly,aroundtheworld!

Wheretostartwithperformance?•  ThestoryofJavaperformance

– Nosinglerecipe(Manyfactors:JVMimplementation,hardware,applicationdesign)

–  JVMsevolve,performanceimproves

•  JVMs“complex,intricate,subtle”

•  Wouldn’titbegreatifitweresimple?-XX:goFaster,-XX:useLessResources

TheIntersection•  Necessary

–  developersneedtoknowifthecodetheywriteaffectsperformance

–  currentlyusingdiversesetoftoolsandapproaches,home-madescripts,duplication,lostlearningopportunities

•  Impossible–  measuringperformanceoftenstatedas“toohard”todo

Necessary Impossible

Definingthe“Impossible”•  Whatisperformancetesting?

–  Oftencalled”Experimentalscience”–  “Testingifasystemaccomplishesitsdesignatedfunctionswithingivenconstraintsregardingprocessingtimeandthroughputrate.”*

•  Goodperformance?Speed,resourcesorablend– Modernlanguageruntimescareaboutmanydifferentmetrics

•  Throughput,StartupTime,Ramp-upTime,CompileTime•  Footprint

–  AverageResidentSetSize–  CompilationMemoryConsumption–  PeakResidentSetSize

*Witteveen,Albert.Performancetesting-apracticalguide(KindleLocations176-177).

WhattomeasureMetricname Whatto

measure?Constraints Inputstovary

Throughput #oftransactions time

Latency Timeforsingletransaction

#oftransactions Workload(increases)

Capacity #ofsimultaneoustransactions

Throughputorlatency Parallelloadonthesystem

Utilization Useofresources workload

Efficiency Throughput/utilization

Scalability Throughputorcapacity

Resources(added)

Degradation Latencyorthroughput

utilization Workload(increases)

Explicitorimplicit‘inputs’tonormalize:HW,OS,systemsetup

BasicSteps•  Setagoal–whichmetric(s)toimprove

•  Measure–buthow?tools?

•  Adjust–applyyourexperiments

•  Measureagain–howexhaustive?

•  Verifygoal–didthemetricsimprove?enough?

AdoptOpenJDKTestinggithub.com/AdoptOpenJDK/openjdk-tests

•  Thewildlydifferent‘fruit’,howtomakethemeasilyconsumable

“ConsolidateandCurate”

functional openjdk perfjckexternalsystem

testkitgen

testNG,cmdlinetester

STF junit&others javatest Assorted

benchmarksjtreg

PerformanceBenchmarks(Large-scaleandMicrobenchmarksatAdoptOpenJDK)

bbenchacme-air libertydt jmhidlespark odm

perf

Assortedbenchmarks

… … … …

Largescale

Microbenchmarks

Introducinggithub.com/AdoptOpenJDK/openjdk-test-tools

•  PerfNext-configure,tuneandlaunchperformancebenchmarks.

•  TestResultsSummaryService(TRSS)-summarizeandvisualizedifferenttestresultsincludingperfresults,pushdifferentsetsoftestresultstoaDB,searchtestandcompareresultsacrossdifferentplatforms,reportondifferencesbetweenjobs

•  Futureservices–resultanalytics,testgeneration,coreanalytics,bugprediction

Tracktheprogressofbenchmarkrunsandverifytheiroutput

TRSS–PerformanceComparison

TRSS–PerformanceComparison

TRSS–RegressionAnalysis

1month

All data

7days

BumbleBench“MicrobenchmarksSimplified”

github.com/AdoptOpenJDK/bumblebench

•  WritingagoodmicrobenchwithanoptimizingJITrunningyourcodeishard– areyoumeasuringwhatyouthinkyouaremeasuring?

•  BumbleBenchisaJavaframeworkthatprovideahookpointtoimplementthebenchmarkpayload

•  Frameworkprovidestheoutertimingloop,scoringinfrastructure,etc.

Conclusion•  Perfishard(notimpossible)

– Highresourcerequirementsforfull-scaletesting– Microbenchmarksdifficulttowrite– Dataisnoisyandsubjecttointerpretation

•  Buildingtoolstomakeperfeasier– TRSS/PerfNext / BumbleBench

•  AdoptOpenJDKgitrepos:openjdk-tests,openjdk-test-tools,bumblebench

•  Comingsoon->trss.adoptopenjdk.net

•  OpenCollaborationleadstogreaterInnovation–  “Innovationiscreativitywithajobtodo.”–JohnEmmerling

AdoptOpenJDK

adoptopenjdk.netAdoptOpenJDK/openjdk-tests @adoptopenjdk

eclipse.org/openj9eclipse/openj9 @openj9

eclipse.org/omreclipse.org/omr @eclipseomr

UpcomingTalks:PerformanceTestingforEveryoneAdoptOpenJDK:EnsuringFreeJavafortheCommunityFuzzyPlansandOtherTestIntegrationsShakingSticksandTestingOpenJDKImplementations

Connect&Collaborate!WebsiteGithubTwitter

8thdaytesting.comsmlambert @ShelleyMLambert