towards a macrobenchmark framework for performance analysis of java applications
TRANSCRIPT
![Page 1: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/1.jpg)
Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems
Budapest University of Technology and EconomicsFault Tolerant Systems Research Group
Towards a Macrobenchmark Framework for Performance Analysis of Java Applications
Gábor Szárnyas, Dániel Varró
![Page 2: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/2.jpg)
Performance issues
Model-Driven Development
Modeling
Codegeneration
Testing
Early validationsTransformations
Scalabilitychallenges
![Page 3: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/3.jpg)
Model Sizes
Models = graphs with 100M–1B elements
o Car industry
o Avionics
o Software analysis
o Cyber-physical systems
Source: Markus Scheidgen, Automated and TransparentModel Fragmentation for Persisting Large Models, 2012
application model size
software models 108
sensor data 109
geo-spatial models 1012
Validation may take hours
![Page 4: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/4.jpg)
Research Goal
Scalable query engine for evaluating complexqueries on large models.
Latest results are presented in tomorrow’s session:
o János Maginecz, Gábor Szárnyas: Sharded Joins for Scalable Incremental Graph Queries
Today: benchmarking
![Page 5: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/5.jpg)
Motivation
Jennifer Widom: Tips for Writing Technical Papers
On performance experiments:
It’s easy to do “hokey” or meaningless experiments, and many papers do.
Many conferences expect experiments.
![Page 6: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/6.jpg)
Benchmarks in Software Engineering
Performance experiments = benchmarks
Common goals
o Compare various tools
o Derive performance metrics
o Assess the scalability of the system
Microbenchmarks
oMethod-level
o Very difficult and not recommended
Macrobenchmarks
o Application-level
![Page 7: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/7.jpg)
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
![Page 8: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/8.jpg)
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
Distributed storage
![Page 9: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/9.jpg)
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
Distributed indexing
Distributed storage
![Page 10: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/10.jpg)
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
Distributed indexing
Distributed storage
Distributed query network
![Page 11: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/11.jpg)
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
![Page 12: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/12.jpg)
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Indexer Indexer Indexer Indexer
Join
Join
Antijoin
![Page 13: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/13.jpg)
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Indexer Indexer Indexer Indexer
Join
Join
Antijoin
Separate processes
![Page 14: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/14.jpg)
Operating System
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
OS OS OS OS
Server 0
Indexer Indexer Indexer Indexer
Join
Join
Antijoin
• File system caching• Scheduled jobs• Swapping
![Page 15: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/15.jpg)
Cloud Noise
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
OS OS OS OS
Server 0
Indexer Indexer Indexer Indexer
Join
Join
AntijoinOther VMs runningon the same server
![Page 16: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/16.jpg)
Parallel Execution
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
OS OS OS OS
Server 0
Indexer Indexer Indexer Indexer
Join
Join
Antijoin
Multiple processes in the same OS
Multithreaded processes
![Page 17: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/17.jpg)
Threats to Validity
Operating system: caching, scheduled jobs
Parallel processes
Multithreaded execution
Environmental noise in the cloud
The hardest: Managed Runtime Environments
o Java Virtual Machine
o .NET CLR
![Page 18: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/18.jpg)
Managed Runtime Environments
Just-in-timecompilation
Runtimeoptimization
![Page 19: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/19.jpg)
The Effect of Warmup
Firstexecution time
Medianexecution timeof 5 executions
Two Java-based query engines
Performmultiple
executions
![Page 20: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/20.jpg)
CASE STUDY:THE TRAIN BENCHMARK
Gábor Szárnyas, Benedek Izsó, István Ráth, Dániel Varró
2016
![Page 21: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/21.jpg)
Database Benchmarks
Criteria for domain-specific benchmarks(Jim Gray, Benchmark Handbook, 1993):
Relevant
Scaleable
Portable
Simple
Two-phase commit, data cubes, ACID transactions
![Page 22: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/22.jpg)
The Train Benchmark
Domain: railway network validation
Goal: measure the scalability of query evaluation
Supports various representations
Randomly generated instance models
o Increasing sizes
o Close to real-world instances
o Faults are inserted during generation
![Page 23: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/23.jpg)
Railway Model
Synthetic model
Customizable model generator
Semaphore
Route
SegmentSwitch
Entry
ExitSwitch position
of the Route
Current
position
of the Switch
![Page 24: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/24.jpg)
Validation Queries
Well-formedness constraints
Queries are looking for error patterns
o Lots of filtering, joins, etc.
Transformations
o Fault injections
o Quick fix-like repair operations
![Page 25: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/25.jpg)
Benchmark Phases
1. Load 3. Transform 4. Revalidate2. Validate
Iteration: × 10Run:× 5
Change set size
Modelincreasing
size
Query Measure-
ments
Framework features
o Automated build and unit testing
o Automated visualization
Warmup
Warmup
![Page 26: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/26.jpg)
Model Transformation
7 OOMs
![Page 27: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/27.jpg)
Some Conclusions
Very hard to tell the actual memory consumption
oManual calls to the garbage collector – not enough
o Profiler – still not enough
o Setting a hard limit is the best approach
Benchmarking is difficult
o Lots of factors can add noise to the results
o Nothing works on the first try
Visualization helps a lot
oWorth investing time to learn R
o Not a nice language, but very productive
![Page 28: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/28.jpg)
Related PublicationsBenchmark framework
Szárnyas, G., Izsó, B., Ráth, I., and Varró, D.,The Train Benchmark for Evaluating the Performance of Continuous Model Validation, SOSYM journal (work-in-progress)
Izsó, B., Szárnyas, G., Ráth, I., and Varró, D.,MONDO-SAM: A Framework to Systematically Assess MDE Scalability, 2nd Workshop on Scalable Model Driven Engineering, 2014
Benchmark applications
Szárnyas, G., Semeráth, O., Ráth, I., and Varró, D.,The TTC 2015 Train Benchmark Case for Incremental Model Validation, Transformation Tool Contest, 2015
Szárnyas, G., Izsó, B., Ráth, I., Harmath, D., Bergmann, G., and Varró, D.,IncQuery-D: A Distributed Incremental Model Query Framework in the Cloud, ACM/IEEE 17th International Conference on Model Driven Engineering Languages and Systems, 2014
Izsó, B., Szárnyas, G., Ráth, I., and Varró, D., IncQuery-D: Incremental Graph Search in the Cloud,Proceedings of the Workshop on Scalability in Model Driven Engineering, 2013
![Page 29: Towards a Macrobenchmark Framework for Performance Analysis of Java Applications](https://reader031.vdocument.in/reader031/viewer/2022022413/58eeed331a28abe23e8b45cd/html5/thumbnails/29.jpg)
Ω