ContinuousPerformance Testing

in Virtual Time

Nikos Baltas & Tony Field

Department of Computing

Imperial College London

Imperial College - Department of Computing

Motivation

Performance analysis:Early stages: performance modellingLate stages: profiling & performance testing Intermediate stages: extract models from code,

calibrate them and analyse...

Can we simplify this? Can we performance test incomplete applications?

Imperial College - Department of Computing

Idea

Integrate models and code

Don't model existing code – execute it in virtual time!

Use performance models that describe the expected response times of missing (or existing) methods

Combine measurements and model simulations to guide the thread progress of the application

Imperial College - Department of Computing

Virtual Time Execution

Imperial College - Department of Computing

VEX

Virtual Execution framework (VEX)

Java applications

Components Thread control Profiling I/O handling

Imperial College - Department of Computing

VEX simulation

Model existing code by profiling it on the simulation host

Map the (possibly scaled) measurements on a virtual timeline

Control the thread progress to enforce a fair scheduling policy in virtual time

Imperial College - Department of Computing

Integration of models to VEX

Don't just scale method durations: replace them

Use time acquired by the simulation of performance models (open queuing networks)

Decouple profiled time from virtual time measurements

Use for either implemented methods or code stubs

Continuous performance testing

Imperial College - Department of Computing

Example

Imperial College - Department of Computing

Issues

How does the model-simulation time relate to the virtual time of executed code?

How do we control the model simulation?

What if the code of the model-described method interacts with the application?

Imperial College - Department of Computing

Remote and local resources

Imperial College - Department of Computing

Model simulation in VEX

Thread adds job Thread state depends

on job state in queue: L-Think: Running/able R-Think: Timed-waiting Queuing: Waiting L-Serv: Running/able R-Serv: Timed-waiting

Thread and job join at a Sink node

Imperial College - Department of Computing

Model description: JMT [1]

[1] M. Bertoli, G. Casale, and G. Serazzi, “JMT: performance engineering tools for system modeling,” SIGMETRICS Perform.Eval.Rev., vol. 36, no. 4, pp. 10–15, 2009.

Imperial College - Department of Computing

Functional consistency

T1

T2

Performancemodel for M()

synchronized(obj) {resume suspend

M() { synchronized(obj) {

wait for T1 to resume

blocked on obj monitor

Deadlock!violate virtual scheduleand resume anyway

} exit monitor obj suspend

enter/exit monitor

} M()

Join with model resume

Imperial College - Department of Computing

Usage

-Xbootclasspath/p:vex.jar –javaagent:jine.jar –agentpath:libjinevex.sojava Class

void foo() {

…..

bar();

…..

}

@virtualtime.ModelPerformance(

jmtModelFilename=“file.jsimg”,

replaceMethodBody=true,

sourceNodeLabel=“Source0”)

Imperial College - Department of Computing

Evaluation

Case study to demonstrate usage and verify functionality of approach

Sample results and parameters

Validate process by comparing predicted results to real times

Study: remote DB server with local cache

Imperial College - Department of Computing

Pure model

Imperial College - Department of Computing

Partially complete code

Imperial College - Department of Computing

Complete code

Imperial College - Department of Computing

Results

Imperial College - Department of Computing

Conclusion

Integration of models and code within VEX

Models and code can be used interchangeably

Continuous performance testing without extracting models from code

Prediction errors due to disregard for background system load and I/O approximation

Imperial College - Department of Computing

Future work

Handle more behaviours (asynchronous I/O)

Lower-level VEX and/or distributed virtual execution

Application on larger scale projects

Release VEX under open source license

Extensions to other languages (C++)

Imperial College - Department of Computing

Questions?

Thank you

Imperial College - Department of Computing

Virtual Time Execution II

Imperial College - Department of Computing

Remote and local resources II

Imperial College - Department of Computing

Local resources visualisation