performance test driven development with oracle coherence
DESCRIPTION
Slides from Coherence SIG London,18 July 2013TRANSCRIPT
Test Driven Development with Oracle Coherence
Alexey Ragozin
London, 18 Jul 2013
Presentation outline
• Motivation
• PTDD philosophy
• Oracle Coherence under test
Coherence Vs. Testing
Small cluster Vs. Big cluster
Areas to keep an eye on
• Automation challenge
• Common pitfalls of performance testing
This code works
Filter keyFilter = new InFilter(new KeyExtractor(), keySet); EntryAggregator aggregator = new Count(); Object result = cache.aggregate(keyFilter, aggregator);
ValueExtractor[] extractors = { new PofExtractor(String.class, TRADE_ID), new PofExtractor(String.class, SIDE), new PofExtractor(String.class, SECURITY), new PofExtractor(String.class, CLIENT), new PofExtractor(String.class, TRADER), new PofExtractor(String.class, STATUS), }; MultiExtractor projecter = new MultiExtractor(extractors); ReducerAggregator reducer = new ReducerAggregator(projecter); Object result = cache.aggregate(filter, reducer);
This code also works
public static class MyNextExpiryExtractor implements ValueExtractor { @Override public Object extract(Object obj) { MyPorfolio pf = (MyPorfolio) obj; long nextExpiry = Long.MAX_VALUE; for(MyOption opt: pf.getOptions()) { if (opt.getExpiry() < nextExpiry) { nextExpiry = opt.getExpiry(); } } return nextExpiry; } @Override public String toString() { return getClass().getSimpleName(); } }
And this also looks Ok
@LiveObject public static class MyLiveObject implements PortableObject { // ... @EventProcessorFor(events={EntryInsertedEvent.class}) public void inserted( EventDispatcher dispatcher, EntryInsertedEvent event) { DefaultCommandSubmitter.getInstance() .submitCommand( "subscription", new MySubscribeCommand(event.getEntry().getKey())); } }
Another slide to scare you
APICache
service
Packet
publisher
Packet
speakerOS
Packet
listener
Packet
receiverOS
Service
thread
Worker
thread
Packet
receiver
Packet
publisher
Packet
speaker
Packet
listener
Packet
receiver
Service
thread
Cache
service
Packet
publisher
Packet
speaker
Packet
listener
APIService
thread
Packet
receiver
Packet
listenerOS OS
Packet
speaker
Packet
publisher
Service
thread
Worker
thread
Serialization
Deserialization
Client thread
Approximate sequence diagram for cache get operation
Functional Vs. Fast
You have paid for Coherence
You have paid for gigabytes of RAM
You have spent time developing solution
and you want to be REALLY FAST
Do not be a hostage of your beliefs
Test early
Test often
PTTD Philosophy
Working cycle
Write simplest functional code
Benchmark it
Improve based on test measurements
Never optimize unless you can measure outcome
Never speculate, measure
Saves time and improves work/life balance
Testing Coherence
Challenges
Cluster required
Sensitive to network
Database is usually part of solution
Massive parallel load generation required
Testing Coherence
Benefits
Pure Java, less hurdle with OS tweaking etc
Nodes are usually plain J2SE processes
you can avoid app server setup pain
No disk persistence
managing data is usually hardest part of test setup
Benchmarking and cluster size
Single node cluster may reveal server side processing issues
Small cluster 2-8 physical servers latency related problems
scalability anomalies
partitioning anomalies
Large cluster > 8 physical servers my terra incognita, so far
Areas to keep eye on
Extractors, queries, indexes • query index usage
• query plans for complex filters
• POF extractors
Server side processing • backing map listeners
• storage side transformations
• cross cache access
Network • effective network throughput
Capacity • large messages in cluster
• Coherence*Extend buffers
Mixed operation loads • cache service thread pool saturation
• cache service lock contention
Scale out • broadcast requests
• hash quality
• 100% utilization of network thread
Automation
“Classic” approach bash + SSH + log scraping
Problems not reusable
short “half-live” of test harness
Java and bash/awk is totally different skill set
Automation
Stock performance test tools Deployment are not covered
Often “web oriented”
Insufficient performance of tool
Automation – Game changer
cloud = CloudFactory.createSimpleSshCloud(); cloud.node("cbox1"); cloud.node("cbox2"); cloud.node("cbox3"); cloud.node("**").touch(); // Say hello cloud.node("**").exec(new Callable<Void>() { @Override public Void call() throws Exception { String jvmName = ManagementFactory.getRuntimeMXBean().getName(); System.out.println("My name is '" + jvmName + "'. Hello!"); return null; } });
Automation – Game changer
NanoCloud - http://code.google.com/p/gridkit/wiki/NanoCloudTutorial
• Managing slave nodes
in-process, local JVM process, remote JVM process
• Deploy free remote execution
• Classpath management automatic master classpath replication
include/exclude classpath elements
• Pain free master/slave communications
• Just works!
Automation – Game changer
Full stack
• Maven – ensure test portability
• Nanocloud – painless remote execution
• JUnit – test enter point
• Java – all test logic staring nodes, starting clients, load generation, result processing …
•Java – all test logic
• Jenkins – execution scheduling
Simple microbench mark
@Before public void setupCluster() { // Simple cluster configuration template // Single host cluster config preset cloud.all().presetFastLocalCluster(); cloud.all().pofEnabled(true); cloud.all().pofConfig("benchmark-pof-config.xml"); // DSL for cache config XML generation DistributedScheme scheme = CacheConfig.distributedSheme(); scheme.backingMapScheme(CacheConfig.localScheme()); cloud.all().mapCache("data", scheme); // Configuring roles cloud.node("storage*").localStorage(true); cloud.node("client").localStorage(false); // Storage nodes will run as separate processes cloud.node("storage*").outOfProcess(true); }
* https://github.com/gridkit/coherence-search-common/blob/master/src/test/java/org/gridkit/coherence/search/bench/FilterPerformanceMicrobench.java
Simple microbench mark
@Test public void verify_full_vs_index_scan() { // Tweak JVM arguments for storage nodes JvmProps.addJvmArg(cloud.node("storage-*"), "|-Xmx1024m|-Xms1024m|-XX:+UseConcMarkSweepGC");
// Generating data for benchmark // ...
cloud.node("client").exec(new Callable<Void>() { @Override public Void call() throws Exception { NamedCache cache = CacheFactory.getCache("data"); System.out.println("Cache size: " + cache.size()); calculate_query_time(tagFilter); long time = TimeUnit.NANOSECONDS.toMicros(calculate_query_time(tagFilter)); System.out.println("Exec time for [tagFilter] no index - " + time); // ... return null; } }); }
* https://github.com/gridkit/coherence-search-common/blob/master/src/test/java/org/gridkit/coherence/search/bench/FilterPerformanceMicrobench.java
Monitoring
Server CPU usage
Process CPU usage
Network bandwidth usage
Coherence threads CPU usage Packet Publisher/Speaker/Receiver
Cache service thread
Cache service thread pool
Coherence MBeans Cache service task backlog
TCMP, *Extend IO throughput
etc
Flavors of testing
Distributed micro benchmarks
Performance regression tests
Bottlenecks analyzing
Performance sign off
Flavors of testing
Distributed micro benchmarks • Micro benchmark using real cluster
• Proving ideas
• To be run manually be developer
Performance regression tests
Bottlenecks analyzing
Performance sign off
Flavors of testing
Distributed micro benchmarks
Performance regression tests • To be run by CI
• Execute several stable test scenarios
• Fixed load scenarios, not for stress testing
• GOAL: track impact of code changes
• GOAL: keep test harness compatible with code base
Bottlenecks analyzing
Performance sign off
Flavors of testing
Distributed micro benchmarks
Performance regression tests
Bottlenecks analyzing • Testing through N-dimensional space of parameters
• Fully autonomous execution of all test grid !!! • Analyzing correlation to pin point bottle neck
• To be performed regularly to prioritize optimization
• Also used to measure/prove effect of optimizations
Performance sign off
Flavors of testing
Distributed micro benchmarks
Performance regression tests
Bottlenecks analyzing
Performance sign off • Execution performance acceptance tests aligned to release goals
• Activity driven by QA
• Can share infrastructure with dev team owned tests
Flavors of testing
Distributed micro benchmarks
Performance regression tests
Bottlenecks analyzing
Performance sign off
Common pit falls
Measuring “exception generation” performance
always validate operation results
write functional test on performance tests
Fixed user Vs. Fixed request rate
serious problems may go unnoticed
Ignoring environment health and side load
Common pit falls
Fixed user Vs. Fixed request frequency
Fixed users 5 threads
5 operations out of 50k will fall out of time envelop
99 percentile would be ~1ms
Fixed request rate 300k operation in total
250k around 1 ms
50k between 1ms and 10 s
99 percentile would be ~9.4 s
Case: Operation mean time: 1ms Throughput: 5k ops/s Test time: 1 minute GC pause 10 seconds in middle of run
Links
• Nanocloud http://code.google.com/p/gridkit/wiki/NanoCloudTutorial
• ChTest – Coherence oriented wrapper for Nanocloud
http://code.google.com/p/gridkit/wiki/ChTest
http://blog.ragozin.info/2013/03/chtest-is-out.html
https://speakerdeck.com/aragozin/chtest-feature-outline
• GridKit @ GitHub https://github.com/gridkit
Thank you
Alexey Ragozin [email protected]
http://blog.ragozin.info - my blog about JVM, Coherence and other stuff