perfomance tuning on go 2.0
Post on 15-May-2015
1.292 Views
Preview:
TRANSCRIPT
© 2010 ThoughtWorks, Inc. All rights reserved.
Performance Tuning on Go 2.0http://www.thoughtworks-studios.com/go/
Yogi Kulkarni
© 2010 ThoughtWorks, Inc. All rights reserved.
GoContinuous integration Release management
© 2010 ThoughtWorks, Inc. All rights reserved.
© 2010 ThoughtWorks, Inc. All rights reserved.
Stage-2Unit Test
Run Unit Test Suite - 1
Run Unit Test Suite - 2
Stage-3Acceptance Test
Accetance Test Suite - 1
Acceptance Test Suite – 2
Stage-1Compile
CompileJob
Pipeline
Run Unit Test Suite - 3
© 2010 ThoughtWorks, Inc. All rights reserved.
Pipeline Dashboard
© 2010 ThoughtWorks, Inc. All rights reserved.
© 2010 ThoughtWorks, Inc. All rights reserved.
Pipeline Dependencies
© 2010 ThoughtWorks, Inc. All rights reserved.
The Problem
© 2010 ThoughtWorks, Inc. All rights reserved.
Slow page loads
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Jobs taking long to get assigned to agents
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Jobs getting rescheduled
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Artifact uploads timing out
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Server VM freezing
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Lots of concurrent activity
© 2010 ThoughtWorks, Inc. All rights reserved.
What was happening under the hood?Profiler revealed the culprits
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Thread blocks
© 2010 ThoughtWorks, Inc. All rights reserved.
Threads blocked on the database connection pool
© 2010 ThoughtWorks, Inc. All rights reserved.
So we introduced asynchronicity
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Big mistake!
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Find the root cause
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
© 2010 ThoughtWorks, Inc. All rights reserved.
Problems
Too many queries
Slow queries
Coarse-grained synchronized blocks
JRuby and Rails issues
© 2010 ThoughtWorks, Inc. All rights reserved.
Problems Solutions
Too many queries Caching
Slow queries Query tuning
Coarse-grained synchronized Reduce lock granularity
JRuby and Rails issues Hack and pray!
© 2010 ThoughtWorks, Inc. All rights reserved.
ObjectiveNo request should take > 100 ms
Most should be under 50 ms
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Setup & Tools
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Conservative hardware
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
“Performance Server”Dell 620 laptop
2 Cores, 2GB RAM
© 2010 ThoughtWorks, Inc. All rights reserved.
Large Dataset5GB merged database
Large config file with 100 pipelinesReal-world data
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Extreme load70 agents
Each Job generating lots of console outputEach Job uploading 10 MB artifacts
Apache Bench / Httperf
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Automation for fast feedbackCapistrano script
Feedback cycle down from hours to 10 min
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
Caching
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Single application-wide cacheDomain objects
DTOsRails view fragments
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
© 2010 ThoughtWorks, Inc. All rights reserved.
EHCacheIn-memory only
Least-Recently-Used eviction policy
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Transaction rollbacks == Incorrect cached dataIgnore cache puts in transactions
Invalidate caches on transaction commit only
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Spring to the rescue
© 2010 ThoughtWorks, Inc. All rights reserved.
Fast deep-cloning of cached objectsJava Deep Cloning Library: http://robust-it.co.uk/clone/index.php
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Locks
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Be suspicious of synchronized methods
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Lock on interned stringsFine granularityProfiler support
Also used as cache keys
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
© 2010 ThoughtWorks, Inc. All rights reserved.
Use ReadWriteLocks for highly contended code
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Use “volatile” fields for lockless reads
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Slow
© 2010 ThoughtWorks, Inc. All rights reserved.
Not synchronized
© 2010 ThoughtWorks, Inc. All rights reserved.
Acquire locks before starting transactionsTo enforce aggregate invariants
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
Stage(Building)
Job – 1(Completed)
Job – 2(Building)
Stage(Building)
Job – 1(Building)
Job – 2(Completed)
Thread 1
Thread 2
Stage(Building)
Job – 1(Building)
Job – 2(Building)
Stage(Building)
Job – 1(Completed)
Job – 2(Cmpleted)
© 2010 ThoughtWorks, Inc. All rights reserved.
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Query Tuning
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Use a large databaseUse the query analyzer to find missing indexes
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Views are great… sometimes10x-20x faster for “TOP n” queries
Depends on database implementation
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Query selectivity is important200 ms
Stages(70,000 rows)
Pipelines(20,000 rows)
BuildStateTransitions(4,000,000 rows)
© 2010 ThoughtWorks, Inc. All rights reserved.
2 ms
Introduce an n+1 query
© 2010 ThoughtWorks, Inc. All rights reserved.
JRuby on Rails Hacks
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Single JRuby runtime + Rails multi-threaded mode
Much lower memory footprintBut completely uncharted territory
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Severe thread blocks in JRubyReported to JRuby community
Fixed immediately in 1.5.0 & 1.5.1
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Rails nested partials Mysterious lock contention
10 concurrent requests take 90 secs to completeStill a mystery!
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Desperate situations, desperate measuresStatically inline partials at build time!
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Routes subsystem is slowCache url_for
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
LoggingReplace Rails BufferedLogger with Java Log4jLogger
Turn off Rails logging in production
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
Fragment cachingGenerate a cache-key that incorprates all fields that are needed to
uniquely identify that object’s current stateLRU cache will evict it on state change
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
© 2010 ThoughtWorks, Inc. All rights reserved.
“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,building]”
Key-identifier Pipeline-name Stage detail Stage detail
© 2010 ThoughtWorks, Inc. All rights reserved.
“pipelineDashboardFragment|dev|1[compile,1,passed][test,1,building]”
“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,passed]”
Key-identifier Pipeline-name Stage detail Stage detail
© 2010 ThoughtWorks, Inc. All rights reserved.
Lessons LearnedFind the root cause
Use a profilerMeasure before and after each fix
Problem Investigation Setup Solution
© 2010 ThoughtWorks, Inc. All rights reserved.
How we got from that to thisINSERT IMAGE
Threads after performance tuning
© 2010 ThoughtWorks, Inc. All rights reserved.
http://www.thoughtworks-studios.com/go/
top related