perfomance tuning on go 2.0
TRANSCRIPT
![Page 1: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/1.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Performance Tuning on Go 2.0http://www.thoughtworks-studios.com/go/
Yogi Kulkarni
![Page 2: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/2.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
GoContinuous integration Release management
![Page 3: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/3.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
![Page 4: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/4.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Stage-2Unit Test
Run Unit Test Suite - 1
Run Unit Test Suite - 2
Stage-3Acceptance Test
Accetance Test Suite - 1
Acceptance Test Suite – 2
Stage-1Compile
CompileJob
Pipeline
Run Unit Test Suite - 3
![Page 5: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/5.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Pipeline Dashboard
![Page 6: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/6.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
![Page 7: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/7.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Pipeline Dependencies
![Page 8: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/8.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
The Problem
![Page 9: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/9.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Slow page loads
Problem Investigation Setup Solution
![Page 10: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/10.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Jobs taking long to get assigned to agents
Problem Investigation Setup Solution
![Page 11: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/11.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Jobs getting rescheduled
Problem Investigation Setup Solution
![Page 12: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/12.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Artifact uploads timing out
Problem Investigation Setup Solution
![Page 13: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/13.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Server VM freezing
Problem Investigation Setup Solution
![Page 14: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/14.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Lots of concurrent activity
![Page 15: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/15.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
What was happening under the hood?Profiler revealed the culprits
Problem Investigation Setup Solution
![Page 16: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/16.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Thread blocks
![Page 17: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/17.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Threads blocked on the database connection pool
![Page 18: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/18.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
So we introduced asynchronicity
Problem Investigation Setup Solution
![Page 19: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/19.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Big mistake!
Problem Investigation Setup Solution
![Page 20: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/20.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Find the root cause
Problem Investigation Setup Solution
![Page 21: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/21.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
![Page 22: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/22.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Problems
Too many queries
Slow queries
Coarse-grained synchronized blocks
JRuby and Rails issues
![Page 23: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/23.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Problems Solutions
Too many queries Caching
Slow queries Query tuning
Coarse-grained synchronized Reduce lock granularity
JRuby and Rails issues Hack and pray!
![Page 24: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/24.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
ObjectiveNo request should take > 100 ms
Most should be under 50 ms
Problem Investigation Setup Solution
![Page 25: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/25.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Setup & Tools
Problem Investigation Setup Solution
![Page 26: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/26.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Conservative hardware
Problem Investigation Setup Solution
![Page 27: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/27.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
“Performance Server”Dell 620 laptop
2 Cores, 2GB RAM
![Page 28: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/28.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Large Dataset5GB merged database
Large config file with 100 pipelinesReal-world data
Problem Investigation Setup Solution
![Page 29: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/29.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Extreme load70 agents
Each Job generating lots of console outputEach Job uploading 10 MB artifacts
Apache Bench / Httperf
Problem Investigation Setup Solution
![Page 30: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/30.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Automation for fast feedbackCapistrano script
Feedback cycle down from hours to 10 min
Problem Investigation Setup Solution
![Page 31: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/31.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Caching
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 32: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/32.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Single application-wide cacheDomain objects
DTOsRails view fragments
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 33: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/33.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
![Page 34: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/34.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
EHCacheIn-memory only
Least-Recently-Used eviction policy
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 35: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/35.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Transaction rollbacks == Incorrect cached dataIgnore cache puts in transactions
Invalidate caches on transaction commit only
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 36: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/36.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Spring to the rescue
![Page 37: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/37.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Fast deep-cloning of cached objectsJava Deep Cloning Library: http://robust-it.co.uk/clone/index.php
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 38: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/38.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Locks
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 39: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/39.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Be suspicious of synchronized methods
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 40: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/40.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Lock on interned stringsFine granularityProfiler support
Also used as cache keys
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 41: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/41.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
![Page 42: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/42.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Use ReadWriteLocks for highly contended code
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 43: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/43.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Use “volatile” fields for lockless reads
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 44: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/44.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Slow
![Page 45: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/45.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Not synchronized
![Page 46: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/46.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Acquire locks before starting transactionsTo enforce aggregate invariants
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 47: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/47.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
Stage(Building)
Job – 1(Completed)
Job – 2(Building)
Stage(Building)
Job – 1(Building)
Job – 2(Completed)
Thread 1
Thread 2
Stage(Building)
Job – 1(Building)
Job – 2(Building)
Stage(Building)
Job – 1(Completed)
Job – 2(Cmpleted)
![Page 48: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/48.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 49: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/49.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Query Tuning
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 50: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/50.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Use a large databaseUse the query analyzer to find missing indexes
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 51: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/51.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Views are great… sometimes10x-20x faster for “TOP n” queries
Depends on database implementation
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 52: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/52.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Query selectivity is important200 ms
Stages(70,000 rows)
Pipelines(20,000 rows)
BuildStateTransitions(4,000,000 rows)
![Page 53: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/53.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
2 ms
Introduce an n+1 query
![Page 54: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/54.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
JRuby on Rails Hacks
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 55: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/55.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Single JRuby runtime + Rails multi-threaded mode
Much lower memory footprintBut completely uncharted territory
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 56: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/56.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Severe thread blocks in JRubyReported to JRuby community
Fixed immediately in 1.5.0 & 1.5.1
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 57: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/57.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Rails nested partials Mysterious lock contention
10 concurrent requests take 90 secs to completeStill a mystery!
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 58: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/58.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Desperate situations, desperate measuresStatically inline partials at build time!
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 59: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/59.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Routes subsystem is slowCache url_for
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 60: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/60.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
LoggingReplace Rails BufferedLogger with Java Log4jLogger
Turn off Rails logging in production
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 61: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/61.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Fragment cachingGenerate a cache-key that incorprates all fields that are needed to
uniquely identify that object’s current stateLRU cache will evict it on state change
Problem Investigation Setup Solution
Caching Locks Query Tuning Jruby & Rails
![Page 62: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/62.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,building]”
Key-identifier Pipeline-name Stage detail Stage detail
![Page 63: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/63.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
“pipelineDashboardFragment|dev|1[compile,1,passed][test,1,building]”
“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,passed]”
Key-identifier Pipeline-name Stage detail Stage detail
![Page 64: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/64.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
Lessons LearnedFind the root cause
Use a profilerMeasure before and after each fix
Problem Investigation Setup Solution
![Page 65: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/65.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
How we got from that to thisINSERT IMAGE
Threads after performance tuning
![Page 66: Perfomance tuning on Go 2.0](https://reader033.vdocument.in/reader033/viewer/2022052823/5555a8fed8b42a52568b47ea/html5/thumbnails/66.jpg)
© 2010 ThoughtWorks, Inc. All rights reserved.
http://www.thoughtworks-studios.com/go/