framework and application benchmarking
TRANSCRIPT
Framework and Application Benchmarking
Paul M. JonesPhpBenelux 2011
@pmjoneshttp://joind.in/2498
Overview
• Benchmarking methodology, results, and interpretation• Server/application• Single frameworks• Framework comparisons
• Resource usage prediction• Engineering analysis
Read These
• “Free to Choose”, Friedman
• “Pleasure of Finding Things Out”, Feynman
• “Fooled By Randomness”, Taleb
About Me
• Dev, Sr Dev, Team Lead, Architect, VP Eng
• PHP since 1999
• Savant template system
• PEAR Group member
• http://paul-m-jones.com
• @pmjones
Full Disclosure
• Solar framework
• Zend framework
• Zend_Db*, Zend_View
• “Rigor and integrity”
• Google Code, now Github
Part 1:In General
Profiling vs Benchmarking
• Profiling applies to sub-systems (component design)
• Benchmarking applies to entire systems (architecture and interaction)
• Profile is to unit test, asbenchmark is to system test
Benchmarking Subjects
• CPU
• RAM
• Disk access
• Database access
• Network access
• Requests/second
• Programmer productivity
• Time to initial implementation
• Time to add new major feature
• Time to fix bugs
-- numeric measurement ---- control for variables --
Why Bother With Benchmarking?
• New site will be linked by Slashdot, Digg, etc.
• “Can we handle the traffic, Dodgers?”
• “Sure thing, boss.”
Why Bother With Benchmarking?
• “...(time to load-test.)”
• How many req/sec expected?
• Load-tested, optimized in a hurry, balanced
Know Your Limitations
• What limits are imposed, and where?
• Where to spend effort when optimizing?
• Profiling your app will not help with non-app limits
• Profiling does not tell you req/sec
The Stack
• Web server• Framework (foundation)• Application
Part 2:Getting Started
Methodology
• Refining and reporting since 2007• Benchmark the basic system• Add a component, benchmark again• Track responsiveness at each point
Tools
• ab, http_load, siege, ...
• Emulate concurrent users hitting a URI
• Reports req/sec, latency, min/max times, etc
ab -c 10 -t 60 http://example.com/
ab
Document Path: /baseline-html/index.htmlDocument Length: 13 bytesConcurrency Level: 10Time taken for tests: 0.19088 secondsComplete requests: 50000Failed requests: 0Total transferred: 15950000 bytesHTML transferred: 650000 bytesRequests per second: 2619.45Transfer rate: 835.60 kb/s received
http_load
4278 fetches, 325 max parallel, 25668 bytes6 mean bytes/connection855 fetches/sec, 5130 bytes/secmsecs/connect: 20.0881 mean, 3006.54 max, 0.099 minmsecs/first-response: 51.3568 mean, 342.488 max, 1.423 minHTTP response codes: code 200 -- 4278
siege
2009-02-26 16:35:22, 256869, 60.28, 3, 0.00, 4261.26, ...
Combined Tooling
• Each tool (ab, http_load, siege) has different options, params, and target specifications
• http://github.com/pmjones/php-framework-benchmarks
./bench/(ab|http_load|siege) target/all.ini
Benchmark Subject
• Run against a test installation • As similar to production as possible• Hardware, OS, server, PHP installation• No database, no application code
Control for Known Variables
• Network latency (use localhost)• Server processes (turn everything off)
Outer Limits Of Responsiveness
• Amazon EC2 “Large” instance, 64-Bit Ubuntu 10.04 (Alestic)
• Apache 2.2 (stock install)
• PHP 5.3.3, APC, Suhosin (stock install)
• “index.html” and “index.php”
HTML and PHP Graph
ab
http_load
siege
0 1250 2500 3750 5000
3148.12
2496.59
2170.31
4262.12
3195.91
2814.62
html php(10 concurrent users, 60 seconds)
HTML and PHP Percentage Change
html php pct
ab 2814.62 2170.31 77.11%
httpload 3195.91 2496.59 78.12%
siege 4262.12 3148.12 73.86%
Control forUnknown Variables
• Run multiple times
• No two runs will be identical; look for trends
• File-based sessions will cause exponentially declining returns
• session_id(‘constant_value’)
Other Problem Indicators
• PHP faster than HTML?
• Byte counts?
• Socket availability?
Benchmarking as Resource Prediction
• About resource planning, not speed (per se)
• With supply of “R” requests/second, you need “S” servers for demand of “T” traffic
• What stack components can you modify to increase “R” so you can handle more “T” with same or fewer “S”?
Part 3:Single Frameworks
Single-Framework Methodology
• Minimal, not full, application• Sane config• No action code• No view layout or helpers• Static view text• No caching• No database
Framework (Foundation)Benchmarking
• Dynamic dispatch cycle as the limiting factor
• Improvements off critical path will not increase responsiveness
• Page cache?
Web Server + PHPFramework
Boot Front Page Action View
Single-Framework Analysis
• Now you know how much you need to improve
• Work on the critical path
• Iterative benchmarking
Benchmarking as Engineering Aid
• Every architecture decision has a cost
• Compare benchmark stats between versions
• Are added features are worth req/sec cost?
• (Thanks, Laura Thomson.)
Part 3:Multiple
Frameworks
Multiple-Framework Comparison
• Examine responsiveness in context of other systems
• Compare architecture costs between systems
• Use the same minimal app to remove variability in application code
Compare Like With Like
• Frameworks to be compared should be reasonably similar in “style” or “class”
• PHP5, design patterns, front & page controllers, controller & view separation, plugins or extensions, no globals
• Cake, Lithium, Solar, Symfony, Yii, Zend
• CI, Kohana
Multiple-Framework Methodology (1/2)
• Minimal application• Reduced or optimized config• Production mode• No debugging or logging • No database connection• No layouts or view helpers
Multiple-Framework Methodology (2/2)
• APC to reduce filesystem access• Restart web server between runs (APC)• 5 runs of 10 concurrent users for 60
seconds, averaged• Calculate req/sec as percent of PHP
(thanks, Richard “Cyberlot” Thomas)
Framework NumbersTarget relative average 1 2 3 4 5
baseline/html 1.2823 3220.56 3227.18 3226.03 3207.75 3222.63 3219.20
baseline/php 1.0000 2511.54 2521.14 2509.00 2506.45 2508.48 2512.63
cake/1.3.7 0.0412 103.46 103.37 103.73 103.93 102.55 103.70
ci/1.7.3 0.2946 739.95 722.08 725.95 722.96 768.37 760.40
kohana/3.0.9 0.2912 731.37 696.83 757.46 743.72 720.10 738.76
lithium/0.9.9 0.0827 207.62 252.87 249.94 35.19 249.53 250.58
solar/1.1.1 0.1609 404.17 405.08 404.33 403.12 399.63 408.68
symfony/1.4.8 0.0784 196.91 197.00 197.77 199.00 194.93 195.87
symfony/2.0.0alpha1 0.1675 420.73 422.02 424.85 422.48 411.20 423.08
symfony/2.0.0pr4 0.0897 225.22 224.59 221.18 225.88 227.73 226.71
yii/1.1.5 0.1891 474.82 500.21 439.74 456.77 494.82 482.57
zend/1.10.2-minapp 0.1136 285.28 279.83 291.61 280.61 287.00 287.37
zend/1.10.2-project 0.0832 208.84 215.74 206.20 209.26 207.75 205.25
Framework Graph (100%)baseline/html
baseline/php
cake/1.3.7
ci/1.7.3
kohana/3.0.9
lithium/0.9.9
solar/1.1.1
symfony/1.4.8
symfony/2.0.0alpha1
symfony/2.0.0pr4
yii/1.1.5
zend/1.10.2-minapp
zend/1.10.2-project
0 0.375 0.750 1.125 1.500
Framework Graph (30%)cake/1.3.7
ci/1.7.3
kohana/3.0.9
lithium/0.9.9
solar/1.1.1
symfony/1.4.8
symfony/2.0.0alpha1
symfony/2.0.0pr4
yii/1.1.5
zend/1.10.2-minapp
zend/1.10.2-project
0 0.10 0.20 0.30
0.08
0.11
0.19
0.09
0.17
0.08
0.16
0.08
0.29
0.29
0.04
What Does This Mean?
• Both more and less than you think it might
• Speed is not the only consideration• Features, intangibles• Unlikely to change minds of devotees
• Even the slowest is still pretty fast• Each AJAX call is one request
Failure Modes
• Apache/Lighty, APC/XCache, mod_php/fcgid
• EC2 load fluctuation between/during runs• Order of testing• Different concurrency or duration• Added features and app code will reduce
dynamic response ... but curve is unknown.
Links
• http://github.com/pmjones/php-framework-benchmarks
• http://code.google.com/p/web-framework-benchmarks
• “New Year’s Benchmarks”:
http://paul-m-jones/archives/238• “Do All Frameworks Suck?”:
http://www.laurathomson.com/2007/07/do-all-framewor.html
• Google: "cargo cult science"
Objections
• “Highly profiled.”• “‘Hello world’ is unrealistic.”• “Not optimized for ‘hello world.’”• “Faster for ‘real world’ applications.”
Avoid Frameworks?
• If framework code is only at 4-30% of max PHP, should we not use frameworks?
• “Slow” relative to what?• Compare like with like, and proposed systems
to existing systems.• You will use a framework, even if you write it
yourself.• The good comes not from the code or the
features, but process standardization.
Thanks!