ruby http clients comparison
TRANSCRIPT
![Page 1: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/1.jpg)
Ruby HTTP clients comparison
Hiroshi Nakamura nahi at Twitter, github
Technical Architect at Appirio Japan
CRuby and JRuby committer Asakusa.rb: http://qwik.jp/asakusarb/
![Page 2: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/2.jpg)
Ruby HTTP Clients Matrix
advantages-and-disadvantages comparison of HTTP client libraries
http://bit.ly/RubyHTTPClients2012
Disclaimer: I'm the author of "httpclient"
![Page 3: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/3.jpg)
Agenda
net/http
16 libraries I picked
Ruby HTTP Clients Matrix
API style
Compatibility
Supported features
Performance Comparisons
My Recommendations
![Page 4: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/4.jpg)
net/http
Net::HTTP::Proxy? net/https?
![Page 5: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/5.jpg)
Ruby HTTP Client libraries
![Page 6: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/6.jpg)
HTTP client libraries I didn’t evaluate
Cannot evaluate • activeresource (Rails specific) • http (under development) • http_request.rb (test doesn't pass) • nestful (no test) • typhoeus (under heavy rewrite)
Obsolete • eventmachine (built-in client is obsolete) • right_http_connection (no update) • simplehttp (no update) • rfuzz (no update)
![Page 7: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/7.jpg)
Evaluation Axis
Project Stats
API style
Compatibility: CRuby, JRuby, Rubinius
Supported features
Connection features
Basic HTTP features
Development support
Advanced features
http://bit.ly/RubyHTTPClientsFeatureTest (test/unit scripts)
![Page 8: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/8.jpg)
Project Stats
![Page 9: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/9.jpg)
![Page 10: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/10.jpg)
![Page 11: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/11.jpg)
![Page 12: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/12.jpg)
Project Stats
![Page 13: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/13.jpg)
API style
![Page 14: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/14.jpg)
![Page 15: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/15.jpg)
sync API (1/5) - Client instance
net/http, mechanize, httpclient, patron, curb, faraday
client = Net::HTTP.new(host, port) p client.get(path).body client = Mechanize.new client = HTTPClient.new client = Patron::Session.new p client.get(url).body curl = Curl::Easy.new(url) curl.http_get p curl.body_str client = Faraday.new(:url => baseurl) p client.get(path).body
![Page 16: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/16.jpg)
sync API (2/5) - Client class
restfulie, excon, httpi
p Restfulie.at(url).get!.body p Excon.get(url).body p HTTPI.get(url).body
![Page 17: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/17.jpg)
sync API (3/5) - Resource
rest-client, rufus-verbs
rs = RestClient::Resource.new('http://example.com') rs['posts/1/comments'].post 'Good article.', :content_type => 'text/plain' ep = EndPoint.new( :host => "resta.farian.host", :port => 80, :resource => "inventory/tools") res = ep.get :id => 1 res = ep.get :id => 2
![Page 18: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/18.jpg)
sync API (4/5) - Include & Customize
httparty, weary
client = Class.new { include HTTParty } p client.get(url) class WearyClient < Weary::Client domain 'http://api.target.org/' get :retrieve, '{path}' end p WearyClient.new.fetch(:path => path).perform.body
![Page 19: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/19.jpg)
sync API (5/5) - Others
open-uri.rb, wrest
# open-uri.rb p open(url) { |f| f.read } # wrest p 'http://www.google.co.jp/'.to_uri.get.body
![Page 20: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/20.jpg)
async API(1/2) - Callback
em-http-request
body = nil EM.run do req = EM::HttpRequest.new( 'http://www.google.com/').get req.callback do body = req.response EM.stop end req.errback do body = nil end end p body
![Page 21: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/21.jpg)
async API(2/2) - Polling
httpclient, weary, wrest
client = HTTPClient.new conn = client.get_async(url) conn.finished? # => false # ... io = conn.pop.content while str = io.read(4096) p str end
![Page 22: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/22.jpg)
parallel API - curb
responses = [] m = Curl::Multi.new urls.each do |url| responses[url] = '' m.add(Curl::Easy.new(url) { |curl| curl.on_body { |data| responses[url] << data data.bytesize } }) end m.perform p responses.map { |e| e.bytesize }
![Page 23: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/23.jpg)
![Page 24: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/24.jpg)
![Page 25: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/25.jpg)
API style
![Page 26: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/26.jpg)
Compatibility
![Page 27: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/27.jpg)
![Page 28: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/28.jpg)
Compatibility
![Page 29: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/29.jpg)
Connection features
![Page 30: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/30.jpg)
![Page 31: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/31.jpg)
3 types of HTTP connections
![Page 32: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/32.jpg)
![Page 33: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/33.jpg)
Keep-Alive in em-http-request body = [] EM.run do conn = EventMachine::HttpRequest.new(server.url) req1 = conn.get(:keepalive => true) req1.callback { body << req1.response req2 = conn.get(:keepalive => true) req2.callback { body << req2.response req3 = conn.get(:keepalive => true) req3.callback { body << req3.response req4 = conn.get(:keepalive => true) req4.callback { body << req4.response EM.stop req4.errback { ... }} req3.errback { ... }} req2.errback { ... }} req1.errback { ... } end
![Page 34: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/34.jpg)
Pipelining in em-http-request
body = [] EM.run do conn = EventMachine::HttpRequest.new(server.url) req1 = conn.get(:keepalive => true) req2 = conn.get(:keepalive => true) req3 = conn.get(:keepalive => true) req4 = conn.get() req1.callback { body << req1.response } req2.callback { body << req2.response } req3.callback { body << req3.response } req4.callback { body << req4.response; EM.stop } req1.errback { ... } req2.errback { ... } req3.errback { ... } req4.errback { ... } end
![Page 35: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/35.jpg)
![Page 36: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/36.jpg)
NO verification by default?!
options[:ssl_ca_file] == nil => VERIFY_NONE
if http.use_ssl? http.verify_mode = OpenSSL::SSL::VERIFY_NONE if options[:ssl_ca_file] http.ca_file = options[:ssl_ca_file] http.verify_mode = OpenSSL::SSL::VERIFY_PEER end end
![Page 37: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/37.jpg)
![Page 38: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/38.jpg)
![Page 39: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/39.jpg)
![Page 40: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/40.jpg)
Connection features
![Page 41: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/41.jpg)
Basic HTTP features
![Page 42: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/42.jpg)
![Page 43: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/43.jpg)
![Page 44: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/44.jpg)
IRI: Internationalized Resource Identifier
uri.rb doesn't support IRI
addressable/uri does
client.get("http://www.ebooks.com/797059/some-kind-of-peace/grebe-camilla-träff-åsa-norlen-paul/")
![Page 45: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/45.jpg)
![Page 46: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/46.jpg)
"Cross-site Cooking" bug in httpclient
httpclient eats this cookie and send it to all *.com site
Mechanize handles it properly like browsers
http://en.wikipedia.org/wiki/Cross-site_cooking
Set-Cookie: TEST=test; path=/; domain=.com
![Page 47: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/47.jpg)
![Page 48: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/48.jpg)
![Page 49: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/49.jpg)
![Page 50: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/50.jpg)
![Page 51: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/51.jpg)
Streaming upload/download
# Chunked upload with Patron client = Patron::Session.new res = client.request(:post, url, {}, :file => path_to_upload) # Chunked download client.get_file(url, path_to_write) # Chunked upload with em-http-request req = EM::HttpRequest.new(url).post :file => path_to_upload # Chunked download req = EM::HttpRequest.new(url).get req.stream do |chunk| p chunk end
![Page 52: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/52.jpg)
![Page 53: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/53.jpg)
![Page 54: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/54.jpg)
Basic HTTP features
![Page 55: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/55.jpg)
Development Support
![Page 56: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/56.jpg)
![Page 57: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/57.jpg)
Response stubbing
# Stubbing response body client = HTTPClient.new client.test_loopback_response << 'Hello!' client.get('http://www.example.com/hello').body #=> "Hello!" # Stubbing HTTP response client.test_loopback_http_response << "HTTP/1.0 302 Found¥r¥nLocation: http://foo/¥r¥n¥r¥n" << "HTTP/1.0 200 OK¥r¥n¥r¥nHello!" client.post('http://www.example.com/todo', :follow_redirect => true, :body => '{}').body #=> "Hello!"
![Page 58: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/58.jpg)
![Page 59: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/59.jpg)
![Page 60: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/60.jpg)
IRB like shell
rest-client, httpclient, wrest
% restclient https://example.com user pass >> delete '/private/resource' % httpclient >> get "https://www.google.com", :q => :ruby % wrest >> 'http://www.google.com?q=ruby'.to_uri.get
![Page 61: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/61.jpg)
Replayable log
rest-client
% RESTCLIENT_LOG=/tmp/restclient.log restclient >> RestClient.get "https://www.google.com/" ... % cat /tmp/restclient.log RestClient.get "https://www.google.com/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate" # => 200 OK | text/html 13354 bytes
![Page 62: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/62.jpg)
![Page 63: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/63.jpg)
Development Support
![Page 64: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/64.jpg)
Advanced features
![Page 65: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/65.jpg)
![Page 66: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/66.jpg)
![Page 67: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/67.jpg)
![Page 68: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/68.jpg)
![Page 69: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/69.jpg)
HTML form handling of Mechanize
agent = Mechanize.new page = agent.get(url) form = page.form('login') form.email = '[email protected]' form.password = 'jKH.P945wruV*qh3' page = agent.submit(form, form.button('submit'))
![Page 70: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/70.jpg)
Advanced features
![Page 71: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/71.jpg)
Testing your client
webmock by Bartosz Blimke (bblimke)
Library for stubbing and setting expectations on HTTP requests in Ruby.
vcr by Myron Marston (myronmarston)
Record your test suite's HTTP interactions and replay them during future test runs for fast, deterministic, accurate tests.
![Page 72: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/72.jpg)
Performance Comparisons
Server Linode Xen VPS (Linode 512) at Fremont, CA Ubuntu 10.10 Apache 2.2, KeepAlive On
Client AWS EC2 (m1.small) at North Virginia (us-east-1b) Ubuntu 12.04 HTTP clients w/ CRuby 1.9.3p286
Multiple downloads of 177B.html and 24MB.zip
Don't take it serious! http://bit.ly/RubyHTTPClientsBenchmarkScript
![Page 73: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/73.jpg)
Multiple 177B downloads
[sec]
![Page 74: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/74.jpg)
[sec]
![Page 75: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/75.jpg)
[sec]
![Page 76: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/76.jpg)
My Recommendations
• Speed is the king => em-http-request, curb w/ multi
• HTML operation, Cookies => Mechanize
• API client => Faraday and adapter based impls
• SSL, Connectivity => httpclient
Check the matrix before you use the libraries
Please let me know when you find incorrect cell
![Page 77: Ruby HTTP clients comparison](https://reader033.vdocument.in/reader033/viewer/2022042700/557ad432d8b42a2c0f8b5350/html5/thumbnails/77.jpg)
Development Timeline