ruby application based on http
DESCRIPTION
Do you have an experience to write a client application based on http? Such as fetching the contacts from email or writing a IM client. Some of the servers provide apis which make life better, but most of them not. So how can we get the data from these servers or communicating with these servers? This talk will teach you how to analysis the packets between client and server and share my experience about how to write the client application and how to do the test and refactor.TRANSCRIPT
Ruby application based on http
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Who am I?
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Richard Huang a.k.a flyerhzm
Work at Ekohe
http://huangzhimin.com
@flyerhzm
http://github.com/flyerhzm
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Http Server
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Oauth
JSONXML
Params Signature
Web Service
RestfulCloud Computing
HTML XHTML
Http protocol
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Request - Response
Header - Body
Client(Browser)
RubyConfChinaServer
Get /session/new
Return session/new page
Post /session
Login success, redirect to /
Get /
Return home page
Ruby API? Official? Community?
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Get data from third-party website
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
if the website provides an api if the website provides an official/community ruby api use the ruby api else write the ruby api by your self endelse write the ruby codes act as a http client (browser)end
What if fetch contacts from gmail?
What if fetch timelines from twitter?
What if fetch friends from facebook?
What if fetch contacts from msn?
What if fetch repositories from github?
Lucky, there are ruby api to do theses!
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
What if fetch contacts from 163 mail?
What if fetch weather from sina?
What if fetch friends from kaixin001?
What if fetch contacts from fetion?
There are no ruby api to do theses!
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Act as a http client (browser)
Analyze the packets sent by http client (browser) in sniffer tools.
Resend same packets by Ruby.
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Sniffer
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
WiresharkDetect any protocol (ssh, ftp, tcp, http, …)
Cross platform (Windows, Linux, Mac OS)
Both raw data and printable text
Not easy to view https encrypted data
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Wireshark
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
filter http one http request
TCP/IPHTTP
HTTP Header/Body
Raw Data andPrintable text
Wireshark
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
one http response
HTTP AnalyzerOnly detect http/https protocol
Concentrated on http protocol (request timing, cookie, …)
Easy to view https encrypted data
Windows version only
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
HTTP Analyzer
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
http request header
http response header
HTTP Analyzer
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
http response
body
HTTP Analyzer
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
http request timing
Get session/new
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Get /session/new HTTP/1.1\r\nHost: rubyconfchina.org\r\nConnect: keep-alive\r\nUser-Agent: Mozilla/5.0 AppleWebKit/553.4\r\nAccept: text/html,application/xhtml+xml,text/html;q=0.9\r\nAccept-Encoding: gzip,deflate\r\nAccept-Language: en-US,en;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n\r\n
Get session/new response
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
HTTP/1.1 200 OK\r\nServer: nginx/0.5.32\r\nDate: Sun, 20 Jun 2010 09:25:08 GMT\r\nContent-Type: text/html; charset=utf-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nCache-Control: private, max-age=0, must-revalidate\r\nContent-Encoding: gzip\r\n\r\n<!DOCTYPE html PUBLIC …>\n<html>\n…
Post /session
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Post /session HTTP/1.1\r\nHost: rubyconfchina.org\r\nConnect: keep-alive\r\nUser-Agent: Mozilla/5.0 AppleWebKit/553.4\r\nReferer: http://rubyconfchina.org/session/new\r\nContent-Length: 106\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept: text/html,application/xhtml+xml,text/html;q=0.9\r\n\r\nauthenticity_token=…&login=flyerhzm&password=…&commit=Log+in
Post /session response
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
HTTP/1.1 302 Moved Temporarily\r\nServer: nginx/0.5.32\r\nDate: Sun, 20 Jun 2010 09:25:31 GMT\r\nContent-Type: text/html; charset=utf-8\r\nConnect: keep-alive\r\nLocation: http://rubyconfchina.org/\r\nCache-Control: no-cache\r\nSet-Cookie: auth_token=; _euruko_session=BAh7CTo\r\nContent-Length: 91\r\n\r\n<html><body>You are being <a href=http://rubyconfchina.org/>redirected</a>.</body></html>
Get /
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Get / HTTP/1.1\r\nHost: rubyconfchina.org\r\nConnect: keep-alive\r\nUser-Agent: Mozilla/5.0 AppleWebKit/553.4\r\nAccept: text/html,application/xhtml+xml,text/html;q=0.9\r\nAccept-Encoding: gzip,deflate\r\nAccept-Language: en-US,en;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nCookie: auth_token=; _euruko_session=BAh7CTo\r\n\r\n
Http protocol
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
HTTP/1.1 200 OK\r\nServer: nginx/0.5.32\r\nDate: Sun, 20 Jun 2010 09:25:08 GMT\r\nContent-Type: text/html; charset=utf-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nCache-Control: private, max-age=0, must-revalidate\r\nContent-Encoding: gzip\r\n\r\n<!DOCTYPE html PUBLIC …>\n<html>\n…
Resend same packets by Ruby
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Resend same packets by Ruby
Send a GET request
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
uri = URI.parse("http://rubyconfchina.org/session/new")http = Net::HTTP.new(uri.host, uri.port)headers = { "User-Agent" => "Mozilla/5.0 AppleWebKit/553.4", "Accept" => "…"}response = http.request_get(uri.request_uri, headers) puts response.body =~ /Login/ # 1942
Resend same packets by Ruby
Parse GET response
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
raise Exception, "get response should success" unless Net::HTTPSuccess === repsonse
if response.body =~ %r(<input name="authenticity_token".*?value="(.*>)" />)m authenticity_token = $1end
Resend same packets by Ruby
Send a POST request
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
uri = URI.parse("http://rubyconfchina.org/session")http = Net::HTTP.new(uri.host, uri.port)body = "authenticity_token=#{authenticity_token}&login=flyerhzm&password=…"headers = { "User-Agent" => "Mozilla/5.0 AppleWebKit/553.4", "Accept" => "…"}response = http.request_post(uri.request_uri, body, headers)
Resend same packets by Ruby
Parse POST response
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
raise Exception, "get response should be redirect" unless Net::HTTPRedirection === repsonse
cookie = response["Set-Cookie"]cookie = parse_cookie(cookie)Location = response["Location"]
Resend same packets by Ruby
Send a GET request
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
uri = URI.parse(Location)http = Net::HTTP.new(uri.host, uri.port)headers = { "User-Agent" => "Mozilla/5.0 AppleWebKit/553.4", "Accept" => "…", "Cookie" => cookie}response = http.request_get(uri.request_uri, headers) raise Exception, "get repsonse should success" unless Net::HTTPSuccess === responseputs response.body =~ /Login/ # nil
Facility
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
MechanizeA ruby library that makes automated web interaction
easy.
Automatically location redirection
Automatically cookies management
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Mechanize
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
a = Mechanize.new do |agent| agent.user_agent_alias = 'Linux Firefox'enda.get('http://rubyconfchina.org/session/new') do |page| puts page.body =~ /Login/ # 1942 home_page = page.form_with(:action => '/session') do |f| f.login = 'flyerhzm' f.password = '…' end.click_button puts home_page.body =~ /Login/ # nilend
Test
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
TestTest http requests
Test http responses
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Test request
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Client Server
GET /ht/sd.aspx?i=1 HTTP/1.1\r\nHost: 221.176.31.39\r\n…\r\n\r\nR fetion.com.cn SIP-C/4.0F: 730020377I: 1Q: 1 RCN: 19D28D4978125CAA4F6E54277BA7D9EFCL: type="pc" ,version="3.6.2020"
SIPP
sid = 730020377i = 0
Test request
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
@fetion.stubs(:sid).returns(730020377)@fetion.stubs(:i).returns(0)@fetion.stubs(:guid).returns("19D28D4978125CAA4F6E54277BA7D9EF")
expected_sipc_message =<<-EOFR fetion.com.cn SIP-C/4.0F: 730020377I: 1Q: 1 RCN: 19D28D4978125CAA4F6E54277BA7D9EFCL: type="pc" ,version="3.6.2020"
SIPPEOFexpected_sipc_message.gsub!("\n", "\r\n").chomp!SipcMessage.register_first(@fetion).should == expected_sipc_message
Copied from Sniffer tool
Test response
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Client Server
HTTP/1.1 200 OK\r\nServer: nginx/0.5.32\r\n…\r\nSet-Cookie: ssic=CBIOAAAm+FiuQgpcnFi; path=/\r\n\r\n<?xml version="1.0" encoding="utf-8" ?><results status-code="200"> <user uri="sip:[email protected];p=6907" /></results>
FakewebA test helper for faking responses to web requests.
No need to modify existing code
No need to write extensive stubs
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Test response
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
FakeWeb.register_uri( :get, 'https://uid.fetion.com.cn/ssiportal/SSIAppSignInV4.aspx? mobileno=15800681509&digest=79cd56b93f21298dc8ae9d26de1258e3d', :body => %Q|<?xml version="1.0" encoding="utf-8" ?><results status-code="200"><user uri="sip:[email protected];p=6907" /></results>|, :set_cookie => Q|ssic=CBIOAAAm+FiuQgpcnFi; path=/|)@[email protected]_code.should == "200"@fetion.sid.should == "730020377"
Copied from Sniffer tool
Test response
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
FakeWeb.register_uri( :get, 'https://uid.fetion.com.cn/ssiportal/SSIAppSignInV4.aspx? mobileno=15800681509&digest=79cd56b93f21298dc8ae9d26de1258e3d', :body => %Q|<?xml version="1.0" encoding="utf-8" ?><results status-code="401" desc="password error" />|, :status => ['401', 'password error'])lambda { @fetion.login }.should raise_exception(Fetion::PasswordError)
Copied from Sniffer tool
Test ProcessDo some stubs
Copy the http request/response as expected message from sniffer tools
Execute the method to generate http request/respones
Test with expected message
TDD (optional)
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
Performance
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
net-http-persistenta thread-safe wrapper for Net::HTTP that performs
persistent connections for you
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
user system total real
Net::HTTP 8.410000 2.400000
10.810000
( 17.333671)
Net::HTTP::Persistent
8,110000 0.880000
8.990000 ( 12.190094)
typhoeusParallel http requests
based on libcurl and libcurl-multi
500ms response, call 20 times
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development
user system total real
Net::HTTP 0.030000 0.010000 0.040000
( 10.054327)
typhoeus 0.020000 0.070000 0.090000
( 0.508817)
Q&A
Thank you
www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development