introduction http via curl
DESCRIPTION
新人向けの社内勉強会でつかった資料。HTTPについて勉強して欲しい。TRANSCRIPT
cURLがよくわかーる
(株)永和システムマネジメント諸橋恭介
2011-09-12(月)
curl- transfer a URL
• curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP).
2011-09-12(月)
why cURL is useful?
• -O で末尾ファイル名で保存
• もうwgetに逃げなくて良い
• -Lもあるぞー
• 連番保存もできるぞ
• curl -O http://example.com/wallpapers/img[01-19].jpg
2011-09-12(月)
why cURL is usefulfor Web Developers
• (telnetよりも)簡単にHTTPが使える
• 標準でstdoutに出る!!
2011-09-12(月)
$ telnet www.google.co.jp 80Trying 74.125.153.105...Connected to www.l.google.com.Escape character is '^]'.
GET / HTTP/1.1Host: www.google.co.jp
<結果>
HTTP
2011-09-12(月)
GET / HTTP/1.1
[method] [request-uri] HTTP/1.1
2011-09-12(月)
method
• GET : リソースを読み取る
• POST: リソースに書き込む
• PUT: リソースを書き換える
• DELETE: リソースを消す
2011-09-12(月)
Host: www.google.co.jp
Host: <ホスト名>
2011-09-12(月)
$ telnet www.google.co.jp 80Trying 74.125.153.105...Connected to www.l.google.com.Escape character is '^]'.
GET / HTTP/1.1Host: www.google.co.jp
<結果>
HTTP
2011-09-12(月)
$ curl -v http://www.google.co.jp* About to connect() to www.google.co.jp port 80 (#0)* Trying 74.125.153.106... connected* Connected to www.google.co.jp (74.125.153.106) port 80 (#0)> GET / HTTP/1.1> User-Agent: curl/7.21.7 ...> Host: www.google.co.jp> Accept: */*>< HTTP/1.1 200 OK< Date: Thu, 08 Sep 2011 06:47:44 GMT< Expires: -1< Cache-Control: private, max-age=0
curl
2011-09-12(月)
curl --headcurl --verbose|-v
(curl --trace-ascii dump)
2011-09-12(月)
$ curl -v http://www.google.co.jp* (ry)> GET / HTTP/1.1> User-Agent: curl/7.21.7 ...> Host: www.google.co.jp> Accept: */*>< HTTP/1.1 200 OK< Date: Thu, 08 Sep 2011 06:47:44 GMT< Expires: -1< Cache-Control: private, max-age=0
HTTP Header
2011-09-12(月)
HTTP Request Headers
• Accept : メディアタイプ, JSONとかXML
• Accept-Language: ほしい言語, jaとかen
• Authorization: 認証情報
• Host: 宛先(共用サーバとかで使う)
• Referer: リンク元
• User-Agent: 使ってるブラウザとか
2011-09-12(月)
$ curl -v -H 'Accept-Language: ja' http://twitter.com | nokogrep -a content 'meta[name=description]'
> GET / HTTP/1.1> User-Agent: curl/7.21.7> Host: twitter.com> Accept: */*> Accept-Language: ja>< HTTP/1.1 200 OK...
あなたにとって最も大切なことと瞬時につなぐ。友だち、専門家、好きな芸能人や最新ニュースをフォローしよう。
2011-09-12(月)
$ curl -v -H 'Accept-Language: en' http://twitter.com | nokogrep -a content 'meta[name=description]'
> GET / HTTP/1.1> User-Agent: curl/7.21.7> Host: twitter.com> Accept: */*> Accept-Language: en>< HTTP/1.1 200 OK...
Instantly connect to what's most important to you. Follow your friends, experts, favorite celebrities, and breaking news.
2011-09-12(月)
$ curl -v \ -H 'Accept: application/json' \ http://jobs.dev/jobs | \ ruby -rpp -rjson -e 'pp JSON.parse(STDIN.read)'
> GET /jobs HTTP/1.1> User-Agent: curl/7.21.7> Host: jobs.dev> Accept: application/json
2011-09-12(月)
< HTTP/1.1 200 OK< Content-Type: application/json; charset=utf-8< X-UA-Compatible: IE=Edge< ETag: "1490b2d47b09a5abb92bd7eb09c4c51e"< Cache-Control: max-age=0, private, must-revalidate< X-Runtime: 0.059816< Connection: keep-alive< Transfer-Encoding: chunked
[{"created_at"=>"2011-09-07T05:56:22Z", "deadline"=>"2011-09-07", "description"=>"若者は泥にまみれろ!!!", "id"=>1, "public"=>true, "title"=>"10年泥のような", "updated_at"=>"2011-09-07T09:59:33Z"}]
2011-09-12(月)
HTTP/1.1 200 OKDate: Mon, 12 Sep 2011 01:27:34 GMTServer: hiStatus: 200 OKETag: "ceedd0ac4f0339411aab820e225afcdf"Content-Type: text/html; charset=utf-8Content-Length: 44188Pragma: no-cacheExpires: Tue, 31 Mar 1981 05:00:00 GMTCache-Control: no-cache, no-store, must- revalidate, pre-check=0, post-check=0X-XSS-Protection: 1; mode=block
HTTP Response
2011-09-12(月)
HTTP/1.1 200 OKDate: Mon, 12 Sep 2011 01:27:34 GMTServer: hiStatus: 200 OKETag: "ceedd0ac4f0339411aab820e225afcdf"Content-Type: text/html; charset=utf-8Content-Length: 44188Pragma: no-cacheExpires: Tue, 31 Mar 1981 05:00:00 GMTCache-Control: no-cache, no-store, must- revalidate, pre-check=0, post-check=0X-XSS-Protection: 1; mode=block
HTTP Response
2011-09-12(月)
Status line
• 200 : 正常処理できた
• 302: リダイレクト(歴史的な経緯による)
• 400: クライアントエラー / 再送不可
• 404: ページが見つかりません。
• 500: サーバ側が悪いエラー/再送可
• 503: 処理負荷高すぎていまは無理
2011-09-12(月)
HTTP Status is app UI
• ex: 退会したユーザのプロフィール
• 退会済みとして正常に表示するから200 ?
• 退会ユーザのは出さないから404 ?
• 410(Gone)というのもあるぞー
2011-09-12(月)
HTTP Response Headers
• Location: 新しいリソースがあればそこを指す
• Server: 相手サーバ
• Content-Type : メディアタイプ, JSONとかXML
• Content-Length: レスポンスのサイズ
• Expire, ETag, Cache-Control: キャッシュ制御
2011-09-12(月)
curl --data|-d
2011-09-12(月)
$ curl -v -d [email protected] \ -d password=hogehoge \ http://example.com/users
POST /users HTTP/1.1User-Agent: curl/7.21.7Host: example.comAccept: */*Content-Length: 36Content-Type: application/x-www-form-urlencoded
POST via curl
2011-09-12(月)
HTTP/1.1 201 CreatedContent-Type: application/jsonDate: Mon, 12 Sep 2011 01:43:41 GMTServer: journey/0.4.0Content-Length: 71Connection: keep-alive
{"email":"[email protected]","_id":"4e6d63cd9815710100000005","profiles":[]}
2011-09-12(月)
curl --user|-u
2011-09-12(月)
$ curl -v -u [email protected]:hogehoge \ http://example.com/account
GET /account HTTP/1.1Authorization: Basic bW9yb0BleGFtcGxlLmpwOmhvZ2Vob2dlUser-Agent: curl/7.21.7Host: example.comAccept: */*
HTTP Authentication
2011-09-12(月)
HTTP/1.1 200 OKContent-Type: application/jsonDate: Mon, 12 Sep 2011 01:51:00 GMTServer: journey/0.4.0Content-Length: 71Connection: keep-alive
{"email":"[email protected]","_id":"4e6d63cd9815710100000005","profiles":[]}
HTTP Authentication
2011-09-12(月)
cookie
• HTTPにおいて、クッキー{Cookie}とは、元々ユーザエージェント(Webブラウザ)によって保存される「小さな」ファイルを指します。
クッキーは、主に「状態管理」のために利用され、具体的には、たとえばオンラインショッピングでの「ショッピングカート」を実現するために利用されています。
http://www.studyinghttp.net/cookies2011-09-12(月)
[res] Set-Cookie[req] Cookie
• Netscape社が開発したクッキーという技術は、Set-CookieとCookieという2種類のHTTPヘッダのやりとりで状態を管理しようとするものです。
http://www.studyinghttp.net/cookies#HeadersForCookie
2011-09-12(月)
2011-09-12(月)
2011-09-12(月)
curl --cookie|-b
2011-09-12(月)
$ curl -q http://jobs.dev |\ nokogrep '#header p.greeting'
Who are you? Sign in
without cookie
2011-09-12(月)
$ curl -v -b '_jobs_session=BAh7CEkiD3Nl(略)' http://jobs.dev |\ nokogrep '#header p.greeting'
GET / HTTP/1.1User-Agent: curl/7.21.7Host: jobs.devAccept: */*Cookie: _jobs_session=BAh7CEkiD3Nl(略)
Cookie & cURL
2011-09-12(月)
HTTP/1.1 200 OKContent-Type: text/html; charset=utf-8X-UA-Compatible: IE=EdgeETag: "8c0c63144925e4cc29b655906c221a3f"Set-Cookie:_jobs_session=BAh7CEkiD3Nl(略)X-Runtime: 0.156343Connection: keep-aliveTransfer-Encoding: chunked
Hi [email protected] Sign out
Cookie & cURL
2011-09-12(月)
まとめ
2011-09-12(月)
$ curl -v \ -d 'user[email][email protected]'\ -d 'user[password]=hogehoge'\ http://jobs.dev/users/sign_in POST /users/sign_in HTTP/1.1User-Agent: curl/7.21.7Host: jobs.devAccept: */*Content-Length: 51Content-Type: application/x-www-form-urlencoded
HTTP/1.1 302 Moved TemporarilyLocation: http://jobs.dev/Content-Type: text/html; charset=utf-8X-UA-Compatible: IE=EdgeCache-Control: no-cacheSet-Cookie: _jobs_session=BAh7CE(略)X-Runtime: 0.217055Connection: keep-aliveTransfer-Encoding: chunked
<html><body>You are being <a href="http://jobs.dev/">redirected</a>.</body></html>
$ curl -v \ -b '_jobs_session=BAh7CE(略)'\ http://jobs.dev/ | \ nokogrep '#header p.greeting'GET / HTTP/1.1User-Agent: curl/7.21.7Host: jobs.devAccept: */*Cookie: _jobs_session=BAh7CE(略)
HTTP/1.1 200 OKContent-Type: text/html; charset=utf-8X-UA-Compatible: IE=EdgeETag: "98d16875bdceb8c5451e7706e6071ece"Cache-Control: max-age=0, private, must-revalidateSet-Cookie: _jobs_session=BAh7CUki(略)X-Runtime: 0.193951Connection: keep-aliveTransfer-Encoding: chunked
Hi [email protected] Sign out
2011-09-12(月)
$ curl -v \ -d 'user[email][email protected]'\ -d 'user[password]=hogehoge'\ http://jobs.dev/users/sign_in POST /users/sign_in HTTP/1.1User-Agent: curl/7.21.7Host: jobs.devAccept: */*Content-Length: 51Content-Type: application/x-www-form-urlencoded
HTTP/1.1 302 Moved TemporarilyLocation: http://jobs.dev/Content-Type: text/html; charset=utf-8X-UA-Compatible: IE=EdgeCache-Control: no-cacheSet-Cookie: _jobs_session=BAh7CE(略)X-Runtime: 0.217055Connection: keep-aliveTransfer-Encoding: chunked
<html><body>You are being <a href="http://jobs.dev/">redirected</a>.</body></html>
$ curl -v \ -b '_jobs_session=BAh7CE(略)'\ http://jobs.dev/ | \ nokogrep '#header p.greeting'GET / HTTP/1.1User-Agent: curl/7.21.7Host: jobs.devAccept: */*Cookie: _jobs_session=BAh7CE(略)
HTTP/1.1 200 OKContent-Type: text/html; charset=utf-8X-UA-Compatible: IE=EdgeETag: "98d16875bdceb8c5451e7706e6071ece"Cache-Control: max-age=0, private, must-revalidateSet-Cookie: _jobs_session=BAh7CUki(略)X-Runtime: 0.193951Connection: keep-aliveTransfer-Encoding: chunked
Hi [email protected] Sign out
2011-09-12(月)
$ curl -v \ -d 'user[email][email protected]'\ -d 'user[password]=hogehoge'\ http://jobs.dev/users/sign_in POST /users/sign_in HTTP/1.1User-Agent: curl/7.21.7Host: jobs.devAccept: */*Content-Length: 51Content-Type: application/x-www-form-urlencoded
HTTP/1.1 302 Moved TemporarilyLocation: http://jobs.dev/Content-Type: text/html; charset=utf-8X-UA-Compatible: IE=EdgeCache-Control: no-cacheSet-Cookie: _jobs_session=BAh7CE(略)X-Runtime: 0.217055Connection: keep-aliveTransfer-Encoding: chunked
<html><body>You are being <a href="http://jobs.dev/">redirected</a>.</body></html>
$ curl -v \ -b '_jobs_session=BAh7CE(略)'\ http://jobs.dev/ |\ nokogrep '#header p.greeting'
GET / HTTP/1.1User-Agent: curl/7.21.7Host: jobs.devAccept: */*Cookie: _jobs_session=BAh7CE(略)
HTTP/1.1 200 OKContent-Type: text/html; charset=utf-8X-UA-Compatible: IE=EdgeETag: "98d16875bdceb8c5451e7706e6071ece"Cache-Control: max-age=0, private, must-revalidateSet-Cookie: _jobs_session=BAh7CUki(略)X-Runtime: 0.193951Connection: keep-aliveTransfer-Encoding: chunked
Hi [email protected] Sign out
2011-09-12(月)
$ curl -v \ -d 'user[email][email protected]'\ -d 'user[password]=hogehoge'\ http://jobs.dev/users/sign_in POST /users/sign_in HTTP/1.1User-Agent: curl/7.21.7Host: jobs.devAccept: */*Content-Length: 51Content-Type: application/x-www-form-urlencoded
HTTP/1.1 302 Moved TemporarilyLocation: http://jobs.dev/Content-Type: text/html; charset=utf-8X-UA-Compatible: IE=EdgeCache-Control: no-cacheSet-Cookie: _jobs_session=BAh7CE(略)X-Runtime: 0.217055Connection: keep-aliveTransfer-Encoding: chunked
<html><body>You are being <a href="http://jobs.dev/">redirected</a>.</body></html>
$ curl -v \ -b '_jobs_session=BAh7CE(略)'\ http://jobs.dev/ | \ nokogrep '#header p.greeting'GET / HTTP/1.1User-Agent: curl/7.21.7Host: jobs.devAccept: */*Cookie: _jobs_session=BAh7CE(略)
HTTP/1.1 200 OKContent-Type: text/html; charset=utf-8X-UA-Compatible: IE=EdgeETag: "98d16875bdceb8c5451e7706e6071ece"Cache-Control: max-age=0, private, must-revalidateSet-Cookie: _jobs_session=BAh7CUki(略)X-Runtime: 0.193951Connection: keep-aliveTransfer-Encoding: chunked
Hi [email protected] Sign out
2011-09-12(月)
• HTTPはだいたいテキスト(一部のbody除く)。読み書きできる。
• cURLやWebブラウザはいい感じにテキストを送るツール。
• 上手く動かない場合には、そのテキストがどうなってるのか覗いてみるんだ。
2011-09-12(月)
• http://www.studyinghttp.net/
• http://www.studyinghttp.net/rfc_ja/rfc2616
• http://amazon.jp/dp/4774142042/morodiary05-22/ref=nosim/
see also
2011-09-12(月)