5 manas gupta - achieving common ground via logging
DESCRIPTION
Achieving Common Ground via Logging Slides from - http://devopsdays.org/events/2013-newyork/proposals/Achieving%20common%20ground%20via%20Logging/TRANSCRIPT
AchievingCommon Ground via
Logging
Second Class Citizens
Why?
Why?(?:(?:(?:\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?:(?![0-9])(?:(?:2[0123]|[01][0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?:(?:(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))|(?(?![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (?(?(?:[\w._/%-]+))(?:\[(?\b(?:[1-9][0-9]*)\b)\])?): (?(?![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?(?:[+-]?(?:[0-9]+))) \[(?(?(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)/(?[0-9]+):(?(?![0-9])(?(?:2[0123]|[01][0-9])):(?(?:[0-5][0-9]))(?::(?(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?(?:[+-]?(?:[0-9]+))))\] (?\S+) (?\S+)/(?\S+) (?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?\S+) (?\S+) (?\S+) (?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?\S+) (?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+))))
Thank you @jordansissel
All good?
Yes. But...
DevOps
SupportProducts
Where to begin?
“Request Completed”
Masterzen - 10 Commandments of Logging
Context
Request (f67a) completed for user
George-Michael (token: weqeqweq)
to fakeblock in 3.16275 seconds.
Context
{ "msg": "Request completed", "runtime_s": 3.16275, "request": { "action": "fakeblock", "accessToken": "yjfhakfsf", "user": "George-Michael" }, "requestId": "f67a"}
Context + Correlation{ .... "msg": "Request completed", "runtime_s": 3.16275, "request": { "action": "faceblock", "accessToken": "yjfhakfsf", "user": "George-Michael" }, "requestId": "f67a", "host": "i-90zas2",}
{ ... "host": "load-balancer", "status": "200", "request": "POST /url/ HTTP/1.1", "http_user_agent": "iphone", "x_requestId": "f67a", "request_time": 3.9}
Ship them!
level : ‘error’
... and class : ‘mailer’
... and host : ‘i-8904’
action : ‘fakeblock’
... and runtime > 2.1
level : ‘error’
... and ...
... and ...
is count > threshold
Critical, Warn, OK?
Wait a sec...
{ "msg": "Request completed", "runtime_s": 3.16275, "request": { "action": "fakeblock", "accessToken": "yjfhakfsf", "user": "George-Michael" }, "requestId": "f67a"}
type : ‘user_log’and
user : ‘George-Michael’
“invalid identifier returned”
But we are a small team...
Defaults work well
• 40 GB index/day
• 3k events/sec indexed
• m1.xlarge x 2
• Chef Cookbook available
• pyes (ES Api in py)
Bonus
Thanks
Manas Gupta@ngrep_q