5 manas gupta - achieving common ground via logging

AchievingCommon Ground via

Logging

Second Class Citizens

Why?

Why?(?:(?:(?:\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?:(?![0-9])(?:(?:2[0123]|[01][0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?:(?:(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))|(?(?![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (?(?(?:[\w._/%-]+))(?:\[(?\b(?:[1-9][0-9]*)\b)\])?): (?(?![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?(?:[+-]?(?:[0-9]+))) \[(?(?(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)/(?[0-9]+):(?(?![0-9])(?(?:2[0123]|[01][0-9])):(?(?:[0-5][0-9]))(?::(?(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?(?:[+-]?(?:[0-9]+))))\] (?\S+) (?\S+)/(?\S+) (?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?\S+) (?\S+) (?\S+) (?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?\S+) (?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+))))

Thank you @jordansissel

All good?

Yes. But...

DevOps

SupportProducts

Where to begin?

“Request Completed”

Masterzen - 10 Commandments of Logging

Context

Request (f67a) completed for user

George-Michael (token: weqeqweq)

to fakeblock in 3.16275 seconds.

Context

{ "msg": "Request completed", "runtime_s": 3.16275, "request": { "action": "fakeblock", "accessToken": "yjfhakfsf", "user": "George-Michael" }, "requestId": "f67a"}

Context + Correlation{ .... "msg": "Request completed", "runtime_s": 3.16275, "request": { "action": "faceblock", "accessToken": "yjfhakfsf", "user": "George-Michael" }, "requestId": "f67a", "host": "i-90zas2",}

{ ... "host": "load-balancer", "status": "200", "request": "POST /url/ HTTP/1.1", "http_user_agent": "iphone", "x_requestId": "f67a", "request_time": 3.9}

Ship them!

level : ‘error’

... and class : ‘mailer’

... and host : ‘i-8904’

action : ‘fakeblock’

... and runtime > 2.1

level : ‘error’

... and ...

... and ...

is count > threshold

Critical, Warn, OK?

Wait a sec...

{ "msg": "Request completed", "runtime_s": 3.16275, "request": { "action": "fakeblock", "accessToken": "yjfhakfsf", "user": "George-Michael" }, "requestId": "f67a"}

type : ‘user_log’and

user : ‘George-Michael’

“invalid identifier returned”

But we are a small team...

Defaults work well

• 40 GB index/day

• 3k events/sec indexed

• m1.xlarge x 2

• Chef Cookbook available

• pyes (ES Api in py)

Bonus

Thanks

Manas Gupta@ngrep_q