un-broken logging - technologyug - leeds - matthew skelton
TRANSCRIPT
![Page 1: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/1.jpg)
Un-Broken Loggingthe foundation of software operability
TechnologUG, Leeds #techugThursday 22th October 2015
Matthew SkeltonSkelton Thatcher Consulting
@matthewpskelton
![Page 2: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/2.jpg)
![Page 3: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/3.jpg)
The way we use logging is (often) broken
How to make our logging more awesome
Why we should care
![Page 4: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/4.jpg)
![Page 5: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/5.jpg)
![Page 6: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/6.jpg)
![Page 7: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/7.jpg)
Matthew Skelton
@matthewpskelton
#techug
![Page 8: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/8.jpg)
![Page 9: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/9.jpg)
![Page 10: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/10.jpg)
![Page 11: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/11.jpg)
![Page 12: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/12.jpg)
@Operability
#operability
WhoOwnsMyOperability.com
![Page 13: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/13.jpg)
confession:
I am a big fan of logging
![Page 14: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/14.jpg)
![Page 15: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/15.jpg)
exceptional situationsedge cases
metricsanalytics‘audits’
…@evanphx
![Page 16: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/16.jpg)
execution trace
![Page 17: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/17.jpg)
BAD STUFF
![Page 18: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/18.jpg)
Logging is often unloved
1. Discontinuous
2. Errors only, or arbitrary
3. ‘Bolted on’
4. No aggregation & search
5. Specify severity up front
![Page 19: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/19.jpg)
GOOD STUFF
![Page 20: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/20.jpg)
How to make logging awesome
1. Continuous event IDs
2. Transaction tracing
3. Log aggregation & search tools
4. Design for logging
5. Decoupled severity
![Page 21: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/21.jpg)
reduce time-to-detectincrease team engagement
increase configurabilityenhance DevOps collaboration
#operability
![Page 22: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/22.jpg)
Background
![Page 23: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/23.jpg)
Autonomous weather station
![Page 24: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/24.jpg)
![Page 25: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/25.jpg)
![Page 26: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/26.jpg)
MRI brain scan imaging
![Page 27: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/27.jpg)
![Page 28: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/28.jpg)
![Page 29: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/29.jpg)
Oil well monitoring
![Page 30: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/30.jpg)
![Page 31: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/31.jpg)
![Page 32: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/32.jpg)
Web-scale systems
![Page 33: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/33.jpg)
![Page 34: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/34.jpg)
![Page 35: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/35.jpg)
![Page 36: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/36.jpg)
![Page 37: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/37.jpg)
![Page 38: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/38.jpg)
![Page 39: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/39.jpg)
logging makes things work
![Page 40: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/40.jpg)
(event sourcing)
(structured logging)
(CQRS)
![Page 41: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/41.jpg)
How is logging usually broken?
![Page 42: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/42.jpg)
Logging is often unloved
1. Discontinuous
2. Errors only, or arbitrary
3. ‘Bolted on’
4. No aggregation & search
5. Specify severity up front
![Page 43: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/43.jpg)
![Page 44: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/44.jpg)
using logging mainly for errors
![Page 45: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/45.jpg)
![Page 46: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/46.jpg)
inconsistent use of logging
![Page 47: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/47.jpg)
![Page 48: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/48.jpg)
logging slows down the software
![Page 49: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/49.jpg)
![Page 50: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/50.jpg)
logging ‘pollutes’ my precious domain model
![Page 51: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/51.jpg)
![Page 52: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/52.jpg)
logging is just for those weird Ops people
![Page 53: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/53.jpg)
![Page 54: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/54.jpg)
logging assumed to be free ($0) to implement
no budget for aggregating logs across machines
![Page 55: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/55.jpg)
![Page 56: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/56.jpg)
log aggregation happens only in Production
logs not available to Devs
![Page 57: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/57.jpg)
![Page 58: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/58.jpg)
fights over log severity levels
![Page 59: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/59.jpg)
![Page 60: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/60.jpg)
poor time synchronisation
![Page 61: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/61.jpg)
![Page 62: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/62.jpg)
Some history, with pirates
![Page 63: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/63.jpg)
![Page 64: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/64.jpg)
![Page 65: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/65.jpg)
![Page 66: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/66.jpg)
![Page 67: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/67.jpg)
weather, course, sightings, latitude, longitude, …
(even when quiet)
![Page 68: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/68.jpg)
John
Har
rison
![Page 69: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/69.jpg)
Why log?
![Page 70: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/70.jpg)
verificationtraceability
accountability
charting the waters
![Page 71: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/71.jpg)
![Page 72: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/72.jpg)
- June 13th –Pirates!!!!
- Weds –Sharks!!!
- 19th Jun –BIGGER sharks!!!!
![Page 73: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/73.jpg)
How to make logging awesome
![Page 74: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/74.jpg)
How to make logging awesome
1. Continuous event IDs
2. Transaction tracing
3. Log aggregation & search tools
4. Design for logging
5. Decoupled severity
![Page 75: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/75.jpg)
![Page 76: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/76.jpg)
Storage I/O
Worker Job
Queue
Upload
![Page 77: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/77.jpg)
Continuous event IDs
![Page 78: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/78.jpg)
How many distinct event types (state transitions) in
your application?
![Page 79: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/79.jpg)
![Page 80: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/80.jpg)
represent distinct states
![Page 81: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/81.jpg)
enum
Human-readable sets: unique values, sparse, immutable
C#, Java, Python, node(Ruby, PHP, …)
![Page 82: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/82.jpg)
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
BasketItemAdded = 60001,
BasketItemRemoved = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}
![Page 83: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/83.jpg)
Technical
Domain
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
BasketItemAdded = 60001,
BasketItemRemoved = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}
![Page 84: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/84.jpg)
BasketItemAdded = 60001
![Page 85: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/85.jpg)
BasketItemAdded = 60001
BasketItemRemoved = 60002
![Page 86: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/86.jpg)
BasketItemAdded = 60001
BasketItemRemoved = 60002
![Page 87: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/87.jpg)
represent distinct states
![Page 88: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/88.jpg)
![Page 89: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/89.jpg)
OrderSvc_BasketItemAdded
![Page 90: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/90.jpg)
![Page 91: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/91.jpg)
Monolith to microservices:debugger does not have the full view
![Page 92: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/92.jpg)
Even with remote debugger, it’s boring to attach and detach
![Page 93: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/93.jpg)
![Page 94: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/94.jpg)
Storage I/O
Worker Job
Queue
Upload
![Page 95: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/95.jpg)
![Page 96: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/96.jpg)
![Page 97: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/97.jpg)
![Page 98: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/98.jpg)
Transaction tracing
![Page 99: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/99.jpg)
‘Unique-ish’ identifier for each request
Passed through downstream layers
![Page 100: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/100.jpg)
Unique-ish ID
![Page 101: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/101.jpg)
What about APM?
![Page 102: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/102.jpg)
APM gives us application insightBUT
How much do we learn? Is APM available on the Dev box?
It’s not just ‘an Ops problem’!
![Page 103: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/103.jpg)
![Page 104: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/104.jpg)
Helps us to understand how the software really works
Small overhead is worth it
![Page 105: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/105.jpg)
Configurable severity levels
![Page 106: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/106.jpg)
Which log level is right?
![Page 107: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/107.jpg)
DEBUG, INFO, WARNING, ERROR, CRITICAL
![Page 108: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/108.jpg)
Log level should *not* be fixed at compile or build time!
![Page 109: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/109.jpg)
Tune log levels
![Page 110: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/110.jpg)
Tune log levels
![Page 111: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/111.jpg)
Tune log levels
![Page 112: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/112.jpg)
{
"eventmappings": {
"events": {
"event": [ {
"id": "CacheServiceStarted",
"severity": { "level": "Information" }
}, {
"id": "PageCachePurged",
"severity": { "level": "Debug" },
"state": { "enabled": false }
}, {
"id": "DatabaseConnectionTimeOut",
"severity": { "level": "Error" }
} ]
}
}
}
![Page 113: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/113.jpg)
![Page 114: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/114.jpg)
Tune severity levels of specific event IDs
![Page 115: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/115.jpg)
![Page 116: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/116.jpg)
Event tracing
Use enumerations (or closest thing)
Technical and Domain event types
Distributed systems: debuggers less useful
Trace calls with ‘unique-enough’ handles
Tune log levels via config
![Page 117: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/117.jpg)
Log aggregation & search tools
![Page 118: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/118.jpg)
![Page 119: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/119.jpg)
![Page 120: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/120.jpg)
![Page 121: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/121.jpg)
![Page 122: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/122.jpg)
Design for log aggregation
![Page 123: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/123.jpg)
develop the software using log aggregation as a first-class thing
![Page 124: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/124.jpg)
![Page 125: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/125.jpg)
![Page 126: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/126.jpg)
stories for testing logging
![Page 127: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/127.jpg)
BasketItemAdded
grep BasketItem
![Page 128: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/128.jpg)
logging is (‘just’) another system component
![Page 129: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/129.jpg)
NTP
![Page 130: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/130.jpg)
Dev and Ops collaboration*
* and testers too!
![Page 131: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/131.jpg)
![Page 132: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/132.jpg)
Where?
![Page 133: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/133.jpg)
![Page 134: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/134.jpg)
auditingcompliance
pre-emptive fault diagnosisperformance
metrics…
![Page 135: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/135.jpg)
Recap
![Page 136: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/136.jpg)
Logging is often unloved
1. Discontinuous
2. Errors only, or arbitrary
3. ‘Bolted on’
4. No aggregation & search
5. Specify severity up front
![Page 137: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/137.jpg)
How to make logging awesome
1. Continuous event IDs
2. Transaction tracing
3. Log aggregation & search tools
4. Design for logging
5. Decoupled severity
![Page 138: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/138.jpg)
logging makes things work
![Page 139: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/139.jpg)
“There is no thought behind aspect-oriented programming”
![Page 140: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/140.jpg)
MINDFUL LOGGING (?!)
![Page 141: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/141.jpg)
database transaction logs
![Page 142: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/142.jpg)
‘Structured Logging’TW: “Adopt” (May 2015)
https://www.thoughtworks.com/radar/techniques/structured-logging
http://gregoryszorc.com/
.NET: http://serilog.net/Java: https://github.com/fluent/fluent-logger-java
![Page 143: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/143.jpg)
sanity
![Page 144: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/144.jpg)
![Page 145: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/145.jpg)
More
Ditch the Debugger and Use Log Analysis Instead
Matthew Skelton
https://blog.logentries.com/2015/07/ditch-the-debugger-and-use-log-analysis-instead/
![Page 146: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/146.jpg)
More
Using Log Aggregation Across Dev & Ops: The Pricing Advantage
Rob Thatcher
https://blog.logentries.com/2015/08/using-log-aggregation-across-dev-ops-the-pricing-
advantage/
![Page 148: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/148.jpg)
Books
operabilitybook.comoperationalfeatures.com
![Page 149: Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton](https://reader031.vdocument.in/reader031/viewer/2022030303/587b990a1a28ab4e4f8b6ff7/html5/thumbnails/149.jpg)
Thank you
http://skeltonthatcher.com/[email protected]
@SkeltonThatcher
+44 (0)20 8242 4103
@matthewpskelton