container logging: best practices
TRANSCRIPT
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Container logging: Best practices for consistency
Carmen Puccio
G P S T E C 3 2 9
Principal Solutions Architect
AWS Partner Program
Shashi Prabhakar
Manager, Partner Solutions Architects, Modernization & Linux
AWS Partner Program
Before we begin, let’s categorize the logs
Typically developers are concerned with this because these logs give insight into application performance
• Example application: /var/log/myapp.log
• Web logs: journalctl -u nginx.service --since today
• Transaction logs
• Others
Typically operations teams are concerned with this because these logs give insight into
the stability of the platform and the availability of services
• Syslog and other OS logs
• Audit logs
• Performance metrics
It’s all over the place
# Apache access log
127.0.0.1 - - [28/Jul/2019:10:22:04 -0300] "GET / HTTP/1.0" 200 2216
# Apache error log
127.0.0.1 - - [28/Jul/2019:10:27:32 -0300] "GET /hidden/ HTTP/1.0" 404 7218[Thu Jun 22 11:35:48 2019] [notice] caught SIGTERM, shutting down
# typical expressjs log output
[Mon, 21 Nov 2019 20:52:11 GMT] 200 GET /foo (1ms)
# /var/log/secure
Sep 27 16:48:13 ip-192-168-2-203 sshd[6181]: Invalid user super from 34.80.19.95 port 53100
# .NET core
Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
at error.Program.Main(String[] args) in /Users/puccio/Desktop/myApp/Program.cs:line 11
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
But you said you want to run containers?
What about logging?
all the things!
Log
What about logging? In reality
all the things . . .
almost
Log
Application logging considerations• Implement a standard and easy-to-configure framework: Log4j2 / Log4net, NLog, Winston
• Log to STDOUT/STDERR
• If files are a must, think about the log names: mylog.yyyy-MM-dd HH:mm:ss.SSS.log
• Use logrotate or configure the docker-daemon.json
• {"log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}
• Use log severity levels: DEBUG, INFO, WARN, and ERROR
• Use a standard structure such as JSON
• Custom formats and raw text logs need custom parsing rules in order to extract data for analysis
• Multiline messages are not your friend; dump them to JSON, even in this case
• This also applies to the schema for the fields. Use consistent names and label your services.
• {"time":"2019-11-04 12:34:56.0000","level":"ERROR","message":"HelloWorld","service":"foo","event":{"error":{"name":"RuntimeError","message":"MissingClass is undefined"}}}
• {"time":"2019-11-04 12:35:56.0000","level":"ERROR","message":"HelloWorld","service":"bar","event":{"error":{"exception":"RuntimeError","message":"MissingClass is undefined"}}}
• Use trace ID, for example, X-Amzn-Trace-Id for HTTP headers via the AWS X-Ray SDK
• X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Sampled=1
What just happened?
Frontend
service
AWS Fargate
Backend service
v2
AWS Fargate
Backend service
v1
AWS Fargate
Error
Logging challenges with containers
• No permanent storage• Containers are not pets. If your container dies,
your logs will most likely be gone too.
• No fixed location from a network
address standpoint• You should not care about the IP addresses of
your containers
• No fixed location from a placement
perspective• Even though you can target nodes with things
such as label selectors, it’s all up to the
scheduler
Log collection modes: Sidecar
app-container Sidecar
Core functionality Peripheral tasks
such as
• Logging
• Proxy
• Configuration
• Others
Instance
AWS Fargate
Amazon Elastic
Kubernetes Service
(Amazon EKS)
Amazon Elastic Container
Service (Amazon ECS)
Log collection modes: DaemonSet
app-container
DaemonSet
Core functionality
Peripheral tasks
such as:
• Logging
• Node monitoring
• Others
Instance
app-container
DaemonSet
Core functionality
Peripheral tasks
such as:
• Logging
• Node monitoring
• Others
Instance
AWS Fargate Amazon EKSAmazon ECS
DaemonSet versus SidecarWho is using what for logging?
DaemonSet Sidecar
Collecting logs Files Stdout & stderr/files
Deployment Standard manifest/deployment Standard manifest/deployment
Multitenant isolation Average? Strong?
ScaleRequests/limits
Scheduling conflicts?
Can lead to significant resource
consumption (requests/limits)
Resource usageLow (each node runs a
container)High (each pod runs a container)
Ability to customizeLow (1 DaemonSet covering all
nodes per node)
High (each pod is configured
separately)
Cluster size? Clusters with few functions? Large clusters? PaaS? SaaS?
Anything else? ? ?
How to consolidate? Enter Fluent Bit
Amazon Elastic Container
Registry (Amazon ECR)
Amazon EKS
Amazon ECS
Fluent Bit
DaemonSet
Amazon Simple Storage
Service (Amazon S3)
Amazon Kinesis
Data FirehoseAmazon Athena
Fluent Bit
DaemonSet
app-container
app-container
Related breakouts
CON309 FireLens
ANT312 Amazon Elasticsearch Service: Everything ingest
ANT332 Use Amazon ES to visualize and monitor containerized applications
Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Carmen Puccio
@CarmenAPuccio
Shashi Prabhakar
@shashikantprabhakar
Please complete the session survey in the mobile app.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s start with the basics
$: docker run --name test -d busybox sh -c "while true; do $(echo date); sleep 1; done”
$: docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3bedb2d04f06 busybox "sh -c 'while true; …" 23 seconds ago Up 22 seconds test
$: docker logs test
Tue Sep 24 13:56:52 UTC 2019
Tue Sep 24 13:56:53 UTC 2019
Tue Sep 24 13:56:54 UTC 2019
Tue Sep 24 13:56:55 UTC 2019
Tue Sep 24 13:56:56 UTC 2019
But what just happened?$: docker logs test
Tue Sep 24 13:56:52 UTC 2019
Tue Sep 24 13:56:53 UTC 2019
Tue Sep 24 13:56:54 UTC 2019
Tue Sep 24 13:56:55 UTC 2019
Tue Sep 24 13:56:56 UTC 2019
• Docker supports different logging drivers used to store and/or stream container stdout and stderr logs of the main container process (pid 1)
• The easiest and most used logging method for containerized applications is to write to the stdout and stderr streams
• By default, the Docker daemon uses the json-file logging driver
$: docker info --format '{{.LoggingDriver}}'
json-file
• When you start a container, you can configure it to use a different logging driver than the Docker daemon’s default, using the --log-driver flag
$: docker run --log-driver none --name test2 -d busybox sh -c "while true; do $(echo date); sleep 1; done”
$: docker inspect -f '{{.HostConfig.LogConfig.Type}}' test2
none
Logging driversDriver Description
none No logs are available for the container, and docker logs do not return any output
local Logs are stored in a custom format designed for minimal overhead
json-file The logs are formatted as JSON, the default logging driver for Docker
syslog Writes logging messages to the syslog facility; the syslog daemon must be running on the host machine
journald Writes log messages to journald; the journald daemon must be running on the host machine
gelf Writes log messages to a Graylog extended log format (GELF) endpoint such as Graylog or Logstash
fluentd Writes log messages to fluentd (forward input); the fluentd daemon must be running on the host machine
awslogs Writes log messages to Amazon CloudWatch Logs
splunk Writes log messages to splunk using the HTTP Event Collector
etwlogs Writes log messages as event tracing for Windows (ETW) events (only available on Windows platforms)
gcplogs Writes log messages to Google cloud platform (GCP) logging
logentries Writes log messages to Rapid7 Logentries
The awslogs log driver
$: docker run --log-driver=awslogs \
--log-opt awslogs-region=us-east-1 \
--log-opt awslogs-group=reinvent \
--log-opt awslogs-create-group=true \
--name test3 -d busybox sh -c "while true; do $(echo date); sleep 1; done"
Amazon ECS & AWS Fargate log configuration
"logConfiguration": {
logDriver": "awslogs","fluentd","gelf","json-file","journald","logentries","splunk","syslog","awsfirelens",
"options": {
"string": "string"
...
},
"secretOptions": [{
"name": "string",
"valueFrom": "string"
}]
}
AWS FargateAmazon ECS
/var/log/containers/POD-NAME_NAMESPACE_CONTAINER-NAME-CONTAINER-ID.log
What about Kubernetes?
• Logs should have a separate storage and lifecycle independent of nodes, pods, or containers
• This concept is called cluster-level logging
• Cluster-level logging requires a separate backend to store, analyze, and query logs
• Kubernetes provides no native storage solution for log data, but you can integrate many existing logging solutions into your Kubernetes cluster
stdout
stderr
app-container
my-pod
Instance
Deleted on pod eviction
Fluent Bit on Amazon EKS
/var/log/containers/POD-NAME_NAMESPACE_CONTAINER-NAME-CONTAINER-ID.log
stdout
stderr
app-container
my-pod
Instance
Fluent Bit
DaemonSet
Read logs Look up
metadata
Kubernetes API Server
Analyze the tag and extract
the following metadata:• Pod name
• Namespace
• Container name
• Container ID
Query Kubernetes API Server in order to obtain extra metadata for the POD in question• Pod ID
• Labels
• Annotations