pdx devops: intro to logstash
DESCRIPTION
TRANSCRIPT
![Page 1: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/1.jpg)
Logstash
![Page 2: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/2.jpg)
MeSysadmin at NetXposure Inc.
“DevOps” because I sit next to the Java engineers (there wasn’t any
desk space next to the other sysadmin)
First grownup job
BS in Microbiology
https://github.com/nickchappell
![Page 3: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/3.jpg)
Splunk
500MB/day limit with demo version
![Page 4: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/4.jpg)
Whaddya do?
tail -f /var/log/**/* | grep somestuff ?
![Page 5: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/5.jpg)
LogstashLog aggregator/router/processor
http://logstash.net/
![Page 6: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/6.jpg)
Elasticsearch +
Logstash +
Kibana
The ELK stack
![Page 7: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/7.jpg)
Logstash internalsWritten in Ruby, but uses JRuby for a runtime
(so it will actually use more than 1 core)
•Logs enter through inputs!•Each incoming log line gets represented internally as an event
with fields!•Filters process events (add, remove or transform fields and their
contents) •Outputs send events elsewhere
Codecs are like filters but can be used inline in inputs/outputs (decode an incoming JSON log line and turn the objects/hashes/arrays into individual event fields)
(show example logstash.conf in repo)
![Page 8: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/8.jpg)
Getting logs into Logstash
rsyslog: http://www.rsyslog.com/ Logstash forwarder: https://github.com/elasticsearch/logstash-forwarder
(show rsyslog config and Puppet code)
...and configure your Logstash server with inputs
(show inpiut section of example logstash.conf)
• TCP/UDP ports for plain syslog • log4J/JMX inputs to take logs directly out of your JVM app • collectd/ganglia • SNMP traps
Set up your machines/devices to send logs....
![Page 9: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/9.jpg)
Processing logs
(show example rsyslog config in VM)
Done with filters
Grok can do lots of heavy lifting!
(show example logstash.conf in repo)
Basic idea: use filters (grok, mutate, etc.) to pull bits of data out of the raw message and put
different pieces into the event's separate fields
You can add, rename and delete an event's fields
http://grokdebug.herokuapp.com/
![Page 10: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/10.jpg)
Exporting logs from Logstash
Tons of different outputs:
One output is Elasticsearch...
•Email •Plain text files on the local machine Logstash is installed on •Nagios/Icinga •Graphite •fluentd/flume/riemann event stream processors •SaaS monitoring/metrics services like Librato, Boundary, PagerDuty
![Page 11: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/11.jpg)
Elasticsearch
http://www.elasticsearch.org/overview/elasticsearch/
![Page 12: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/12.jpg)
ElasticsearchWritten in Java, based on Apache Lucene
Main competitor for Lucene-based search engines is Solr
Can be used as a JSON-based document database
Has HTTP APIs for sending/receiving data
Has several client libraries for sending/receiving data of different languages (Ruby, Python, Java, PHP, etc.)
![Page 13: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/13.jpg)
ElasticSearch
(show ElasticSearch kopf plugin page)
Really easy to cluster (multicast networking FTW)!
Individual documents are grouped into indexes
Indexes are broken into shards
Shards are shared with other nodes in the cluster
Masters coordinate reads/writes and shard replication
(show ElasticSearch node status JSON page)(show example logstash.conf in repo)
![Page 14: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/14.jpg)
Kibana
http://www.elasticsearch.org/overview/kibana/
![Page 15: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/15.jpg)
KibanaHTML5+JS+CSS dashboard/search tool for ElasticSearch
Really easy to deploy and configure to talk to Elasticsearch!
(show Kibana dashboard in browser)
Browser talks directly to Elasticsearch
(show example Kibana config.js)
Rashid Khan's Monitorama PDX 2014 workshop: http://vimeo.com/95297576
https://github.com/elasticsearch/kibana/blob/master/sample/nginx.conf
https://github.com/elasticsearch/kibana/blob/master/sample/apache_ldap.conf
![Page 16: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/16.jpg)
That's cool and all, but so what?
Centralized system logging for admins is one use case...
Centralized application logs for your devs is a lot more compelling!
![Page 17: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/17.jpg)
Demo repository
What the repo has: •1 Puppet master VM •1 Kibana VM •1 Logstash VM •2 Elasticsearch VMs •2 VMs running rsyslog to generate log dataWhat you'll need: •Vagrant •VirtualBox •An Ubuntu 13.10 base box
https://github.com/nickchappell/logstash_pdxdevops
Further instructions in README
![Page 18: PDX DevOps: Intro to Logstash](https://reader033.vdocument.in/reader033/viewer/2022051210/54c637694a7959e43f8b462e/html5/thumbnails/18.jpg)
Community#logstash on freenode
#elasticsearch on freenode
#kibana on freenode
https://github.com/elasticsearch/
https://github.com/elasticsearch/elasticsearch
https://github.com/elasticsearch/logstash
https://github.com/elasticsearch/kibana