![Page 1: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/1.jpg)
S
Finding the needle in the haystack with ELK
Elasticsearch for Incident Handlers and Forensic Analysts
![Page 2: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/2.jpg)
Whoami
S Working for the Belgian Government my own company S Incident Handling S Malware analysis
S Forensics (network + system)
S Open Source minded
S Creator of MISP – Malware Information Sharing Platform
S Creator of pystemon – pastebin monitoring tool
S Core organizer of the FOSDEM conference for many years
S Contact me: [email protected]
![Page 3: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/3.jpg)
S
Finding the needle in the haystack with ELK
Elasticsearch for Incident Handlers and Forensic Analysts
![Page 4: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/4.jpg)
image by James Lumb
![Page 5: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/5.jpg)
What tools do you use?
S Text logs
S notepad
S Grep
S awk / sed / cut
S MS Excel / OOo Calc
![Page 6: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/6.jpg)
image by velorichard.wordpress.com
![Page 7: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/7.jpg)
Optimizing
S grep -F log.txt
S zgrep -F log.txt
S zgrep -f patterns.txt -F log.txt
S find "$LOGS_DIR" -iname "*.gz" -print0 | parallel --gnu -0 -n1 -P8 zgrep -f patterns.txt –F > result-all.txt
S Fast for single search, however no column lookup !
![Page 8: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/8.jpg)
Optimizing
S MySQL / MS Access
S Splunk S free = 500MB/day
S ELSA – Enterprise Log Search and Archive S Limitation of the # of columns
S ${COMMERCIAL_TOOL}
![Page 9: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/9.jpg)
Trick for Splunk Addicts
S Limit is 500 MB /day
S 3 license violations allowed per month
S Set the date to 00:01 AM
S Index as much as possible 24h/day for 3 days (while loops are your friend)
S Enjoy searching
![Page 10: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/10.jpg)
logstash kibana
Trick for all = ELK
S Elasticsearch Logstash Kibana
S Index as much as you want
S No limit on volume, speed or position of the moon
S Open Source, Free to use, commercial support
![Page 11: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/11.jpg)
Configurations
S https://github.com/cvandeplas/ELK-forensics
S Repository with Logstash and Kibana configurations
S Mactime, BlueCoat, Mail IMSS, IWSVA, IIS, SuperTimeline, Plaso, …
S http://christophe.vandeplas.com/2014/06/setting-up-single-node-elk-in-20-minutes.html
S Our focus today: S Forensics and Incident Handling
S Batch-Import
![Page 12: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/12.jpg)
![Page 13: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/13.jpg)
![Page 14: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/14.jpg)
![Page 15: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/15.jpg)
![Page 16: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/16.jpg)
![Page 17: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/17.jpg)
![Page 18: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/18.jpg)
S
How does it work?
![Page 19: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/19.jpg)
logstash kibana
Trick for all = ELK
S Elasticsearch Logstash Kibana
S Index as much as you want
S No limit on volume, speed or position-of-the-moon-licensing
S Open Source, Free to use, commercial support
![Page 20: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/20.jpg)
Inputs
S Inputs & codecs S collectd, drupal_dblog, elasticsearch, eventlog, exec, file,
ganglia, gelf, gemfire, generator, graphite, heroku, imap, invalid_input, irc, jmx, log4j, lumberjack, pipe, puppet_facter, rabbitmq, rackspace, redis, relp, s3, snmptrap, sqlite, sqs, stdin, stomp, syslog, tcp, twitter, udp, unix, varnishlog, websocket, wmi, xmpp, zenoss, zeromq
S cloudtrail, collectd, compress_spooler, dots, edn, edn_lines, fluent, graphite, json, json_lines, json_spooler, line, msgpack, multiline, netflow, noop, oldlogstashjson, plain, rubydebug, spool
S Outputs
S Filters
![Page 21: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/21.jpg)
Input Example
S I usually don’t use “file” as input
S Keeps a reference to the position in the file
S TCP socket is the easiest for me
S ncat log01.lab.internal 18001 < logfile.log!
![Page 22: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/22.jpg)
Outputs
S Inputs & codecs
S Outputs S boundary, circonus, cloudwatch, csv, datadog,
datadog_metrics, elasticsearch, elasticsearch_http, elasticsearch_river, email, exec, file, ganglia, gelf, gemfire, google_bigquery, google_cloud_storage, graphite, graphtastic, hipchat, http, irc, jira, juggernaut, librato, loggly, lumberjack, metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty, pipe, rabbitmq, rackspace, redis, redmine, riak, riemann, s3, sns, solr_http, sqs, statsd, stdout, stomp, syslog, tcp, udp, websocket, xmpp, zabbix, zeromq
S Filters
![Page 23: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/23.jpg)
Output Example
![Page 24: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/24.jpg)
Filters
S Inputs & codecs
S Outputs
S Filters S advisor, alter, anonymize, checksum, cidr, cipher, clone,
collate, csv, date, dns, drop, elapsed, elasticsearch, environment, extractnumbers, fingerprint, gelfify, geoip, grep, grok, grokdiscovery, i18n, json, json_encode, kv, metaevent, metrics, multiline, mutate, noop, prune, punct, railsparallelrequest, range, ruby, sleep, split, sumnumbers, syslog_pri, throttle, translate, unique, urldecode, useragent, uuid, wms, wmts, xml, zeromq
![Page 25: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/25.jpg)
Filter Example
![Page 26: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/26.jpg)
Filter Example
![Page 27: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/27.jpg)
Grok
S Named regular expressions to match patterns/extract data.
S Logstash ships with lots of patterns ! https://github.com/elasticsearch/logstash/tree/master/patterns
S Test app: http://grokdebug.herokuapp.com
![Page 28: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/28.jpg)
Testing complex Groks
![Page 29: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/29.jpg)
Data Enrichment with Filters
S Extract fields: csv, grok, kv!
S Extract date!
S Modify using mutate!
S Enrich with S Geoip
S User-agent
S Urldecode
S Translate
S …
![Page 30: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/30.jpg)
Geoip
![Page 31: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/31.jpg)
Geoip
![Page 32: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/32.jpg)
User-Agent
![Page 33: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/33.jpg)
User-Agent
![Page 34: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/34.jpg)
Translate
![Page 35: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/35.jpg)
Translate
![Page 36: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/36.jpg)
Ruby as last resort
* There might be a better way to do this, but ruby and I are not really friends yet
![Page 37: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/37.jpg)
Data Enrichment with Filters
S Extract fields: csv, grok, kv!
S Extract date!
S Modify using mutate!
S Enrich with S Geoip
S User-agent
S Urldecode
S Translate
S …
![Page 38: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/38.jpg)
logstash kibana
Trick for all = ELK
S Elasticsearch Logstash Kibana
S Index as much as you want
S No limit on volume, speed or season-licensing
S Open Source, Free to use, commercial support
![Page 39: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/39.jpg)
Elasticsearch
S Wikipedia: Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License.
S Very very fast
S Adding an node = easier than extremely easy
![Page 40: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/40.jpg)
Elasticsearch
S Be cautious
S No security by default
S Auto-discovery, auto-distribution if other node is present
S Elastic HQ plugin S cd /usr/share/elasticsearch/bin!S ./plugin -install royrusso/elasticsearch-HQ!
![Page 41: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/41.jpg)
logstash kibana
Trick for all = ELK
S Elasticsearch Logstash Kibana
S Index as much as you want
S No limit on volume, speed or horoscope-licensing
S Open Source, Free to use, commercial support
![Page 42: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/42.jpg)
Kibana
S Fancy GUI
S Extremely easy to build up a dashboard
S Gives good overview over data
S Powerful, but limited in capability
S For more: write a python script or use REST API
![Page 43: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/43.jpg)
DO NOT PRESS
THIS BUTTON
![Page 44: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/44.jpg)
Search syntax
S Apache Lucene Search syntax
S title:foo title:"foo bar”
S title:"foo bar” AND body:"quick fox”
S (title:"foo bar" AND body:"quick fox") OR title:fox
S title:foo -title:bar
S title:foo*bar
S time_taken:[10000 TO 999999999]
http://www.lucenetutorial.com/lucene-query-syntax.html
![Page 45: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/45.jpg)
Load dashboards
![Page 46: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/46.jpg)
Filter
![Page 47: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/47.jpg)
![Page 48: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/48.jpg)
S
Performance
![Page 49: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/49.jpg)
Performance goals
S Focus Incident Handling and Forensics
S Max speed of indexing
S Max speed of searching
S During indexation search may be slow
S No need for redundancy
S So don’t use this advice for operations-live-production
![Page 50: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/50.jpg)
Performance Logstash
S Memory setting: (/etc/default/elasticsearch) S LS_HEAP_SIZE="500m"!
S Command line flag: S -w or –filterworkers AMOUNT_OF_CORES (default: 1)!
S Each extra filter slows it down S Grok aka regex = slow
S Prefer csv, kv
S Use the least possible wildcards (* or +)!
S Geoip = slow but very practical
S User-agent = slow, often practical
![Page 51: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/51.jpg)
Performance Elasticsearch
S Memory setting (/etc/default/elasticsearch) S ES_HEAP_SIZE=12g => set to half of RAM (max 32 GB)
S Disable redundancy (/etc/elasticsearch/elasticsearch.yml)
S index.number_of_replicas: 0!
S Shards for number of nodes (/etc/elasticsearch/elasticsearch.yml) S index.number_of_shards: 1
S Increase memory buffer for search S indices.memory.index_buffer_size: 50%!
![Page 52: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/52.jpg)
Perf. Elasticsearch Indexes
S Open Index = memory usage + disk usage Closed Index = disk usage, so close index when not needed
S Per case new indexes Similar logs in the same index, but use a field “host” to differentiate investigations S system timelines: logstash-%{[case]}-%{[type]}
S mail logs: logstash-%{[case]}-%{[type]}-%{+YYYY.MM}
S proxy logs: logstash-%{[case]}-%{[type]}-%{+YYYY.MM.dd}
S curl -XPOST 'localhost:9200/logstash-${case}*/_close' curl -XPOST 'localhost:9200/logstash-${case}*/_open'!
![Page 53: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/53.jpg)
Performance Kibana
S Each block/graph is extra search
S So 10 graphs equals 10 simultaneous searches
1. First select small date/time window
2. Test your search on small data set
3. Add filters
4. Zoom out on date/time
5. Dig deeper
![Page 54: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/54.jpg)
Keep in mind
S Logstash is (relatively) SLOW
S Finished? Close the index, do NOT delete it
S Or save JSON to files (output plugin Logstash), re-index them later
S Node++ = Speed++
![Page 55: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/55.jpg)
S
Forensic analysis
![Page 56: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/56.jpg)
Plaso
S Plaso = the new log2timeline and more
S log2timeline.py win7-64-nfury-10.3.58.6.dump /path/to/disk/image
S psort.py -o elastic win7-64-nfury-10.3.58.6.dump
![Page 57: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/57.jpg)
ELK-forensics
S https://github.com/cvandeplas/ELK-forensics
S Logstash configs
S Kibana dashboards
S Mactime, Log2timeline csv, BlueCoat, Mail IMSS, IWSVA, IIS
S More to come
![Page 58: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/58.jpg)
Other interesting projects using Elasticsearch
S Moloch – Open Source large scale IPv4 full PCAP capturing, indexing and database system. https://github.com/aol/moloch
S Mozdef – PoC – automate IH process and facilitate real-time activities - https://github.com/jeffbryner/MozDef
S Suricata – Exports data in EVE format (JSON). Great to visualize malware activity from sandbox
![Page 59: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/59.jpg)
![Page 60: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/60.jpg)
![Page 61: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/61.jpg)
![Page 62: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/62.jpg)
![Page 63: Finding the needle in the haystack with ELK - SANS · PDF fileFinding the needle in the haystack with ELK ... metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty,](https://reader030.vdocument.in/reader030/viewer/2022021423/5a9f97717f8b9a7f178d0363/html5/thumbnails/63.jpg)
S
Places to be? • https://github.com/cvandeplas/ELK-forensics • http://www.elasticsearch.org/overview/elkdownloads/ • http://logstash.net/ • https://groups.google.com/forum/#!forum/logstash-users