real-time data analysis using elk

REAL TIME DATA ANALYSIS USING ELK

@jettroCoenradie

Jettro Coenradie

http://amsterdam.luminis.eu

REAL TIME DATA ANALYSISUSING ELK

Real time log analysis

Introduction of ELK components

Who is abusing my blog?

Lessons learned from IDMC project

Good to know

REAL TIME LOG ANALYSIS

indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET CLR 4.0.30319)"46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"61.135.219.2 - - [21/Apr/2015:03:53:28 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;http://www.youdao.com/help/reader/faq/topic006/;1 subscribers;)"46.4.132.226 - - [21/Apr/2015:03:53:33 +0200] "GET /2008/09/23/when-good-guys-start-looking-like-bullies/ HTTP/1.0" 200 54917 "http://www.gridshore.nl/2008/09/27/does-professionalization-kill-open-source/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://

http://serverfault.com/questions/11028/do-you-have-any-useful-awk-and-grep-scripts-for-parsing-apache-logs


# tail -fn 100 access-log-2014-04-22



# tail -fn 100 access-log-2014-04-22

awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ } END { for (i in ipcount) { printf "%15s - %d\n", i, ipcount[i] } }' access-log-2015-04-21


EVERY NIGHT A BATCH USING WEBALIZER

GOOGLE ANALYTICS

WHAT IS REAL TIME?

THERE IS ALWAYS A DELAY

HOW MUCH DELAY CAN YOU ACCEPT?

ARCHITECTURE OF DELAY

accesslogs shipper Queue Logstash

elasticsearch

Monitor Send Retrieve

Storeforwarderlogstashbeaver

RedisKafka

DATA LIFECYCLE

DATA LIFECYCLE

Obtain

DATA LIFECYCLE

Obtain Transform

DATA LIFECYCLE

Obtain Transform Store

DATA LIFECYCLE

Obtain Transform Store Use

DATA LIFECYCLE


Learn

DATA LIFECYCLE: ELK


Learn

DATA LIFECYCLE: ELK


Learn

Logstash

DATA LIFECYCLE: ELK


Learn

Logstash Logstash

DATA LIFECYCLE: ELK


Learn

Logstash Logstash Elasticsearch

DATA LIFECYCLE: ELK


Learn

Logstash Logstash Elasticsearch Kibana

DATA LIFECYCLE: ELK


Learn

Logstash Logstash Elasticsearch Kibana

YOU

INTRODUCTION OF ELK COMPONENTS

LOGSTASH: COMPONENTSfile

syslog

redis log4j

websockettwitter grok

mutate

drop

clone

geoipelasticsearch

file

graphite

statsd

LOGSTASH: COMPONENTSfile

syslog

redis log4j

Input

websockettwitter

Filter

Output

grok

mutate

drop

clone

geoipelasticsearch

file

graphite

statsd

LOGSTASH: COMPONENTSfile syslog redislog4jInput web

sockettwitter

Filter

Output

grok mutatedropclone geoip

elasticsearch filegraphite statsd

ELASTICSEARCH

ELASTICSEARCH

cluster

ELASTICSEARCH

cluster

Node Node Node

ELASTICSEARCH

cluster

Node Node NodeIn

dex

Inde

xIn

dex

Inde

xIn

dex

Inde

x

ELASTICSEARCH

cluster

Node Node NodeIn

dex

Inde

xIn

dex

Inde

xIn

dex

Inde

x

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

ELASTICSEARCH

cluster

Node Node NodeIn

dex

Inde

xIn

dex

Inde

xIn

dex

Inde

x

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

Mapping

ELASTICSEARCH

cluster

Node Node NodeIn

dex

Inde

xIn

dex

Inde

xIn

dex

Inde

x

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

Mapping

Search API

ELASTICSEARCH

cluster

Node Node NodeIn

dex

Inde

xIn

dex

Inde

xIn

dex

Inde

x

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

shar

dsh

ard

Mapping

Search API

Aggregations

AGGREGATIONS

AGGREGATIONS

27.159.213.10 - - [25/Feb/2015:03:35:57 +0100] "GET /2008/04/29/using-ehcache-and-verifying-that-it-works-with-jpa-and-springframework/ HTTP/1.1" 200 18720 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.0.3705)"

AGGREGATIONS


GET

AGGREGATIONS


AGGREGATIONS

78.5.169.90 - - [15/Apr/2015:03:08:17 +0200] "POST /wp-login.php HTTP/1.1" 403 3538 "http://www.gridshore.nl/wp-login.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML like Gecko) Chrome/24.0.1312.56 Safari/537.17"

AGGREGATIONS


POST

AGGREGATIONS


AGGREGATIONS

175989 133343 2008 2POSTGET HEAD PUT

AGGREGATIONS

Date histogram

Feb Mar Apr

311344 395654 157623

AGGREGATIONS

Date histogram

Feb Mar Apr

311344 395654 157623

Cardinality [client ip] 11848 26152 9064

GET /gridshore-logs-*/_search?search_type=count{ "aggs": { "byDate": { "date_histogram": { "field": "@timestamp", "interval": "month" }, "aggs": { "uniqueVisitors": { "cardinality": { "field": "clientip" } } } } }}

KIBANA

Discover

KIBANA

Discover

Visualise

KIBANA

Discover

Visualise

Analyse

Discover

Visualise

Analyse

WHO IS ABUSING MY BLOG?

OBTAINING LOGS

daily rolling file

OBTAINING LOGS

daily rolling file

shell script ftp

OBTAINING LOGS

daily rolling file

shell script ftp

logstash

OBTAINING LOGS

daily rolling file

shell script ftp

logstash

elasticsearch

1. input { 2. file { 3. path => "/access-log-*" 4. type => "apache" 5. start_position => "beginning" 6. } 7. }

OBTAIN


OBTAIN

files to import


OBTAIN

used for filtering


OBTAIN

start reading from


71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"

1. filter { 2. grok { 3. match => { "message" => "%{COMBINEDAPACHELOG}" } 4. remove_field => ["message"] 5. } 6. grok { 7. match => { "request" => "%{URIPATH:request_noparam}"} 8. } 9. geoip { 10. source => "clientip" 11. } 12. useragent { 13. source => "agent" 14. target => "useragent" 15. remove_field => ["agent"] 16. } 17. date { 18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] 19. } 20.}

TRANSFORM


remove parsed message

TRANSFORM


TRANSFORM

extra parse of request


TRANSFORM

request => /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3

request_noparam => /wp-content/plugins/scripts-gzip/gzip.php


TRANSFORM

add geo information


TRANSFORM

parse useragent fields


TRANSFORM

agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36

useragent => {"name": "Safari", "os": “Mac OS X 10.10.2”, "os_name": “Mac OS X”, "device": "Other", "major": "537", "minor": "36" }


TRANSFORM

take timestamp from log

STORE1. output {2. if "_grokparsefailure" not in [tags] {3. elasticsearch {4. protocol => "transport"5. host => "localhost:9300"6. cluster => "jc-play"7. index => "gridshore-logs-%{+YYYY.MM}"8. manage_template => false9. template_name => "gridshore-logs"10. }11. }12.}


in case of an error


use faster binary protocol


format of index to create:gridshore-logs-2015.02


provide our own index template

Integrated Disease Management Control

LESSONS LEARNED

DATA ENHANCEMENT

PROBLEM WITH DATES

WHAT CANNOT BE DONE

THINGS ABOUT AGE

GOOD TO KNOW

GETTING BIG

SMAP - Soil Moisture Active Passive

http://smap.jpl.nasa.gov/mission/why-it-matters/

MonitorDrought

PredictFloods

AssistCrop

ProductivityWeatherForecasting

VERIZON

https://speakerdeck.com/bhaskarvk/elastic-on-15-500-billion-documents-and-counting

“We offer technology products and solutions that transform the way our customers connect, collaborate and innovate”

VERIZON

https://speakerdeck.com/bhaskarvk/elastic-on-15-500-billion-documents-and-counting

“We offer technology products and solutions that transform the way our customers connect, collaborate and innovate”

Store massive logging data

Store in high rate

Query in acceptable rate

VERIZON

128 Nodes

8 cores - 64 Gb RAM - 6 x 1TB disk

10+ Bilion documents a day

Over 500 Billion documents total

SAVING YOUR DASHBOARDS

WHAT ABOUT SECURITY

Elastic shield

FUTURE DIRECTIONS

LOGSTASH

LOGSTASH

• API for pipeline

LOGSTASH


• Internal / persistent queues

LOGSTASH


• Internal / persistent queues

• Clustered logstash

ELASTICSEARCH

ELASTICSEARCH

• Better error responses

ELASTICSEARCH


• Reindex API

ELASTICSEARCH


• Reindex API

• Changes API

KIBANA

KIBANA

• Formatting output: numbers, currency, urls, video

KIBANA


• Edit and save or pin filters

KIBANA



• Choose your own colours in charts

KIBANA



• Choose your own colours in charts

• Create API for custom plugins

SUMMARISE

SUMMARISE

• Real time data analysis

SUMMARISE


• Obtain and transform data using logstash

SUMMARISE



• Index data in elasticsearch

SUMMARISE




• Show data using Kibana

SUMMARISE




• Show data using Kibana

• What Kibana does well and what not

MORE INFORMATION

@jettroCoenradie

[email protected]://amsterdam.luminis.eu/news/

https://www.elastic.co/products

mailto:[email protected]

real-time data analysis using elk

Technology