elasticsearch, logstash, kibana technical...

Elasticsearch, Logstash, KibanaTechnical Walk-Through

Mark Walkom, Hat Wearer @warkolm

www.elastic.co2

Elasticsearch

www.elastic.co3

Elasticsearch Terminology

•A node is a single Elasticsearch instance, a single JVM

•Multiple nodes can form a cluster

•A cluster can manage multiple indices

•A cluster is agile & self managing

•Clusters often 3-10 nodes but can scale to 100s of nodes

•Clusters can have Petabytes of data

•Clusters can be federated for larger scale

www.elastic.co4

an open source, distributed, scalable,

highly available, document-oriented, RESTful

full text search engine

with real-time search and analytics capabilities

built on lucene and java

Elasticsearch is...

www.elastic.co5

an open source, distributed, scalable, highly available, document-oriented,

RESTful, full text search engine with real-time search and analytics capabilities

Elasticsearch is...

Apache 2.0 License  

https://www.apache.org/licenses/LICENSE-2.0

www.elastic.co6

an open source, distributed, scalable, highly available, document-oriented,

RESTful, full text search engine with real-time search and analytics capabilities

Elasticsearch is...

www.elastic.co7

an open source, distributed, scalable, highly available, document-oriented,

RESTful, full text search engine with real-time search and analytics capabilities

Elasticsearch is...

www.elastic.co8

an open source, distributed, scalable, highly available, document-oriented,

RESTful, full text search engine with real-time search and analytics capabilities

Elasticsearch is...

www.elastic.co9

an open source, distributed, scalable, highly available, document-oriented,

RESTful, full text search engine with real-time search and analytics capabilities

Elasticsearch is...

Source:  http://json.org/

www.elastic.co10

an open source, distributed, scalable, highly available, document-oriented,

RESTful, full text search engine with real-time search and analytics capabilities

Elasticsearch is...

Source:  https://httpwg.github.io/asset/http.svg

www.elastic.co11

an open source, distributed, scalable, highly available, document-oriented,

RESTful, full text search engine with real-time search and analytics capabilities

Elasticsearch is...

www.elastic.co12

an open source, distributed, scalable, highly available, document-oriented,

RESTful, full text search engine with real-time search and analytics capabilities

Elasticsearch is...

www.elastic.co13

Search

Search with Elasticsearch

www.elastic.co14

CRUD

www.elastic.co15

CRUD

www.elastic.co16

CRUD

www.elastic.co17

CRUD

www.elastic.co18

Searching

www.elastic.co19

Searching

www.elastic.co20

Aggregation

Analytics with Elasticsearch

www.elastic.co21

Aggregations

GET /person/person/_search?search_type=count{  

"aggs": {  

"by_country": {  

"terms": {  

"field": "address.country"  

}  

}  

}  

}{ ..., "aggregations" : {   "by_country" : {   "buckets" : [ {   "key" : "England",   "doc_count" : 30051   }, {   "key" : "Germany",   "doc_count" : 30004   }, {   "key" : "France",   "doc_count" : 15034   }, {   "key" : "Spain",   "doc_count" : 14912   } ]}}}

17%

17%

33%

33%

EnglandGermanyFranceSpain

www.elastic.co22

Histograms

GET /person/person/_search?search_type=count{  

"aggs": {  

"by_date": {  

"date_histogram": {  

"field": "dateOfBirth",  

"interval": "year",  

"format": "yyyy"  

}  

}  

}  

}

{ ..., "aggregations": {  

"by_date": {  

"buckets": [  

{  

"key_as_string": "1960",  

"key": -946080000000,  

"doc_count": 39  

},  

{  

"key_as_string": "1961",  

"key": -630720000000,  

"doc_count": 12677  

},  

{  

"key_as_string": "1962",  

"key": -315360000000,  

"doc_count": 12936  

}, ...  

]  

}  

}}0

7500

15000

22500

30000

1940 1950 1960 1970 1980 1990 2000 2010

www.elastic.co23

A Lot More

www.elastic.co24

More than search

Elasticsearch

www.elastic.co25

Text Analysis - Analyzers

• Tokenizer

Breaks the text into tokens and produces a token stream Example: keyword, whitespace, regex, etc...

• Token Filter

Acts on the token stream - can drop and modify existing tokens, or add new ones. Example: lowercase, stopword, ngram, etc..

www.elastic.co26

Free steak knives!

• Relational documents Parent/child Nesting

• Suggestion API Predictive typing/search

• Highlighting Emphasise results, e.g. <em>w00t</em>

• Percolators - search for searches Does this document match this search?

www.elastic.co27

Geo Search

• Geo points and shapes Polygon Polygon with holes Multi polygon

• Bounding boxes, distance from point, distance in a range

• Supports multiple coordinate formats; “location”: { "lat" : 41.12, "lon" : -71.34 } "location" : “41.12,-71.34” "location" : [-71.34, 41.12]

www.elastic.co28

Elasticsearch & Hadoop

www.elastic.co29

Elasticsearch for Apache Hadoop™

www.elastic.co30

Logstash

www.elastic.co31

Logstash

Logstash

Input OutputFilter

? ?

collect and split alter and enrich store and visualise

www.elastic.co32

Logstash

www.elastic.co33

Logstash

71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1" 301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"

www.elastic.co34

Logstash110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 123.125.71.35 - - [16/Feb/2014:09:49:02 -0500] "GET /blog/tags/release HTTP/1.1" 200 40693 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 50.150.204.184 - - [16/Feb/2014:09:49:37 -0500] "GET /images/googledotcom.png HTTP/1.1" 200 65748 "http://www.google.com/search?q=https//:google.com&source=lnms&tbm=isch&sa=X&ei=4-r8UvDrKZOgkQe7x4CICw&ved=0CAkQ_AUoAA&biw=320&bih=441" "Mozilla/5.0 (Linux; U; Android 4.0.4; en-us; LG-MS770 Build/IMM76I) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30" 207.241.237.225 - - [16/Feb/2014:09:50:06 -0500] "GET /blog/tags/examples HTTP/1.0" 200 9208 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 200.49.190.101 - - [16/Feb/2014:09:50:10 -0500] "GET /reset.css HTTP/1.1" 200 1015 "-" "-" 200.49.190.100 - - [16/Feb/2014:09:50:08 -0500] "GET /blog/tags/web HTTP/1.1" 200 44019 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1" 200.49.190.101 - - [16/Feb/2014:09:50:12 -0500] "GET /style2.css HTTP/1.1" 200 4877 "-" "-" 200.49.190.101 - - [16/Feb/2014:09:50:19 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1" 66.249.73.185 - - [16/Feb/2014:09:51:19 -0500] "GET /reset.css HTTP/1.1" 200 1015 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:51:26 -0500] "GET /blog/tags/munin HTTP/1.1" 200 9746 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:51:47 -0500] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:52:34 -0500] "GET /blog/geekery/eventdb-ideas.html HTTP/1.1" 200 11418 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 67.214.178.190 - - [16/Feb/2014:09:53:19 -0500] "GET / HTTP/1.0" 200 37932 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0" 67.214.178.190 - - [16/Feb/2014:09:53:30 -0500] "GET /blog/geekery/installing-windows-8-consumer-preview.html HTTP/1.0" 200 8948 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0" 207.241.237.220 - - [16/Feb/2014:09:53:47 -0500] "GET /blog/tags/projects HTTP/1.0" 200 28370 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 46.105.14.53 - - [16/Feb/2014:09:53:48 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/" 207.241.237.227 - - [16/Feb/2014:09:53:50 -0500] "GET /blog/geekery/soekris-gpio.html HTTP/1.0" 200 9587 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "http://en.wikipedia.org/wiki/Xvfb" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:35 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Win64; x64; Trident/6.0)" 66.249.73.185 - - [16/Feb/2014:09:54:44 -0500] "GET /doc/index.html?org/elasticsearch/action/search/SearchResponse.html HTTP/1.1" 404 294 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 207.241.237.228 - - [16/Feb/2014:09:54:54 -0500] "GET /blog/tags/defcon HTTP/1.0" 200 24142 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 207.241.237.101 - - [16/Feb/2014:09:54:58 -0500] "GET /blog/tags/regex HTTP/1.0" 200 14888 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 87.169.99.232 - - [16/Feb/2014:09:56:12 -0500] "GET /presentations/puppet-at-loggly/puppet-at-loggly.pdf.html HTTP/1.1" 200 24747 "https://www.google.de/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 209.85.238.199 - - [16/Feb/2014:09:56:18 -0500] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 3 subscribers; feed-id=14171215010336145331)" 209.85.238.199 - - [16/Feb/2014:09:56:31 -0500] "GET /test.xml HTTP/1.1" 200 1370 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=11390274670024826467)" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /blog/geekery/ssl-latency.html HTTP/1.1" 200 17147 "http://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CE4QFjAE&url=http%3A%2F%2Fwww.semicomplete.com%2Fblog%2Fgeekery%2Fssl-latency.html&ei=ZdEAU9mGGuWX1AW09IDoBw&usg=AFQjCNHw6zioJpizqX8Q0YpKKaF4zdCSEg&bvm=bv.61535280,d.d2k" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:29 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 66.249.73.135 - - [16/Feb/2014:09:57:36 -0500] "GET /blog/geekery/vmware-cpu-performance.html HTTP/1.1" 200 12908 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 46.105.14.53 - - [16/Feb/2014:09:58:48 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/" 218.30.103.62 - - [16/Feb/2014:09:59:36 -0500] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:09:59:41 -0500] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:09:59:46 -0500] "GET /projects/fex/ HTTP/1.1" 200 14352 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 74.125.40.20 - - [16/Feb/2014:09:59:53 -0500] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "FeedBurner/1.0 (http://www.FeedBurner.com)" 71.212.224.97 - - [16/Feb/2014:10:00:05 -0500] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://suckless.org/rocks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:05 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 218.30.103.62 - - [16/Feb/2014:10:00:07 -0500] "GET /projects/xdotool/xdotool.xhtml HTTP/1.1" 304 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 108.174.55.234 - - [16/Feb/2014:10:00:16 -0500] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "-" 218.30.103.62 - - [16/Feb/2014:10:00:28 -0500] "GET /blog/geekery/c-vs-python-bdb.html HTTP/1.1" 200 11388 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 121.107.188.202 - - [16/Feb/2014:10:00:28 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 218.30.103.62 - - [16/Feb/2014:10:00:52 -0500] "GET /blog/productivity/better-zsh-xterm-title-fix.html HTTP/1.1" 200 10185 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:01:14 -0500] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:01:37 -0500] "GET /blog/geekery/puppet-facts-into-mcollective.html HTTP/1.1" 200 9872 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 198.46.149.143 - - [16/Feb/2014:10:01:44 -0500] "GET /blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 9316 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 198.46.149.143 - - [16/Feb/2014:10:01:44 -0500] "GET /blog/geekery/solving-good-or-bad-problems.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 10756 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 218.30.103.62 - - [16/Feb/2014:10:01:57 -0500] "GET /blog/geekery/jquery-interface-puffer.html%20target= HTTP/1.1" 200 202 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:02:19 -0500] "GET /blog/geekery/ec2-reserved-vs-ondemand.html HTTP/1.1" 200 11834 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 66.249.73.135 - - [16/Feb/2014:10:02:37 -0500] "GET /blog/web/firefox-scrolling-fix.html HTTP/1.1" 200 8956 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://www.haskell.org/haskellwiki/Xmonad/Frequently_asked_questions" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0” 66.249.73.135 - - [16/Feb/2014:10:03:25 -0500] "GET /blog/tags/bdb HTTP/1.1" 200 23099 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" 107.170.41.69 - - [16/Feb/2014:10:03:31 -0500] "GET /?flav=atom HTTP/1.1" 200 32352 "-" "Feedbin - 1 subscribers" 50.16.19.13 - - [16/Feb/2014:10:03:43 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "http://www.semicomplete.com/blog/tags/puppet?flav=rss20" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 46.105.14.53 - - [16/Feb/2014:10:03:50 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/"

www.elastic.co35

Logstash

input { stdin { } }

filter { grok { match => { "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}' } }

date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] locale => en }

geoip { source => "clientip" }

useragent { source => "agent" target => "useragent" }}

output { stdout { codec => rubydebug } }

www.elastic.co36

Logstash{ "message" => "71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] \"GET /admin HTTP/1.1\" 301 566 \"-\" \"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\"", "@version" => "1", "@timestamp" => "2011-05-18T08:48:10.000Z", "host" => "bender.local", "clientip" => "71.141.244.242", "ident" => "-", "auth" => "kurt", "timestamp" => "18/May/2011:01:48:10 -0700", "verb" => "GET", "request" => "/admin", "httpversion" => "1.1", "response" => 301, "bytes" => 566, "referrer" => "\"-\"", "agent" => "\"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\"", "geoip" => { "ip" => "71.141.244.242", "country_code2" => “US", "city_name" => "San Francisco", "timezone" => "America/Los_Angeles", "location" => [ [0] -122.4194, [1] 37.7749 ] }, "useragent" => { "name" => "Firefox", "os" => "Windows XP", "os_name" => "Windows XP", "device" => "Other", "major" => "3", "minor" => "6", "patch" => "3" }}

www.elastic.co37

Logstash

input { stdin {} }

filter { grok { match => [ message, "%{COMBINEDAPACHELOG}" ] }}

output { elasticsearch { protocol => “http”

host => “bender” }}

www.elastic.co38

Kibana

www.elastic.co39

Kibana

•Kibana 4 is a total re-architecture from 3

Nodejs + javascript

Zazzier UI

Single binary that serves itself

•Lots more functionality via aggregations

•Extensible - plugins coming real soon

www.elastic.co40

Kibana

www.elastic.co41

Kibana

www.elastic.co42

Kibana

www.elastic.co43

Found - ESaaS

•Fully Managed and Monitored Infrastructure

Automated Backups

HA - Replication and Failover

•GUI Driven, User Friendly*

•Sydney AZ very, very soon

www.elastic.co44

Elastic: Commercial Plugins

•Marvel: Monitor your Cluster

Currently KB3 based front end. v2.0 will be KB4.

•Shield: For Security

ACLs, RBAC via AD or LDAP, SSL, IP filtering, Auditing

•Watcher: Alerting on your data

Email and webhook push notifications

•More coming soon!

www.elastic.co45

Goodies

•Curator: index management https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

•Puppet & Chef modules https://forge.puppetlabs.com/elasticsearch https://github.com/elastic/cookbook-elasticsearch/

• logstash forwarder: low overhead collector https://github.com/elastic/logstash-forwarder

•grokdebugger: log pattern matching http://grokdebug.herokuapp.com/

www.elastic.co46

More Goodies

•Github: https://github.com/elastic

•Docs: http://www.elastic.co/guide/

•Forums: https://discuss.elastic.co

• IRC channels #elasticsearch, #logstash, #kibana, #beats on Freenode

•We’re hiring! jobs@elastic.co, drop me an email/DM or come say Hi :)

Thanks!

Mark Walkom, Hat Wearer @warkolm