large scale log analytics with solr (from lucene revolution 2015)

OCTOBER 13-16, 2016 • AUSTIN, TX

Large Scale Log Analytics with SolrRafał Kuć and Radu Gheorghe

Sematext Group

3

01About Us

RaduRafał

Logsene

4

02Agenda

Logstash + Solr

rsyslog + Solr

rsyslog + Redis + Logstash + Solr

Solr

5

01Flow in Logstash

/var/log/apache.log

redis

https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png

input

6

01Flow in Logstash

/var/log/apache.log

redis

https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png

plain

{json}

input

codec

7

01Flow in Logstash

/var/log/apache.log

redis

Rafał @kucrafal

grok{

"user": "Rafał","twitter": "@kucrafal"

}

- w $numberOfWorkers

https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png

plain

{json}

input

codec

filter

8

01Flow in Logstash

/var/log/apache.log

redis

Rafał @kucrafal

grok{

"user": "Rafał","twitter": "@kucrafal"

}

- w $numberOfWorkers

https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png

workers => 2

plain

{json}

input

codec

filter

output

9

01Simple Config https://github.com/sematext/lucene-revolution-samples/tree/master/2015

input {

file {

path => "/opt/logs/example.log"

start_position => "beginning"

}

}

output {

solr_http {

solr_url => "http://localhost:8983/solr/gettingstarted"

flush_size => 5000

workers => 4

}

}

bin/plugin install logstash-output-solr_http

apache combined logs

10

01Base Result

11

01Parse JSONinput {

file {

path => "/opt/logs/example.log.parsed"

start_position => "beginning"

…filter {

json {

source => "message"

}

}

output {

solr_http {

…

apache combined logs in JSON

bin/logstash -f logstash.conf -w 4 # filterWorkers=4

12

01JSON Result

input {

file {

path => "/opt/logs/example.log"

start_position => "beginning"

…filter {

grok {

match => [ "message", "%{COMBINEDAPACHELOG}" ]

}

}

output {

solr_http {

…

13

01Grok

14

01Grok Result

15

01Flow Options

https://upload.wikimedia.org/wikipedia/commons/thumb/b/bb/Gorilla-server.svg/2000px-Gorilla-server.svg.pnghttps://www.elastic.co/assets/blt69f6410148efbab8/logstash.png

16

01Flow Options (cont.)

http://www.hanselman.com/blog/content/binary/Windows-Live-Writer/ef572a4c3e50_13F7B/redis_logo_a83f44f3-708d-4fad-aa6e-6eb0d6f82001.pnghttps://upload.wikimedia.org/wikipedia/commons/thumb/f/f8/Question_mark_alternate.svg/2000px-Question_mark_alternate.svg.png

or Kafka or *MQ or...

something light here

rsyslog

rsyslog

rsyslog

17

01Flow in rsyslog

/var/log/apache.log

syslog socket

input

18

01Flow in rsyslog

/var/log/apache.log

syslog socketmain queue (RAM+Disk)

inputqueue.typequeue.size...

19

01Flow in rsyslog

/var/log/apache.log

syslog socketmain queue (RAM+Disk)

inputqueue.typequeue.size...

queue.workerThreads(filter, parse and send events)

20

01Flow in rsyslog

/var/log/apache.log

syslog socketmain queue (RAM+Disk)

inputqueue.typequeue.size...

queue.workerThreads(filter, parse and send events)

queue.dequeueBatchSize

21

01Flow in rsyslog

/var/log/apache.log

syslog socketmain queue (RAM+Disk)

inputqueue.typequeue.size...

queue.workerThreads(filter, parse and send events)

queue.dequeueBatchSize

rsyslog_solr.py

rsyslog_solr.py

rsyslog_solr.py

action

template {JSON}

22

01Flow in rsyslog

/var/log/apache.log

syslog socketmain queue (RAM+Disk)

inputqueue.typequeue.size...

queue.workerThreads(filter, parse and send events)

queue.dequeueBatchSize

rsyslog_solr.py

rsyslog_solr.py

rsyslog_solr.py

action

template {JSON}

23

01Simple Config (1/2) https://github.com/sematext/lucene-revolution-samples/tree/master/2015

module(load="imfile")

module(load="omprog")

input(type="imfile"

File="/opt/logs/example.log"

Tag="apache:")

main_queue(

queue.highWatermark="100000"

queue.lowWatermark="50000"

queue.maxDiskSpace="5g"

queue.fileName="solr_action"

queue.spoolDirectory="/opt/rsyslog/queues"

queue.saveOnShutdown="on"

queue.workerThreads="4"

queue.dequeueBatchSize="500"

)

apache combined logs

24

01Simple Config (2/2)template(name="json_lines" type="list" option.json="on") {

constant(value="{")

constant(value="\"timestamp\":\"")

property(name="timereported" dateFormat="rfc3339")

constant(value="\",\"message\":\"")

property(name="msg")

...

constant(value="\",\"syslog-tag\":\"")

property(name="syslogtag")

constant(value="\"}\n")

}

action(

type="omprog"

binary="/opt/rsyslog/rsyslog_solr.py"

template="json_lines"

)

get from https://github.com/rsyslog/rsyslog/tree/master/plugins/external/solr

25

01Base Result

26

01Base Result

15% rsyslog,4x1% rsyslog_solr.py

27

01Base Result

15% rsyslog,4x1% rsyslog_solr.py

125MB rsyslog, 4x15MB rsyslog_solr.pyDepends on queue. Here up to 100K events in RAM

28

01JSON Config# same main queue settings and modules

input(type="imfile"

File="/opt/logs/example.log.parsed"

Tag="apache:")

module(load="mmnormalize")

action(type="mmnormalize"

rulebase="/opt/rsyslog/json.rb"

)

template(name="json_lines" type="list") {

property(name="$!root") constant(value="\n")

}

action(type="omprog"

...

apache combined logsalready parsed in JSON

version=2

rule=:%root:json%

29

01JSON Result

30

01Normalizing Config

input(type="imfile"

File="/opt/logs/example.log"

Tag="apache")

action(type="mmnormalize"

rulebase="/opt/rsyslog/apache_combined.rb"

)

template(name="json_lines" type="list") {

property(name="$!all-json")

constant(value="\n")

}

version=2

rule=:%[

{"type": "word", "name": "clientip"},

{"type": "literal", "text": " "},

...

{"type": "char-to", "name": "agent", "extradata": "\""},

{"type": "literal", "text": "\""},

{"type": "rest", "name": "blob"}

]%

31

01Normalizing Result

32

01Normalizing “Should Scale”*

sys

tem log

d -ng

performance depends mostly on log length and not on the number of rules:http://blog.gerhards.net/2013/01/performance-of-liblognormrsyslog-parse.html

rule=apache_combined:%[

{"type": "word", "name": "clientip"},

...

{"type": "char-to", "name": "agent", "extradata": "\""},

{"type": "literal", "text": "\""},

{"type": "rest", "name": "blob"}

]%

rule=apache_common:%[

{"type": "word", "name": "clientip"},

...

{"type": "number", "name": "bytes"},

{"type": "rest", "name": "blob", "priority": 65535}

]%

...

33

01Normalizing with Five Rulesinput(type="imfile"

File="/opt/logs/example*"

Tag="apache")

action(type="mmnormalize"

rulebase="/opt/rsyslog/multiple_rules.rb"

)

if $!root <> "" then {

set $.final-json = $!root;

} else {

set $.final-json = $!all-json;

}

template(name="json_lines" type="list") {

property(name="$.final-json") constant(value="\n")

}

34

015 Rules Result

35

01OK, so this works:

rsyslog

rsyslog

rsyslog

36

01How about this:

rsyslog

rsyslog

rsyslog

37

01rsyslog.confmodule(load="imfile")

module(load="omhiredis")

input(type="imfile"

File="/opt/logs/example.log"

Tag="apache:")

template(name="json_lines" type="list" option.json="on") {...}

main_queue(queue.workerthreads="1"

queue.dequeueBatchSize="100"

queue.size="10000")

action(type="omhiredis"

mode="publish"

key="rsyslog_logstash"

template="json_lines")

./configure --enable-omhiredis

small&light queue

38

01logstash.conf

input {

redis {

data_type => "channel"

key => "rsyslog_logstash"

batch_count => 100

}

}

output {

solr_http {

...

}

}

JSON codec is implied

39

01Combined Result

rsyslog 1%

Redis 2%

Logstash 200%

rsyslog 10MB (10K queue)

Redis 1000MB (configurable)

Logstash 380MB

40

015-Rule Normalizing Result

rsyslog 100%

Redis 2%

Logstash 200%

rsyslog 30MB

Redis 1000MB

Logstash 450MB

41

01Shipper conclusions

rsyslog

rsyslog

rsyslog

rsyslog

rsyslog

rsyslog

easy setup; flexibleheavy

light; fastless flexible&easy

offloads buffers and Logstash processing;flexible and efficientsetup and maintenance overhead

42

01Solr Tuning Agenda

Schema and config adjustments

Time-based collections

Tiered cluster (e.g. hot vs cold nodes)

43

01Schema: Two Kinds of Fields

message:failed

"docValues": true"omitNorms": true,

"omitTermFreqAndPositions": true

44

01Schema: Two Kinds of Fields

message:failed

"docValues": true"omitNorms": true,

"omitTermFreqAndPositions": true

+20 to 100% capacity* 10% faster indexing*

* http://blog.sematext.com/2014/11/17/solr-presentations-lucene-solr-revolution/

45

01Commits

"updateHandler.autoSoftCommit.maxTime": 5000

"updateHandler.autoCommit.maxTime": 60000<ramBufferSizeMB>200</ramBufferSizeMB>

5s feels near-realtime while searching

Flush to disk every minute or 200MB

46

01Commits

"updateHandler.autoSoftCommit.maxTime": 5000

"updateHandler.autoCommit.maxTime": 60000<ramBufferSizeMB>200</ramBufferSizeMB>

5s feels near-realtime while searching

Flush to disk every minute of 200MB

+10% capacity; 10% faster indexing*

47

01Time-Based Collections

indexing, merges,most searches

doesn’t change => cache friendly can be optimized

delete without triggering merges

48

01Time-Based Collections

indexing, merges,most searches

doesn’t change => cache friendly=> can be optimized

delete without triggering merges

20-30x capacity; less indexing degradation*

* http://www.slideshare.net/sematext/side-by-side-with-elasticsearch-solr-part-2

49

01Tiered Cluster

hot1

hot2

cold1

cold2

cold3

cold4

50

01Tiered Cluster

hot1

hot2

cold1

cold2

cold3

cold4

51

01Tiered Cluster

hot1

hot2

cold1

cold2

cold3

cold4

ADDREPLICA

52

01Tiered Cluster

hot1

hot2

cold1

cold2

cold3

cold4

53

01Tiered Cluster

hot1

hot2

cold1

cold2

cold3

cold4

54

01Tiered Cluster

hot1

hot2

cold1

cold2

cold3

cold4

55

01Tiered Cluster

hot1

hot2

cold1

cold2

cold3

cold4

quick recent searches and indexing rare lengthy requests

56

01Tiered Cluster

cold1

cold2

cold3

cold4

quick recent searches and indexing rare lengthy requests

hot1

hot2

buffer for indexing spikes

57

01Tiered Cluster

cold1

cold2

cold3

cold4

quick recent searches and indexing rare lengthy requests

hot1

hot2

buffer for indexing spikes

less shards per collectionand the cluster is still balanced

58

01Tiered Cluster

cold1

cold2

cold3

cold4

quick recent searches and indexing rare lengthy requests

hot1

hot2

buffer for indexing spikes

less shards per collectionand the cluster is still balanced

CPU++

RAM++IO++

59

01Wrap-Up

60

01Wrap-Up

DocValues

commits

61

01Wrap-Up

DocValues

commits

https://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png

62

01Wrap-Up

DocValues

commits

https://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png

63

01Wrap-Up

DocValues

commits

http://www.funnyshirts.net/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/z/o/zombies-hate-fast-food-funny-tshirt-preview.pnghttps://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png

64

01Wrap-Up

DocValues

commits

http://www.funnyshirts.net/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/z/o/zombies-hate-fast-food-funny-tshirt-preview.pnghttps://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png

rsyslog

65

01Wrap-Up

DocValues

commits

http://www.funnyshirts.net/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/z/o/zombies-hate-fast-food-funny-tshirt-preview.pnghttps://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png

rsyslog

rsyslog

rsyslog

rsyslog

66

01Questions?

Rafał Kuć@kucrafalrafal.kuc@sematext.com

Radu Gheorghe@radu0gheorgheradu.gheorghe@sematext.com

Sematext@sematexthttp://sematext.com

67

01Questions?

Rafał Kuć@kucrafalrafal.kuc@sematext.com

Radu Gheorghe@radu0gheorgheradu.gheorghe@sematext.com

Sematext@sematexthttp://sematext.com

we’re hiring, too!