elasticsearch logstash kibana meetup
Post on 06-Aug-2015
291 Views
Preview:
TRANSCRIPT
Who Am I
••••
•
•
•
•
•
••••
••••••••
•••••
•••
•••••••
RAM, CPU type/cores, DISK, Networks matter a lot, But the cluster design, data structures of documents and queries has a huge impact on the Elasticsearch Clusters and your search experiences.
I have experienced it a lot. You can check out some of the scenarios here : https://www.found.no/foundation/crash-elasticsearch/
•
•
••
•
•
•
•
•
••
•••••
•••
••••
•••
•••
••
••••
•••
•••
Centralize
Analyze Monitor
Share
• •
•
•••
•••
#!/bin/bash
sudo apt-get purge openjdk-\*
wget --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz
tar -xvf jdk-7u79-linux-x64.tar.gz
sudo mkdir -p /usr/lib/jvm
sudo mv ./jdk1.7.0_79 /usr/lib/jvm/
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.7.0_79/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.7.0_79/bin/javac" 1
sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.7.0_79/bin/javaws" 1
sudo chmod a+x /usr/bin/java
sudo chmod a+x /usr/bin/javac
sudo chmod a+x /usr/bin/javaws
sudo chown -R root:root /usr/lib/jvm/jdk1.7.0_79
sudo update-alternatives --config java
##########Skip above if you have java already available.
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.4.deb
sudo dpkg -i elasticsearch-1.4.4.deb
sudo update-rc.d elasticsearch defaults 95 10
script.disable_dynamic: false
cluster.name: give_your_cluster_name
node.name:"es-master-3"
node.master: true
node.data: false
node.max_local_storage_nodes: 1
index.number_of_shards: 4
index.number_of_replicas: 1
bootstrap.mlockall: true
transport.tcp.port: 9300
http.enabled:false
discovery.zen.minimum_master_nodes:2
discovery.zen.ping.timeout:10s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts:["es-master-1:9300","es-master-2:9300", "es-master-3:9300"]
action.disable_delete_all_indices: false
action.destructive_requires_name: true
marvel.agent.exporter.es.hosts: ['es-monitor-1:6200']
/etc/elasticsearch/elasticsearch.yml
Configuration for Master Node.
Give half of total available RAM to ES:vim /etc/init.d/elasticsearchES_HEAP_SIZE=2g ( on a 4 GB RAM, the more heap size the better performance, But need to keep sweet spot of 32 GB in mind)
…
ShipperLogstash
ShipperLogstash
ShipperLogstash
BrokerRedis/RabbitMQ
IndexerLogstash Elasticsearch
Using Logstash Shippers and Redis or RabiitMQ as a broker
img source: https://deviantony.wordpress.com/2014/05/19/centralized-logging-with-an-elk-stack-elasticsearch-logback-kibana/
Using Logstash Forwarder : Lumberjack Protocol
Don’t want to use any agent/shipper for system events logging??
Use Rsyslog:
1. Send logs directly to Elasticsearch: omelasticsearch http://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html
2. Use base logs sending : Create and edit the file /etc/rsyslog.d/logstash.conf with
*.* @logserver.example.com:5544 and parse the logs on centralized logstash server.
Rivers are finally removed:https://github.com/elastic/elasticsearch/pull/11568#event-332821650
Want to get a deep dive into Elasticsearch??
Join us @ 3rd Delhi Elasticsearch Meetup
http://www.meetup.com/Delhi-Elasticsearch-Meetup/events/223470631/
top related