elkingyour postgresql database infrastructure...>what’s need to be done on the postgres...
TRANSCRIPT
Infrastructure at your Service.
Elking your PostgreSQL Database Infrastructure
Infrastructure at your Service.
30.06.17Elking your PostgreSQL Database InfrastructurePage 2
About me
Arnaud BerbierSenior Consultant
+41 79 128 91 45
Agenda
1.Playground Infrastructure
2.Elastic Search Stack
3.PostgreSQL Monitoring
4.Conclusion
30.06.17Page 3
Elking your PostgreSQL Database Infrastructure
Playground Infrastructure
30.06.17Elking your PostgreSQL Database InfrastructurePage 4
> Big Picture
> Use case and deployment process
Big Picture
30.06.17
Playground Infrastructure
Page 5
Elking your PostgreSQL Database Infrastructure
Elastic Stack 5.4.1 Postgres Clusters 10.0 PGPool II
PG Standby Node
PG Primary Node PGPool Node 0
PGPool Node 1
Hea
rtbe
at
SSH192.168.56.0/24
Version 5.1.18 r114002 (Qt5.6.2)
HealthCheck
Version 2.3.0
Stre
amin
g R
eplic
atio
n
vmtestpgsql02192.168.56.210/32
vmtestpgpool02192.168.56.230/32
vmtestpgpool01192.168.56.220/32
Kibana
Elastic Node 0
Elastic Search
Beats
Beats
Beats
vmtestpgsql01192.168.56.200/32
vmtestelastic01192.168.56.100/32
Ship
Ship
Inge
st
Visualize
Use case and deployment process
30.06.17
Playground Infrastructure
Page 6
Elking your PostgreSQL Database Infrastructure
Database Clusters
Elastic Stack Server
Dashboard import
Metric Agent Deployment
PGPool Metric Agent Deployment
Elastic Search Stack
30.06.17Elking your PostgreSQL Database InfrastructurePage 7
> What does ELK mean ?
> What’s a Beat ?
> Is all the stack Open Source ?
> How much costs the cloud offering ?
> ELK is a complete monitoring platform - Elastic Search Stack
> At the beginning Elastic Stack was commonly known as ELK
> E - Elastic Search> Store, Search and Analyze
> L – Logstach> Collect, Enrich and Transport
> K – Kibana> Explore, Visualize and Share
What does ELK means ?
30.06.17
Elastic Search Stack
Page 8
Elking your PostgreSQL Database Infrastructure
DATA DATADATA DATA
INFORMATIONDATA
DATATRANS
INFORMATION
What does ELK means ?
30.06.17
Elastic Search Stack
Page 9
Elking your PostgreSQL Database Infrastructure
Kibana
Elastic Search
Logstash Beats
Elastic Cloud
Graph
Alerting
Monitoring
Reporting
Security
User InterfaceVisualization
> It evolved to be a full Monitoring Stack
Store, Index & Analyzse
Ingest
X-Pack
> Lightweight Data Shippers to ingest data into Elastic
What’s a Beat ?
30.06.17
Elastic Search Stack
Page 10
Elking your PostgreSQL Database Infrastructure
Is all the stack open source ?
30.06.17
Elastic Search Stack
Page 11
Elking your PostgreSQL Database Infrastructure
> Components support over subscriptions
Is all the stack open source ?
30.06.17
Elastic Search Stack
Page 12
Elking your PostgreSQL Database Infrastructure
Open Source Basic Gold Platinum
Elas
ticSe
arch Elasticsearch
Kibana
Beats
Logstash
X-P
ack Monitoring
Multicluster
Configurable Data retention
X-Se
curi
ty Authentication + Encryption
Role-Based access ctrl
Audit
Encryption at rest support
> Components support over subscriptions
> Details : https://www.elastic.co/subscriptions
Is all the stack open source ?
30.06.17
Elastic Search Stack
Page 13
Elking your PostgreSQL Database Infrastructure
Open Source Basic Gold Platinum
X-P
ack
Alerting via Watcher
Reporting
Graph Analytics & Visualization
Dev Tools – Search Profiler
Supp
ort Support Coverage BH 24x7
Response TimesCrit. 4hL2: 1dL3: 2d
Crit. 1hL2: 4hL3: 1d
Unlimited # of Incidents
> Elastic Cloud Offering – Host Elastic Search and Kibana
How much cost the cloud offering ?
30.06.17
Elastic Search Stack
Page 14
Elking your PostgreSQL Database Infrastructure
PostgreSQL Monitoring
30.06.17Elking your PostgreSQL Database InfrastructurePage 15
> Postgres Cluster Settings
> What’s need to be done on the Postgres Clusters> Filebeat log harvesting
> The elastic stack process owner needs access to the Postgres log files> log_file_mode in pg_settings parameters
Postgres Cluster Settings
30.06.17
PostgreSQL Monitoring
Page 16
Elking your PostgreSQL Database Infrastructure
postgres=# select name, setting, unit from pg_settings where name like '%log_file%' or name like '%log_dir%';
name | setting | unit
----------+---------------------------------------------+------
log_directory | /app/postgres/admin/dbixchange01/pg_log |
log_file_mode | 0776 |
log_filename | postgresql-%a.log |(3 rows)
> This correspond to file mode 666
> What’s need to be done on the Postgres Clusters> Metric Beat postgresql
> A dedicated user for DB connection > Allowed access to the database
> Elastic servers need to be accessible by the deployed beat agents> Firewall> Network
Postgres Cluster Settings
30.06.17
PostgreSQL Monitoring
Page 17
Elking your PostgreSQL Database Infrastructure
> Easy to deploy> Ingest lots of data of any log type> Already have predefined metric such as
Pro & contra
30.06.17
Conclusion
> Most of the required features needs subscription
Page 18
Elking your PostgreSQL Database Infrastructure
Infrastructure at your Service.
30.06.17
We look forward to working with you!
Page 19
Elking your PostgreSQL Database Infrastructure
Arnaud BerbierSenior Consultant
+41 79 128 91 [email protected]
Any questions? Please do ask