simple log monitoring with prometheus and grafana · syslog logs, apache and other webserver logs,...
TRANSCRIPT
![Page 2: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/2.jpg)
About Me● DevOps engineer of PT. Gits Indonesia (https://www.gits.id)
● Cloud Explorer
● FOSS Community member : opensuse, opensuse-id, KLaS, etc
● Fortune Teller
● Contact me :
● Blog :
● https://medium.com/@sitidy
![Page 3: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/3.jpg)
BackgroundLog monitoring systemLight resourceRich content of DashboardAlternate for Elasticsearch Logstash and Kibana Stack
![Page 4: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/4.jpg)
How About Grafana Loki?
<>
Nope, we will not talk about it in this class
![Page 5: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/5.jpg)
Open-source systems monitoring and alerting toolkit originally built at SoundCloud.Since its inception in 2012, many companies and organizations have adopted Prometheus. The project has a very active developer and user community.It is now a standalone open source project and maintained independently of any company.Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.
<> What is Prometheus?
![Page 6: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/6.jpg)
<> Grok Exporter● Grok is a tool to parse crappy unstructured log data
into something structured and queryable. ● Heavily used in Logstash to provide log data as input
for ElasticSearch.● Grok ships with about 120 predefined patterns for
syslog logs, apache and other webserver logs, mysql logs, etc.
● It is easy to extend Grok with custom patterns.● The grok_exporter aims at porting Grok from the ELK
stack to Prometheus monitoring. The goal is to use Grok patterns for extracting Prometheus metrics from arbitrary log files.
![Page 7: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/7.jpg)
<> Grok Exporter● https://github.com/fstab/grok_exporter● Log → grok_exporter → prometheus?
![Page 8: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/8.jpg)
<> Unstructured to Structured● Unstructured :− ERROR 30.07.2016 14:37:03 alice 1.5− WARNING 30.07.2016 14:37:33 alice 2.5− ERROR 30.07.2016 14:43:02 bob 2.5− ERROR 30.07.2016 14:45:59 alice 2.5
![Page 9: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/9.jpg)
<> Unstructured to Structured● Structured :− LOGLEVEL: ERROR− DATE: 30.07.2016− TIME: 14:37:03− USER: alice− NUMBER: 1.5
![Page 10: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/10.jpg)
<> grok_exporter config file● Input:− type: file
path: ./example.log● grok:− patterns_dir: ./logstash-patterns
● metrics:− type: counter− name: grok_example_lines_total− help: Counter metric example.− match: '%{LOGLEVEL} %{DATE} %{TIME}
%{USER} %{NUMBER}'
![Page 11: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/11.jpg)
Lorem ipsum doloret is amet
One articleLorem ipsum dolor sit amet, consectetuer adipiscing elit.
Two articleLorem ipsum dolor sit amet, consectetuer adipiscing elit.
Three articleLorem ipsum dolor sit amet, consectetuer adipiscing elit.
Four articleLorem ipsum dolor sit amet, consectetuer adipiscing elit.
![Page 12: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/12.jpg)
Lorem ipsum doloret is amet
One articleLorem ipsum dolor sit amet, consectetuer adipiscing elit.
Two articleLorem ipsum dolor sit amet, consectetuer adipiscing elit.
Three articleLorem ipsum dolor sit amet, consectetuer adipiscing elit.
![Page 13: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/13.jpg)
DEMO
![Page 14: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/14.jpg)
DEMOTools:● grok_exporter● http://grokconstructor.appspot.c
om/do/match
![Page 15: Simple Log Monitoring with Prometheus and Grafana · syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns. The grok_exporter](https://reader033.vdocument.in/reader033/viewer/2022052814/60a028811248727175128803/html5/thumbnails/15.jpg)
Thank you!
We Are Hiringhttps://gits.id/career