![Page 1: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/1.jpg)
LLNL-PRES-XXXXXXThis work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC
Elastic Stack Installation & Configuration
Anna Gassen, Ciara Goetze, James Gadson IIITeam G Code
![Page 2: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/2.jpg)
LLNL-PRES-xxxxxx
2
Objective
▪ Install and configure Elastic Stack on the Academy clusters
▪ Gather logs from all nodes
▪ Develop some insightful searches
▪ Research data analysis concepts
![Page 3: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/3.jpg)
LLNL-PRES-xxxxxx
3
Elastic StackYou know, for search
▪ Our clusters produce more than 1500 log messages per minute
▪ Comprised of six open-source tools: Elasticsearch, Logstash, Kibana, Beats, X-Pack, Elastic Cloud
▪ Allows quick analyzation, visualization, and mining of millions of log files
▪ Identify trends, statistics, and abnormalities
![Page 4: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/4.jpg)
LLNL-PRES-xxxxxx
4
Logstash
▪ Collects data from many different sources at the same time
▪ Filters and parses each message, converts it into a common format for easier analysis
▪ Aggregates and transports data to Elasticsearch (or the software of your choice)
![Page 5: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/5.jpg)
LLNL-PRES-xxxxxx
5
Filebeat
▪ A lightweight log file shipping agent
▪ Part of the Beats family of data shippers
▪ Communicates directly with Logstash or Elasticsearch
▪ Easily forwards and centralizes log files
![Page 6: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/6.jpg)
LLNL-PRES-xxxxxx
6
Elasticsearch
▪ Full-text search engine that searches and centrally stores data
▪ Quickly find, retrieve, and analyze big volumes of data
▪ Distributed and highly scalable
▪ Near real time search
▪ Uses RESTful API, JSON, and Lucene
![Page 7: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/7.jpg)
LLNL-PRES-xxxxxx
7
Kibana
▪ Data visualization tool for log and time series analytics
▪ Makes navigation and monitoring of logs more intuitive
▪ Provides numerous graph and dashboard options to display information
![Page 8: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/8.jpg)
LLNL-PRES-xxxxxx
8
![Page 9: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/9.jpg)
LLNL-PRES-xxxxxx
9
![Page 10: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/10.jpg)
LLNL-PRES-xxxxxx
10
Approach
5
Boron
10.811
B
28
Nickel
58.6934
Ni
86
Radon
222.018
Rn
lgw1lgw2
lgw3
![Page 11: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/11.jpg)
LLNL-PRES-xxxxxx
11
Number of Documents per Node
![Page 12: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/12.jpg)
LLNL-PRES-xxxxxx
12
Failed Login Attempts
![Page 13: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/13.jpg)
LLNL-PRES-xxxxxx
13
Root vs Non-Root Logins
![Page 14: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/14.jpg)
LLNL-PRES-xxxxxx
14
Number of Documents per Day
![Page 15: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/15.jpg)
LLNL-PRES-xxxxxx
15
Martian Source Warnings
![Page 16: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/16.jpg)
LLNL-PRES-xxxxxx
16
Future work
▪ Research Logstash pipeline configuration options
▪ Utilize Beats and X-Pack
▪ Perform more complex Elasticsearch queries
▪ Configuring Elastic Stack to be useful to future Academy interns
![Page 17: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/17.jpg)
LLNL-PRES-xxxxxx
17
Acknowledgements
▪ David Fox
▪ Geoff Cleary
▪ Pam Hamilton
▪ Bryan Dixon
▪ Richard Randall
![Page 18: Anna Gassen, Ciara Goetze, James Gadson III Team G Code · Elasticsearch Full-text search engine that searches and centrally stores data Quickly find, retrieve, and analyze big volumes](https://reader033.vdocument.in/reader033/viewer/2022043006/5f9103d36e100657b30e7c64/html5/thumbnails/18.jpg)