WHY TALK ABOUT LOGS?They are all around us
We use them to debug our software every day
Is that all tere is?
ELASTICSEARCHThe database
document orienteddistributedshardedreplicated*
timestamp partitioning*
Java & lucene
PARSING LOGSfilter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } }
PARSING LOGS(̂?#regex designed to parse VyOS kernel log)(?#some global parsing, like timestamp, fitlter, interfaces, and so on)(?<time>[̂ ]* [̂ ]* [ ̂]*) (?<host>[̂ ]*) (?<vyos_sylog_facility>[̂: ]*)?: \[(?<vyos_fw_filter_name>[̂\[]*)\] ?IN=( |(?<vyos_in_interface>[̂ ]*) )OUT=( |(?<vyos_out_interface>[̂ ]*) )(MAC=( |(?<vyos_mac_addres>[̂ ]*) ))?SRC=( |(?<vyos_source_ip_address>[̂ ]*) )DST=( |(?<vyos_dstination_ip_address>[̂ ]*) )LEN=( |(?<vyos_len>[̂ ]*) )TOS=( |(?<vyos_tos>[̂ ]*) )PREC=( |(?<vyos_prec>[̂ ]*) )TTL=( |(?<vyos_ttl>[̂ ]*) )ID=( |(?<vyps_packet_id>[̂ ]*) )(?<vyos_packet_flags>[̂ |(PROTO)]*)? ?PROTO=( |(?<vyps_ip_protocol>[̂ ]*))(?#here comes the fun part, different parser for different interesting packet types, regex if and positive lookbehind matching each type of interesting )(?:(?<=(TCP))((?#tcp specific matchers) ?SPT=( |(?<vyos_source_port>[̂ ]*) )DPT=( |(?<vyos_destination_port>[̂ ]*) )WINDOW=( |(?<vyos_tcp_window>[̂ ]*) )RES=( |(?<vyos_res>[̂ ]*) )(?<vyps_tcp_state>[̂(URGP)]* ).*)|(?:(?<=(UDP))((?#udp specific matchers) ?SPT=( |(?<vyos_source_port>[̂ ]*) )DPT=( |(?<vyos_destination_port>[̂ ]*) ).*)|(?:(?<=(ICMP))((?#icmp specific matchers) TYPE=( |(?<vyos_icmp_type>[̂ ]*) )CODE=( |(?<vyos_icmp_code>[̂ ] )).*)|(.*))))$
CUSTOM SOLUTION?Implement custom data collection within the application
Populate data with Servlet filters or Spring AOP
Index data in Elasticsearch
BENEFITSLogging that meets your needs
Ability to extract analytical data
Near real-time event tracking
HTTP DATABandwith costs money
Web site scrapers are common
Serving non-compressed data is expensive
Identification of scrapers can reduce cost
3RD PARTY API TRACKINGLog all requests and responses
Monitor performance
Monitor availability
Provide extra troubleshooting data
BENEFITS OVER COMMERTIALSOLUTIONSno software coston site or cloud installfull control over datafully customizable
THANK YOUelastic.co
Fluentd
Apache Flume