© 2 0 1 9 s p l u n k i n c . db connect: automating the h
TRANSCRIPT
© 2 0 1 9 S P L U N K I N C .
© 2 0 1 9 S P L U N K I N C .
DB Connect: Automating theH-E-Double Hockey Sticks Out Of It
Ryan MossPrincipal Security Engineer | Verizon
During the course of this presentation, we may make forward‐lookingstatements regarding
future events or plans of the company. We caution you that such statements reflect our
current expectations and estimates based on factors currently known to us and that actual
events or results may differ materially. The forward-lookingstatements made in the this
presentation are being made as of the time and date of its live presentation. If reviewed after
its live presentation, it may not contain current or accurate information. We do not assume
any obligation to update any forward‐lookingstatements made herein.
In addition, any information about our roadmap outlines our general product direction and is
subject to change at any time without notice. It is for informational purposes only, and shall
not be incorporated into any contract or other commitment.Splunk undertakes no obligation
either to develop the features or functionalities described or to include any such feature or
functionality in a future release.
Splunk, Splunk>, Turn Data Into Doing, The Engine for Machine Data, Splunk Cloud, Splunk
Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States
and other countries. All other brand names, product names, or trademarks belong to their
respective owners. © 2019 Splunk Inc. All rights reserved.
Forward-LookingStatements
© 2 0 1 9 S P L U N K I N C .
© 2 0 1 9 S P L U N K I N C .
Why are You Here?
Why this will benefit you.
© 2 0 1 9 S P L U N K I N C .
Why are You Here?
Talk about specific use cases involving the audit logs as well as ad-hoc queries
Makes new inputs and connection creation more simplistic
Allows you to focus on other areas of Splunk
Make your DBA’s happier
Why this will benefit you
© 2 0 1 9 S P L U N K I N C .
“ I absolutely LOVE it when I
get to manually input
connections and inputs in
DB Connect!”
Said no one. Ever.
© 2 0 1 9 S P L U N K I N C .
Introductions
Who is this guy, and what is he going to talk about?
© 2 0 1 9 S P L U N K I N C .
Introductions
Principal Security Engineer
Working on Splunk for the past 6+ years
Worked on on-prem and Cloud deployments
Had one of the fastest cloud deployments
Not a DBA (sorry)
Who is this guy?
© 2 0 1 9 S P L U N K I N C .
Agenda 1) What are the use cases?Possible use cases
2) How do I do it?Splunk HF
DB Repo
SPL
Scripts
3) Demo
4) Key Takeaways
5) Q & A
© 2 0 1 9 S P L U N K I N C .
What Can I Automate?
When and what can you automate
© 2 0 1 9 S P L U N K I N C .
What are the Use Cases?
DB Audit logs for Oracle and MSSQL
• Can automate the connections and the inputs
Ad-hoc queries (connections)
• Can automate the connections
When and what can you automate
© 2 0 1 9 S P L U N K I N C .
That’s Nice, But How Do I Do It?
How to automate DB Connect
© 2 0 1 9 S P L U N K I N C .
That’s Nice, But How Do I Do It?
Need to have some things in place before automation
• Splunk Heavy Forwarder
• DB to be used as a repo for DB information (server name and instance)
• Standardized port configuration
• Good naming convention
Utilize SPL to make your scripting easier
• Create scheduled reports to pull DB information from repo
• Use SPL to preformat the data
Use scripts to combine all these into usable connections and inputs
How to automate DB Connect
© 2 0 1 9 S P L U N K I N C .
That’s Nice, But How Do I Do It?
Size the HF Accordingly
• We use 12 core 16 GB RAM virtual host (works with ~650 connections pulling every 3 min)
Need to change the default sockets and threads
• Settings found in $SPLUNK_HOME\etc.\system\local\server.conf
• Need to change maxSockets and maxThreads under [httpsServer] stanza
• Splunk defaults to “0”. If set to “0”, Splunk automatically sets [maxSockets | maxThreads] to one third of the maximum allowable [open files | threads] on the host
• Can set it to unlimited by setting it to -1 (BE CAREFUL!)
Splunk HF
© 2 0 1 9 S P L U N K I N C .
1. Splunk HFSplunk HF
That’s Nice, But How Do I Do It?
© 2 0 1 9 S P L U N K I N C .
That’s Nice, But How Do I Do It?
Need to have a database repository that you can pull from
• Can be pulled from a CMDB
Need a standardized naming convention
• Naming convention includes the hostname as well as the instance name
• End of instance name include last digits of port
– Example: myhostisawsome\sqlprod84
DON’T USE DEFAULT PORT
• Create a port standard (specific ports for dev/qa/prod)
• Use high port numbers that are not common
– Example 489 for Prod, 488 for QA, 487 for Dev
Last two digits of instance are port number
• Example: myhostisawesome\sqlprod84 – 84 are last two digits of port
• Complete port for connection would be 48984
DB Repo
© 2 0 1 9 S P L U N K I N C .
DB Repo
That’s Nice, But How Do I Do It?
Examples Names:
Hosts\Instances Port Number
awesomesauce\sqlprod33 48933
coolbeans\sqlqa55 48855
winser01\sqldev84 48784
coolserver\sqlprod99 48999
pitattack\sqlqa22 48822
beandip\sqldev7 48707
© 2 0 1 9 S P L U N K I N C .
That’s Nice, But How Do I Do It?
Use the dbxquery on your HF
Utilize eval commands to pre-process the data
Use outputlookup to output the results to a CSV
Save the search as a scheduled report
Utilize SPL
© 2 0 1 9 S P L U N K I N C .
Utilize SPL
That’s Nice, But How Do I Do It?
Example query:
| dbxquery query="select * FROM [dbrepo].[dbo].[sqlinstances]"
connection="mssql_winser01selxa_sqldev84"
| eval instance_name_modified = replace(sqlinstances,"\\\\","_")
| eval instance_name_modified = "mssql_" +
instance_name_modified
| eval instance_name_modified = lower(instance_name_modified)
| rex field=instance_name_modified
"(?<server_name>mssql_\w+_sql[qa|dev|prod]*)(?<old_port>\d{1,2})“
| eval initialLength = len(tostring(old_port))
| eval port = "0".tostring(old_port)
| eval port = substr(port,initialLength,2)
| eval instance_name = server_name + port
| table instance_name
| outputlookup sql_server_instances.csv
© 2 0 1 9 S P L U N K I N C .
That’s Nice, but How Do I Do it?
Example output
Utilize SPL
© 2 0 1 9 S P L U N K I N C .
That’s Nice, but How Do I Do It?
Script will combine everything you did previously
Automates the connection, inputs as well as creating the checkpoint file
Utilize scripts
© 2 0 1 9 S P L U N K I N C .
Utilize Scripts
That’s Nice, But How Do I Do It?
Example Script:
#!/bin/bash
##### Shutdown Splunk Service in Prep to Update Files #####
#/opt/splunk/bin/splunk stop
sudo systemctl stop Splunkd.service
##### Remove Quotes and instance_name from file and output to
txt file #####
sed 's/\"//g'
/opt/splunk/etc/apps/search/lookups/sql_server_instances.csv >
sqlserver_list.txt
sed -i '/instance_name/d' sqlserver_list.txt
© 2 0 1 9 S P L U N K I N C .
Example Script (cont’d):
##### Separate out DEV QA PROD #####
cat sqlserver_list.txt | grep sqldev > mssql_server_dev_instance.txt
cat sqlserver_list.txt | grep sqlqa > mssql_server_qa_instance.txtcat sqlserver_list.txt | grep sqlprod > mssql_server_prod_instance.txt
##### Read contents of instance file and put it in a variable #####
mapfile -t devStanza < mssql_server_dev_instance.txt
mapfile -t qaStanza < mssql_server_qa_instance.txtmapfile -t prodStanza < mssql_server_prod_instance.txt
##### Copy the DB Connections to search for new additions #####
cp
/opt/splunk/etc/apps/splunk_app_db_connect/local/db_connections.conf db_connections.txt
##### Copy the DB Inputs file to search for new additions #####
cp /opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.conf
db_inputs.txt
Utilize Scripts
That’s Nice, But How Do I Do It?
© 2 0 1 9 S P L U N K I N C .
Example Script (cont’d):
##### Create the DEV connection file #####
for dstanza in ${devStanza[@]}; do
if grep -Fq "$dstanza" db_connections.txtthen
echo -n ""
else
echo "
[$dstanza]connection_type = generic_mssql
database = master
disabled = 0
fetch_size = 10000
host = $dstanzaidentity = sql_service_account
jdbcUseSSL = true
port = $dstanza
readonly = true
timezone = UTC"fi
done > dev_connection.txt
Utilize Scripts
That’s Nice, But How Do I Do It?
© 2 0 1 9 S P L U N K I N C .
Example Script (cont’d):
##### Only show correct Hostname and Port #####
sed -i 's/host = mssql_/host = /g' prod_connection.txt
sed -i 's/port = mssql_[^_]*_sqlprod/port = 489/g' prod_connection.txtcat prod_connection.txt | sed -r 's/(host\s+=\s+\w+)_.*/\1/' >
prod_connection_test.txt; mv prod_connection_test.txt
prod_connection.txt
##### Write the connections to the db_connections file #####cat prod_connection.txt >>
/opt/splunk/etc/apps/splunk_app_db_connect/local/db_connections.conf
cat qa_connection.txt >>
/opt/splunk/etc/apps/splunk_app_db_connect/local/db_connections.conf
cat dev_connection.txt >> /opt/splunk/etc/apps/splunk_app_db_connect/local/db_connections.conf
Utilize Scripts
That’s Nice, But How Do I Do It?
© 2 0 1 9 S P L U N K I N C .
Example Script (cont’d):
##### Create the DEV inputs file #####
for dstanza in ${devStanza[@]}; do
if grep -Fq "$dstanza" db_inputs.txt
then
echo -n ""
else
echo "
[$dstanza]
batch_upload_size = 1000
connection = $dstanza
disabled = 0
fetch_size = 300
index = mssql
index_time_mode = dbColumn
input_timestamp_column_number = 1
interval = 2-59/5 0,2-23 * * *
max_rows = 0
max_single_checkpoint_file_size = 1048576
mode = rising
query = SELECT *\\
FROM sys.fn_get_audit_file ('C:\\\\SplunkAudit\\\\SplunkAudit*',default,default)) \\
WHERE event_time > ?\\
ORDER BY event_time ASC
query_timeout = 300
sourcetype = mssql:audit
tail_rising_column_number = 1"
fi
done > dev_inputs.txt
Utilize Scripts
That’s Nice, But How Do I Do It?
© 2 0 1 9 S P L U N K I N C .
Example Script (cont’d):
##### Write inputs to SQL inputs file #####
cat prod_inputs.txt >>
/opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.confcat qa_inputs.txt >>
/opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.conf
cat dev_inputs.txt >>
/opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.conf
Utilize Scripts
That’s Nice, But How Do I Do It?
© 2 0 1 9 S P L U N K I N C .
Utilize Scripts
That’s Nice, But How Do I Do It?
Example Script (cont’d):
##### Create the PROD checkpoint files #####
for stanza in ${prodStanza[@]}; do
if [ ! -e /opt/splunk/var/lib/splunk/modinputs/server/splunk_app_db_connect/$st
anza ]
then
echo '{"value":"1970-01-01
00:00:00.00","appVersion":"3.1.4","columnType":93,"timestamp":"1970-01-01T00:00:00.000-04:00"}' >
/opt/splunk/var/lib/splunk/modinputs/server/splunk_app_db_connect/$st
anza
fi
done
##### Start up Splunk #####
#/opt/splunk/bin/splunk start
sudo systemctl start Splunkd.service
© 2 0 1 9 S P L U N K I N C .
Demo
© 2 0 1 9 S P L U N K I N C .
© 2 0 1 9 S P L U N K I N C .
1. DB Connect doesn’t have to be a manual process
2. Create a DB Repo for your hosts\instances
3. Utilize SPL and scripts to automate the process
4. Can be used in Oracle as well as MSSQL
Key Takeaways
RATE THIS SESSION
Go to the .conf19 mobile app to
© 2 0 1 9 S P L U N K I N C .
You!
Thank
© 2 0 1 9 S P L U N K I N C .
Q&A
Ryan Moss | Principal Security Engineer