best practices for splunk ssl (the sslippery slope revisited)splunk’s default ssl posture 5 but...
TRANSCRIPT
![Page 1: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/1.jpg)
Copyright©2015SplunkInc.
DuaneWaddleGeorgeStarcherDefensePointSecurity
BestPracticesforSplunkSSL(TheSSLipperySlopeRevisited)
![Page 2: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/2.jpg)
SSLRefresher
2
Authenticationoftheserver(theserveriswhotheysaytheyare)OptionalauthenticationoftheclientBulkencryptionofdataintransitSeveralmovingparts,“CAs”,“keys”,“CSRs”,“certs”Weoftensay"SSL"whenwemean"TLS".TrueSSLiseffectivelydead.(ORISIT!?)
![Page 3: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/3.jpg)
SplunkArchitectureandSSL
3
Splunkweb(SSLtobrowsers)Splunk-to-splunkdatatransfer(forwarderstoindexers)SplunkdRESTport(Inter-Splunk)
DeploymentClient/DeploymentServerRESTAPI/SDKsDistributedSearch
LDAPconnectionsClustering
![Page 4: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/4.jpg)
Splunk’sdefaultSSLposture
4
Theout-of-the-boxconfiguration:Allcertificatesaregeneratedonadefault-shippedCAconfigurationSplunkwebdoesnotuseSSLSplunkdusesSSLfortheRESTport-withcertificateverificationdisabledNoSSLdatainputs/outputsaredefinedSplunkdLDAPcanuseSSL-againwithnocertificateverification
![Page 5: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/5.jpg)
Splunk’sdefaultSSLposture
5
ButforSplunk6.3.3.4/6.3.4ApparentlyturnsoffSSLv3bydefault(undocumented)Breakscommunicationwithsomeolderforwarders/patchlevels(like6.1.0)sslVersions=tls,ssl3
Don'tusethisasanexcusenottoupgrade!http://www.splunk.com/view/SP-CAAAPKV(advisory)http://blogs.splunk.com/2016/04/06/splunk-maintenance-releases-and-patch-to-address-the-drown-openssl-vulnerability/(blogpostw/d/llinks)
![Page 6: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/6.jpg)
6
Type of exchange Client function Server function Encryption Certificate Authentication
Common Name checking
Type of data exchanged
Browser to Splunk Web Browser Splunk Web NOT enabled by default
dictated by client (browser)
dictated by client (browser)
search term results
Inter-Splunk communication
Splunk Web splunkd enabled by default NOT enabled by default NOT enabled by default search term results
Forwarding splunkd as a forwarder splunkd as an indexer NOT enabled by default
NOT enabled by default NOT enabled by default data to be indexed
Inter-Splunk communication
splunkd as a deployment client
splunkd as deployment server
enabled by default NOT enabled by default NOT enabled by default configuration data
Inter-Splunk communication
splunkd as a search head splunkd as search peer Enabled by default NOT enabled by default NOT enabled by default search data
http://docs.splunk.com/Documentation/Splunk/latest/Security/AboutsecuringyourSplunkconfigurationwithSSL
![Page 7: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/7.jpg)
Whythisstuffmatters
ADPSpenetrationtesterfoundhimselfonarandomLinuxboxasanunprivilegeduser.Thisboxwas:
RunningSplunkforwarderasroot...withthedefaultadmin/changemepassword...anddefaultSSLconfigs,trustinganycertificate
7
![Page 8: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/8.jpg)
Whythisstuffmatters
Hewasableto:UsetheRESTAPItochangethedeploymentserverIP(tohisbox)RestarttheforwarderDownloadanapptotheforwarderthatstartedareverserootshellPivotfromrootonthatboxtodownloadingthesite'sChefrepoLiftacopyofalloftheirrecipes,includingAWSAPIkeys
8
Moralofthestory:
WhoevercontrolsyourDScontrolstheusersrunningyourforwarders.
![Page 9: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/9.jpg)
(some)BestPracticesChecklist
RunSplunkforwardersasanunprivilegeduserChangeforwarderadminpasswordsEnablestrongSSLauthenticationbetweenDSclientandDSserverUsehostbasedfirewalltolimitoutboundconnectionstotrustedIPsPickanappropriatecipherSuiteUsewildcardSSLcertswithcautionDecideonFIPSmodeearlyonandtalktoSplunkfirstNotrunning6.3yet?Stillrunningonthedefaultcerts?
https://answers.splunk.com/answers/395886/for-splunk-enterprise-splunk-light-and-hunk-pre-63.html
9
![Page 10: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/10.jpg)
CommercialCAorPrivateCA?
CommercialRootcertsareineveryone'sbrowseralreadyCostsrealmoney(potentiallyaLOTifyouuseECC)Potentialrenewalnightmare
PrivateYouhavetorunaCA(likelyalreadyare...)Free(ish)RootcertsmustbedistributedYoucandoverylongexpirations(intheory)
10
![Page 11: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/11.jpg)
HowmanycertsdoIneed?
11
Splunkweb-SearchHead:A3rd partyCAcertanditsroot/intermediates
Splunkd:Arootcertanditsintermediates-(either3rdpartyorprivate)OneperSplunkServernonSearchHead
(oroneperroleinlargeenvs)OnethrowawaycertificateforalloftheSplunkUFstoshare
![Page 12: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/12.jpg)
OurExampleArchitecture
12
![Page 13: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/13.jpg)
CreateSplunkServerKey&CSR
13
$ mkdir $SPLUNK_HOME/etc/auth/myOrg $ cd $SPLUNK_HOME/etc/auth/myOrg $ openssl req -nodes -newkey rsa:2048 -keyout
splunk-srv1.web.key -out splunk-srv1.csr $ openssl rsa -in splunk-srv1.web.key -des3 -out splunk-srv1.key
GettheCArootcertificatechainandputinauth/myOrgascacert.crt.
CopytheCA-returnedcrtfiletoauth/myOrg/splunk-srv1.crt
![Page 14: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/14.jpg)
TheSearchHead-SplunkWeb
14
$ cd $SPLUNK_HOME/etc/auth/myOrg
$ cat splunk-srv1.crt cacert.crt > splunk-srv1.web.pem
$ vi $SPLUNK_HOME/etc/system/local/web.conf
[settings]
enableSplunkWebSSL = 1
httpport = 8443
privKeyPath = etc/auth/myOrg/splunk-srv1.web.key
caCertPath = etc/auth/myOrg/splunk-srv1.web.pem
sslVersions = tls, -tls1.0
cipherSuite = ?
![Page 15: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/15.jpg)
Architecture-Status
15
![Page 16: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/16.jpg)
Indexers
16
FirstmakecertificatesasyouwouldforSplunkWeb.
$ cd $SPLUNK_HOME/etc/auth/myOrg $ openssl req -nodes -newkey rsa:2048 -keyout splunk-idx01.key -out splunk-idx01.csr $ openssl rsa -in splunk-idx01.web.key -des3 -out splunk-idx01.key
TomaketheIndexerformatted.PEM:$ cat splunk-idx01.crt splunk-idx01.key cacert.crt > splunk-idx01.pem
![Page 17: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/17.jpg)
TheIndexer-Inputs.conf
17
vi $SPLUNK_HOME/etc/system/local/inputs.conf
[splunktcp-ssl://9998] disabled = 0
[SSL] password = <REDACTED> rootCA = $SPLUNK_HOME/etc/auth/myOrg/cacert.crt serverCert = $SPLUNK_HOME/etc/auth/myOrg/splunk-idx01.pem sslVersions = tls, -tls1.0 cipherSuite = ? requireClientCert = true|false
![Page 18: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/18.jpg)
TheForwarder
18
FirstmakecertificatesasyouwouldforSplunkWeb.Thiscanbedoneonyourdeploymentserver.
$ cd $SPLUNK_HOME/etc/auth/myOrg/forwarder $ openssl req -nodes -newkey rsa:2048 -keyout splunk-forwarder.web.key -out splunk-forwarder.csr
Onforwardersonly,makethekeypassword"password"forreasons... $ openssl rsa -in splunk-forwarder.web.key -des3 -out splunk-forwarder.key
Throwawaysplunk-forwarder.web.key $ cat splunk-forwarder.crt splunk-forwarder.key cacert.crt > splunk-forwarder.pem
Copythesplunk-forwarder.pemandcacert.crttoyourForwarder(s):YesyoucoulduseanAPPforthis.
![Page 19: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/19.jpg)
ForwardertoIndexer-Outputs.conf
19
vi $SPLUNK_HOME/etc/system/local/outputs.conf
[tcpout] defaultGroup = myIndexers
[tcpout:myIndexers] server = splunk-idx01.myorg.com:9998 sslCertPath = $SPLUNK_HOME/etc/auth/myOrg/splunk-forwarder.pemsslPassword = password # For Reasons sslRootCAPath = $SPLUNK_HOME/etc/auth/myOrg/cacert.crt sslVerifyServerCert = true sslCommonNameToCheck = splunk-idx01.myorg.com
![Page 20: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/20.jpg)
Gotcha-ForwardertoIndexer
20
IfyoumistypethesslRootCAPathargumentinoutputs.conf,theforwarderwilldefaulttonot-SSLwhentryingtotalktoindexer.Theerrorontheindexerwilllooklikethefollowing:
6-23-2014 20:46:48.918 +0000 ERROR TcpInputProc - Error encountered for connection from src=10.0.1.57:41778. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
![Page 21: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/21.jpg)
Architecture-Status
21
![Page 22: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/22.jpg)
TheDeploymentServer
22
FirstmakecertificatesasyouwouldforSplunkWeb.
$ cd $SPLUNK_HOME/etc/auth/myOrg
$ openssl req -nodes -newkey rsa:2048 -keyout splunk-d.web.key -out splunk-d.csr $ openssl rsa -in splunk-d.web.key -des3 -out splunk-d.key
TomaketheDeploymentServerformattedpem:$ cat splunk-d.crt splunk-d.key cacert.crt > splunk-d.pem
![Page 23: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/23.jpg)
TheDeploymentServer-server.conf
23
vi $SPLUNK_HOME/etc/system/local/server.conf
[sslConfig]
caCertFile = cacert.crt
caPath = $SPLUNK_HOME/etc/auth/myOrg
sslKeysfile = splunk-d.pem
sslKeysfilePassword = <REDACTED>
sslVersions = tls, -tls1.0
cipherSuite = ?
requireClientCert = false
![Page 24: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/24.jpg)
SplunkForwarder-DSClients
24
vi $SPLUNK_HOME/etc/system/local/server.conf
[sslConfig]
caCertFile = cacert.crt
caPath = $SPLUNK_HOME/etc/auth/myOrg
sslKeysfile = splunk-forwarder.pem
sslKeysfilePassword = password # Reasons
sslVersions = tls, -tls1.0
sslVerifyServerCert = true
sslCommonNameToCheck = splunk-d.myorg.com
![Page 25: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/25.jpg)
Architecture-Status
25
![Page 26: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/26.jpg)
Splunk-ServertoServer
26
vi $SPLUNK_HOME/etc/system/local/server.conf
[sslConfig] caCertFile = cacert.crt caPath = $SPLUNK_HOME/etc/auth/myOrg
sslKeysfile = splunk-srvXX.pem sslKeysfilePassword = <REDACTED>
sslVersions = tls, -tls1.0 cipherSuite = ?
requireClientCert = false sslVerifyServerCert = true sslCommonNameList = splunk-srv01.myorg.com, splunk-d.myorg.com, splunk-idx01.myorg.com, splunk-idx02.myorg.com, ...
![Page 27: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/27.jpg)
Architecture-Status
27
![Page 28: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/28.jpg)
SplunkLDAPS
28
EachLDAPstrategyhasanSSLtoggleon/off InGUI,it’sacheckbox Inauthentication.conf,eachLDAPstanzaneedsSSLEnabled=1
MinimumCertificatesettingsin$SPLUNK_HOME/etc/openldap/ldap.conf TLS_REQCERT demand
TLS_CACERT /opt/splunk/etc/auth/LDAProotcert.crt
TLS_CIPHER_SUITE (equivalenttocipherSuite)
![Page 29: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/29.jpg)
IndexerClustering
29
IndexerclusteringusesbothRESTAPIandadedicatedclusterdatatransferportCerts&configforRESTAPIareallcoveredaboveSSLsignatureandcommonnamecheckingoccurBEFOREpass4SymmKeychecking
Protip:Ifbuildingaclusterfromscratch,usethesamesplunk.secretonallclusternodesConvertingaclusterfromdefaultcertstoproductioncertscanbebrittle
EnablesslVerifyServerCertandsslCommonNameListLASTsslCommonNameListneedstolistallpossibleRESTcommunicationspartners
Allindexers,clustermaster,licenseserver,andsearchheads...
![Page 30: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/30.jpg)
IndexerClustering-SSLDataTransfer
30
Minimaldocumentation-onlyonereferencetoitinthedocshttp://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverconf
Inserver.confcommentoutreplication-portstanzaandadd:
[replication_port-ssl://8002] password = <REDACTED> rootCA = $SPLUNK_HOME/etc/auth/myOrg/cacert.crt serverCert = $SPLUNK_HOME/etc/auth/myOrg/splunk-idx01.pem
Trythisoutinatestclusterfirst!ThisisNOTacommonsettinginthewild
![Page 31: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/31.jpg)
SHCandKVStore
31
SHC-sameRESTportrulesapplyaswithindexerclusteringKVStorehasitsownSSLconfigstanzainserver.conf:
[KVstore] caCertPath = … sslKeysPath = … sslKeysPassword = …
DocsmentiontheseONLYworkinFIPSmode-needsmoretesting
![Page 32: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/32.jpg)
ThankYou!
32
Other resources Splunk IRC ( EFNet #splunk )
Splunk Answers ( http://answers.splunk.com ) Splunk community wiki ( http://wiki.splunk.com ) Splunk User Group Slack ( http://splunk402.com/chat/ )
http://www.georgestarcher.com/ http://www.duanewaddle.com/
Past (and future!) virtual.conf presentations:
http://wiki.splunk.com/Virtual_.conf
![Page 33: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/33.jpg)
Bonus Material Deleted Scenes
![Page 34: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/34.jpg)
BeyourownCertificateAuthority
34
• WewilluseECCcryptoforhigherperformance• StartoutbymakingaCARootkeyandcertificate.• VeryhelpfulSplunkBlogspostbyJoseHernandez
http://blogs.splunk.com/2014/06/03/generate-elliptical-curve-certkeys-for-splunk/
• Youwillbepromptedforpassphrasesformultiplekeys– Keepthemsecret– Keepthemsafe– Useadifferentpassphraseforeverykey
![Page 35: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/35.jpg)
CreatetheCARootKey&Cert-ECC
35
$ cd $SPLUNK_HOME/etc/auth/myOrg
$ splunk cmd openssl ecparam -name "prime256v1" -genkey |
splunk cmd openssl ec -des3 -out CAroot.key
Enter PEM pass phrase: <abc123>
Verifying - Enter PEM pass phrase: <abc123>
$ splunk cmd openssl req -key CAroot.key -sha1 -subj
"/CN=Splunk Root CA/O=myOrg" -new -x509 -days 3650
-set_serial 1 -out cacert.crt
Enter pass phrase for CAroot.key: <abc123>
![Page 36: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/36.jpg)
CreateSplunkServerKey&CSR-ECC
36
$ splunk cmd openssl ecparam -name "prime256v1" -genkey -out splunk-d.web.key $ splunk cmd openssl ec -des3 -in splunk-d.web.key -out splunk-d.key Enter PEM pass phrase: <def234> Verifying - Enter PEM pass phrase: <def234> $ splunk cmd openssl req -key splunk-d.key -subj
"/CN=splunk-d.myorg.com/O=myOrg" -new -out splunk-d.csr
![Page 37: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/37.jpg)
SigntheSplunkCertusingRootCert-ECC
37
$ splunk cmd openssl x509 -req -days 1095 -in splunk-d.csr -CA cacert.crt -CAkey CAroot.key -set_serial 02 -out splunk-d.crt Signature ok
subject=/CN=splunk-d.myorg.com/O=myOrg
Getting CA Private Key
Enter pass phrase for CAroot.key: <abc123>
Now we have a keyfile (both encrypted and not) and a cert issued by our CA
![Page 38: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/38.jpg)
SomeotherwaystobeyourownCA
38
ActiveDirectoryCertificateServicesFedoraCertificateServerhttp://pki.fedoraproject.org/wiki/PKI_Main_PageAlsoapartoftheFreeIPAsuite(CommerciallyasRedHatDirectoryServer)
![Page 39: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/39.jpg)
BonusMaterial
39
SplunkBlog:http://blogs.splunk.com/2014/06/03/generate-elliptical-curve-certkeys-for-splunk/
Troubleshooting:http://mikeberggren.com/post/28429473721/chain-check
![Page 40: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/40.jpg)
Testconnectivitywithopenssls_client
40
OpenSSL has a built-in SSL client that you can use to do basic connectivity testing. Works ‘just like TELNET’ but over SSL No certificate verification by default, but you can get it to dump the presented certs so you can check them by hand. It will also dump TLS protocol version and negotiated cipher specification $ openssl s_client -connect 10.10.10.10:8089 -showcerts
The returned certs can be checked in plaintext by copypasting into a file and running $ openssl x509 -text -noout -in xxxx.crt
![Page 41: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/41.jpg)
ForwardertoLBIndexers-Outputs.conf-1
41
vi $SPLUNK_HOME/etc/system/local/outputs.conf
(or use an app)
[tcpout]
defaultGroup = myIndexers
[tcpout:myIndexers]
maxQueueSize = 128MB
useACK = true
autoLB = true
server = splunk-idx01.myorg.com:9998, splunk-idx02.myorg.com:9998
sslCertPath = $SPLUNK_HOME/etc/auth/myOrg/splunk-forwarder.pemsslPassword = <REDACTED>
sslRootCAPath = $SPLUNK_HOME/etc/auth/myOrg/cacert.crt
![Page 42: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/42.jpg)
ForwardertoLBIndexers-Outputs.conf-2
42
vi $SPLUNK_HOME/etc/system/local/outputs.conf
(or use an app)
[splunk-idx01.myorg.com]
sslVerifyServerCert = true sslCommonNameToCheck = splunk-idx01.myorg.com
[splunk-idx02.myorg.com]
sslVerifyServerCert = true sslCommonNameToCheck = splunk-idx02.myorg.com
![Page 43: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/43.jpg)
File formats can and will trip you up
43
DifferentareasofSplunkuseSSLkeyfiles/certfilesformattedslightlydifferently
SplunkalwaysexpectsPEMencodedcerts&keysSomeCAswillsendDERandyou’llhavetoconvertSomewillsendPKCS7,PKCS12,orevenstrangerfiles
Splunkwebv6.1.xandolderhasCherryPydependenciesSSLkeyfilemustbeunencryptedSSLkeyandSSLcertmustbeinseparatefiles
Splunkdexpectskey/cert/root-certallinonefile
![Page 44: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/44.jpg)
HandlingPKCS7packagedcerts
44
SometimeshappensfromaSSLadmingrabbingcertsfromComodoandoftenhasthewholecertificatechain.Yeah,thishappenedtoGeorgehelpingsomeonerebuildtheirSplunk.
Starts like: -----BEGIN PKCS7-----
MIIOewYJKoZIhvc
Tochangetheformat:openssl pkcs7 -inform PEM -in $PKCS7_FILE -outform PEM -print_certs > splunk-srv1.pem
Copythefilesplunk-srv1.pemtocacert.pemvicacert.pemanddeletethefirstcertificateandsavethefileCopythefilesplunk-srv1.pemtosplunk-srv1.crtanddeletethelasttwocertificatesandsavethefile
![Page 45: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/45.jpg)
HandlingPKCS12formattedcerts
45
Sometimesyou’llevengetPKCS12(.pfx)filesbackfromthecertificateauthority/SSLadminPKCS12filesmaycontainbothcertsandkeysTochangetheformat:• openssl pkcs12 -in $PKCS7_FILE -out splunk-srv1.pem
• Take the resulting .pem file, and break it up into different files for each part – CA Root / Intermediate certs – Your issued certs – Keys (if any)
• You “should” be able to tell which is which by the common name and issuer – If not, run each through ‘openssl x509 -text -noout -in <file>’
![Page 46: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/46.jpg)
Certificateverificationvscommon-namematching
46
• Unique,butcomplementary,partsoftheSSLauthenticationscheme• SplunkcandosameCAverificationwithoutcommon-namematching• SplunkCNmatchingdoesrequireCAverificationbetrue• Certificateverificationisacryptographicoperation.
– Doesacert’ssignaturebyitsissuercryptographicallyverifywhencheckedusingtheissuer’spublickey?
• Common-Namematchingcomesnext– DoestheCN=inthecertificatematchtheCNyouareexpecting?– BrowsersdothiscomparisonagainsttheDNShostnameintheURL– Splunkdoesthisbyhardcodedconfigurationentry
![Page 47: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/47.jpg)
errors:)
47
ThisisfromenablingsslVerifyServerCert=trueandscrewingupaclusterpeer’scertonpurpose
09-07-201400:51:55.619-0400ERRORSSLCommon-Certificatedoesn'tverify,err=1909-07-201400:51:55.619-0400INFONetUtils-SSLConnectioncouldnotbemade-serverauthenticationerror09-07-201400:51:55.619-0400WARNHTTPClient-SSL_ServerAuthErrorconnectingto=104.131.13.214:808909-07-201400:51:55.619-0400WARNHTTPClient-Connectto=104.131.13.214:8089timedout;exceeded30sec
![Page 48: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/48.jpg)
moreerrors
48
Thisisfrom(againonpurpose)puttinginafalseCommonNameToCheck
09-07-201415:53:33.771-0400ERRORSSLCommon-Commonnamedoesn'tmatchservercertcommonname=splunk-d.myorg.com.Triedtomatchaaa.bbb.cc.09-07-201415:53:33.771-0400WARNHTTPClient-SSLConnectioncouldnotbemade-serverauthenticationfailed09-07-201415:53:33.771-0400WARNHTTPClient-SSL_ServerAuthErrorconnectingto=splunk-d.myorg.com:808909-07-201415:53:33.771-0400WARNHTTPClient-Connectto=splunk-d.myorg.com:8089timedout;exceeded30sec
![Page 49: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/49.jpg)
CipherSuiteErrors
49
RanintoanerrorsettingupSSLonindexercluster.AfterenablingnewcertontheCM,errordoinga'splunkapplycluster-bundle'Splunkd.logonCMshowserror:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher
Somewiresharkinglater...cipherSuite = HIGH fixed it
![Page 50: Best Practices for Splunk SSL (The SSLippery Slope Revisited)Splunk’s default SSL posture 5 But for Splunk 6.3.3.4 / 6.3.4 Apparently turns off SSLv3 by default (undocumented) Breaks](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec52a4bad0cc77157272f52/html5/thumbnails/50.jpg)