passwords*are** forchumps - splunkconf · limitaonsof splunksso 8!...

Copyright © 2014 Splunk Inc.

David Veuve SE, Splunk

Passwords are for Chumps

Who Am I?

2

!   David Veuve – Sales Engineer for Major Accounts in Northern California

! [email protected] !   Former Splunk Customer (For 3 years, 3.x through 4.3) !   Security Guy !   Primary Author of Splunk Search Usage app !   David on Splunk Answers

Agenda

3

!   Why Single Sign On (SSO)? !   SeUng up SSO on Windows !   SeUng up SSO on Linux !   SeUng up SSO via SAMLv2 !   A liWle something extra !   Wrap up

!   All config files (where possible for Windows) will be posted to GitHub at the end of the presenta[on

Disclaimer

4

During the course of this presenta[on, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cau[on you that such statements reflect our current expecta[ons and

es[mates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,

please review our filings with the SEC. The forward-‐looking statements made in the this presenta[on are being made as of the [me and date of its live presenta[on. If reviewed a`er its live presenta[on, this presenta[on may not contain current or accurate informa[on. We do not assume any obliga[on to update any forward-‐looking statements we may make. In addi[on, any informa[on about our roadmap outlines our general product direc[on and is subject to change at any [me without no[ce. It is for informa[onal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obliga[on either to develop the features or func[onality described or to

include any such feature or func[onality in a future release.

What is Wrong with Passwords

5

!   Diminish adop[on !   Dispropor[onately discourage the users you really want

–  Execu[ves/Managers, Business Users

!   Fundamentally insecure

Detail: Passwords are Fundamentally Insecure

6

!   People write them on post-‐it notes

!   People create simple ones !   People type them into phishing websites

!   People reuse them across many websites –  hWp://xkcd.com/792/

hWp://xkcd.com/936/

Benefits of Single Sign On

7

!   Easier adop[on !   More secure !   Facilitates High Availability

–  Search Head Pooling works beWer with SSO enabled ê  Allows you to fail over without a user no[cing

Limita[ons of Splunk SSO

8

!   Single Sign On depends on an external proxy that will handle the authen[ca[on piece, and then pass the username in an HTTP header to Splunk

!   Even with Single Sign On handling authen[ca[on, we s[ll need an LDAP connec[on to assign users to individual roles. This is not typically an issue for internal deployments, but is a greater issue for SAML deployments –  Can cover standard roles [To be filled in]

Single Sign On -‐ Defini[on

9

!   Single sign-‐on (SSO) is mechanism whereby a single ac6on of user authen6ca6on and authoriza6on can permit a user to access all computers and systems where he has access permission, without the need to enter mul6ple passwords –  hWp://www.opengroup.org/security/sso/

!   In prac[ce: Users are automa[cally logged in without typing in a password

Common Single Sign On Methods

10

!   Ac[ve Directory –  AD has supported SSO via NTLM and others for years

!   Kerberos –  Core to Ac[ve Directory and widely used in Linux / OSX

!   SAML –  Commonly used for online systems

!   Smart Card (or One Time Password) –  Can be implemented by one of the above, or a hook into Ac[ve Directory to

intercept and service authen[ca[on accounts

!   Several others employing similar core theories

How to Decide Which Method

11

!   Windows Server Environment: –  Windows Authen[ca[on –  Easiest setup in my experience

!   Linux Server Environment: –  Kerberos –  S[ll easy

! Splunk hosted via external cloud (or with 3rd party SSO such as Okta, PingIden[ty, etc.): –  SAML –  Most Challenging approach

!   3rd Party Proxy / Load Balancer –  Likely Kerberos, but depends on product

Splunk Setup

Splunk Setup Steps

13

1.  Set up LDAP Authen[ca[on 2.  Map LDAP Groups 3.  Update server.conf 4.  Update web.conf

LDAP Configura[on

14

!   Frequently done by Splunk Users –  hWp://docs.splunk.com/Documenta[on/Splunk/6.1.3/Security/ConfigureLDAPwithSplunkWeb

!   From Splunk Web, Access Controls

server.conf and web.conf Setup

15

! server.conf –  trustedIP Indicates that the local splunkd will trust the user coming from

splunkweb ê  (Remember that indexers implicitly trust the search head, so this only happens on the search head)

! web.conf –  trustedIP Indicates that splunkweb will trust the user coming from your

upstream proxy/other device –  SSOMode Indicates whether local logons are allowed –  remoteUser Indicates what header parameter the user string will be put into

Security Quick Tip

16

!   Limit the number of trusted IPs you have configured on splunkweb, as they will be able to masquerade as any user

!   If you have tools.proxy.on = true, and see your worksta[on’s IP address in /debug/sso, turn off tools.proxy.on and don’t add every worksta[on to the trustedIP list

Demo – Splunk Setup

17

Demo -‐ Splunk LDAP Setup

18

Demo – server.conf

19

! server.conf – Refers to the local splunkd –  Remember that splunkweb running on the same box will communicate with

splunkd via 127.0.0.1

Demo – web.conf

20

! web.conf – Refers to the local splunkweb –  SSOMode

ê  Permissive – allows either SSO or direct access to splunkd ê  Strict – SSO only (cannot log in with local auth seUngs – if locked out, must modify via conf files)

–  trustedIP ê  IP of Proxy

–  remoteUser ê  Parameter containing username

–  tools.proxy.on ê  Required for old versions of Apache. This is turned on in a bunch of examples, but for none of the systems I’ve used has it actually been necessary

Windows Op[on

Core Technologies at Play

22

!   Func[oning Splunk Install !   Ac[ve Directory Infrastructure !   IIS Web Server (2012 R2 in my test, but known to work at least through 2008) –  Plarorm addons:

ê  ARR –  hWp://www.iis.net/downloads/microso`/applica[on-‐request-‐rou[ng

ê  ISAPI Module ê  ISAPI Filters Module

–  Free Third Party ê  ISAPI_Rewrite3 –  hWp://www.isapirewrite.com/ –  Allows you to add authen[cated user name to header

High Level Process

23

1.  Configure Authen[ca[on for IIS Site 2.  Configure Reverse Proxy for IIS Site 3.  Configure URL_Rewrite to empty Accept Encoding

–  Workaround for UI quirk

4.  Configure ISAPI_Rewrite3 to put REMOTE-‐USER header

Windows Authen[ca[on Diagram

24

!   Users will hit the IIS Server, which will authorize them via Integrated Windows Authen[ca[on

!   Requests will then be proxied to Splunk ! Splunk will perform authoriza[on via LDAP Groups

!   Users will get a seamless authen[ca[on and authoriza[on experience, and be greeted by the Splunk page!

Challenges

25

!   By default, Splunk will use gzip encoding, but that doesn’t work with IIS ARR rou[ng rules. As a result, we need to store the original Accept Encoding in a header, wipe it, and then replace it. That will be seen in the example

!   IIS does not support wri[ng the authen[cated user informa[on into a header. This is why we need the external ISAPI_Rewrite3 Lite module. Fortunately, we can use the free Lite module by offloading the rou[ng

!   (Neither of these issues exist on Linux, or should exist on 3rd party proxies or load balancers)

Why Third Party (ISAPI_Rewrite3 Lite)

26

!   ISAPI_Rewrite3 by Helicon is a great way to port configura[ons over from Apache

!   In par[cular, it allows us to set a header a`er the authen[ca[on part completes, which is not possible out of the box with IIS

!   There are two versions of ISAPI_Rewrite3 – free and commercial –  For this configura[on, we only need the free version. The commercial

version adds addi[onal proxy capabili[es which are delivered by IIS ARR

Demo – Enabling Authen[ca[on

27

Demo – Enabling Reverse Proxy

28

Demo – Configure URL Rewrite

29

Demo – Workaround for URL Rewrite Quirk

30

Demo -‐ Helicon

31

Demo – Successful SSO Debug

32

Demo – Successful Logon

33

Troubleshoo[ng

34

! Wireshark – Verify that communica[on to your search head has the proper field populated

!   Debug page –  hWp://YourIISServer/debug/sso

!   IIS Detailed Debug Logs –  By default, IIS will only show you the major error code (e.g., 500). If you turn

on detailed logs, it will also show the more detailed logs, e.g.: ê  HTTP Error 500.52 -‐ URL Rewrite Module Error. Outbound rewrite rules cannot be applied when the content of the HTTP response is encoded ("gzip")

Troubleshoo[ng with Wireshark

35

!   Capture relevant traffic (port 8000) !   Then look for the actual headers being passed in the HTTP message

Troubleshoo[ng with Debug SSO

36

!   Great source for ensuring your seUngs are correct

!   Look par[cularly for the SSO Mode, trustedIPs and the Remote user HTTP Header. This has to be the same as what is seen inWireshark

!   Hopefully your setup will lookjust like this

Troubleshoo[ng with IIS Logs

37

!   By default IIS logs aren’t very helpful. While troubleshoo[ng, turn on detailed logs for your site

!   Just click on Error Logs, then Edit Feature SeUngs, then Detailed Logging

Linux Op[on

Core Technologies

39

!   Working Splunk Installa[on !   Linux Kerberos !   Apache Web Server

–  mod_auth_kerb –  mod_proxy –  mod_rewrite

!   Ac[ve Directory (or other Kerberos Store)

High Level Process

40

!   Create AD Service Account !   Create keytab !   Configure Linux Host Kerberos !   Configure Apache to use mod_auth_kerb !   Configure Apache to revers proxy using mod_proxy !   Configure Request Header to set Remote User

Linux Authen[ca[on Diagram

41

!   Users will hit the Apache Server, which will authorize them via Kerberos to AD



Challenges

42

!   Biggest challenge with this approach is that there are many different sets of instruc[ons on the internet. This approach, end to end, worked in my environment

Demo – Create AD User

43

!   Nothing complex about the user account – can be anything

Demo – Create Keytab

44

!   Copy-‐paste from internet. Note that this will reset the password ! ktpass -‐princ {PRINCIPAL NAME} -‐mapuser {username@fqdn} -‐crypto {YourChoice} -‐ptype KRB5_NT_PRINCIPAL -‐pass {LookAtMyLongPassword} -‐out {Path\to\keytab}

Demo -‐ Configure Linux Host Kerberos

45

!   Change the realm to your local realm !   Note that this should probably match your users’ desktop config – i.e., if they log into mydomain.local and you’re hos[ng this site on mydomain.com, you will need to configure IE/Firefox/Chrome to try a kerberos Auth

Demo -‐ Configure Apache to use auth_kerb

46

!   Change the realm and AuthName to your local realm/domain FQDN

!   Configure the Krb5KeyTab to where you copied the file over from your domain controller

!   KrbMethodK5Passwd allows users without kerberos to authen[cate via password

!   Require valid-‐user tells Apache that authen[ca[on is required

Demo -‐ Configure Apache to Reverse Proxy

47

!   This leverages and requires mod_proxy to work, but is a preWy straighrorward config beyond that

!   The last two lines are the heart of the config – behind the scenes, take anything going to myserver/* and send a parallel request to hWp://127.0.0.1:8000/*

!   If moun[ng your web path at a different directory, consider the root_endpoint seUng

! hWp://www.davidveuve.com/tech/proxying-‐splunk-‐with-‐ssl/

Demo – Configure Remote User Header

48

!   Unlike with Windows, here we can leverage a simple config to insert the remote user into the REMOTE-‐USER header

!   In seUng this up, I tried several aWempts to get the remote_user properly inserted – this is the one that finally worked

Demo – PuUng it all together

49

!   All the configura[on for my environment lives in /etc/hWpd/conf.d/splunksso.conf

!   The en[re configura[on is here →

Troubleshoo[ng

50

!   Paralleling the Windows troubleshoo[ng, there are three great tools for troubleshoo[ng on Linux: –  Apache Logs (hey, it’s super easy to Splunk those!) –  Debug SSO Splunk Endpoint –  tcpdump

Troubleshoo[ng with Apache Logs

51

!   Make sure your keytab is in the right path! !   Make sure your web server name matches your principal name!


52


!   Look par[cularly for the SSO Mode, trustedIPs and the Remote user HTTP Header. This has to be the same as what is seen in tcpdump

!   Hopefully your setup will look just like this

Troubleshoo[ng with tcpdump

53

!   Great to verify that the reverse proxy actually works and that the seUngs are correct

!   Look par[cularly for the the Remote user HTTP Header

SAML Op[on

Core Technologies

55

!   Working Splunk Installa[on !   Linux Host (CentOS 6.0 for this demo)

–  yum install xmlsec1 xmlsec1-‐openssl xmlsec1-‐openssl-‐devel openssl hWpd mod_ssl

–  Install EPEL on your RHEL-‐type box to get the xmlsec1s –  Lasso

!   Apache Web Server –  mod_auth_mellon

!   SAMLv2 Iden[ty Provider –  Recommend that to get started, you leverage a known working partner such

as Okta (used here) or PingIden[ty. Then adapt to your own SAMLv2

High Level Process

56

!   Install host dependencies !   Set up Iden[ty Provider (e.g., Okta/PingIden[ty/etc.) !   Set up mellon config !   Set up mod_auth_mellon config

!   Based almost completely on Paul Stout’s excellent guide: hWp://blogs.splunk.com/2013/10/09/splunk-‐sso-‐using-‐saml-‐through-‐okta/

SAMLv2 Authen[ca[on Diagram

57

!   Users will hit the Okta Server, which will authorize them and then forward them (via POST) to the Splunk server, which does not have to be accessible to Okta (can be behind the VPN)



Challenges

58

!   The provided versions of mod_auth_mellon / lasso only work for hWpd 2.2. There will be a conflict if you try to install on 2.4, and when I tried a newer version of mod_auth_mellon (0.7.0 instead of 0.5.0) it never worked, and never errored out –  Recommend that you set up first on 2.2 (RHEL or equivalent 5.x or 6.x,

verify with hWpd -‐v) as it’s a known working version

!   SAMLv2 is a notoriously finicky setup with lots of moving parts. Recommend that you start with a known working combina[on (e.g., Okta has a no-‐limit free version for a single app), then make incremental changes to move to your own implementa[on

On Groups

59

!   The major downside to SAMLv2 in Splunk is that it will only handle authen[ca[on. You will s[ll need to set up groups to handle authoriza[on, which would require an LDAP connec[on

Demo – Install Host Dependencies

60

! wget hWp://dl.fedoraproject.org/pub/epel/6/x86_64/epel-‐release-‐6-‐8.noarch.rpm !   rpm -‐ivh epel-‐release-‐6-‐8.noarch.rpm !   yum install hWpd xmlsec1 xmlsec1-‐openssl xmlsec1-‐openssl-‐devel mod_ssl openssl !   Disable or tune selinux (/etc/selinux/config) !   Set your hostname to match your principal name (e.g., splunk.dvsplunk.com) ! wget hWps://dev.entrouvert.org/redhat/6/RPMS/x86_64/lasso-‐2.3.6-‐1.el6.x86_64.rpm ! wget hWps://dev.entrouvert.org/redhat/6/RPMS/x86_64/

mod_auth_mellon-‐0.5.0-‐1.el6.x86_64.rpm !   rpm -‐ivh lasso-‐2.3.6-‐1.el6.x86_64.rpm !   rpm -‐ivh mod_auth_mellon-‐0.5.0-‐1.el6.x86_64.rpm

Demo – Set up Iden[ty Provider (IdP)

61

!   Very easy with Okta –  Add Applica[on –  Provide URL –  Provide Default Relay State and username

Demo – Grab IdP Metadata

62

!   Also very easy with Okta:

Demo – Set up Mellon Config

63

!   Paul Stout’s previously-‐linked-‐to guide includes a handy script that will set up the suppor[ng mellon files for Splunk:

Demo – Set up mod_auth_mellon

64

!   The Mellon config is preWy straighrorward, and very copy-‐pasteable

!   For an explana[on of the ProxyPass configura[on, please see the Linux Config sec[on

Troubleshoo[ng

65

!   The recommended troubleshoo[ng tools for this configura[on are iden[cal to those for normal Linux systems: –  Apache Logs (hey, it’s super easy to Splunk those!) –  Debug SSO Splunk Endpoint –  tcpdump

Troubleshoo[ng with Apache Logs

66

!   Make sure your keytab is in the right path! !   Make sure your web server name matches your principal name!


67


!   Look par[cularly for the SSO Mode, trustedIPs and the Remote user HTTP Header. This has to be the same as what is seen in tcpdump

!   Hopefully your setup will look just like this

Troubleshoo[ng with tcpdump

68

!   Great to verify that the reverse proxy actually works and that the seUngs are correct

!   Look par[cularly for the Remote user HTTP Header

Shameless Plug

Splunk Search Usage

70

! Splunk Search Usage Analysis and Adop[on Tracking, with security reports

!   hWp://www.davidveuve.com/go/ssu

Wrap Up

Wrap Up

72

!   Three Op[ons for Single Sign On: –  Windows Web Server – Easy –  Linux Web Server – Easy –  SAML – Achievable, recommend a packaged solu[on if you need this

(e.g., Okta, PingIden[ty, etc.)

!   SSO gives you more security, greater adop[on, and less headache !   You can probably set this up in your environment in < 1 hr !   Check out the Splunk Search Usage app to beWer understand users and broaden adop[on!

Config Files – GitHub

73

!   That was a lot of material, right? !   Get all the configs here: hWp://www.davidveuve.com/go/conf-‐sso

hWp://xkcd.com/565/

THANK YOU

passwords*are** forchumps - splunkconf · limitaons*of* splunksso 8!...

Documents

passwords*are** forchumps - splunkconf · limitaonsof splunksso 8!...