multiple splunk as a service (msaas)

WHITE PAPER

MULTIPLE SPLUNK AS A SERVICE (MSAAS)An Architecture Framework for Multitenant Splunk Deployments

WHITE PAPER

2Multiple Splunk as a Service (MSaaS)

Table of ContentsIntroduction ................................................................................................................................................................................................................... 3

Architecture ................................................................................................................................................................................................................... 4

MSaaS Architecture Features and Benefits............................................................................................................................................... 4

Service-Oriented Approach to Deployment ............................................................................................................................................. 5

Architectural Concepts ..................................................................................................................................................................................... 6

Splunk Enterprise Architectural Concepts ......................................................................................................................................... 6

MSaaS Logical Architecture .................................................................................................................................................................................... 8

Search Tier ............................................................................................................................................................................................................ 9

Indexing Tier ......................................................................................................................................................................................................... 11

Data Collection Tier ........................................................................................................................................................................................... 12

Namespace Best Practices .............................................................................................................................................................................. 13

MSaaS Roles ......................................................................................................................................................................................................... 13

Summary: Architectural Components ......................................................................................................................................................... 14

MSaaS Physical Architecture .................................................................................................................................................................................. 15

Search Tier ............................................................................................................................................................................................................ 15

Indexing Tier ......................................................................................................................................................................................................... 16

Data Collection Tier ........................................................................................................................................................................................... 16

Administration .............................................................................................................................................................................................................. 17

Deployment .......................................................................................................................................................................................................... 17

Management and Monitoring .................................................................................................................................................................................. 19

Licensing and Chargeback .............................................................................................................................................................................. 20

Credentials ............................................................................................................................................................................................................ 20

Example Implementation ......................................................................................................................................................................................... 20

Architecture .......................................................................................................................................................................................................... 21

Authentication and Authorization ......................................................................................................................................................... 23

Licensing ......................................................................................................................................................................................................... 23

Monitoring and Management .................................................................................................................................................................. 23

Automated Provisioning ........................................................................................................................................................................................... 24

Post-implementation Conclusions ........................................................................................................................................................................ 24

Key Observations ................................................................................................................................................................................................ 25

Additional Resources ................................................................................................................................................................................................. 25

Summary ......................................................................................................................................................................................................................... 26

WHITE PAPER


IntroductionToday data is growing at an ever-accelerating rate,

and it will continue to do so into the foreseeable

future. The use of Splunk Enterprise to monitor and

analyze large volumes of data continues to grow as

well, driving increased demand for Splunk Enterprise.

In this time of rapid growth, administrators are

increasingly called upon to provision new Splunk

Enterprise instances for their internal or external

customers. Service providers often provision separate

instances of Splunk Enterprise for each of their many

customers, and enterprises commonly set up separate

instances for each business unit that requires Splunk

Enterprise within their organization.

Organizations with large-scale, multitenant Splunk

Enterprise deployments need to provide data

segregation and access control for individual tenants

to meet regulatory requirements or internal security

policies. In addition, they need a scalable solution

that can successfully handle the volume of data and

the growing number of instances under management.

These organizations strive to speed deployment and

manage both deployment and upgrade risk, all while

controlling administrative costs. They need a cost-

efficient approach that reduces the marginal cost of

each additional Splunk Enterprise instance and helps

optimize their total cost of ownership of the platform.

Multiple Splunk as a Service (MSaaS) is an

architectural framework that proposes a multi-

instance approach to supporting multiple internal or

external customers. Although multiple customers can

be supported on a single Splunk Enterprise instance,

the multi-instance approach is inherently more

scalable and provides essential data segregation

capabilities in a multi-tenant environment. The Splunk

Enterprise licensing model, together with indexing

volume tracking for each individual instance, can

be used as a basis for flexible chargeback plans for

individual clients. Leveraging this multi-instance

approach can provide an economical and scalable

solution for managing multitenant deployments.

The MSaaS approach proposes an automated,

on-demand request process to provision Splunk

Enterprise instances as needed for individual clients,

or to tailor deployments for operational reasons

within an organization. Investing in the design and

implementation of an MSaaS architecture benefits

ROI when deploying new Splunk Enterprise instances.

The automated deployment process can be very time

efficient—taking minutes or hours to deploy, integrate,

and test a new Splunk Enterprise instance.

Deployment is packaged and modular, and it is

consistent across all instances. This consistency

reduces the risk of introducing errors and simplifies

managing the deployed instances. Integration with

a version control system (VCS) provides reliable

tracking and control of the deployed configuration

files, reduces the risks associated with periodic

configuration changes, and enables rollback to a

known stable state when carrying out changes.

Using a configuration management system (CMS)

provides ease of deployment and scalability for

large implementations and promotes consistency

and reliability. The MSaaS architecture is highly

flexible and supports custom configuration for each

deployed Splunk Enterprise instance.

This document describes a conceptual framework

for designing an MSaaS architecture that supports

multitenant Splunk Enterprise deployments as

a service. The sections that follow describe the

MSaaS architecture, outline typical administration

and deployment tasks, and describe an example

implementation of this architecture. This document,

intended as a starting point in designing and

implementing an MSaaS deployment, provides

a conceptual framework rather than a complete

implementation plan, followed by a description of

an example customer implementation.

WHITE PAPER


ArchitectureThe MSaaS architectural framework defines a

conceptual approach to deploying Splunk Enterprise

instances as a service. When implemented, the

framework described in this document enables

the automated, centralized deployment of Splunk

Enterprise instances through a service-request

model. In the MSaaS architecture, multiple islands

are created, with each island containing a complete

traditional Splunk Enterprise deployment for an

individual tenant (a customer or business unit).

The underlying capabilities of each Splunk Enterprise

deployment are unchanged. By implementing MSaaS

with the architectural framework described in this

document, organizations can design a service-

oriented implementation that can quickly and easily

deploy new Splunk Enterprise instances. Each

instance retains the capabilities and distributed

deployment options of a normal Splunk Enterprise

instance, and each instance can be sized to meet

the performance requirements for the use cases

it supports.

MSaaS Architecture Features and Benefits

The MSaaS architectural framework is designed

to guide the implementation of a service-oriented

approach for automatically deploying Splunk

Enterprise instances for customers. Each customer—

for example, a distinct enterprise business unit—

requires a Splunk Enterprise deployment, and an

island is created for each deployment. A single island

can service multiple customers if their respective

segregation requirements permit; there is no need to

deploy them as multiple tenants.

Key features of the MSaaS architecture include

the following:

• Service-request model. When a new Splunk

instance is needed, customers submit a service

request that describes the use cases and attributes

of the new instance. For example, attributes might

include the replication factor, search factor,

recovery point objective (RPO) and recovery

time objective (RTO) for disaster recovery,

backup requirements, retention plan, and daily

indexing volume.

• Automated deployment of each Splunk Enterprise instance. Predefined scripts developed

using a configuration management system are

used to automatically deploy new Splunk

Enterprise instances.

• Highly scalable deployments. The MSaaS

architecture improves Splunk Enterprise’s inherent

scalability by partitioning large deployments—

those that support multiple organizations,

span multiple data centers or multiple geographic

locations, and index very large volumes—into

islands with low marginal overhead per-island.

The number of islands that can be created is not

limited, and each island can be scaled in the usual

manner by which Splunk Enterprise scales

through dynamically provisioning multiple

clustered or individual indexers and search heads

to handle the required workload.

• Multitenancy. The MSaaS architecture supports

multitenancy by deploying a set of Splunk

Enterprise components running on distinct

computing resources for each tenant, ensuring

complete functionality along with full segregation

between tenants. The architecture can also

support a unified view of multiple islands up

to the entire environment, and it delivers the

ability to track resource usage for each tenant so

that utilization-based chargeback models can

be implemented.

• Segregation of data, flexible jurisdiction. The

MSaaS architecture maintains separation of data

for Splunk Enterprise instances. Data routing

rules can be implemented to control the flow

of data to specific indexers, and searches can be

performed over specific data sets. Flexible

jurisdiction capabilities enable hierarchical or

WHITE PAPER


overlapping groupings of islands and the data they

contain, and role-based access control (RBAC) can

be used to implement jurisdiction, restricting

access to data, as needed.

• Flexible and customizable instances. The MSaaS architecture imposes no fixed

requirements on the underlying Splunk Enterprise

instances that make up each island. Instances

can scale from very small to as large as necessary,

and they can be customized to fulfill a wide

range of use-case requirements.

Implementing automated, centralized deployment

of Splunk Enterprise instances provides significant

advantages for organizations that need to administer

a number of Splunk instances. This service-oriented

approach helps organizations:

• Improve time to value. Once the MSaaS approach

is implemented, administrators can simply submit

a service request specifying the characteristics of

the new instance, and the new Splunk Enterprise

instance can be automatically installed and running.

• Reduce the marginal effort to deploy each instance. Once MSaaS is implemented, all that is

required to deploy a new Splunk Enterprise

instance is its characteristics. The marginal effort

to deploy each new instance is minimal.

• Manage deployment and upgrade risk. The MSaaS

service-based approach reduces the risk of human

error and makes the deployment process

consistent for all Splunk Enterprise instances.

Common scripts are used for configuration, and

a version control system (VCS) is used to maintain

copies of these scripts, configuration files, Splunk

Apps, and any other required artifacts, providing

an audit trail and a consistent deployment process.

Upgrades can also be easily deployed to all

Splunk Enterprise instances from a centralized

server, ensuring the deployment remains

consistent across the data center.

Service-Oriented Approach to Deployment

With a service-oriented approach, deployment of new

Splunk Enterprise instances is streamlined: Once the

MSaaS framework is in place, administrators provision

and deploy a new island through a service request.

A first step in designing the MSaaS architecture is

analysis of the attributes that are required for each

Splunk Enterprise instance. For example, typical

attributes can include:

• Data indexing daily volume

• Licensing and chargeback requirements

• Storage volume

• Retention policies

• Resilience requirements

• App provisioning

• Disaster recovery requirements

• Data segregation and jurisdiction requirements

Data segregation and jurisdiction requirements drive

the data collection configuration for each island. A

service request for a new island includes a list of data

sources (also known as end points), an estimate of

the daily data volume they generate, and a method

by which data can be identified as required or not

required by the island in question. Each end point can

send data to any island, and each collected event can

be sent to multiple islands or ignored.

After the set of island attributes are defined, a

mechanism for specifying a request should be

provided. For example, an online form can be used

to help define each attribute. Predefined attribute

sets that define common instance types can facilitate

the users’ selection of these instance types.

As a final step in the MSaaS design, scripts are

created to map the specification of the new instance

to the automated deployment process. Configuration

management tools are used to automate the task of

creating the Splunk Enterprise components that are

required for the new instance. VCS tools are used to

maintain control of scripts, configuration files, add-

ons, and apps that are used for the instances.

WHITE PAPER

6

When the MSaaS architecture is implemented, it can

service requests. Each service request contains details

on the attributes for the new instance. Once the

request is submitted, the automated process creates

and deploys the Splunk Enterprise components

required for that service.

Architectural Concepts

The MSaaS framework uses normal Splunk Enterprise

architectural concepts: Each island in the MSaaS

architecture contains a complete, traditional

Splunk Enterprise deployment. Building on top of

the traditional architecture, the MSaaS framework

adds the new concepts of islands, bridges, routing,

jurisdictions, and the MSaaS deployment server.

The following sections first provide a brief overview

of the Splunk Enterprise architecture and then

introduce the MSaaS architectural concepts.

For a more thorough description of the Splunk

Enterprise architecture, refer to the Splunk Enterprise documentation.

Splunk Enterprise Architectural Concepts At a high level, each Splunk Enterprise deployment

includes a data collection tier, an indexing tier, and a

search tier. The data collection tier consumes data.

The indexing tier parses and indexes the incoming

data and services search requests from the search

tier. And the search tier requests interactive or

scheduled searches from the indexing tier on the

indexed data, and presents, consolidates, or stores

their results.

For small deployments, a single computer can handle

the resource requirements of all three tiers. However,

more typically the functionality is split across multiple

specialized Splunk Enterprise components running

on separate computing platforms (see Figure 1).

Each component handles one or more roles, such

as indexing or search. Additional components can be

added to each tier to scale it horizontally to meet

the performance requirements. For more information,

see the Splunk Enterprise Distributed

Deployment Manual.

Figure 1. Example Splunk Enterprise architecture with search tier, indexing tier and data collection tier.

Search Tier

Indexing Tier

Firewall Router

Data Collection Tier

Search Head Cluster

Indexer

Forwarder

Indexer Indexer Indexer

Forwarder Forwarder

http://docs.splunk.com/Documentation

http://docs.splunk.com/Documentation

http://docs.splunk.com/Documentation/Splunk/6.2.4/Deploy/Distributedoverview

http://docs.splunk.com/Documentation/Splunk/6.2.4/Deploy/Distributedoverview

WHITE PAPER


The three tiers contain the following components:

• Data Collection Tier: Forwarder. The forwarder

component consumes data and forwards it

to the indexing tier either directly or through

intermediate forwarders. For configurations

with multiple indexers, the data streams from

the forwarders can be load balanced across

several indexers. Load balancing helps

performance, scalability and indexing-tier

resilience. If one or more of the indexers become

unavailable, the forwarder can send its data to

another indexer.

• Indexing Tier: Indexer. Each indexer component

or its aggregate (an indexer cluster) indexes

incoming data and performs searches across

its indexes.

Figure 2. Example Splunk Enterprise architecture using clustering to provide increased scalability and resilience.

• Search Tier: Search Head. Each search head

component or its aggregate (a search head

cluster) distributes the search queries to

its indexing tier, consolidates and finalizes the

search results, and presents them to the user.

Environments of all sizes can deploy clustering for

increased resilience and availability (see Figure

2). Larger environments commonly include a

global license master and a deployment server for

administrative convenience.

DeploymentServer

Search Head Cluster

ClusterMembers

Peer Nodes(Search Peers)

Indexer Cluster

Each cluster membercommunicates with the peer nodes.

Each forwarder communicateswith the peer nodes.

Forwarders

Master ClusterNode

Captain

LicenseServer

WHITE PAPER


Figure 3. An island contains a single end-to-end Splunk Enterprise deployment.

Main components include the search head cluster,

indexer cluster, deployment server and license master:

• Search Head Cluster. A search head cluster is a

group of three or more search heads that share

the search-tier workload. Members of the search

head cluster share configurations, job scheduling

and search artifacts. One cluster member,

designated the captain, coordinates all cluster

wider activities. If the member serving as captain

goes down, another member automatically takes

its place.

• Indexer Cluster. Indexer clusters are groups of

Splunk Enterprise indexers that keep multiple

copies of replicated data. Index replication

prevents data loss and promotes data availability.

Splunk Enterprise indexer clusters feature

automatic failover. If an indexer fails, the surviving

indexers continue to perform indexing and

searches across all of the data. The work of

the indexer cluster is managed by its cluster

master node.

• Deployment Server. The deployment server

centrally manages configuration files and apps

across the Splunk Enterprise deployment.

• License Master. The license master tracks indexed

data volume and enforces Splunk Enterprise’s

licensing model.

MSaaS Logical Architecture

An island is a logical construct that defines a single

end-to-end Splunk Enterprise deployment (see

Figure 3). Each island contains one or more indexers

and zero or more search heads. Optionally, each

island can also contain a license master, deployment

server, a cluster master node, and zero or more job

servers (non-interactive search heads dedicated to

running scheduled jobs). Each island also controls the

configuration of a set of forwarders. Each forwarder

can send data to any number of islands. However,

there is a single island that controls each forwarder’s

configuration through its deployment server.

An island’s search heads and indexers can be

clustered. Islands can also include passive standby

instances of the license master, deployment server

and cluster master node for resilience.

Islands maintain data segregation. Each island

maintains its own indexed data, and islands do not

share information with each other. Each island also

leverages the enterprise authentication system to

manage permissions and roles, providing secure

access to its data.

Each island contains any number of apps or add-ons.

These can include standard or customized Splunk

Apps and Add-Ons, and also island-specific or site-

specific versions. The configuration management

and deployment automation systems maintain the

complete set of apps and add-ons.

Site-Aware Cluster

...

Island LicenseManager(optional)

DeploymentServer(optional)

Island SearchHead Cluster

Island JobServer Cluster(optional)

Indexer 1Cluster MasterNode

Indexer n

WHITE PAPER


Figure 4. Islands provide segregation of data.

Search Tier The search tier in the MSaaS architectural framework consists of two distinct components: island search

heads (whether clustered or not) and bridges. The island search heads service only their island and cannot

access data indexed by other islands. This design enables the creation of separate jurisdictions of data,

providing data segregation that may be required to meet regulatory requirements or internal enterprise

security and privacy policies. In the configuration shown in Figure 4, for example, data indexed and stored

on Island 1 cannot be seen by Island 2.

Data Sources

SearchHead

Indexer IndexerIndexer

Island 1


Data Jurisdiction 2

Island 2

Data Sources

SearchHead

Data Jurisdiction 1

WHITE PAPER


A bridge is similar to an island but has no indexing or data collection capabilities or components and does not

require access to a license server. Bridges are used to provide search access to indexed data on one or more

islands and can be deployed to access data across multiple islands. In the example shown in Figure 5, Island 1 and

Island 2 use a bridge to create a new data jurisdiction that includes the data indexed on both islands.

Figure 5. Bridges are used to create data jurisdictions that span multiple islands.

The search heads in both the island and the bridge are shown as a single component in Figure 5, but they can be deployed as clusters or as multiple, distinct search heads, as required.

Search Head

Bridge

Data Sources

SearchHead


Island 1


Data Jurisdiction 2

Island 2

Data Sources

SearchHead

Data Jurisdiction 1

WHITE PAPER


The MSaaS architectural framework also supports hierarchical and overlapping data jurisdictions (see Figure 6). The framework enables the creation of data jurisdictions that provide the data segregation required to support separate customer use cases while simultaneously allowing wider or global visibility needed for certain use cases, such as an enterprise security monitoring use case.

Cluster master node resilience is not implemented

in the core Splunk Enterprise software. However,

the island can be configured with either a warm

standby or cold standby cluster master node that

can be brought into service if the primary master

node fails. A warm standby is a cluster master node

that is running and ready to take over immediately if

the active cluster master node fails. A cold standby

is a similarly configured system that is normally

not running but can be started if the cluster master

node fails. The appropriate choice of the warm or

cold standby configuration depends on the disaster

recovery RTO for that island. The indexing cluster

continues to function as normal during the recovery

of the cluster master node, because it can sustain a

period of cluster master node downtime without loss

of functionality.

Indexing Tier The MSaaS architectural framework recommends a

clustered indexing tier. The indexing tier in an island

contains one or more indexers, a cluster master

node, and an optional redundant standby cluster

master node—identical to a standard distributed

Splunk Enterprise deployment. The specific indexing

components deployed and their configuration

are determined by the island’s service attributes,

including the required resiliency and disaster recovery

capabilities.

Even if multiple clustered indexers are not initially

required for resilience and disaster recovery, the

indexing tier should be deployed as an atrophied

site-aware cluster—consisting of only an indexer and

a cluster master node—to accommodate automatic

site-aware replication if resilience or disaster recovery

is required in the future.

Figure 6. Bridges can be used to create overlapping and hierarchical data jurisdictions.

Data Jurisdiction 1Data Jurisdiction 2

Data Jurisdiction 3

Island 1 Island 2 Island 3

Bridge 3

Bridge 1 Bridge 2

Search Head Cluster

Search Head Cluster

Search Head Cluster

Indexer IndexerIndexer Indexer IndexerIndexer Indexer IndexerIndexer

WHITE PAPER


Data Collection Tier The data collection tier in the MSaaS architectural

framework consists of end points that generate data

and forwarders. A forwarder is a Splunk Enterprise

instance that forwards data to an indexer, either

directly or through any number of intermediate

forwarders. An instance that receives data from a

forwarder is called a receiver. Intermediate forwarders

are usually deployed to meet network-topology or

security requirements or to enhance data safety where

there is a risk of data loss due to network performance

issues and a shortage of in-flight data buffering.

Different types of forwarders can be deployed to meet

the performance and use case requirements for a

particular configuration.

• Universal Forwarder. A universal forwarder is a

dedicated, streamlined Splunk Enterprise instance

that contains only the essential components

required to send data. Universal forwarders have

no parsing, searching, indexing or alerting

capabilities. Their primary role is to send unparsed

data to indexers or intermediate forwarders. Of

particular relevance to the MSaaS framework,

universal forwarders can be used to implement

routing of data to different indexers. However, given

that the universal forwarder does not parse the data

in any way before forwarding it to a receiver,

content-based routing cannot be performed by

universal forwarders.

• Heavy Forwarder. A heavy forwarder is a full Splunk

Enterprise instance with some features disabled to

achieve a smaller computing-resources footprint.

Heavy forwarders parse data before forwarding it

to a receiver and can support content-based routing.

Unlike universal forwarders, heavy forwarders can

index data locally and can optionally be configured

to perform local searches and alerting.

Data routing is a key requirement of the MSaaS

architectural framework. A single end point in the

configuration can have multiple data sources and

generate multiple types of events. These events

may need to be routed to different islands or be

filtered out to eliminate the indexing of superfluous

events (see Figure 7). For example, certain security-

related events may be routed to indexers on an

MSaaS administrative island that provides security

monitoring for all managed islands, with other events

being routed to islands deployed for the customer-

specific use cases.

A more detailed, complete discussion of routing

implementation is beyond the scope of this document.

Although islands are separated from each other,

Figure 7. Forwarders are used to implement data routing.

their use cases can overlap and they may need to

share data or data sources. This can be achieved by

consolidating the islands that need to share data

into a larger island. However, this may not always be

possible if jurisdiction constraints or data segregation

requirements do not permit this sharing. In these

cases, data routing in the data collection tier can be

used to control the distribution of multiple copies of

the data in question to the appropriate islands. The

volume of data routed to each island is counted once

against each destination island’s license. Thus, events

sent to multiple indexers are counted multiple times,

once per destination indexer.

Data sources indexed by a given island are partitioned

into specific indexes based on security, ownership,

visibility constraints, and retention and resilience

requirements. Index design and routing is an

island-specific construct, and these attributes can

differ across islands depending on the use cases

implemented by each island.

Indexers

Customer-Specific Island Global AdministrativeIsland

Security Events

HeavyForwarder

Customer-SpecificEvents

WHITE PAPER


Namespace Best Practices Splunk Enterprise implementers and users define

and maintain a large number of named entities—

knowledge objects and their consolidation into

apps or add-ons in addition to a wide range of

deployment-related constructs such as roles, users

and indexes.

As discussed in the chapter “Develop naming

conventions for knowledge objects” in the Splunk

Enterprise Knowledge Manager Manual, Splunk’s

recommended best practice is that Splunk

implementers and users develop and enforce

naming conventions for knowledge objects. These

conventions facilitate the reuse and sharing of

knowledge objects across a single deployment and,

by extension, across multiple Splunk Enterprise

deployments.

In an MSaaS deployment, the significance of

maintaining a common naming convention and

namespace across the entire deployment is increased

significantly due to the fact that the deployment’s

requirements are likely to evolve in a way that can

require the consolidation of multiple instances

into a single instance. If naming conventions and a

common namespace are not strictly adhered to across

all named entities, such a consolidation becomes

difficult, at best, or even impossible without significant

rework and a high risk of failure and data loss.

Thus, conventions for the naming of all Splunk

entities—knowledge objects, indexes, sources, source

types, roles, users and anything else—must be defined

and strictly enforced to help ensure that a single

namespace is maintained across the entire MSaaS

deployment.

MSaaS Roles Security for the MSaaS framework is implemented

through the combined concepts of islands,

jurisdictions and roles. Standard authentication and

authorization services used by the enterprise, such

as LDAP or Active Directory, are typically utilized to

manage user access throughout the deployment.

Splunk Enterprise users are typically assigned to

roles, each of which is associated with a set of

capabilities. Splunk’s best practices define a set of

common roles that can be used to partition and

authorize users in a systematic way across multiple

deployments. Table 1 lists these roles and their

corresponding responsibilities.

Role

Administrator

Search Expert

Email User

User

Project Manager

Developer

Responsibility

Design and execute the architecture and deployment; manage users and provide support; administer OS, network and infrastructure

Develop searches and analyze results; help users resolve search-related issues and understand data sources

Request searches from search expert and receive reports by email

Run saved searches and utilize dashboards to gain domain-related insights

Define project goals and set requirements; mange projects and coordinate across departments

Customize and configure the Splunk Apps and Add-Ons and develop the site- and island-specific apps and add-ons as required; implement configuration file changes; customize and develop dashboards, scripted inputs, hooks to data sources, and external inputs; use Splunk APIs and scripting tools

Table 1. Common roles and responsibilities.

http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Developnamingconventionsforknowledgeobjecttitles

http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Developnamingconventionsforknowledgeobjecttitles

WHITE PAPER


ServiceAn IT service provided to one or more internal business units, including all applications that are managed separately in addition to wider infrastructure concepts that are provisioned to customers.

JurisdictionAn entity that defines a set of constraints on access to data contained in one or more islands.

MSaaS deployment server

A centralized system responsible for installing, configuring and updating the islands’ configurations.

In addition to the above roles, Splunk Enterprise

administrators can create custom roles to help assign

specific capabilities to users, as needed to meet

security requirements. For example, roles can be

defined with attributes such as the following:

• Default list of indexes to be searched

• Restricted list of indexes

• Restrictions on the data that can be searched

with a search filter

• Restrictions on the time window for which

searches are allowed

• Inherited roles or capabilities

Roles are defined for each island and are used to

enforce data access controls within a single island.

In the MSaaS architecture, a single global instance of

Splunk Enterprise’s configuration file is maintained.

This configuration file, which specifies the user roles

for each island, is centrally managed and propagated

to each island by the MSaaS Deployment Server. (For

a description of the MSaaS Deployment Server, see

the “Deployment” section on page 21, in the chapter

on Administration).

When a user logs into a specific island, that user’s

role within that island is determined from the local

copy of the configuration file and the user is assigned

the appropriate permissions and capabilities for that

island. A user’s role and corresponding permissions

can vary from island to island, enabling the

enforcement of flexible data segregation and security

policies. For more information on users and role-

based access control, see the section “Use access

control to secure Splunk data” in the Securing Splunk

Enterprise manual.

Component

Bridge

Definition

A type of island that consists of search tier components only and no indexing or data collection capabilities; used to provide search access to data on two or more islands.

IslandA single end-to-end Splunk Enterprise deployment with indexing cluster, search head cluster, and optional license, deployment server, and job servers.

Primary island Forwarder attribute: An island whose deployment server configures the forwarder.

Service attributesProperties such as replication factor, search factor, disaster recovery RPO and RTO, security, backup, storage tier, performance, retention plan, and daily volume.

TenantAn independent user of Splunk Enterprise. Normally an individual business unit or an external customer.

Summary: Architectural Components Table 2 contains a summary list of the primary concepts and components introduced in the MSaaS logical

architecture. For a complete list of Splunk terms, see the online Splexicon.

Table 2. Concepts introduced in the MSaaS architectural framework.

http://docs.splunk.com/Documentation/Splunk/latest/Security/UseaccesscontroltosecureSplunkdata




http://docs.splunk.com/Splexicon

WHITE PAPER


MSaaS Physical Architecture

The island is the basic deployment unit for the MSaaS

architecture. Each island is self-contained and is

deployed with the apps and add-ons it needs for its

use cases. It includes any number of search heads,

indexers, job servers and cluster master nodes,

together with an optional deployment server or

license master. Islands should also include redundant

components, if needed to meet their resilience

requirements.

The following sections describe the physical

architecture of the search, indexing, and data

collection tiers.

Search Tier In the MSaaS architectural framework, the search tier

of each island can contain both interactive search

heads and non-interactive jobs servers. Use of

search head clustering is recommended to provide

search tier resilience and scalability (see Figure 8).

Configuring a search head cluster with multiple

search heads enables these cluster members to run

searches across a common set of indexers, referred to

as the search head cluster’s search peers.

Figure 8. Search head clustering provides resilience for the search tier.

In a search head cluster, one cluster member is

designated the captain. The captain is a search

head, no different from the others, but it is

responsible for coordinating job scheduling and

replication activities between the cluster members.

Any search head cluster member can perform the

role of captain, but there is just one captain at any

time. When a captain fails, another search head

is elected as captain, providing resiliency for the

search cluster.

In addition to the search heads, each search

head cluster also contains a deployer (not to be

confused with the deployment server). This Splunk

Enterprise component distributes apps, add-ons

and other configuration updates to the cluster

members. In addition, a load balancer can be used

with the search head cluster for resilience and

simplicity of access, as described in the “Search

head clustering architecture” section of the Splunk

Enterprise Distributed Search documentation.

Load Balancer

Search Head Cluster

ClusterMembers

Search Peers

Deployer

Captain

http://docs.splunk.com/Documentation/Splunk/6.2.4/DistSearch/SHCarchitecture

http://docs.splunk.com/Documentation/Splunk/6.2.4/DistSearch/SHCarchitecture

WHITE PAPER


Indexing Tier In the MSaaS architectural framework, the indexing

tier consists of one or more indexers with a nominally

optional cluster master node. Each indexer is assigned

to a single specific island to ensure data segregation

and to facilitate licensing.

Four basic indexing tier configurations provide

a range of data replication and disaster recovery

options, with each island able to select the

configuration model that best suits its needs.

• Non-replicated, single site

• Replicated, single site

• Non-replicated, multiple sites

• Replicated, multiple sites

Clustering and multisite clustering are supported in

the MSaaS architectural framework to provide data

replication and disaster recovery capabilities for an

island. Clustering provides replication of data, with

two or more indexers maintaining copies of the

indexed data. Multisite clustering provides disaster

recovery capabilities by deploying two or more

indexers at geographically separate sites. Configuring

each site with a full copy of data helps protect against

a site-wide failure.

As mentioned previously, if the island’s resilience

requirements do not dictate redundancy or disaster

recovery, the island’s indexing tier should still be

configured as an atrophied site-aware cluster

consisting of a single indexer and a cluster master

node. This configuration facilitates conversion of

the indexing tier to include redundancy or disaster

recovery if these are later required.

Each indexing cluster can include a standby cluster

master node to provide failover protection. Standby

cluster master nodes are configured with copies

of the primary cluster master node’s static state

prior to failover. Whenever changes are made to the

cluster master node’s configuration, the changes

must propagate to the standby cluster master node

to maintain the identical state. In addition, the island

must be configured to locate the replacement cluster

master node after a failure occurs. A load balancer,

DNS-based failover, or another technique can

be used to facilitate failover between the cluster

master node and the standby cluster master node,

enabling the cluster’s indexing peer nodes and the

search heads to locate the standby cluster master

node after failover and recognize it as the now-

current master. No loss of functionality is expected

during the failover process, because the indexing

cluster continues to function normally without a

functioning cluster master node until the standby

cluster master node is fully functional.

Data Collection Tier Each data collection dataflow in the MSaaS

architectural framework terminates with a Splunk

Enterprise forwarder that transfers data to an

indexer. Each forwarder is either a universal

forwarder or a heavy forwarder, depending on

whether the data it handles requires content-

based processing to reach its correct destination.

If content-based processing is required, a heavy

forwarder is used. By default, universal forwarders

are used.

Data sources in the MSaaS architecture can be

divided into two classes: devices and general-

purpose systems. Each class of data source

is accommodated differently in the MSaaS

architectural framework.

• Devices. Devices are those data sources that

generate log messages, have little or no local

storage capacity, and have no general-purpose

computing capability to run additional

software to support log collection. These

devices are normally configured to send their

data to forwarders that reside on general-

purpose systems that then forward the

information to the designated indexers, either

directly or through intermediate forwarders.

In addition to the Splunk Enterprise forwarder,

if required, these general-purpose systems can

run additional software to collect the data and

improve data safety. For example, in the case of

syslog data, a syslog server, such as syslog-ng,

can be used. Similarly, if the data is in the form

WHITE PAPER


of files residing on the data source and the data

source supports a file transfer protocol, a compatible

software client can be used to pull the data file from

the data source.

• General-purpose systems. General-purpose

systems often have storage and computing

capacity to run additional software. If they do—IT

policies permitting—these systems run a Splunk

Enterprise forwarder to collect data. It forwards

this data to either an intermediate forwarder

running on another general-purpose system, which

is usually dedicated to this task, or directly to the

indexing tier.

A Splunk Enterprise configuration file is used to

define how forwarders send data on to receivers. A

single forwarder can have multiple configuration files,

and the settings from these multiple configuration

files are combined to determine event routing. In the

MSaaS architectural framework, custom configuration

files are created to handle the various data routing

and data segregation requirements for the islands.

These configuration files can be defined globally or

for each jurisdiction or any collection of islands. To

maintain consistency, these files are stored in the

MSaaS VCS. If changes are required, these edits can

be made centrally and then pushed to the forwarders.

Administration

Administrative tasks for an MSaaS deployment can

be classified into two primary categories: deploying

new Splunk Enterprise instances, and managing

and monitoring the deployed instances. VCS and

configuration management toolsets are needed to

develop, maintain and run an MSaaS implementation.

In addition, a dedicated administrative island is

recommended to oversee and manage all other

islands in the deployment and to maintain a site-wide

security posture.

The following sections discuss deployment and

ongoing management of an MSaaS implementation,

along with related licensing, chargeback and

credential management for the deployment.

Deployment

The MSaaS framework enables the fully automated

deployment of Splunk islands and defines a

centralized MSaaS deployment server consisting of

repositories of binaries and MSaaS-specific content,

configuration management tools, and a VCS to

maintain the configuration text files (see Figure 9).

WHITE PAPER


Figure 9. The MSaaS deployment server maintains binaries in a package manager and configuration files in a VCS.

Configuration management tools are used to

automate the deployment of new MSaaS islands and

to roll out updates to existing islands. The MSaaS

framework allows a choice of any commercial,

enterprise-class configuration management system.

For example, a single configuration management

system that supports both Linux and Windows

systems can be used. Alternatively, two separate

configuration management systems—one for Linux

and one for Windows—can be used, at the cost of

additional management complexity.

A single, centralized MSaaS deployment server (MDS)

is used to manage the deployment of islands. This

system is responsible for distributing and installing

the Splunk Enterprise binaries, configuration files, and

apps, and it also maintains the static configuration

of each Splunk Enterprise component. The MDS also

updates binaries for each island when necessary.

Each island can contain a Splunk Enterprise

deployment server (DS) of its own to process

configuration updates for all Splunk Enterprise

components on that island. With this deployment

approach, the centralized MDS deploys configuration

files to the DS on each island, which then propagates

these files to the Splunk Enterprise components on

the island and the forwarders that island controls. If

the island does not include its own DS, the MDS must

maintain each individual component on that island.

...Standard

Island 1–Specific

Customized

Island n–Specific

Enterprise

Apps, Add-Ons and Configuration Files

Configuration ManagementSystem Files

ConfigurationManagement

SystemUniversalForwarderVersions

splunkdVersions

Binary Package ManagerVersion Control System

MSaaS Deployment Server

IntermediateForwarders

Forwarders

Customer IslandsAdministration andSecurity Island

Operational SourceCode, Scripts andConfiguration Files

Routing andTransformation

Table

Bridges

OperationalBinaries

WHITE PAPER


The MDS server maintains two primary repositories:

one containing MSaaS-specific content and the other

containing binaries. The MSaaS-specific content

is stored in a VCS to maintain consistency across

deployments, enable version tracking and control,

and provide an audit trail.

The following entities needed for an MSaaS

deployment are typically maintained in the VCS:

• Splunk Apps and Add-Ons. Standard and

customized apps or add-ons, or those specific to a

subset of the islands.

• Operational source code, scripts andconfiguration files. Items that help maintain the

deployment, implement customized administration

tasks or perform other operational tasks.

• Routing and transformation tables. Configuration

files that control the routing and data segregation

requirements for the islands.

• Configuration management system files. Files

specific to the configuration management

system that is used, such as Chef recipes, Ansible

playbooks or Puppet language scripts.

• Island-specific configuration files. Includes

pointers to island-specific applications and other

applications that an island uses.

The MDS maintains the deployed binaries using one

or more package management tools, such as the YUM

package-management utility or Windows Installer. In

addition to the Splunk binary packages, this repository

should include any operational binaries that are needed

to implement the island’s functional requirements, such

as a Syslog server implementation, file transfer software

or file integrity monitoring software.

The MSaaS framework is platform-agnostic, and

it imposes no requirements on the use of specific

configuration management, package management

or VCS software. Any enterprise-class VCS can be

used to maintain the consistency of the configuration

files. Similarly, any configuration management system

or package management system with sufficient

capabilities can be used to automate the island

deployment. This flexibility enables enterprises to

capitalize on their existing in-house expertise and

reduces their learning curve when implementing an

MSaaS deployment.

Management and Monitoring

Deploying a separate MSaaS administrative island is

recommended in the MSaaS architectural framework.

This administrative island includes a Splunk Enterprise

instance and is used primarily to monitor and maintain

all the other MSaaS islands in the deployment, including

its global security posture. If indexing-volume-based

chargeback is required, the administrative island

can track the usage of each island and generate the

appropriate reports for this purpose.

In other respects, ongoing management and

monitoring of an MSaaS deployment is similar to that

of any Splunk Enterprise deployment. Administrators

can use the command line or Splunk Web interfaces,

and they continue to monitor the standard

configuration files as for a traditional distributed

Splunk Enterprise deployment. In addition, custom

apps can be built to perform any site-specific tasks

required for an individual deployment.

The monitoring console (MC) can play a part in

managing an MSaaS deployment. The MC provides

insights into various performance and usage statistics

for the Splunk Enterprise components deployed across

the multiple islands. The MC can be configured for

distributed mode, enabling administrators to log into

one instance and view performance information for

all instances in the deployment. It is also possible to

maintain distinct views for each island by using MC with

tagged groups.

In addition, the implementation of an MSaaS

deployment should include developing an MSaaS

administration app to consolidate information that is

not covered by the MC.

For example, this app could track the activities

of the MDS, report on per-island indexing for the

purpose of chargeback, and track the health of the

WHITE PAPER


different islands.

Licensing and Chargeback Licensing for the MSaaS architectural framework

uses the standard Splunk Enterprise licensing

model with license pools (see Figure 10). Licenses

are aggregated into license stacks, managed by

a license master and each pool provides the total

licensing volume for a single island. Additional

licenses can be added to the license master’s

managed licenses dynamically, increasing capacity

as needed. Similarly, license pools can be increased

as required.

Figure 10. Each island is allocated a license pool.

In the MSaaS architectural framework, the license

is centrally purchased and controlled. While in

principle a local license master can be deployed

on each island, the MSaaS architectural framework

recommends that islands rely on a global license

master. Each island is allocated a specific license

pool from a license stack managed by the global

license master, based on its indexing volume

requirements. The license usage can form the basis

for a number of chargeback schemes that can

depend on the license reserved or utilized by each

island, or a combination of both.

Credentials The credentials for each island can be maintained

by the enterprise’s central identity management

system. The allocation of credentials for each role in

each island is carried out as part of the automated

deployment process. A detailed discussion of this

subject is beyond the scope of this paper.

Example Implementation

Schuberg Philis, a European business technology

company focused on supporting mission-critical

applications for its customers, implemented the

MSaaS framework described in this paper. It was

facing growing data volumes and a need for multiple

Splunk Enterprise instances implemented as a flexible

and adaptable solution that could handle its workload

and rapidly deploy new instances. The primary

objectives were a scalable and high-performance

system; high availability and resilience for the

instances; data security, with isolation between

customers and RBAC to control access to data; and a

flexible solution with automated deployment.

...

...

License

Indexer ClusterIndexer Cluster

LicensePool 1

LicensePool n

Island 1 Island n

License Master

WHITE PAPER


Functional requirements of the MSaaS

implementation included:

• Automated deployment and update of multiple

Splunk Enterprise instances with packaged

binaries, scripts and configuration files.

• Flexible, scalable design, with each instance

customized as needed.

- Instances scalable from very small to large,

as necessaray.

- Archiving, backup, and resilience requirements

defined per customer; no single point of

failure dual-datacenter configurations, indexing

tier resilience and storage resilience.

- Indexing tier replication-based data protection.

• Strict data segregation, with no shared resources

nor shared information between individual

customer instances. Cross-jurisdiction access

possible only when explicitly enabled, for functions

such as security.

• Jurisdiction hierarchy supported, with the

ability for one data jurisdiction to include other

jurisdictions.

• Multiple licensing models, including central license

masters and a license pool per island or

independent licenses managed by customer

islands if needed, to meet individual customer

requirements.

• Credentials maintained in the enterprise

identity management system, using OpenAM

for authentication and single sign-on and Active

Directory for authorization of users.

Architecture

Figure 11 shows the architecture implemented by

Schuberg Philis. The Apache CloudStack cloud

computing platform provides virtualization capabilities

and creates the underlying cloud infrastructure.

Primary components include multiple islands, each

implementing a full Splunk Enterprise deployment for

a single customer; a Schuberg Philis internal island;

a bridge used for security information and event

management (SIEM); a global license master; and

an Active Directory server. Nagios IT infrastructure

monitoring solution monitors the IT infrastructure. The

OpenAM open-source authentication, authorization,

entitlement and federation software platform, acts as

an authentication proxy to provide secure, single sign-

on access to the systems for clients.

Supporting systems include Chef systems and cloud

infrastructure automation framework for configuration

management, the Yum automatic updater and

package installer and remover for RPM systems for

package management, the GitHub Enterprise version

control repository, an orchestration server running

internally-developed orchestration scripts, and a file

server serving up artifacts. All change management

and version control are automated using GitHub

Enterprise and configuration management is

automated using Chef. The artifact server and Yum

repository store and distribute binaries, and the

orchestration server runs scripts that facilitate the

deployment process.

Of note, the following components are not

implemented in this example: a Splunk Enterprise

deployment server on each island and dedicated job

servers. Schuberg Philis does not require dedicated

job servers for their deployment. And the deployment

servers are not needed, because deployment tasks

are handled by the Chef software, which deploys

each component.

WHITE PAPER


Figure 11. MSaaS architecture implemented at Schuberg Philis.

Schuberg PhilisInternal Island

...

... ...

Support Systems

Deploy

Customer Islands

IdentifyInfrastructureAlerts

InfrastructureMonitor

IdentityManagement

Server

Self-LicensedInternal Island

SecurityBridge

Single Sign-OnAuthentication

Proxy

ConfigurationManagement

System

PackageRepository

VersionControlSystem

OrchestrationServer

ArtifactServer

GlobalLicenseMaster

WHITE PAPER


This architecture was deployed in a dual data

center environment. Both the replication factor

and search factor are two (that is, two copies of

searchable data are maintained), and data backups

are not implemented. Search heads are configured

in an active-standby configuration for resilience

in the event of a search head failure. No routing is

implemented in the current deployment. Universal

forwarders are used, rather than heavy forwarders,

and each forwarder sends data to a single island. SSL

certificates are used to ensure the security of data

sent from forwarders to customer islands.

Authentication and Authorization Single sign-on is implemented through the OpenAM

authenticating proxy, which enforces certificate-

based, dual-factor authentication of users to securely

control external access to the islands. OpenAM

interfaces into the Schuberg Philis’ Active Directory

server. When a user logs in, OpenAM authenticates

the user and then sends the user name to the

Active Directory server. The Active Directory server

maintains a per-island mapping between the Active

Directory groups and the Splunk Enterprise roles.

All credentials for the islands are managed through

OpenAM and Active Directory; no user credentials are

stored on the islands.

This approach enhances scalability through ease of

administration while maintaining controlled access

and segregation. Administrators sign on once and

are authorized to access a specific island. They can

then transparently switch between islands—and

be authorized for each island they visited—without

requiring re-authentication. This design makes

administration of a multiple-island deployment very

easy, and the approach seamlessly scales as the

number of islands increases.

Licensing Two centralized license masters are deployed: one

to maintain the managed service provider (MSP)

licenses for Schuberg Philis, and one to maintain

their not-for-resale (NFR) license. The MSP licenses

are used for the customer islands, using a separate

license pool per island. The NFR license is used on

an island dedicated to the company’s infrastructure,

employee training and proofs of concepts.

Each individual customer-island is assigned its

own license pool to ensure license violations from

that customer-island do not affect the operation

of other islands. In addition, one customer is using

the full features of Splunk Enterprise. Because the

MSP license does not allow end-user customers to

build dashboards or have full access to the Splunk

Enterprise interface, it is necessary to deploy a

separate license master on that customer’s island.

Switching to a private-license model for this customer

was trivial; implementation consisted of simply

pointing that customer’s island to its own license

master rather than to the global license master.

Monitoring and Management Nagios software and the Schuberg Philis internal

island, which hosts a Splunk Enterprise dashboard,

tracks license utilization and indexing, and monitors

the Splunk Enterprise deployment. In addition to

managing the MSaaS infrastructure, the internal

island ingests logs from the global license manager.

The license pools for the customer islands are one

component that requires careful monitoring. When

a license violation occurs, the Nagios software raises

an alert. Initially, these alerts occurred often, leading

Schuberg Philis to proactively monitor the license

usage for each customer.

To help prevent indexes from growing out of bounds

and exhausting the available storage, a maximum

index-size is set to limit storage utilization. 250GB

of storage is assigned to each indexer for four

60GB indexes and a minimal 1MB for the default

index, which is normally not used. A volume-based

retention policy is used and if the indexing quota is

exceeded, older events are automatically dropped.

This approach scales well, as retention volume can be

controlled centrally for easy administration of multi-

island deployments. A default storage allocation per

indexer is centrally specified, and it customized to

accommodate non-standard customer requirements

when deploying a new island.

WHITE PAPER


Automated Provisioning

When a new customer requests the provisioning of

an island, the service attributes are defined and the

automated provisioning process is triggered. The

attributes include the resilience and disaster recovery

requirements, licensing and storage volume, island

specific SSL certificates, retention requirements,

and data sources and data segregation requirements

for this customer. The automated provisioning

process then implements the specified attributes

and deploys the island.

An internally-developed orchestration script is used to

implement the automated provisioning of islands. This

script uses Apache CloudStack APIs, Chef cookbooks

and the Yum repository to deploy and configure the

software components for the infrastructure. Different

Splunk Apps are deployed on each island, based

on the service attributes of that island. Schuberg

Philis have evaluated the open-source Terraform

orchestrator to replace the internally developed

orchestration scripts and have found it suitable.

A hierarchy of Chef cookbooks is used for island

configuration. A global Splunk Enterprise cookbook

is used to set default parameters for all islands.

In addition, island-specific cookbooks are used

to override the default parameters, as needed, to

meet any specific configuration requirements for a

particular island.

Specifically, the orchestration script and cookbooks

automate the following tasks:

1. Create the island

2. Instantiate the server

3. Install and configure the Splunk

Enterprise software

4. Configure the cluster

5. Create and manage the data disk and indexes

6. Configure data replication between search heads

7. Configure security (firewall rules, SSL setup)

8. Configure monitoring (monitoring of processes,

connectivity, cluster health and Splunk

Enterprise alerts)

9. Configure single sign-on

10. Install Splunk Enterprise Apps

Change management and patching are also

automated in the Schuberg Philis deployment.

GitHub Enterprise version control software handles

all version control. When the Splunk Enterprise

software requires upgrading, new versions of the

appropriate binaries are deployed to the Yum

repository. The artifact server stages the files, and

the new binaries are deployed to the islands after

applying operating system patches and any Splunk

Enterprise software patches by the orchestration

server. This approach provides central management

for software upgrades and facilitates the automatic

upgrade of customer islands.

Post-implementation Conclusions

Implementing the MSaaS architecture at Schuberg

Philis was a success, with 15 deployed customer

islands (as of publication). Before embarking on

the MSaaS based design and implementation, the

company was confident of the Splunk Enterprise

software capabilities but required a strategy to help it

efficiently deploy and manage multiple instances. The

MSaaS-based Splunk Enterprise deployment enabled

Schuberg Philis to successfully provide support for

mission-critical applications for multiple customers

without having to allocate significant resources for

its ongoing maintenance, and the manageability,

performance and scalability of the solution met

their requirements. The dynamic schema-on-the-

fly functionality provided high performance for

search queries, and the superior reporting, graphing,

and search capabilities enabled Schuberg Philis to

efficiently monitor applications and support

its customers.

The biggest benefit of the MSaaS architecture

deployed at Schuberg Philis is the huge efficiency

achieved by the automatic deployment of new Splunk

Enterprise instances for customers. Deployment time

WHITE PAPER


takes approximately one hour per island. In addition,

by having a scripted deployment, the potential for

human errors and misconfigurations are minimal.

The use of a VCS provides a valuable audit trail for

tracking new deployments, upgrades and changes to

configurations. This flexible solution enables Schuberg

Philis to efficiently provide a scalable solution for a

growing number of customers with very little Splunk-

related marginal overhead per customer.

The MSaaS architecture enables Schuberg Philis

to track and understand per-customer usage.

Although they are currently not implementing per-

customer utilization chargeback, they can easily

add this functionality at any time if their business

requirements change.

Schuberg Philis has deployed a bridge for security

information and event management (SIEM) and

developed an app to monitor its security posture. This

bridge provides visibility into events on all customer

islands, and it has a proven ability to detect security

breaches. The security app can also be deployed to

customer islands to help maintain the islands’ security.

Currently, the Schuberg Philis implementation does

not use complex routing. No heavy forwarders are

used and all the universal forwarders send data to a

single island. However, certain potential use cases are

leading them to consider adding heavy forwarders to

their infrastructure. In one case they are considering

using Splunk DB Connect to integrate a customer’s

structured data sources with the customer’s island.

In another case, a customer has sensitive information

that would need to be filtered out before sending

the data from the customer’s site to their island.

The MSaaS architecture inherently supports these

use cases.

Key Observations A key observation from Schuberg Philis is the value

of a central user directory and single sign-on

capabilities to provide management at scale. These

features are a prerequisite in enabling administrators

to securely access the islands and transparently

switch between customer islands while maintaining

authorization control.

In addition, use of a cloud virtualization framework

with API access is a prerequisite for scalability and

dynamic orchestration. Physical provisioning of

the infrastructure would be too difficult and time

consuming to enable the expeditious deployment

of new islands.

One final key observation from Schuberg Philis is

the importance of investing adequately in training

the system’s implementers and in designing the

implementation and its supporting environment

before embarking on an implementation project of

this scale. While Splunk Enterprise is relatively easy

to install and use, distributed architecture design and

implementation is non-trivial and any poor design

decisions would be multiplied and spread with each

new deployed instance. Training can help reduce the

risk, speed the deployment, and assist in getting the

administrative team up and running efficiently. In

addition to expertise with Splunk Enterprise software,

expertise with the software products used to automate

the deployment—including the VCS, configuration

management system, package management tools and

any cloud infrastructure software—is also essential.

Additional Resources

This document provides the conceptual framework

for implementing the MSaaS architecture. Once

implemented, this architecture provides significant

savings in Splunk Enterprise deployment and

administration overhead. However, designing,

deploying and administering an MSaaS

implementation is a significant undertaking. Splunk

offers a variety of support and training options to help

ensure an efficient and successful implementation.

The Splunk Professional Services team has experience

with Splunk Enterprise deployments of all sizes, and

this team of experts can provide a range of services

to meet the specific needs of each organization.

For example, they can provide architectural design

guidance and a technical assessment of the proposed

Splunk environment, working along with local Splunk

administrators. If preferred, the Splunk Professional

Services team can take the lead on project

management, monitoring and managing the project.

WHITE PAPER

Download Splunk for Free or explore the online sandbox. Whether cloud, on-premises, or for large or small teams,

Splunk has a deployment model that will fit your needs. Learn more.

© 2017 Splunk Inc. All rights reserved. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. WP-Splunk-MSaaS-Arch-104

www.splunk.comLearn more: www.splunk.com/asksales

To help ensure a successful deployment and the

continued effective administration of the MSaaS

architecture, local Splunk administrators require a

thorough understanding of the technical concepts

along with practical skills. Splunk Education Services

offers direct, focused training programs that set the

stage for long-term success and efficiency. Investing

in education builds productivity and competence, and

it reduces the risk of problems caused by user error.

Splunk Education Services offers flexible training

options including instructor-led virtual classes, in-

person classes and self-paced eLearning. In addition,

Splunk Education Services offers dedicated on-site

classes and can create a custom curriculum to match

the specific training requirements of an organization.

Summary

The MSaaS conceptual architecture described in

this paper supports multitenant Splunk Enterprise

deployment as a service. This scalable multi-instance

approach features individual islands—complete, end-

to-end Splunk Enterprise deployments—that provide

data segregation for tenants while allowing global

visibility as needed for a range of functions including

security. An automated, service-request model

quickly provisions new Splunk Enterprise instances as

needed. Integration with configuration management

and VCS tools provides scalability and consistent

implementation for large installations, reducing

deployment and upgrade risks and simplifying

ongoing administration. This cost-efficient approach

can reduce the marginal cost of each additional

deployed instance and reduce the total cost of

ownership for large multitenant deployments.

http://www.splunk.com/en_us/download.html

http://www.splunk.com

http://www.splunk.com

http://www.splunk.com/asksales

multiple splunk as a service (msaas)

Documents