supporangand upgradinglarge deployments* · pdf fileexpedia’sstoryandnextsteps...

Copyright © 2014 Splunk Inc.

Erik Andresen System Engineer II Expedia, Inc.

SupporAng and Upgrading Large Deployments

Disclaimer

2

During the course of this presentaAon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauAon you that such statements reflect our current expectaAons and

esAmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,

please review our filings with the SEC. The forward-‐looking statements made in the this presentaAon are being made as of the Ame and date of its live presentaAon. If reviewed aPer its live presentaAon, this presentaAon may not contain current or accurate informaAon. We do not assume any obligaAon to update any forward-‐looking statements we may make. In addiAon, any informaAon about our roadmap outlines our general product direcAon and is subject to change at any Ame without noAce. It is for informaAonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaAon either to develop the features or funcAonality described or to

include any such feature or funcAonality in a future release.

PresentaAon Agenda

!   About the Speaker !   Expedia, Inc.: A Global Enterprise !   Components and Symbols

!   StarAng Out Small – Single Server !   SeparaAng FuncAons – MulAple Tiers

!   Expanding Horizontal – MulAple Servers !   Expanding Datacenters – MulA-‐Datacenter Instances !   Headed to the Cloud – AWS Instances

!   SeparaAng Teams – Pod Instances !   Up Next for Expedia’s Splunk Environment 3

About the Speaker

4

About the Speaker

5

!   System Engineer II, Monitoring Tools Team, Expedia, Inc. –  As Splunk Administrator, helps to manage the enAre Splunk Environment –  The team provides monitoring, analyAcs, and automaAon soluAons for

Expedia –  Has worked for Expedia since September 2012

!   Previously contracted at MicrosoP –  OperaAons Engineer with the Xbox OperaAons Center (XOC) –  Worked for four years in various roles from Tier I to Tier 2.5 – SME

!   Graduated from and worked at the University of Washington –  Business AdministraAon Degree – Management InformaAon Systems, 2008 –  Windows System Administrator, Electrical Engineering Department

Expedia, Inc.: A Global Enterprise

6


7


8 1Includes eLong bookable properties

2comScore Worldwide Average Monthly UV data, TTM 2Q14


9

Components and Symbols

10

Splunk Components Splunk FuncAonaliAes User Symbols LocaAons Symbols


11

Icon Descrip,on

Splunk Server – A server that acts as both a search head and an indexer that allows users to search, aggregate, and manipulate machine data that is locally stored.

Search Head – A dedicated Splunk instance uAlized by users to search, aggregate, and manipulate machine data that is stored on Splunk indexer(s).

Indexer – A dedicated Splunk instances that holds machine data in a proprietary map-‐reduce repository and enables it to be searched/returned to the search head.

Forwarder – A package installed onto a client server that sends data into Splunk in a format that allows it to be indexed and subsequently searched by a user.

Splunk Components


12

Icon Descrip,on

Deployment Server – A dedicated Splunk instance uAlized by the operaAon team to deploy new and update exisAng Splunk based apps.

Licensing Server – A dedicated or combined Splunk instance that hosts the licensing file(s) that search heads and Indexers connect to report their usage.

Master Cluster Node – A dedicated Splunk instance that manages a cluster, handles were searches are directed and how data is indexed onto the Indexers.

Job Server – A dedicated Splunk instance that runs scheduled searches only and outputs the results to summary indexes, alerts, and reports.

Splunk Components


13

Splunk FuncAonaliAes

Icon Descrip,on

Search – An ad-‐hoc search, usually started by a user through the Splunk UI or API, that is started on a search head and runs against the indexer(s).

Saved Search – A search, either scheduled or not scheduled, that is saved on a search head and runs against the indexer(s).

Alert – A saved and scheduled search that is on a search head and runs against the Indexer(s), which is setup to alert, via email, scripts, etc. when a criteria is met.

Dashboard – A visual display that shows the results from an inline or saved search through bar charts, line charts, radial gauges, single values, etc.

App and Add-‐On – A package that is installed onto a search head or indexer that provides addiAonal funcAonaliAes and features on top of the base install.


14


Icon Descrip,on

Indexes – A locaAon that stores the events sent from the forwarders to the indexers that are normally stored for short Ameframes.

Summary Indexes – A locaAon that stores summary events that are retrieved from the indexers that are normally stored for longer Ameframes.

Accelerated Reports – A data backed summary created off of an accelerated search that automaAcally creates a data stored alongside the buckets in the original index.


15


Icon Descrip,on

Log File Input – An input from a server, normal in the form of a log or text file, that indexes events into a Splunk instance

Network Input – An input from networking devices, normally through Syslog streaming, that indexes events into a Splunk instance

Scripted Input – An input that pulls informaAon from a source, including log files, databases, etc., that indexes events into a Splunk instance


16

User Symbols

Icon Descrip,on

Read-‐Only Users – A Splunk role that grants minimal access to run UI-‐based searches, save private searches, and create private dashboards

Power Users – A Splunk role that adds to the permissions of the read-‐only users role to include more concurrent searches and longer search Ameframes

Power Plus Users – A Splunk role that enables addiAonal like real-‐Ame and accelerated searches, public dashboards, etc. features to the other roles

Splunk Admin – A Splunk role that has full admin rights and capabiliAes within the Splunk instances

Splunk Support – Technically not a Splunk role, instead a group of people that work for Splunk and provide product support


17

LocaAons Symbols

Icon Descrip,on

Datacenter – A Splunk environment that is locally hosted within a company owned or leased locaAon

Cloud – A Splunk environment that is remotely hosted within a cloud-‐based locaAon like Amazon Web Services (AWS)

Office Building – A locaAon were employees work and interact with and support a Splunk environment

Network – ConnecAons between Splunk servers and forwarders, datacenters, the cloud, and users

StarAng Out Small – Single Server SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps

StarAng Out Small – Single Server

19

!   A single installaAon of Splunk, running either on a physical server or virtual machine !   Normally no customizaAon of user access roles, parsing sehngs, or other

configuraAons

!   Due to simple need, normally running a default installaAon is all that is needed

SecAon Overview


20

Data Tier

Search and Indexing Tier

Environment Diagrams Splunk FuncAonality


21

Data Tier

Search and Indexing Tier

Environment Diagrams Splunk Roles


22

Drawbacks •  Limited by the resources (cpu,

memory, storage, etc.) of a single physical server or virtual machine

•  Cannot grow horizontal or verAcally without standing up a separate instance

•  Everybody’s search and indexing load is on the same server

•  No way to take parAal outages for maintenances or refreshes

Benefits •  Simple to setup and manage

configuraAons •  Ideal for proof of concepts and

demos of features and funcAonaliAes

•  Ideal for local installs (laptop and desktop) to complete tesAng and troubleshooAng

•  InstallaAon can be done on a physical server or virtual machine


23

Expedia’s Story: !   Single server instances were primarily used as a proof of concept in the early days

of using Splunk

!   There are a couple single server instances of Splunk running within AWS for producAon and lab uses

!   Single server instances are also used on our laptops to test configuraAons and troubleshooAng issues with logs formahng

Up Next: !   The first “ProducAon” instance of Splunk was setup as mulAple Aer environment.

This was due to the planning esAmates to index two to three terabytes a day

Expedia’s Story and Next Steps

SeparaAng FuncAons – MulAple Tiers SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps

SeparaAng FuncAons – MulAple Tiers

25

!   A Splunk installaAon is broken up into two parts, searching and indexing, that are broken up verAcally into two Aers

!   Normally referred to has a three Aer environment, which is made up of the search and indexing Aers, with the third Aer being the data or forwarder Aer.

!   The search Aer consists of Splunk search heads and the indexing Aer consists of Splunk indexers

!   Each Aer consists of a single server with similar hardware specificaAons, except for more storage space in the server acAng as an Indexer

SecAon Overview


26

Environment Diagrams

Data Tier

Indexing Tier

Search Tier

Splunk FuncAonality


27


Data Tier

Indexing Tier

Search Tier

Splunk Roles


28

Drawbacks •  SAll limited by the resources (cpu,

memory, storage, etc.) of a single physical server or virtual machine for each Aer

•  Cannot grow horizontal without standing up a separate instance

•  Everybody’s search and indexing load is on the same set of servers

•  No way to take parAal outages for maintenances or refreshes

Benefits •  SAll prely simple to setup and

manage configuraAons •  Break up the funcAonaliAes that

could impact each other •  Provides addiAonal security as

access to the indexer can be locked down to admins only

•  InstallaAon can be done on a physical server or virtual machine


29

Expedia’s Story: !   Several mulAple Aer instances, with single servers in each Aer, for the first year or

so of using Splunk !   There are a couple mulAple Aer instances, with single servers in each Aer, of Splunk

running within AWS for producAon and lab uses

Up Next: !   APer a handful of teams had successfully stood up several mulAple Aer instances,

they were combined together into a single “ProducAon” instance. !   Due to the combined search and index load, we also started expanding both Aers

horizontally


Expanding Horizontal – MulAple Servers

SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps


31

!   The two Aers, search and indexing Aers, are now expanded horizontally with addiAonal installaAons of Splunk

!   Each Aer can consists of the same or different number of servers depending on where the demand is coming

!   If search load is heavier then index load, add addiAonal search heads into the search Aer

!   If index load is heavier then search load, add addiAonal indexers into the indexing Aer

!   Normally a good Ame to start customizing user access roles, parsing sehngs, or other configuraAons

!   Normally a good Ame to start looking at deploying a separate license and deployment server(s) to handle the addiAonal Splunk servers and forwarders

SecAon Overview


32


Data Tier

Indexing Tier

Search Tier

Splunk FuncAonality


33


Data Tier

Indexing Tier

Search Tier Splunk Infrastructure Tier


34


Data Tier

Indexing Tier

Search Tier

Splunk Roles


35

Drawbacks •  InstallaAon recommended to be

done only on a physical servers •  Setup and management becomes

more complex with mulAple servers in each Aer

•  Everybody’s search and indexing load is sAll on the same set of servers

Benefits •  SAll a prely simple to setup and

manage configuraAons •  Expansion of one or both Aers to

beler support load •  Considerably less limited by the

resources (cpu, memory, storage, etc.) as each Aer can be expanded as needed

•  ParAal outage can be done for maintenances or refreshes

•  Start taking advantage of Splunk’s other funcAonaliAes (deployment, license, heavy forwarder, etc.)


36

Expedia’s Story: !   Three major instances that are both mulAple Aers and servers including our

producAon, PPE/DR, and lab environments !   In addiAon to our main instances, we have three other small mulAple Aers and

servers instances hosted within our datacenters !   There is also a Splunk instances within AWS that is running with mulAple indexers

that are separate from a single search head

Up Next: !   With several team moving towards a mulA-‐datacenter setup for their services,

worked started to update our environment to allow cross-‐site availability and searching


Expanding Datacenters – MulA-‐Datacenter Instances SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps

Expanding Datacenters – MulA-‐Datacenter Instances

38

!   A mulA-‐Aer and server instance is setup within two or more separate datacenters that have customer facing service running in them

!   It is recommended that the forwarders only send data to the indexing Aer within their datacenter

!   Once the data is indexed, you can either use index and forward or the new mulAsite clustering feature to send across mulAple sites

!   Setup the search heads within each datacenter to have both sets of indexers as search peers to allow mulAsite searching

!   Remember to look at high availability of the license and deployment servers to handle the Splunk servers and forwarders within each environment

SecAon Overview


39


Data Tier

Indexing Tier

Search Tier


40


Data Tier

Indexing Tier

Search Tier


41

Drawbacks •  Setup and management becomes

more complex with mulAple servers in each Aer and mulAple instances that are interconnected

•  InstallaAon recommended to be done only on a physical servers

•  Depending on setup, the WAN link between datacenters could become overloaded

•  Everybody’s search and indexing load is on the same set of servers

Benefits •  Adds the ability to expand one or

both Aers to beler support and index load

•  Expansion can also be done within each datacenter as search and index load increases

•  Minimal, if any, outages comes from maintenances or refreshes as the clustering ensures the data is always available

•  High availability can be setup for all funcAons of Splunk


42

Expedia’s Story: !   There are only a limited number of search heads that can search across our two

datacenters

!   The primary purpose for mulA-‐datacenter searching is for operaAonally and security purposes and it is available to a select few users

!   There is no cross-‐site indexing of data, instead all data from servers within each datacenter is stored on indexers in the same datacenter

!   There are no mulAsite clustering or high availability setup within our environment for any piece of the Splunk environment



43

Up Next: !   ConAnued research of the opAons for allowing addiAonal users to use cross-‐site

searching and possibly opening it up to every user

!   The new mulAsite clustering feature is being looked at as part of our post-‐upgrade project to Splunk 6.1

!   With team’s moving into the cloud, instances are being created within each AWS VPC were we have producAon and lab servers

!   Due to rapid growth of the environment, impact due to team’s search and index load is being seen, so a re-‐architecAng of the producAon instances is under way to setup pod-‐based instances


Headed to the Cloud – AWS Instances SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps

Headed to the Cloud – AWS Instances

45

!   A single or mulA-‐Aer instance that is setup with the cloud, i.e. AWS, that indexes data from forwarders installed on the other virtual servers

!   It is recommended that the forwarders only send data into instance hosted within the same VPC to reduce networking cost between VPCs or datacenters

!   Remember to setup Splunk license and deployment server(s), can be on a search head, single instance, or standalone, within each VPC unless you are able to open up access to your datacenters

!   Remember to setup local Splunk accounts, for each role, within each VPC unless you are able to open up corporate authenAcaAon

SecAon Overview


46


Data Tier

Indexing Tier

Search Tier


47


Data Tier

Indexing Tier

Search Tier


48

Drawbacks •  If you follow the recommendaAon

to have instances within each VPC, you will have many environments that people have to search against

•  Need separate license and deployment servers in each VPC, unless you open up connects to the your datacenter

Benefits •  Index AWS logs without paying for

networking costs to get it back to your datacenter

•  Single Aer and mulA-‐Aer with single or many servers can be created depending on search and index load

•  Expansion can be done on each or both Aers within each VPC as needed


49

Expedia’s Story: !   There are eight separate instances of Splunk running within several AWS VPCs

throughout the world !   The Splunk instances support both producAon and lab environments running within

the AWS VPCs

Up Next: !   Current plans include upgrading the Splunk instances to Splunk 6.1 and expanding

either or both Aers as needed !   Ongoing invesAgaAon into open up the centralized license and deployment servers

to communicate with the various AWS VPCs


SeparaAng Teams – Pod Instances SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps

SeparaAng Teams – Pod Instances

51

!   Several mulA-‐Aer and server instances are setup within the same datacenter that allows team specific data to be indexed into separate pods

!   It is recommended that the forwarders only send data to the indexing Aer for that team’s pod to reduce duplicate licensing usage

!   Remember to look at high availability of the license and deployment servers to handle the Splunk servers and forwarders within each environment and pod

SecAon Overview


52


Data Tier

Indexing Tier

Search Tier


53


Data Tier

Indexing Tier

Search Tier


54

Drawbacks •  Will most likely involve addiAonal

servers to support the Splunk infrastructure

•  Setup and management becomes even more complex with pods in each datacenter and instances that are interconnected

•  Complexity also comes if other teams need to search against a pod that is not for their team

Benefits •  Expansion of one or both Aers to

beler support search or index load for each team

•  Adds the ability to separate the possible impact of expensive search load or large indexing volumes

•  Separate maintenances can be completed on each pod when it is best for the team

•  Some pods can be setup with clustering or new mulAsite clustering feature


55

Expedia’s Story: !   Working on building out five pod environments within our producAon datacenter !   The new environment will be made up of four team specific pods and an “other”

pod that will contain the rest of the teams !   All pod environments will be fully searchable for our Enterprise InformaAon

Security Team and Global OperaAons Center to monitor for security and service issues

!   All users within Expedia will have limited searchability to each pod instances to allow upstream or downstream invesAgaAons

Expedia’s Story


56

Up Next: !   The process being completed in our producAon datacenter will be repeat in our

PPE/DR datacenter next year so that the two datacenters mirror each other

!   The “other” pod will be monitored for search and index capacity issues that would warrant creaAng another team specific pod

!   Looking at ways to expand our usage of Splunk to provide addiAonal service offerings and possibly decommission old applicaAons and tools

!   Working on conAnual improvements to the Splunk environment, funcAonaliAes, features, and users

Expedia’s Story

Up Next for Expedia’s Splunk Environment

57


58

!   SecAon Overview !   Usage of Splunk and Community Apps !   MulAsite FuncAonaliAes Between Instances

!   Improved Access Control !   AutomaAon of App Deployments and ConfiguraAons

!   AutomaAon of Forwarder Deployments !   Centralized License, Deployment, and AuthenAcaAon Support for AWS

SecAon Topics


59

!   Splunk’s usage within Expedia was and sAll is to provide a centralized locaAon for log aggregaAon. It also allows users without access to the producAon servers to look at their service’s logs

!   Its organic growth has caused the environment to grow from 3TB when it was first stood up four years ago to 4TB in September 2012 to 8TB in September 2013 to 16TB in September 2014

!   It has recently become a stateless monitoring planorm of the logs indexed into the environment through dashboards, alerts, and reports

!   With our recent and ongoing upgrades to the current version and improvements to the architecture, the following slides explain what we are currently doing or would like to do next

SecAon Overview


60

Background: !   Usage of the Splunk and community created apps is limited to a couple that provide

simple funcAonality for both the users and admins !   Around 15 addiAonal applicaAons and tools are supported by our team and several

of them could be replaced with Splunk apps

Requirement: !   Provide Splunk apps that offer monitoring of our infrastructure, capacity

management for Windows, Linux, and VMware servers, and PCI and security compliance

!   Apps need to offer direct replacements of the features currently being used or extend the funcAonality past what the current applicaAons and tools can provide

Usage of Splunk and Community Apps


61

Solu,ons: ! Monitoring of Infrastructure

! Capacity Management


AppDynamics Cisco Security Suite Extra Hop Monitoring of JVM Solr Monitoring Splunk App for MicrosoP Exchange

Splunk App for Puppet Splunk Deployment Monitor Splunk for Keynote Splunk for Nagios Splunk Support for AcAve Directory

Splunk App for Windows Infra. Splunk Add-‐on for MicrosoP Windows Splunk app for Unix and Linux

Splunk Add-‐on for Unix and Linux Splunk app for VMware


62

Solu,ons: ! PCI and Security Compliance

! Extend FuncAonality


Splunk App for AWS Splunk App for Enterprise Security Splunk App for PCI Compliance Splunk for Blue Coat ProxySG

Splunk for Juniper Firewalls Splunk for Juniper SA Splunk for Sourcefire Splunk for Squid

AutomaAon TesAng -‐ SyntheAc TransacAons System Center OperaAons Manager IntegraAon Sideview UAls Splunk App for HadoopOps Splunk DB Connect

Splunk for Asset Discovery Splunk for ServiceNow Splunk Hadoop Connect Splunk ODBC


63

Background: !   With limited excepAon, there are no cross-‐site searching setup that allows a user to

search for their logs in each datacenter from a single locaAon !   There is no high availability or mulA-‐site indexing setup that crosses between our

three datacenters

Requirement: !   Provide the ability to search across each datacenter without impacAng the

e-‐commerce traffic that is on the same WAN link !   Provide mulA-‐site indexing that allows seamless failover between instances and

minimal impact during maintenances

MulAsite FuncAonaliAes Between Instances


64

Solu,ons: !   InvesAgate into taking a slice of the WAN link between our three datacenters for cross-‐

site search and index traffic !   The slice would have a throlling limit so that e-‐commerce traffic would not be impact by

Splunk running wild, similar our Big Data setup !   Even with the throlling limit, Aght user permissions will be upheld to ensure concurrent

search limits and search result size do not cause other users impact

!   With the pod setup, high availability and mulA-‐site clustering would be deployed on a team-‐by-‐team basis

!   Setup automated failover and maintenance processes to enable minimal impact to the users

MulAsite FuncAonaliAes Between Instances


65

Background: !   Only the default access roles (users, power, and admin) that are built into Splunk,

along with their default sehngs, are used in our environment !   The sehngs for searchable indexes by default is enabled for every index, which

causes issues with large search results if an index value is not included

Requirement: !   Ensure that all users have the features and funcAonaliAes they need to do their jobs

while safeguarding the Splunk environment !   Create custom access roles to allow the separaAon of Splunk’s features

and funcAonaliAes

Improved Access Control


66

Solu,ons: !   Research into the need for addiAonal, non-‐default, access roles that limit or add

to the exisAng default access roles !   UpdaAng access roles that, for example, would remove real-‐Ame searching

from the power role and creaAng a new role that grants real-‐Ame searching to select users

!   CreaAng new access roles that, for example, would increasing concurrent search limits, adding access to security specific indexes and apps, etc

!   New and updated access roles would be specific to individual instances, instead of for the enAre environment

!   Regular audits of the memberships of all access roles would be completed to ensure accurate access to the environment

Improved Access Control


67

Background: !   Only a limited number of search head and indexer apps are checked into Perforce

and none of them use the deployment server !   The majority of the saved searches, dashboards, alerAng, etc. are setup on the

search heads without peer review or code checks before they are deployed

Requirement: !   Ensure that all search head and indexer apps are checked into Perforce to allow

proper version control and back-‐ups !   Provide fast and seamless deployments of search head and Indexer apps while

sAll safeguarding the Splunk environment with automated peer reviews and code checks

AutomaAon of App Deployments and ConfiguraAons


68

Solu,ons: !   Re-‐enable the built-‐in deployment funcAonality on the search heads and indexers

which is currently disabled

!   Setup a secAon in our Perforce depot to handle search head and indexer apps and configuraAons in a centralized locaAon and enable back-‐ups

!   InvesAgate into an automated process to sync from Perforce to the deployment servers once an app is checked in

!   InvesAgate into an automated peer review and code check process that ensure no common issues (i.e. inline searches in dashboards, no index/sourcetype/host values in searches, etc.) are deployed out

AutomaAon of App Deployments and ConfiguraAons


69

Background: !   A semi-‐automated process to deploy monitoring apps is setup that pulls checked in

apps from Perforce and syncs them to the deployment servers !   The creaAon of all monitoring apps goes through a request process that is handled

and completed by my team

Requirement: !   Ensure that all monitoring apps are checked into Perforce to allow proper version

control and back-‐ups !   Provide fast and seamless deployments of monitoring apps while sAll safeguarding

the Splunk environment

AutomaAon of Forwarder Deployments


70

Solu,ons: !   Setup a new secAon in our Perforce depot to handle monitoring apps and

configuraAons in a centralized locaAon and enable back-‐ups

!   All exisAng monitoring apps are fully audited for proper setup and usefulness and the serverclass.conf file is rebuilt from scratch

!   InvesAgate into an automated process to sync from Perforce to the deployment servers once an app is checked in

!   InvesAgate into an automated peer review and code check process that ensure no common issues (i.e. backup files indexed, parsing issues, etc.) are deployed out

AutomaAon of Forwarder Deployments


71

Background: !   The various AWS instances run their own local licenses and deployment servers and

do not Ae into our central license and deployment servers !   AuthenAcaAon for the producAon instances is local and does not Ae into our

Corporate AD/LDAP systems

Requirement: !   Securely allow connecAons from the various AWS instances to our central license

and deployment servers hosted in our datacenters !   Remove the need for local accounts and allow authenAcaAon to our corporate AD/

LDAP systems like our main instances

Centralized License, Deployment, and AuthenAcaAon Support for AWS


72

Solu,ons: !   InvesAgate into the possibiliAes to open up the external firewalls to allow inbound

connecAons only from our AWS instances of Splunk

!   Create external CNAMEs for each funcAonality so that all of the cloud-‐based Splunk instances can point to them and not need to be updated for changes

!   Setup a secAon in our Perforce depot to handle search head, indexer, and forwarder apps for all of the AWS instances

!   Delete all local accounts and non-‐standard access roles, then add in the normal security group to role mappings like our main instances

Centralized License, Deployment, and AuthenAcaAon Support for AWS

QuesAons and Answers

THANK YOU

supporang*and* upgrading*large* deployments* · pdf fileexpedia’s*story*and*nextsteps*...

Documents

supporangand upgradinglarge deployments* · pdf fileexpedia’sstoryandnextsteps...