WEST Interactive’s Deployment Approach and DevOps Practices

— WSO2 Customer Webinar —

Preston Gross-Rhode - Systems Engineer, West Corporation

Chathura Kulasinghe - Senior Solutions Engineer, WSO2

Agenda● WEST corporation● The deployment ● Production hardening● Scalability, Availability and Disaster Recovery ● Lifecycle management, versioning and artifact distribution● DevOps automation ● Troubleshooting and Debugging the production system● Enhancements planned

The need of WEST InteractiveTo build a,

○ Service Oriented, ○ Multi-channel and Multi-client Platform ○ which leverages common Protocols, ○ Languages and Interfaces

web, mobile, sms, ivr, b2b

The Design of the solution

WSO2 API Manager

3rd Party System Units

WSO2 Message Broker

WEST System Units

WSO2 Enterprise Service Bus

WSO2 Data Services Server

WSO2 Application Server

WSO2 Identity Server

Deployment : Clusters and the deployment topology

LB 3rd Party System Units3rd Party

System Units3rd Party System Units

Data

Cen

ter -

01

Data

Cen

ter -

02

WEST Interactive 3rd PartiesClients

Production Hardening - Deployment ModelThe deployment model chosen by WEST interactive,

• Was not necessarily based on WSO2. • Deployed on own VMs on-premise, • Built some system development prototyping environments on AWS.

Because, WSO2 products

• Are JVM based, • Support many operating systems,• and many Infrastructure

Production Hardening - VM based Tuning - WEST

Hardware specification of a particular VM

• Quad Core• 16 GB of RAM• 45 GB of hard disk space

With JVM Based tuning as,

• -Xms8g -Xmx8g -XX:MaxPermSize=2g

Production Hardening - VM based Tuning vs WSO2 Spec

Hardware specification of a particular VM

• Quad Core• 2 Cores

• 16 GB of RAM• 4 GB of RAM

• 45 GB of hard disk space• 10 GB of hard disk space

With JVM Based tuning as,

• -Xms8g -Xmx8g -XX:MaxPermSize=2g• -Xms2g -Xmx2g -XX:MaxPermSize=1g

VM (Virtual Machine)

CPU 2CPU 1

CPU 1

2 GB RAM2 GB RAM

OS

OSFiles Space

JDK W

SO2

Bina

ry{Z

IP a

rchi

ve e

xtra

cted

}

Carbon/JVM instance

Production Hardening - Performance StatisticsExample:

Records against a Cluster of WSO2 Enterprise Service Bus with 4 instances

Transactions Per Hour (TPH) Transactions Per Second (TPs)

Average load 660,120 184

Historical spikes 2,520,000 700

Load tests 7,200,000 2000

WSO2 ESB

η

Scalability, Availability and Disaster Recovery - Performance vs Capacity

The general equation is,

n = ⌈{ η + η (30/100) } / ω⌉In this deployment n = 4,

4 = ⌈{ η + η (30/100) } / ω⌉If ESB capacity = 2000 TPS,

4 = ⌈{ η + η (30/100) } / 2000⌉Therefore,

η < 6154

WSO2 ESB

η

Scalability, Availability and Disaster Recovery - The real capacityη < 6154 TPS

Therefore,

• Occasional peak upto 6154 TPS

Targeted,

• 6154/2 TPS ( 3077 TPS )• for the entire setup• which is the capacity,• of 2 ESB instances• considering HA

WSO2 ESB

WSO2 ESB

6154/2 TPS

Data Center - 01

Data Center - 02

Scalability, Availability and Disaster Recovery : 2 Data Centers

LB 3rd Party System Units3rd Party

System Units3rd Party System Units

Data

Cen

ter -

01

Data

Cen

ter -

02

WEST Interactive 3rd PartiesClients

Scalability, Availability and Disaster RecoveryOur design of the solution attempted to

• drive state below the SOA framework, • as to steer heavily towards linear scaling

since true clustered communication between the units is not required.

This applies to

• all units apart from Message Broker • those require Clustering, Monitoring • and Client-side load-balancing (as a critical requirement).

LCM, Versioning and Artifacts distribution

Systems Development

Application Development

Quality Assurance Pre-prod Production

Puppet

Chef

Perfo

rce

SVN Deployment ArtifactsConfiguration Artifacts

Automation

Automation

DevOps and Automation

• Current puppet implementation revolves around the automated phone systems platform

• In near future, WSO2 platform will be moved into this infrastructure

• A bonus of the WSO2 product slate is that configuration and settings are similar across products

• This simplifies this effort.

DevOps and Automation : The Typical WSO2 Product

Carbon Kernel

Features Collection

Carbon Kernel

Identity Management

Features

Carbon Kernel

Integration Features

Carbon Kernel

Application ServerFeatures

Identity Server ESB Application Server

Carbon Kernel

Clustering Server-framework Registry

Logging Data-sources Transports User-management Multi-Tenancy

Admin-console Feature-manager

Dep-sync

DevOps and Automation : Configuration Files and The Kernel

Systems Development

Application Development

Quality Assurance Pre-prod Production

Puppet

Chef

Perfo

rce

SVN

Carbon Kernel

Configuration files

Configuration file templates

Configuration properties of different servers (key-value)

Troubleshoot and Debug the production system

• A large percentage of services revolve around the ESB• WSO2 ESB is used to handle errors from other systems

gracefully, which would otherwise create negative consequences.

• This follows general Integration Error Handling patterns• Explained comprehensively at http://goo.gl/BzL8L9

Troubleshoot and Debug the production system

• Otherwise the errors are debugged using the WIRE and DEBUG logs …

• and dynamic level logging within the services as well as packet captures as needed

• We have employed Bamboo automated Testing suite with SOAPUI test cases …

• to run every night against all known testable services, (currently around 140 services)

Enhancements planned• Integrate WSO2 Governance Registry … • into WEST Interactive’s devops effort … • along with the other devops tools.

Systems Development

Application Development

Quality Assurance Pre-prod Production

Puppet

Chef

Perfo

rce

SVN

WSO2 Governance Registry

{ QA }

{ Pre-prod }

{ Production }

Enhancements planned• Extend the Identity Server installation for many new user

management tasks. • Interested in what the new carbon core will bring to the table in

terms of performance and capability.

CONTACT US !