Common blind spots on the

Journey to Production

Vijay Raghavan Aravamudhan

Code/People Agitator @

ThoughtWorks Technologies (Chennai)

Email: vraravam@thoughtworks.com

Github: vraravam

Twitter: @avijayr1

Is it ….

CI/CD?

Delivery timelines?

Quality?

Maintainability?

Roadmap?

Customer / End User

Team – both dev team + ops

Architecture + Tech Stack

Delivery process

v2+

In the context of:

● A new product?

● For existing users?

● Personalization + Analytics

● A/B Testing [for product ideas]

● Non-clustered architecture

● Multiple responsibilities packaged into one “app”

● Sticky sessions

● Blocking requests even for long-running steps

● Hard-coded IP addresses for endpoint URLs

● Hard-coded “linked server” IP addresses in

database (SQL Server)

Do you have the right…

• Skillsets?

• Composition?

• Motivation?

• Goal?

SCM: {VSS, TFS}, {SVN, CVS}, DVCS

CI: Jenkins/Hudson, TeamCity, GoCD, CircleCI,

TravisCI

CD: GoCD

Testing: QTP, SpecFlow, Cucumber, Selenium

Perf: LoadRunner, SilkTest, Locust.io, Gatling,

Apache Bench, Wrk

Deployment: Gradle, Maven, Ant, Psake, Rake

• Central vs Distributed

• Branch per feature

• Trunk/Master

• Short-lived POC-style branches

• Checkout

• Clean

• Compile

• Run unit tests

• Run js tests

• Code coverage

metrics

• Run integration

tests

• Package

• Deploy to

Functional Test

env

• Run Functional

Tests

● Build time goes up as the codebase grows

● Time for feedback is longer

● More complex CI setup

● Checkout, Clean, Compile, Run unit tests

(parallellize), Run js tests, Headless tests, Collect

Code coverage metrics, Package

● Deploy to Functional Test env, Run FT

o Split randomly or by functional vertical into a

build/test grid (ala Selenium Grid)

Core product is chugging along

Customer1 gets a forked version - 3 month release

cycle

Customer2 gets another fork - 4/5 month release

cycle

Each gets the cumulative feature-set only once both

a complete

Both teams diverge in tools and process

Domain knowledge gets siloed

Technology-based career growth might become

stunted

Feature-merge/Integration hell

● Combine both teams at least for design stage so

that each understands what other’s client wants

o Rotate frequently among teams to cross-

pollinate knowledge (tools + techniques)

o “Software artisans”

● Use Dependency Injection (based on tech stack)

+ Feature Toggles

● Feature-branch based development

o Use feature branches @ SCM-level for

architectural changes while delivering BAU

Active development across branches

⇒

More CI pipelines

Think beyond “application deployment”

Think “environment deployment” or

“ecosystem deployment”

Use tools like Ansible, Vagrant, Puppet, Chef,

Docker*

• Db changes should be developed alongside the

story

• It should also be part of the commits into the

SCM

• Scripts should always be incremental in nature

• As part of the CI build, ensure both roll-forward

and roll-back works

Use tools like flywaydb, dbDeploy

● Most teams assume that the app-layer will be

enough to ensure data integrity

● What happens if the app is replaced by a new

app - the db will live on, correct?

● Data validations should also be applied at the db

level - for eg foreign keys, unique constraints,

non-null checks, case-sensitive checks

● ACID Transactionality should be ensured whether

or not an ORM is used

• Use tools like active_sanity (rails gem)

• Obfuscated database snapshot from production

uploaded into non-prod env for testing on weekly

basis

• [Unfortunately] Yet another checkpoint before

pushing to production

• App should be clustered, and cluster-aware

• Deployments should not have any manual

intervention (including DB)

• DB changes should be backwards compatible

(n+1)th release can cleanup temporary stuff

from nth release

• API-changes should be backwards compatible

Dont have a “long tail” of multi-version support

● SOX/PCI Compliance

● Data at rest

● Data in transit

● Threat Vectors

● Attack surface

OWASP Guidelines