agile.2013.effecting.a.dev ops.transformation.at.salesforce

Effecting a DevOps Transformation at Salesforce.comAgile 2013

Dave MangotArchitect, Infrastructure Engineering

@davemangot

in/dmangot

Karthik RajanVP, Infrastructure Engineering

@kabbyr

in/karthikrajan

Safe Harbor

Safe harbor statement under the Private Securities Litigation Reform Act of 1995:

This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services.

The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site.

Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.

Karthik Rajan

Responsible for the infrastructure platform for running the salesforce.com service -

“Infrastructure as Code”

5 years @ salesforce.com, in various roles - developer to management

Photographer. Petrolhead.

Collector of many many gadgets

Dave Mangot

• Mainly focused on automation, monitoring, orchestration and lastly...cultural change!

• Systems Engineer 15+ years, a variety of companies big (Cable & Wireless, Sun Microsystems) and small (Tagged.com, Terracotta)

The 5 Why’s:

• Why are we the same? Different?• Why didn’t we transform TechOps on the first try?

• Why are we succeeding?• Why we still have challenges

• Why are you here?

Why are we the same? Why are we different?

http://flic.kr/p/eeWbP8 http://flic.kr/p/eeWbGp

A little story...

It was the year 2000

Number of people

fast innovativesmart

Number of Major Releases per year

7 years later

rapid success

59,300+ Customers

10 B transactions/Quarter

tons of people

it was getting more difficult to deliver

Features Delivered per Team

Days between Major Releases

Number of Major Releases in 2006

2007 – Birth of ADM

Major R&D wide Agile Transformation to ADM

On time delivery?

Last waterfall release

But what about infrastructure?

More importantly, here’s a story of a typical start-up

We’ve got an app…

We’ve got a customer or two…

Web

DB

LogsConf

Data

Stat

Hlth

ID

App

Othr

AppLogsConf

Data

Stat

Hlth

ID

App

Othr

LogsConf

Data

Stat

Hlth

ID

App

Othr

Customer

Admin

I’ve got a few more customers…

DB LogsConf

Data

Stat

Hlth

ID

App

Othr

FW

LBLB

FW

Now we’re making money…

DB LogsConf

Data

Stat

Hlth

ID

App

Othr

FW

LBLB

FWFW

FWFW

FW

LBLB

LBLB

DB LogsConf

Data

Stat

Hlth

ID

App

Othr

Salesforce.com Architecture

Why didn’t we transform TechOps on the first try?

Delivering on business priorities

Scaling through hiring

Still a relatively small team

But thenRapid success continues

100,000+ Customers

1 B transactions/day

tons and tons and tons people

So can we continue to

innovate at scale?

Lack of visibility

http://upload.wikimedia.org/wikipedia/commons/thumb/4/46/Morning_Fog_at_GGB.JPG/448px-Morning_Fog_at_GGB.JPG

Resource Bottlenecks

Lack of responsiveness, lack of team alignment on priorities

Unpredictable completion of projects and initiatives

2012 – Infrastructure Take 2 ADM + DevOps

Take 2a

Developers supporting Operations (part time)

Take 2b

Infrastructure Engineering

TechOps splits into

Operations

Take 2c

Clouds

Embedded teams

How to use DevOps?

No Crisis

The 3 Ways – Gene Kim

The First Way emphasizes the performance of the entire system

The Second Way is about creating the right to left feedback loops.

The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure;

C A M S

Culture

Automation

http://upload.wikimedia.org/wikipedia/commons/6/61/Differential_free.png

Metrics

http://www.aosabook.org/images/graphite/composer-ui.png

Sharing

Why are we succeeding?

The Toyota Way - Kata

http://upload.wikimedia.org/wikipedia/commons/7/70/Iaido2.jpg

ADM for Cultural Change - DevOps Kata

•Daily Standup - #8 “Encourage effective two way communication and other means to drive out fear throughout the organization so that everybody may work effectively and more productively for the company.”

•Sprint Retrospective - # 13 “Institute a vigorous program of education and self-improvement”

•Sprint Demo - #9 “Break down barriers between departments. People in research, design, sales, and production must work as a team, in order to foresee problems of production and usage that may be encountered with the product or service.”

Cloud Realignment

http://www.freefoto.com/images/46/01/46_01_49---Clouds_web.jpg

Theory of Constraints as per Goldratt

http://images4.wikia.nocookie.net/__cb20080623095705/starwars/it/images/thumb/d/d4/JawaNEGAS.jpg/331px-JawaNEGAS.jpg

Hack Day

Internal DevOps Mini-Conference

http://www.flickr.com/photos/gazeronly/7645165306/

Infrastructure Development Lifecycle

http://www.freefoto.com/images/46/01/46_01_49---Clouds_web.jpg

Artifact Y Deployed to Environment X from release store

Test Results marked in ReleaseDB

Eligible forpromotionto next environment

Environment specific tests are Run(increasing clock time)

PASS?

Ejected fromprocessFAIL?

Testing and Deployment

Virtualization

Embeds

http://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Embedded_Artillery_Shell_at_Fort_Sumter_%287639234388%29.jpg/800px-Embedded_Artillery_Shell_at_Fort_Sumter_%287639234388%29.jpg

Scrum Master Training

Failure? Still?

http://theintentionalwriter.com/wp-content/uploads/2013/01/head_in_hands.jpg

Front Door Process

http://upload.wikimedia.org/wikipedia/commons/thumb/e/ef/Alnwick_Castle_a_frontal_door.JPG/450px-Alnwick_Castle_a_frontal_door.JPG

Bugs

http://upload.wikimedia.org/wikipedia/commons/thumb/9/9b/Cotton_Harlequin_Bugs.jpg/757px-Cotton_Harlequin_Bugs.jpg

Why we still have challenges

Why are you here?

Present and Future Challenges

• Bringing Agile into traditional IT Ops

• Bringing IT Ops in with Infra Eng and R&D

• Re-educating workforce

• Recruiting

• Scaling Securely