DevOps as Relationship ManagementHow to keep systems of services happy

@jamesurquhartDirector of ProductDell Cloud Manager

http://gigaom.com/author/jurquhart

I want to answer three questions:

Why are the relationships between service components important in modern software systems?

What one amazing strategy can you use to manage your software systems

No, really, what are some tactics that work and are being used today?

May 6, 2010

”[A] large fundamental trader chose to execute [a $4.1B] sell program via an automated execution algorithm ('Sell Algorithm')."

- Findings Regarding The Market Events of May 6, 2010http://www.sec.gov/news/studies/2010/marketevents-report.pdf

"...the Sell Algorithm…executed the sell program extremely rapidly in just 20 minutes.”

The market responded, and trading volume increased…

"... [The Sell Algorithm] responded to the increased volume by increasing the rate at which it was feeding the orders into the market."- Findings Regarding The Market Events of May 6, 2010

http://www.sec.gov/news/studies/2010/marketevents-report.pdf

Market A

Automatic Trading

Algorithm 1

Market A

Automatic Trading

Algorithm 1

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market A

Automatic Trading

Algorithm 1

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Market A

Automatic Trading

Algorithm 1

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Automatic Trading

Algorithm 5

Automatic Trading

Algorithm 6

Market A

Automatic Trading

Algorithm 1

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Automatic Trading

Algorithm 5

Automatic Trading

Algorithm 6

Quick!What was the root

cause?

Market A

Automatic Trading

Algorithm 1

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Automatic Trading

Algorithm 5

Automatic Trading

Algorithm 6

?

Automatic Trading

Algorithm N

Automatic Trading

Algorithm N

Market A

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Automatic Trading

Algorithm 5

Automatic Trading

Algorithm 6

Automatic Trading

Algorithm N

"May 6 was…an important reminder of the interconnectedness of our derivatives and securities markets, particularly with respect to index products."

- Findings Regarding The Market Events of May 6, 2010http://www.sec.gov/news/studies/2010/marketevents-report.pdf

Interconnectedness

How do we avoid another “flash crash” (or at least make one

less likely)?

Here’s what the SEC did…

First, they visualized the

system…

…(albeit, post mortem).

Then, they took action…

They installed “circuit breakers”

They defined rollback protocols

They inserted agents into the flow

Why?

Because…While high speed trading software contained the logic by which actions took place, the dynamic ways in which that software and market systems interacted defined the emergent behaviors of the system itself.

The SEC didn’t “own” and could not dictate all aspects of behavior of those agents. They did, however, have jurisdiction over key integration points, and could insist certain rules be enforced at those interfaces

It was in the best interest of the agent owners/operators to follow these new rules and protocols, as it was how their revenues and profits were derived.

Because (simplified)…Agents contain and execute logic, but interactions drive systemic behavior.

Nobody owns the system, but one can own or influence key interactions that are relevant to desired outcomes

Well defined interfaces enable cooperative behavior, but feedback mechanisms ensure it.

Back to those questions:

Why are the relationships between service components important in modern software systems?

What one amazing strategy can you use to manage your software systems

No, really, what are some tactics that work and are being used today?

Why are the relationships between service components important in modern software systems?

Because such relationships–numerous and

dynamic as they are–define system

behavior.

What one amazing strategy can you use

to manage your software systems?

I believe I have a simple rule…

…that takes tremendous

discipline and work to execute well:

Visualize the

system…

…but take action at the agent

level

No, really, what are some tactics that work

and are being used today?

Visualize the

system…

V.

Expansionist Reductionist

Remember,you are part of the

system.

…but take action at the agent

level

Install “circuit breakers”

Define rollback protocols

Insert agents into flow

Remember,you are part of the

system.

Thank you!

Questions?