the new it: rethinking a 60 year old industry self-service changes the game for departments; it...
TRANSCRIPT
The New IT: Rethinking a
60 Year Old Industry
1
Geoff Halprin The SysAdmin Group
[email protected] @geofficate
Driving IT Conference Copenhagen
November 2017
(C) Copyright 2015-2017, Geoff Halprin. All Rights reserved.(C) Copyright 2015-2017, Geoff Halprin. All Rights reserved.
2This is not a talk about containers… but it sort of is!
3This is not a talk about startups… but it sort of is!
4This is a talk about enterprise IT
The Old World is Ending
Act I
5
The Age of the Dinosaurs
6
Software Development in the 80s
http://www.flickr.com/photos/cblue98/6679374097/
Highly Formalised, Unidirectional WorkflowThe Waterfall Model
‣ Key Attributes: • Left-to-right progress. Each phase completed before next
commenced (in theory).• Right-to-left traceability. Code can be traced back to requirements.
7
Requirements Architecture Design Code Test Release
‣ Input:
‣Requirements from user
‣Output:
‣RequirementsDefinitionDocument(RDD)
‣ Input:
‣RDD
‣Output:
‣SystemArchitectureDescription(SAD)
‣ Input:
‣ SAD
‣Output:
‣DetailedDesignDescription(DDD)
‣ Input:
‣DDD
‣Output:
‣Code
‣ Input:
‣RDD, SAD, DDD, Code
‣Output:
‣Test Reports
‣ Input:
‣Code, Test Reports
‣Output:
‣Product
Lie #1: It was always a bad guess
8
Lie #2: The world does not stand still
9
Lie #3: Not all developers are equal
10
Lie #4: The A-team versus the B-team
11
The Root CausesThe “Big M” Methodologies
‣ Human Resources • Eschew the difference in productivity of programmers.
‣ Wrong Model • “All the creativity is at the beginning, the rest is just construction.”
‣ High Risk, Hence High Formality • Such large investments require significant controls to (attempt to)
ensure that the value is delivered.
12
We have known these methodologies were wrong for decades, and yet we still use them in most
enterprises
13
This waterfall approach to projects extends from inception to retirement
It extends beyond the Software Development boundary
It extends beyond the IT boundary
14
Physical infrastructure investment was cost-optimised, and so bespokeInfrastructure Investment
‣ Expensive, bespoke Production infrastructure and applications
‣ Different infrastructure for Development and Test environments
‣ Long delivery and commissioning lead times
‣ Large, complex production acceptance processes
15
Reflects physical hardware costs
The Capex view of the world is about ensuring delivery (visible success) of long-running projects through “high ceremony” governance
Capex Budget Allocation
‣ Budget allocation, funding gates • Fight to keep the project “above the line” for next financial year
• Control release of funds to “ensure” delivery
‣ Project management • Seen as responsible for delivery
‣ Operational acceptance • Compliance checklist as gatekeeper
‣ Alarm response • Defined during build - with no production experience
16
Agile fails in many enterprises because is must still deal with traditional funding and operations models
The Agile Sandwich
17
Agile
Project
Release 1 Release 3
Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5
Release 2
Value realised over time
2 weeks
WaterfallWaterfall
There has Been a Seismic Shift
Act II
18
New technologies have enabled new practices that affect project delivery,
risk management, and altered the nature of good IT management irreversibly.
19
Technology Trends
20
Technologies
Technical Practices
Business Practices
Enabling self-service changes the game for departments; it moves infrastructure from Capex to Opex
Cloud Computing - Key Attributes
‣ Independent, compartmentalised (isolated) tenancies
‣ Self-service and automated fulfilment
‣ On-Demand
‣ Ephemeral
‣ API based services (CRUD)
21
Cloud is important because it enables self-service, incremental investment, and new risk management
practices
22
But it also introduces new failure and resilience postures
23
The cloud requires a new approach to risk managementRisk and Resiliency
‣ Failure is inevitable - at cloud scale it a mathematical certainty • Cloud is unreliable, but offers infrastructure diversity
‣ Human led processes: (1) do not scale, (2) are not fast enough • Autonomous response is required
‣ In the cloud, the application must deal with failure (service resilience) • There is no separate infrastructure responsibility; the application is
responsible for all failure modes. This includes network functions.
‣ Being able to deploy is more important than keeping the platform operating • Infrastructure diversity and rapid repair requires automated deployment
24
The Inception of Cloud ComputingAPIs
25
All teams will henceforth expose their data and functionality through service interfaces.Teams must communicate with each other through these interfaces.There will be no other form of inter-process communication… no back-doors whatsoever……All service interfaces, without exception, must be designed from the ground up to be externalizable…No exceptions.
Anyone who doesn’t do this will be fired.Thank you have a nice day!
— Jeff Bezos, 2002
APIs are important because they create an ecosystem
An internal ecosystem: they enable decentralisation of the SDLC
A wider ecosystem: they also enable Internet based commerce in software components (Software as a Service)
The enterprise no longer has to construct complete solutions
26
Corollary: Enterprises are no longer limited by the velocity of the IT department
27
But they introduce the potential for significant complexity and multi-party dependencies
28
Microservices
29
In short, the microservice architectural style isan approach to developing a single application as a suite of small services, each running in itsown process and communicating with lightweightmechanisms, often an HTTP resource API. These services are build around business capabilitiesand independently deployable by fully automated deployment machinery. There is a bare minimumof centralised management of these services,which may be written in different programming languages and use different data storage technologies.”
— Martin Fowler
Microservices encourage a move from centralised construction of large, complex systems towards decentralised assembly of discrete components.
30
Systems Must Take Three Design Aspects Into ConsiderationThe Three Planes of Code
31
Execution Plane
𝜇S-A-02𝜇S-A-01
AZ-West AZ-EastImplementation Plane
𝜇S-B-01𝜇S-B-02𝜇S-A-01𝜇S-A-02
Design Plane
𝜇S 𝜇S
𝜇S 𝜇S
𝜇S
Microservices are important because they enable discrete component delivery, reduce integration overheads, and simultaneously increase velocity
and quality.
32
But they introduce the potential for significant complexity and far more dynamic production
configuration changes
33
New Technical Practices
34
Technologies
Technical Practices
Business Practices
Leveraging these new technologies requires new approaches to IT across the lifecycle
A Digital Delivery Model
35
Arch/ Design
Build/EvolveImplement
Operate
ITIL ➔ SRE
Bespoke ➔ Cloud
Design First ➔ LibraryConstruction ➔ Assembly
Waterfall ➔ Agile/DevOpsMonoliths ➔ Microservices
TOGAF has not evolved to support the digital enterpriseTOGAF is Feeling Dated
36
TOGAF
Technical Reference Architecture (TRM)has not been updated since pre-cloud era
TOGAF development methodology isheavy, document based, and slow.
Does not reflect cloud-nativeoperational requirements
Does not reflect modern service(and microservice) based ecosystem
Does not reflect modern iterativesoftware development practices
Does not reflect customerexperience led design thinking
Architecture must move from gatekeeper to enablerEnterprise Architecture as Librarian
‣ Software development is now: • Incremental and iterative• Assembled from component microservices, rather than constructed
from as a self-contained monolith
‣ Enterprise architecture must evolve • The model I propose is that of librarian, or concierge• Support and encourage good behaviour in distributed teams
• Advisor, not gatekeeper
37
Business Value Faster and More Reliably Agile Versus Waterfall
38
Project
Release 1 Release 3
Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5
Release 2
Agile Project
Value realised over time
2 weeks
Project
SolutionDefinition Design & Build
RDD
Waterfall-based Project
SAD DDD Code
Test
Release
Value realised over time
2 years?
Agile fails in many enterprises because is must still deal with traditional funding and operations models
The Agile Sandwich
39
Agile
Project
Release 1 Release 3
Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5
Release 2
Value realised over time
2 weeks
WaterfallWaterfall
Continuous Delivery and Feedback LoopsDevOps: A New Model
40
Requirements Architecture Design Code Test Release Operate
Continuous Integration
Continuous Delivery
DevOps
Waterfall/Agile Software Development
Continuous Deployment
DevOps is a new, expansive paradigm, where software evolution is data driven, by continual feedback loops
DevOps is About More Than Agile
41
DevOps turns a number of traditional IT practices on their headThe Big DevOps Behaviours
‣ Shared ownership and high collaboration
• vs. organisational silos and hand-offs
‣ Continuous Integration and Delivery • vs. high cost integration testing
‣ Risk management by embracing change
• vs. fear of change
‣ Ephemeral infrastructure • vs. build once, hand crafted
“snow flakes”
‣ Automation • vs. manual fulfilment
‣ Feedback loops and data driven responses
• vs. waterfall design of new features and releases
42
Tools: Driving Speed and Consistency
43
Jenkins, Bamboo, Teamcity
Continuous Integration
Git, Mercurial, Subversion
Source Code Control
Crucible, Gerritt
Human Review
Coverity, Findbugs
Static Analysis
Clover, Emma
Code Coverage
Code Review
Eclipse, Visual Studio
Development
Rally, Jira, Bugzilla
ALM
JUnit, SOAPUI
Unit
Silenium, Pacts, ServerSpec
Integration
Loadrunner, SOAPUI
Stress/Volume
Test Automation
Cucumber, Selenium
Acceptance
Make, Ant, Rake, Maven
Build
Nexus, Artefactory
Artefact Repository
Packer, Docker
Packaging
Continuous Delivery
Vagrant, Heat
IaaS
Infrastructure
Cloud Foundry, OpenShift, Docker
PaaS
Puppet, Chef, Ansible
Deployment
Release Mgmt
Consul, HAProxy, Envoy
Activation
New Relic, Sensu, Logstash, Spunk
Monitoring
Management/Ops
Only the left edge is “writable”. Everything else is read-only.DevOps Best Practice: The Hard Left Edge
44
Everything other than Production is a GuessDecoupling Deployment from Activation
45
DevOps manages risk through increasing the rate of change rather than aversion to it
DevOps Risk Management
‣ Deploying is king • Deployment must be painless.• You have deployed the same thing several times before it gets to Production.
• Don’t guess - use real production data!
‣ Deployment is decoupled from activation • Risk is managed via activation controls
- Canaries, 1% of the user base, waves of activation, etc.
‣ Deployment is not “one size fits all” • It is a rail yard of interconnecting steps and micro-processes.
46
DevOps places paramount importance on automation and quality across the entire idea-to-production pipeline
The Release Pipeline
47
Requirements Architecture/ Design Code Test Release Operate
Feature FlagsA/B TestingAutomated failure responses
Ephemeral InfrastructureCD - push on green
On-demand environmentsCI - nightly builds
Test-Driven DevelopmentSmaller, more frequent releasesAutomated build from source artefacts
Just enough designRefactoring
MVP, MMFReal world feedbackEvolution of product
The cloud has enabled new ways of managing server builds, software release, and server entropy
Infrastructure as Code
‣ Cattle, not pets • Servers are built on demand via automation.• Logging into the server is seen as failure.
‣ Release through parallel infrastructure • Build the new version on new infrastructure; stage transition between
environments (zero downtime).
‣ Ephemeral (transient) infrastructure • Thrown away when it is no longer needed.• This eliminates entropy - a major source of change failure.
48
Inverting key decisions and limitationsA Clash of Operations Cultures
49
Traditional Ops DevOps
‣ Manual configuration changes to critical infrastructure.
‣ Application architectures defined by network design.
‣ Bespoke infrastructure built once, then maintained.
‣ Risk managed through change windows.
‣ Processes biased towards “build once”.
‣ Automated deployment to all environments.
‣ Network design defined by application architectures.
‣ Ephemeral infrastructure created for each new deployment.
‣ Risk managed through progressive activation.
‣ Processes re-engineered for high volume, rapid throughput of changes.
Applying software engineering to system operationsSRE: ITSM at Cloud Scale
‣ The purpose of operations is to protect production • Reliability, Availability, Serviceability (RAS)• These principles haven’t changed since the 60s
‣ In the 90s, the world began adopting ITIL • Process oriented controls• Assumption: human fulfilment• Assumption: large, up-front, project-based delivery • Assumption: bespoke solutions and infrastructure
‣ The cloud needs a different model • Site Reliability Engineering is the application of the same principles at cloud scale.
50
51
Industry Trends
52
Technologies
Technical Practices
Business Practices
Incremental, Data and Experiment Driven Product DevelopmentLean Enterprise
‣ Minimum Viable Product,Minimum Marketable Features
‣ Business Model Canvas
‣ Learn (guess), Test (prototype),Engage (trial), Measure, Pivot
53
The Role of Central IT Must EvolveDecentralised Computing
‣ The Problems: • Central IT is too slow and too restricting
• JIT infrastructure - Cloud only requires a credit card
‣ Shadow IT is now Departmental IT • BUT: Data security/compliance obligations remain: PII, GDPR, LI, etc.
‣ IT must evolve from gatekeeper to service provider • Changing role of architecture: concierge/librarian
‣ The Big Issues: • Data governance, duplication, and compliance• Duplication of effort
54
A Stark Contrast
Act III
55
Digital enterprises value flexibility and speed to market mostDigital Companies Prioritise Flexibility
56
A comprehensive re-engineering of the enterprise is requiredThe Transformation Journey
57
From:Bespoke infrastructureManual fulfilmentProject based deliveryITILWaterfallSOA/ESBInside-out
To:CloudAutomationProduct evolutionSite Reliability Eng.DevOpsAPIsOutside-in
Realising the benefits of digital technologies will require significant changes to the company operating model
An Unprecedented Challenge
‣ A transformation like no other • It affects the entire operating model (DNA) of the company.• A product-led transformation will fail.
- But new generation products are a critical element of transformation.
• An operating model transformation is required.
‣ This is about putting customer experience at the centre of the new world
58
The Right Discussion
59
The investment model begets the risk and governance model which begets the operating model
A New Operating Model
60
Investment Model Risk/Governance Model Operating Model
The incumbent operating model reflects the physical infrastructure investment model of the past 40 years
The Incumbent ICT Operating Model
‣ Expensive, bespoke production infrastructure and applications • Long delivery cycle times• Maintained in-situ across a lifespan of many years• Other environments do not match production due to cost
• Resistant to automation due to highly customised nature of each host• Subjected to outsourcing in order to control costs
‣ Project based delivery • New = complex, expensive, risky• Separate project team
• Everything must be delivered (via capex) during build phase
‣ Clear separation of business, development, QA, security, and production teams • Handoffs, engagement artefacts, lack of end-to-end responsibility
61
Digital infrastructure is virtual, incremental, and on-demandThe Digital Operating Model
‣ Cloud • Highly scalable, self-service environments, build on commodity hardware.
‣ High levels of automation • Infrastructure build, testing, release, deployment, activation• Rapid real world feedback
‣ Product based delivery • Continuous stream of small incremental improvements (frequent, rapid)• Industrialised view of change
‣ Integrated, cross-functional teams • High collaboration and shared goals
‣ Digital technologies • Re-engineering of core processes against new technology backdrop
62
The modern enterprise now prizes agility and experimentation above stability and ROI
It’s About Risk and Enablement
‣ This is about risk models
‣ The old risk model reflected the highly bespoke nature of construction • It focussed on protecting the (very large) investment
‣ The new world is software on commodity hardware • The investment is now incremental• Agility is far more valuable
‣ The risk, governance, and operating models must evolve
63
This is about the re-engineering of the enterprise from a “create” bias
to an “update” bias
64
