technical debt - sotr14 - clarkie

Technical DebtIs your codebase approaching the fiscal cliff?

Andy Clarke clarkie@concrete.cc http://clarkie.io @clarkieclarkie

This is me

• Software Engineer for 8 years

• Never worked on a greenfield project (at work)

• Technical Architect @ Concrete

Outline

• What is Technical Debt?

• Debt Accumulation

• Debt Reduction

• Q + A

– Ward Cunningham, OOPSLA 1992

“Shipping first time code is like going into debt; !

a little debt speeds development so long as it is paid back promptly with a rewrite.”

– Intel, 2009

“Technical Debt is loosely defined as the volume of poorly written lines of code, poorly refactored, not following coding standards, not supported

with sufficient unit tests, and the amount of code duplication.”

– Michael (Doc) Norton, 2009

“Messy Code is not Technical Debt”

Technical DebtIs your code base project a ‘bag of spanners’?

Andy Clarke clarkie@concrete.cc http://clarkie.io @clarkieclarkie

Where does it come from?

• No vision

• No design

• Operational

• Experimentation

Where to keep your debt

• Code

• Persistent storage

• Architecture

• Infrastructure

Debt AccumulationCase Study: how Concrete got to where we are today

• A single platform to manage projects, plans, people, content, all from one place in the cloud.

• Enterprise collaboration

What is Concrete today?

• First client

• Need a picture library

In the beginning

• How do you create two new client sites?

• Easy

• Ctrl+C, Ctrl+V

In the beginning

• A client needs feature X

• A new client signs and requires feature X

Adding Features

• 2010 - 10 clients sites

• New client = new developer

• Cross selling features

• Bug fixes?

• 12 million lines of code!

Scaling

What we have today

• 30 production sites & 30 test sites

• Monthly releases

• Full LNSR HA stack

• Code deployed in sub 1 minute

• An understanding of our debt

What we have today

• 9 new applications

• A vision

• A product owner

• An architect

What we have today

0

10

20

30

40

50

60

70

80

90

2005

-04

2005

-07

2005

-10

2006

-01

2006

-04

2006

-07

2006

-10

2007

-01

2007

-04

2007

-07

2007

-10

2008

-01

2008

-04

2008

-07

2008

-10

2009

-01

2009

-04

2009

-07

2009

-10

2010

-01

2010

-04

2010

-07

2010

-10

2011

-01

2011

-04

2011

-07

2011

-10

2012

-01

2012

-04

2012

-07

2012

-10

2013

-01

2013

-04

2013

-07

2013

-10

2014

-01

2014

-04

Thou

san

ds

Total

Debt Reductionwhat we did to enable us to grow

How did we change

• Dev sites

• SubVersion

• Continuous Delivery

Continuous Delivery

Continuous Delivery

Continuous Delivery

• SVN post commit hooks

• Windows batch file & scheduled task

• Jenkins

Continuous Delivery

Consolidated Codebase

• 10 clients

• 6 applications

• 12,000,000 lines of code

Consolidated Codebase

• 30 clients

• 12 applications

• 72,000,000 million lines of code (estimate)

• 800,000 (actual)

Consolidated Codebase

• New framework

• Moved ‘core’ applications to a new repository

• ‘Core’ deployed independently

• Integration issues

• Shared database adds complexity (more debt)

Consolidated Codebase

• 3 new clients

• Usual copy and paste or…

• ConcretePlatform v2

Consolidated Codebase

• Merged ‘core’ back into a site

• Manually diff databases

• Merged everything on deployment

• Still deploying 60 copies of code base

• Deployments could take up to 1 hour

Consolidated Codebase

‘CP2’

New Clients

‘Coring’

Consolidated Codebase

• True single codebase (so close!)

• Deployed 10 times

• Each instance can run any client site

• Configuration driven

• Designed at SOTR 2013!

404 Handlers

• Tidy up static assets

• Enabled migration to Railo

404 Handlers

• <cfdirectory>

• QoQ

• Redirect or fail

Keeping dev sites up to date

• Code

• Database

• Configuration

Release Process• Monthly releases

• Run pre_core.sql

• Run pre_client.sql for each client db

• Deploy code (anything up to 30 minutes)

• Run post_core.sql

• Run post_client.sql for each client db

Database Rebuilds• Take back up from live

• Copy backup to dev server

• Drop the dev db

• Restore the backup from live

• ‘Massage’ the data

• Perform the release

• Every night

Database Rebuilds

Local Development

• Self installed + Configured

• Virtualbox Image shared

• Vagrant + Puppet

Monitoring and Metrics

• Statsd & Graphite

• Make it easy

• <cfset statsd.increment(“app.sessions.expired”) />

• Think about your metric structure

Monitoring and Metrics

• Logstash and Kibana

• Make it easy

Monitoring and Metricsfile { type => "statsD" path => "/var/www/*/WEB-INF/railo/logs/statsd.log" }

grok { type => “statsD" match => [ "message", ‘"%{DATA:level}" ,"%{DATA:thread}" ,"%{DATA:date}" ,"%{DATA:time}" ,”%{DATA:application}" ,"%{GREEDYDATA:app_message}"'] }

Monitoring and Metrics

So what’s next?

More to do• Decouple our backend services from the

databases

• Decouple the front end applications from the asset storage

• Consolidate our web servers

• Improve application usage metrics

• Unit testing - thanks Kev!

• Migrate to AWS

Codebase VCS Deployment Monitoring Logging

Crawl Individual _old FTP Nagios <cfmail

Walk ‘Core’ SubVersion Batch files Pingdom + PagerDuty <cflog

Run 1 Git Jenkins Cedexis + Statsd Kibana

Baby steps

Conclusion

• Technical debt isn’t bad

• Debt can give you a quicker time to market

• There isn’t a silver bullet

• You have to fight your corner

• It’s worth paying it off

Thanks

• clarkie@concrete.cc

• http://clarkie.io

• @clarkieclarkie