atlascamp 2014: building a production ready connect add-on
DESCRIPTION
Atlassian Connect add-ons are SaaS applications. Building and running them means planning with operations in mind: where should you host your add-on? What's the best way to deploy it? How can you monitor it once it's live? How much will it cost to run? We'll draw from Atlassian's experiences building Who's Looking for OnDemand (a production Connect add-on installed in 750 instances with 15,000 active users) to explore tips and best practices to help answer these questions and more.TRANSCRIPT
![Page 1: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/1.jpg)
June 3-5, 2014 | Berlin, Germany
![Page 2: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/2.jpg)
Robin Fernandes, Dev Team Lead, Atlassian
Building a Production-Ready Connect Add-On
![Page 3: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/3.jpg)
Who’s Looking?
19 500 users 950 hosts
![Page 4: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/4.jpg)
Overview / Stack
× many!
× many!
Who’s Looking?
Multi-tenancy
Scalabilityhttps://bitbucket.org/atlassian/whoslooking-connect
![Page 5: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/5.jpg)
What Connect ProvidesTenant registration
Authentication & tenant context
Webhooks
Integrated look & feel
Local dev & test
…Handled by Connect!
![Page 6: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/6.jpg)
A Production-Ready Add-On
Hosting
Upgrading
Scaling
Monitoring
…
Lots of decisions…
![Page 7: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/7.jpg)
Hosting
![Page 8: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/8.jpg)
Stateless Request Processing
Load Balancer
Data Store
Hosting on a PaaS can encourage
statelessness, which enables
horizontal scalability and
improves resilience.
Requests
app processes
(dynos)
whoslooking.herokuapp.com
![Page 9: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/9.jpg)
Rich ecosystem of external services
3rd Party Services
![Page 10: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/10.jpg)
Basic deployment is just a git push
Low Barrier to Entry
> git push heroku master
Free tiers allow you to experiment at no cost
![Page 11: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/11.jpg)
Testing
![Page 12: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/12.jpg)
TestingUnit testing: business as usual
Browser tests: similar to in-process add-ons
Run add-on and host app locally
Atlassian SDK, WebDriver, Atlassian Page Objects…
Add post-deployment verification tests
Reliable sanity tests, run after every deployment
Must be runnable against a production instance
![Page 13: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/13.jpg)
Testing & Muti-tenancyMock out products’ REST APIs to test multi-tenant interactions
(see Who’s Looking source on Bitbucket for examples)
ac-mock-tenants.herokuapp.com
Connect add-on
![Page 14: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/14.jpg)
Load TestingLoad test on free tier, and extrapolate
Use e.g. New Relic to monitor the add-on under load
![Page 15: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/15.jpg)
Load Testing
Tool: ac-load-test
Sets up Gatling-based load tests for Connect add-ons
Simulates authenticated requests from 1000s of hosts & users
Connect add-on
ac-mock-tenants> ./sbt runac-load-test
(Gatling)
heavy load
bitbucket.org/atlassianlabs/ac-load-test.g8
Mock OD instances
JWT-signed requests
![Page 16: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/16.jpg)
Load Testing
> ./sbt run … ================================================================================ 2014-05-17 17:36:22 20s elapsed ---- A mixture of iframe and XHR requests -------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / running: 200 / done:0 ---- Requests ------------------------------------------------------------------ > Global (OK=899 KO=0 ) > iframe-request (OK=200 KO=0 ) > token-based-xhr-request (OK=699 KO=0 ) ================================================================================ …
bitbucket.org/atlassianlabs/ac-load-test.g8
![Page 17: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/17.jpg)
Deployment
![Page 18: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/18.jpg)
Staged Deployments
whoslooking-dev. herokuapp.com
dev
run PDV tests
deploy
![Page 19: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/19.jpg)
Staged Deployments
whoslooking-stg. herokuapp.com
stg
run PDV tests
promote from dev to stg
(using pipelines)
Green dev deployment?
+ soak time
![Page 20: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/20.jpg)
Staged Deployments
whoslooking. herokuapp.com
run PDV tests
All good on stg?
prod
promote from stg to prod
(using pipelines)
![Page 21: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/21.jpg)
Further Precautions
the goggles…
![Page 22: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/22.jpg)
Things can still go wrong!Log management
Set up alerts on log patterns, search logs
Add-ons: Logentries, FlyData, Papertrail…
Feature flags
Turn new features on & off via configuration
Metrics / analytics
Get insight into application behaviour to detect issues
Libraries for in-memory metrics: Simon, Metrics, Servo…
Add-ons for externalised analytics: New Relic, StatsMix, Hosted Graphite…
![Page 23: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/23.jpg)
Healthcheck & Monitoring
http://whoslooking.herokuapp.com/healthcheck
Implement a Health Check
REST resource that runs sanity-checks & reports results
Monitor it (Pingdom, New Relic…)
Example:
(Checks Postgres & Redis connections, reports config
settings, resource usage, weekly/daily active users/hosts)
![Page 24: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/24.jpg)
Healthcheck & Monitoring
![Page 25: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/25.jpg)
To build a production-ready add-on…Design for multi-tenancy and horizontal scalability
Ensure deploying is easy
Make the most of 3rd party services
Load test to help estimate running cost
Implement a health check resource & monitor it
Run post-deployment verification tests
Start simple!
![Page 26: AtlasCamp 2014: Building a Production Ready Connect Add-on](https://reader033.vdocument.in/reader033/viewer/2022060109/5557f4aed8b42a200d8b4b2b/html5/thumbnails/26.jpg)
Who’s Looking Stats
2-4 dynos (average ~3) 5 external services (3 free, 2 paid) 19800 active users last week 960 active instances last week 3500 average rpm (~7500 peak) <1s average page load time !
$89.50 / month (~$0.005 / user / month)
Slides:
go.atlassian.com/prod-connect
[email protected] / @rewbs