building internet-scale applications
TRANSCRIPT
![Page 1: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/1.jpg)
Building Internet-scale Applications:The Beginning
By @gaveen At #RefreshColombo 22 September 2011
![Page 2: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/2.jpg)
Gaveen Prabhasara
twitter.com/gaveen
SysAdmin
DevOps
Works at Thinkcubegaveen.owain.org
github.com/gaveen
![Page 3: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/3.jpg)
Standing on theShoulders of Giants
If I claim to be a wise man, wellIt surely means that I don't know
“
“
- Kansas Carry On Wayward Son
![Page 4: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/4.jpg)
This is a 25 minute talk
Which sounds like a collection of
Random thoughts about
Building Internet-scale Systems
You probably already know
![Page 5: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/5.jpg)
Internet-scale
What is it?
Applications
Does not always mean WebApps
![Page 6: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/6.jpg)
Internet-scale > Enterprise-scale
Facebook,YouTube, Twitter,Gmail, Flikr
Eg:
Case: Facebook750M+ active users350M+ daily users100B+ daily hits30B+ monthly objects shared2.5M+ websites integrated130TB+ daily logs*Sources: http://goo.gl/zT1bL
http://goo.gl/cdFPI
Case: Akamai95800+ servers1000+ networks70+ countries15-30% WWW traffic*Source: http://goo.gl/yVmG9
![Page 7: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/7.jpg)
Culture
Culture
Culture
How
![Page 8: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/8.jpg)
Know Your Trade
Learn
Acquire domain-specific knowledge
Learn how people build similar services
Learn about Technologies
Stay updated
RTFM / Use Search Engines ;)
![Page 9: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/9.jpg)
Get Right People
Team
Find balance in the team
Know their strengths
Nurture them
Set a proper environment
Don't step on others toes
![Page 10: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/10.jpg)
Go for the Best Solution
Architecture & Design
Have a solid architecture
Find a good engineering design
Don't over-engineer / over-complicate
Avoid architecture degradation
![Page 11: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/11.jpg)
Right Tool for the Right Job
Technology
You favourite tool can't do everything. Face it
'Prejudice against' & 'religious for' isn't good for you
Brand names aren't for worshiping
Cloud isn't a magical silver bullet
This is Spar... the Internet. It's big
![Page 12: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/12.jpg)
Do Things Right
Process
Use best practices
Automate. Automate. Automate.
Plan for scaling from the start
Don't let the process be a burden
Little more detail...
![Page 13: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/13.jpg)
Test All the Time or better,do Continuous Integration
Little more detail...
Testing isn't a QA job
Let Devs take responsibility
XP, TDD, BDD, Cucumber, etc.
Don't push untested code
Works-in-my-machine doesn't count
Process
![Page 14: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/14.jpg)
Fight the Smell, Fight Anti-Patterns
Little more detail...
Re-factoring is good
Code smell isn't something to behold
Don't let the code age on you
Don't let the architecture degrade
Bad code is bad code
Process
![Page 15: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/15.jpg)
Use Version Control
Little more detail...
Use a proper VCS
Git, SVN, Hg, Bzr, etc.
Helps automation
No version control, no deploying
Learn to use a VCS properly
Process
![Page 16: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/16.jpg)
Use Configuration Management
Little more detail...
Infrastructure is Code, therefore Automate
User a Configuration Management System
Puppet, Chef, CFEngine, MCollective, etc.
One-off work is (usually) a myth
Managed Infra opposed to mucked-about infra
DevOpsProcess
![Page 17: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/17.jpg)
Security isn't a Task
Little more detail...
You can't apply security later
Both infra & code needs to be secure
“Noboday wants to hack me” is a myth
HTTPS isn't a fad
# chmod 777 kills puppies
SELinux / AppArmor isn't your enemy
Process
![Page 18: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/18.jpg)
Scaling starts from the beginning
Little more detail...
Plan early. Start from basics.
Balance load & plan for HA
Use Caching
Do stress / load testing
Not every technology scales gracefully
Use messaging, async jobs when needed
Etc., etc.
Process
![Page 19: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/19.jpg)
Monitor & Measure
Little more detail...
Capture important metrics
Measure, analyze & improve
Monitoring sucks. But still...
Home grown a NMS aren't necessarily better
Have notification alerting in place
Process
![Page 20: Building Internet-scale Applications](https://reader034.vdocument.in/reader034/viewer/2022052623/559d16451a28abca018b480a/html5/thumbnails/20.jpg)
Q & A
Anybody who wants ISO's of the following Linux distros,meet me with USB drives after the session.
BackTrack, CentOS, Debian, Fedora, Hanthana, MeeGo, Mint,Ubuntu, Ubuntu Server