"in love with open source : past, present and future" : keynote osdconf 2014
DESCRIPTION
OPEN SOURCE DEVELOPERS CONFERENCE http://osdconf.in/ ★ April 26-27th, Noida ★ Keynote Session By Piyush Kumar (Lead of Infrastructure and Website Operations at MakeMyTrip)TRANSCRIPT
![Page 1: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/1.jpg)
OSDConf 2014 “In love with Open Source : Past,
Present and Future…“
~/Piyush
http://piyush.me
![Page 2: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/2.jpg)
In real open source, you have the right to control your own destiny. ~Linus Torvalds
OSDConf 2014 : http://piyush.me 2
![Page 3: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/3.jpg)
Mostly Everyone Uses/Loves Open Source Directly or Indirectly
• Linux
• Android
• Mozilla Firefox
• LibreOffice/OpenOffice
• Apache HTTP server
• Asterisk (PBX)
• VLC media player
• Wordpress
OSDConf 2014 : http://piyush.me 3
![Page 4: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/4.jpg)
4 C’s of Open Source
• Community • Collaboration • Collective Ownership of intellectual property
rights – Distributed Credit
• Continuous, public process
– Software releases part of the process – Open peer review – Public Bug Tracking – Testing
OSDConf 2014 : http://piyush.me 4
![Page 5: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/5.jpg)
My entry to open source…
• I started with using Linux
• then started using a classic LAMP stack as Dev
• And then in sea of Infrastructure (WebOps) it was open source all around
• Now playing with BigData Stack again … its OSS … making our life easier !
OSDConf 2014 : http://piyush.me 5
![Page 6: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/6.jpg)
Journey….
• 2003-2004 -> Linux User back there in my college days
• 2005 -> became first Red Hat Certified Engineer of college
• 2006-2008 -> NexTag Journey as QE
• 2008-2013 -> MakeMyTrip, Website Operations Team
• 2013-till now -> BI / Lead Data Platform Team
OSDConf 2014 : http://piyush.me 6
![Page 7: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/7.jpg)
2006 -> 2008
• Started as QE – Quality Engineer at NexTag.com (27 months) part of QA-Infrastructure team having worked on managing QA Servers, Testing of NexTag Application (Search Indexes, Cache System, Monitoring Applications) and Automation
• Performance Testing of OS, Apache HTTP server, Tomcat + Every New Release
• Managing F5 BigIP LTM in QA & Google Mini (Search Engine)
OSDConf 2014 : http://piyush.me 7
![Page 8: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/8.jpg)
Key Learning's • COMMUNITY : IRC #centos channel freenode
– Inspired by awesome z00dax : Karanbir Singh
• HABIT : IRC #noc channel for production updates @ NexTag Infra • Learned the art of
– ./configure && make && make install
• Love for BASH #!/bin/bash • SOA – Service Oriented Architecture • Performance Testing - Jmeter/Replay ( Load / Stress ) • Automated Testing – Sahi / Selenium • Code Coverage - Cobertura • Importance of managing Infrastructure like your web applications
– Continuous updates / improvements – Automation – Monitoring
OSDConf 2014 : http://piyush.me 8
![Page 9: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/9.jpg)
2008 -> till date
• I played various roles in Web Operations (which demanded multidisciplinary skills )
– System Engineer
– Security Operations
– Lead Infrastructure
– ISM – Information Security Manager
– Lead DataPlatformTeam
OSDConf 2014 : http://piyush.me 9
hacker@makemytrip
![Page 10: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/10.jpg)
DDoS Protection
Firewall - ASA
IPS – Intrusion Detection / Prevention System
WAF – Web Application Firewall
Load Balancer
Web Server
Engine / SOA / APIs
RDBMS - Transactional Database
NoSQL
Caching Technologies
Virtualization / Cloud Stack
Email / DNS / AUTH / Config Management / CI/Deployment
CDN – content delivery network
Log Management - SIEM
Typical Web Infra Stack
Obviously including Hardware, Network & Power OSDConf 2014 : http://piyush.me 10
![Page 11: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/11.jpg)
MMT OSS powered Tech Stack • Programming Languages
– JAVA / PHP / Python
• Operating System – CentOS 5/6 • Caching proxy
– ATS – Apache Traffic Server
• Webserver – Apache HTTPD – Nginx
• Application Server – Apache Tomcat / Jetty – JBoss
• RDBMS – MySQL
OSDConf 2014 : http://piyush.me 11
![Page 12: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/12.jpg)
• NoSQL – SOLR / ElasticSearch (Lucene) – Search – Cassandra – Couchbase
• Monitoring – Graphite / Diamond – Nagios – NOCMATE-Rx
• Lmetric – Logging Infrastructure – Syslog – Logstash – ElasticSearch – Kibana – Web Dashboard
OSDConf 2014 : http://piyush.me 12
![Page 13: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/13.jpg)
• Cheetah – Selenium , TestNG , Pyvnc2swf – Jenkins
• Neerikshan – Observium (network monitoring)
• Version Control – GIT, SVN , CVS – GitLab – Git Management
• Continuous Integration – Jenkins
• Performance Testing – Jmeter / ab / soapui
• IDE – Eclipse / vim
OSDConf 2014 : http://piyush.me 13
![Page 14: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/14.jpg)
• Virtualization – XenServer
• Configuration Management - Puppet • Application Frameworks
– Spring (JAVA) – Laravel (PHP) – Django / Flask (Python)
• Knowledge Management – Twiki / Wordpress
• CMS – Drupal
• Security – OSSIM / AlienVault – OWASP ZAP
• …..and many more ….
OSDConf 2014 : http://piyush.me 14
![Page 15: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/15.jpg)
Cheetah
• cheetah : Test automation and business service monitoring tool
– uses Selenium and Jenkins(Hudson) integration for Real Time monitoring of website with nice Reporting and Alerting
– record sessions using Pyvnc2swf
OSDConf 2014 : http://piyush.me 15
![Page 16: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/16.jpg)
Few other hacks
• BizEye SynC Tool
• MyClient
• Image Thumbnail Processing
– ImageMagick
• MyIdeas
– Drupal
OSDConf 2014 : http://piyush.me 16
![Page 17: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/17.jpg)
LBManager • Managing LTM and GTM using Django /
PyControl
• Single interface for managing the LoadBalancers hosted in multi-colocations / IDCs
OSDConf 2014 : http://piyush.me 17
![Page 18: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/18.jpg)
Graphite/Diamond & NOCMATE-Rx • Monitoring for Operational Excellence
– Easy interface to collect data / Metrics – Trends, Dashboards, Alerting , Reporting – User specific/LOB specific dashboards
• Desi principle: If it moves, we track it :) • “You cannot manage what you do not measure and
to measure you need to monitor” • 5 Categories of Metrics
– System | Network | Security | Business | Application
• Poller’s – DBPoller, SNMP, Script etc.
OSDConf 2014 : http://piyush.me 18
![Page 19: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/19.jpg)
Graphite : Trends / Analysis / Dashboards
OSDConf 2014 : http://piyush.me 19
![Page 20: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/20.jpg)
Lmetric (Logging Metric) • Centralized Log and event storage and search !
• Powered by :
– Logstash
– ElasticSearch
– Kibana – Web Dashboard
OSDConf 2014 : http://piyush.me 20
![Page 21: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/21.jpg)
Personare : BigData Stack • To collect valuable user interaction information for
subsequent use in personalization, recommendations, analysis and reporting.
• Layered Architecture (Lambda):
– Batch Layer : Hadoop M/R
– Speed Layer : Storm NRT Computation
– Serving Layer
OSDConf 2014 : http://piyush.me 21
![Page 22: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/22.jpg)
Personare : Architecture
OSDConf 2014 : http://piyush.me 22
![Page 23: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/23.jpg)
OSS powered BigData Stack • Log Collection: Logstash, Syslog-NG
• Distributed messaging system : Kafka , RabbitMQ
• ETL : Pentaho/ Talend
• Hadoop Ecosystem
• NRT Computation: Twitter Storm
• Document-Oriented NoSQL DB : Couchbase
• Distributed Search: ElasticSearch
• Distributed messaging system : Kafka , RabbitMQ
• NoSQL : Cassandra, Redis, Neo4J (Graph)
• API Management : WSO2 API Manager
• Programming Languages : Java , Python, R
OSDConf 2014 : http://piyush.me 23
![Page 24: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/24.jpg)
My next hope ! …goal ?
• Would I be able to apply what all I have learned in technology through OSS to solve some issues faced by masses around the world ..?
– Using technology to make world a better place for everyone !
and I am sure OSS will have major role to play in it !!
OSDConf 2014 : http://piyush.me 24
![Page 25: "In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014](https://reader034.vdocument.in/reader034/viewer/2022052618/55496322b4c905e94e8b5a27/html5/thumbnails/25.jpg)
Thanks!
Questions (If any ) ?
Contact:
@piykumar
http://piyush.me
OSDConf 2014 : http://piyush.me 25