coscup 2013 : continuous integration on top of hadoop
Post on 10-May-2015
945 Views
Preview:
DESCRIPTION
TRANSCRIPT
Continuous Integration on top of Hadoop
Wisely Chen and Neal Lee
Saturday, August 3, 13
Agenda
• Who I am
• Problem
• Solution
• Demo
• Q&A
Saturday, August 3, 13
Who I am
• Wisely Chen ( thegiive@gmail.com )
• Release manager of Yahoo![Taiwan] shopping and data team
• Loves to promote open source tech in Taiwan
• Hadoop Summit 2013 San Jose
• Ruby and Rails : Coscup 2006, Ubisunrise 2007, OSDC 2007
• Puppet : PHPConf 2012 , RubyConf 2012
• Release Practice : Webconf 2013, Coscup 2012
Saturday, August 3, 13
Who I am
• Neal Lee (@neal_lee)
• Data Engineer at Yahoo![Taiwan] data team
• Aims to build a easy to use self-service BI platform connecting to Hadoop.
Saturday, August 3, 13
EC Data Team拍賣/商城/購物中心
站台流量/點擊/使用者行為 追蹤
Transactional data Tracking data
DataHighway
Data Warehouse/Data Mart
DataInfra BI
Platform
Report
RecommendationAPI
Machine Learning
Serve
Saturday, August 3, 13
Problem : Debug
Saturday, August 3, 13
Problem : Performance
Saturday, August 3, 13
Solution
Saturday, August 3, 13
Continuous Integration
Saturday, August 3, 13
Continuous Integration
• A software engineering practice • Maintain code repos
• Automate the build
• Make the build self-testing
• Everyone commit to the baseline everyday
• Every commit should be a build
• Test in a clone of production environment
• Make it easy to get the latest deliverables
• Everyone can see the result of latest build
• Automate deployment
Saturday, August 3, 13
We focus on• A software engineering practice
• Maintain code repos
• Automate the build
• Make the build self-testing
• Everyone commits to the baseline everyday
• Every commit should be a build
• Test in a clone of production environment
• Make it easy to get the latest deliverables
• Everyone can see the result of latest build
• Automate deployment
Saturday, August 3, 13
CI on Hadoop Flow
CodeUnitTest
PerformanceTest Deploy Doc Execution
Saturday, August 3, 13
One Click Deploy
CommitUnitTest
PerformanceTest Deploy Doc Execution
Saturday, August 3, 13
Toolset
CommitUnitTest
PerformanceTest Deploy Doc Execution
Vaidya
BASH
Saturday, August 3, 13
System diagram
CI Master
GitHub
Alpha
CI Slave
Beta Cluster
HadoopJobTracker
CI Slave Hadoop nodeHadoop
nodeHadoop
nodeHadoop
nodeSlave Node
Prod ClusterGateway
Saturday, August 3, 13
Unit Test
CommitUnitTest
PerformanceTest Deploy Doc Execution
Saturday, August 3, 13
PigUnit
• A simple xUnit framework
• No cluster set up is required in local mode
• Unit testing, regression testing, and rapid prototyping on the fly
Saturday, August 3, 13
Using PigUnit
• After
• Coding
• Write PigUnit test case
• Run local PigUnit test
• Push to cluster
• Run Pig on cluster
• Get right result !
• Before
• Coding
• Manual local test
• Push to cluster
• Run Pig on cluster
• Get right result !
Saturday, August 3, 13
Unit test is live doc
• Unit test is runnable live doc
• Pass test case and meet previous requirement
Saturday, August 3, 13
Flexible
• Pig can use PigUnit
• MapReduce can use MapUnit
• Hive can use hive_test
Saturday, August 3, 13
Performance Test
CommitUnitTest
PerformanceTest Deploy Doc Execution
Saturday, August 3, 13
Vaidya
• Rule based performance diagnosis of M/R jobs
• Extensible framework
• You can add your own rules
• Write complex rules using existing rules
Saturday, August 3, 13
Performance Test
Pig JobPig Job History
Vaidya
VaidyaRule
4
Pig Job Conf
NotifyUser
3
Performance result
Next CI Stage
1
1
22
2
5
1. Exec pig job with sampling data on beta server2. Vaidya read job history,conf,rule to check performance problem3. If ok, create performance result4. If job has performance issue, notify user5. Go to next CI stage
Sampling data
1
Saturday, August 3, 13
Vaidya Rule<Diagnos)cTest>
<Title><![CDATA[Balanaced Reduce Partitioning]]></Title>
<ClassName> <![CDATA[ org.apache.hadoop.vaidya.postexdiagnosis.tests.BalancedReducePar77oning ]]></ClassName>
<Descrip)on> <![CDATA[This rule tests as to how well the input to reduce tasks is balanced]]></Descrip)on>
<Importance><![CDATA[High]]></Importance>
<SuccessThreshold><![CDATA[0.20]]></SuccessThreshold>
<Prescrip)on><![CDATA[advice]]></Prescrip)on>
</Diagnos)cTest>
See if the reduce job is balance or not
Rule importance
Diagnose success threshold
Test Java Class
Saturday, August 3, 13
Deploy
CommitUnitTest
PerformanceTest Deploy Doc Execution
Saturday, August 3, 13
Deploy
• Deploy to production cluster
• Easy to rollback
• Create a git tag
• Auto doc generating
• Each release should map to a ticket
• Auto comment in Bugzilla
Saturday, August 3, 13
Auto comment in bugzilla
Repo url
Release Note
Issue statuschange
Saturday, August 3, 13
Auto create git tag
Release Note[Bug xxx] log....
Git Tag
Saturday, August 3, 13
DEMO
Saturday, August 3, 13
Demo
• Demo1 : Unit test fail
• Demo2 : Unit test success
• Demo3 : Check performance test
• Demo4 : Auto generate Doc
• Demo5 : Notify user
Saturday, August 3, 13
Demo
Saturday, August 3, 13
Conclusion
• CI will revolutionize your workflow
• CI will boost your productivity
Saturday, August 3, 13
Saturday, August 3, 13
Logic Debug
• Map/Reduce job oJen takes a lot of )me for execu)on
• Repeated Map/Reduce execu)on cost a lot of )me during logic debugging phase
• Need a way to find out logic problem before execu)on produc)on job
•Coding
ManualTest
Exec
Get Bug
Saturday, August 3, 13
Performance
• Map/Reduce performance is hard to es)mate before execu)on
• Production Grid computing resource is shared by all Yahoos
• Bad performance will affect other Yahoos Grid jobs
• Putting bad performance code on production grid is guilty
• We manually investigate the job performance before we actually execute it on production Grid
CodingManual
TestManual
investgateGet Bug
Saturday, August 3, 13
top related