node.js in a heterogeneous system
DESCRIPTION
TRANSCRIPT
![Page 1: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/1.jpg)
Node.js !in a heterogeneous system
Alexey Migutsky : Senior Software Developer @ Lohika
![Page 2: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/2.jpg)
What do we do?�
![Page 3: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/3.jpg)
What do we do?�PaaS for BigData analytics in cloud environments.
Ubix.io
![Page 4: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/4.jpg)
What do we use?�
![Page 5: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/5.jpg)
What do we use?�
Scala
Python
Bash
Javascript
![Page 6: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/6.jpg)
What do we use?�
BDAS (Spark + Shark)
AWS
Black Voodoo Magic™
Velocipede™
Brains
![Page 7: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/7.jpg)
What’s next?�
![Page 8: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/8.jpg)
What’s next?�
Node & Maven
CI
Velocipede™
Jenkins & Node testing
![Page 9: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/9.jpg)
?�
![Page 10: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/10.jpg)
Why Node.js?
![Page 11: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/11.jpg)
Why Node.js? Fast and easy Easy websockets support Plays nice with AWS We use it as a web-tier orchestrator
![Page 12: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/12.jpg)
Node & Maven
![Page 13: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/13.jpg)
Node & Maven We use maven (pom.xml) We use npm (packege.json) We have node+maven projects!
![Page 14: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/14.jpg)
Project structure
![Page 15: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/15.jpg)
![Page 16: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/16.jpg)
Project structure + Subprojects are maven modules + Subprojects use common code base + Easy « build process » + Easy deployments - Uber root package.json - Mangled dependencies
![Page 17: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/17.jpg)
Custom modules
![Page 18: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/18.jpg)
Custom modules Live in root /node_modules right now Planning to move to separate git repos Tested with subprojects
![Page 19: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/19.jpg)
Tests
![Page 20: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/20.jpg)
Tests Mocha + Sinon + Chai Sinon-chai + mocha-sinon Chai-as-promised Rewired Jenkins TAP plugin
![Page 21: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/21.jpg)
Test example describe('##getConfiguration', function () {! it('should return promise and resolve with data ', function (done) {! var promise = sysStore.getConfiguration('somekey');! promise.should.eventually.become({key: 'somekey'}) !
" .then(function () {! mockDb.getItem.should.be.calledWithMatch( !
! mockTables.system, !" " {key: sinon.match.string} !" "); !
}) ! .should.notify(done);! });! });
![Page 22: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/22.jpg)
Test Env How-to
![Page 23: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/23.jpg)
Test Env How-to Goals: - run from root using mvn test - run from sub using npm test - human-readable result logs - result >> in console and file - Jenkins support
http://www.mindmeister.com/346337860
![Page 24: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/24.jpg)
package.json
"devDependencies": { "mocha": "*", "sinon": "1.6.*", "chai": "*", "sinon-chai": "*", "mocha-sinon": "*", "chai-as-promised" : "*", "rewire" : "*" }, "scripts" : { "test" : "./runtest.sh" }
Test Env How-to
![Page 25: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/25.jpg)
runtest.sh
#!/bin/bash!mocha $(find test -name '*.js') -R tap | !tee test/testresults.tap; (exit ${PIPESTATUS[0]})
Test Env How-to
The whole command runs in a single line. Wrapped here for display purpose J
![Page 26: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/26.jpg)
pom.xml (maven-antrun-plugin)
<execution> ! <id>test-nodejs</id> ! <phase>test</phase> ! <configuration> ! <tasks name="Run mocha tests"> ! <exec dir="${basedir}”! executable="npm” ! failonerror="true"> ! <arg value="test"/>! </exec> ! </tasks> ! </configuration> ! <goals> ! <goal>run</goal> ! </goals> !</execution>
Test Env How-to
![Page 27: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/27.jpg)
Build process
![Page 28: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/28.jpg)
Build process
- No npm install on build server (too slooow)
- Fixed deps versions (update by-hand)
- All deployable deps are checked in!
- Build phase only copies deps in subprojects
http://www.futurealoof.com/posts/nodemodules-in-git.html
![Page 29: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/29.jpg)
Deployment
![Page 30: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/30.jpg)
Deployment
- Builds are stored at S3
- Copied to instances by orchestration tools
- Managed by upstart
![Page 31: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/31.jpg)
upstart (server.conf) #this should be put in /etc/init/server.conf on Ubuntu !description ”Node.js server" !author "Alexey Migutsky" !!# used to be: start on startup !# until we found some mounts weren't ready yet while booting: !start on started mountall !stop on shutdown !!# Automatically Respawn: !respawn !respawn limit 20 5 !!# Max open files are @ 1024 by default. Bit few. !limit nofile 32768 32768 !!script !
"# Not sure why $HOME is needed, but we found that it is: "!"export HOME="/root” !"exec /usr/local/bin/node /usr/local/server/app.js >> /var/log/server-err.log 2>&1 !
end script !!post-start script !end script
Deployment
http://kvz.io/blog/2009/12/15/run-nodejs-as-a-service-on-ubuntu-karmic/
![Page 32: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/32.jpg)
Deployment hint
Have a look at http://pm2.io/
![Page 33: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/33.jpg)
Continuous Integration
![Page 34: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/34.jpg)
![Page 35: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/35.jpg)
« hack some node.js » npm test
![Page 36: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/36.jpg)
push code
![Page 37: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/37.jpg)
push code
![Page 38: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/38.jpg)
push code
build & test
![Page 39: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/39.jpg)
push code
build & test
store
![Page 40: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/40.jpg)
push code
build & test
store
![Page 41: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/41.jpg)
push code
build & test
store
![Page 42: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/42.jpg)
push code
build & test
store
deploy
![Page 43: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/43.jpg)
That’s it!
![Page 44: Node.js in a heterogeneous system](https://reader034.vdocument.in/reader034/viewer/2022042613/5404c8528d7f729e768b48ce/html5/thumbnails/44.jpg)
That’s it!
Thanks
@mr_mig_by bit.ly/mr_mig