node.js at cloudkick - paul.querna.org · node.js • express • api • services • realtime....
TRANSCRIPT
![Page 1: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/1.jpg)
Node.js at CloudkickPaul QuernaJuly 26, 2011
![Page 2: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/2.jpg)
What is Cloudkick?
• Cloud Server Management
• Dashboard across providers
• Metadata container (tags, etc)
• Monitoring as a Service
• Acquired by Rackspace in December 2010
![Page 3: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/3.jpg)
Python, Python, Python!
• Django
• UI
• API
• Twisted
• Services
• Orbited
• Realtime
![Page 4: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/4.jpg)
Cloudkick Classic
Webapp
Orbited
RabbitMQ ServicesUser
MySQL Solr Cassandra
monitoring system (complicated)
![Page 5: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/5.jpg)
What’s wrong?
• Django vs Twisted
• Twisted in general
• RabbitMQ
• orbited is dead
• Multi-Region durability
![Page 6: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/6.jpg)
Node.js
• Express
• API
• Services
• Realtime
![Page 7: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/7.jpg)
New Architecture
Node.jsUser
Cassandra
monitoring system (complicated)
![Page 8: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/8.jpg)
In Production
![Page 9: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/9.jpg)
Running
• runit / daemontools
![Page 10: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/10.jpg)
Configuration
• Puppet
• local_settings.js (django style)
![Page 11: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/11.jpg)
Load Balancer
• apache 2.2, event mpm, mod_ssl, mod_proxy_balancer
• Want to remove it.
• ssl/tls termination (sad face)
• consider stunnel+haproxy. for now.
• examining stud+haproxy:
• https://github.com/bumptech/stud
![Page 12: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/12.jpg)
Internals
![Page 13: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/13.jpg)
package.json
![Page 14: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/14.jpg)
"async": "0.1.9", "cassandra-client": "0.2.1",
"express": "2.4.0", "ipv6": "0.0.1", "jade": "0.10.7",
"logmagic": "0.1.4", "long-stack-traces": "0.1.2",
"metrics-ck": "0.0.1", "node-dev": "0.1.5", "node-uuid": "1.2.0", "nodelint": "0.4.0", "optimist": "0.2.0",
"sax": "0.1.2", "showdown": "0.0.1",
"strtok": "0.1.1", "swiz": "0.3.1",
"thrift": "0.6.0-1", "whiskey": "0.4.0"
![Page 15: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/15.jpg)
npm, private registry
• Replicate Issac’s CouchDb
• ~/.npmrc
• always-auth = true
• registry = https://npm.example.com/
![Page 16: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/16.jpg)
npm
• Prevent Publishing:
• "private" : true,
• "publishConfig" : { "registry": "http://127.0.0.1:33333/__invalid__registry_do_not_publish"},
![Page 17: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/17.jpg)
Dealing with Patches
• Fork on github.
• Patch in branch.
• Publish to private NPM registry.
• Pray upstream accepts.
• Switch back to public package once released.
![Page 18: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/18.jpg)
Flow Control
• Async
• https://github.com/caolan/async
![Page 19: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/19.jpg)
Cassandra Client
• CQL
• It’s like SQL, but for NoSQL (troll face)
• Relational and Timeseries data
• https://github.com/racker/node-cassandra-client
![Page 20: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/20.jpg)
Logging
• Log Magic:
• https://github.com/pquerna/node-logmagic/
• logmagic.route("mylib.*", logmagic.DEBUG, "stdout")
![Page 21: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/21.jpg)
Metrics
• https://github.com/mikejihbe/metrics
• Metrics Broker:
• Collect across multiple processes
• (not open source yet, working on it)
![Page 22: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/22.jpg)
Testing
• Whiskey:
• https://github.com/cloudkick/whiskey
• http://cast-project.org/test-coverage/
![Page 23: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/23.jpg)
Linting
• node-jshint
• Google Closure Linter:
• http://code.google.com/p/closure-linter/
![Page 24: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/24.jpg)
Continuous Integration
• BuildBot against
• Production (v0.4.10-99b210d7eac)
• Stable Branch (v0.4.x)
• Dev (master / v0.5.x)
• (need to add --use-uv)
![Page 25: Node.js at Cloudkick - paul.querna.org · Node.js • Express • API • Services • Realtime. New Architecture User Node.js Cassandra monitoring system (complicated) In Production](https://reader036.vdocument.in/reader036/viewer/2022062414/5f76a56872056967d21b116e/html5/thumbnails/25.jpg)
Questions?
• Slides:
• http://paul.querna.org/slides/