CONVERTING TO NODEHow I converted a medium sized Rails project to Node.js

BACKGROUND

• CTO of Ideal Candidate

• MVP existed when I joined

• Author of Haraka and a few other Node.js libraries

• Background in Perl and Anti-Spam

THE RAILS APP• Unicorn + Thin + Rails

• CoffeeScript with Angular.JS

• SASS+Compass for stylesheets

• HAML for templates

• MySQL + MongoDB

• Sidekiq for background tasks

WHY?

APP SIZE

• 5700 Lines of Ruby (170 files)

• 2500 Lines of CoffeeScript (119 files)

• 3800 Lines of SASS (42 files)

• 1500 Lines of HAML (100 files)

FINAL GOAL

• Express + minimal middleware

• PostgreSQL hand coded with referential integrity

• Jade templates

• SCSS for CSS

• Javascript + Angular.js frontend

HOW?It’s all about the information, Marty

RAILS GIVES YOU• Routes

• Middleware

• DB abstraction layer (aka Model)

• Views

• Controllers

• Configuration

• Plugins

• A directory structure

• Unit tests

EXPRESS GIVES YOU

• Routes

• Middleware

• That’s it

• (This scares a lot of people)

DIRECTORY STRUCTURElib - general JS libraries ⌞ model - database accessors public - files that get served to the front end ⌞ assets - files that are considered part of the app ⌞ javascripts ⌞ stylesheets ⌞ images routes - files that provide express routes for HTML ⌞ api/v1 - files providing REST API endpoints (JSON) views - Jade templates app.js - entry point

RAILS ASSET PIPELINE

• Rails does a lot for you, but it’s f ’ing confusing

• HAML Template Example:

• = javascript_include_tag :application

• You’d think this adds <script src=“application.js”>, but no.

RAILS ASSET PIPELINE

• Easy if you understand it

• Too much magic for my liking

• But… the overall effect is good. So I copied some of it.

JADE EQUIVALENT

each script in javascripts script(src=script, type=“text/javascript”)

APP.JS:

var js = find.fileSync(/\.(coffee|js)$/, __dirname + ‘/public/assets/javascripts/controllers') .map(function (i) { return i.replace(/^.*\/assets/, ‘/assets') .replace(/\.coffee$/, ‘.js') });app.all('*', function (req, res, next) { res.locals.javascripts = js; next();});

SERVING COFFEESCRIPT// developmentapp.use(require('coffee-middleware')({ src: __dirname + '/public', compress: false,}));!// Productionvar all_js = coffee_compiler.generate(js, true);!var shasum = crypto.createHash('sha256');shasum.update(all_js);var js_sha = shasum.digest('hex');!js = ['/assets/javascripts/application-' + js_sha + '.js'];!app.get('/assets/javascripts/application-' + js_sha + '.js', function (req, res) { res.type('js'); res.setHeader('Cache-Control', 'public, max-age=31557600'); res.end(all_js);});

SERVING SCSS

• Node-sass module serves .scss files

• Doesn’t serve up .sass files (weird, huh?) !# mass-convert.bashfor F in `find . -name \*.sass`; do O=“${F/.sass/.scss}” sass-convert -F sass -T scss “$F” “$O” git rm -f “$F” git add “$O”done

SCSS APP.JSfunction compile_css () { console.log("Recompiling CSS"); resources.watchers.forEach(function (w) { w.close(); }); resources.watchers = [];! resources.css = sass.renderSync({ file: __dirname + '/public/assets/stylesheets/application.scss', includePaths: [__dirname + '/public/assets/stylesheets'], });! var shasum = crypto.createHash('sha256'); shasum.update(resources.css); var css_sha = shasum.digest('hex');! resources.stylesheets[0] = '/assets/stylesheets/application-' + css_sha + '.css'; resources.stylesheets[1] = '//fonts.googleapis.com/css?family=Open+Sans:400,700,800,300,600';! find.fileSync(cssregexp, __dirname + '/public/assets/stylesheets').forEach(function (f) { resources.watchers.push(fs.watch(f, compile_css)); })}

SCSS APP.JS

compile_css();!// Dev and Production the sameapp.get(/\/assets\/stylesheets\/application-(\w+)\.css/, function (req, res) { res.type('css'); res.setHeader('Cache-Control', 'public, max-age=31557600'); res.end(resources.css);});

ADDING COMPASS

• Ruby CSS framework

• Luckily only the CSS3 part used

• CSS3 code is just SASS files

• Once I figured this out, copied them into my project, et voila!

CONVERTING HAML TO JADE• Both indent based

• HAML: %tag{attr: “Value”} Some Text .person .name Bob

• JADE: tag(attr=“Value”) Some Text .person .name Bob

• Other subtle differences too

CONVERSION TOOL: SUBLIME TEXT

• Afterthought: I should have written something in Perl

• Regexp Find/Replace

• ^(\s*)% => $1 (fix tags)

• (\w)\{(.*)\} => $1($2) (attribute curlys)

• (\w):\s*([“‘]) => $1=$2 (attribute key: value)

THINGS LEFT TO FIX• Helpers: = some_helper

• Text on a line on its own - Jade treats these as tags

• Nested attributes: %tag{ng:{style:’…’,click:’…’},class:’foo’} -> tag(ng-style=‘…’, ng-click=‘…’, class=‘foo’)

• Making sure the output matched the Ruby/HAML version was HARD - HTML Diff tools suck

HELPERS BECAME MIXINS

• Standard Rails Helpers: = form_for @person do |f| f.label :first_name f.text_field :first_name %br! f.label :last_name f.text_field :last_name %br! f.submit

• Custom Rails helpers stored in app/helpers/ folder • http://api.rubyonrails.org/classes/ActionView/Helpers/FormHelper.html

JADE MIXINS• Very powerful. Poorly documented. • Standalone or can have block contents // implementation mixin form(action) form(accept-charset="UTF-8", action=action, method="POST") input(name="utf8" type="hidden" value="&#x2713;") block // usage: +form(‘/post/to/here’) input(type=“text”,name=“hello”,value=“world”)

• Supports js code mixin app_error(field) if (errors && errors[field]) div(class="err_msg " + field) each error in errors[field] span= error

JADE: NO DYNAMIC INCLUDES

• HAML/Helpers can do: = popup ‘roles/new'

• Jade needed: +popup(‘roles/new’) include ../roles/new

CREATING THE MODEL• Hand coded db.js layer developed over time from previous

projects

• One file per table (generally) in lib/model/*.js "use strict"; var db = require('./db'); exports.get = function get (id, cb) { db.get_one_row("SELECT * FROM Answers WHERE id=$1", [id], cb); } exports.get_by_submission_id = function (submission_id, cb) { db.query("SELECT * FROM Answers\ WHERE submission_id=$1\ ORDER BY question_id", [submission_id], cb); }

WHY CONTROL THE SQL?exports.get_avg_team_size = function (company_id, role_id, months, cb) { var role_sql = role_id ? ' AND role_id=$2' : ' AND $2=$2';! if (months == 0 || months == '0') { months = '9000'; // OVER NINE THOUSAND }! var sql = "SELECT avg(c) as value\ FROM (\ SELECT g.month, count(e.*)\ FROM generate_series(\ date_trunc('month', now() - CAST($3 AS INTERVAL)),\ now(),\ INTERVAL '1 month') g(month)\ LEFT JOIN employees e ON e.company_id=$1" + role_sql + "\ AND (start_date, COALESCE(end_date, 'infinity'))\ OVERLAPS\ (g.month, INTERVAL '1 month')\ GROUP BY g.month\ HAVING count(e.*) > 0\ ) av(month,c)"; db.get_one_row(sql, [company_id, role_id, months + " months"], cb);}

AND FINALLY…• Routes

• Run Rails version of App

• Open Chrome Dev Console Network tools

• Hit record

• Find all routes and implement them

CREATING ROUTES/MODELS

• While I glossed over this, it was the bulk of the work

• Each endpoint was painstakingly re-created

• This allowed me to get a view of the DB layout

• And fix design bugs in the DB layoutfind.fileSync(/\.js$/, __dirname + '/routes').forEach(function (route_file) { require(route_file);});

BACKGROUND TASKS

• Currently using Sidekiq, which uses Redis as a queue

• Used for downloading slow data feeds

• Node.js doesn’t care if downloads are slow

• So I punted on background tasks for now

• If I need them later I will use Kue (see npmjs.org)

DEPLOYMENT

• Linode + Nginx + Postgres 9.3.1 + Runit + Memcached

• /var/apps and deploy_to_runit for github autodeploy

• Monitoring via Zabbix

• 60M used vs 130M for Rails

NEXT STEPS

• Convert coffeescript code to plain JS - I find coffeescript too much of a pain

• Implement graceful restarts using cluster

• Consider porting CSS to Bootstrap so we get mobile support