drupal as a data purveyor, part ii

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Drupal as a Data Purveyor

Layering Drupal with emerging technologies to create a performant, scalable data purveyor.


ASTONISH DESIGNC H A N G E G A M E

Diana Montalion Dupuis@dianadupuis

Sam Heuck@samheuck

Andrew Elster@filmknurd


ASTONISH DESIGNC H A N G E G A M E

We have a new website: astonishdesign.com

@Astonish_Design


An open-source content management PHP/MySQL framework designed for rapid development of content-rich web applications.


Reasons (among many)

• Growth: evolving tool

• Foundation skill set

• User features


http://jdhancock.com/ • JD Hancock• http://farm9.staticflickr.com/8322/8031897271_9c63e48a29_b.jpg

http://jdhancock.com/

http://farm9.staticflickr.com/8322/8031897271_9c63e48a29_b.jpg


“Big” Data

"Big" in relation to the amount of content Drupal is designed

to manage


Volume


Velocity

BIKE WALLS• http://www.bikewalls.com

http://www.bikewalls.com


Variety


Content

•Every kind of data•Coming from everywhere


Current Issues

•Bootstrapping•Bottlenecks - Normalized - Join Happy•Lack of solution architecture


ViewsRapid

DevelopmentPlatform

Visitors& ContentAuthors

Miners

Who is the user?


Welcome to the future

When modularizing the architecture creates performant applications for all content and all users.


Traditional Optimizations


DuplicateData

RapidDevelopment

Platform

API

DenormalizedDocument Storage for READS

NormalizedRelational for WRITES

Data Mining

Content & DataCreation


Cost of Bootstrap• Hooks allow modules to

customize behavior

• Drupal calls hooks during each stage of bootstrap

• More modules, more hooks called, slower bootstrap


Massive Callstacks


Bootstrap on Every Request


ViewsRapid

DevelopmentPlatform


Miners

Backbone.js

Twisted

JS

PhotonVert.xCelluloid::IO

PYTHON

PHP

JAVA

RUBY


ViewsRapid

DevelopmentPlatform


Miners

Backbone.js

Twisted

JS


PYTHON

PHP

JAVA

RUBYSources


API and Node.JS


Why Proxy External Sources?

OK

your.domai

GET

OK

other.domain

OPTION

Yes

GET


RESTful

Robert Tadlock • http://www.flickr.com/photos/rtadlock/2716877199/

http://apigee.com/about/api-best-practices Since you are building the API, you can make it any way you want, but following some good basic REST conventions is a good idea.

It will make it easier to integrate with front end frameworks.

http://www.flickr.com/people/rtadlock/

http://www.shutterstock.com

http://apigee.com/about/api-best-practices


Pick a FrameworkExpress or Restify

•http://expressjs.com•http://mcavage.github.com/node-restify

http://expressjs.com

http://mcavage.github.com/node-restify


Pick a Connector

Tony Hisgett • http://www.flickr.com/photos/hisgett/

http://mongoosejs.com/Mongoose

Alternatives•Mongolia•Mongo Skin•Native MongoDB Driver

http://www.flickr.com/people/rtadlock/

http://www.flickr.com/photos/hisgett/6964614700/

http://mongoosejs.com


ExampleExpress & Mongoose


Create a Resource


Create an Endpointhttp://domain/resources

{ [ { “someField”: “blah blah blah” “anotherField”: 2 }

...

] }

http://api.domain.com/resources


On the Front-end


ViewsRapid

DevelopmentPlatform


Miners

Backbone.js

Twisted

JS


PYTHON

PHP

JAVA

RUBYSources


Isn’t JS Slow?Yes but....

1. JS and devices continue to improve

2. We've improved the overall round trip: faster data, smaller payload, no bootstrap. It's still faster overall.


Organize Your CodeOn the front-end, you could write everything as AJAX calls and jQuery DOM manipulation, but that will get messy very quickly as your application grows in sophistication.

In the interests of maintainability and sanity, use a framework that organizes your code and abstracts away the DOM. There are many to choose from. A new one probably came out this morning.


We use Ember because it is straightforward to use, powerful, flexible, and still fairly light weight. Backbone has been around for a while and is quite nice, but we like how Ember handles garbage collection and the other goodies it ships with.

What Why Howhttp://www.emberjs.com Why Discourse Uses Ember •Official Guides

•Resources•Fire Up Ember•Ember Camp•Darthdeus V. Ember•

http://www.emberjs.com

http://eviltrout.com/2013/02/10/why-discourse-uses-emberjs.html

http://www.emberjs.com/guides

http://accidentaltechnologist.com/javascript/33-great-resources-get-started-emberjs/

https://peepcode.com/products/emberjs

http://lanyrd.com/2013/ember-camp/

http://darthdeus.github.com/


Results fromRich Snippets

For Algebraix, all the data comes from an external source, so it makes sense to run the application on the client side, as opposed to importing the data into Drupal first.

It also makes the control panel feel more responsive. No more click and hope while waiting for the interface to receive data. Now we've created a conversation with the user.


Essential question ...

Why use Drupal?

Don’t use Drupal if you don’t need it.


The role of the themer

Front engineer


Questions?

drupal as a data purveyor, part ii

Technology