from big gorm-centered into a cloud of fast redis nodes

From bigGORM-centeredinto a cloud offast nodes

Jorge Uriarte Aretxaga@jorgeuriarte

http://www.slideshare.com/gailen/

About me

About me

@jorgeuriartejorge.uriarte@gailen.es

Origins

In the beginning...

In the beginning...

Well-known territory

Well-known territory

JVM based!

Well-known territory

Old good SQL& RDBMs

JVM based!

Well-known territory

Old good SQL& RDBMs

JVM based!

GORM magic!

Productive development environment

Productive development environment

Affordable learning curve

Productive development environment

Affordable learning curve

Good-enough performance

Evolution

“Mmmhhh... I wouldn’t build it that way

nowadays...”Enrique Amodeo

“Mmmhhh... I wouldn’t build it that way

nowadays...”Enrique Amodeo

“Mmmhhh...nowadays...

neither would I!”Me

“Mmmhhh... I wouldn’t build it that way

nowadays...”Enrique Amodeo

New brands& countries

Reality Check

Reality bites

Reality bites

-server -Xmx1024m -XX:MaxPermSize=128m -XX

:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:

+UseTLAB -XX:+CMSIncrementalMode

-XX:+CMSIncrementalPacing -XX:CMSIncrement

alDutyCycleMin=0 -XX:CMSIncrementalDutyCyc

le=10 -XX:MaxTenuringThreshold=0

-XX:SurvivorRatio=256 -XX:CMSInitiatingOcc

upancyFraction=60 -XX:+DisableExplicitGC

-server -Xmx1024m -XX:MaxPermSize=128m -XX

:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:

+UseTLAB -XX:+CMSIncrementalMode

-XX:+CMSIncrementalPacing -XX:CMSIncrement

alDutyCycleMin=0 -XX:CMSIncrementalDutyCyc

le=10 -XX:MaxTenuringThreshold=0

-XX:SurvivorRatio=256 -XX:CMSInitiatingOcc

upancyFraction=60 -XX:+DisableExplicitGC -server

-Xms3g -Xmx4g

-XX:MaxPermSi

ze=128m -XX:+U

seParNewGC

-XX:+UseConcMa

rkSweepGC -XX:

+UseTLAB -XX:+

CMSIncremental

Mode

-XX:+CMSIncrem

entalPacing -X

X:CMSIncrement

alDutyCycleMin

=0

-XX:CMSIncreme

ntalDutyCycle=

10 -XX:MaxTenu

ringThreshold=

0

-XX:SurvivorRa

tio=256 -XX:CM

SInitiatingOcc

upancyFraction

=60

-XX:+DisableEx

plicitGC

Productivity slows down...

Fragility...

Evolution stopped...

Slow development

Slow development

Fragile system... prone to errors

Slow development

Fragile system... prone to errors

Tuning hell with JVM heap

Slow development

Fragile system... prone to errors

Tuning hell with JVM heap

Terracota cluster issues

Slow development

Fragile system... prone to errors

Tuning hell with JVM heap

Terracota cluster issues

Overall excessive complexity

Change?Surgery?Refactor?

keep tuning?

Meanwhile... in another mental thread...

A solution looking for a problem?

But get back to the previous point...

Will you dare jump off it?

But get back to the previous point...

• New side-project• Potentially deadly traffic• Still to be tighly connected with our core system

• New side-project• Potentially deadly traffic• Still to be tighly connected with our core system•Solutions?

•Dedicated system?•More machines & tuning?•Catch that train!

Proof of concept and... JUMP!

Some details?

Prototype- show me the code -

RedisFast Controllers

Ajax (fewer requests)

Fast importer from existing XMLs

Quick & dirty (1w)

Groovy!!!(XML parsing, JSON

rendering, grails controllers, ...)

Elasticsearch

• Simply groovy.util.XmlSlurper

• Simply groovy.util.XmlSlurper• Two importers, really

• Simply groovy.util.XmlSlurper• Two importers, really• Bulk

• Simply groovy.util.XmlSlurper• Two importers, really• Bulk• Incremental (continuous)

• Simply groovy.util.XmlSlurper• Two importers, really• Bulk• Incremental (continuous)

• Born as a temporary hack...

• Simply groovy.util.XmlSlurper• Two importers, really• Bulk• Incremental (continuous)

• Born as a temporary hack...soon to give way to a better system

• Redis master/slaves replication

• Redis master/slaves replication• Fast

• Redis master/slaves replication• Fast• Trivial

• Redis master/slaves replication• Fast• Trivial• Fast

• Redis master/slaves replication• Fast• Trivial• Fast• Trivial

• Redis master/slaves replication• Fast• Trivial• Fast• Trivial• Did I mention fast and trivial?

Deadly simple Grails app

Deadly simple Grails app

Deadly simple Grails app

Deadly simple Grails app

“Look mum, no GORM!”

From ‘searchable’ to ElasticSearch

• Fat-trimmed new fronts• Fast, *light* application• Information still safe in the core system

• Much more capacity with less resources• Simple integration

Wait... where did the red arrows go?

New system

Old system

Subdomain based write-through

Subdomain based write-through

• Login, session, tracking...

Subdomain based write-through

• Login, session, tracking...• Purchases, payments, subscriptions

Subdomain based write-through

• Login, session, tracking...• Purchases, payments, subscriptions• Everything goes to our *good old core system*

Subdomain based write-through

• Login, session, tracking...• Purchases, payments, subscriptions• Everything goes to our *good old core system*• ACID, consistency, GORM, SQL, transactions...

Subdomain based write-through

• Login, session, tracking...• Purchases, payments, subscriptions• Everything goes to our *good old core system*• ACID, consistency, GORM, SQL, transactions...• ...but now with a small fraction of the load

Evaluation

Evaluation

Terracota/Clustering approach vs “new thing”

Evaluation

Terracota/Clustering approach vs “new thing”

Memory, speed, size, codebase

Evaluation

Terracota/Clustering approach vs “new thing”

Memory, speed, size, codebase

Decoupled search system

Evaluation

Terracota/Clustering approach vs “new thing”

Memory, speed, size, codebase New problems,

synchronization

Decoupled search system

Productivity increasing again...

New landscape

New landscapeGet rid of XML importer(event driven pub/subs...?)

New landscapeGet rid of XML importer(event driven pub/subs...?)

Delete old code (yeah!)

New landscapeGet rid of XML importer(event driven pub/subs...?)

Keep pushing for speed

Delete old code (yeah!)

New landscapeGet rid of XML importer(event driven pub/subs...?)

Keep pushing for speed

Delete old code (yeah!)

Keep it simple (this time)

Just ask me... in two years

@jorgeuriarte

External sources

PITAhttp://www.quicksales.com.au/ad/p-i-t-a-pain-in-the-ass-car-number-plate/2606513

Hamsterhttp://www.flickr.com/photos/sualk61/3117477410/

Cliff viewhttp://www.flickr.com/photos/epsos/4376727123/sizes/l/

Knight in the white horsehttp://www.flickr.com/photos/photoplod/7982134342/

Snailhttp://www.flickr.com/photos/raphaelquinet/693813637/

Fast snailhttp://www.flickr.com/photos/zorro13/4086173313

Timanfaya devilhttp://www.flickr.com/photos/yelacis/6941891989/

Light patternshttp://www.flickr.com/photos/aerosolhalos/6676192013/

Kick in the top of the cliffhttp://www.flickr.com/photos/4elevenpix/5056089861/

Homer evolutionhttp://www.flickr.com/photos/photonquantique/1858685882/

Lego machinehttp://www.flickr.com/photos/djimison/3058818283/

Landscapehttp://www.flickr.com/photos/blmiers2/6112610781/

Examhttp://www.flickr.com/photos/albertogp123/5843577306/

Future is pasthttp://www.flickr.com/photos/tind/7533681980/

"From big GORM-centered into a cloud of fast nodes" por Jorge Uriarte se encuentra bajo una

Licencia Creative Commons Atribución-CompartirIgual 3.0 Unported.