cartodb inside out

CartoDBInside Out

Jorge Sanz [email protected] @xurxosanzSASIG OSGeo.pt Lisbon 2015

A technical overview of a web mapping platform

Agenda Why

CartoDB nowadays

Architecture of the platform

Components

Setup

How to serve dynamic data at scale

No maps in this talk sorry

https://congosto.cartodb.com/viz/e5da12e2-9fe7-11e4-bc43-0e853d047bba/embed_map

https://congosto.cartodb.com/viz/e5da12e2-9fe7-11e4-bc43-0e853d047bba/embed_maphttps://congosto.cartodb.com/viz/e5da12e2-9fe7-11e4-bc43-0e853d047bba/embed_map

http://elpais.com/elpais/2015/08/20/media/1440070260_752772.html

http://elpais.com/elpais/2015/08/20/media/1440070260_752772.htmlhttp://elpais.com/elpais/2015/08/20/media/1440070260_752772.html

Why @jatorre and @saleiva start Vizzuality in April 2008

Company focused on visualization projects

Every geospatial project had the same stack:

Database + Map Server + APIs + client code

CartoDB: internal product to deploy faster this geo-stack

Mission

With CartoDB everybody can analyse,visualise and share insights on location data.We are democratising location intelligence

CartoDB nowadays A VC founded company of ~70 people (NY + Madrid)

A leading mapping service with +130.000 users

An Open Source product that can be deployed

on the cloud or on just one server (on-premises)

Full stack solution from storage to client side libraries

Its a mapping editor supported by a platform (APIs)

CartoDB architecture

http://www.di.unito.it/~schifane/smellymaps/explorer.html

http://www.di.unito.it/~schifane/smellymaps/explorer.htmlhttp://www.di.unito.it/~schifane/smellymaps/explorer.html

https://team.cartodb.com/u/piensaenpixel/viz/8a6530e0-5239-11e5-9cc4-0e0c41326911/public_map

https://team.cartodb.com/u/piensaenpixel/viz/8a6530e0-5239-11e5-9cc4-0e0c41326911/public_maphttps://team.cartodb.com/u/piensaenpixel/viz/8a6530e0-5239-11e5-9cc4-0e0c41326911/public_map

The CartoDB platform

Spatial database for users NoSQL database for

configurations and cache Rendering, imports,

database and other services Cache and Content

Delivery Network JavaScript Library Map Editor

Platform

On-premisesor in the Cloud

Data servicesYour data

Editor

The CartoDB platform

CartoDB.js Odyssey.jsTorque.js Leaflet /Gmaps

PostgreSQL

PostGIS

SQL API

Redis

Maps API

TorqueMapnik

Varnish Caching

Fastly CDN

Car

toD

B o

n A

maz

on

el

asti

c R

ecip

es

Car

toD

B o

n

Am

azo

n e

last

ic

Rec

ipes

PostgreSQL

PostGIS

CartoDB Server API

Rest API Imports API

CartoDB Editor

Browser

Cloud setup 1 - load balancer (nginx)

1 - caching (varnish)

N - UI servers (CartoDB UI)

N - database servers (user data)

1 - Metadata server (CartoDB metadata)

1 - redis server (+slave)

N - Maps API + API SQL servers

Postgres + Postgis Geodata, users and editor metadata storage

Tiny Well Known Binary

CartoDB - Postgres extension

Redis key-value highly performant storage

configuration for APIs

cache

Backend services Ruby on Rails

Import API (ogr2ogr)

Background management tasks

Invalidation service Node.js + HA Proxy + Consul

Invalidates caches by request

Multi-agent, replicated

SQL API node.js

Exposes an entry point for running queries

Acts as an export API

Maps API Windshaft node.js + mapnik

renders tiles from CartoCSS + SQL

SQL API client

nginx web server

load balancer

Varnish + Fastly/GCP web cache

Content Delivery Network

CartoDBMap Editor

the easiest web mapping tool

ever

Map Editor Ruby on Rails + JavaScript

Consumes the Imports, Maps, SQL API and its own API

Highly focused on UX-UI

Manage datasets (schema, privacy, etc)

Manage maps (basemap, infowindows,

other visual elements, privacy, etc)

Define layer data source with filters and SQL editor

Define symbology through wizards and CartoCSS editor

Publish maps as embeds, links or with CartoDB.js

CartoDB.js Based on jQuery + Backbone + Underscore

Leverages Leaflet or Google Maps API

Easy interaction with the SQL and Maps API

Manages transparently utf-grids for interactivity

Allows easy customizations on infowindows,

layer SQL and CartoCSS definitions, etc

Serving data at scale

https://scordivano.cartodb.com/viz/f55a3268-0c29-11e3-827e-5fd91394b353/embed_map

https://scordivano.cartodb.com/viz/f55a3268-0c29-11e3-827e-5fd91394b353/embed_maphttps://scordivano.cartodb.com/viz/f55a3268-0c29-11e3-827e-5fd91394b353/embed_map

http://mappingkat.github.io/indian-removal/

Story of a tile: first request

Story of a tile: next request

Story of a tile: insert & next request

Cache invalidation

Write queries fire a trigger that updates

cdb_tablemetadata and starts the invalidation

process for the tiles that use that table

Summary CartoDB is an Open Source, full stack mapping product

Runs on a single server or distributed

Delivers to thousands of users the superpowers of Postgis

and Mapnik, not just pretty maps but also new insights

Provides tools for users and developers

Its designed to scale, but we are always improving it

We are hiring!!

http://cartodb.com/jobs

http://cartodb.com/jobshttp://cartodb.com/jobs

CartoDBInside Out

Jorge Sanz [email protected] @xurxosanzSASIG OSGeo.pt Lisbon 2015http://bit.ly/cartodb-sasig2015

Muito obrigado! Questions?

http://bit.ly/cartodb-sasig2015http://bit.ly/cartodb-sasig2015

Links https://cartodb.com

https://github.com/CartoDB/cartodb

https://github.com/CartoDB/cartodb.js

https://github.com/CartoDB/cartodb-postgresql

https://github.com/CartoDB/Windshaft-cartodb

https://github.com/CartoDB/CartoDB-SQL-API

http://docs.cartodb.com

http://cartodb.readthedocs.org

https://cartodb.comhttps://cartodb.comhttps://github.com/CartoDB/cartodbhttps://github.com/CartoDB/cartodbhttps://github.com/CartoDB/cartodb.jshttps://github.com/CartoDB/cartodb.jshttps://github.com/CartoDB/cartodb-postgresqlhttps://github.com/CartoDB/cartodb-postgresqlhttps://github.com/CartoDB/Windshaft-cartodbhttps://github.com/CartoDB/Windshaft-cartodbhttps://github.com/CartoDB/CartoDB-SQL-APIhttps://github.com/CartoDB/CartoDB-SQL-APIhttp://docs.cartodb.com/http://docs.cartodb.com/http://cartodb.readthedocs.orghttp://cartodb.readthedocs.org