more than websites: php and the firehose @datasift (2013)

And The Firehose

More Than Websites

Saturday, 23 March 13

@Introduce YourselvesSaturday, 23 March 13

@@stuherbert

What is

Sift throughsocial data

Twitter firehose, Facebook, bitly clicks,news, videos, comments

and more

Gain insights using augmentations

Language, gender, trends, links,sentiment, salience & entity analysis

and more

RealtimeGet matching data within

secondsof it being posted

HistoricsSearch our social data archive

going back toJanuary 2010

Pull the datafrom our servers

via HTTP/1.1 streamingor websockets

Let us pushdata to you

Have the data delivered directlyto your servers

or into your databases

in numbers

30Sources of social data

and data augmentations

Up to 20,000Number of new pieces of data

ingested into DataSiftevery second

3 TerabytesAmount of new data added

to the Historics archiveevery week

12Different wayswe can deliver

data to you

1Average number of seconds

to pass the datathrough DataSift

12Number of servicesdata passes through

inside DataSift

25Number of engineerswho write code for

the DataSift platform

5Primary programming languages:C++, Node, PHP, Python, Scala

154Private GitHub repos

@Our GitHub Repositories

Java & Scala

C & C++

JS & Node

Unclassified

Python

Shell Script

0 15 30 45 60

Architecture

Three majordata pipelines

+ supporting services

Data ArchivingAdds new data to the

Historics Archive

Filtering PipelineFiltering and delivery of data

in realtime

Playback PipelineFiltering and delivery of data

from the Historics Archive

@DataSift Technical Architecture

UltrahoseArchiver

PickleNode

Node Shard

PickleNode

Node Shard

ACL(with interaction

counter)

Stream Recorder

MonitoringAggregator

EDRs(licensed contentmetrics)

ControlChannels

UltrahoseArchiver

100%Prism

PickleFilteringEngine

Twitter

FacebookWikipedia

RedditLexisNexisMeltwaterEstimize

@lorenzoalbertonDataSift Architecture 2.2

Links Resolution+ OpenGraph

+ Twitter Cards+ Metadata

DeletesProcessor

Input Streams

NewsCredBoardReader

MySpaceSuperFeeder

Augmentation Pipeline

PickleNode

Node Shard

PickleNode

Node Shard

MonitoringKafkaQueue

EventsStorage

counter)

tracker

LimitManager

AuthenticationManager

NotificationService

APIStream .Manager .DB

Definition .Manager .DB

CSDL Compiler,Validator,

Normaliser

HistoricsScheduler

RecordingScheduler

PushScheduler

InteractionTargets

Mapping

FilteringTardis Pickle

InteractionTargets

Mapping

Hadoop

Titan Historics

Map/Reduce

HBase Cluster

Region 1 Region 2 Region N...

Data Node Data Node Data Node Data Node Data Node100%

StopPUB

License Manager DB

Billing Pipeline DB

MaskManager DB

ConnectionManager

Time Machine + InsightsPost-Processing, Stream Analytics

jobsDB

chunksDB

chunkselector

jobtracker

WorkerSnapshotter

Buffered Streams Redis

Worker

NodeMeteor

Real-timeStreams

HTTP Request

GET batch

PUSHScheduler

subscription X

subscription Y

job queuePUSHProducer

SubscriptionsDB

PUSHDelivery

HTTP(S) POST(S)FTP

Amazon S3DynamoDB

Microsoft AzureMongoDB

Exports andAnalytics

WebSockets

HTTPStreaming

Delivery Subscriptions

ConnectionsStorage

kafka-consumer

Oracle

Stream results

CouchDB

PickleDB .DB

KafkaKafka

Historical Queries

@datasift

Goblin Head

Goblin HeadGoblin TailGoblin Tail

Goblin Tail

Interaction GenerationInteraction Generation

3rd party APIs

Demographics Trends Analysis

Sentiment Analysis

Named Entities

Topics Analysis

Language Detection

KloutScore + Profile

OgreOgreOgreOgreOgreOgre

IBM Cognos

HDFSArchiver

Data ingestion + Augmentation

Bit.ly

StreamSplitter/Joiner

Deduper

Msg splitter

Google BigQuery

Stream results

Cloud Storage

BI tools

@Filtering Pipeline

UltrahoseArchiver

PickleNode

Node Shard

PickleNode

Node Shard

counter)

Stream Recorder

ControlChannels

UltrahoseArchiver

100%Prism

Twitter

FacebookWikipedia

DeletesProcessor

Input Streams

NewsCredBoardReader

MySpaceSuperFeeder

PickleNode

Node Shard

PickleNode

Node Shard

EventsStorage

counter)

tracker

LimitManager

NotificationService

Normaliser

HistoricsScheduler

RecordingScheduler

PushScheduler

InteractionTargets

Mapping

InteractionTargets

Mapping

Hadoop

Titan Historics

Map/Reduce

HBase Cluster

StopPUB

License Manager DB

Billing Pipeline DB

MaskManager DB

ConnectionManager

jobsDB

chunksDB

chunkselector

jobtracker

WorkerSnapshotter

Worker

NodeMeteor

Real-timeStreams

HTTP Request

GET batch

PUSHScheduler

subscription X

subscription Y

SubscriptionsDB

PUSHDelivery

HTTP(S) POST(S)FTP

Amazon S3DynamoDB

WebSockets

HTTPStreaming

ConnectionsStorage

kafka-consumer

Oracle

Stream results

CouchDB

PickleDB .DB

KafkaKafka

Historical Queries

@datasift

Goblin Head

Goblin Tail

3rd party APIs

Sentiment Analysis

Named Entities

Topics Analysis

Language Detection

IBM Cognos

HDFSArchiver

Bit.ly

Deduper

Msg splitter

Google BigQuery

Stream results

Cloud Storage

BI tools

Twitter

FacebookWikipedia

Digg Links Resolution+ OpenGraph

DeletesProcessor

Input Streams

NewsCredBoardReader

MySpaceSuperFeeder

Goblin Head

Goblin Tail

3rd party APIs

Sentiment Analysis

Named Entities

Topics Analysis

Language Detection

Bit.ly

Deduper

Msg splitter

+ Twitter Cards+ MetadataOgreOgreOgreOgreOgre

PickleNode

Node Shard

PickleNode

Node Shard

counter)

Stream Recorder

ControlChannels

100%Prism

PickleFilteringEngine push

PickleNode

Node Shard

PickleNode

Node Shard

counter)

WorkerSnapshotter

Worker

NodeMeteor

Real-timeStreams

HTTP Request

GET batch

PUSHScheduler

subscription X

subscription Y

SubscriptionsDB

PUSHDelivery

HTTP(S) POST(S)FTP

Amazon S3DynamoDB

WebSockets

HTTPStreaming

kafka-consumer

OracleCouchDB

IBM CognosGoogle BigQuery

Cloud Storage

BI tools

@Data Archiving Pipeline

UltrahoseArchiver

PickleNode

Node Shard

PickleNode

Node Shard

counter)

Stream Recorder

ControlChannels

UltrahoseArchiver

100%Prism

Twitter

FacebookWikipedia

DeletesProcessor

Input Streams

NewsCredBoardReader

MySpaceSuperFeeder

PickleNode

Node Shard

PickleNode

Node Shard

EventsStorage

counter)

tracker

LimitManager

NotificationService

Normaliser

HistoricsScheduler

RecordingScheduler

PushScheduler

InteractionTargets

Mapping

InteractionTargets

Mapping

Hadoop

Titan Historics

Map/Reduce

HBase Cluster

StopPUB

License Manager DB

Billing Pipeline DB

MaskManager DB

ConnectionManager

jobsDB

chunksDB

chunkselector

jobtracker

WorkerSnapshotter

Worker

NodeMeteor

Real-timeStreams

HTTP Request

GET batch

PUSHScheduler

subscription X

subscription Y

SubscriptionsDB

PUSHDelivery

HTTP(S) POST(S)FTP

Amazon S3DynamoDB

WebSockets

HTTPStreaming

ConnectionsStorage

kafka-consumer

Oracle

Stream results

CouchDB

PickleDB .DB

KafkaKafka

Historical Queries

@datasift

Goblin Head

Goblin Tail

3rd party APIs

Sentiment Analysis

Named Entities

Topics Analysis

Language Detection

IBM Cognos

HDFSArchiver

Bit.ly

Deduper

Msg splitter

Google BigQuery

Stream results

Cloud Storage

BI tools

UltrahoseArchiver

Twitter

FacebookWikipedia

DeletesProcessor

Input Streams

NewsCredBoardReader

MySpaceSuperFeeder

Goblin Head

Goblin Tail

3rd party APIs

Sentiment Analysis

Named Entities

Topics Analysis

Language Detection

Bit.ly

Deduper

Msg splitter

HBase Cluster

Region 1 Region 2 Region N... HDFSArchiver

+ Twitter Cards+ MetadataOgreOgreOgreOgreOgre

@Playback Pipeline

UltrahoseArchiver

PickleNode

Node Shard

PickleNode

Node Shard

counter)

Stream Recorder

ControlChannels

UltrahoseArchiver

100%Prism

Twitter

FacebookWikipedia

DeletesProcessor

Input Streams

NewsCredBoardReader

MySpaceSuperFeeder

PickleNode

Node Shard

PickleNode

Node Shard

EventsStorage

counter)

tracker

LimitManager

NotificationService

Normaliser

HistoricsScheduler

RecordingScheduler

PushScheduler

InteractionTargets

Mapping

InteractionTargets

Mapping

Hadoop

Titan Historics

Map/Reduce

HBase Cluster

StopPUB

License Manager DB

Billing Pipeline DB

MaskManager DB

ConnectionManager

jobsDB

chunksDB

chunkselector

jobtracker

WorkerSnapshotter

Worker

NodeMeteor

Real-timeStreams

HTTP Request

GET batch

PUSHScheduler

subscription X

subscription Y

SubscriptionsDB

PUSHDelivery

HTTP(S) POST(S)FTP

Amazon S3DynamoDB

WebSockets

HTTPStreaming

ConnectionsStorage

kafka-consumer

Oracle

Stream results

CouchDB

PickleDB .DB

KafkaKafka

Historical Queries

@datasift

Goblin Head

Goblin Tail

3rd party APIs

Sentiment Analysis

Named Entities

Topics Analysis

Language Detection

IBM Cognos

HDFSArchiver

Bit.ly

Deduper

Msg splitter

Google BigQuery

Stream results

Cloud Storage

BI tools

counter)

WorkerSnapshotter

Worker

NodeMeteor

Real-timeStreams

HTTP Request

GET batch

PUSHScheduler

subscription X

subscription Y

SubscriptionsDB

PUSHDelivery

HTTP(S) POST(S)FTP

Amazon S3DynamoDB

WebSockets

HTTPStreaming

kafka-consumer

OracleCouchDBIBM Cognos

Google BigQuery

Cloud Storage

BI tools

InteractionTargets

Mapping

InteractionTargets

Mapping

Hadoop

Titan Historics

Map/Reduce

HBase Cluster

Data Node Data Node Data Node Data Node Data Node

jobsDB

chunksDB

chunkselector

jobtracker

Stream results

Historical Queries

HDFSArchiver

Stream results

@Written In PHP

UltrahoseArchiver

PickleNode

Node Shard

PickleNode

Node Shard

counter)

Stream Recorder

ControlChannels

UltrahoseArchiver

100%Prism

Twitter

FacebookWikipedia

DeletesProcessor

Input Streams

NewsCredBoardReader

MySpaceSuperFeeder

PickleNode

Node Shard

PickleNode

Node Shard

EventsStorage

counter)

tracker

LimitManager

NotificationService

Normaliser

HistoricsScheduler

RecordingScheduler

PushScheduler

InteractionTargets

Mapping

InteractionTargets

Mapping

Hadoop

Titan Historics

Map/Reduce

HBase Cluster

StopPUB

License Manager DB

Billing Pipeline DB

MaskManager DB

ConnectionManager

jobsDB

chunksDB

chunkselector

jobtracker

WorkerSnapshotter

Worker

NodeMeteor

Real-timeStreams

HTTP Request

GET batch

PUSHScheduler

subscription X

subscription Y

SubscriptionsDB

PUSHDelivery

HTTP(S) POST(S)FTP

Amazon S3DynamoDB

WebSockets

HTTPStreaming

ConnectionsStorage

kafka-consumer

Oracle

Stream results

CouchDB

PickleDB .DB

KafkaKafka

Historical Queries

@datasift

Goblin Head

Goblin Tail

3rd party APIs

Sentiment Analysis

Named Entities

Topics Analysis

Language Detection

IBM Cognos

HDFSArchiver

Bit.ly

Deduper

Msg splitter

Google BigQuery

Stream results

Cloud Storage

BI tools

FacebookWikipedia

DeletesProcessor

Input Streams

3rd party APIs

Sentiment Analysis

Named Entities

Topics Analysis

Language Detection

OgreOgreOgreOgreOgreOgreOgreOgreOgreOgreOgre

Bit.ly

WorkerSnapshotter

Worker

HTTP Request

GET batch

PUSHScheduler

subscription X

subscription Y

job queue

SubscriptionsDB

PUSHDelivery

HTTP(S) POST(S)FTP

Amazon S3DynamoDB

kafka-consumer

OracleCouchDBIBM Cognos

Google BigQuery

Cloud Storage

BI toolsMonitoringAggregator

LimitManager

NotificationService

License Manager DB

Billing Pipeline DB

MaskManager DB

AuthenticationManager DB

Stream .Manager .DB

RecordingScheduler

100%Every piece of data

is handled by our PHP code

in realtime

What we do in

Marketing websiteRuns on Drupal

Our main webappCustomer signup,stream creation,

account management

Our external APIOur main interface with customers

Boring!That’s all very standard stuff,

well understood

The interesting uses are behind the scenes

Behind the scenes?Are you mad?!?

Everyone knows that PHPis only for building websites!

Internal servicesAPIs that support our data pipelines

User management, billing,data security

Data assemblyConvert incoming data

into common ‘interaction’structure

100%Every piece of data

is handled by our PHP code

in realtime

Push deliveryOutbound delivery of data

to customers’ serversand into their databases

1 MP3/secHow much data we can deliverto a single EC2 micro-instance

500Number of simultaneousdeliveries to customers

every second

HornetOur EvilTestTool(tm)

Designed to melt thedata centre

StorytellerOur functional test tool

Brings user stories to lifeFires up VMs, deploys code,

tests services

Reproducibly

Our HistoryDataSift grew out of TweetMeme

Our ProductPHP is superb

at handlingunstructured data

Our CustomersPHP can talk to

any server, database / datastorethat we want to deliver data to

Our PeopleSeveral ‘names’ from PHP community

PHP is a language most engineers know

Our TimePHP is a great language

to build high-quality codevery very quickly

Our PerformancePHP is fast enough

for data assembly work

and is getting faster with every major release

Our SanityOur PHP applications require

less Ops timethan any of the others

frameworks

Rolled our ownFrink & Stone

Right choice for usWe’re not part of the target demographic

for the major PHP frameworks

(nor the minor ones, tbh)

FrinkTweetmeme’s framework

built to handlemillions of tweeted links

Built for speedStripped down tothe bare essentials

a reaction to experiencewith early Zend Framework

JobqueuesLong-running daemon processes

Worker processes handle data queuesManager process monitors workers

StoneFoundation of ourin-house test tools

Hornet and Storyteller

Built for speedPowers our fake Twitter firehose

used for testing

Built for inspectionAllows us to measure

activity normally hiddenby libraries and PHP extensions

tools & utilities

PHP 5.3.latestCompiled in-house

Extensions statically-linkedfor performance

ZeroMQ extensionTransport layer for our pipelines

APC extensionShared memory for app metrics

PHP is too slow without an opcache

Lack of APC has prevented usmoving to PHP 5.4

XHProf extensionFor profiling code

Skews the resultsless than Xdebug

Redis extensionBuffering and queueing

(being phased out)

XdebugFor code coverage metrics(and readable vardump()s!)

PHPunitFor all our unit tests

phpdoc2For code documentation

(although nobody reads it -code is king)

MavenFor building all

release RPM packages

JenkinsContinuous integration

RPMPackages for deployment

into dev, test, staging, and production

Thank youPS: We’re hiring :-)

more than websites: php and the firehose @datasift (2013)

Technology

flex 4 components from the firehose

drupal theming from a firehose

nick halstead datasift cognitive match doing it all again...

resource: the torch or the firehose - mit opencourseware

drinking from the firehose: web weaving and acquisitions

marketing flow firehose - visual marketing

data-driven marketing roadshow datasift - march 27, 2014

amazon kinesis data firehose api reference - api...

sipping from the firehose

photography - a sip from a firehose!

datasift september '12 release overview

stewart townsend, datasift 'know your customer - stop...

staying dry while drinking from the technology firehose

drinking from the social media firehose

2016 11...

aws october webinar series - introducing amazon kinesis...

amazon kinesis firehose - firehose

introduction to amazon kinesis firehose - aws august webinar...

firehose instability in heliospheric and astrophysical...

rf firehose friction revised