genn.ai (源内)Realtime processing for everyone

1

Takeshi Nakano@tf0054

http://genn.ai/

Who am I?Takeshi NAKANO

Senior Researcher / Architect.Co-authered Getting started with Solr in Japanese.Co-authered Hadoop Hacks in Japanese.

2

Writing “Getting started with Kafka” in Japanese.

http://genn.ai/

Agenda

What is genn.ai? Realtime processingUsecases

3

Let’s use genn.aiDownload & get your keyDefine a schema and a filterMake it real!

A usecase of genn.ai

4

CustomerA

CustomerB

CustomerC

Realtime Analysis Platform

じゃらんWeb Sites

Client(Hotels)

MarketerAnalyst

Visual editor for genn.ai

http://genn.ai/

GungnirThrift Server

GungnirRest Server

GungnirCosole

Stormnimbus

Stormsupervisor

GungnirClient

on finagle

Kafkabroker

Stormsupervisor

StormsupervisorKafka

broker

Kafkabroker

GungnirThrift Server

on finagle

GungnirRest Server

GungnirRest Serveron finagle

MongoDBMetaStore

TupleTopic

Query

genn.ai UIQuery design

view

Query

User accountTuple schema

Topology

JSON Tuple Tuple Tuple

Kafka

MongoDB

HDFS

User accountTuple schema

6

Structure of genn.ai

http://dev.genn.ai/

Preparing for starting.

http://dev.genn.ai/

Preparing for starting.

> cd gungnir-client> bin/gungnir -u tf0054 -p 2e2232073109

Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-82013/10/14 22:34:58 com.twitter.finagle.Init$ apply情報: Finagle version 6.5.1 (rev=59..9cde41) built at 20130626-111057

Gungnir server connected... ec2-xxx.amazonaws.com/54.238.99.212:9290Welcome tf0054gungnir>

Define your schema!CREATE TUPLE userAction ( userId STRING, hotelId STRING);

CREATE TUPLE commitAction ( userId STRING, hotelId STRING, checkin TIMESTAMP('yyyy-MM-dd'), nights INT);

STRING             Java type string

TINYINT            Java type byteSMALLINT          Java type shortINT                 Java type intBIGINT              Java type longFLOAT              Java type floatDOUBLE            Java type double

TIMESTAMP        long value

LISTMAPSTRUCT

FROM userAction AS ua, commitAction AS ca USING kafka_spout()EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user');

From EMIT

Define your filter! (1/4)

FROM userAction AS ua, commitAction AS ca USING kafka_spout()JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo')JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel')EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user');

From JOIN(user)

JOIN(hotel)

EMIT

Define your filter! (2/4)

FROM userAction AS ua, commitAction AS ca USING kafka_spout()JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo')JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel')INTO stream_0;FROM stream_0EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user');

From JOIN(user)

JOIN(hotel) stream_0

EMIT(user)

Define your filter! (3/4)

FROM userAction AS ua, commitAction AS ca USING kafka_spout()JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo')JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel')INTO stream_0;FROM stream_0EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user');FROM stream_0BEGIN GROUP BY hotelId EACH count() AS count, * EMIT hotelId, hotelname, photo, count USING kafka_emit('${TOPOLOGY_ID}_hotel')END GROUP;

From JOIN JOIN

EMIT(user)

stream_0

GROUPBY

EMIT(hotel)

Define your filter! (4/4)

EACH sum(nights)EACH avg(nights)

SUBMIT TOPOLOGY;

DESC TOPOLOGY;

SHOW TOPOLOGIES;

STOP TOPOLOGY xxx;

DROP TOPOLOGY xxx;

Register your filter! and ...gungnir> SUBMIT TOPOLOGY;OKgungnir> DESC TOPOLOGY;{

"id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", "createTime":"2013-09-16T16:35:47.432Z", "summary":{

...

}

}

gungnir> STOP TOPOLOGY 52373363e4b0a525adadb4f2 ;

gungnir> DELETE TOPOLOGY 52373363e4b0a525adadb4f2 ;

TRACK userAction {"userId":"siniida","hotelId":"226979"};TRACK commitAction {"userId":"siniida","hotelId":"226979","checkin":"2013-10-18","nights":1};

Testing it!

> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_user ; or> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_hotel;

gungnir> DESC TOPOLOGY;{

"id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", ...

... USING kafka_emit('${TOPOLOGY_ID}_user')

"hotelId" : "128767"

"hotelId" : "356564"

"hotelId" : "226979"

"hotelId" : "325073"

"hotelId" : "357090"

"hotelId" : "126942"

"hotelId" : "310294"

"hotelId" : "367746"

"hotelId" : "232896"

"hotelId" : "220231"

"hotelId" : "163175"

"hotelId" : "367856"

"hotelId" : "360863"

"hotelId" : "119747"

*** We are changing this name because “TRACK” isn’t fit to its function! (advised in GOTO berlin 2013. Thx.)*** The function of this command is to make and post a event to gennai.

Knowing http tuple receiver

> curl -v -H "Content-type: application/json" -X POST \

-d '{"userId":"tf0054","hotelId":"226979"}' \

http://dev.genn.ai:9191/gungnir/v1.0/track/525e67f8e4b0e7cfca98d040/userAction

gungnir> DESC USER;{

"id":"525e67f8e4b0e7cfca98d040", "name":"siniida", "password":"CmGOdN2UU/Xtmk0wRCZKzDy/BN65LAHPCp2+1KMDOOM=", "createTime":"2013-10-16T10:18:32.849Z"

}

gungnir>

Design of genn.aiBuilt on Storm

Converts properly (e.g. building AST)The Good Parts (e.g. Supporting parallelism setting) ..

genn.ai wants friends!genn.ai codes will be open-sourced in the future.We are looking for co-challenging partners for improving genn.ai.

You can use genn.ai on your web sites completely for free (infrastructure etc..) if we can share the dream together.

Please feel free to contact us!http://genn.ai/ tf0054@gmail.com

The future of genn.ai

http://genn.ai/

Thanks!

19

20