druidmeetup@seoul 0906
TRANSCRIPT
![Page 2: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/2.jpg)
DRUID MEETUP@SEOUL
WELCOME TO DRUID WORLD
THE JOURNEY OF 9 MONTHS
![Page 3: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/3.jpg)
DRUID MEETUP@SEOUL
DRUID OVERVIEW
REALTIME
BROKER HISTORICAL
![Page 4: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/4.jpg)
DRUID MEETUP@SEOUL
ARCHITECTURE - BATCH INGESTION
HDFS
HISTORICAL NODE
HISTORICAL NODE
HISTORICAL NODE
BROKER NODE
Segments
Queries
![Page 5: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/5.jpg)
DRUID MEETUP@SEOUL
ARCHITECTURE - STREAMING INGESTION
REALTIME NODE
HISTORICAL NODE
HISTORICAL NODE
HISTORICAL NODE
BROKER NODE
Segments
QueriesStreaming
![Page 6: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/6.jpg)
DRUID MEETUP@SEOUL
ARCHITECTURE - LAMBDA
REALTIME NODE
HISTORICAL NODE
HISTORICAL NODE
HISTORICAL NODE
BROKER NODE
Segments
QueriesStreaming
HDFS
![Page 7: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/7.jpg)
DRUID MEETUP@SEOUL
REAL WORLD IS CRUEL
![Page 8: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/8.jpg)
DRUID MEETUP@SEOUL
PROBLEMS
▸ For Data Scientist
Arbitrary and Interactive exploration of time series data
▸ Scalability and Performance
Ad-hoc query on trillions of events
▸ Characteristics of the data
Dynamic Columns
Numeric data with Array Type
![Page 9: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/9.jpg)
DRUID MEETUP@SEOUL
DATA LAKE
http://www.kdnuggets.com/2015/09/data-lake-vs-data-warehouse-key-differences.html
![Page 10: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/10.jpg)
DRUID MEETUP@SEOUL
STAT FUNCTION FOR DATA SCIENTIST
▸ HISTOGRAM (MEDIAN)
▸ MEAN
▸ STDDEV
▸ RANGE
▸ AREA
▸ MIN/MAX/SUM
![Page 11: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/11.jpg)
DRUID MEETUP@SEOUL
DATA EXPLODING
// column size 100 ~ 1000 KEYS = [“a”,”b”,”c”……] VALUES = [3.25,45.443,103.2…..]
“a” = 3.25“b” = 45.443“c” = 103.2...
![Page 12: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/12.jpg)
DRUID MEETUP@SEOUL
REAL WORLD ARCHITECTURE
DATA NODE #1
DATA NODE #70
OVERLORDMIDDLE MANAGE
#1
COORDINATOR
MYSQL
HA PROXY
MEMCACHED#2
BROKER NODE
#1
BROKER NODE
#1
MEMCACHED#3
MEMCACHED#1
HISTORICAL NODE #1
HISTORICAL NODE #70
MIDDLE MANAGE
#50
ZK1
ZK2
ZK3
![Page 13: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/13.jpg)
DRUID MEETUP@SEOUL
BUT, WE DID IT
ORC INGESTION
VIRTUAL COLUMNS
JDBC FIREHOSE
STATS-EXTENTIONS
OPTIMIZATION
QUERY
![Page 14: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/14.jpg)
DRUID MEETUP@SEOUL
NO PAIN, NO GAIN
![Page 15: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/15.jpg)
DRUID MEETUP@SEOUL
MAY THE FORCE BE WITH YOU
![Page 16: Druidmeetup@seoul 0906](https://reader031.vdocument.in/reader031/viewer/2022030317/587069301a28ab48378b5ad5/html5/thumbnails/16.jpg)
Q&A
THANK YOU
DRUID MEETUP@SEOUL