streaming architecture with hdp & elk
TRANSCRIPT
ìReal-timestreamprocessing
15/11/15AlainDouangpraseuth
$(whoami)
ì AlainDouangpraseuth
ì So$wareConsultant–France
fr.linkedin.com/in/alain-douangpraseuth-b5577972
@douangpraseuth
15/11/15AlainDouangpraseuth
Agenda
ì Real-Dmestreamprocessing
ì Storm,NoSQLandHadoop
ì HortonworksDataPlaLorm
ì Usecase«Real-DmeanalysisofTweets»
ì AmbariDemo
ì Real-DmestreamprocessingDemo
ì Q&A
15/11/15AlainDouangpraseuth
Real-timestreamprocessing
ì BigDataì TonsofinformaDonarrivingeverysecondì Processingbigvolumesofdataisnotenough
ì FastDataì Datahastobeprocessedfastì ReacttochangingbusinesscondiDons
15/11/15AlainDouangpraseuth
Real-timestreamprocessing
ì Basicsconceptsì StreamProcessingì StreamingAnalyDcs
ì StreamprocessingsoluDonsì DesignedtohandlehighvolumeinrealDmeì Scalable,highlyavailableandfaulttolerant
architecture
ì AnalysisofdatainmoDon
15/11/15AlainDouangpraseuth
Real-timestreamprocessing
ì Domainsì Tradingì FrauddetecDonì Systemmonitoringì Manyotherexamples
15/11/15AlainDouangpraseuth
Storm,NoSQLandHadoop
ì Stormì IsahighlydistributedrealDmecomputaDonsystem
ì NoSQLì ElasDcsearchì VisualizaDontool,Kibana
ì Hadoopì HDFSì YARN+MapReduce2
15/11/15AlainDouangpraseuth
HortonworksDataPlatform
15/11/15AlainDouangpraseuth
Usecase«Real-timeanalysisofTweets»
5-nodeHadoopCluster
Twi_erData
YARN+MapReduce2
Storm
Zookeeper
HDFS
LogData …Ambari
SOURCES REAL-TIMEVISUALIZATION
NoSQL
REAL-TIMEINDEX
REAL-TIMEPROCESS
Hive
SQLVISUALIZATION
15/11/15AlainDouangpraseuth
AmbariDemo
ì 5-nodeHadoopcluster
ì HDP2.3withAmbari2.1
ì Cloud-EC2withRHEL7.1
15/11/15AlainDouangpraseuth
Real-timestreamprocessingDemo
15/11/15AlainDouangpraseuth
Q&A
15/11/15AlainDouangpraseuth