tie big data panel
DESCRIPTION
TRANSCRIPT
![Page 2: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/2.jpg)
![Page 3: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/3.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
![Page 4: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/4.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Store files
![Page 5: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/5.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Process
data
![Page 6: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/6.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Ingest data
![Page 7: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/7.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Store records & tables
![Page 8: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/8.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
High level domain specific
language
![Page 9: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/9.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Chain together complex workloads
![Page 10: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/10.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Schedule them
![Page 11: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/11.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Columnar format + metadata
![Page 12: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/12.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
End user queries
![Page 13: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/13.jpg)
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Coordinate within
system
![Page 14: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/14.jpg)
The pattern repeated
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
HiPal
Hive Databee Databee
Scribe
Hive HBase
Zookeeper
![Page 15: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/15.jpg)
The pattern repeated
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Hive Oozie Oozie
Data
Highway
Pig & Hive HBase
Zookeeper
![Page 16: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/16.jpg)
The pattern repeated
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Azkaban Azkaban
Sqoop
Kafka
Pig Voldemort
Zookeeper
![Page 17: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/17.jpg)
The pattern repeated
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Hue Hue
Hive Oozie Oozie
Sqoop
Flume
Hive / Pig HBase
Zookeeper
Cloudera’s Distribution Including Apache Hadoop
![Page 18: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/18.jpg)
Project summary
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Topic Project(s)
File storage HDFS
Record storage Hbase, Hypertabe, Accumulo
Metadata storage Hive, Hcatalog
Batch data processing MapReduce
Streaming data processing S4, Storm
Graph processing Giraph, X-Rime
Query language Hive
Dataflow language Pig
Database integration Sqoop
Event data collection Flume, Scribe
Test & assembly Bigtop
Distributed lock Zookeeper
Web access Hue
Workflow Oozie, Azkaban
File format Avro, RCFile, Protocol Buffers, Sequence File
![Page 19: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/19.jpg)
![Page 20: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/20.jpg)
![Page 21: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/21.jpg)
BIG DATA
PO
SS
IBL
E
anything
with
is
![Page 22: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/22.jpg)
![Page 23: TiE Big Data panel](https://reader034.vdocument.in/reader034/viewer/2022052504/54c67efc4a7959a4368b46c9/html5/thumbnails/23.jpg)
Ce
leb
rate
Ne
xt
Satu
rday