Download - Hadoop on Cloud: Why and How?
![Page 1: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/1.jpg)
1© Cloudera, Inc. All rights reserved.
Hadoop on Cloud. Why and How?Andrei Savu | Tech Lead, Cloudera DirectorSilicon Valley Cloud Computing Group | Nov 18, 2015
![Page 2: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/2.jpg)
2© Cloudera, Inc. All rights reserved.
About me
Tech Lead on Cloudera Director
Previously founder of axemblr.com
Contributed to Apache Whirr (PMC) & jclouds.
Twitter: @andreisavu
![Page 3: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/3.jpg)
3© Cloudera, Inc. All rights reserved.
Cloudera Directorcloudera.com/director
Deploy and manage enterprise-grade Hadoop in the cloud
AWS & Google CloudExtensible via plugins
![Page 4: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/4.jpg)
Journey to the Cloud
![Page 5: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/5.jpg)
5© Cloudera, Inc. All rights reserved.
Do you use a public or private cloud?
How do you run and manage Hadoop?
![Page 6: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/6.jpg)
6© Cloudera, Inc. All rights reserved.
What is this talk about?
State of the WorldArchitectural PatternsImagine the Future
![Page 7: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/7.jpg)
7© Cloudera, Inc. All rights reserved.
Gartner's 2015 Hype Cycle for Emerging Technologies (source)
Advanced AnalyticsHybrid CloudInternet of Things
![Page 8: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/8.jpg)
8© Cloudera, Inc. All rights reserved.
Hybrid Clouds
Cloud ExchangeApplication PortabilityPrivate-PublicPublic-Public
![Page 9: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/9.jpg)
9© Cloudera, Inc. All rights reserved.
Cloud Wars
AWSMicrosoft AzureGoogle CloudVMWareOpenstacketc.
![Page 10: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/10.jpg)
10© Cloudera, Inc. All rights reserved.
Data has Mass and Gravity
![Page 11: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/11.jpg)
11© Cloudera, Inc. All rights reserved.
Hadoop EnvironmentsOn-Premise versus Cloud
On-Premise CloudStorage Direct Attached Direct Attached or Object Store
Data Not shared across clusters Shared across multiple clusters
Sizing Fixed-size Dynamic based on load
Usage Model All users share cluster Clusters created as needed for apps/users
Resource Management (YARN)
HDFS
Process Discover Model Serve
Industry Standard Servers (CPU, Memory, & Direct Attached Storage)
Resource Management (YARN)
HDFS
Process Discover Model Serve
Industry Standard Servers (CPU & Memory)
Object Storage
![Page 12: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/12.jpg)
12© Cloudera, Inc. All rights reserved.
Cloud providers shipping distributions of Hadoop
IntegrationUnlock Query EnginesMigration workloads
Is that a sustainable advantage? Or just a temporary stop gap?
![Page 13: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/13.jpg)
13© Cloudera, Inc. All rights reserved.
Maturity level
On-prem vs. CloudMonitoringDev / Test / ProdAvailabilityDurability
![Page 14: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/14.jpg)
14© Cloudera, Inc. All rights reserved.
Common Architectural Patterns in the Cloud
Object Storage
Source Data Seed Data Backup/DR
ETL/MODELING(Spark, MapReduce)
• Short-running clusters• Elastic workload• No local storage
necessary
|WASB |SWIFT |BLOB
• Long-running clusters• Sized to demand• Some local storage
BI/ANALYTICS(Impala, Solr)
• Fixed clusters • Periodic sync• Default to local
storage
APP DELIVERY(HBase, Kudu)
![Page 15: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/15.jpg)
15© Cloudera, Inc. All rights reserved.
Cluster lifecycle management
Create / TerminateDiscoveryMetadataMonitoring
![Page 16: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/16.jpg)
16© Cloudera, Inc. All rights reserved.
Work Queue
WorkflowsDispatchTrackingDecoupledFault Tolerant
![Page 17: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/17.jpg)
17© Cloudera, Inc. All rights reserved.
Common Architectural Patterns in the Cloud
Object Storage
Source Data Seed Data Backup/DR
ETL/MODELING(Spark, MapReduce)
• Short-running clusters• Elastic workload• No local storage
necessary
|WASB |SWIFT |BLOB
• Long-running clusters• Sized to demand• Some local storage
BI/ANALYTICS(Impala, Solr)
• Fixed clusters • Periodic sync• Default to local
storage
APP DELIVERY(HBase, Kudu)
![Page 18: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/18.jpg)
18© Cloudera, Inc. All rights reserved.
Multi-user
SecureIsolatedFriendly
![Page 19: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/19.jpg)
19© Cloudera, Inc. All rights reserved.
Elastic
Grow or shrinkBusiness hoursNumber of usersStorage vs. ComputeCost efficient
![Page 20: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/20.jpg)
20© Cloudera, Inc. All rights reserved.
Common Architectural Patterns in the Cloud
Object Storage
Source Data Seed Data Backup/DR
ETL/MODELING(Spark, MapReduce)
• Short-running clusters• Elastic workload• No local storage
necessary
|WASB |SWIFT |BLOB
• Long-running clusters• Sized to demand• Some local storage
BI/ANALYTICS(Impala, Solr)
• Fixed clusters • Periodic sync• Default to local
storage
APP DELIVERY(HBase, Kudu)
![Page 21: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/21.jpg)
21© Cloudera, Inc. All rights reserved.
Advanced Monitoring
LatencyResource utilizationConsistent performance
![Page 22: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/22.jpg)
22© Cloudera, Inc. All rights reserved.
High availability and failure domains
Data durabilityRepair within SLAHost-to-instance
![Page 23: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/23.jpg)
23© Cloudera, Inc. All rights reserved.
Backup and disaster recovery
Object store centricActive-Standby
![Page 24: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/24.jpg)
24© Cloudera, Inc. All rights reserved.
Imagine the Future
Portable ExperienceSelf-serviceSelf-healingGranular SecurityAdvanced GovernanceComplete Management
What’s your vision?
![Page 26: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/26.jpg)
26© Cloudera, Inc. All rights reserved.
Resources
Cloudera Director: http://www.cloudera.com/director
Interested in API level integration and scripting?
https://github.com/cloudera/director-sdk
https://github.com/cloudera/director-scripts
Interested in integration with another cloud platform?
https://github.com/cloudera/director-spi
https://github.com/cloudera/director-google-plugin
![Page 27: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/27.jpg)
27© Cloudera, Inc. All rights reserved.
What’s new in Cloudera Director 1.5?http://blog.cloudera.com/blog/2015/08/whats-new-in-cloudera-director-1-5/
Get Started
AWS Reference Guide
GCP Reference Guide
Try It Out
AWS Quickstart
Resources
![Page 28: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/28.jpg)
Cloudera Director Screenshots
![Page 29: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/29.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 30: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/30.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 31: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/31.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 32: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/32.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 33: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/33.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 34: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/34.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 35: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/35.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 36: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/36.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 37: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/37.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 38: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/38.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 39: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/39.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 40: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/40.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 41: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/41.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 42: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/42.jpg)
© 2014 Cloudera, Inc. All rights reserved.
![Page 43: Hadoop on Cloud: Why and How?](https://reader031.vdocument.in/reader031/viewer/2022030304/587850061a28ab68198b630b/html5/thumbnails/43.jpg)
© 2014 Cloudera, Inc. All rights reserved.