the elephant in the cloud · operationalizing this new technology as use case move into production...

1

May 2014

Big Data Summit

The Elephant in the Cloud

Changing Shape of Data

Confidential and Proprietary, Qubole Inc. page 4

Emergence of Hadoop


Scalability on

Commodity Hardware

Democratization of

Data Processing

Impediments


Investment Risk as upfront investment needed to discover value of data

Execution Risk in order to come up to speed on the technology and then integrating it

Operationalizing this new technology as use case move into production

Enabling Accessibility to this technology in the enterprise

Benefits of the Cloud


On-demand and Turn Key without the hassles

Flexible in supporting different workloads and use cases and growing as

the enterprise moves from PoC to Production

Accessible in multiple regions and geographies

6

Use Hadoop on the Cloud to Discover and

Accelerate Big Data Use Cases

Cloud and Big Data


De-risk Hadoop & Big Data


80 node PoC Cluster to start with

that grew to 3000 node Cluster

Platform from 2007 to 2011

Took 3 months to get the cluster in

place and the base software

deployed on it

Took another 9 months to make it a

true strategic platform

Cloud based Hadoop as Service


Zero upfront investment in

infrastructure

Instantaneous scaling because of

the cloud

Hadoop becomes a strategic

platform very quickly

De-risk Hadoop & Big Data


System Mgmt

Hadoop

Scheduler

Hive/PIG

MonitoringGUI

Interfaces

(ODBC/JDBC)

Data Connectors

Lots of moving parts and open source technologies needed before getting to a data processing platform

Cloud based Hadoop as Service


System Mgmt Hadoop Scheduler(Oozie)Hive/PIG

Mahout/Weka

MonitoringGUI(Hue) Interfaces

(ODBC/JDBC)

Data Connectors

(MongoAdaptor..)

Fully Integrated Turn Key Platform that enables you to discover the ROI quickly with low execution

and investment risk

Challenges in Operationalizing Hadoop


Managing Growth placed on the infrastructure as usage grows

Managing Unpredictability in today’s agile development environment where use cases

change leading to different types of demands from the infrastructure

Managing Open Source as there is constant innovation and internal operations teams

have to continue to monitor open source contributions and distros

Cloud based Hadoop as a Service


Self Managed the growth of the infrastructure is matched with the growth of usage

and data dynamically and in an on demand manner

Flexibility and Elasticity of the cloud ensures instant availability of different machine

types for different use cases

Open Source Innovation is managed by the service provider while enterprises can focus

on deriving actionable insights from their data

Cloud Accessibility


Cloud based Hadoop as a Service


Sharing and Collaboration of data and analysis

is easily enabled across:

Geographies

Organizations

Supply Chains

Objections to a Cloud Service


Is TCO higher in the rent model vs pay upfront model?

Prices on the cloud keep dropping due to economies of scale and competition

Storage: 3 cents/GB/month (70% drop in the last 4 weeks)

Accenture study found broadly better performance in the same price on the cloud

http://www.accenture.com/SiteCollectionDocuments/PDF/Accenture-

Hadoop-Deployment-Comparison-Study.pdf

It is already more cost effective to run on the cloud and it is just going to get better

Objections to a Cloud Service


Is security and compliance there?

For security use encryption

For compliance AWS has many new products such as auditability,

Even the CIA runs workloads on the cloud

http://www.crn.com/news/cloud/240163382/amazon-wins-600-million-cia-

cloud-deal-as-ibm-withdraws-protest.htm

17

Cloud Based Hadoop Services provide a

Quick path to a Big Data Platform that

Adapts to the needs of an organization while

Reducing Costs and

Reducing Failure Risk

Cloud and Big Data


Thank You

Ashish Thusoo

[email protected]