the state of the art for openstack data processing (hadoop on openstack) - atlanta
DESCRIPTION
Update on Sahara as of the OpenStack Icehouse releaseTRANSCRIPT
The State of OpenStackData Processing: Sahara,Now and in Juno
Sergey Lukjanov (Mirantis)Matthew Farrellee (Red Hat)John Speidel (Hortonworks)
Agenda
• Sahara overview• Icehouse release• HDP plugin updates• Juno plans
Agenda
• Sahara overview• Icehouse release• HDP plugin updates• Juno plans
OpenStack Data Processing: Sahara
Mission: To provide a scalable data processing stack and associated management interfaces.
• provision and operate Hadoop clusters • schedule and operate Hadoop jobs
Hadoop - Big Data Platform
© http://hortonworks.com/hadoop/yarn/
Trends
http://www.google.com/trends/
Use cases
• Self-service provisioning of Hadoop clusters• Utilization of unused compute capacity for
bursty workloads• Dev -> Stage -> Prod lifecycle• Run Hadoop workloads in few clicks without
expertise in Hadoop ops
Architecture overview
Data Sources
Savanna Python Client RE
ST A
PI
Cluster Configuration
Manager
Horizon
Keystone
Auth
Data Access Layer
Swift
Savanna Pages
HadoopVM
Vendors Plugins
HadoopVM
HadoopVM
HadoopVM
Resources Orchestration
Manager
Job Sources Job
Manager
Heat
Nova
Glance
Cinder
Neutron
Trove DB
Sahara status
• Official integrated OpenStack project• Supported Hadoop distros:
• Vanilla Apache Hadoop• Hortonworks Data Platform• Intel Distribution• Cloudera Distribution in blueprint
• Included into OpenStack distros:• RDO - openstack.redhat.com• Mirantis OpenStack - software.mirantis.com
Contributors
Agenda
• Sahara overview• Icehouse release• HDP plugin updates• Juno plans
Icehouse release
142 bugs fixed
Icehouse release
57 blueprints
Icehouse release
32 people
Icehouse release
Standard process
Icehouse release
Dozens more in the client!
Icehouse release
Tempest helps us manage our API
Icehouse release
Sahara easily deployed with DevStack
Icehouse release
Hadoop 2 available via all plugins
© http://hortonworks.com/hadoop/yarn/
Icehouse release
• HBase (and Sqoop) available via HDP plugin• Spark images w/ diskimage-builder (full plugin in review)
• Heat for provisioning• i18n translation started• Neutron namespaces w/ rootwrap• Guest agent implementation started
Elastic Data Processing (EDP) is Sahara’s take on data processing workflow management.
Goal - let end users (those w/ high value questions to answer) get answers about data without having to know a single thing about cluster management.
“Customers launch millions of Amazon EMR clusters every year.”http://aws.amazon.com/elasticmapreduce/
Elastic Data Processing update
Elastic Data Processing update
Available with the Hortonworks Data Platform plugin
Elastic Data Processing update
Support for external HDFS
Elastic Data Processing update
MapReduce.Streaming
and Java actions
Elastic Data Processing update
Job relaunch, with new data and parameters
Command line interface overview
If you can do it with the Dashboard, you can do it from the command-line
Blueprint: python-savannaclient-cli
Command line interface overview
Image management
$ sahara...Positional arguments: <subcommand>
image-add-tag Add a tag to an image.image-list Print a list of available images.image-register Register an image from the Image index.image-remove-tag Remove a tag from an image.image-show Show details of an image.image-unregister Unregister an image.
Command line interface overview
Node group, cluster and job templates$ sahara
node-group-template-create Create a node group...node-group-template-delete Delete a node group...node-group-template-list Print a list of available...node-group-template-show Show details of a node...cluster-template-create Create a cluster template.cluster-template-delete Delete a cluster template.cluster-template-list Print a list of available...cluster-template-show Show details of a cluster...job-template-create Create a job template.job-template-delete Delete a job template.job-template-list Print a list of job...job-template-show Show details of a job...
Command line interface overview
Data sources and job binaries$ sahara... <subcommand> data-source-create Create a data source that provides job input receives job output. data-source-delete Delete a data source. data-source-list Print a list of available data... data-source-show Show details of a data source. job-binary-create Record a job binary. job-binary-delete Delete a job binary. job-binary-list Print a list of job binaries. job-binary-show Show details of a job binary.
Command line interface overview
Clusters and jobs$ sahara... <subcommand>
cluster-create Create a cluster.cluster-delete Delete a cluster.cluster-list Print a list of available clusters.cluster-show Show details of a cluster.job-createjob-delete Delete a job.job-list Print a list of jobs.job-show Show details of a job.
Agenda
• Sahara overview• Icehouse release• HDP plugin updates• Juno plans
HDP Plugin Overview
• Full support for all Sahara Functionality• Nova and Neutron network• Cluster Scaling• Scale Up• Swift Integration• Cinder Support• Data Locality• EDP
• Apache Ambari REST API’s used for clusterprovisioning
• Monitoring/Management of clusters via Ambari• Full support for multiple HDP stacks• HDP pre-installed or generic VM images
HDP 1.3.2● NameNode● Secondary NameNode● DataNode● HDFS● ZooKeeper ● Ambari Server/Agent● HCatalog● Sqoop● Job Tracker● Task Tracker● MapReduce● Hive● MySQL● Pig● WebHCat Server● Oozie● Ganglia● Nagios● HBase
HDP Plugin Stack Support
HDP 2.0.6● History Server● MapReduce 2 / YARN● Resource Manager● YARN Client
HDP 2.1● Storm● Falcon
Coming Soon!
Availa
ble
Availa
ble
HDP 2.1 +● SOLR● Cascading
Roadmap
HDP Disk Images
• Disk Image Builder offers consistent approach for image creation• HDP Plugin provides images and scripts for (CentOS, RHEL):
• Plain • 1.3.2• 2.0.6• 2.1 (coming soon)
• Pre-Packaged images (1.3.2, 2.0.6) provide images with HDP packages pre-installed for accelerated provisioning, reduced network traffic
• Image Build Scripts allow images to be customized• Security• Custom Packages• O/S Settings
Ambari Blueprints• Two primary goals of Ambari Blueprints
• Ability to export a complete description of a running cluster
• Provide API based cluster installations based on a self- contained cluster description
• Blueprints contain cluster topology and configuration information
• Enables Interesting use cases between physical and virtual, including OpenStack/Sahara
Agenda
• Sahara overview• Icehouse release• HDP plugin updates• Juno plans
Juno roadmap
• Further integration with OpenStack ecosystem:• Distributed architecture• Guest agents• EDP enhancements• Merge dashboard to Horizon
To be discussed and confirmed at Design Summit
Design Summit Sessions
7 Sessions: Thursday 1:30 - Friday 10:30
http://goo.gl/lQXtUS
Agenda
Q&A
Cluster and EDP workflowsRarely
Infrequently
Occasionally
Commonly
Occasionally
Frequently