Download - Highway to the Information Zone (Andy Cross)
![Page 1: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/1.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Highway to the Information Zone
Solving 3 key challenges of building Big Data Solutions in the Cloud
@andybareweb
![Page 2: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/2.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Huge thanks to our sponsors & partners!
![Page 3: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/3.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Big Data core ethos: Distribute workload to achieve throughput on IO bound operations
Flat files + Compute = Azure
![Page 4: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/4.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
GA managed Hadoop 2 Hadoop on Microsoft AzureFamiliar tools such as Hive, Pig, OozieAdditional BoB Microsoft ecosystem tooling with .net SDK
Powershell and .net for provisionExecution with .net and powershell for Hive
Paired with Hortonworks HDP for on-premises Hadoop; compatible with all major Hadoop implementationsCombined with Excel and traditional Microsoft BI stack for compelling solutions
HDInsight – Hadoop as a Service
![Page 5: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/5.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Simple Programming style for efficient distribution
A cluster topology designed for resilience and efficiency
What is Hadoop?
MAP REDUCE
Name Node & Job Tracker
Data Node & Task Tracker
Data Node & Task Tracker
Data Node & Task Tracker……
![Page 6: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/6.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Apply innovative expressions of logic over stored mass of data
![Page 7: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/7.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Position in Cloud
![Page 8: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/8.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Blank Canvas• Windows Azure Subscription
– Capacity to provision HDInsight– Capacity to provision Storage Account
![Page 9: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/9.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Challenge 1: Cluster Provision
![Page 10: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/10.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
We need somewhere to Execute!
• Powershell / C# / xpat CLI
• All these give further configuration options including – Boost performance by increasing IOPs – stripe data across many Storage
Accounts– Manage cluster specific features; core-site, mapred-site and hdfs-site
![Page 11: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/11.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
DEMOProvision a customised HDInsight cluster via powershell
![Page 12: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/12.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Centralised Resources
![Page 13: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/13.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
HDFS
Mount Azure Blob Storage; consume from Hadoop
![Page 14: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/14.jpg)
Provision
Execute
De-provision
![Page 15: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/15.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Shard Data to boost performance
Shard source data across Azure storage accounts, giving over 5000 IOPS per HDInsight cluster
![Page 16: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/16.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Isolate logs best practice
Use a state storage account for logs, creating automatically at the same time as cluster creation
![Page 17: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/17.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Challenge 2: Data Ingress
![Page 18: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/18.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Windows Azure Storage Blobs– Equivalent to Azure Blob Storage
• Mounted as HDFS compatible file system– Hadoop can read/write directly with
– Azure Blobs
Explanation of WASB
ANDYC2014
![Page 19: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/19.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
DEMOFile upload to new WASB location; Hadoop fs –cat /path/to/file
![Page 20: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/20.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
In reality you will have a file pipeline; my solution is Cloud Data Sync Agent
![Page 21: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/21.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Challenge 3: Run a query!
![Page 22: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/22.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• .net Map Reduce SDK• Programmatically express logic• Implement three main classes• Job execution from a console
application
• Hive query language• Create Table myTable location ‘/path’• Select * from myTable• Powershell execution
![Page 23: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/23.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
DEMOHive and .net
![Page 24: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/24.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#
![Page 25: Highway to the Information Zone (Andy Cross)](https://reader034.vdocument.in/reader034/viewer/2022051512/540092fa8d7f724c088b4df4/html5/thumbnails/25.jpg)
Premium community conference on Microsoft technologies itcampro@ itcamp14#