acadgild webinar - the correct time to switch to hadoop

presents

Webinar on

The Correct Time to Switch to Hadoop

Presented by: Shajee

© copyright ACADGILD

Brief Intro About AcadGild: CEO – Vinod Dham, Father of Pentium

2Big Data and Hadoop Development

• ACADGILD is a technology education start-up which provides online courses in

latest technologies like FrontEnd, FullStack, Big-Data, Android etc.

• Started by IIT/IIM alumni

• Our aim is to provide job ready skills to millions of high school and college

graduates, and working professionals.

Course Title© copyright ACADGILD

Is it the correct time to switch yourCareer with Hadoop?

3Big Data and Hadoop Development

© copyright ACADGILD

Agenda Points

4Big Data and Hadoop Development

Sl No. Agenda Title

1 What is Big Data?

2 3 Vs of Big Data

3 From the Pen of Eric Schmidt- Ex-CEO, Google

4 Exploding Data Problem

5 Solution for Data Explosion – Hadoop

6 Core Components of Hadoop Cluster

7 Hadoop Ecosystem

Sl No. Agenda Title

8 Execution of First MapReduce Application

9 Job Prospects in Different Sectors

10 % Growth in Different Profiles

11 Companies Looking for Big Data Skills

12 Big Data-Related Job Titles

13 IDG Enterprise Big Data Research

14 Petrol Dataset Analysis using Pig

© copyright ACADGILD

What is Big Data?

5Big Data and Hadoop Development

© copyright ACADGILD

3 Vs of Big Data

6Big Data and Hadoop Development

Data Complexity

VolumeData Size

VelocitySpeed of Change

VarietyData Sources

• Terabytes• Records• Transactions• Table/Files

• Batch• Near-Time• Real-Time• Streams

• Structured• Unstructured• Semi-structured• All of the above

© copyright ACADGILD

From the Pen of Eric Schmidt- Ex-CEO, Google

7Big Data and Hadoop Development

Every two days now we create as much information as we did from the dawn of civilization up until  2003, according to Schmidt. That’s something like five Exabyte of data.

© copyright ACADGILD

Exploding Data Problem

8Big Data and Hadoop Development

• Big Data constitutes a large data set in PBs & ZBs which cannot be processed by a single machine within expected time frame.

© copyright ACADGILD

Solution for Data Explosion - Hadoop

9Big Data and Hadoop Development

• Need a new System:• With new database management other than Relational Database, capable of

handling unstructured as well as structured data.• To process huge datasets on large clusters of computers, than on a single system.• To manage clusters in which: • Nodes fail frequently• Number of nodes keep changing• Common infrastructure which is:• Efficient• Easy to use• Reliable

Hadoop is that new system !!

© copyright ACADGILD

Core Components of Hadoop Cluster

10Big Data and Hadoop Development

Hadoop 2.x Core Components

HDFS YARN

Storage Processing

NameNode

DataNode

Resource Manager

Node Manager

Master Layer

Slave Layer

© copyright ACADGILD

Hadoop Ecosystem

11Big Data and Hadoop Development

Data Life Cylce &

GovernanceFalcon,Atlas

Data WorkFlow

SqoopFlumeKafka

Nfs

Provisi-oning

ManagingAmbari

OutBreakZooKeeper

SchedulingOozie

AdministrationAuthenticationAuthorization

AuditingData

ProtectionRangerKnoxAtlas

Governance Integration DATA ACCESS SECURITY OPERAT-

IONS

HDFS – Hadoop Distributed File System

YARN: Data Operating System

Batch

Map-Reduce

Script

Pig

NoSQL

HBase

Stream

Storm

SQL

Hive

Search

Solr

In-Mem

Spark

© copyright ACADGILD 12Big Data and Hadoop Development

Let’s execute our First MapReduce application

© copyright ACADGILD

Job Prospects in Different Sectors

13Big Data and Hadoop Development

• According to Forbes, the top five industries who are hiring Big Data-related skills are Professional, Scientific and Technical Services, IT, Manufacturing, Finance, Insurance and Retail.

• The graph below shows the distribution of job openings in the above-mentioned sectors:

© copyright ACADGILD

% Growth in Different Profiles

14Big Data and Hadoop Development

• Forbes also reported that the demand for sales representatives skilled in selling Big Data solutions are going through the roof and will continue to do so into 2016 as well as the upcoming years.

• Big Data-related jobs like Information Security Analysts, Management Analysts, Management Analysts and Information Security Analyst continue to be in high demand.

© copyright ACADGILD

Companies Looking for Big Data Skills

15Big Data and Hadoop Development

Companies Looking for Big Data Skills:• EMC2, IBM, Cisco, Oracle are just a few of the top companies who are looking

for Big Data skills set. • Here’s a distribution of job requirements of the top ten Big Data employers

today, according to Wanted Analytics.

© copyright ACADGILD

Big Data-Related Job Titles

16Big Data and Hadoop Development

• Here are some job titles that would provide you with full range of opportunities when looking for Big Data-related jobs.

• Take a look at it and expand your search:

© copyright ACADGILD

IDG Enterprise Big Data Research

17Big Data and Hadoop Development

• According to IDG Enterprise Big Data Research, many organizations plan to invest in skill sets necessary for Big Data deployments, including Data Scientists, Data Architects, Data Analysts, Data Visualizers, Research Analysts, and Business Analysts in the next 12-18 months.

© copyright ACADGILD 18Big Data and Hadoop Development

Petrol Dataset Analysis using Pig

© copyright ACADGILD 19Big Data and Hadoop Development

© copyright ACADGILD

Contact Info:

o Website : http://www.acadgild.com

o LinkedIn : https://www.linkedin.com/company/acadgild

o Facebook : https://www.facebook.com/acadgild

o Support: support@acadgild.com

20Big Data and Hadoop Development

© copyright ACADGILD 21Big Data and Hadoop Development

Thank You