Download - Hadoop-2.6.0 Slides
Taming Big Data with HadoopKul Subedi
Introduction
● What is Big Data?
Properties of Big Data
Cont...
● Large and growing data files● Commonly measured in terabytes or
petabytes● Unstructured data● May not fit in a “relational database model”● Derived from Users, Applications, Systems,
and Sensors
Problem: Data Throughput Mis-match
● Standard spinning hard drive 60-120 MB/sec● Standard solid state hard drive 250-500
MB/sec● Hard drive capacity growing● Online data growth
Moving data on and off of disk is the bottleneck.
Cont...● One Terabyte (TB) of data will take 10,000
seconds (approximately 167 minutes)● One TB of data will take 2000 seconds
(approximately 33 minutes) to read data at 500 MB/sec (solid state)
One TB is a “small” file sizeThe need for parallel data access is essential for Big data.
Problem(1): Scaling
Agenda● Hadoop Definition● Hadoop Ecosystem● History● Hadoop Design Principle● HDFS and MapReduce (Demo)● Conclusion
Definition
● A framework of open-source tools, libraries, and methodologies for the distributed processing of large data sets
● Scale up from single servers to thousands of machines, each offering local computation and storage
Cont...
● The project includes ❏ Hadoop Common❏ HDFS❏ YARN❏ MapReduce
Cont...
● Other Hadoop-related projects❏ Pig❏ Hive❏ Tez❏ Spark❏ HBase❏ Ambari etc
Hadoop Usage Modes● Administrators❏ Installation❏ Monitor/Manage System❏ Tune System
● End Users❏ Design MapReduce Applications❏ Import/Export Data❏ Work with various Hadoop Tools
Hadoop History
● Developed by Doug Cutting and Michael J. Cafarella
● Based On Google MapReduce technology● Designed to handle large amounts of data
and be robust● Donated to Apache Software Foundation in
2006 by Yahoo
Cont...
Application areas:❏ Social media❏ Retail❏ Financial services❏ Web Search❏ Everywhere there is large amounts of
unstructured data
Cont...
Prominent Users:❏ Yahoo!❏ Facebook❏ Amazon❏ Ebay❏ American Airlines ❏ The New York Time, and many others
Design Principles
● Moving computation is cheaper than moving data
● Hardware will fail, manage it● Hide execution details from the user● Use streaming data access● Use a simple file system coherency model
Cont...
What Hadoop is not: A replacement for SQL, always fast and efficient, good for quick ad-hoc querying
HDFS
HDFS-Architecture1.Where do I read or write data?2.Use these data nodes.
NameNode
● Only one per cluster “master node”● Stores meta information of “filesystem” such
as Filename, permission, directories, blocks● Keep in RAM for fast access● Persisted to disk● The namenode is the brain of the outfit
DataNode
● Many per cluster, “slave node”● Stores individual file “blocks” but knows
nothing about them, accept the block name● Reports regularly to NameNode “Hey I am alive, and I have these blocks”
HDFS Presents
● Transparency● Replication
HDFS Properties● Files are immutable
No updates, no appends● Disk access is optimized for sequential
readsStore data in large “blocks” 128 MB default
● Avoid Corruption“blocks” are verified with checksum when
stored and read
Cont...● High throughput
Avoid contention, have system share as little information and resources as possible
● Fault TolerantLoss of a disk, or machine, or rack of
machines should not lead to data loss
Client Reading From HDFS
Client Writing To HDFS
Demo● File System Health check❏ hdfs fsck /
● List the file system content❏ $hdfs dfs -ls /
● Create a directory❏ $hdfs dfs -mkdir /data1
● Upload file to HDFS❏ $hdfs dfs -put input.txt /in
Cont...
● Input directory in HDFS: /in● Output directory in HDFS: /output
NameNode High Availability
● NFS Filer● Quorum Journal Manager (QJM)
MapReduce● A programming model for processing large
Data in Distributed fashion over cluster of commodity machines
● Introduced by Google● Uses two key steps: mapping and reducing
Cont...● Almost all data can be mapped into <key,
value> pairs somehow● Your keys and values may be of any type:
string, integers, dummy types, and <K,V> pairs themselves and so on
● Scale-free programming❏ If a program works for a 1KB file, it can
work for any file size
see spot run run spot runsee the cat
run spot run
see spot run
see the cat
see,1 spot,1 run,1
see,1 see,1
spot,1 spot,1
run,1 run,1run,1
the,1
cat,1
InputSplit
Map
Shuffle
see,1 spot,1 run,1
see,1 the,1 cat,1
see,2
spot,2
run,3
the,1
cat,1
Reduce
see,2 spot,2 run,3the,1 cat,1
Output
Data Flow
MapReduce Word Count Data Flow
Example: Hello World Program
● wget www.gutenberg.org/files/2600/2600.txt● python mapper.py < input.txt | sort | python
reducer.py● cat *.txt | python mapper.py | sort | python
reducer.py
Demo
● cat input1.txt | python mapper.py● cat input1.txt | python mapper.py | sort● cat input1.txt | python mapper.py | sort |
python reducer.py● cat input1.txt | python mapper.py | sort |
python reducer.py > output.txt
How to run job in Cluster?
● Using Streaming Interface❏ hadoop jar /opt/hadoop/hadoop-2.6.0
/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar -file ./mapper.py -mapper ./mapper.py -file ./reducer.py -input /in/input.txt -output /output/run1
Web Interfaces
● NameNode: http://10.0.0.160:50070● ResourceManager: http://10.0.0.160:8088● /opt/hadoop/hadoop-2.6.0
/share/hadoop/hdfs/webapps/hdfs contains view
Prerequisites
● Java● ssh● rsync
Installation
● wget apache.osuosl.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0-src.tar.gz
● wget apache.osuosl.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0-src.tar.gz.mds
Cont...
● wget apache.osuosl.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
● wget apache.osuosl.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz.mds
Integrity Check
● md5sum hadoop-2.6.0.tar.gz● cat hadoop-2.6.0.tar.gz.mds | grep -i md5
Startup Code
● https://github.com/kpsubedi/BigData
OpenSource
● Apache Hadoop http://hadoop.apache.org/
Commercial Big Data Players
● Hortonworks http://hortonworks.com/● Cloudera http://www.cloudera.
com/content/cloudera/en/home.html● MAPR https://www.mapr.com/● Others
Conclusion
● Thank you
References
● Apache Hadoop http://hadoop.apache.org/
● Hortonworks http://hortonworks.com/
● Cloudera http://www.cloudera.com/content/cloudera/en/downloads.html
● MAPR https://www.mapr.com/
● The Google File System http://static.googleusercontent.com/media/research.google.com/en/us/archive/gfs-sosp2003.pdf
References (1)
● MapReduce: Simplified Data Processing on Large Clusters http://static.googleusercontent.com/media/research.google.com/en/us/archive/mapreduce-osdi04.pdf