hadoop and its applications at the state and university library, scape information day, 25 june 2014

Per Møldrup-Dalum State and University Library

SCAPE Information Day State and University Library, Denmark, 2014-06-25

Hadoop and its applications at the State and University Library

• A bit on Hadoop in general • A bit on our experience in deploying Hadoop at the

library

2

Agenda

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

• MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Senjay Ghemawat, 2004

• In 2005 Cutting and Cafarella created Hadoop at Yahoo! • Now an Apache project • Commercial distributions, community editions, DIY

3

Origins

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

4

Map/Reduce

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

MAP

REDUCE

5

Lorem ipsum

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

• Count addresses that have fruits etc in their street name • Kirsebærhaven • Jordbærvej • Nødde allé

• Result • Kirsebær: 1203 • Nødder: 34 • Jordbær: 543

6

The Zoo

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

HDFS – data locality MapReduce

•••

7

Hadoop at the Library

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

• Blade servers with no local storage • Storage exclusively on NAS • We‘ve done several experiments

8

Can it be done?

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

Existing infrastructure

CPU Storage

4 CPU nodes • Two 6-core CPU • Intel® Xeon® Processor

X5670 with 12M Cache, 2.93 GHz, and 6.40 GT/s Intel® QPI

• 96GB RAM • 2Gbit Ethernet interface • CentOS • NFS mount point on NAS for

HDFS • Reachable NAS storage: ~4PB

9

Cluster topology

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

Science Museum/Science & Society Picture Library

10

Cloudera Hadoop Distribution

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

11

Interface

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

• http://hadoop.apache.org • http://www.cloudera.com • http://static.googleusercontent.com/media/research.g

oogle.com/en//archive/mapreduce-osdi04.pdf

12

References

This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).

13 This work was partially supported by the SCAPE Project. The SCAPE project is co‐funded by the European Union under FP7 ICT‐2009.4.1 (Grant Agreement number 270137).