making pig fly

8/10/2019 Making Pig Fly

1/36

Hortonworks Inc. 2011

Daniel Dai (@daijy)Thejas Nair (@thejasn)

Page 1

Making Pig FlyOptimizing Data Processing on Hadoop


2/36


What is Apache Pig?

Page 2Architecting the Future of Big Data

Pig Latin, a high leveldata processinglanguage.

An engine that

executes Pig Latinlocally or on aHadoop cluster.

Pig-latin-cup pic from http://www.flickr.com/photos/frippy/2507970530/


3/36


Pig-latin example


Query : Get the list of web pages visited by userswhoseage is between 20 and 29 years.

USERS = loadusersas (uid, age);

USERS_20s = filterUSERS by age >= 20 andage


4/36


Why pig ?


Faster developmentFewer lines of codeDont re-invent the wheel

Flexible

Metadata is optionalExtensibleProcedural programming

Pic courtesy http://www.flickr.com/photos/shutterbc/471935204/


5/36


Pig optimizations


Ideally user should not have to botherReality

Pig is still young and immaturePig does not have the whole picture

Cluster configurationData histogram

Pig philosophy: Pig is docile


6/36


Pig optimizations


What pig does for youDo safe transformations of query to optimizeOptimized operations (join, sort)

What you doOrganize input in optimal wayOptimize pig-latin queryTell pig what join/group algorithm to use


7/36 Hortonworks Inc. 2011

Rule based optimizer


Column prunerPush up filterPush down flatten

Push up limitPartition pruningGlobal optimizer



Column Pruner


Pig will do column pruning automatically

Cases Pig will not do column pruningautomatically

No schema specified in load statement

A = loadinput as(a0, a1, a2);B = foreachA generatea0+a1;C = orderB by$0;StoreC intooutput;

Pig will prunea2 automatically

A = loadinput;B = orderA by$0;C = foreachB generate$0+$1;StoreC intooutput;

A = loadinput;A1 = foreachA generate$0, $1;B = orderA1 by$0;C = foreachB generate$0+$1;StoreC intooutput;

DIY



Column Pruner


Another case Pig does not do columnpruningPig does not keep track of unused column aftergrouping

A = loadinput as(a0, a1, a2);B = groupAby a0;C = foreach B generate SUM(A.a1);Store C into output;

DIY

A = loadinput as(a0, a1, a2);A1 = foreachA generate$0, $1;B = groupA1by a0;C = foreach B generate SUM(A.a1);Store C into output;



Push up filter


Pig split the filter condition before push

A

Join

a0>0 && b0>10

B

Filter

A

Join

a0>0

B

Filter b0>10

Original query Split filter condition

A

Join

a0>0

B

Filter b0>10

Push up filter



Other push up/down


Push down flatten

Push up limit

Load

Flatten

Order

Load

Flatten

Order



Partition pruning


Prune unnecessary partitions entirelyHCatLoader

2010

2011

2012

HCatLoaderFilter

(year>=2011)

2010

2011

2012

HCatLoader(year>=2011)



Intermediate file compression


Pig Script

map 1

reduce 1

map 2

reduce 2

Pig temp file

map 3

reduce 3

Pig temp file



Enable temp file compression


Pig temp file are not compressed bydefaultIssues with snappy (HADOOP-7990)LZO: not Apache license

Enable LZO compressionInstall LZO for HadoopIn conf/pig.properties

With lzo, up to > 90% disk saving and 4x queryspeed up

pig.tmpfilecompression = truepig.tmpfilecompression.codec = lzo



Multiquery


Combine two or more map/reduce jobinto one

Happens automaticallyCases we want to control multiquery: combine toomany

Load

Group by $0 Group by $1

Foreach Foreach

Store Store

Group by $2

Foreach

Store


16/36



Implement the right UDF


Algebraic UDFInitialIntermediateFinal

A = loadinput;B0 = groupA by$0;C0 = foreachB0 generategroup, SUM(A);StoreC0 intooutput0;

MapInitial

Combiner

Intermediate

ReduceFinal



Implement the right UDF


Accumulator UDFReduce side UDFNormally takes a bag

BenefitBig bag are passed inbatchesAvoid using too muchmemory

Batch size

A = loadinput;B0 = groupA by$0;C0 = foreachB0 generategroup,my_accum(A);StoreC0 intooutput0;

my_accum extendsAccumulator {publicvoidaccumulate() {// take a bag trunk

}publicvoidgetValue() {

// called after all bag trunks areprocessed

}}

pig.accumulative.batchsize=20000


19/36


Memory optimization


Control bag size on reduce side

If bag size exceed threshold, spill to disk

Control the bag size to fit the bag in memory ifpossible

reduce(Text key, Iteratorvalues, )

Mapreduce:

Iterator

Bag of Input 1 Bag of Input 2 Bag of Input 3

pig.cachedbag.memusage=0.2


20/36


Optimization starts before pig


Input formatSerialization formatCompression


21/36


Input format -Test Query


> searches = load aol_search_logs.txt'using PigStorage() as(ID, Query, );

> search_thejas = filtersearches byQuery

matches'.*thejas.*';

> dump search_thejas;

(1568578 , thejasminesupperclub, .)


22/36


Input formats


RunTime (sec)

PigStorage

LzoPigStorage

PigStorage W Type

AvroStorage (has types)


23/36


Columnar format


RCFileColumnar format for a group of rowsMore efficient if you query subset ofcolumns


24/36


Tests with RCFile


Tests with load + project + filter out allrecords.Using hcatalog, w compression,types

Test 1Project 1 out of 5 columnsTest 2

Project all 5 columns


25/36


RCFile test results


Plain Text

RCFile


26/36


Cost based optimizations


Optimizations decisions based onyour query/dataOften iterative process

Runquery

Measure

Tune


27/36


Hash Based Agg

Use pig.exec.mapPartAgg=true to enable

Map task

Cost based optimization - Aggregation


Map

(logic)


28/36


Cost based optimizationHash Agg.


Auto off featureswitches off HBA if output reduction isnot good enough

Configuring Hash AggConfigure auto off feature -pig.exec.mapPartAgg.minReduction

Configure memory used -

pig.cachedbag.memusage


29/36


Cost based optimization - Join


Use appropriate join algorithmSkew on join key - Skew joinFits in memoryFR join


30/36


31/36


Parallelism of reduce tasks


Runtime

4.0

6.0

8.0

24.0

48.0

256.0

Number of reduce slots = 6Factors affecting runtimeCores simultaneously used/skewCost of having additional reduce tasks

Cost based optimization keep data


32/36


Cost based optimizationkeep datasorted


Frequent joins operations on samekeysKeep data sorted on keysUse merge join

Optimized group on sorted keysWorks with few load functionsneedsadditional i/f implementation


33/36


Optimizations for sorted data


Join 2

Join 1

Sort2

Sort1


34/36


Future Directions


Optimize using statsUsing historical stats w hcatalogSampling


35/36


Questions


?


36/36

making pig fly

Documents