HPAT: High Performance Analytics with Scripting Ease-of-Use

Ehsan TotoniIntel Labs, USA

ehsan.totoni@intel.com

Todd A. AndersonIntel Labs, USA

todd.a.anderson@intel.com

Tatiana ShpeismanIntel Labs, USA

tatiana.shpeisman@intel.com

AbstractBig data analytics requires high programmer productiv-ity and high performance simultaneously on large-scaleclusters. However, current big data analytics frameworks(e.g. Apache Spark) have prohibitive runtime overheadssince they are library-based. We introduce a novel auto-parallelizing compiler approach that exploits the charac-teristics of the data analytics domain such as the map/re-duce parallel pattern and is robust, unlike previous auto-parallelization methods. Using this approach, we build HighPerformance Analytics Toolkit (HPAT), which parallelizeshigh-level scripting (Julia) programs automatically, gener-ates efficient MPI/C++ code, and provides resiliency. Fur-thermore, it provides automatic optimizations for scriptingprograms, such as fusion of array operations. Thus, HPAT is369× to 2033× faster than Spark on the Cori supercomputerand 20× to 256× times on Amazon AWS.

Keywords Big Data Analytics, High Performance Comput-ing, Automatic Parallelization

1. IntroductionBig data analytics applies advanced analytics and machinelearning techniques to gain new insights from large data sets,which are gathered from sources such as sensors, web, logfiles, and social media. Big data analytics allows users to ex-tract knowledge from this data and make better and fasterdecisions. However, supporting fast decision making neces-sitates rapid development of the application by domain ex-pert programmers (i.e., high productivity) and low executiontime (i.e., high performance). High productivity in the dataanalytics domain requires scripting languages such as MAT-LAB, R, Python, and Julia since they express mathematicaloperations naturally and are the most productive languagesin practice [14, 30]. High performance requires efficient exe-

[Copyright notice will appear here once ’preprint’ option is removed.]

cution on large-scale distributed-memory clusters due to ex-treme dataset sizes.

Currently, there is a significant productivity and per-formance gap in the big data analytics domain. A typicalHigh Performance Computing (HPC) approach of writinglow-level codes (e.g. MPI/C++) is beyond the expertiseof most data scientists and is not practical in their inter-active workflows. Existing big data analytics frameworkssuch as Apache Hadoop [37] and Apache Spark [40] pro-vide better productivity for big data analytics on clustersusing the MapReduce programming paradigm [15]. How-ever, this productivity comes at the cost of performance asthese frameworks are orders of magnitude slower than hand-written MPI/C++ programs [11, 25, 31]. A fundamental is-sue is that these frameworks are library-based, requiring aruntime system to coordinate parallel execution across allthe nodes. This leads to high runtime overheads - for ex-ample, the master node managing execution is typically abottleneck.

We propose a novel auto-parallelizing compiler approachfor this domain that does not suffer from the shortcomingsof prior methods [9, 13, 22, 38]. These efforts failed in prac-tice due to the complexity of the problem, especially fordistributed-memory architectures [17, 18, 21, 34]. The mainchallenge is that the compiler needs to know which datastructures and loops of the program should be parallelized,and how they should be mapped across the machine. Pre-vious algorithms for automatic parallelization typically startby analyzing array accesses of the program and finding allpossible array and computation distributions for loop-nests.Then, distributions are selected based on heuristics and/orcost models. A fundamental problem is that the approximatecost models and heuristics cannot find the best combinationof distributions from a potentially large search space reli-ably, leading to significantly lower performance comparedto manual parallelization. In contrast, we develop a data flowalgorithm that exploits domain knowledge as well as high-level semantics of mathematical operations to find the bestdistributions, but without using approximations such as costmodels. Hence, our approach is robust and matches manualparallelization.

1 2017/4/12

arX

iv:1

611.

0493

4v2

[cs

.DC

] 1

0 A

pr 2

017

In this paper, we use this approach to build High Per-formance Analytics Toolkit (HPAT)1. HPAT is a compiler-based framework for big data analytics on large-scale clus-ters that automatically parallelizes high-level analytics pro-grams and generates scalable and efficient MPI/C++ code.To the best of our knowledge, HPAT is the first compiler-based system that can parallelize analytics programs auto-matically and achieve high performance. Our contributionscan be summarized as follows:

• A robust domain-specific automatic parallelization algo-rithm (§4)

• Domain-specific optimization techniques (§4.2)• A system design addressing various challenges such as

parallel I/O code generation (§3)• A domain-specific automatic checkpointing technique

(§5)

Our evaluation demonstrates that HPAT is 369× to 2033×faster than Spark on the Cori supercomputer [3] and providessimilar performance to hand-written MPI/C++ programs.HPAT is also 20×-256× faster on Amazon AWS (§7).

2. Logistic Regression ExampleWe introduce HPAT and compare it to the state-of-the-art us-ing the logistic regression machine learning algorithm. Lo-gistic regression uses the logistic sigmoid function to derivea model for data classification [8]. This model is updatediteratively using the gradient descent optimization method.Figure 1a presents the HPAT code for logistic regression.The @acc hpat macro annotation and the DataSource con-struct are HPAT-specific, but the rest of the code is high-level sequential Julia code which uses matrix/vector oper-ations (corresponding to “vectorized” mathematical nota-tion). HPAT automatically parallelizes and optimizes thisprogram and generates efficient MPI/C++. HPAT’s auto-parallelization algorithm uses domain knowledge to inferdistributions for all variables and computations accurately;for example, vector w is replicated on all processors whilecolumns of matrix points are divided in block manner, whichis the same strategy for manual parallelization of this pro-gram. Previous auto-parallelization methods cannot achievethis since there is a large distribution space even for thissimple program; for example, even computations on w aredata-parallel and could be distributed without violation ofdependencies, but parallel performance suffers. HPAT alsofuses all computation steps in the iteration loop together toachieve best data locality.

The current state-of-the-art for big data analytics frame-works is the library approach, which can have significantoverheads. We use Apache Spark to demonstrate this ap-proach since it is a widely used big data analytics frame-

1 HPAT is available online as open-source at https://github.com/

IntelLabs/HPAT.jl.

using HPAT

@acc hpat function logistic_regression(iters ,file)points = DataSource(Matrix{Float64},HDF5 ,"points",

file)labels = DataSource(Vector{Float64},HDF5 ,"labels",

file)D,N = size(points)labels = reshape(labels ,1,N)w = 2*rand(1,D)-1for i in 1:iters

w -= ((1./(1+ exp(-labels.*(w*points)))-1).*labels)*points ’

endreturn w

end

weights = logistic_regression (100,"mydata.hdf5")

(a) HPAT logistic regression example (Julia).

from pyspark import SparkContext

D = 10 # Number of dimensions# read batch of points for faster Numpy computation.def readPointBatch(iterator):

strs = list(iterator)matrix = np.zeros((len(strs), D + 1))for i, s in enumerate(strs):

matrix[i] = np.fromstring(s.replace(’,’,’ ’),sep=’ ’)

return [matrix]

if __name__ == "__main__":sc = SparkContext(appName="PythonLR")points = sc.textFile(sys.argv [1]).mapPartitions(

readPointBatch).cache()iterations = int(sys.argv [2])w = 2*np.random.ranf(size=D)-1

def gradient(matrix , w):Y = matrix[:, 0] # unpack point labelsX = matrix[:, 1:] # unpack point coordinatesreturn ((1.0 / (1.0 + np.exp(-Y * X.dot(w))) - 1.

0) * Y * X.T).sum (1)

def add(x, y):x += yreturn x

for i in range(iterations):w -= points.map(lambda m: gradient(m, w)).reduce(

add)sc.stop()

(b) Spark logistic regression example (Python)

Figure 1: Logistic regression example codes.

2 2017/4/12

Figure 2: Performance of Logistic Regression on AmazonAWS (c4.8xlarge instances, 256M 10-feature samples, 20iterations).

work – other frameworks are similar in principle. Figure 1bpresents the Spark version, which initializes a resilient dis-tributed dataset (RDD) called points. An RDD is essentiallya distributed data structure that is distributed in one dimen-sion. In addition, the computation is written in terms of mapand reduce operators. In each iteration, the task schedulerof Spark runtime library in master node divides map/reduceoperators into tasks, and sends these tasks to the executor(slave) nodes. The operator closures which include the re-quired data (w in this case) are also serialized and sent. Onexecutors, each task is initialized and the RDD data is deseri-alized from the data cache into Python (Numpy) objects forexecution. Finally, the result of the reduce operator is sentback to the master node. This sequence is repeated for everyiteration since the library has to return the results of reduc-tions to the user context and is not aware of the iteration loopin the Python code. In essence, each iteration is a separatejob that is launched as a wave of small tasks; scheduling andserialization overheads incur repeatedly. This is a fundamen-tal problem with this library design and cannot be mitigatedeasily – significant Spark development effort has not closedthe performance gap with MPI/C++ codes (Spark has over1000 contributors [2]).

Figure 2 demonstrates the performance of logistic regres-sion algorithm in Spark, MPI/Python, HPAT, and MPI/C++.HPAT is 35× faster than Spark and is close to the hand-written MPI/C++ code. The gap between Spark and theMPI/Python code reveals the magnitude of overheads inSpark. The gap between MPI/Python code and MPI/C++is mostly due to locality advantages of fused loops as op-posed to executing a sequence of Numpy operations. Bridg-ing the performance gap with MPI/C++ codes is crucial forbig data analytics; for example, 91% of Spark users choseperformance among the most important aspects for them ina Spark survey [1]. Furthermore, distributed libraries are im-plemented in languages such as Java and Scala since reflec-

tion, serialization, and isolation features of a sandbox likeJava Virtual Machine (JVM) are required, but it can have sig-nificant overheads [23, 24, 28]. In contrast, HPAT achievesscripting productivity and HPC performance simultaneouslyusing an auto-parallelizing compiler approach.

3. HPAT OverviewIn this section, we provide an overview of our target appli-cation domain and the HPAT compilation pipeline.

Domain Characteristics: The goal of HPAT is to au-tomatically parallelize common analytics tasks such asdata-parallel queries and iterative machine learning algo-rithms. The main domain characteristic we exploit is thatthe map/reduce parallel pattern inherently underlies the tar-get application domain. This characteristic is assumed incurrent frameworks like Spark as well. Hence, distributionof parallel vectors and matrices is mainly one-dimensional(1D) for analytics tasks (i.e. matrices are “tall-and-skinny”).Furthermore, the workload is uniform across the data set(which can be extended by providing load balancing). Theseassumptions do not necessarily apply to other domains.For example, multi-dimensional distribution is often bestin many physical simulation applications. Note that whiletraditional HPC applications are sometimes developed overmany years, analytics programs need to be developed in aslittle as a few minutes in some cases to support the interac-tive workflow of data scientists (which makes productivityof scripting languages critical). Hence, analytics programsare often significantly shorter and simpler than many HPCapplications.

HPAT Coding Style: HPAT supports the high-level script-ing syntax of the Julia language, which is intuitive to domainexperts. However, users need to follow certain coding styleguidelines to make sure HPAT can analyze and parallelizetheir programs automatically:

• The analytics task is in functions annotated with @acchpat.

• I/O (e.g. reading input samples) is performed throughHPAT (using the DataSource and DataSink constructs).

• The data-parallel computations are in the form of high-level matrix/vector computations or comprehensionssince HPAT does not parallelize sequential loops.

• Julia’s column-major order should be followed for mul-tidimensional arrays since HPAT parallelizes across lastdimensions. For example, this means that features of asample are in a column of the samples matrix.

HPAT Compiler Pipeline: HPAT uses the @acc macroprovided by Julia’s CompilerTools package to configureHPAT’s compilation pipeline as shown in Figure 3. Thesolid boxes are HPAT components. The macro pass desugarsHPAT extensions (such as DataSource) into function calls

3 2017/4/12

Figure 3: HPAT’s compilation pipeline.

and type annotations to enable compilation by Julia. The Ju-lia compiler then performs further desugaring and type infer-ence on the function’s intermediate representation (IR). TheDomain-Pass transforms HPAT extensions into a form moreconducive to optimization by the subsequent Domain-IR andParallel-IR passes. The Domain-IR and Parallel-IR passesare provided by Julia’s ParallelAccelerator package [5].The Domain-IR pass identifies operators (e.g. vector-vectorelement-wise addition) and other constructs in the IR whosesemantics are parallelizable. The Parallel-IR pass takes thedifferent kinds of parallel constructs found by Domain-IRand transforms those into a common representation calledparfor (that represents a tightly nested for loop whose it-erations can all be performed in parallel) and performs fu-sion and other optimizations on the IR. These two passesare described in more detail in Section 6. Given this paral-lel representation, the Distributed-Pass partitions the arraysand parfors for distributed-memory execution and generatescommunication calls. HPAT Code Generation takes advan-tage of several hooks provided by CGen (part of ParallelAc-celerator) to generate C++ code.

4. Automatic ParallelizationOur novel auto-parallelization algorithm exploits domainknowledge in a data flow framework [27] to find data andcomputation distributions. The goal of the algorithm is tofind a consistent map/reduce view of the program that doesnot violate the high-level parallel semantics of any mathe-matical operation. Since the underlying parallel pattern ismap/reduce, each array should be either distributed in 1Dblock fashion (1D B) or replicated on all processors (REP).We also support a 2D block-cyclic distribution (2D BC) forthe less common square matrix computations, but this re-quires a user annotation. To give some intuition, 1D blockdistribution is typically applied to arrays such as datasets andmap operations on those datasets, whereas a replicated dis-tribution is used for an array containing a model and is oftenassociated with a reduction. We also know that data remap-ping is not needed in this domain so array distributions areglobal. On the other hand, the distribution of each computa-

tion step can be determined based on its high-level semantics(e.g. reductions) and the distribution of its inputs/outputs.

We construct our data flow framework as follows. Themeet-semilattice of distributions is defined as:L = {1D B, 2D BC,REP}, REP ≤ 2D BC ≤ 1D B

⊥ = REP , > = 1D BThe properties to infer are defined as

Pa : A→ L,Pp : P→ Lwhere A is the set of arrays in the program, P is the set ofparfors, Pa specifies distributions for arrays, and Pp speci-fies distributions for parfors. Other operations such as matrixmultiply and library calls also have similar properties, whichwe omit here for brevity. Next, we define the set of transferfunctions F for each node type based on its parallel seman-tics, which are used to update the properties. In essence, thisframework provides equation

(Pa,Pp) = F(Pa,Pp)which can be solved using a fixed-point iteration algorithm(repeatedly walks over the IR until quiescence). The initialdistributions are assigned as 1D B for all arrays and parfors(the top element in the semi-lattice). We design the trans-fer functions to be monotone, which is consistent with thesemantics of the operations in this domain since data remap-pings are not common. However, it is possible to remap databy inserting special remapping nodes that copy data to newarrays (left for future work). Monotonicity ensures that thefixed-point iteration algorithm converges to the least solu-tion [27], which means that higher values like 1D B are pre-served as much as possible while satisfying all the equations.This ensures maximum parallelization for the program. Pro-gram control flow (e.g. branches) can be safely ignored sincethe distributions do not change across different paths in thisdomain.Transfer Function: Array Assignment - The distribution ofleft-hand side and right-hand side of an assignment on arraysshould have the same distribution, which is the meet (∧) oftheir previously inferred distributions:

fl=r : Pa(l) = Pa(r) = Pa(l) ∧ Pa(r)Transfer Function: Calls - When the distribution algorithmencounters a call site, the algorithm determines whether itknows the transfer function for the function being called. Ifso, the call site is considered to be a known call; otherwise, itis called an unknown call. For unknown calls, such as func-tions from external modules not compiled through HPAT, thedistribution algorithm conservatively assumes the involvedarrays need to be REP. If the function has parallel semanticsfor arrays, the user needs to provide the information. Con-versely, distribution transfer functions are built into a HPATknownCallProps table for many Julia and HPAT operations.Common examples include Julia’s array operations (e.g. re-shape, array set, array get, array length), and HPAT’s datastorage functions.

funknown call g(x) : Pa(x) = REPfknown call g(x) : Pa(x) = Pa(x) ∧ knownCallProp(g)

4 2017/4/12

Transfer Function: Returned Array - For return statementsof the top-level function, the set of arrays being returned areeach flagged as REP since returned arrays need to fit on asingle node and this output typically represents a summa-rization of a much larger data set. If larger output is required,the program should write the output to storage. This is a use-ful domain-specific heuristic that facilitates compiler analy-sis.

freturn x : Pa(x) = REPTransfer Function: Matrix Operations - Matrix/matrixand matrix/vector multiply operations (GEMM/GEMV callnodes) have more complex transfer functions.

flhs=gemm(x,y) : GemmTransfer(lhs, x, y)Figure 4 illustrates the portion of the GEMM transfer

function that is exercised during auto-parallelization of thelogistic regression example of Figure 1a, as well as a 2D BCcase. These formulas are derived based on our matrix dis-tribution and layout assumptions, and semantics of GEMMoperations. Since samples are laid out in columns and wedo not split sample features across processors (1D B), anyvector or row of a matrix computed using reduction acrosssamples is inferred as REP. For example, the result of the in-ner formula in Logistic Regression’s kernel is multiplied bythe transpose of sample points and used to update the param-eters (w− = (· · · . ∗ labels) ∗ points′). Using this analysis,the algorithm infers that the output of the operation is REP ifboth inputs are 1D B and the second one is transposed. Thisalso means that the inputs can stay 1D B. In this case, a re-duction is also inferred for the node (which eventually turnsinto MPI Allreduce in the backend). Furthermore, since thevector w is used in a dot product with matrix columns inw ∗ points, it should be REP and the matrix stays as 1D B.Transfer Function: Parfor - As described in Section 3,parfor nodes represent data-parallel computation and requirespecial handling during distribution analysis.

fparforx : applyParforRules(x)Figure 5 illustrates the transfer function for parfors. For

clarity, this figure is simplified and only shows how the com-mon case of 1D B arrays are handled. As with array distri-bution, we start by assuming that the parfor is 1D B untilproven otherwise. First, we analyze the body of the parforand extract all the array access/indexing operations. We theniterate across all the array accesses. Since HPAT parallelizes1D B arrays and parfors across the last dimension, the indexvariable of the last loop of the parfor is used for testing. First,we check if the index expression for the last dimension of thearray access (i.e., index exprN) is identical to the parfor’s in-dex variable allocated to the last dimension of the loop nest.If so and the array being accessed is REP then the parfor it-self becomes REP (the meet of two distributions is chosen).Second, we check whether the last dimension’s parfor loopindex variable is used directly or indirectly (e.g. temp = par-for.LoopsNests[end].index var; index expr1 = temp + 1) inany of the array access index expressions for the first N-1 di-

Procedure GemmTransfer(lhs,x,y)if is1D(x) ∧ is1D(y) ∧ ¬isTransposed(x) ∧isTransposed(y) then// e.g. (· · · .*labels)*points’ -

reduction across samples

Pa(lhs) = REPelse if ¬is2D(x) ∧ is1D(y) ∧¬isTransposed(y) ∧ is1D(lhs) then// e.g. w*points - dot product with

sample features

Pa(x) = REPelse if ¬isREP(x) ∧ ¬isREP(y) ∧ ¬isREP(lhs)∧ (is2D(x) ∨ is2D(y) ∨ is2D(lhs)) then// If any array is 2D, all arrays

should be 2D

Pa(lhs) = Pa(x) = Pa(y) = 2D BCelse// all replicated if no rule applied

Pa(lhs) = Pa(x) = Pa(y) = REP

return

Figure 4: HPAT distribution analysis for matrix multiply.

Procedure applyParforRules(parfor)distribution = 1DmyArrays = ∅arrayAccesses =

extractArrayAccesses(parfor.body)parforIndexVar = parfor.LoopNests[end].index varfor each arrayAccess ∈ arrayAccesses do

if parforIndexVar =arrayAccess.index expr[end] then

myArrays = myArrays ∪arrayAccess.array

distribution =distribution ∧ Pa(arrayAccess.array)

endif parforIndexVar ∈

arrayAccess.index expr[1:end-1] thendistribution = REP

endif isDependent(arrayAccess.index expr[1:end],

parforIndexVar) thendistribution = REP

endendPp(parfor) = distribution

for each array ∈ myArrays doPa(array) = distribution

endreturn

Figure 5: HPAT inference rules for parfors.

5 2017/4/12

mensions. If so, then the parfor must be REP. These tests areconservative but do not typically prevent parallelization ofcommon analytics codes. For accesses to 2D BC arrays, theabove algorithm has a straight-forward extension that con-siders not one but the last two parfor index variables andarray index expressions.

Algorithm: SGDsamples = read input()w = initial w()for each iteration do

w = gradient update(samples, w)endreturn w

Figure 6: Typical gradient descent method.

4.1 Effectiveness on Data Analytics ProgramsThe main reason HPAT’s heuristics are effective is thatdata analytics programs typically produce a summary oflarge data sets. In the case of machine learning algorithms,this summary is the weights of the trained model. Morespecifically, many large-scale machine learning algorithmsoptimization methods such as stochastic gradient descent(SGD) [10]. Hence, their structure can be represented as inFigure 6. Parameter set w is updated iteratively using gra-dient updates in order to minimize a form of cost functionon samples. HPAT’s analysis can infer that samples is dis-tributed since it will be accessed in a data parallel manner.It will also infer that w is replicated since it is updated usinga form of reduction. For example, variable w in Figure 1ais updated by a matrix/vector multiplication that implies areduction.

4.2 Domain-specific OptimizationsBefore distributed code generation, HPAT applies two noveldomain-specific optimization heuristics. HPAT performsthese since typical compiler cost models cannot reliablyconclude that the complex transformations described be-low are beneficial; for example, we found ICC and GCCunable to perform these optimizations on any of our bench-marks. Hence, HPAT’s use of domain knowledge is critical.For the first heuristic, we observe that matrix multiplications(GEMM calls) with at least one “tall-and-skinny” input arecommon in data analytics algorithms (e.g. w ∗ points inFigure 1). Ideally, these GEMM calls should be replacedwith equivalent loop-nests and fused with other operationsto achieve a single pass through data points and intermediateresults:

HEURISTIC 1. GEMM operations with one or more 1D Binputs are replaced with loop-nests and fusion is called onthe basic block.

using HPAT

@acc hpat function kmeans(numCenter , iters , file)points = DataSource(Matrix{Float64},HDF5 ,"points",

file)D,N = size(points)centroids = rand(D, numCenter)for l in 1:iters

dist = [Float64[sqrt(sum(( points[:,i]-centroids[:,j]).^2)) for j in 1:numCenter] for i in 1:N]

labels = Int[indmin(dist[i]) for i in 1:N]centroids = Float64[ sum(points[j,labels.==i])/

sum(labels.==i) for j in 1:D, i in 1:numCenter]

endreturn centroids

end

centroids = kmeans (5,100,"mydata.hdf5")

Figure 7: HPAT K-Means example.

To enable fusion, HPAT arranges the loop-nests so thatthe long dimension is outer-most. This optimization causesall mathematical operations of the logistic regression algo-rithm in Figure 1 to be fused. Hence, each data point isloaded just once, which improves performance by maximiz-ing locality. Furthermore, memory consumption is improvedsince saving intermediate data into memory is avoided.

The second heuristic is based on the observation thatloops over data sets and their intermediate results can oc-cur inside other loops. For example, the centroids calcula-tion in Figure 7 is written using nested comprehensions thatinclude passes over points and labels inside. This preventsmaximum loop fusion and causes multiple passes over thosedata sets. Furthermore, extra communication is then gener-ated for each iteration of the outer loop-nest instead of asinge communication call for all data exchanges. Thus, werearrange loop-nests to avoid this problem:

HEURISTIC 2. Parfors with REP distribution that have1D B parfors inside are interchanged and fusion is calledon the basic block.

HPAT performs loop fission on the REP parfor beforeinterchange since the body may have more statements. Thisoptimization maximizes fusion in the kmeans example ofFigure 7 and ensures a single pass over the data set.

4.3 Automatic Parallel I/OFigure 8 demonstrates how HPAT’s compilation pipelinetranslates DataSource syntax to parallel I/O code (DataSinkis similar). Macro-Pass desugars the syntax into a HPATplaceholder special function call (e.g. HPAT h5 read) andtype-annotates the output array so that Julia’s type infer-ence can work. Domain-Pass generates function calls to getthe size of the array and allocations so that Domain-IR andParallel-IR can optimize the program effectively. Alloca-tions and size variables are critical information that enablefusion and elimination of intermediate arrays.

6 2017/4/12

Figure 8: HPAT’s DataSource compilation pipeline.

Distributed-Pass enables parallel I/O by distributing theinput array among nodes and adding the start and end in-dices for each dimension. HPAT Code Generation’s func-tion call replacement mechanism generates the backendHDF5 code (currently MPI/C++) for placeholders such asHPAT h5 read. HPAT also supports text files using MPI I/Obecause many big data files are stored as text.

4.4 Distributed-Memory TranslationDistributed-Pass transforms the function for distributed-memory execution and generates communication calls af-ter auto-parallelization provides distributions. The passalso inserts calls into the IR to query the number of pro-cessors and to get the node number. Allocations of dis-tributed arrays are divided among nodes by inserting codeto calculate size based on the number of processors (e.g.mysize = total/num pes). Distributed (1D B) parfors aretranslated by dividing the iterations among processors usingnode number and number of processors and updating ar-ray indices accordingly. For example, A[i] is replaced withA[i −mystart] where mystart contains the starting indexof the parfor on the current processor. Furthermore, for par-fors with reductions, communication calls for distributed-memory reductions are generated.

Furthermore, matrix/vector and matrix/matrix multipli-cation (GEMM calls) need special handling. For instance,w ∗ points in Figure 1a does not require communicationsince replication of w makes it data-parallel, while (· · · . ∗labels) ∗ points′ requires an allreduce operation since botharguments are distributed. The Distributed-Pass makes thisdistinction by using the parallelization information providedby previous analyses. In the first case, the first input is repli-cated while the second input and the output are distributed.In the second case, both inputs of GEMM are distributed butthe output is replicated.

4.5 Backend Code GenerationOur approach facilitates using various backends, since Distributed-Pass returns a high-level parallel IR that enables flexiblecode generation. Our current prototype supports MPI/C++and MPI/OpenMP/C++, which we found to be suitable for

using HPAT

@acc hpat function calcNaiveBayes(num_classes ,file_name)points = DataSource(Matrix{Float64},HDF5 ,"/points

",file_name)labels = DataSource(Vector{Float64},HDF5 ,"/labels

", file_name)coeffs = HPAT.NaiveBayes(points , labels ,

num_classes)return coeffs

end

Figure 9: HPAT library call example.

many analytics workloads. However, some cases might re-quire using a backend with an adaptive runtime system, dueto scheduling and load balancing requirements. Hence, onemight use Charm++ [7] or HPX [20] as the backend forHPAT. Evaluation of these backends for HPAT is left forfuture work.

On the other hand, Spark uses a runtime system withDAG scheduling, which is required for implementation ofcomplex operations (such as shuffling) on top of Spark’smap/reduce core. However, these operations do not necessar-ily need runtime scheduling. In general, Spark surveys showthat only a small fraction of users run irregular workloadssuch as graph workloads on Spark [1, 2]. Nevertheless, ourapproach enables the use of low-overhead HPC runtime sys-tems, avoiding Spark’s overheads.

4.6 Automatic Utilization of Distributed-MemoryLibraries

Libraries are essential components of productive analyticsplatforms. For example, Spark provides MLlib [6], whichis implemented using its RDD data format. Since HPAT iscompiler based, it can take advantage of distributed-memorylibraries without requiring changes to their data structuresand interfaces. On the other hand, only libraries that imple-ment interoperation with Spark’s RDDs can be used withSpark; this is a fundamental limitation for library-basedframeworks.

Figure 9 shows example code that calls a library. Thisfunction call goes through the compilation pipeline as a spe-cial known function call, i.e., it has entries in HPAT anal-ysis tables. Most importantly, the HPAT parallelization al-gorithm knows that the input arrays can be 1D B, whilethe output array is REP. If parallelization is successful, theDistributed-Pass adds two new arguments to the functioncall; the first index and the last index of the input array oneach node. HPAT’s CGen extension uses a MPI/C++ coderoutines for code generation of the call. Currently, HPATsupports ScaLAPACK and Intel R© Data Analytics Acceler-ation Library (Intel R© DAAL) [4] as an alternative to MLlibfor machine learning algorithms.

Performance evaluation of analytics programs that use li-braries (e.g. MLlib or DAAL) is beyond the scope of this

7 2017/4/12

using HPAT

@acc hpat function matrix_multiply(file1 ,file2 ,file3)@partitioned(M,2D);M = DataSource(Matrix{Float64},HDF5 ,"/M", file1)x = DataSource(Matrix{Float64},HDF5 ,"/x", file2)y = M*xy += 0.1*randn(size(y))DataSink(y,HDF5 ,"/y", file3)

end

Figure 10: HPAT matrix multiply example.

paper. Comparing libraries is non-trivial in general; data an-alytics functions can be implemented using multiple algo-rithms with different computational complexities. Our ob-jective is to generate efficient code for scripting codes whichare dominantly used in this area.

4.7 2D ParallelizationHPAT requires a simple annotation to assist the automaticparallelization algorithm for the less common cases that re-quire 2D parallelization. Consider the example program inFigure 10 (a real-world case requested by a user). The pro-gram reads two (multi-terabyte) matrices, multiplies them,adds some random value to the result, and writes it back tostorage. The user specifies that matrix M requires 2D parti-tioning. HPAT infers that x and y matrices also need 2D par-titioning as well. Furthermore, the related intermediate vari-ables in the AST (such as the random matrix created) andthe parfors are also 2D partitioned. In the backend, HPATgenerates MPI/C++ code which calls parallel HDF5 for I/Oand ScaLAPACK (PBLAS component) for matrix multipli-cation.

Manually developing efficient parallel code for this pro-gram is challenging. ScaLAPACK requires block-cyclic par-titioning of input and output data but HDF5 provides ablock-based read/write interface for parallel I/O. Hence, thegenerated code has to sets 96 indices, since there are threeI/O operations; each operation requires four hyperslab selec-tions including corner cases, and each hyperslab selectionrequires start, stride, count and block size indices for twodimensions. In addition, ScaLAPACK’s legacy interface isFortran-based and has its own intricacies. As a result, thegenerated MPI/C++ code is 525 lines. Therefore, manuallydeveloping this code is highly error-prone for domain ex-perts and HPAT’s automation is significant.

This use case demonstrates a fundamental advantage ofour compiler approach. Library-based frameworks are basedon fixed distributed data structures with specific partitioningand layout formats. For example, Spark’s RDDs are 1Dpartitioned and the whole framework (e.g block managerand scheduler) is based on this 1D partitioning. Supportingdifferent partitionings and layouts is easy for a compiler, butis difficult for a library.

5. CheckpointingResiliency is essential for long-running iterative machinelearning algorithms. The challenge is to provide low-overheadfault tolerance support without significant programmer in-volvement. Previous work on automatic checkpointing can-not achieve minimal checkpoints since, for example, thosesystems do not have the high-level knowledge that modelsare replicated and one copy is enough [29, 32, 36]. Spark’sapproach is to use lineage to restart shortest possible tasksbut this is made possible only by a system design withhigh overheads. HPAT provides automated application-levelcheckpointing (ALC) based on domain characteristics.

For iterative machine learning applications, only thelearning parameters and the loop index need to be check-pointed since the data points are read-only. Moreover, theselearning parameters are replicated on all processors so onlyone copy needs to be checkpointed. We use these domaincharacteristic in designing HPAT’s checkpointing capabil-ity. Hence, HPAT checkpointing assumes a typical analyt-ics function containing a single outer-loop and having theform: initialization, iteration loop, results. In the initializa-tion phase, input is loaded and variables initialized, whichestablish the invariants for entry into the loop. The body ofthe outer-loop can be arbitrarily complex including contain-ing nested loops. In the results phase, the outputs of the loopare used to compute the final result of the function. This ap-proach is readily extensible to support multiple outer-loopsas well.

If enabled, HPAT adds a checkpointing pass to the com-pilation pipeline after Domain-Pass. The checkpointing passfirst locates the outer-loop and analyzes it to determinewhich variables are live at entry to the loop (including theloop index) and are written in the loop. This set of iteration-dependent variables are saved as part of a checkpoint. Thepass creates a new checkpoint function tailored to this set ofvariables and inserts a call to that function as the first state-ment of the loop. In this function, MPI rank zero comparesthe time since the last checkpoint was taken with the nextcheckpoint time as calculated using Young’s formula [39].If it is time to take a checkpoint, rank zero calls the HPATruntime to start a checkpointing session, write each variableto the checkpoint, and then end the session. The HPAT run-time records the time to take the checkpoint and uses thisinformation to improve the estimated checkpoint time that isinput to Young’s formula. At the exit of the outer-loop, thepass also inserts a call to the HPAT runtime to indicate thatthe checkpointed region has completed and that any savedcheckpoints can be deleted.

To restart a computation, the programmer calls the HPATrestart routine and passes the function to be restarted andthe original arguments. The HPAT compiler creates a restartversion of the function that is identical to the original butwith the addition of checkpoint restore code before the entryto the loop. This checkpoint restore code finds the saved

8 2017/4/12

checkpoint file and loads the iteration-dependent variablesfrom the checkpoint. In this way, the initialization code isperformed again (restoring read-only variables), and the loopfast-forwards to the last successfully checkpointed iteration.An important consideration is that the iterations should bedeterministic since some might be re-executed during restart.

Consider the logistic regression example in Figure 1a; westore only the loop index i and w in the checkpoint whereasthe full set of live data would include points and labelsand would result in checkpoints orders of magnitude larger.As far as we are aware, this checkpointing approach thatexploits domain knowledge by for example re-executing theinitialization phase is novel. A key insight is that HPAT canperform the analysis for this minimal checkpointing, while alibrary approach like Spark is unable to do so.

6. ParallelAccelerator InfrastructureHPAT relies on the ParallelAccelerator package [5] for dis-covering potential parallelism in dense array operationsof Julia. ParallelAccelerator consists of three main com-piler passes, Domain-IR, Parallel-IR, and CGen. Domain-IR looks for operations and other constructs in Julia’s IRthat have different kinds of parallel semantics and then re-places those operations with equivalent Domain-IR nodesthat encode those semantics. Some of the most common par-allelism patterns in Domain-IR are map, reduce, Cartesianmap, and stencil. For example, Domain-IR would identifyunary vector operations (such as -, !, log, exp, sin, and cos)and binary, element-wise vector-vector or vector-scalar op-erations (such as +, -, *, /, ==, !=, <, and >) as havingmap semantics. Likewise, Julia’s sum() and prod() functionswould be identified by Domain-IR as having reduce seman-tics. Domain-IR identifies comprehensions within the Juliacode as having Cartesian map semantics.

The Parallel-IR pass lowers Domain-IR nodes to a com-mon representation called parfor. Once in this represen-tation, Parallel-IR performs parfor fusion between parforscoming from potentially dissimilar Domain-IR nodes. Thisfusion process reduces loop overhead and eliminates manyintermediate arrays, helping the program to have better lo-cality. There are three main components of the parfor repre-sentation: loop nests, reductions, and body. Every parfor hasa loop nest that represents a set of tightly nested for loops. Itis typical for the number of such loops to match the numberof dimensions of the array on which the parfor operates. Theparfor reductions component is only present when the par-for involves a reduction and encodes the variable holding thereduction value along with its initial value and the functionused to combine reduction elements. Finally, the body of theparfor contains code to compute the result of the parfor for asingle point in the iteration space. After the Parallel-IR passis finished, CGen converts the IR to OpenMP-annotated Ccode in the backend.

Table 1: Benchmark sizes and parameters.

Benchmark Input size/iterations on Cori AWS

Kernel Density 2B/(NA) 256M/(NA)Linear Regression 2B/20 (10 features, 4 models) 256M/20Logistic Regression 2B/20 (10 features) 256M/20K-Means 320M/20 (10 features, 5 centroids) 64M/20

7. EvaluationWe compare the performance of Spark, HPAT and hand-written MPI/C++ programs on the Cori supercomputerat NERSC [3] and Amazon AWS cloud. Cori is a CrayXC40 supercomputer that includes advanced features fordata-intensive applications such as large memory capac-ity and high I/O bandwidth. Each node (Phase I) has twosockets, each of which is a 16-core Intel R©Xeon R©E5-2698v3 2.3GHz (Haswell architecture). The memory capacityof each node is 128GB. On AWS, we use the compute-optimized C4 instances (c4.8xlarge with 36 vCPUs), whichfeature Intel R©Xeon R©E5-2666 v3 (Haswell) processors.Each instance has 60GB of memory. We use PlacementGroups which provide low latency, full bisection, 10Gbpsbandwidth between instances. We use Intel R©C++ Compiler(ICC) 17.0 for backend C++ compilation. The default Spark2.0.0 installation on Cori is used which is tuned and sup-ported.

We use the benchmarks listed in Table 1 for evaluation.We believe they are representative of many workloads indata analytics; related studies typically use the same or simi-lar benchmarks [11, 19, 28, 31]. Benchmark sizes are chosenso that they fit in the memory, even with excessive memoryusage of Spark, to make sure Spark’s performance is not de-graded by accessing disks repeatedly. HPAT is capable ofgenerating MPI/OpenMP but currently, it turns OpenMP offand uses MPI-only configuration since OpenMP code gener-ation is not tuned yet. We use one MPI rank per core (withouthyperthreading). The Spark versions of the benchmarks arebased on the available examples in Spark’s open-source dis-tribution. We developed the MPI/C++ programs by simplydividing the problem domain across ranks equally and ensur-ing maximum locality by fusing the loops manually. ParallelI/O times are excluded from all results. MPI/C++ codes areabout 6× longer in lines of code compared to HPAT codes.

Figure 11 compares the performance of Spark, manualMPI/C++, and HPAT on 64 nodes (2048 cores) of Cori andfour nodes of Amazon AWS. HPAT is 369×-2033× fasterthan Spark for the benchmarks on Cori and is only 2×-4×slower than manual MPI/C++. In addition, HPAT is 20×-256× faster than Spark on Amazon AWS. The lower perfor-mance of Spark on Cori is expected since the master nodeis a bottleneck and prevents scaling. Kernel Density demon-strates the largest performance gap among the benchmarks;HPAT is 2033× faster than Spark on Cori and 256× onAWS. The reason is that computation per element is small

9 2017/4/12

(a) Cori, 64 nodes. (b) Amazon AWS, 4 instances (c4.8xlarge).

Figure 11: Performance comparison of Spark, manual MPI/C++, and HPAT. Note the logarithmic scales.

and the Spark overheads such as serialization/deserializationand master-executor coordination are amplified.

Automatic parallelization by HPAT matches the manualparallelization for all of the benchmarks perfectly, whichdemonstrates the robustness of our auto-parallelization algo-rithm. Furthermore, loop structures are identical to the man-ual versions which demonstrates the success of HPAT’s fu-sion transformations. The performance gap between HPATand MPI/C++ codes can be attributed to the the generatedcode being longer and containing extra intermediate vari-ables; this makes code generation phase of the backend C++compiler more challenging. The optimization reports of ICCsupport this hypothesis. We hope to match the performanceof manual codes in future work.

We use the Python interface of Spark in this paper asbaseline since scripting languages are preferred by domainexperts. One may argue that using Scala, which is staticallycompiled, might be significantly faster in Spark. We testedthis hypothesis and found that Scala is moderately faster forsome benchmarks (e.g. K-Means) but HPAT is still severaltimes faster than Spark. For other benchmarks (e.g. LogisticRegression) Python is actually faster since the computationis inside Numpy operations, which have optimized nativebackends (Anaconda distribution). Furthermore, the flexibil-ity of Numpy allows batched processing while Scala’s breezelinear algebra library does not have this capability.

We use an ADMM LASSO solver [35] to evaluate the ef-fectiveness of our auto-parallelization method on a complexalgorithm. The code is originally written in Python and par-allelized using mpi4py by a domain expert. However, themanual parallelization method sacrifices accuracy for easierparallelization. On the other hand, HPAT is able to paral-lelize the code effectively and accurately. Figure 12 demon-strates the scaling on up to 64 nodes. The slowdown of the

Figure 12: Strong scaling of ADMM LASSO. Note the log-arithmic scales.

MPI/Python code running in parallel is partially due to ac-curacy loss which forces the algorithm to run up to specifiedmaximum number of iterations. The success of HPAT’s auto-parallelization on this algorithm provides confidence aboutthe effectiveness of our approach, since we do not expectanalytics algorithms to be significantly more complex thanthis case in practice.

Compiler feedback and control: Previous compiler ap-proaches are hard to understand and control by users sincethey use complex heuristics and cost models. HPAT’s au-tomatic parallelization algorithm inherently facilitates usercontrol since it is deterministic and can be controlled easily.For example, HPAT provides the operations that caused eachREP inference. The user is then able to change paralleliza-tion behavior by explicitly annotating parallelization for ar-

10 2017/4/12

rays or providing more information for operations (e.g. par-allelization for library calls not previously known to HPAT).

8. Related WorkPrevious studies demonstrate that current big data analyticsframeworks are orders of magnitude slower than hand-tunedMPI implementations [11, 25, 31]. To improve the perfor-mance of data analytics frameworks, some previous stud-ies have focused on more efficient inter-node communica-tion [19, 41] but they do not address the fundamental per-formance bottlenecks resulting from the library approach.HPAT solves this problem using a novel automatic paral-lelization approach.

Automatic parallelization is studied extensively in HPC,but it is known that auto-parallelizing compilers cannotmatch the performance of hand-written parallel programs formany applications [9, 17, 18, 34, 38]. For example, previousstudies have tried to automate data alignment (TEMPLATEand ALIGN directives) and distribution (DISTRIBUTE di-rective) steps in High Performance Fortran (HPF) with lim-ited success [13, 21, 22]. More specifically, our distributionanalysis algorithm can be compared with the frameworkby Kennedy and Kremer [22]. This framework performs asearch of possible alignment and distribution combinationsfor loop-nests; a performance model helps predict the com-bination with the best performance. However, this frame-work cannot find the best combination reliably due to the in-accuracies of the performance model. Conversely, the HPATalgorithm leverages domain knowledge and assigns distribu-tions to arrays and computations based on semantics of dif-ferent high-level operations. To the best of our knowledge,this is the only algorithm to use a data flow formulation andis therefore novel. In general, previous work targeting sci-entific computing applications relies on complex analysis,performance models and approximations that are known tohave challenges in a practical setting [21, 34]. In contrast,our algorithm does not rely on models and approximations,and is therefore accurate (matches manual parallelization).

Distributed Multiloop Language (DMLL) provides com-piler transformations on map-reduce programs on distributedheterogeneous architectures [11]. However, HPAT startsfrom higher level scripting code, which might have oper-ations like matrix-multiply that do not have a simple map-reduce translation. More importantly, DMLL relies on userannotations and a simple partitioning analysis pass for datapartitioning but HPAT is fully automatic. We believe that ouriterative partitioning analysis algorithm produces more effi-cient code since it does not parallelize all potentially paralleloperations like DMLL’s approach. This can affect commu-nication cost on distributed-memory operations significantlysince DMLL broadcasts local data structures. This could bethe reason for DMLL’s much smaller speedups over Sparkon CPU clusters.

Distributed Halide is a domain-specific compiler for im-age processing that translates high-level stencil pipelinesinto parallel code for distributed-memory architectures [16].Unlike HPAT, 2D partitioning and near neighbor communi-cation are the norm and not 1D partitioning and reductions.Moreover, Halide requires user “schedule” for optimizationand distributed execution while HPAT is fully automatic.

Several previous efforts such as Delite [33], Copper-head [12], and Intel R©Array Building Blocks [26] pro-vide embedded domain-specific languages (DSLs) that arecompiled for parallel hardware. HPAT’s design is simi-lar in many aspects, but HPAT targets data analytics fordistributed-memory architectures (with more accurate auto-parallelization). Furthermore, HPAT uses the abstractions ofthe host language and avoids introducing new abstractionsto the programmer as much as possible.

Systems that automate application-level checkpointingto some degree have been proposed before [29, 32, 36].For example, in the method by Plank et al. [29], the pro-grammer adds checkpoint locations and the system excludesdead variables and read-only data from checkpoints. In con-trast, HPAT’s scheme is domain-specific and, for example,uses the knowledge that the learning parameters in HPATare replicated; checkpointing them does not require an MPIconsistency protocol. HPAT also completely automates thecheckpointing process whereas other systems require pro-grammer effort to some degree.

9. Conclusion and Future WorkLibrary-based big data analytics frameworks such as Sparkprovide programmer productivity but they are much slowerthan hand-tuned MPI/C++ codes due to immense runtimeoverheads. We introduced an alternative approach based on anovel auto-parallelization algorithm, which is implementedin High Performance Analytics Toolkit (HPAT). HPAT pro-vides the best of both worlds: productivity of scripting ab-stractions and performance of efficient MPI/C++ codes. Ourevaluation demonstrated that HPAT is 369×-2033× fasterthan Spark. We plan to expand HPAT to provide more dataanalytics features and use cases. For example, providing sup-port for sparse computations, data frames (heterogeneous ta-bles), out-of-core execution is under investigation. Most ofthese features need research on multiple layers; from script-ing abstractions to compilation techniques and code gener-ation. We are also building a prototype HPAT system forPython.

References[1] 2015. Apache Spark Survey 2015 Report. http://go.

databricks.com/2015-spark-survey/. (2015).

[2] 2016. Apache Spark Survey 2016 Report. http://go.

databricks.com/2016-spark-survey/. (2016).

[3] 2016. Cori Supercomputer at NERSC. http://www.nersc.gov/users/computational-systems/cori/. (2016).

11 2017/4/12

[4] 2016. Intel Data Analytics Acceleration Library. https:

//software.intel.com/en-us/daal. (2016).

[5] 2016. ParallelAccelerator Julia package. https://github.com/IntelLabs/ParallelAccelerator.jl. (2016).

[6] 2016. Spark Machine Learning Library (MLlib)Guide. http://spark.apache.org/docs/latest/

mllib-guide.html. (2016).

[7] Bilge Acun, Abhishek Gupta, Nikhil Jain, Akhil Langer,Harshitha Menon, Eric Mikida, Xiang Ni, Michael Rob-son, Yanhua Sun, Ehsan Totoni, Lukasz Wesolowski, andLaxmikant Kale. 2014. Parallel Programming with Migrat-able Objects: Charm++ in Practice (SC’14).

[8] Christopher M. Bishop. 2006. Pattern Recognition andMachine Learning (Information Science and Statistics).Springer-Verlag New York, Inc.

[9] William Blume, Ramon Doallo, Rudolf Eigenmann, JohnGrout, Jay Hoeflinger, Thomas Lawrence, Jaejin Lee, DavidPadua, Yunheung Paek, Bill Pottenger, Lawrence Rauchw-erger, and Peng Tu. 1996. Parallel Programming with Polaris.Computer 29, 12 (1996).

[10] Leon Bottou. 2010. Large-scale machine learning withstochastic gradient descent. In Proceedings of COMP-STAT’2010.

[11] Kevin J. Brown, HyoukJoong Lee, Tiark Rompf, Arvind K.Sujeeth, Christopher De Sa, Christopher Aberger, and KunleOlukotun. 2016. Have Abstraction and Eat Performance, Too:Optimized Heterogeneous Computing with Parallel Patterns.In CGO.

[12] Bryan Catanzaro, Michael Garland, and Kurt Keutzer. 2011.Copperhead: Compiling an Embedded Data Parallel Lan-guage (PPoPP ’11).

[13] Siddhartha Chatterjee, John R. Gilbert, Robert Schreiber, andShang-Hua Teng. 1995. Optimal Evaluation of Array Expres-sions on Massively Parallel Machines. ACM Trans. Program.Lang. Syst. 17, 1 (1995).

[14] J. C. Chaves, J. Nehrbass, B. Guilfoos, J. Gardiner, S. Ahalt,A. Krishnamurthy, J. Unpingco, A. Chalker, A. Warnock,and S. Samsi. 2006. Octave and Python: High-Level Script-ing Languages Productivity and Performance Evaluation. InHPCMP Users Group Conference, 2006.

[15] Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: sim-plified data processing on large clusters. Commun. ACM(2008).

[16] Tyler Denniston, Shoaib Kamil, and Saman Amarasinghe.2016. Distributed Halide (PPoPP ’16).

[17] R. Eigenmann, J. Hoeflinger, and D. Padua. 1998. On theautomatic parallelization of the Perfect Benchmarks(R). IEEETransactions on Parallel and Distributed Systems 9, 1 (1998).

[18] Seema Hiranandani, Ken Kennedy, Chau Wen Tseng, andScott Warren. 1994. The D Editor: A New Interactive ParallelProgramming Tool (Supercomputing ’94).

[19] N. Islam, W. Rahman, X. Lu, D. Shankar, and D. Panda.2015. Performance Characterization and Acceleration of In-Memory File Systems for Hadoop and Spark Applications on

HPC Clusters. In 2015 IEEE International Conference on BigData.

[20] Hartmut Kaiser, Thomas Heller, Bryce Adelstein-Lelbach,Adrian Serio, and Dietmar Fey. 2014. HPX: A Task BasedProgramming Model in a Global Address Space (PGAS ’14).

[21] Ken Kennedy, Charles Koelbel, and Hans Zima. 2007. TheRise and Fall of High Performance Fortran: An HistoricalObject Lesson (HOPL III).

[22] Ken Kennedy and Ulrich Kremer. 1998. Automatic Data Lay-out for Distributed-memory Machines. ACM Trans. Program.Lang. Syst. 20, 4 (1998).

[23] David Lion, Adrian Chiu, Hailong Sun, Xin Zhuang, NikolaGrcevski, and Ding Yuan. 2016. Don’t Get Caught inthe Cold, Warm-up Your JVM: Understand and EliminateJVM Warm-up Overhead in Data-Parallel Systems. In OSDI.USENIX Association.

[24] Martin Maas, Tim Harris, Krste Asanovic, and John Kubia-towicz. 2015. Trash Day: Coordinating Garbage Collection inDistributed Systems. In HotOS.

[25] Frank McSherry, Michael Isard, and Derek G Murray. 2015.Scalability! but at what COST? (HotOS).

[26] Chris J. Newburn, Byoungro So, Zhenying Liu, Michael Mc-Cool, Anwar Ghuloum, Stefanus Du Toit, Zhi Gang Wang,Zhao Hui Du, Yongjian Chen, Gansha Wu, Peng Guo,Zhanglin Liu, and Dan Zhang. 2011. Intel’s Array BuildingBlocks: A Retargetable, Dynamic Compiler and EmbeddedLanguage (CGO ’11).

[27] Flemming Nielson, Hanne R Nielson, and Chris Hankin.2015. Principles of program analysis. Springer.

[28] Kay Ousterhout, Ryan Rasti, Sylvia Ratnasamy, ScottShenker, and Byung-Gon Chun. 2015. Making Sense of Per-formance in Data Analytics Frameworks (NSDI’15).

[29] J. S. Plank, M. Beck, and G. Kingsley. 1995. Compiler-Assisted Memory Exclusion for Fast Checkpointing. IEEETechnical Committee on Operating Systems and ApplicationEnvironments 7, 4 (1995).

[30] Lutz Prechelt. 2000. An Empirical Comparison of SevenProgramming Languages. IEEE Computer 33, 10 (Oct. 2000),23–29.

[31] Jorge Luis Reyes-Ortiz, Luca Oneto, and Davide Anguita.2015. Big Data Analytics in the Cloud: Spark on Hadoop vsMPI/OpenMP on Beowulf. In INNS Conference on Big Data2015.

[32] Martin Schulz, Greg Bronevetsky, Rohit Fernandes, DanielMarques, Keshav Pingali, and Paul Stodghill. 2004. Im-plementation and Evaluation of a Scalable Application-LevelCheckpoint-Recovery Scheme for MPI Programs (SC ’04).

[33] Arvind K. Sujeeth, Kevin J. Brown, Hyoukjoong Lee, TiarkRompf, Hassan Chafi, Martin Odersky, and Kunle Oluko-tun. 2014. Delite: A Compiler Architecture for Performance-Oriented Embedded Domain-Specific Languages. ACMTrans. Embed. Comput. Syst. 13, 4s (2014).

[34] Abdul Waheed, Michael Frumkin, Jerry Yan, Haoqiang Jin,and Michelle Hribar. 1998. A Comparison of Automatic Par-

12 2017/4/12

allelization Tools/Compilers on the SGI Origin 2000. SC’98(1998).

[35] Bo Wahlberg, Stephen Boyd, Mariette Annergren, and YangWang. 2012. An ADMM algorithm for a class of total varia-tion regularized estimation problems. IFAC Proceedings Vol-umes 45, 16 (2012), 83–88.

[36] Panfeng Wang, Xuejun Yang, Hongyi Fu, Yunfei Du, ZhiyunWang, and Jia Jia. 2008. Automated Application-level Check-pointing Based on Live-variable Analysis in MPI Programs(PPoPP ’08). 273–274.

[37] Tom White. 2012. Hadoop: The definitive guide. O’ReillyMedia, Inc.

[38] Robert P. Wilson, Robert S. French, Christopher S. Wilson,Saman P. Amarasinghe, Jennifer M. Anderson, Steve W. K.Tjiang, Shih-Wei Liao, Chau-Wen Tseng, Mary W. Hall,Monica S. Lam, and John L. Hennessy. 1994. SUIF: AnInfrastructure for Research on Parallelizing and OptimizingCompilers. SIGPLAN Not. 29, 12 (1994).

[39] John W. Young. 1974. A First Order Approximation to theOptimum Checkpoint Interval. Commun. ACM 17, 9 (Sept.1974), 530–531.

[40] Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin,Scott Shenker, and Ion Stoica. 2010. Spark: Cluster Com-puting with Working Sets. (2010).

[41] Bingjing Zhang, Yang Ruan, and Judy Qiu. 2015. Harp:Collective communication on hadoop (IC2E).

13 2017/4/12