foreach iterators - lewis

7/28/2019 Foreach Iterators - Lewis

1/25

foreach + iterators

Bryan Lewis Steve Weston

Revolution ComputingNew Haven, CT USA

Rmetrics 2009
http://find/


2/25

Outline

iterators

foreach

Experimenting with existing packages
http://find/


3/25

iterators

An S3 class with tools for iterating over various R data structures:

Conceptually like while loops

Defined by a nextElem function

Like iterators in Java and other languages
http://find/


4/25

Simple Examples

it


5/25

Another example

iquery


6/25

foreach

New looping methods for R

An abstract interface to parallel computing Python/Haskell-like list comprehensions
http://find/http://goback/


7/25

Foreach Syntax

foreach(iterator,...) %dopar% {statements

}
http://find/


8/25

Example

> foreach (j=1:4) %dopar% { j }

[[1]]

[1] 1

[[2]][1] 2

[[3]]

[1] 3

[[4]]

[1] 4
http://find/


9/25

Examples

> foreach (j=1:4,.combine=c) %dopar% { j }

[ 1 ] 1 2 3 4

> foreach (j=icount(4),.combine=+) %dopar% { j }

[1] 10

Note the difference withsum (1:4).
http://find/


10/25

Another Example

> library (randomForest)

> x y rf


11/25

The %dopar% operator

%dopar% is a registration API for parallel back-ends:

doSEQ (the default backend)

doMC (multicore package)

doNWS doSNOW
http://find/


12/25

The %dopar% operator

%dopar% is a registration API for parallel back-ends:

doSEQ (the default backend)

doMC (multicore package)

doNWS doSNOW

doRHIPE?

doRMPI?

. . .


13/25

Foreach tries to parse R syntax reasonably

> z < - 2

> f foreach (j=1:4, .combine=c) %dopar% { f (j) }

[1] 1.732051 2.000000 2.236068 2.449490
http://find/


14/25

List comprehension

> foreach (j=-2:2,.combine=c) %:% when (j>=0)

+ %dopar% sqrt (j)

[1] 0.000000 1.000000 1.414214


15/25

Nesting

Foreach loops can be nested. Nesting admits at least two

interesting cases:

Easy loop unrolling

Easy multi-paradigm parallelism

L lli
http://find/


16/25

Loop unrolling

Compare (100 iterations of 5 parallel tasks):x


17/25

Multi-paradigm parallelism

> require (doSNOW)

> cl registerDoSNOW (cl)

> foreach (j=, .packages=doMC) %dopar% {+ foreach (k=) %dopar% {

+ registerDoMC ()

+ ...

+ }

+ }

E l V i l b kt ti


18/25

Example: Very simple backtesting

simpleRule


19/25

Brute-force parameter optimization

# Define a return series Ra for the instrument# (below we use the closing price of MSFT), and

# benchmark series Rb

M


20/25

Now in parallel, by rows...

M


21/25

Parallelizing parts of an existing package

Basic idea

Profile code with Rprof (profr is a nice wrapper that visualizes

the results)

Examine bottlenecks for apply-like statements and for loops

with independent code blocks

Rewrite for loops without side-effects as required (may require

a custom combine function)

Unlock the namespace, provisionally replace target

function(s) and experiment (a nice trick)

Example: ipred
http://find/


22/25

Example: ipred

(Work through the ipred replacement functions in the filecvx.R.)

Appendix: Fun map/reduce examples
http://find/


23/25

Appendix: Fun map/reduce examples

Succint map/reduce...from the mapReduce package by

Christopher Brown:

mapReduce


24/25

mapReduce sequential and parallel examples

# An example

mapReduce (cyl, mean(mpg), mean(hp),data=mtcars, applyfun=sapply)

# With multicore:

require (mutlicore)

mapReduce (cyl, mean(mpg), mean(hp),

data=mtcars, applyfun=mclapply)

# With SNOW parSapply:

require (snow)

cl


25/25

mapReduce parallel examples

# With Rmpi mpi.parSapply:

require (Rmpi)

x

foreach iterators - lewis

Documents