r e p r o d u c i b l e ? a r e my r e s u l ts...dat acamp p aral l el p rogrammi ng i n r l'...
TRANSCRIPT
![Page 1: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/1.jpg)
DataCamp Parallel Programming in R
Are my resultsreproducible?
PARALLEL PROGRAMMING IN R
Hana SevcikovaUniversity of Washington
![Page 2: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/2.jpg)
DataCamp Parallel Programming in R
Random numbers in R
Many statistical applications involve random numbers (RNs)
Examples: MCMCs in Bayesian methods, bootstrap, simulations
For reproducibility:
Set seed of a random number generator (RNG) prior to running the codeset.seed(1234)
rnorm(3)
[1] -1.2070657 0.2774292 1.0844412
rnorm(3)
[1] -2.3456977 0.4291247 0.5060559
set.seed(1234)
rnorm(3)
[1] -1.2070657 0.2774292 1.0844412
rnorm(3)
[1] -2.3456977 0.4291247 0.5060559
![Page 3: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/3.jpg)
DataCamp Parallel Programming in R
Naive (non)reproducibility in parallel codelibrary(parallel)
cl <- makeCluster(2)
set.seed(1234)
clusterApply(cl, rep(3, 2), rnorm)
[[1]]
[1] -1.891091 -1.351767 -1.456848
[[2]]
[1] 1.7346577 0.7855641 -2.2319774
set.seed(1234)
clusterApply(cl, rep(3, 2), rnorm)
[[1]]
[1] 0.4432499 -0.7896067 0.2659675
[[2]]
[1] 0.2229560 0.8323269 -0.4092570
![Page 4: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/4.jpg)
DataCamp Parallel Programming in R
Incorrect way of generating RNs in parallel code
Using set.seed(), the RNG is initialized only on the master.
Workers start with a clean environment, thus no RNG seed set.
What happens when we set the RNG on each worker?
clusterEvalQ(cl, set.seed(1234))
clusterApply(cl, rep(3, 2), rnorm)
[[1]]
[1] -1.2070657 0.2774292 1.0844412
[[2]]
[1] -1.2070657 0.2774292 1.0844412
![Page 5: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/5.jpg)
DataCamp Parallel Programming in R
Another incorrect way of generating RNs in parallel code
Quick and dirty solution:
NOT RECOMMENDED!!!
for (i in 1:2) {
set.seed(1234)
clusterApply(cl, sample(1:10000000, 2), set.seed)
print(clusterApply(cl, rep(3, 2), rnorm))
}
[[1]]
[1] 0.078249533 0.003019703 -1.314239709
[[2]]
[1] 1.3955357 -0.9935141 -0.3740712
[[1]]
[1] 0.078249533 0.003019703 -1.314239709
[[2]]
[1] 1.3955357 -0.9935141 -0.3740712
![Page 6: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/6.jpg)
DataCamp Parallel Programming in R
Let's practice!
PARALLEL PROGRAMMING IN R
![Page 7: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/7.jpg)
DataCamp Parallel Programming in R
Parallel random numbergenerators
PARALLEL PROGRAMMING IN R
Hana SevcikovaUniversity of Washington
![Page 8: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/8.jpg)
DataCamp Parallel Programming in R
Random Number Generators (RNGs)
Important parameters of an RNG:
long period (preferably > 2 )
good structural (distributional) properties in high dimensions
These parameters should hold when used in distributed environment
100
![Page 9: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/9.jpg)
DataCamp Parallel Programming in R
L'Ecuyer Multiple Streams RNG
A good quality RNG with multiple independent streams proposed by Pierre
L'Ecuyer et al. (2002), RngStreams
Period 2
Streams have seeds 2 steps apart
Parallel parts of user computation can use independent and reproducible
streams
Direct interface in R: rlecuyer, rstream
In R core: RNGkind("L'Ecuyer-CMRG")
191
127
![Page 10: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/10.jpg)
DataCamp Parallel Programming in R
Using L'Ecuyer RNG in parallel
Setting an RNG seed for cluster cl:
Initializes a reproducible independent stream on each worker
clusterSetRNGStream(cl, iseed = 1234)
![Page 11: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/11.jpg)
DataCamp Parallel Programming in R
Reproducibility in the parallel package
In parallel: one stream per worker
Creates constraints on reproducibility
Results only reproducible if:
1. process runs on clusters of the same size
2. process does not use load balancing, e.g. clusterApplyLB()
![Page 12: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/12.jpg)
DataCamp Parallel Programming in R
Let's practice!
PARALLEL PROGRAMMING IN R
![Page 13: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/13.jpg)
DataCamp Parallel Programming in R
Reproducibility in foreachand future.apply
PARALLEL PROGRAMMING IN R
Hana SevcikovaUniversity of Washington
![Page 14: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/14.jpg)
DataCamp Parallel Programming in R
doRNG: backend for foreach
![Page 15: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/15.jpg)
DataCamp Parallel Programming in R
Using doRNG via %dorng%library(doRNG)
library(doParallel)
registerDoParallel(cores = 3)
set.seed(1)
res1 <- foreach(n = rep(2, 5), .combine = rbind) %dorng% rnorm(n)
set.seed(1)
res2 <- foreach(n = rep(2, 5), .combine = rbind) %dorng% rnorm(n)
identical(res1, res2)
[1] TRUE
![Page 16: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/16.jpg)
DataCamp Parallel Programming in R
Using doRNG via %dopar%
Note: Cannot be used with the %doSEQ% backend.
library(doRNG)
library(doParallel)
registerDoParallel(cores = 3)
registerDoRNG(1)
res3 <- foreach(n = rep(2, 5), .combine = rbind) %dopar% rnorm(n)
set.seed(1)
res4 <- foreach(n = rep(2, 5), .combine = rbind) %dopar% rnorm(n)
c(identical(res1, res3), identical(res2, res4))
[1] TRUE TRUE
![Page 17: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/17.jpg)
DataCamp Parallel Programming in R
Summary of using doRNG
Two ways of including doRNG into foreach:
1. Using %dorng%:
advantage of being explicit about using the L’Ecuyer’s RNG
2. Using %dopar% and registering doRNG:
easy to make code/packages reproducible by only prepending
registerDoRNG()
doRNG can be used with any parallel backend, including doFuture.
![Page 18: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/18.jpg)
DataCamp Parallel Programming in R
future.apply
Uses independent streams of the L’Ecuyer’s RNG
As in doRNG, generates one stream per task
Need only to assign future.seed argumentlibrary(future.apply)
plan(sequential)
res5 <- future_lapply(1:5, FUN = rnorm, future.seed = 1234)
plan(multiprocess)
res6 <- future_lapply(1:5, FUN = rnorm, future.seed = 1234)
identical(res5, res6)
[1] TRUE
![Page 19: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/19.jpg)
DataCamp Parallel Programming in R
Let's practice!
PARALLEL PROGRAMMING IN R
![Page 20: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/20.jpg)
DataCamp Parallel Programming in R
Finishing Touch
PARALLEL PROGRAMMING IN R
Hana SevcikovaSenior Research Scientist, University of Washington
![Page 21: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/21.jpg)
DataCamp Parallel Programming in R
Recommended R packages
parallel (core package)
No need for dependencies on other packages
Important to understand as other packages are built on it
Often yields best performance
Reproducible results: only on clusters of the same size with no load balancing
![Page 22: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/22.jpg)
DataCamp Parallel Programming in R
Recommended R packages (cont.)
foreach (with doParallel, doFuture)
Higher level programming
Intuitive syntax in form of for loops
Results reproducible via doRNG
future.apply (based on future)
Unifies many parallel backends into one interface
Intuitive apply()-like syntax
Results always reproducible
![Page 23: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/23.jpg)
DataCamp Parallel Programming in R
Getting the best performance
Minimize amount of communication (sending repeatedly big data is bad!)
Use scheduling and load balancing appropriate for your application (e.g. group
tasks into chunks evenly distributed across workers)
Use cluster size appropriate for your hardware (i.e. number of physical cores)
![Page 24: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/24.jpg)
DataCamp Parallel Programming in R
![Page 25: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/25.jpg)
DataCamp Parallel Programming in R
![Page 26: r e p r o d u c i b l e ? A r e my r e s u l ts...Dat aCamp P aral l el P rogrammi ng i n R L' E cuyer Mul t i pl e S t reams RNG A good quality RNG with multiple independent streams](https://reader034.vdocument.in/reader034/viewer/2022050508/5f990a55f3597b384927345b/html5/thumbnails/26.jpg)
DataCamp Parallel Programming in R
Final Slide
PARALLEL PROGRAMMING IN R