Introduction to Parallel Processing with Python

1

2 Source - https://computing.llnl.gov/tutorials/parallel_comp/

What is Parallel Computing?

Serial Computing

What is Parallel Computing?

3

Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through multiple processors

Parallelized Hardware

4

Nearly all processors now have parallelized processing

architectures

Eight-core CPUs on now selling for mainstream consumers

Intel® Core™ i7-5960X: $1000 (2014)

AMD Ryzen 2700X: $300 (2018)

HPCs – Built for Parallelization

5

• HPCs employ often 2-4 server-grade CPUs per node• 8 – 16 processor cores per CPU• Shared memory on each node for all processors

• Distributed memory architecture• Nodes are connected via a 56-100 Gbps network• Memory is shared between nodes through some API

• MPI is most commonly used

Global Interpreter Lock

6

Global Interpreter Lock

7

• The Python interpreter is not fully thread-safe.

• In order to support multi-threaded Python programs, there’s a global lock, called the global interpreter lock or GIL, that must be held by the current thread before it can safely access Python objects.

• Without the lock, even the simplest operations could cause problems in a multi-threaded program

• For example, when two threads simultaneously increment the reference count of the same object, the reference count could end up being incremented only once instead of twice.

• Therefore, only one thread is run at a time.

So how can one effectively parallelize their code?

8

Enter: multiprocessing

Time to switch over to Jupyter Notebook

Installing a Conda Environment for Keras and TensorFlow with Jupyter Support

9

$ module load python/3.6.1-2-anaconda

$ conda create --name py3.6-multiprocess -–clone root$ source activate py3.6-multiprocess$ conda install –c conda-forge multiprocess

$ ipython kernel install --user --name py3.6-multiprocess --display-name=“Custom"