introduction to parallel computingkarypis/parbook...all-reduce & prefix-sum scatter and gather...
TRANSCRIPT
![Page 1: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/1.jpg)
Introduction to Parallel Computing
George KarypisBasic Communication Operations
![Page 2: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/2.jpg)
OutlineImportance of Collective Communication OperationsOne-to-All BroadcastAll-to-One ReductionAll-to-All Broadcast & ReductionAll-Reduce & Prefix-SumScatter and GatherAll-to-All Personalized
![Page 3: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/3.jpg)
Collective Communication Operations
They represent regular communication patterns that are performed by parallel algorithms.
Collective: Involve groups of processorsUsed extensively in most data-parallel algorithms.The parallel efficiency of these algorithms depends on efficient implementation of these operations.They are equally applicable to distributed and shared address space architecturesMost parallel libraries provide functions to perform themThey are extremely useful for “getting started” in parallel processing!
![Page 4: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/4.jpg)
MPI Names
![Page 5: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/5.jpg)
One-to-All Broadcast &All-to-One Reduction
![Page 6: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/6.jpg)
Broadcast on a Ring Algorithm
![Page 7: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/7.jpg)
Reduction on a Ring Algorithm
![Page 8: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/8.jpg)
Broadcast on a Mesh
![Page 9: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/9.jpg)
Broadcast on a Hypercube
![Page 10: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/10.jpg)
Code for the BroadcastSource: Root
![Page 11: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/11.jpg)
Code for BroadcastArbitrary Source
![Page 12: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/12.jpg)
All-to-All Broadcast & Reduction
![Page 13: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/13.jpg)
All-to-All Broadcast for Ring
![Page 14: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/14.jpg)
All-to-All Broadcast on a Mesh
![Page 15: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/15.jpg)
All-to-All Broadcast on a HCube
![Page 16: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/16.jpg)
All-Reduce & Prefix-Sum
![Page 17: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/17.jpg)
Scatter & Gather
![Page 18: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/18.jpg)
Scatter Operation on HCube
![Page 19: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/19.jpg)
All-to-All Personalized (Transpose)
![Page 20: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/20.jpg)
All-to-all Personalized on a Ring
![Page 21: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/21.jpg)
All-to-all Personalized on a Mesh
![Page 22: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/22.jpg)
All-to-all Personalized on a HCube
![Page 23: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/23.jpg)
All-to-all Personalized on a HCubeImproved Algorithm
Perform log(p) point-to-pointcommunication steps
Processor i communicates with processor iXORj duringthe jth communication step.
![Page 24: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/24.jpg)
Complexities
![Page 25: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/25.jpg)
![Page 26: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/26.jpg)
![Page 27: Introduction to Parallel Computingkarypis/parbook...All-Reduce & Prefix-Sum Scatter and Gather All-to-All Personalized. Collective Communication Operations](https://reader035.vdocument.in/reader035/viewer/2022062507/5fe07e24b732713fd12350f5/html5/thumbnails/27.jpg)