![Page 1: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/1.jpg)
Fat-trees and Dragonflies – A perspective
on topologies
Sven-Arne Reinemo
Simula Research Laboratory
HPC Advisory Council Switzerland Workshop
March 15, 2012
![Page 2: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/2.jpg)
o Frank Olaf Sem-Jacobsen
o Mathias Myrland
o Professor Tor Skeie
o Eitan Zahavi, Mellanox
o Gilad Shainer, Mellanox
![Page 3: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/3.jpg)
o Motivation
o The fat-tree topology
o The dragonfly topology
o Group topologies for the dragonfly
o Blocking, cost, and scalability
o Conclusion
![Page 4: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/4.jpg)
o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested as an alternative.
o In that context we would like to answer the following questions:
o Is the dragonfly a viable alternative to the fat-tree?
o How do they compare in fundamental properties such as blocking and scalability?
o What about cost?
o What dragonfly configurations are attractive?
![Page 5: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/5.jpg)
oMost common topology for InfiniBand clusters.
oCan be routed deadlock free without additional
resources such as virtual lanes.
oFault-tolerant through its path diversity.
oFull bisection bandwidth for arbitrary
permutations.
oScalable, but costly when going beyond to
levels.
oPerformance suffer slightly due to static
routing.
![Page 6: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/6.jpg)
m-port n-tree k-ary n-tree hybrid tree
![Page 7: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/7.jpg)
o Recently proposed by John Kim et al. in [1].
o Caused discussion in the HPC community about is suitability for IB and as an exascale topology.
o The dragonfly is a hierarchical topology with the following properties:
o Several groups are connected together using all to all links, i.e. each group has at least one link directly to each other group.
o The topology inside each group can be any topology. The recommendation in [1] is the flattened butterfly.
o Focus on reducing the number of long links and network diameter.
o Requires non-minimal global adaptive routing and advanced congestion look ahead for efficient operation.
[1] John Kim et al. “Technology-Driven, Highly-Scalable Dragonfly Topology” in proceedings of the
35th
International Symposium on Computer Architecture, 2008.
![Page 8: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/8.jpg)
The recommendation in [1] is to keep:
Inter group
topology
Group0
Groupg-1
Group1
Switch
0
Switch
1
Switcha-1
0 …. p-1
0 …. h-1
a * h links to other groups
Intra group
topology
a ³ 2p ³ 2h
![Page 9: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/9.jpg)
Dragonfly conforming to versus an unrestricted
dragonfly. a ³ 2p ³ 2h
![Page 10: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/10.jpg)
In our studies we will:
oconsider the flattened butterfly as a group
topology.
oconsider the 2d torus as a group
topology.
oconsider a fat-tree as a group topology.
orelax the requirement to
compare it on equal terms with the fat-
tree.
a ³ 2p ³ 2h
![Page 11: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/11.jpg)
A single flattened butterfly group with 8 terminals and 8 external
connections.
Fully connected, but requires non-minimal routing for path
diversity and load balancing.
S0 S1 S2 S3
![Page 12: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/12.jpg)
Statistics for 100 random permutations on a 25 760 ports
dragonfly with flattened butterfly groups, buildt using 36-port
switches.
![Page 13: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/13.jpg)
Good for nearest neighbour traffic, but bisection bandwidth does
not scale for arbitrary traffic.
![Page 14: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/14.jpg)
Statistics for 100 random permutations on a 25 760 ports
dragonfly with 2d torus groups, built using 36-port switches.
![Page 15: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/15.jpg)
Rather than building a complete fat-tree we use the fat-tree as
the group topology in the dragonfly.
![Page 16: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/16.jpg)
oAs shown, the viable dragonfly groups are
the a flattened butterfly and the fat-tree.
oWe will compare these dragonfly
configurations with a 3- and 4-level fat-tree.
oWe look at how the blocking and cost of
these topologies develop as the size
increases.
![Page 17: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/17.jpg)
Comparing the dragonflies with a 3- and 4-tier fat-tree.
![Page 18: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/18.jpg)
Scaling of the 4-tier fat-tree and two hybrid dragonflies.
![Page 19: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/19.jpg)
Key results:
o The topology with best cost/performance depends on the size.
o The fat-tree gives the best cost/worst-case-blocking
performance.
o The dragonfly is almost as good as the fat-tree when
considering cost/average-blocking performance.
Remember:
o The dragonfly require support for adaptive routing and
congestion look ahead for optimal behaviour, this is not
supported by any existing hardware.
o The dragonfly requires multiple virtual channels for deadlock
avoidance.
![Page 20: Fat-trees and Dragonflies A perspective on topologies · o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested](https://reader033.vdocument.in/reader033/viewer/2022060517/604997df04f7f3261d09f20c/html5/thumbnails/20.jpg)