trinity: a distributed graph engine on a memory cloud
DESCRIPTION
TRANSCRIPT
![Page 1: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/1.jpg)
Trinity: A Distributed Graph Engine on a Memory Cloud
Speaker: LIN Qianhttp://www.comp.nus.edu.sg/~linqian/
![Page 2: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/2.jpg)
Graph applications
Online query processing Low latencyOffline graph analytics High throughput
![Page 3: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/3.jpg)
Online queries
Random data accesse.g., BFS, sub-graph matching, …
![Page 4: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/4.jpg)
Offline computations
Performed iteratively
![Page 5: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/5.jpg)
Insight: Keeping the graph in memory
at least the topology
![Page 6: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/6.jpg)
Trinity
Online query + Offline analytics
![Page 7: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/7.jpg)
Random data access problem in large graph
computationGlobally addressable distr. memory
Random access abstraction
![Page 8: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/8.jpg)
Belief
High-speed network is more availableDRAM is cheaper
In-memory solution become practical
![Page 9: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/9.jpg)
“Trinity itself is not a system that comes with comprehensive built-in graph computation modules.”
![Page 10: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/10.jpg)
Trinity cluster
![Page 11: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/11.jpg)
Stack of Trinity system modules
User define: Graph schema, Communication protocols, Computation paradigms
![Page 12: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/12.jpg)
Memory cloud
Partition memory space into trunksHashing
![Page 13: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/13.jpg)
Memory trunks
2p > m1. Trunk level parallelism
2. Efficient hashing
![Page 14: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/14.jpg)
Hashing
Key-value storep-bit value i [0, 2∈ p – 1]
Inner trunk hash table
![Page 15: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/15.jpg)
Data partitioning and addressing
Benefit:Scalability Fault-tolerance
![Page 16: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/16.jpg)
Modeling graph
Cell: value + schemaRepresent a node in a cell
![Page 17: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/17.jpg)
TSL
Object-oriented cell manipulationData integration
Network communication
![Page 18: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/18.jpg)
Online queries
Traversal basedNew paradigm
![Page 19: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/19.jpg)
Vertex centric offline analytics
Restrictive vertex centric model
![Page 20: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/20.jpg)
Message passing optimization
Create a bipartite partition of the local graph
Buffer hub vertices
![Page 21: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/21.jpg)
A new paradigm for offline analytics
1. Aggregate answers from local computations
2. Employ probabilistic inference
![Page 22: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/22.jpg)
Circular memory management
• Aim to avoid memory gaps between large number of key-value pairs
![Page 23: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/23.jpg)
Fault tolerance
Heartbeat-based failure detectionBSP: checkpointing
Async.: “periodical interruption”
![Page 24: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/24.jpg)
Performance
![Page 25: Trinity: A Distributed Graph Engine on a Memory Cloud](https://reader035.vdocument.in/reader035/viewer/2022062613/540eae6f8d7f72767e8b4f20/html5/thumbnails/25.jpg)
Performance (cont.)