cloudclustering: toward an iterative data processing pattern on the cloud
TRANSCRIPT
![Page 1: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/1.jpg)
CloudClustering
Ankur Dave*, Wei Lu†, Jared Jackson†, Roger Barga†
*UC Berkeley†Microsoft Research
Toward an Iterative Data Processing Pattern on the Cloud
![Page 2: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/2.jpg)
Wei Lu Jared Jackson Roger BargaAnkur Dave
![Page 3: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/3.jpg)
Background
MapReduce and its successors reimplement communication and storage
But cloud services like EC2 and Azure natively provide these utilities
Can we build an efficient distributed application using only the primitives of the cloud?
![Page 4: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/4.jpg)
Design Goals
• Efficient • Fault tolerant• Cloud-native
![Page 5: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/5.jpg)
Outline
Windows AzureK-Means clustering algorithmCloudClustering architecture– Data locality– Buddy system
EvaluationRelated work
![Page 6: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/6.jpg)
Windows AzureVM instances
Blob storage
Queue storage
![Page 7: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/7.jpg)
Outline
Windows AzureK-Means clustering algorithmCloudClustering architecture– Data locality– Buddy system
EvaluationRelated work
![Page 8: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/8.jpg)
K-Means algorithm
Iteratively groups points into clusters
Centroids
Initialize
Assign points to centroids
Recalculate centroids
Points moved?
![Page 9: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/9.jpg)
K-Means algorithmInitialize
Partition work
Recalculate centroids
Points moved?
Assign points to centroids
Compute partial sums
![Page 10: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/10.jpg)
Outline
Windows AzureK-Means clustering algorithmCloudClustering architecture– Data locality– Buddy system
EvaluationRelated work
![Page 11: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/11.jpg)
Architecture
⋮
Centroids:
Initialize
Partition work
Recalculate centroids
Points moved?
Assign points to centroids
Compute partial sums
![Page 12: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/12.jpg)
Outline
Windows AzureK-Means clustering algorithmCloudClustering architecture– Data locality– Buddy system
EvaluationRelated work
![Page 13: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/13.jpg)
Data locality
Multiple-queue pattern unlocks data localityTradeoff: Complicates fault tolerance
⋮
Single-queue pattern is naturally fault-tolerantProblem: Not suitable for iterative computation
![Page 14: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/14.jpg)
Outline
Windows AzureK-Means clustering algorithmCloudClustering architecture– Data locality– Buddy system
EvaluationRelated work
![Page 15: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/15.jpg)
Handling failureInitialize
Partition work
Recalculate centroids
Points moved?
Assign points to centroids
Compute partial sums
Stall? Buddy system
![Page 16: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/16.jpg)
Buddy system
Buddy system provides distributed fault detection and recovery
![Page 17: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/17.jpg)
Buddy system
Spreading buddies across Azure fault domains provides increased resilience to
simultaneous failure
1
2
3
Fault domain
![Page 18: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/18.jpg)
Buddy system
Cascaded failure detection reduces communication and improves resilience
…
vs.
![Page 19: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/19.jpg)
Outline
Windows AzureK-Means clustering algorithmCloudClustering architecture– Data locality– Buddy system
EvaluationRelated work
![Page 20: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/20.jpg)
Evaluation
Linear speedup with instance count
![Page 21: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/21.jpg)
Evaluation
Sublinear speedup with instance sizeReason: I/O bandwidth doesn’t scale
![Page 22: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/22.jpg)
Outline
Windows AzureK-Means clustering algorithmCloudClustering architecture– Data locality– Buddy system
EvaluationRelated work
![Page 23: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/23.jpg)
Related work
• Existing frameworks (MapReduce, Dryad, Spark, MPI, …) all support K-Means, but reimplement reliable communication
• AzureBlast built directly on cloud services, but algorithm is not iterative
![Page 24: CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud](https://reader035.vdocument.in/reader035/viewer/2022062707/5583f3dcd8b42afa438b4cd2/html5/thumbnails/24.jpg)
Conclusions
• CloudClustering shows that it's possible to build efficient, resilient applications using only the common cloud services
• Multiple-queue pattern unlocks data locality• Buddy system provides fault tolerance