cache management for in memory analytics - eie | …jeiezhang/document/talk_cache.pdfoutline 2 1....
TRANSCRIPT
![Page 1: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/1.jpg)
Cache Management for In‐Memory Analytics
1
Jun ZHANGOct 15, 2018
![Page 2: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/2.jpg)
Outline
2
1. Introduction
2. LRC: Dependency‐aware caching
3. OpuS: Fair cache sharing in multi‐tenant cloud
4. SP‐Cache: Load balancing for cluster caching
5. Summary
![Page 3: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/3.jpg)
My Research Interests
Wireless CommunicationsMachine learning for wireless communications Cellular‐enabled UAV communications
Edge IntelligenceMobile edge computing Edge learning
Distributed Computing Systems Big data analytics systems Distributed learning
Recruiting (@PolyU) 2~3 PhD, 1 Postdoc
3
![Page 4: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/4.jpg)
Big Data
4
![Page 5: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/5.jpg)
Big Data
5
![Page 6: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/6.jpg)
6
![Page 7: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/7.jpg)
Big Data Analytics
7
• Cluster‐Computing Frameworks• (December 2011) [1]
• (May 2014) [2]In‐memory cluster computing
[1] J. Dean and S. Ghemawat. “MapReduce: Simplified data processing on large clusters.” In Proc. The 6th Symposium on OperatingSystems Design and Implementation (OSDI), pp.137–150, Dec. 2004.[2] M. Zaharia, M. Chowdhury, et al. “Resilient distributed datasets: A fault‐tolerant abstraction for in‐memory cluster computing.” InNSDI, 2012.
![Page 8: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/8.jpg)
Memory Speeds up Computation
Caching input data speeds up tasks by orders of magnitude. [M. Zaharia, 2012]
127 sec / iteration
6 sec / iteration
8
![Page 9: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/9.jpg)
In‐Memory Data Analytics
9
![Page 10: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/10.jpg)
Cache Management
Cache is limitedWhat to cache? [INFOCOM 17]
Cache is shared resource How to fairly share cache among users? [ICDCS 18]
Distributed caches Load imbalance How to balance the load? [SC 18]
10
![Page 11: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/11.jpg)
1. LRCDependency‐aware Caching Management
11
[3] Y. Yu, W. Wang, J. Zhang, and K. B. Letaief, "LRC: Dependency‐aware cache management in data analytics clusters," in Proc. IEEE INFOCOM 2017, Atlanta, GA, May 2017.
![Page 12: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/12.jpg)
Cache Management
12
Crucial for in‐memory data analytics systems. Well studied in many systems
CDN (Akamai)
Objective: optimize the cache hit ratio Maximize the chance of in‐memory data access.
AS1 AS2 AS3
AS4 AS5
AS6 AS7 AS8 AS9
BrowserCache
EdgeCache
OriginCache BackendUser
![Page 13: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/13.jpg)
Existing Solutions
13
Least Recently Used (LRU) policy [R. L. Mattson, 1970]
Evicts the data block that has not been used for the longest period. Widely employed in prevalent systems, e.g., Spark, Tez and Alluxio.
Least Frequently Used (LFU) policy [M. Stonebraker, 1971]
Evicts the data block that has been used the least times.
Summary: “guessing” the future data access patterns basedon historical information (access recency or frequency).
![Page 14: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/14.jpg)
Data Dependency Reveals Access Patterns
14
Application Semantics Data dependency structured as a Directed Acyclic Graph (DAG)
Available to the cluster scheduler before the job starts
Data access follows the dependency DAG. The future is not totally unpredictable.
![Page 15: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/15.jpg)
Data Dependency Reveals All‐or‐Nothing Property
15
All‐or‐Nothing: a computing task can only be sped up when its dependent data blocks are all cached in memory. E.g. To compute a block in B, all blocks of A are required. Cache hits of
only part of the three blocks makes no difference.
Cache hit ratio is not appropriate metric for cache performance.
![Page 16: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/16.jpg)
Inefficiency of Existing Cache Polices
16
• Oblivious to the data access pattern: – Inactive data (no future access) cannot be evicted timely.– In our measurement studies, inactive data accounts for >77% of the
cache space for >50% of time.
• Oblivious to the all‐or‐nothing property:– Achieving a high cache hit ratio does not necessarily speed up the
computation.
• Challenge: How to exploit the data dependency information (DAGs) to clear the inactive data efficiently and factor in the all‐or‐nothing property?
![Page 17: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/17.jpg)
LRC: Dependency‐Aware Cache Management
17
Reference count: defined for each data block as the number of downstream tasks depending on it. Dynamically changing over time:
Least Reference Count (LRC) policy: when the cache is full, always evict the data with the least reference count. Inactive data (w/ zero reference count) is evicted first, e.g., block B.
1
1
1
2 0
![Page 18: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/18.jpg)
Effective Cache Hit Ratio
18
Factor in the all‐or‐nothing property?
Effective cache hit ratio: A cache hit is effective when it speeds up the computation, i.e., when all the depended blocks are cached.
![Page 19: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/19.jpg)
Tailor LRC to Optimize Effective Cache Hit Ratio
19
A reference to a data block is only “counted” when it effectively speeds up the computationE.g., the reference to block D for computation of block F is not counted if block C is evicted from the cache.
1
1
2 1
![Page 20: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/20.jpg)
Spark Implementation
20
![Page 21: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/21.jpg)
Evaluations: Workload Characterization
21
• Cluster setup: 20‐node Amazon EC2 cluster. • Instance type: m4.large. Dual‐core 2.4 GHz Intel Xeon® E5‐2676 v3
(Haswell) processors and 8 GB memory.• Workloads: Typical applications from SparkBench [4].
[4] M. Li, J. Tan, Y. Wang, L. Zhang, and V. Salapura, “Sparkbench: a comprehensive benchmarking suite for in memory data analytic platform spark,” in Proc. 12th ACM International Conf. on Comput. Frontiers, 2015.
Not all applications benefit from the improvement of cache management.
![Page 22: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/22.jpg)
Evaluations: Effective Cache Hit Ratio
22
![Page 23: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/23.jpg)
Evaluations: Application Runtime
23
![Page 24: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/24.jpg)
Evaluations: Summary
24
• LRC speeds up typical workloads by up to 60%.
![Page 25: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/25.jpg)
Conclusions
25
• It is effective to leverage the dependency DAGs to optimize the cache management
• Effective cache hit ratio is a better cache performance metric– To account for the all‐or‐nothing property
• LRC – a dependency‐aware cache management policy– Optimizes the effective cache hit ratio– Speeds up typical workloads by up to 60%
![Page 26: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/26.jpg)
2. FairnessOpuS: Fair and Efficient Cache Sharing
26
[5] Y. Yu, W. Wang, J. Zhang, Q. Weng, and K. B. Letaief, “OpuS: Fair and efficient cache sharing for in‐memory data analytics,” in Proc. IEEE Int. Conf. Distrib. Comput. Syst. (ICDCS), July 2018.
![Page 27: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/27.jpg)
Cache Sharing
Multiple users/applicationsand limited cache resources.
Objective: fairness and efficiency (cache utilization).
27
![Page 28: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/28.jpg)
What we want
Fair: Isolation guarantee: no worse than isolation. Strategy‐proofness: cannot cheat to improve.
Efficient: Pareto efficiency: impossible to improve a user without hurting others.
28
![Page 29: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/29.jpg)
Isolated Cache?
Fair: Equal and dedicated.
Inefficient: Duplication. Users might not be able to fully utilize theirquota.
29
![Page 30: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/30.jpg)
Global Sharing?
Efficient: No duplication; no waste.
Unfair: Low‐demand users get arbitrarily small allocations Prone to strategic behavior.
E.g., LRU, LFU
30
![Page 31: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/31.jpg)
Challenge: Free‐riding
Non‐exclusively shared. No user caches files of common interest, in the hope of free‐
riding.
Solutions for others resources, e.g., CPU and network, do not work.
31
![Page 32: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/32.jpg)
Max‐min Fair Allocation
Maximize the minimum cache allocation across all users.
Suffer from free‐riding strategies:
3 hit/sec
2 hit/sec
2.5 hit/sec
2.5 hit/sec
Total budget: 2 unit 32
![Page 33: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/33.jpg)
FairRide: State‐of‐the‐art [6]
Built atop max‐min fairness; block free‐riders with a chosen probability.
Users can still cheat to improve by other forms of cheating.
FairRide
33[6] Q. Pu, H. Li, M. Zaharia, A. Ghodsi, and I. Stoica, “FairRide: nearoptimal, fair cache sharing,” in Proc. USENIX NSDI, 2016.
![Page 34: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/34.jpg)
Summary of Existing Solutions
Isolation guarantee
Strategy‐proofness
Pareto‐efficiency
IsolationGlobalMax‐minFairRide Near max‐min
34
It was proved in [6] that no allocation can achieve all the three properties due to free‐riding.
![Page 35: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/35.jpg)
OpuS
OpuS: Two‐stage Opportunistic Sharing
1. Stage I: share the cache. High efficiencyStrategy‐proof
2. Isolation not guaranteed?
3. Stage II: isolation.and high probability of IG.
35
![Page 36: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/36.jpg)
OpuS
Stage I: cache sharing High efficiency and high chance of isolation guarantee Proportional fairness
Cache hit ratio
36
![Page 37: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/37.jpg)
OpuS
Stage I: cache sharing Strategy‐proof
Charge each user a “tax”: the loss of others due to its existence. Free‐riding is not “free”!
Discourage any kind of cheating: the utility increment is always less than the extra tax paid.
VCG mechanism
Others’ utility without user i’s existence
37
![Page 38: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/38.jpg)
OpuS
Stage I: cache sharing The VCG tax is charged via blocking:
Stage II: Isolation Reduce to isolation if IG is violated.
Non‐blocking prob.
38
![Page 39: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/39.jpg)
Our Contribution
Isolation guarantee
Strategy‐proofness
Pareto‐efficiency
IsolationGlobalMax‐minFairRide Near max‐minOpuS Near optimal
o Isolation guarantee.o Immune to any form of cheating.o Near‐optimal efficiency.
39
![Page 40: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/40.jpg)
Implementation
40
![Page 41: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/41.jpg)
Evaluations
Strategy‐proofness
Cheating only leads to worse performance41
![Page 42: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/42.jpg)
Evaluations
Near‐optimal efficiency
42
![Page 43: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/43.jpg)
Wrap up – Fair Cache Sharing
Cache sharing uniquely faces the free‐riding challenge.
OpuS: a cache allocation algorithm for shared clouds. Dis‐incentivize all cheating strategies, including free‐ridingNear‐optimal efficiency
43
![Page 44: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/44.jpg)
3. Load BalancingSP‐Cache: Load‐balanced, Redundancy‐free Cluster Caching with Selective Partition
4444
[7] Y. Yu, R. Huang, W. Wang, J. Zhang, and K. B. Letaief, “SP‐Cache: Load‐balanced, Redundancy‐free Cluster Caching with Selective Partition,” accepted by IEEE/ACM Int. Conf. High Perform. Comput., Netw., Storage, and Analysis (SC), Nov. 2018.
![Page 45: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/45.jpg)
The Challenge of Load Imbalance
Spark, Piccolo, TensorFlow, …
A B
Hot files
45
![Page 46: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/46.jpg)
Root Causes
1. Skewed file popularity. 2. Hot files are usually of large sizes.
Distribution of file popularity and size observed in a Yahoo! cluster.
46
![Page 47: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/47.jpg)
47
Load‐Imbalance Cancels out the Caching Benefit
The read latencies with and without caching as the load of hot files increases.
5x latency reduction
Network congestion dominates
![Page 48: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/48.jpg)
48
Resort to redundant caching:(1) copy multiple replicas(2) create coded partitions.
Replication: High cache overheadCoding: High compute overhead.
Problems of Existing Solutions
![Page 49: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/49.jpg)
49
Replicate hot files, e.g., Scarlett [8].
Applications
A
Serve the requests to file A with of prob. of 1/3.
Other files are evicted: lower cache hits
Applications
A AA
More options
Existing Solutions – 1: Selective Replication
[8] G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris, “Scarlett: coping with skewed content popularity in MapReduce clusters,” in Proc. ACM Eurosys, 2011.
![Page 50: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/50.jpg)
50
High memory redundancy: each replica incurs an additional memory overhead of 1×.
1 2 3 4 5Replicas # of the top 10% popular files
0
2
4
6
Avg
. rea
d la
tenc
y (s
)
0
20
40
Mem
ory
over
head
(%)
Avg. read latencyMemory overhead Linearly increasing cost
Sublinear improvement
Existing Solutions – 1: Selective Replication
![Page 51: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/51.jpg)
51
Storage codes, e.g., EC‐Cache [9].
( ) coding scheme:
Divide
.
.
Any out of the partitions can recovery the original file.
Original file
Encode
.
.( )(Evenly)
Existing Solutions – 2: Erasure Coding
[9] K. Rashmi, M. Chowdhury, J. Kosaian, I. Stoica, and K. Ramchandran, “EC‐Cache: Load‐balanced, low‐latency cluster caching with onlineerasure coding.” in Proc. USENIX OSDI, 2016.
![Page 52: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/52.jpg)
52
Applications
A B
Applications
A1 A3 B1
A
encoding A1
A2
A3
The traffic to file A is distributed to three servers.
ReadA1
A2
decodeA
Smaller partitions
A2 B2 B3
(2, 3)
Existing Solutions – 2: Erasure Coding
![Page 53: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/53.jpg)
53
Decoding/encoding overhead
20 50 100 150 200 250File size (MB)
0
15
30
Dec
odin
g ov
erhe
ad (%
)
Existing Solutions – 2: Erasure Coding
![Page 54: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/54.jpg)
54
A simple yet effective approach –
splitting files without parity partitions.
No redundancy.No decoding/encoding overhead.
Spreading the load to multiple partitions.Read/write parallelism.
Our Solution: File Partition
![Page 55: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/55.jpg)
3 9 15 21 27Partition #
1
1.5
2
2.5
Avg
. rea
d la
tenc
y (s
)
55
Partition is unnecessary (and even troublesome) forthe trivial many small files. Selective partition
Network overhead: TCP connections.
Stragglers.
3 9 15 21 27Partition #
1
1.5
2
2.5
Avg
. rea
d la
tenc
y (s
)
W/o stragglersW/ stragglers
Challenges
17‐22x improvement over non‐partitioning
Network overhead
No need for too many partitions
![Page 56: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/56.jpg)
56
Divide files in proportion to their expected loads. size popularity
Scale factor(design parameter)
Intuition: Differentiate the vital few from the trivial many. A few hot files contribute the most to the load imbalance. Mitigate the hotspots without amplifying the straggler
effects.
SP‐Cache: Selective Replication
![Page 57: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/57.jpg)
57
To minimize the straggler effects, should be just large enough to mitigate the hotspots.
Solution:1. Set up a model without stragglers.2. Initialize .3. Gradually increase , until the improvement of
average read latency become marginal.
Configuration of the Scale Factor
![Page 58: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/58.jpg)
58
A tight upper bound to approximate the average read latency.
0 0.5 1 1.5 2 2.5Scale factor for SP-Cache
0.6
0.8
1
1.2
1.4
Avg
. rea
d la
tenc
y (s
)
0 0.5 1 1.5 2 2.5Scale factor for SP-Cache
0.6
0.8
1
1.2
1.4
Avg
. rea
d la
tenc
y (s
)
Upper boundExperiment
Find the “Elbow” Point
“Elbow” Point
![Page 59: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/59.jpg)
59
Implementation
![Page 60: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/60.jpg)
60
5 10 15 20 25File request rate
0
2
4
6
8
Avg
. lat
ency
(s)
Selective ReplicationEC-CacheSP-Cache
5 10 15 20 25File request rate
0
5
10
15
20
Tail
(95th
) lat
ency
(s) Selective Replication
EC-CacheSP-Cache
Average latency Tail latency
Improvement up to 50% and 70%, respectively Improvement up to 55% and 60%, respectively
Evaluations – Read Latency
![Page 61: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/61.jpg)
61
5 10 15 20 25 30Servers sorted by load
10
20
30
40
50
60
70
80
Dat
a re
ad (G
B)
5 10 15 20 25 30Servers sorted by load
10
20
30
40
50
60
70
80D
ata
read
(GB
)
5 10 15 20 25 30Servers sorted by load
10
20
30
40
50
60
70
80
Dat
a re
ad (G
B)
Selective replication EC‐Cache
SP‐Cache
1.18 0.44
0.18
Evaluations – Loads
Imbalance factor
![Page 62: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/62.jpg)
62
Load imbalance: reasons and impacts.
Selective partition: algorithms and analysis.
Compared to the‐state‐of‐the‐art: improve both average and tail latencies by up to 50%; improve the load‐balance by 2.45x.
Wrap up – Load Balanced Caching
![Page 63: Cache Management for In Memory Analytics - EIE | …jeiezhang/document/Talk_Cache.pdfOutline 2 1. Introduction 2. LRC: Dependency‐aware caching 3. OpuS: Fair cache sharing in multi‐tenant](https://reader030.vdocument.in/reader030/viewer/2022040608/5ec5c828a42c801031179662/html5/thumbnails/63.jpg)
63
Summary
LRC: dependency‐aware caching management Exploit dependency DAGs. Effective cache hit ratio as the metric.
Opus: fair cache sharing built atop the VCG mechanism Strategy‐proof. Isolation guarantee. Near optimal efficiency.
SP‐Cache: efficient load balancing with selective partition Redundancy‐free and coding‐free. Reduces latency and improves load balancing with less
memory footprint.