![Page 1: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/1.jpg)
DHTs and ShardingAurojit Panda
![Page 2: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/2.jpg)
Announcements
![Page 3: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/3.jpg)
Announcements
• Fill out the Github consent form.
![Page 4: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/4.jpg)
Announcements
• Fill out the Github consent form.
• Fill out the Github account association form.
![Page 5: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/5.jpg)
Announcements
• Fill out the Github consent form.
• Fill out the Github account association form.
• Lab 0 is due tonight. Hopefully all of you are done.
![Page 6: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/6.jpg)
Announcements
• Fill out the Github consent form.
• Fill out the Github account association form.
• Lab 0 is due tonight. Hopefully all of you are done.
• Contacting me: please send e-mail to [email protected]
![Page 7: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/7.jpg)
Announcements
• Fill out the Github consent form.
• Fill out the Github account association form.
• Lab 0 is due tonight. Hopefully all of you are done.
• Contacting me: please send e-mail to [email protected]
• I do not check the main NYU e-mail address all that often.
![Page 8: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/8.jpg)
Clarifying Final Project• Some concern about how projects will be graded.
![Page 9: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/9.jpg)
Clarifying Final Project• Some concern about how projects will be graded.
• Not really looking to be adversarial: ideal outcome is everyone gets an A.
![Page 10: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/10.jpg)
Clarifying Final Project• Some concern about how projects will be graded.
• Not really looking to be adversarial: ideal outcome is everyone gets an A.
• Main bar: write a proposal that we both agree on.
![Page 11: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/11.jpg)
Clarifying Final Project• Some concern about how projects will be graded.
• Not really looking to be adversarial: ideal outcome is everyone gets an A.
• Main bar: write a proposal that we both agree on.
• Important to scope it so that you have some chance of finishing things.
![Page 12: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/12.jpg)
Clarifying Final Project• Some concern about how projects will be graded.
• Not really looking to be adversarial: ideal outcome is everyone gets an A.
• Main bar: write a proposal that we both agree on.
• Important to scope it so that you have some chance of finishing things.
• Important to include something you would enjoy working on.
![Page 13: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/13.jpg)
Clarifying Final Project• Some concern about how projects will be graded.
• Not really looking to be adversarial: ideal outcome is everyone gets an A.
• Main bar: write a proposal that we both agree on.
• Important to scope it so that you have some chance of finishing things.
• Important to include something you would enjoy working on.
• Beyond that main metric is whether you put in the effort.
![Page 14: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/14.jpg)
Clarifying Final Project• Some concern about how projects will be graded.
• Not really looking to be adversarial: ideal outcome is everyone gets an A.
• Main bar: write a proposal that we both agree on.
• Important to scope it so that you have some chance of finishing things.
• Important to include something you would enjoy working on.
• Beyond that main metric is whether you put in the effort.
• Expect that many projects won't be done in time. That is fine.
![Page 15: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/15.jpg)
Clarifying Final Project• Some concern about how projects will be graded.
• Not really looking to be adversarial: ideal outcome is everyone gets an A.
• Main bar: write a proposal that we both agree on.
• Important to scope it so that you have some chance of finishing things.
• Important to include something you would enjoy working on.
• Beyond that main metric is whether you put in the effort.
• Expect that many projects won't be done in time. That is fine.
• Some ideas will fail to bear out. That is also fine.
![Page 16: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/16.jpg)
Some Comments on Summaries• Thanks for sending summaries, looked over most of them.
![Page 17: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/17.jpg)
Some Comments on Summaries• Thanks for sending summaries, looked over most of them.
• A few observations about mechanics:
![Page 18: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/18.jpg)
Some Comments on Summaries• Thanks for sending summaries, looked over most of them.
• A few observations about mechanics:
• Please use the subject and e-mail address on the website.
![Page 19: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/19.jpg)
Some Comments on Summaries• Thanks for sending summaries, looked over most of them.
• A few observations about mechanics:
• Please use the subject and e-mail address on the website.
• Please write your summary in the body of the e-mail, do not send t.
![Page 20: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/20.jpg)
Some Comments on Summaries• Thanks for sending summaries, looked over most of them.
• A few observations about mechanics:
• Please use the subject and e-mail address on the website.
• Please write your summary in the body of the e-mail, do not send t.
• A few observations about content:
![Page 21: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/21.jpg)
Some Comments on Summaries• Thanks for sending summaries, looked over most of them.
• A few observations about mechanics:
• Please use the subject and e-mail address on the website.
• Please write your summary in the body of the e-mail, do not send t.
• A few observations about content:
• Sorry about Slicer, it was not as self-contained as I thought it would be.
![Page 22: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/22.jpg)
Some Comments on Summaries• Thanks for sending summaries, looked over most of them.
• A few observations about mechanics:
• Please use the subject and e-mail address on the website.
• Please write your summary in the body of the e-mail, do not send t.
• A few observations about content:
• Sorry about Slicer, it was not as self-contained as I thought it would be.
• Would have liked to see a little more on how the papers relate to each other.
![Page 23: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/23.jpg)
Office Hour Etiquette • Sorry, I know this sounds bad for you, but necessary for my sanity.
![Page 24: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/24.jpg)
Office Hour Etiquette • Sorry, I know this sounds bad for you, but necessary for my sanity.
• My office hours are on Thursday from 5 - 6PM or by appointment.
![Page 25: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/25.jpg)
Office Hour Etiquette • Sorry, I know this sounds bad for you, but necessary for my sanity.
• My office hours are on Thursday from 5 - 6PM or by appointment.
• Please don't come to my office for class help outside of these circumstances.
![Page 26: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/26.jpg)
Office Hour Etiquette • Sorry, I know this sounds bad for you, but necessary for my sanity.
• My office hours are on Thursday from 5 - 6PM or by appointment.
• Please don't come to my office for class help outside of these circumstances.
• Fine to come for other things like:
![Page 27: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/27.jpg)
Office Hour Etiquette • Sorry, I know this sounds bad for you, but necessary for my sanity.
• My office hours are on Thursday from 5 - 6PM or by appointment.
• Please don't come to my office for class help outside of these circumstances.
• Fine to come for other things like:
• Research or advice about random things.
![Page 28: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/28.jpg)
Office Hour Etiquette • Sorry, I know this sounds bad for you, but necessary for my sanity.
• My office hours are on Thursday from 5 - 6PM or by appointment.
• Please don't come to my office for class help outside of these circumstances.
• Fine to come for other things like:
• Research or advice about random things.
• Informing me about new/better ice cream places.
![Page 29: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/29.jpg)
Now onto more fun stuff
![Page 30: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/30.jpg)
Two Problems - One Solution?
• Distributed data storage and management.
• Load distribution.
![Page 31: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/31.jpg)
Two Problems - One Solution?
• Distributed data storage and management.
• Load distribution.
![Page 32: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/32.jpg)
DHTs
![Page 33: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/33.jpg)
Some History
• 1999: Shawn Fanning and Sean Parker create Napster
![Page 34: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/34.jpg)
Some History
• 1999: Shawn Fanning and Sean Parker create Napster
• 2000: Metallica + RIAA + other sue Napster.
![Page 35: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/35.jpg)
Some History
• 1999: Shawn Fanning and Sean Parker create Napster
• 2000: Metallica + RIAA + other sue Napster.
• 2001: Napster shuts down network due to injunction.
![Page 36: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/36.jpg)
Some History
• 1999: Shawn Fanning and Sean Parker create Napster
• 2000: Metallica + RIAA + other sue Napster.
• 2001: Napster shuts down network due to injunction.
• What? Why?
![Page 37: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/37.jpg)
Napster: Original Design
Yogi
Huckelberry
Snuffles
Penelope
Track 1
Track 2Track 1: SnufflesTrack 2: Penelope
![Page 38: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/38.jpg)
Napster: Original Design
Yogi
Huckelberry
Snuffles
Penelope
Track 1
Track 2Track 1: SnufflesTrack 2: Penelope
![Page 39: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/39.jpg)
Napster: Original Design
Yogi
Huckelberry
Snuffles
Penelope
Track 1
Track 2
![Page 40: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/40.jpg)
Centralization affects Survivability
• Easier to shutdown infrastructure.
![Page 41: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/41.jpg)
Centralization affects Survivability
• Easier to shutdown infrastructure.
• Infrastructure is more susceptible to failure.
![Page 42: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/42.jpg)
Centralization affects Survivability
• Easier to shutdown infrastructure.
• Infrastructure is more susceptible to failure.
• But also much easier to build and manage.
![Page 43: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/43.jpg)
An Alternative to Centralization
Yogi
Huckelberry
Snuffles
Penelope
Track 1
Track 2
... ...
......
...
![Page 44: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/44.jpg)
An Alternative to Centralization
Yogi
Huckelberry
Snuffles
Penelope
Track 1
Track 2
... ...
......
...
Any problems with this approach?
![Page 45: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/45.jpg)
Distributed Hash Tables• Rather than replicating data, distribute it across nodes.
• Requirements:
• Allow lookup without requiring a centralized index.
• Allow nodes to enter and exit with relative ease.
• Several proposals: Chord, CAN, Kadmelia, ...
• Several in use: Kadmelia used for the BitTorrent trackers.
![Page 46: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/46.jpg)
Chord
![Page 47: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/47.jpg)
Consistent Hashing
• Consider a hash table with b buckets and n items.
![Page 48: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/48.jpg)
Consistent Hashing
• Consider a hash table with b buckets and n items.
• Add a new bucket, so there are now b + 1 buckets.
![Page 49: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/49.jpg)
Consistent Hashing
• Consider a hash table with b buckets and n items.
• Add a new bucket, so there are now b + 1 buckets.
• How many items need to move?
![Page 50: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/50.jpg)
Consistent Hashing• Depends on what hashing algorithm you use.
• But one pretty common strategy (with some additional details missing)
h:16 buckets
![Page 51: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/51.jpg)
Consistent Hashing• Depends on what hashing algorithm you use.
• But one pretty common strategy (with some additional details missing)
h:
h[k] = v
16 buckets
![Page 52: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/52.jpg)
Consistent Hashing• Depends on what hashing algorithm you use.
• But one pretty common strategy (with some additional details missing)
h:
h[k] = v
16 buckets
h[hash(k) % len(h)] = v
![Page 53: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/53.jpg)
Consistent Hashing• Depends on what hashing algorithm you use.
• But one pretty common strategy (with some additional details missing)
h:
h[k] = v
16 buckets
h[hash(k) % len(h)] = v
h[hash(k) % 16] = v
![Page 54: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/54.jpg)
Consistent Hashing• Depends on what hashing algorithm you use.
• But one pretty common strategy (with some additional details missing)
h:
h[k] = v
16 buckets
h[hash(k) % len(h)] = v
h[hash(k) % 16] = v
v
![Page 55: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/55.jpg)
Consistent Hashing
h:
h[k] = v
16 buckets
h[hash(k) % len(h)] = v
h[hash(k) % 16] = v
x = h[k]
v
![Page 56: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/56.jpg)
Consistent Hashing
h:
h[k] = v
16 buckets
h[hash(k) % len(h)] = v
h[hash(k) % 16] = v
x = h[k]
x = h[hash(k) % len(h)]
v
![Page 57: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/57.jpg)
Consistent Hashing
h:
h[k] = v
16 buckets
h[hash(k) % len(h)] = v
h[hash(k) % 16] = v
x = h[k]
x = h[hash(k) % len(h)]
x = h[hash(k) % 16]
v
![Page 58: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/58.jpg)
Consistent Hashing
h:20 buckets
x = h[k]
x = h[hash(k) % len(h)]
x = h[hash(k) % 16]
Added 4 buckets
x = h[k]
x = h[hash(k) % len(h)]
x = h[hash(k) % 20]
v
![Page 59: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/59.jpg)
Consistent Hashing
h:20 buckets
x = h[k]
x = h[hash(k) % len(h)]
x = h[hash(k) % 16]
Added 4 buckets
x = h[k]
x = h[hash(k) % len(h)]
x = h[hash(k) % 20]
v
How many items need to move?
![Page 60: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/60.jpg)
Consistent Hashing
• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments.
![Page 61: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/61.jpg)
Consistent Hashing
• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments.
• Monotonicity: When buckets are added items move from old to new.
![Page 62: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/62.jpg)
Consistent Hashing
• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments.
• Monotonicity: When buckets are added items move from old to new.
• No items move between old buckets.
![Page 63: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/63.jpg)
Consistent Hashing
• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments.
• Monotonicity: When buckets are added items move from old to new.
• No items move between old buckets.
• Also require other properties (e.g., balance) which are more traditional.
![Page 64: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/64.jpg)
Consistent Hashing
![Page 65: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/65.jpg)
Consistent Hashing• Map each bucket to a point in the unit interval.
![Page 66: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/66.jpg)
Consistent Hashing• Map each bucket to a point in the unit interval.
• How? Use a hash function hb.
![Page 67: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/67.jpg)
Consistent Hashing• Map each bucket to a point in the unit interval.
• How? Use a hash function hb.
• Map each key to a point in the unit interval.
![Page 68: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/68.jpg)
Consistent Hashing• Map each bucket to a point in the unit interval.
• How? Use a hash function hb.
• Map each key to a point in the unit interval.
• How? Use another hash function hk.
![Page 69: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/69.jpg)
Consistent Hashing• Map each bucket to a point in the unit interval.
• How? Use a hash function hb.
• Map each key to a point in the unit interval.
• How? Use another hash function hk.
• Map key k to bucket b such that: |hk(k) - hb(b)| is minimal.
![Page 70: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/70.jpg)
Consistent Hashing• Map each bucket to a point in the unit interval.
• How? Use a hash function hb.
• Map each key to a point in the unit interval.
• How? Use another hash function hk.
• Map key k to bucket b such that: |hk(k) - hb(b)| is minimal.
• Is this sufficient for monotonicity?
![Page 71: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/71.jpg)
Chord
• Use consistent hashing to build a distributed hash table.
• Keys must be allocated across different nodes.
• Balanced, i.e., no more than (1 + ε)K/N keys go to a single node.
• Minimize amount of data that needs to be synchronized.
• Especially important when nodes leave or join.
![Page 72: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/72.jpg)
Chord Slow Lookup
2
351
Successor pointer
![Page 73: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/73.jpg)
Chord Slow Lookup
2
351
Successor pointer
lookup(k)hash(k) = 7
![Page 74: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/74.jpg)
Chord Slow Lookup
2
351
Successor pointer
lookup(k)hash(k) = 7
![Page 75: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/75.jpg)
Chord Join
2
351
Successor pointer
2: ...
3: ... 4: ...27: ...
37: ... 42: ...
0: ...
![Page 76: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/76.jpg)
Chord Join
2
351
Successor pointer
8
2: ...
3: ... 4: ...27: ...
37: ... 42: ...
0: ...
![Page 77: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/77.jpg)
Chord Join
2
351
Successor pointer
8
2: ...
3: ... 4: ...27: ...
37: ... 42: ...
0: ...
Predecessor pointer
![Page 78: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/78.jpg)
Chord Join
2
351
Successor pointer
8
2: ...
3: ... 4: ...27: ...
37: ... 42: ...
0: ...
Predecessor pointer
![Page 79: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/79.jpg)
Chord Join
2
351
Successor pointer
8
2: ...
27: ...37: ...
42: ...0: ...
Predecessor pointer
3: ... 4: ...
![Page 80: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/80.jpg)
Chord Leave
2
351
Successor pointer
8
2: ...
27: ...37: ...
42: ...0: ...
Predecessor pointer
3: ... 4: ...
![Page 81: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/81.jpg)
Chord Leave
2
351
Successor pointer
2: ...
27: ...37: ...
42: ...0: ...
Predecessor pointer
![Page 82: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/82.jpg)
Chord Leave
2
351
Successor pointer
2: ...
27: ...37: ...
42: ...0: ...
Predecessor pointer
Oops: Need successor list to fix this problem
![Page 83: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/83.jpg)
Chord Leave
2
351
Successor pointer
2: ...
27: ...37: ...
42: ...0: ...
Predecessor pointer
![Page 84: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/84.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
![Page 85: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/85.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
![Page 86: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/86.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
finger[k] = succ(n+ 2k�1) mod 2m)<latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit>
![Page 87: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/87.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
finger[k] = succ(n+ 2k�1) mod 2m)<latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit>
Finger
![Page 88: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/88.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
finger[k] = succ(n+ 2k�1) mod 2m)<latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit>
Finger
1
![Page 89: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/89.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
finger[k] = succ(n+ 2k�1) mod 2m)<latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit>
Finger
1 2
![Page 90: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/90.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
finger[k] = succ(n+ 2k�1) mod 2m)<latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit>
Finger
1 2
3
![Page 91: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/91.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
finger[k] = succ(n+ 2k�1) mod 2m)<latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit>
Finger
1 2
3
4
![Page 92: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/92.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
finger[k] = succ(n+ 2k�1) mod 2m)<latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit>
Finger
1 2
3
4
5
![Page 93: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/93.jpg)
Fingers20
3
6
8
10
12
1416
182022
24
26
28
30
32
34
38 3.lookup(k)hash(k) = 42
m=6 (2m = 64)
finger[k] = succ(n+ 2k�1) mod 2m)<latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit><latexit sha1_base64="o0lJPfhPEwESip2b8DuHRmlkxGM=">AAACHnicbVDLSgMxFM34tr6qLt0Ei6CIZUYU3QiiG5cKVoV2LJn0Tg1NMkNyRyzDfIkbf8WNC0UEV/o3po+FrwMXTs65l9x7olQKi77/6Y2Mjo1PTE5Nl2Zm5+YXyotLFzbJDIcaT2RiriJmQQoNNRQo4So1wFQk4TLqHPf8y1swViT6HLsphIq1tYgFZ+ikZnm3gXCHeSx0G0xR74T0gA4km3FerGu6Sbev885WUGzQhkpa7qU2muWKX/X7oH9JMCQVMsRps/zeaCU8U6CRS2ZtPfBTDHNmUHAJRamRWUgZ77A21B3VTIEN8/55BV1zSovGiXGlkfbV7xM5U9Z2VeQ6FcMb+9vrif959Qzj/TAXOs0QNB98FGeSYkJ7WdGWMMBRdh1h3Ai3K+U3zDCOLtGSCyH4ffJfcrFdDfxqcLZTOTwaxjFFVsgqWScB2SOH5ISckhrh5J48kmfy4j14T96r9zZoHfGGM8vkB7yPLztQoTU=</latexit>
Finger
1 2
3
4
5
6
![Page 94: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/94.jpg)
Some Concerns
• What happens with simultaneous joins and leaves?
• Can you ever loose track of a node?
• Can you ever get to a state where a node joins and then disappears?
![Page 95: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/95.jpg)
An aside on hashing
• Why SHA-1/SHA-256/...?
![Page 96: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/96.jpg)
An aside on hashing
• Why SHA-1/SHA-256/...?
• Do we need cryptographic hash functions for this purpose?
![Page 97: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/97.jpg)
An aside on hashing
• Why SHA-1/SHA-256/...?
• Do we need cryptographic hash functions for this purpose?
• There are other hash functions that are much faster to compute.
![Page 98: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/98.jpg)
An aside on hashing
• Why SHA-1/SHA-256/...?
• Do we need cryptographic hash functions for this purpose?
• There are other hash functions that are much faster to compute.
• Why are cryptographic hash functions well suited to this purpose?
![Page 99: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/99.jpg)
Two Problems - One Solution?
• Distributed data storage and management.
• Load distribution.
![Page 100: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/100.jpg)
Sharding and Load Balancing
![Page 101: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/101.jpg)
Why Shard?• Services like databases rely on several resources.
![Page 102: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/102.jpg)
Why Shard?• Services like databases rely on several resources.
• Storage (to persist), memory (to cache), CPU (to compute), etc.
![Page 103: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/103.jpg)
Why Shard?• Services like databases rely on several resources.
• Storage (to persist), memory (to cache), CPU (to compute), etc.
• An individual node might not have sufficient resources to handle load.
![Page 104: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/104.jpg)
Why Shard?• Services like databases rely on several resources.
• Storage (to persist), memory (to cache), CPU (to compute), etc.
• An individual node might not have sufficient resources to handle load.
• For example, the database might be too big to fit on one server.
![Page 105: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/105.jpg)
Why Shard?• Services like databases rely on several resources.
• Storage (to persist), memory (to cache), CPU (to compute), etc.
• An individual node might not have sufficient resources to handle load.
• For example, the database might be too big to fit on one server.
• For example, many updates to a single row mean not enough CPU for all.
![Page 106: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/106.jpg)
Why Shard?• Services like databases rely on several resources.
• Storage (to persist), memory (to cache), CPU (to compute), etc.
• An individual node might not have sufficient resources to handle load.
• For example, the database might be too big to fit on one server.
• For example, many updates to a single row mean not enough CPU for all.
• Sharding is one way to spread the load around.
![Page 107: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/107.jpg)
Load Balancing
• Sometimes one cannot partition the data, and instead must replicate it.
![Page 108: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/108.jpg)
Load Balancing
• Sometimes one cannot partition the data, and instead must replicate it.
• Also provides resilience: no replication no data after failure.
![Page 109: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/109.jpg)
Load Balancing
• Sometimes one cannot partition the data, and instead must replicate it.
• Also provides resilience: no replication no data after failure.
• Load balancing decides who serves any given request.
![Page 110: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/110.jpg)
Load Balancing
• Sometimes one cannot partition the data, and instead must replicate it.
• Also provides resilience: no replication no data after failure.
• Load balancing decides who serves any given request.
• Who scales the load balancer?
![Page 111: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/111.jpg)
Load Balancing
• Sometimes one cannot partition the data, and instead must replicate it.
• Also provides resilience: no replication no data after failure.
• Load balancing decides who serves any given request.
• Who scales the load balancer?
• Interesting question but won't think about this today.
![Page 112: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/112.jpg)
Both Problems are Somewhat Related
![Page 113: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/113.jpg)
Both Problems are Somewhat Related
• Where should a request be routed?
![Page 114: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/114.jpg)
Both Problems are Somewhat Related
• Where should a request be routed?
• Where should something be stored?
![Page 115: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/115.jpg)
Some Useful Properties
• Related data stays together/related requests get routed to the same machine.
x = 1 x = 1
![Page 116: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/116.jpg)
Some Useful Properties
• Related data stays together/related requests get routed to the same machine.
x = 1 x = 1
set x = 2
![Page 117: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/117.jpg)
Some Useful Properties
• Related data stays together/related requests get routed to the same machine.
x = 2 x = 1
set x = 2
set x = 2
![Page 118: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/118.jpg)
Some Useful Properties
• Related data stays together/related requests get routed to the same machine.
x = 2 x = 1
set x = 2 get x
set x = 2
![Page 119: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/119.jpg)
Some Useful Properties
• Related data stays together/related requests get routed to the same machine.
x = 2 x = 1
set x = 2
set x = 2
![Page 120: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/120.jpg)
Some Useful Properties
• Related data stays together/related requests get routed to the same machine.
x = 2 x = 1
set x = 2get x
set x = 2
![Page 121: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/121.jpg)
Some Useful Properties
• Related data stays together/related requests get routed to the same machine.
x = 2 x = 1
set x = 2get x
set x = 2
When are requests/data related?
![Page 122: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/122.jpg)
Some Useful Properties
• Ability to add more capacity without disruption.
x = 1 x = 1
![Page 123: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/123.jpg)
How can we do this?
![Page 124: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/124.jpg)
Sometimes Consistent Hashing is Not Enough
• Consistent hashing works to give the previous two properties.
![Page 125: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/125.jpg)
Sometimes Consistent Hashing is Not Enough
• Consistent hashing works to give the previous two properties.
• Bigger problem is that a single data unit, or request might be more popular.
![Page 126: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/126.jpg)
Sometimes Consistent Hashing is Not Enough
• Consistent hashing works to give the previous two properties.
• Bigger problem is that a single data unit, or request might be more popular.
• Common observance for web workloads.
![Page 127: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/127.jpg)
Sometimes Consistent Hashing is Not Enough
• Consistent hashing works to give the previous two properties.
• Bigger problem is that a single data unit, or request might be more popular.
• Common observance for web workloads.
• Random hashing is insufficient for this.
![Page 128: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/128.jpg)
Sometimes Consistent Hashing is Not Enough
• Consistent hashing works to give the previous two properties.
• Bigger problem is that a single data unit, or request might be more popular.
• Common observance for web workloads.
• Random hashing is insufficient for this.
• Very little control over what ends up loaded.
![Page 129: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/129.jpg)
Sometimes Consistent Hashing is Not Enough
• Consistent hashing works to give the previous two properties.
• Bigger problem is that a single data unit, or request might be more popular.
• Common observance for web workloads.
• Random hashing is insufficient for this.
• Very little control over what ends up loaded.
• Too hard to reverse engineer an appropriate hash function.
![Page 130: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/130.jpg)
Slicer Like Solutions• Retain some structures from consistent hashing.
![Page 131: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/131.jpg)
Slicer Like Solutions• Retain some structures from consistent hashing.
• Map keys to ranges in some interval.
![Page 132: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/132.jpg)
Slicer Like Solutions• Retain some structures from consistent hashing.
• Map keys to ranges in some interval.
• Assign key ranges to individual nodes.
![Page 133: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/133.jpg)
Slicer Like Solutions• Retain some structures from consistent hashing.
• Map keys to ranges in some interval.
• Assign key ranges to individual nodes.
• Respond to different scenarios by
![Page 134: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/134.jpg)
Slicer Like Solutions• Retain some structures from consistent hashing.
• Map keys to ranges in some interval.
• Assign key ranges to individual nodes.
• Respond to different scenarios by
• Merging adjacent ranges.
![Page 135: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/135.jpg)
Slicer Like Solutions• Retain some structures from consistent hashing.
• Map keys to ranges in some interval.
• Assign key ranges to individual nodes.
• Respond to different scenarios by
• Merging adjacent ranges.
• Partitioning ranges.
![Page 136: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/136.jpg)
A Few Challenges• Assigning a key to a partition is no longer stateless.
• Need to record this information somewhere.
• Need to make sure that you can safely update this configuration.
• Despite multiple assigners which might fail or arrive at inopportune moments.
• Despite arbitrary delays.
• One of the things the paper addresses using CAS in Section 4.1.
![Page 137: DHTs and Sharding...• David Karger, et al. 1997 at MIT: Mechanism to minimize reassignments. • Monotonicity: When buckets are added items move from old to new. • No items move](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3df9ecb1a455fce36b9cd/html5/thumbnails/137.jpg)
Conclusion• Problem: How to spread request/data/... across machines.
• Rather how to ensure that everyone agrees on what machine to go to.
• Elegant solution: Do it with hashing, no state.
• Elegant but some limits on what can be achieved.
• General solution: Rely on state, enforce any policy on how things are spread.
• Now you have two problems: how to distribute and keep state in sync?