[@indeedeng] boxcar: a self-balancing distributed services protocol
DESCRIPTION
Video available at: http://www.youtube.com/watch?v=E1ok08TVxDw Indeed's flagship job search product has evolved over the years to meet new challenges. It began as a single, monolithic web application. This grew larger and increasingly complex as we built new features. To remedy this growing problem, we implemented a service-oriented architecture to improve system availability, scalability, and maintainability. We examined common practices for service-oriented architectures, and we discovered ways to improve on the state of the art. We developed these ideas into a new framework called Boxcar. In this talk, we will discuss the scaling problems we solved, the innovative ideas behind boxcar, and how we built the scalable architecture that we now use throughout our systems. R.B. Boyer is a Software Engineer who has been with Indeed since late 2007. Over the years he has worked on a variety of projects, including distributed storage, authentication, and service architectures.TRANSCRIPT
![Page 1: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/1.jpg)
BoxcarA self-balancing distributed services protocol
![Page 2: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/2.jpg)
R.B. BoyerSoftware Engineer
Resume
![Page 3: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/3.jpg)
I help people get jobs.
![Page 4: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/4.jpg)
I solveinteresting problems.
![Page 5: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/5.jpg)
Boxcar was the solutionto a problem:
![Page 6: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/6.jpg)
Building
![Page 7: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/7.jpg)
How we build products
Simple
Fast
Comprehensive
Relevant
![Page 8: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/8.jpg)
How we build products
Simple
Fast
Comprehensive
Relevant
![Page 9: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/9.jpg)
How we build systems
Simple
Fast
Resilient
Scalable
![Page 10: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/10.jpg)
“I want my application to be more complicated”
- No one ever
Simple
![Page 11: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/11.jpg)
Complexity creates confusion
![Page 12: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/12.jpg)
Complexity creates confusion
Confusion breeds bugs
![Page 13: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/13.jpg)
“I want my application to be slower”
- No one ever
Fast
![Page 14: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/14.jpg)
conducted a speed test
![Page 15: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/15.jpg)
+500 milliseconds of latency per search
![Page 16: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/16.jpg)
20% fewer searches
![Page 17: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/17.jpg)
Speed is a feature
![Page 18: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/18.jpg)
“I want my users to experience outages”
- No one ever
Resilient
![Page 19: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/19.jpg)
Programs crash
![Page 20: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/20.jpg)
Programs crash
Machines die
![Page 21: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/21.jpg)
Minimize vulnerability toany failure
![Page 22: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/22.jpg)
“My system will only need to support 10 users”
- No one ever
Scalable
![Page 23: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/23.jpg)
Scale with MORE machines
![Page 24: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/24.jpg)
Scale with MORE machines
Not BIGGER machines
![Page 25: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/25.jpg)
TL;DR:
![Page 26: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/26.jpg)
![Page 27: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/27.jpg)
Jobs Sites Job
Seekers
Indeed
![Page 28: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/28.jpg)
Aggregation Job Search
Jobs Sites Job
Seekers
![Page 29: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/29.jpg)
Aggregation
Job Search
![Page 30: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/30.jpg)
Aggregation
Job Search
Challenge!
![Page 31: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/31.jpg)
Challenge: SimpleFastResilientScalable
keep this
![Page 32: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/32.jpg)
Options:
![Page 33: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/33.jpg)
Share data access?
![Page 34: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/34.jpg)
Example:Shared Database
![Page 35: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/35.jpg)
Main Database
Shared Database
![Page 36: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/36.jpg)
Main Database
Main Application
Shared Database
![Page 37: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/37.jpg)
Main Database
Main Application
Analysis Tool
Shared Database
![Page 38: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/38.jpg)
Main Database
Main Application
Analysis Tool
Shared Database
BillingApplication
![Page 39: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/39.jpg)
Main Database
Main Application
Analysis Tool
BillingApplication
Intern Project
Shared Database
![Page 40: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/40.jpg)
Main Database
Main Application
Analysis Tool
BillingApplication
Intern Project
Other Intern Project
Shared Database
![Page 41: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/41.jpg)
Main Database
Main Application
Analysis Tool
BillingApplication
Intern Project
Other Intern Project
Email Tool
Shared Database
![Page 42: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/42.jpg)
Main Database
Main Application
Analysis Tool
BillingApplication
Intern Project
Other Intern Project
Email Tool
Shared Database
This is an anti-pattern
![Page 43: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/43.jpg)
On a long enough timeline...
![Page 44: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/44.jpg)
Maintenance Nightmare
![Page 45: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/45.jpg)
Share data access
![Page 46: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/46.jpg)
Share data accessInsulate data from
consumers
![Page 47: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/47.jpg)
Main Database
Main Application
Analysis Tool
BillingApplication
Intern Project
Other Intern Project
Email Tool
Shared Database
![Page 48: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/48.jpg)
Main Application
Analysis Tool
BillingApplication
Intern Project
Other Intern Project
Email Tool
Insulated Database
Main Database
Main Service
![Page 49: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/49.jpg)
Service?
![Page 50: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/50.jpg)
Service
ClientClientClientClient
![Page 51: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/51.jpg)
Service
NET
WO
RK
ClientClientClientClient
![Page 52: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/52.jpg)
Service
NET
WO
RK Icky
TechnicalStuff
ClientClientClientClient
![Page 53: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/53.jpg)
Service
Caches
Databases
Logging
Business Logic
...
Icky TechnicalDetails
Client API
ClientClientClientClient
NET
WO
RK
![Page 54: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/54.jpg)
Client API
Service.getJobs([12345, 62])
![Page 55: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/55.jpg)
Icky Technical Details
SELECT * FROM jobs AS jLEFT JOIN companyinfo AS ci ON j.id=ci.job_idLEFT JOIN locations AS loc ON loc.id=j.location_idWHERE j.id IN (12345, 62)
![Page 56: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/56.jpg)
Service Oriented
Architecture
![Page 57: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/57.jpg)
ServiceOriented
Architecture
![Page 58: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/58.jpg)
![Page 59: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/59.jpg)
Boxcar
![Page 60: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/60.jpg)
Boxcar is a...self-balancing
distributed
services
protocol
![Page 61: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/61.jpg)
Origin Story
![Page 62: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/62.jpg)
There was a life before Boxcar
![Page 63: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/63.jpg)
There were services before Boxcar
![Page 64: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/64.jpg)
Pick one:
![Page 65: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/65.jpg)
Doc Service
![Page 66: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/66.jpg)
Document Serving Serviceaka “Doc Service”
http://go.indeed.com/docservice
![Page 67: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/67.jpg)
Doc Service controlsaccess to JOBS
![Page 68: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/68.jpg)
Building Blocks
Webapp
DocService
Docstore
Wants jobs
Controls access to jobs
Stores jobs
![Page 69: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/69.jpg)
Build it
![Page 70: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/70.jpg)
Webapp
![Page 71: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/71.jpg)
Webapp
Doc Service
Docstore
![Page 72: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/72.jpg)
Webapp
Doc Service
Docstore
![Page 73: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/73.jpg)
Webapp
Doc Service
Docstore
Mission Accomplished
![Page 74: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/74.jpg)
But is it good?
![Page 75: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/75.jpg)
How we build systems
Simple
Fast
Resilient
Scalable
![Page 76: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/76.jpg)
Goodness Metric
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
![Page 77: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/77.jpg)
Webapp
Doc Service
Docstore
![Page 78: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/78.jpg)
Webapp
Doc Service
Docstore
![Page 79: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/79.jpg)
Webapp
Doc Service
Docstore
Resilient✘
![Page 80: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/80.jpg)
Add Resilience
![Page 81: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/81.jpg)
Webapp
Doc Service
Docstore
![Page 82: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/82.jpg)
Webapp
Doc Service
Docstore
Webapp
Doc Service
Docstore
![Page 83: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/83.jpg)
Front-end Load Balancer
Webapp
Doc Service
Docstore
Webapp
Doc Service
Docstore
![Page 84: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/84.jpg)
Siloed Stacks
Webapp
Doc Service
Docstore
Webapp
Doc Service
Docstore
Front-end Load Balancer
![Page 85: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/85.jpg)
Siloed Stacks
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
✓
✓
?✓
![Page 86: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/86.jpg)
Scaling Silos
Webapp
Doc Service
Docstore
Webapp
Doc Service
Docstore
Front-end Load Balancer
![Page 87: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/87.jpg)
Scaling Silos
Webapp Webapp
Doc Service
Docstore
Webapp
Doc Service
Docstore
Front-end Load Balancer
![Page 88: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/88.jpg)
Scaling Silos
Webapp
Doc Service
Docstore
Webapp
Doc Service
Docstore
Webapp
Doc Service
Docstore
Front-end Load Balancer
![Page 89: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/89.jpg)
Need bigger and
bigger machines
![Page 90: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/90.jpg)
Vertical Scaling
![Page 91: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/91.jpg)
Siloed Stacks
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable✘
✓
✓
✓
![Page 92: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/92.jpg)
Siloed Stacks
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable✘
✓ Services Version 1
✓
✓
![Page 93: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/93.jpg)
Improve scalability
![Page 94: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/94.jpg)
Webapp
Doc Service
Docstore
Webapp
Doc Service
Docstore
Webapp
Doc Service
Docstore
Front-end Load Balancer
![Page 95: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/95.jpg)
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
Doc Service Load Balancer
![Page 96: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/96.jpg)
Per-Service Balancer
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
Doc Service Load Balancer
![Page 97: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/97.jpg)
Per-Service Balancer
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
~
✓
??
![Page 98: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/98.jpg)
2x Bandwidth✘
Webapp
Doc Service
Doc Service Load Balancer
Proxying isn’t free
![Page 99: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/99.jpg)
Per-Service Balancer
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
~
✓
?✘
![Page 100: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/100.jpg)
Resilience
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
SINGLE POINT OF FAILURE
![Page 101: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/101.jpg)
Need two balancers
![Page 102: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/102.jpg)
Need two balancers
...and a way to balance between them?
![Page 103: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/103.jpg)
Load Balancer Balancing
Master / Slave
Share IP address
Heartbeat between nodes
Complex
![Page 104: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/104.jpg)
Resilience
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
Doc Service Load BalancerDoc Service Load Balancer
![Page 105: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/105.jpg)
![Page 106: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/106.jpg)
Best explained by our Operations folks:
“Redundant Array of Inexpensive Datacenters”
http://go.indeed.com/raid
![Page 107: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/107.jpg)
Per-Service Balancer
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
~
✓
✓
✘
![Page 108: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/108.jpg)
Per-Service Balancer
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
~
✓
✓ Services Version 2
✘
![Page 109: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/109.jpg)
Reduce network waste
![Page 110: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/110.jpg)
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
Doc Service Load Balancer
![Page 111: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/111.jpg)
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
![Page 112: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/112.jpg)
Naive Round Robin
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
![Page 113: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/113.jpg)
Naive Round Robin
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
✓
✓
??
![Page 114: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/114.jpg)
Direct Connections
1x Bandwidth✓
Webapp
Doc Service
![Page 115: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/115.jpg)
Naive Round Robin
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
✓
✓
?
✓
![Page 116: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/116.jpg)
Server A Server B
![Page 117: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/117.jpg)
Server A Server B
![Page 118: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/118.jpg)
Server A Server B
REQUEST
![Page 119: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/119.jpg)
Server A Server B
REQUEST✘
![Page 120: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/120.jpg)
Server A Server B
REQUEST✘
![Page 121: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/121.jpg)
Server A Server B
REQUEST
![Page 122: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/122.jpg)
Server A Server B
REQUEST
![Page 123: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/123.jpg)
Naive Round Robin
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable✓
✓
✓
✓
![Page 124: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/124.jpg)
Naive Round Robin
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable✓
✓
✓
✓
![Page 125: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/125.jpg)
Naive Round Robin
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
Balanced
✓
✓
?
✓
✓
![Page 126: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/126.jpg)
Slow Fast
![Page 127: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/127.jpg)
Slow Fast
![Page 128: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/128.jpg)
Slow Fast
![Page 129: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/129.jpg)
Slow Fast
![Page 130: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/130.jpg)
Slow Fast
![Page 131: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/131.jpg)
Slow Fast
![Page 132: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/132.jpg)
Slow Fast
![Page 133: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/133.jpg)
Slow Fast
![Page 134: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/134.jpg)
Slow Fast
![Page 135: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/135.jpg)
Slow Fast
![Page 136: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/136.jpg)
Slow Fast
![Page 137: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/137.jpg)
Can’t keep up
Slow Fast
![Page 138: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/138.jpg)
Naive Round Robin
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
Balanced
✓
✓
✓
✓
✘
![Page 139: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/139.jpg)
Naive Round Robin
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
Balanced
✓
✓
✘
NOPE
✓
✓
![Page 140: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/140.jpg)
Ensure balance
![Page 141: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/141.jpg)
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
Doc Service Load Balancer
![Page 142: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/142.jpg)
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
Distribute!
![Page 143: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/143.jpg)
Webapp WebappWebapp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
B BB
![Page 144: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/144.jpg)
WebApp
WebApp
WebApp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
B B B
![Page 145: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/145.jpg)
WebApp
WebApp
WebApp
Doc Service
Docstore
Doc Service
Docstore
Front-end Load Balancer
B B B
Boxcar
![Page 146: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/146.jpg)
Naive Round Robin
Per-Service Balancer
![Page 147: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/147.jpg)
The Boxcar balancing algorithm is simple
![Page 148: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/148.jpg)
Servers assign numeric value to connections
Clients use the connection with the lowest numeric value to service each request
Gist
![Page 149: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/149.jpg)
Server A
![Page 150: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/150.jpg)
Server A
Server A
![Page 151: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/151.jpg)
Server ASlot 0
Slot 1
Slot 2
Slot 3
Slot 4...
![Page 152: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/152.jpg)
Just numbers
No limit
NOT a priority
ONLY used for balancing
Server ASlot 0
Slot 1
Slot 2
Slot 3
Slot 4
...
Slot Numbers
![Page 153: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/153.jpg)
LOW slot numbers
are the
BEST slot numbers
![Page 154: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/154.jpg)
Server ASlot 0 USEDSlot 1 USEDSlot 2
Slot 3
Slot 4 USED
...
Client 2
Hello!
![Page 155: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/155.jpg)
Server ASlot 0 USEDSlot 1 USEDSlot 2 USEDSlot 3
Slot 4 USED
...
Client 2Slot 2
![Page 156: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/156.jpg)
Client 2
![Page 157: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/157.jpg)
Client 2
![Page 158: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/158.jpg)
Client 2
![Page 159: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/159.jpg)
Client 2Slot 0
Slot 2
Slot 12
Slot 29
Slot 30
Slot 57
Server
A
Server
B
![Page 160: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/160.jpg)
Client 2Slot 0
Slot 2
Slot 12
Slot 29
Slot 30
Slot 57
Server
A
Server
B
long-lived connections
![Page 161: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/161.jpg)
Clients are greedy
205
MINE!
![Page 162: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/162.jpg)
Clients are greedy
205MINE!
Want best connections
Continually look for better connections
Close worst connections
![Page 163: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/163.jpg)
Background thread maintains the
connection pool
![Page 164: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/164.jpg)
Client 2Slot 0
Slot 2
Slot 12
Slot 29
Slot 30
Slot 57
Server
A
Server
B
![Page 165: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/165.jpg)
Client 2Slot 0
Slot 2
Slot 12
Slot 29
Slot 30
Slot 57
Server
A
Server
B
Slot17
![Page 166: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/166.jpg)
Client 2Slot 0
Slot 2
Slot 12
Slot 29
Slot 30
Server
A
Server
B
✘
Slot17
Slot 57✘
![Page 167: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/167.jpg)
Client 2Slot 0
Slot 2
Slot 12
Slot 29
Slot 30
Server
A
Server
B
Slot17
![Page 168: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/168.jpg)
Client 2Slot 0
Slot 2
Slot 29
Slot 30
Server
A
Server
B
Slot17
Slot 12
![Page 169: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/169.jpg)
Client 2Slot 0
Slot 2
Slot 17
Slot 29
Slot 30
Server
A
Server
BSlot 12
![Page 170: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/170.jpg)
Client 2Slot 0
Slot 2
Slot 17
Slot 29
Slot 30
Server
A
Server
BSlot 12
Continues forever
![Page 171: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/171.jpg)
Client 2Slot 0 ACTIVESlot 2 ACTIVESlot 12 [idle]
Slot 29 ACTIVESlot 30 [idle]
Slot 57 [idle]
GetJobs()
Incoming Requests
![Page 172: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/172.jpg)
Client 2Slot 0 ACTIVESlot 2 ACTIVESlot 12 ACTIVESlot 29 ACTIVESlot 30 [idle]
Slot 57 [idle]
GetJobs()
Incoming Requests
![Page 173: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/173.jpg)
Connections NOTestablished on-demand
![Page 174: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/174.jpg)
Client 2Slot 0 ACTIVESlot 2 ACTIVESlot 12 ACTIVESlot 29 ACTIVESlot 30 ACTIVESlot 57 ACTIVE
GetJobs()
Requests to Busy Pool
![Page 175: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/175.jpg)
Client 2Slot 0 ACTIVESlot 2 ACTIVESlot 12 ACTIVESlot 29 ACTIVESlot 30 ACTIVESlot 57 ACTIVE
GetJobs()✘ERROR!
Requests to Busy Pool
![Page 176: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/176.jpg)
Sizing the pool properly is imperative!
![Page 177: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/177.jpg)
Servers assign numeric value to connections
Clients use the connection with the lowest numeric value to service each request
Gist Redux
![Page 178: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/178.jpg)
Balanced load is emergent behavior
![Page 179: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/179.jpg)
Load Balancing Simulations
![Page 180: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/180.jpg)
Client XServer A
Server B
![Page 181: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/181.jpg)
0Client XServer A
slot 0
Server B
![Page 182: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/182.jpg)
Client X0
Server Aslot 0
Server B
![Page 183: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/183.jpg)
Client X0
0
Server Aslot 0
Server Bslot 0
![Page 184: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/184.jpg)
Client X00
Server Aslot 0
Server Bslot 0
![Page 185: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/185.jpg)
1
Client X00
1
Server Aslot 0slot 1
Server B
slot 1slot 0
![Page 186: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/186.jpg)
Steady-state balance
Client X00
Server Aslot 0
Server Bslot 0
![Page 187: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/187.jpg)
Client X00
Server Aslot 0
Server Bslot 0
New Clients Join
![Page 188: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/188.jpg)
Client Y
Client X00
Server Aslot 0
Server Bslot 0
![Page 189: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/189.jpg)
Client Y
Client X00
1
1
Server Aslot 0slot 1
Server B
slot 1slot 0
![Page 190: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/190.jpg)
Client Y11
Client X00
Server Aslot 0slot 1
Server B
slot 1slot 0
![Page 191: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/191.jpg)
Client Y11
Client X00
2
2
Server Aslot 0slot 1slot 2
Server B
slot 1slot 2
slot 0
![Page 192: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/192.jpg)
Client Y11
Client X00
Server Aslot 0slot 1
Server B
slot 1slot 0
![Page 193: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/193.jpg)
Client Y11
Client Z
Client X00
Server Aslot 0slot 1
Server B
slot 1slot 0
![Page 194: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/194.jpg)
Client Y11
Client Z
Client X00
2
2
Server Aslot 0slot 1slot 2
Server B
slot 1slot 2
slot 0
![Page 195: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/195.jpg)
Client Y11
Client Z22
Client X00
Server Aslot 0slot 1slot 2
Server B
slot 1slot 2
slot 0
![Page 196: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/196.jpg)
Client Y11
Client Z22
Client X00
3
3
Server Aslot 0slot 1slot 2
Server B
slot 1slot 2slot 3
slot 3
slot 0
![Page 197: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/197.jpg)
Client Y11
Client Z22
Client X00
Steady-state balance
Server Aslot 0slot 1slot 2
Server B
slot 1slot 2
slot 0
![Page 198: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/198.jpg)
Client Y11
Client Z22
Client X00
Server Aslot 0slot 1slot 2
Server B
slot 1slot 2
slot 0
Server Failure
![Page 199: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/199.jpg)
Client Y1
Client Z2
Client X0
Server Aslot 0slot 1slot 2
Server Bslot 0
![Page 200: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/200.jpg)
Client Y1
Client Z2
Client X0
Server Aslot 0slot 1slot 2
Server B
![Page 201: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/201.jpg)
Client Y1
Client Z2
Client X0
Server Aslot 0slot 1slot 2slot 3
3
Server B
![Page 202: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/202.jpg)
Client Y1
Client Z23
Client X0
Server Aslot 0slot 1slot 2slot 3
Server B
![Page 203: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/203.jpg)
Client Y1
Client Z23
Client X0
Server Aslot 0slot 1slot 2slot 3
4
slot 4
Server B
![Page 204: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/204.jpg)
Client Y1
Client Z23
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4
Server B
![Page 205: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/205.jpg)
Client Y1
Client Z23
Client X04
Server Aslot 0slot 1slot 2slot 3
5
slot 4slot 5
Server B
![Page 206: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/206.jpg)
Client Y15
Client Z23
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server B
![Page 207: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/207.jpg)
Client Y15
Client Z23
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Steady-state balance
Server B
![Page 208: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/208.jpg)
Client Y15
Client Z23
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Steady-state balance
Server B
![Page 209: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/209.jpg)
Client Y15
Client Z23
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server B
Server Restored
![Page 210: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/210.jpg)
Client Y15
Client Z23
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server B
![Page 211: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/211.jpg)
Client Y15
Client Z23
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server Bslot 0
0
![Page 212: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/212.jpg)
Client Y15
Client Z2
0 < 3
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server Bslot 0
![Page 213: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/213.jpg)
Client Y15
Client Z20
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server Bslot 0
![Page 214: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/214.jpg)
Client Y15
Client Z20
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server Bslot 0
![Page 215: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/215.jpg)
Client Y15
Client Z02
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server Bslot 0
![Page 216: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/216.jpg)
Client Y15
Client Z02
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server B
slot 1slot 0
1
![Page 217: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/217.jpg)
Client Y1
1 < 5
Client Z02
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4slot 5
Server B
slot 1slot 0
![Page 218: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/218.jpg)
Client Y11
Client Z02
Client X04
Server Aslot 0slot 1slot 2slot 3slot 4
Server B
slot 1slot 0
![Page 219: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/219.jpg)
Client Y11
Client Z02
Client X04
Server B
slot 1slot 2
slot 0
2
Server Aslot 0slot 1slot 2slot 3slot 4
![Page 220: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/220.jpg)
Client Y11
Client Z02
Client X0
2 < 4
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2slot 3slot 4
![Page 221: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/221.jpg)
Client Y11
Client Z02
Client X02
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
![Page 222: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/222.jpg)
Client Y11
Client Z02
Client X02
Steady-state balance
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
![Page 223: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/223.jpg)
Client Y11
Client Z02
Client X02
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
Client Shutdown
![Page 224: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/224.jpg)
Client Y11
Client Z02
Client X02
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
![Page 225: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/225.jpg)
Client Z02
Client X02
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
![Page 226: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/226.jpg)
Client Z02
Client X02
1
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
![Page 227: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/227.jpg)
Client Z0
1 < 2
Client X02
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
![Page 228: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/228.jpg)
Client Z01
Client X02
Server B
slot 1slot 2
slot 0
Server Aslot 0
![Page 229: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/229.jpg)
Client Z01
Client X021
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1
![Page 230: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/230.jpg)
Client Z01
Client X0
1 < 2
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1
![Page 231: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/231.jpg)
Client Z01
Client X01
Server B
slot 1slot 0
Server Aslot 0slot 1
![Page 232: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/232.jpg)
Client Z01
Client X01
Steady-state balance
Server B
slot 1slot 0
Server Aslot 0slot 1
![Page 233: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/233.jpg)
Client Z01
Client X01
Server B
slot 1slot 0
Server Aslot 0slot 1
Client Rejoins
![Page 234: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/234.jpg)
Client Z01
Client X01
Client Y
Server B
slot 1slot 0
Server Aslot 0slot 1
![Page 235: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/235.jpg)
Client Z01
Client X01
Client Y2
2
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
![Page 236: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/236.jpg)
Client Z01
Client X01
Client Y22
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
![Page 237: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/237.jpg)
Client Z01
Client X01
Server B
slot 1slot 2
slot 0
Server Aslot 0slot 1slot 2
Client Y22
Steady-state balance
![Page 238: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/238.jpg)
Why does this Balance?
![Page 239: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/239.jpg)
Connections are like running water
seeking lower ground
![Page 240: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/240.jpg)
Slots
Servers
Connections
![Page 241: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/241.jpg)
Slots
Servers
Connections
![Page 242: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/242.jpg)
Slots
Servers
Connections
![Page 243: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/243.jpg)
Slots
Servers
Connections
![Page 244: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/244.jpg)
Slots
Servers
Connections
![Page 245: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/245.jpg)
Slots
Servers
Connections
![Page 246: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/246.jpg)
Slots
Servers
Connections
![Page 247: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/247.jpg)
Slots
Servers
Connections
![Page 248: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/248.jpg)
Slots
Servers
Connections
![Page 249: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/249.jpg)
Slots
Servers
Connections
![Page 250: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/250.jpg)
Slots
Servers
Connections
![Page 251: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/251.jpg)
Slots
Servers
Connections
![Page 252: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/252.jpg)
Slots
Servers
Roughly Equal Distribution
Connections
![Page 253: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/253.jpg)
Edge cases
![Page 254: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/254.jpg)
Client Z01
Client X01
Server B
slot 1slot 0
Server Aslot 0slot 1
![Page 255: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/255.jpg)
Balancedbut not ideal
Client Z01
Client X01
Server B
slot 1slot 0
Server Aslot 0slot 1
![Page 256: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/256.jpg)
Client Z
Client X01
Server Aslot 0slot 1
Server Bslot 0
![Page 257: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/257.jpg)
Client Z
Client X01
Server Aslot 0slot 1
Server Bslot 0 EMPTY
POOL!
![Page 258: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/258.jpg)
Client Z
Client X01
Server Aslot 0slot 1
Server Bslot 0 EMPTY
POOL!
Resilient✘
![Page 259: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/259.jpg)
Fix by adding entropy
![Page 260: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/260.jpg)
Fix by adding entropy
aka “Table Shaking”
![Page 261: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/261.jpg)
Servers regularly hang up on connections
Table Shaking
![Page 262: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/262.jpg)
Servers regularly hang up on connections
Clients expect failed connections
Table Shaking
![Page 263: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/263.jpg)
Servers regularly hang up on connections
Clients expect failed connections
Failures are retried on new connections
Table Shaking
![Page 264: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/264.jpg)
Servers regularly hang up on connections
Clients expect failed connections
Failures are retried on new connections
Bad configurations are less likely
Table Shaking
![Page 265: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/265.jpg)
Table Shakingturns this
Client Z01
Client X01
Server B
slot 1slot 0
Server Aslot 0slot 1
![Page 266: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/266.jpg)
Into this
Client Z01
Client X01
Server B
slot 1slot 0
Server Aslot 0slot 1
![Page 267: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/267.jpg)
Client Z
1
Client X0
Server Aslot 0slot 1
Server Bslot 0
![Page 268: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/268.jpg)
Client Z
1
Client X0
Server Aslot 0slot 1
Server Bslot 0
YAY!
YAY!
![Page 269: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/269.jpg)
Balancing Tricks:
Handicapping
![Page 270: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/270.jpg)
Handicapping isServer Self-quarantine
![Page 271: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/271.jpg)
Handicapping
Exploit slot number assignment
![Page 272: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/272.jpg)
Handicapping
Exploit slot number assignment
Unhealthy servers inflate slot numbers
![Page 273: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/273.jpg)
Handicapping
Exploit slot number assignment
Unhealthy servers inflate slot numbers
Clients naturally avoid these servers
![Page 274: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/274.jpg)
Slots
Servers
Connections
![Page 275: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/275.jpg)
Slots
Servers
Connections
Unhealthy
![Page 276: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/276.jpg)
Slots
Servers
Connections
![Page 277: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/277.jpg)
Slots
Servers
Connections
![Page 278: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/278.jpg)
Slots
Servers
Connections
![Page 279: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/279.jpg)
Slots
Servers
Connections
Unhealthy
![Page 280: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/280.jpg)
Slots
Servers
Connections
![Page 281: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/281.jpg)
Slots
Servers
Connections
![Page 282: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/282.jpg)
Slots
Servers
Connections
graceful degradation
![Page 283: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/283.jpg)
Is Boxcar good?
![Page 284: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/284.jpg)
Boxcar
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
Balanced
✓
✓
✓
?
?
![Page 285: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/285.jpg)
Clients are pessimistic
![Page 286: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/286.jpg)
Clients are pessimistic
Failure is expected
![Page 287: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/287.jpg)
Boxcar
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
Balanced
✓
✓
✓
?
✓
![Page 288: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/288.jpg)
![Page 289: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/289.jpg)
Balance ConnectionsNot Requests
![Page 290: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/290.jpg)
Balancing Review:
Naive Round Robin
![Page 291: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/291.jpg)
Slow Fast
![Page 292: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/292.jpg)
Slow Fast
![Page 293: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/293.jpg)
Slow Fast
![Page 294: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/294.jpg)
Slow Fast
![Page 295: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/295.jpg)
Slow Fast
![Page 296: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/296.jpg)
Slow Fast
![Page 297: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/297.jpg)
Slow Fast
![Page 298: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/298.jpg)
Slow Fast
![Page 299: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/299.jpg)
Slow Fast
![Page 300: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/300.jpg)
Slow Fast
![Page 301: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/301.jpg)
Slow Fast
![Page 302: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/302.jpg)
Can’t keep up
Slow Fast
![Page 303: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/303.jpg)
The problem was thatrequests (connections)
piled up
![Page 304: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/304.jpg)
Boxcar has a fixed number of connections
![Page 305: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/305.jpg)
Boxcar has a fixed number of connections
there’s nothing to pile up
![Page 306: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/306.jpg)
SlowServer
FastServer
![Page 307: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/307.jpg)
SlowServer
FastServer
Client
Slot 9Slot 7
7 9
![Page 308: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/308.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 309: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/309.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 310: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/310.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 311: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/311.jpg)
SlowServer
FastServer
0 1
Slot 9Slot 7
Client
7 9
![Page 312: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/312.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 313: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/313.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 314: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/314.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 315: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/315.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 316: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/316.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 317: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/317.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 318: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/318.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 319: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/319.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 320: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/320.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 321: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/321.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 322: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/322.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 323: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/323.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 324: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/324.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 325: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/325.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 326: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/326.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 327: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/327.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 328: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/328.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 329: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/329.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
![Page 330: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/330.jpg)
SlowServer
FastServer
Slot 9Slot 7
Client
7 9
2 requests 4 requests
![Page 331: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/331.jpg)
Slow servers handle fewer requests
![Page 332: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/332.jpg)
No overloaded servers
![Page 333: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/333.jpg)
All requests are serviced
![Page 334: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/334.jpg)
Load balancing is probabilistic
![Page 335: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/335.jpg)
Boxcar
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
Balanced
✓
✓
✓
✓
✓
![Page 336: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/336.jpg)
Boxcar
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable
Balanced
✓
✓
✓
✓
✓
![Page 337: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/337.jpg)
Good enough for Indeed
![Page 338: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/338.jpg)
Services well over a
BILLION requestsevery day
![Page 339: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/339.jpg)
Fundamental technology
![Page 340: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/340.jpg)
Powering over 20 different services
![Page 341: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/341.jpg)
In production since 2009
![Page 342: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/342.jpg)
![Page 343: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/343.jpg)
ServiceOriented
Architecture
![Page 344: [@IndeedEng] Boxcar: A self-balancing distributed services protocol](https://reader034.vdocument.in/reader034/viewer/2022052616/5562d5b6d8b42a49398b4f84/html5/thumbnails/344.jpg)
Q & A