missing the forest for the trees hpca2020 - sigarch · 1 missing the forest for the trees:...
TRANSCRIPT
![Page 1: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/1.jpg)
1
Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers
Daniel Richins1,2, Dharmisha Doshi2, Matthew Blackmore2, Aswathy Thulaseedharan Nair2, Neha Pathapati2, Ankit Patel2, Brainard Daguman2, Daniel Dobrijalowski2, Ramesh Illikkal2, Kevin Long2, David Zimmerman2, Vijay Janapa Reddi1,3
1The University of Texas at Austin 2Intel 3Harvard University
International Symposium on High Performance Computer Architecture 25 February 2020
![Page 2: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/2.jpg)
Missing the Forest for the Trees The AI Tax
2
AI
![Page 3: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/3.jpg)
Missing the Forest for the Trees The AI Tax
2
AI
![Page 4: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/4.jpg)
Missing the Forest for the Trees The AI Tax
2
AI
The forest is the AI tax
![Page 5: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/5.jpg)
Missing the Forest for the Trees The AI Tax
3
The AI tax includes all the compute and infrastructure in an AI application that is necessary to enable the AI to execute but that isn’t AI itself.
![Page 6: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/6.jpg)
Missing the Forest for the Trees The AI Tax
3
The AI tax includes all the compute and infrastructure in an AI application that is necessary to enable the AI to execute but that isn’t AI itself.
Artificial IntelligenceTime
Excit
emen
t
![Page 7: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/7.jpg)
Missing the Forest for the Trees The AI Tax
3
The AI tax includes all the compute and infrastructure in an AI application that is necessary to enable the AI to execute but that isn’t AI itself.
Artificial IntelligenceTime
Excit
emen
t
Pre Post
![Page 8: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/8.jpg)
1. AI Tax - A Case Study a. Definition b. Video Analytics c. Analysis
2. AI Acceleration - Anticipating Future Bottlenecks a. Emulation Technique b. Results c. What's Breaking?
3. Optimization - Better Performance at Lower TCO a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
Outline
4
![Page 9: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/9.jpg)
1. AI Tax - A Case Study a. Definition b. Video Analytics c. Analysis
2. AI Acceleration - Anticipating Future Bottlenecks a. Emulation Technique b. Results c. What's Breaking?
3. Optimization - Better Performance at Lower TCO a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
Outline
4
![Page 10: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/10.jpg)
1. AI Tax - A Case Study a. Definition b. Video Analytics c. Analysis
2. AI Acceleration - Anticipating Future Bottlenecks a. Emulation Technique b. Results c. What's Breaking?
3. Optimization - Better Performance at Lower TCO a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
Outline
4
![Page 11: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/11.jpg)
1. AI Tax - A Case Study a. Definition b. Video Analytics c. Analysis
2. AI Acceleration - Anticipating Future Bottlenecks a. Emulation Technique b. Results c. What's Breaking?
3. Optimization - Better Performance at Lower TCO a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
Outline
4
![Page 12: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/12.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
AI Tax
5
![Page 13: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/13.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
AI Tax - Definition
6
![Page 14: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/14.jpg)
Missing the Forest for the Trees
7
![Page 15: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/15.jpg)
Missing the Forest for the Trees
7
![Page 16: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/16.jpg)
Missing the Forest for the Trees
7
AI Tax
![Page 17: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/17.jpg)
Missing the Forest for the Trees
7
AI TaxSupporting compute, storage, network, software infrastructure, etc. together constitute the AI Tax.
![Page 18: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/18.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
AI Tax - Video Analytics
8
![Page 19: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/19.jpg)
Face Recognition Algorithm
9
![Page 20: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/20.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
![Page 21: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/21.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
Video Stream
![Page 22: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/22.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
Video Stream
![Page 23: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/23.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
Video Stream Ingestion
![Page 24: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/24.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
FrameVideo
Stream Ingestion
![Page 25: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/25.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
FrameFace
DetectionVideo
Stream Ingestion
![Page 26: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/26.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
Face Thumbnail
FrameFace
DetectionVideo
Stream Ingestion
![Page 27: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/27.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
Face Thumbnail
FrameFace
Detection
Feature Extraction
Video Stream Ingestion
![Page 28: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/28.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
Vector
Face Thumbnail
FrameFace
Detection
Feature Extraction
Video Stream Ingestion
![Page 29: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/29.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
Vector
Face Thumbnail
FrameFace
Detection
Feature ExtractionClassification
Video Stream Ingestion
![Page 30: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/30.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
Vector
Face Thumbnail
FrameFace
Detection
Feature ExtractionClassification
Video Stream Ingestion
![Page 31: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/31.jpg)
Face Recognition Algorithm
9
Face Recognition is Google’s FaceNet as a data center application.
Vector
Face Thumbnail
Frame
Identity
Face Detection
Feature ExtractionClassification
Video Stream Ingestion
![Page 32: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/32.jpg)
Face Recognition Algorithm
9
User Application
Face Recognition is Google’s FaceNet as a data center application.
Vector
Face Thumbnail
Frame
Identity
Face Detection
Feature ExtractionClassification
Video Stream Ingestion
![Page 33: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/33.jpg)
Face Recognition Algorithm
9
User Application
Face Recognition is Google’s FaceNet as a data center application.
Vector
Face Thumbnail
Frame
Identity
Face Detection
Face Detection
Feature ExtractionFeature
ExtractionClassificationClassification
Video Stream Ingestion
AI Compute
![Page 34: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/34.jpg)
Face Recognition Data Center Deployment
10
Classification
Ingestion Face Detection
Video Stream
Identity Feature Extraction
User Application
AI Compute
![Page 35: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/35.jpg)
Face Recognition Data Center Deployment
10
Classification
Ingestion Face Detection
Feature Extraction
User Application
AI Compute
![Page 36: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/36.jpg)
Ingest/Detect
Face Recognition Data Center Deployment
10
User
Identification
Application
AI Compute
![Page 37: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/37.jpg)
Ingest/Detect
Face Recognition Data Center Deployment
10
User
Identification
Application
Producers
ConsumersAI Compute
![Page 38: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/38.jpg)
IdentificationIdentification
Ingest/DetectIngest/DetectIngest/Detect
Face Recognition Data Center Deployment
10
User
Identification
Application
Producers
ConsumersAI Compute
![Page 39: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/39.jpg)
IdentificationIdentification
Ingest/DetectIngest/DetectIngest/Detect
Face Recognition Data Center Deployment
10
User
Identification
Application
Brokers
Producers
ConsumersAI Compute
![Page 40: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/40.jpg)
Experimental Setup Hardware
11
![Page 41: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/41.jpg)
Experimental Setup Hardware
11
2x Intel Xeon Platinum 8176 2x 28 cores, 2.10 GHz, 2x 38.5 MB LLC
384 GB DDR4 SDRAM
1x Intel SSD P4510 2.85 GB/s read 1.10 GB/s write
100 Gbps Ethernet
![Page 42: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/42.jpg)
Experimental Setup Hardware
11
![Page 43: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/43.jpg)
Experimental Setup Face Recognition
12
We allocate one core per container. Hence, a server runs 56 containers.
![Page 44: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/44.jpg)
Identification
Experimental Setup Face Recognition
12
We allocate one core per container. Hence, a server runs 56 containers.
Ingest/Detect x56
x56
![Page 45: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/45.jpg)
Experimental Setup Face Recognition
12
We allocate one core per container. Hence, a server runs 56 containers.
Ingest/Detect Identification
840 total producers 1680 total consumers
![Page 46: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/46.jpg)
Experimental Setup Face Recognition
12
We allocate one core per container. Hence, a server runs 56 containers.
Brokers get their own server. This grants them full network and storage bandwidth.
Ingest/Detect Identification
840 total producers 1680 total consumers
![Page 47: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/47.jpg)
Experimental Setup Face Recognition
12
We allocate one core per container. Hence, a server runs 56 containers.
Broker
Brokers get their own server. This grants them full network and storage bandwidth.
Ingest/Detect Identification
840 total producers 1680 total consumers
![Page 48: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/48.jpg)
Experimental Setup Face Recognition
12
We allocate one core per container. Hence, a server runs 56 containers.
Brokers get their own server. This grants them full network and storage bandwidth.
Ingest/Detect Identification
840 total producers 1680 total consumers
Broker
3 brokers
![Page 49: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/49.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
AI Tax - Analysis
13
![Page 50: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/50.jpg)
Application Progress Event Logging
14
while True: frame = queue.get()
producer.send(faces) faces = detect_faces(frame)
![Page 51: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/51.jpg)
Application Progress Event Logging
14
while True: frame = queue.get() start = time.time()
end = time.time() producer.send(faces)
faces = detect_faces(frame)
![Page 52: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/52.jpg)
Application Progress Event Logging
14
while True: frame = queue.get() start = time.time()
end = time.time()
size = sys.getsizeof(faces) log = { 'start': start, 'end': end, 'size': size } logger.info(log)
producer.send(faces)
faces = detect_faces(frame)
![Page 53: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/53.jpg)
Application Progress Event Logging
14
while True: frame = queue.get() start = time.time()
end = time.time()
size = sys.getsizeof(faces) log = { 'start': start, 'end': end, 'size': size } logger.info(log)
producer.send(faces)
faces = detect_faces(frame)
Logging is designed to raise the level of abstraction. We view application progress from the data center perspective.
![Page 54: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/54.jpg)
Face Detection Latency
15
![Page 55: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/55.jpg)
Face Detection Latency
15
Latency Breakdown
Ingestion DetectionBrokers Identification
![Page 56: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/56.jpg)
Face Detection Latency
15
Latency Breakdown
Ingestion DetectionBrokers Identification
5.4%
AI Tax
![Page 57: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/57.jpg)
Face Detection Latency
15
Latency Breakdown
Ingestion DetectionBrokers Identification
21.3%
5.4%
AI Tax
AI Compute
![Page 58: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/58.jpg)
Face Detection Latency
15
Latency Breakdown
Ingestion DetectionBrokers Identification
35.9%
21.3%
5.4%
AI Tax
AI Tax
AI Compute
![Page 59: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/59.jpg)
Face Detection Latency
15
Latency Breakdown
Ingestion DetectionBrokers Identification
37.4%
35.9%
21.3%
5.4%
AI Tax
AI Tax
AI Compute
AI Compute
![Page 60: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/60.jpg)
Process Breakdowns
16
![Page 61: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/61.jpg)
Ingestion
100%
AI AI Tax
Process Breakdowns
16
![Page 62: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/62.jpg)
Face Detection
58%42%
AI AI Tax
Ingestion
100%
AI AI Tax
Process Breakdowns
16
![Page 63: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/63.jpg)
Face Detection
58%42%
AI AI Tax
Identification
12%
88%
AI AI Tax
Ingestion
100%
AI AI Tax
Process Breakdowns
16
![Page 64: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/64.jpg)
Face Detection
58%42%
AI AI Tax
Identification
12%
88%
AI AI Tax
Ingestion
100%
AI AI Tax
Process Breakdowns
16
Pre- and post-processing are heavily utilized within stages.
![Page 65: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/65.jpg)
AI Tax
17
Time
Excit
emen
t
Pre PostAI
![Page 66: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/66.jpg)
AI Tax
18
Time
Excit
emen
t
Pre PostAIAIAI
![Page 67: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/67.jpg)
AI Tax
19
Time
Excit
emen
t
Pre PostAIAIAI
![Page 68: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/68.jpg)
AI Tax
19
Time
Excit
emen
t
Pre PostAIAIAI
![Page 69: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/69.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
AI Acceleration
20
![Page 70: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/70.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
AI Acceleration - Emulation Technique
21
![Page 71: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/71.jpg)
Acceleration Emulation
22
Classification
Brokers
Ingest/Detect
Identification
![Page 72: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/72.jpg)
Acceleration Emulation
22
Classification
Brokers
Dial an Accelerator Speed
![Page 73: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/73.jpg)
Acceleration Emulation
23
while True:
faces = detect_faces(frame) start = time.time()
end = time.time()
sys.getsizeof(faces)
frame = queue.get()
producer.send(faces)
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size': size
size =
![Page 74: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/74.jpg)
Acceleration Emulation
23
while True:
faces = detect_faces(frame) start = time.time()
end = time.time()
sys.getsizeof(faces)
frame = queue.get()
producer.send(faces)
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size': size
size =
![Page 75: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/75.jpg)
Acceleration Emulation
23
while True:
start = time.time()
end = time.time()
sys.getsizeof(faces)
frame = queue.get()
producer.send(
time.sleep(avg_time)
faces)
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size': size
size =
![Page 76: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/76.jpg)
Acceleration Emulation
23
while True:
start = time.time()
end = time.time()
sys.getsizeof(faces)
frame = queue.get()
producer.send(
time.sleep(avg_time)
faces)
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size': size
size =
![Page 77: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/77.jpg)
Acceleration Emulation
23
while True:
start = time.time()
end = time.time()
frame = queue.get()
producer.send(
time.sleep(avg_time)
os.urandom(avg_size))
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size': size
avg_size size =
![Page 78: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/78.jpg)
Acceleration Emulation
23
while True:
start = time.time()
end = time.time()
frame = queue.get()
producer.send(
time.sleep(avg_time)
os.urandom(avg_size))
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size': size
avg_size size =
![Page 79: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/79.jpg)
Acceleration Emulation
24
while True:
start = time.time()
end = time.time()
frame = queue.get()
producer.send(
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size':
os.urandom(avg_size))
time.sleep(avg_time)
size = avg_size
size
![Page 80: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/80.jpg)
Acceleration Emulation
24
while True:
start = time.time()
end = time.time()
frame = queue.get()
producer.send(
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size':
os.urandom(avg_size))
time.sleep(avg_time)
size = avg_size
size
![Page 81: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/81.jpg)
Acceleration Emulation
24
while True:
start = time.time()
end = time.time()
frame = queue.get()
producer.send(
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size':
os.urandom(avg_size))
time.sleep(avg_time/speedup)
size = avg_size
size
![Page 82: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/82.jpg)
Acceleration Emulation
24
while True:
start = time.time()
end = time.time()
frame = queue.get()
producer.send(
log = { 'start': start, 'end': end,
} logger.info(log)
'size': 'size':
os.urandom(avg_size))
time.sleep(avg_time/speedup)
size = avg_size
size
With faster processing, we feed frames into the system faster to maximize throughput
![Page 83: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/83.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
AI Acceleration - Results
25
![Page 84: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/84.jpg)
Accelerated AI: Reduced Latency and Increased Throughput
26
![Page 85: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/85.jpg)
Accelerated AI: Reduced Latency and Increased Throughput
26
Late
ncy
(ms)
0
100
200
300
400
500
600
700
1x 2x 4x 6x 8x
Ingest/Detect Broker Identify
Fram
es p
er S
econ
d (x
1000
)
0
10
20
30
40
50
60
70Throughput
![Page 86: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/86.jpg)
Accelerated AI: Reduced Latency and Increased Throughput
26
Late
ncy
(ms)
0
100
200
300
400
500
600
700
1x 2x 4x 6x 8x
Ingest/Detect Broker Identify
Fram
es p
er S
econ
d (x
1000
)
0
10
20
30
40
50
60
70Throughput
![Page 87: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/87.jpg)
Accelerated AI: Reduced Latency and Increased Throughput
26
Late
ncy
(ms)
0
100
200
300
400
500
600
700
1x 2x 4x 6x 8x
Ingest/Detect Broker Identify
Fram
es p
er S
econ
d (x
1000
)
0
10
20
30
40
50
60
70Throughput
![Page 88: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/88.jpg)
Accelerated AI: Reduced Latency and Increased Throughput
26
Late
ncy
(ms)
0
100
200
300
400
500
600
700
1x 2x 4x 6x 8x
Ingest/Detect Broker Identify
Fram
es p
er S
econ
d (x
1000
)
0
10
20
30
40
50
60
70Throughput
![Page 89: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/89.jpg)
Accelerated AI: Reduced Latency and Increased Throughput
26
Late
ncy
(ms)
0
100
200
300
400
500
600
700
1x 2x 4x 6x 8x
Ingest/Detect Broker Identify
Fram
es p
er S
econ
d (x
1000
)
0
10
20
30
40
50
60
70Throughput
At 8x speedup, the average latency goes to infinity. The longer the experiment runs, the greater the latency.
![Page 90: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/90.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
AI Acceleration - What’s Breaking?
27
![Page 91: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/91.jpg)
Three Big Systems
28
![Page 92: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/92.jpg)
Three Big Systems
28
Compute
![Page 93: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/93.jpg)
Three Big Systems
28
Compute Network
![Page 94: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/94.jpg)
Three Big Systems
28
Compute Network Storage
![Page 95: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/95.jpg)
Three Big Systems
28
Compute Network Storage
![Page 96: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/96.jpg)
Three Big Systems
28
Compute Network Storage
?
![Page 97: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/97.jpg)
Three Big Systems
28
Compute Network Storage
? ?
![Page 98: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/98.jpg)
Explaining the Bottleneck
29
![Page 99: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/99.jpg)
Network Utilization
0%
1%
2%
3%
4%
5%
6%
7%
1x 2x 4x 6x 8x
Broker Read Broker Write
Explaining the Bottleneck
29
![Page 100: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/100.jpg)
Network Utilization
0%
1%
2%
3%
4%
5%
6%
7%
1x 2x 4x 6x 8x
Broker Read Broker Write
Explaining the Bottleneck
29
![Page 101: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/101.jpg)
Network Utilization
0%
1%
2%
3%
4%
5%
6%
7%
1x 2x 4x 6x 8x
Broker Read Broker Write
Explaining the Bottleneck
29
![Page 102: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/102.jpg)
Storage Utilization
0%
10%
20%
30%
40%
50%
60%
70%
1x 2x 4x 6x 8x
Network Utilization
0%
1%
2%
3%
4%
5%
6%
7%
1x 2x 4x 6x 8x
Broker Read Broker Write
Explaining the Bottleneck
29
![Page 103: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/103.jpg)
Storage Utilization
0%
10%
20%
30%
40%
50%
60%
70%
1x 2x 4x 6x 8x
Network Utilization
0%
1%
2%
3%
4%
5%
6%
7%
1x 2x 4x 6x 8x
Broker Read Broker Write
Explaining the Bottleneck
29
![Page 104: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/104.jpg)
Storage Utilization
0%
10%
20%
30%
40%
50%
60%
70%
1x 2x 4x 6x 8x
Network Utilization
0%
1%
2%
3%
4%
5%
6%
7%
1x 2x 4x 6x 8x
Broker Read Broker Write
Explaining the Bottleneck
29
![Page 105: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/105.jpg)
Storage Utilization
0%
10%
20%
30%
40%
50%
60%
70%
1x 2x 4x 6x 8x
Network Utilization
0%
1%
2%
3%
4%
5%
6%
7%
1x 2x 4x 6x 8x
Broker Read Broker Write
Explaining the Bottleneck
29
As storage utilization approaches the limits of the devices, it becomes the limiting factor to performance.
![Page 106: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/106.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
Optimization
30
![Page 107: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/107.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
Optimization - Fixing the Bottleneck
31
![Page 108: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/108.jpg)
Fixing the Bottleneck
32
![Page 109: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/109.jpg)
Fixing the Bottleneck
32
Additional Drives
Late
ncy
(ms)
0
50
100
150
200
1 Driv
e
2 Driv
es
3 Driv
es
4 Driv
es
8x 12x 16x 24x 32x
![Page 110: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/110.jpg)
Fixing the Bottleneck
32
Additional Drives
Late
ncy
(ms)
0
50
100
150
200
1 Driv
e
2 Driv
es
3 Driv
es
4 Driv
es
8x 12x 16x 24x 32x
Additional Brokers
Late
ncy
(ms)
0
50
100
150
200
3 Brok
ers
4 Brok
ers
6 Brok
ers
8 Brok
ers
![Page 111: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/111.jpg)
Fixing the Bottleneck
32
Additional Drives
Late
ncy
(ms)
0
50
100
150
200
1 Driv
e
2 Driv
es
3 Driv
es
4 Driv
es
8x 12x 16x 24x 32x
Additional Brokers
Late
ncy
(ms)
0
50
100
150
200
3 Brok
ers
4 Brok
ers
6 Brok
ers
8 Brok
ers
![Page 112: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/112.jpg)
Fixing the Bottleneck
32
Additional Drives
Late
ncy
(ms)
0
50
100
150
200
1 Driv
e
2 Driv
es
3 Driv
es
4 Driv
es
8x 12x 16x 24x 32x
Additional Brokers
Late
ncy
(ms)
0
50
100
150
200
3 Brok
ers
4 Brok
ers
6 Brok
ers
8 Brok
ers
![Page 113: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/113.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
Optimization - Edge Data Centers
33
![Page 114: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/114.jpg)
Advantages of an Edge Data Center
34
![Page 115: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/115.jpg)
Advantages of an Edge Data Center
34
Smaller corporations are finding edge data centers more economical than the cloud.
![Page 116: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/116.jpg)
Advantages of an Edge Data Center
34
Smaller corporations are finding edge data centers more economical than the cloud.
Edge data centers offer lower latency by serving local users.
![Page 117: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/117.jpg)
Advantages of an Edge Data Center
34
Smaller corporations are finding edge data centers more economical than the cloud.
Edge data centers offer lower latency by serving local users.
Edge data centers can be built to target a specific application domain.
https://www.networkworld.com/article/2926448/7-key-criteria-for-defining-edge-data-centers.html
http://blog.cushwake.com/americas/life-on-the-edge-the-new-normal-for-data-centers.html
https://www.vxchnge.com/blog/what-is-an-edge-data-center
Sources
![Page 118: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/118.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
Optimization - Two Designs
35
![Page 119: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/119.jpg)
Edge Data Center Node Allocation
36
![Page 120: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/120.jpg)
Edge Data Center Node Allocation
36
We need to allocate enough brokers to handle 32x speedup.
![Page 121: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/121.jpg)
Edge Data Center Node Allocation
36
We need to allocate enough brokers to handle 32x speedup.
Consumers30
Producers15
Brokers8
![Page 122: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/122.jpg)
Edge Data Center Node Allocation
36
Consumers578
Producers289
Brokers157
We need to allocate enough brokers to handle 32x speedup.
Consumers30
Producers15
Brokers8
![Page 123: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/123.jpg)
Targeted Data Center Design Optimizing for Total Cost of Ownership
37
![Page 124: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/124.jpg)
Targeted Data Center Design Optimizing for Total Cost of Ownership
37
Homogeneous Heterogeneous
![Page 125: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/125.jpg)
Targeted Data Center Design Optimizing for Total Cost of Ownership
37
Homogeneous Heterogeneous
56 core 1 Drive 100 GbE
![Page 126: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/126.jpg)
Targeted Data Center Design Optimizing for Total Cost of Ownership
37
Homogeneous Heterogeneous
56 core 1 Drive 100 GbE
Compute Broker
![Page 127: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/127.jpg)
Targeted Data Center Design Optimizing for Total Cost of Ownership
37
Homogeneous Heterogeneous
56 core 1 Drive 100 GbE
Compute Broker
160 switches
![Page 128: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/128.jpg)
Targeted Data Center Design Optimizing for Total Cost of Ownership
37
Homogeneous Heterogeneous
56 core 1 Drive 100 GbE
Compute Broker
160 switches 28+14 switches
![Page 129: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/129.jpg)
Targeted Data Center Design Optimizing for Total Cost of Ownership
37
Homogeneous Heterogeneous
56 core 1 Drive 100 GbE
Compute Broker
TCO
160 switches 28+14 switches
![Page 130: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/130.jpg)
Heterogeneous Edge Data Center
38
![Page 131: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/131.jpg)
Heterogeneous Edge Data Center
38
Compute Node
![Page 132: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/132.jpg)
Heterogeneous Edge Data Center
38
Compute Node Broker Node
![Page 133: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/133.jpg)
Heterogeneous Edge Data Center
38
Compute Node Broker Node
![Page 134: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/134.jpg)
Heterogeneous Edge Data Center
38
Compute Node Broker Node
![Page 135: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/135.jpg)
Heterogeneous Edge Data Center
38
Compute Node Broker Node
![Page 136: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/136.jpg)
Heterogeneous Edge Data Center
38
Compute Node Broker Node
10 GbE
![Page 137: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/137.jpg)
Heterogeneous Edge Data Center
38
Compute Node Broker Node
10 GbE
![Page 138: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/138.jpg)
Heterogeneous Edge Data Center
38
Compute Node Broker Node
10 GbE
![Page 139: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/139.jpg)
Heterogeneous Edge Data Center
38
Compute Node Broker Node
10 GbE
![Page 140: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/140.jpg)
Heterogeneous Edge Data Center
38
Compute Node Broker Node
10 GbE50 GbE
![Page 141: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/141.jpg)
Heterogeneous Edge Data Center Networking
39
![Page 142: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/142.jpg)
Heterogeneous Edge Data Center Networking
39
100 Gbps Switch 100 Gbps Switch
100 Gbps 100 Gbps 100 Gbps 100 Gbps
![Page 143: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/143.jpg)
Heterogeneous Edge Data Center Networking
39
100 Gbps Switch 100 Gbps Switch
100 Gbps 100 Gbps 100 Gbps 100 Gbps
Broker Node
50 Gbps
![Page 144: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/144.jpg)
Heterogeneous Edge Data Center Networking
39
100 Gbps Switch 100 Gbps Switch
100 Gbps 100 Gbps 100 Gbps 100 Gbps
Broker Node
50 Gbps
![Page 145: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/145.jpg)
Heterogeneous Edge Data Center Networking
39
100 Gbps Switch 100 Gbps Switch
100 Gbps 100 Gbps 100 Gbps 100 Gbps
40 Gbps 40 Gbps
Broker Node
50 Gbps
![Page 146: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/146.jpg)
Heterogeneous Edge Data Center Networking
39
100 Gbps Switch 100 Gbps Switch
100 Gbps 100 Gbps 100 Gbps 100 Gbps
40 Gbps 40 Gbps
Broker NodeCompute Node
50 Gbps
10 Gbps
![Page 147: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/147.jpg)
Heterogeneous Edge Data Center Networking
39
100 Gbps Switch 100 Gbps Switch
100 Gbps 100 Gbps 100 Gbps 100 Gbps
40 Gbps 40 Gbps 40 Gbps40 Gbps
Broker NodeCompute Node
50 Gbps
10 Gbps
![Page 148: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/148.jpg)
Comparing Total Cost of Ownership
40
We assume a three-year amortization of costs.
![Page 149: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/149.jpg)
Comparing Total Cost of Ownership
40
We assume a three-year amortization of costs.
0%
20%
40%
60%
80%
100%
Compute Networking Power Overall
Homogeneous Heterogeneous
![Page 150: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/150.jpg)
Comparing Total Cost of Ownership
40
We assume a three-year amortization of costs.
0%
20%
40%
60%
80%
100%
Compute Networking Power Overall
Homogeneous Heterogeneous
89%100%
![Page 151: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/151.jpg)
Comparing Total Cost of Ownership
40
We assume a three-year amortization of costs.
0%
20%
40%
60%
80%
100%
Compute Networking Power Overall
Homogeneous Heterogeneous
23%
89%100%100%
![Page 152: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/152.jpg)
Comparing Total Cost of Ownership
40
We assume a three-year amortization of costs.
0%
20%
40%
60%
80%
100%
Compute Networking Power Overall
Homogeneous Heterogeneous
100%
23%
89%100%100%100%
![Page 153: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/153.jpg)
Comparing Total Cost of Ownership
40
We assume a three-year amortization of costs.
0%
20%
40%
60%
80%
100%
Compute Networking Power Overall
Homogeneous Heterogeneous
84%100%
23%
89%100%100%100%100%
![Page 154: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/154.jpg)
Comparing Total Cost of Ownership
40
We assume a three-year amortization of costs.
0%
20%
40%
60%
80%
100%
Compute Networking Power Overall
Homogeneous Heterogeneous
84%100%
23%
89%100%100%100%100%
$12.9 million $10.8 million
![Page 155: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/155.jpg)
Comparing Total Cost of Ownership
40
We assume a three-year amortization of costs.
0%
20%
40%
60%
80%
100%
Compute Networking Power Overall
Homogeneous Heterogeneous
84%100%
23%
89%100%100%100%100%
$12.9 million $10.8 million
The targeted, heterogeneous data center incurs 16% lower total cost of ownership
![Page 156: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/156.jpg)
Comparing Total Cost of Ownership
40
We assume a three-year amortization of costs.
0%
20%
40%
60%
80%
100%
Compute Networking Power Overall
Homogeneous Heterogeneous
84%100%
23%
89%100%100%100%100%
$12.9 million $10.8 million
The targeted, heterogeneous data center incurs 16% lower total cost of ownership
Designing the data center to match the needs of the application, we created a better data center at lower cost
![Page 157: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/157.jpg)
1. AI Tax a. Definition b. Video Analytics c. Analysis
2. AI Acceleration a. Emulation Technique b. Results c. What's Breaking?
3. Optimization a. Fixing the Bottleneck b. Edge Data Centers c. Two Designs
4. Conclusion
Conclusion
41
![Page 158: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/158.jpg)
Calls to Action
42
![Page 159: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/159.jpg)
Calls to Action
• To fully understand AI applications, we must consider the overhead of the AI tax in end-to-end performance.
42
![Page 160: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/160.jpg)
Calls to Action
• To fully understand AI applications, we must consider the overhead of the AI tax in end-to-end performance.
• As we accelerate AI, we must consider new bottlenecks that manifest as AI tax.
42
![Page 161: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/161.jpg)
Calls to Action
• To fully understand AI applications, we must consider the overhead of the AI tax in end-to-end performance.
• As we accelerate AI, we must consider new bottlenecks that manifest as AI tax.
• We cannot limit our view of AI to microarchitectural considerations. We need data center-level optimizations to address data center-level bottlenecks.
42
![Page 162: Missing the Forest for the Trees HPCA2020 - SIGARCH · 1 Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers Daniel Richins1,2, Dharmisha](https://reader034.vdocument.in/reader034/viewer/2022043023/5f3e7183dfadaf4cc5070185/html5/thumbnails/162.jpg)
43
Thank You