ebay object storage at scale - snia · 2019. 12. 21. · 22. challenge 5: solutions swift/keystone...

1

eBay Object Storage at Scale

Gilbert BouzeidLead Product Manager Data Storage

2

ebayToday

$22.6Bof GMV

14KGlobalEmployees

61% InternationalRevenue

182M Active Buyers

Q2 2019 numbers

3

Object Storage

4

What is Object Storage?

Object Storage is a data storage architecture that manages data as objects, as opposed to other storage architectures like file systems and block storage.

5

The Object Storage Journey at eBay

1. A need was identified

Seller HubeBay Site Objects Analytics / ElasticSearch Index Backup

6

The Object Storage Journey at eBay

2. Object Storage Solutions:

● 2007: EPS, eBay Picture Service● 2008: EMC Storage ATMOS● 2012: zStore● 2014: Swift

7

Why Swift Object Storage?

● Scalable and Reliable● Pure software based solution with REST API● Elastic● Replication● Multiple policies● Stateless Service● Runs on heterogeneous commodity hardware

8

Swift Object Storage High-Level Deployment Diagram

9

Swift Object Stats

2MContainers

5 PBStorage

2BObjects

30Gbps

25KRequests/

sec

900Accounts

10

Swift Object StorageUse Cases

Analytics DataDeployment PackageGeneric Images/Files storageLoggingPCI / PIIBackup / ArchivalData Movement (between Zones)TransactionLogging

SellerHub

ES Index

Backup

PCI / PII

Image Search

Zone Transfer

Deployment

ReturnShipping

Label

11

Swift Object Storage and Kubernetes

● ECR (eBay Container Registry) backed by Swift

● Leveraging the existing Swift Cluster from Kubernetes

● Swift Running on Kubernetes (Bring your own Swift)

● Creating Swift Accounts Seamless to Kubernetes end users

12

Object Storage:Challenges and Solutions

13

Challenge 1

● One Global Swift Object Storage (distributed) handling a Huge number of Use Cases!

● Large number of concurrent access● Infrastructure Limitations

Swift ProxySwift Nodes

LB

😐😐

14

Challenge 1: Solutions

● Customer Education/Best Practices● Split Swift Services● Efficient Use of Hardware● SLB

Swift Object

Swift Proxy

Swift Object

Swift Proxy

HTTPS TLS 1.2 Request

ProxyAccountContainer

SLB

Object role

SLB

CPU, Memory, SSD

Dense storage disks

15

Challenge 2

● Free offering● Burst number of Requests● Uncontrolled Usage● No Quota

16


● QoS is essential● Requests rate limit● Account quota

17

Challenge 3

● 100’s of Parallel Downloads from Same Account!

● Same Object too!● Limited Disk IOPS on the Object Nodes

18


● Object Chunking● Caching Solution

Swift Object

Swift Proxy

Swift Object

LB

VarnishCache

Swift Proxy

L7

SLB

19

Challenge 4

● Geo Availability● High Availability● Disaster Recovery (DR) Capability

20


● Multi-Regional Keystone

Swift ProxySwift Proxy

Swift ProxySwift Object

SLB

GTM / DNS





CacheCache CacheCacheCacheCache

DC1 DC2 DC3

SLB SLB

21

Challenge 5

● Multi-Tenancy● Zone Isolation

22


● Swift/Keystone Handshake● New Proxy Middleware

KeystoneToken (scoped with zone and IP)


Zone Isolation

IP Restriction

Modified token

ACL restriction

Middlewarepipeline


Project scoped with zone

LBIP Forwarding

23

Challenge 6

● Data Archival / Backup● Longer Retention● Low Cost● High Durability● Large Amount of Data

24


● Erasure Coding● Storage Policies● 175% less storage overhead● High PUTs, Low GETs

25

Object Storage:Alerting and Monitoring

26

Alerting and Monitoring

● Elasticsearch Kibana (ELK) & Prometheus● Synthetic Tests● Zabbix● Metrics

27


● Stats

28


● Metrics

29


● Performance

30

Thank You

Q & A Session

eBay Object Storage at ScaleSlide Number 2Object StorageWhat is Object Storage?The Object Storage Journey at eBayThe Object Storage Journey at eBayWhy Swift Object Storage?Swift Object Storage High-Level Deployment DiagramSwift Object StatsSwift Object StorageUse CasesSwift Object Storage and KubernetesObject Storage:Challenges and SolutionsChallenge 1Challenge 1: SolutionsChallenge 2Challenge 2: SolutionsChallenge 3Challenge 3: SolutionsChallenge 4Challenge 4: SolutionsChallenge 5Challenge 5: SolutionsChallenge 6Challenge 6: SolutionsObject Storage:Alerting and MonitoringAlerting and MonitoringAlerting and MonitoringAlerting and MonitoringAlerting and MonitoringThank You

Q & A Session

ebay object storage at scale - snia · 2019. 12. 21. · 22. challenge 5: solutions swift/keystone...

Documents