![Page 1: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/1.jpg)
Ceph, Open Source, and the Path to Ubiquity in
StorageAACS – 2014 Apr 02
![Page 2: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/2.jpg)
Copyright © 2014 by Inktank | Private and Confidential
WHO?
2
Patrick McGarry
Director, Community - Inktank
/. -> ALU -> P4 -> Inktank
scuttlemonkey
FOSS 4 LYFE!
![Page 3: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/3.jpg)
Copyright © 2014 by Inktank | Private and Confidential
AGENDA
3
INDUSTRY MUSINGS
INTRO TO CEPH
ARCHITECTURE
COMMUNITY
INKTANK CEPH ENTERPRISE
![Page 4: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/4.jpg)
Copyright © 2014 by Inktank | Private and Confidential
THE FORECAST
By 2020over 15 ZB of data will be stored.1.5 ZB are stored today.
4
![Page 5: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/5.jpg)
Copyright © 2014 by Inktank | Private and Confidential
THE PROBLEM
Existing systems don’t scale
Increasing cost and complexity
Need to invest in new platforms ahead of time
2010 2020
IT Storage Budget
Growth of data
5
![Page 6: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/6.jpg)
Copyright © 2014 by Inktank | Private and Confidential
THE SOLUTION
PAST: SCALE UP
FUTURE: SCALE OUT
6
![Page 7: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/7.jpg)
INTRO TO CEPH
![Page 8: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/8.jpg)
Copyright © 2013 by Inktank | Private and Confidential
HISTORICAL TIMELINE
8
RHEL-OSP & RHEV Support FEB 2014
MAY 2012Launch of Inktank
OpenStack Integration 2011
2010Mainline Linux Kernel
Open Source 2006
2004 Project Starts at UCSC
Production Ready Ceph SEPT 2012
2012CloudStack Integration
OCT 2013Inktank Ceph Enterprise Launch
Xen Integration 2013
![Page 9: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/9.jpg)
Copyright © 2014 by Inktank | Private and Confidential
ARCHITECTURAL COMPONENTS
9
RGWA web services
gateway for object storage, compatible
with S3 and Swift
LIBRADOSA library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOSA software-based, reliable, autonomous, distributed object store comprised ofself-healing, self-managing, intelligent storage nodes and lightweight monitors
RBDA reliable, fully-distributed block device with cloud
platform integration
CEPHFSA distributed file
system with POSIX semantics and
scale-out metadata management
APP HOST/VM CLIENT
![Page 10: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/10.jpg)
Copyright © 2014 by Inktank | Private and Confidential
CEPH UNIFIED STORAGE
FILE SYSTEM
BLOCK STORAGE
OBJECT STORAGE
Keystone
Geo-ReplicationNative API
10
Multi-tenant
S3 & Swift
OpenStack
Linux Kernel
iSCSI
Clones
Snapshots
CIFS/NFS
HDFS
Distributed Metadata
Linux Kernel
POSIX
![Page 11: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/11.jpg)
Copyright © 2014 by Inktank | Private and Confidential
TRADITIONAL STORAGE VS. CEPH
Single Purpose
TRADITIONALENTERPRISE STORAGE
Multi-Purpose, Unified
Hardware Distributed Software
Single Vendor Lock-in Open
Hard Scale Limit Exabyte Scale
11
![Page 12: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/12.jpg)
Copyright © 2014 by Inktank | Private and Confidential
CEPH WITH OPENSTACK
OPEN STACK
KEYSTONE API
SWIFT API CINDER APIGLAN
CE API
NOVAAPI
CEPH STORAGE CLUSTER(RADOS)
CEPH OBJECT GATEWAY
(RGW)
CEPH BLOCK DEVICE(RBD)
HYPERVISOR
(Qemu/KVM)
12
![Page 13: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/13.jpg)
Copyright © 2014 by Inktank | Private and Confidential
WEB APPLICATION STORAGE
WEB APPLICATION
APP SERVER
APP SERVER
APP SERVER
CEPH STORAGE CLUSTER(RADOS)
CEPH OBJECT GATEWAY
(RGW)
CEPH OBJECT GATEWAY(RGW)
13
APP SERVER
S3/Swift S3/Swift S3/Swift S3/Swift
![Page 14: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/14.jpg)
Copyright © 2014 by Inktank | Private and Confidential
MULTI-SITE OBJECT STORAGE
WEB APPLICATIONAPP
SERVER
CEPH OBJECT GATEWAY
(RGW)
14
CEPH STORAGE CLUSTER
(US-EAST)
WEB APPLICATIONAPP
SERVER
CEPH OBJECT GATEWAY
(RGW)
CEPH STORAGE CLUSTER
(EU-WEST)
![Page 15: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/15.jpg)
Copyright © 2014 by Inktank | Private and Confidential
ARCHITECTURE
![Page 16: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/16.jpg)
Copyright © 2014 by Inktank | Private and Confidential
ARCHITECTURAL COMPONENTS
16
RGWA web services
gateway for object storage, compatible
with S3 and Swift
LIBRADOSA library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOSA software-based, reliable, autonomous, distributed object store comprised ofself-healing, self-managing, intelligent storage nodes and lightweight monitors
RBDA reliable, fully-distributed block device with cloud
platform integration
CEPHFSA distributed file
system with POSIX semantics and
scale-out metadata management
APP HOST/VM CLIENT
![Page 17: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/17.jpg)
17
DISK
FS
DISK DISK
OSD
DISK DISK
OSD OSD OSD OSD
FS FS FSFSbtrfsxfsext4
MMM
17
![Page 18: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/18.jpg)
18
M
M
M
HUMAN
18
![Page 19: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/19.jpg)
19
Monitors:Maintain cluster mapProvide consensus for distributed decision-making
Must have an odd numberThese do not serve stored objects to clients
MOSDs:•One per disk (recommended)•At least three in a cluster•Serve stored objects to clients•Intelligently peer to perform replication tasks•Supports object classes
19
![Page 20: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/20.jpg)
20
M
M
M
CLIENT
0110
datametadata
20
![Page 21: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/21.jpg)
21
Metadata ServerManages metadata for a POSIX-compliant shared filesystem
Directory hierarchyFile metadata (owner, timestamps, mode, etc.)
Stores metadata in RADOSDoes not serve file data to clients
Only required for shared filesystem
21
![Page 22: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/22.jpg)
CRUSH Pseudo-random placement algorithm Ensures even distribution Repeatable, deterministic Rule-based configuration Replica count Infrastructure topology Weighting
CRUSH
22
![Page 23: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/23.jpg)
23
10 10 01 01 10 10 01 11 01 10
10 10 01 01 10 10 01 11 01 10
hash(object name) % num pg
CRUSH(pg, cluster state, rule set)
23
![Page 24: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/24.jpg)
24
CLIENT
??
24
![Page 25: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/25.jpg)
25
25
![Page 26: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/26.jpg)
26
26
![Page 27: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/27.jpg)
27
CLIENT
??
27
![Page 28: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/28.jpg)
![Page 29: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/29.jpg)
Ceph Developer Summit
29
• Recent: “Giant”
• March 04-05• wiki.ceph.com • Virtual
(irc, hangout, pad, blueprint, youtube)
• 2 days (soon to be 3?)
• Discuss all work
• Recruit for your projects!
![Page 30: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/30.jpg)
New Contribute Page
30
• http://ceph.com/community/Contribute
• Source tree• Issues• Share
experiences• Standups• One-stop shop
![Page 31: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/31.jpg)
New Ceph Wiki
31
![Page 32: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/32.jpg)
Accepted as a mentoring organization 8 mentors from Inktank & Community http://ceph.com/gsoc2014/ Student applications being evaluated Hope to turn this into academic outreach
Google Summer of Code 2014
32
![Page 33: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/33.jpg)
Ceph Days
33
• inktank.com/cephdays
• Recently:London, Frankfurt, NYC, Santa Clara
• Aggressive program
• Upcoming:Sunnyvale, Austin, Boston, Kuala Lumpur
![Page 34: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/34.jpg)
Meetups
34
• Community organized
• World wide• Wiki• Ceph-
community• Goodies
available• Logistical
support• Drinkup to
tradeshow
![Page 35: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/35.jpg)
We haven’t forgotten! Looking for potential founding members Especially important to keep the IP clean
Ceph Foundation
35
![Page 36: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/36.jpg)
Coordinated Efforts
36
• Always need help
• CentOS SIG• OCP• Xen• Hadoop• OpenStack• CloudStack• Ganetti• Many more!
![Page 37: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/37.jpg)
Copyright © 2014 by Inktank | Private and Confidential
http://metrics.ceph.com
37
![Page 38: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/38.jpg)
THE PRODUCT
![Page 39: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/39.jpg)
Copyright © 2014 by Inktank | Private and Confidential39
INKTANK CEPH ENTERPRISEWHAT’S INSIDE?
Ceph Object and Ceph Block
Calamari
Enterprise Plugins (2014)
Support Services
Subscription-based
Priced on capacity
Single price for all protocols
![Page 40: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/40.jpg)
![Page 41: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/41.jpg)
Copyright © 2014 by Inktank | Private and Confidential
Ceph Inktank Ceph Enterprise
Open-Source Ceph ✔ ✔
Object Storage ✔ ✔
Block Storage ✔ ✔
File System ✔
Management API ✔ ✔
Management GUI ✔
Hyper-V Support Q3 2014
SNMP Support Q3 2014
24x7 Support ✔
Bug Prioritization ✔
FEATURE COMPARISON
41
![Page 42: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/42.jpg)
Copyright © 2014 by Inktank | Private and Confidential
ROADMAPINKTANK CEPH ENTERPRISE
42
1.2 2.0
CEPH
CALAMARI
PLUGINS
Erasure Coding
RHEL7 Support
Cache Tiering
User Quotas
RADOS Management
Analytics Hosted/SaaS
SNMP, Hyper-V
Ceph 0.77 Firefly Ceph 0.87 “H-Release”
April 2014 September 2014
CephFS
Ubuntu 14.04 Support
VMware
HDFS Support
iSCSI
Intelligent Objects
QoS
2015
![Page 43: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/43.jpg)
Copyright © 2014 by Inktank | Private and Confidential
Emperor Giant
H
I
J
Inktank Ceph Enterprise v1.1 (Dumpling LTS until May 2015)
Inktank Ceph Enterprise v1.2 (Firefly LTS until November 2015)
RELEASE SCHEDULE
43
2013 2014 2015
Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2
FireflyDumpling
![Page 44: Ceph, Open Source, and the Path to Ubiquity in Storage - AACS Meetup 2014](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6d8614a7959b7618b45bc/html5/thumbnails/44.jpg)
Copyright © 2014 by Inktank | Private and Confidential
QUESTIONS?
44