webinar - dreamobjects/ceph case study
DESCRIPTION
Slides from our DreamObjects/Ceph webinar. This webinar was guest hosted by Kyle Bader of DreamHost.TRANSCRIPT
![Page 1: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/1.jpg)
DreamObjects Cloud Object Storage
Powered by Ceph
![Page 2: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/2.jpg)
About Me
• Senior systems engineer at dreamhost • Been with dreamhost since 2007 • Systems lead for DreamObjects • Excited about storage, distributed systems,
networking, linux internals, automation and monitoring
• I built my first ceph cluster in april, 2010
twitter: @mmgaggle
![Page 3: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/3.jpg)
Anatomy of DreamObjects
![Page 4: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/4.jpg)
Nuts and Bolts - Storage Nodes
• Intel nehalem or AMD lisbon • 32 GB of ram • 8-12 enterprise 3.5" sas disks • LSI megaraid 1078/2108 • 5x Gb nics • IPMI 2.0 BMC
![Page 5: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/5.jpg)
Nuts and Bolts - Support Nodes
• Intel westmere-ep • 8 GB ram • 3x Gb nics • 4x SAS drives • IPMI 2.0 BMC
ceph monitors, radosgw, proxies, object sync
![Page 6: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/6.jpg)
Tubes
• IPv6 native • Dual-stack application proxy / balancer • Application, rados, and cluster networks • One gig links, bonded • LLDP for discovery
![Page 7: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/7.jpg)
Traffic
• IPv6 and IPv4 api endpoints • IPv6 to radosgw nodes • Stud for ssl termination • HAproxy for load balancing • 10Gb backend and frontend
![Page 8: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/8.jpg)
Ceph Numbers
• Two plus petabytes of raw storage • Triplicate replication • 900+ OSDs • 7 nines durability • Single datacenter
![Page 9: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/9.jpg)
CRUSH Configuration
• Row, rack, node, and disk fault tolerant • Weighted, even distribution of data • Object striping • Online expansion and contraction
![Page 10: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/10.jpg)
OSD Configuration
• Single disk RAID0 • Two partitions per RAID0, data + journal • Writeback enabled • Disk cache disabled • Using XFS for filesystem
![Page 11: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/11.jpg)
Application Tuning
• No keepalive on apache • Increase apache thread pool • Increase radosgw thread pool • Raise radosgw ulimits
![Page 12: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/12.jpg)
Argonaut
• Lots of failure testing • Tested on large clusters (~900 osds) • Weight osds in/out of the cluster
![Page 13: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/13.jpg)
Bobtail
• Tunables for less aggressive remapping • CNAME support • OSD performance ++
![Page 14: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/14.jpg)
Control and Management
• chef • pdsh • runit/upstart • jenkins • sensu • collectd • graphite • logstash
![Page 15: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/15.jpg)
Chef
• Bootstrap cluster, new nodes • Packages and configuration • Creates, destroys and encrypts osds • Search node metadata, restful api • Roles mapped to pdsh genders • User and ssh key management • Loads more..
![Page 16: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/16.jpg)
pdsh
Parallel distributed shell • Ceph monitors are a natural control point • pdsh from monitor to other nodes • ssh-agent + ssh agent forwarding • Works when chef api doesn't
![Page 17: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/17.jpg)
runit/upstart
Keep ceph daemons running • ceph-osd • ceph-mon • radosgw
![Page 18: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/18.jpg)
Jenkins
• Continuous api functional tests • Monitor chef convergence • Generates alerts on failures
![Page 19: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/19.jpg)
Sensu
• Relatively new monitoring system • Open source • Written in ruby • State stored in Redis • Pub-sub with local agent • Handlers monitor state and generate alerts
![Page 20: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/20.jpg)
collectd
System metrics • cpu • memory • disks • entropy • interfaces
![Page 21: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/21.jpg)
graphite
• bucky - collectd, statsd, metricsd api • carbon cache • whisper for metric storage (sequential writes) • graphite webui
![Page 22: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/22.jpg)
logstash
• lumberjack agent to ship logs to logstash • logstash graphite output • logstash elasticsearch output • elasticsearch restful api • kibana webui sits on top of elasticsearch
![Page 23: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/23.jpg)
Future
• East coast region • Pool replication to another datacenter • Object expiration, root site support • Content delivery network
..not necessarily in that order ;)
![Page 24: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/24.jpg)
Thanks
Questions?
kyle.bader [at] dreamhost.com @mmgaggle
![Page 25: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/25.jpg)
Inktank’s Professional Services Consulting Services:
• Technical Overview • Infrastructure Assessment • Proof of Concept • Implementation Support • Performance Tuning
Support Subscriptions: • Pre-Production Support • Production Support
A full description of our services can be found at the following:
Consulting Services: http://www.inktank.com/consulting-services/
Support Subscriptions: http://www.inktank.com/support-services/
![Page 26: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/26.jpg)
Check out our upcoming and on demand webinars from Inktank
Upcoming: Advanced Features of Ceph Distributed Storage (delivered by Sage Weil, creator of Ceph)
February 12, 2013 10:00AM PT, 12:00PM CT, 1:00PM ET http://www.inktank.com/news-events/webinars/
On Demand:
Getting Started with Ceph http://www.inktank.com/news-events/webinars/ Intro to Ceph with OpenStack http://www.inktank.com/news-events/webinars/
![Page 27: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/27.jpg)
Contact Inktank
[email protected] and 1-855-INKTANK Don’t forget to follow us on: Twitter: https://twitter.com/inktank Facebook: http://www.facebook.com/inktank YouTube: http://www.youtube.com/inktankstorage
![Page 28: Webinar - DreamObjects/Ceph Case Study](https://reader034.vdocument.in/reader034/viewer/2022042601/54b762b24a79598a618b45d6/html5/thumbnails/28.jpg)
THANK YOU!