openstack trove in production at ebay - trove day 2014
DESCRIPTION
Presentation at OpenStack Trove Day 2014 by Mat Lowery, Software Engineer and Auston McReynolds, Software Engineer from eBayTRANSCRIPT
Session Name-click to change
August 19, 2014
Auston McReynolds, Software Engineer, eBayMat Lowery, Software Engineer, eBay
Trove in Production at eBay
OpenStack Trove Day
The eBay Inc. Portfolio
PLATFORM
8/19/14 tesora.com 2
Private Cloud at eBay
4
eBay Private Cloud: Like a Public Cloud
8/19/14 tesora.com
• One cloud on shared infra• Like a public cloud for all eBay Inc properties
• Multi-tenant, isolated• Different tenants with different needs
• Frictionless, on-demand access• Reduce time to market• Enable innovation
tesora.com 5
Architecture: Region vs. AZ
8/19/14
Dashboard behind global load balancer VIP
Region A
Swift
Regi
on
Svcs
AZ
Serv
icesKeystone Nova CinderNeutron
Glance Dashboard MySQL Misc
Region B
Swift
Regi
on
Svcs
AZ
Serv
icesKeystone Nova CinderNeutron
Glance Dashboard MySQL Misc
Region C
Swift
Regi
on
Svcs
AZ
Serv
icesKeystone Nova CinderNeutron
Glance Dashboard MySQL Misc
tesora.com 6
Customizations
8/19/14
• VPCs• Enforce obligations, restrictions, and capabilities• Associated with Keystone projects• OpenStack resources mapped to VPC (via metadata)• Image metadata• Host aggregate metadata
• Enforcement via custom middleware or scheduler
• Keystone• LDAP• Auto-tenancy
tesora.com 7
Customizations
8/19/14
• Horizon• Most use the custom Horizon dashboard vs. API• Connect To (uses image metadata)• Copy datastore root password• Disable UI functionality based on AZ capabilities or
Datastore• Multi-AZ tenant switcher glues the clouds together
8
Packaging and Deployment
8/19/14 tesora.com
• Deployment unit: virtualenv tarball• virtualenv --no-site-packages –relocatable• Simplifies rollback• Not perfect: doesn’t handle OS libs
• Rollout automation by Puppet, Foreman• Auditing of changes• Host overrides for testing in isolation
tesora.com 9
Monitoring
8/19/14
• Metrics• Periodic gathering based on service DBs• Share with end users• Alerts to engineers
• SLAs• Ensure being met via sampling; simulates cloud user• Nova boot• Swift upload and download
• Capture logs on failures• Alerts to engineers
• Graphite for graphing and Zabbix for alerting• Logstash/Elasticsearch/Kibana stack
Trove at eBay
tesora.com 11
Trove at eBay – The Beginning
8/19/14
• Mid-2013• 1 developer• Stack on Folsom• Not integrated, still Red Dwarf• MySQL only• Single region• No backups
tesora.com 12
Trove at eBay – Now
8/19/14
• Team (all ATCs)• Icehouse• Multiple Datastores• MySQL 5.5, 5.6• MongoDB 2.4• Cassandra 2.0• Couchbase 2.2• Redis 2.8
tesora.com 13
Trove at eBay – Now - Continued
8/19/14
• Multi-region + HA, with limited VPC availabilty
• Backups for MySQL, MongoDB, & Couchbase• Contributions to Trove:• Couchbase Impl + Backups• Multi-Datastore Horizon• Clusters (in-flight)• + many more (incl. 400+ code-reviews)
tesora.com 14
Deployment Details
8/19/14
• HA• Stable* release cuts• Puppet + Foreman for Control Plane• Fabric for Guests
• Shared infra, sans RabbitMQ• RabbitMQ HA Mirrored
tesora.com 15
Deployment Details
8/19/14
• KVM• Ubuntu• Network: Bridged or Overlays• Cinder Volumes• Venv’d Vanilla Guest Agent• Diskimage-Builder• w/ reference & custom elements• Image per Datastore
tesora.com 16
Deployment Details
8/19/14
• Designate (soon)• Zabbix• w/o Heat
tesora.com 17
Challenges
8/19/14
• With shared infra:• Nova quotas vs. Trove quotas• Horizon (compute duplicity, how to connect, etc.)• True ownership of spawned assets• Failure; who’s fault?• Scheduling
tesora.com 18
Challenges
8/19/14
• With Trove:• Upgrades/Rollouts• RabbitMQ• Feature support across Datastores• Multi-region Datastore deployments
tesora.com 19
What’s Next
8/19/14
• Clusters!• Advanced scheduling• Additional monitoring• Performance tuning• More management extensions (UI + API)• API-driven Datastore differentiation
eBay Trove Team is Hiring
http://linkd.in/1sSVbHv or [email protected]
Questions