art van scheppingen, senior support engineer become a ... · become a mongodb dba how to automate,...
Post on 22-May-2020
8 Views
Preview:
TRANSCRIPT
Confidential
Become a MongoDB DBA
How to Automate, Monitor and Manage your MongoDB Servers
Art van Scheppingen, Senior Support Engineer
Confidential
Agenda
☐ MongoDB for DBAs☐ Deploying MongoDB☐ Automation☐ Monitoring☐ Managing☐ Orchestration systems
Confidential
MongoDB for DBAs
Confidential
MongoDB: why should I install it?
☐ Not everyone is happy with (My)SQL☐ Some storage needs differ from what (My)SQL can offer☐ Complex problems require different storage systems
☐ Alternative data storage☐ Polyglot persistence
Confidential
Polyglot Persistence?
Does a carpenter only use one tool?
Confidential
Polyglot Persistence?
Carpenters use many tools
Confidential
Monoglot Persistence
Confidential
Polyglot Persistence
Confidential
How different is MongoDB from MySQL?
☐ Document store (JSON)☐ Flexible schemas☐ Out of the box read scaling (replicaSets)☐ Out of the box write scaling (sharding)☐ Enthusiastic development community
Confidential
Eventual consistency
Confidential
How similar is MongoDB to MySQL?
☐ You still need to do the database basics☐ Deployment☐ Management☐ Monitoring☐ Backups☐ Scaling/Sharding
☐ It is less different than you think!
Confidential
Deploying MongoDB
Confidential
Install MongoDB
☐ Install MongoDB ☐ Download packages☐ Set up repositories
☐ Watch log file☐ /var/log/mongodb/mongod.log
[initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 },
Confidential
Configure MongoDB
☐ MongoDB Configuration file is in YAML# network interfacesnet: port: 27017# bindIp: 127.0.0.1 # Listen to local interface only, comment to listen on all interfaces.
☐ Restart mongod
Confidential
Install a second node
☐ Install packages☐ Change configuration
Confidential
Install a third node
☐ Install packages☐ Change configuration☐ This is getting boring…☐ And what if we deploy a sharded cluster with five shards?
Confidential
Automation
Confidential
Deployment Automation
☐ Deployment automation tools☐ Chef☐ Puppet☐ Ansible☐ Saltstack
Confidential
Chef
☐ Use mongodb3 cookbook☐ https://supermarket.chef.io/cookbooks/mongodb3☐ Others are not well maintained
☐ Handles configuration☐ Supports AWS☐ Supports replication and sharding☐ Supports mms_automation_agent
Confidential
Puppet
☐ Puppetlabs MongoDB module☐ https://github.com/puppetlabs/puppetlabs-mongodb☐ Still in beta: may change in the future☐ Well documented
☐ Handles☐ Installation☐ Configuration☐ Replication & sharding☐ Crude user management (admin user)
Confidential
Ansible
☐ Ansible MongoDB example☐ https://github.com/ansible/ansible-examples/tree/master/mongodb
☐ Installs a sharded MongoDB cluster☐ Expects RHEL/Centos 6 hosts☐ Covers only one use case
☐ You can alter a playbook, right?
Confidential
Ansible (cont.)
☐ Stouts.mongo Ansible role☐ https://github.com/Stouts/Stouts.mongodb
☐ Handles☐ Installation☐ Configuration☐ Replication
☐ Supports mms_automation_agent
Confidential
Saltstack
☐ MongoDB Formula☐ https://github.com/saltstack-formulas/mongodb-formula
☐ Handles☐ Installation☐ Configuration☐ Replication & Sharding
Confidential
Monitoring & Trending
Confidential
Confidential
Monitoring vs Trending
☐ Monitoring☐ Keeps an eye on your systems☐ Will alert if a threshold is met
☐ Trending☐ Insight into the system internals☐ Trends can warn you before anything has happened yet☐ Keep historical state/data
Confidential
Monitoring: Availability
☐ Do more than just opening a connection☐ Measure true status of nodes and cluster☐ Test read/write☐ Open essential databases and collections☐ Keep an eye on the replication lag☐ Increase oplog size?
☐ Check the full topology
Confidential
Monitoring Tools
☐ Open Source☐ Nagios☐ Zabbix
☐ Subscription based☐ MongoDB Cloud Manager☐ VividCortex☐ ClusterControl
Confidential
Nagios
☐ Nagios-MongoDB☐ https://github.com/mzupan/nagios-plugin-mongodb/☐ Performs some very important checks☐ Replication lag☐ Lock time percentage☐ Index miss ratio
Confidential
Zabbix
☐ MongoDB Zabbix monitoring plugin☐ https://github.com/nightw/mikoomi-zabbix-mongodb-monitoring☐ All the necessary metrics and more☐ Entries in oplog
☐ Pre-canned triggers
Confidential
Trending: Why do we need trends?
☐ Trending☐ Plot trends of key (performance) metrics☐ Find problems before they arise☐ Pre-emptive problem management
☐ Trending tools☐ Granularity of sampling☐ More datapoints = better
Confidential
Trending: What metrics to store?
☐ Resource monitoring☐ CPU☐ Memory☐ IO capacity☐ Diskspace
☐ Replication monitoring☐ Replication status☐ Replication lag☐ Oplog size and usage
Confidential
Trending: Solutions
☐ Trending tools☐ Statsd/Grafana☐ Cacti☐ Zabbix
☐ Subscription based☐ MongoDB Cloud Manager☐ VividCortex☐ ClusterControl
Confidential
Cacti
☐ Percona MongoDB Monitoring Templates☐ https://www.percona.com/doc/percona-monitoring-plugins/1.1
/cacti/mongodb-templates.html
Confidential
Cacti
Confidential
Managing
Confidential
Types of backups
☐ Logical backups☐ Dump of your data
☐ Physical backups☐ File(system) copy of your data
Confidential
Logical backups
☐ Mongodump☐ MongoDB Backup☐ Mongob☐ Hotbackup (only available in Percona Server and in beta)
Confidential
Logical backups: mongodump
☐ Mongodump☐ BSON dump of the data☐ OEM tool☐ Works great but needs some wrapping
Confidential
Logical backups: MongoDB Backup
☐ MongoDB Backup☐ https://www.npmjs.com/package/mongodb-backup☐ Nodejs backup solution☐ CLI and API☐ Can stream backups
Confidential
Logical backups: Mongob
☐ Mongob☐ https://github.com/cmpitg/mongob☐ Python based CLI tool☐ MongoDB instance or bz2 target☐ Can copy data between collections☐ Incremental backups☐ Rate limiting
Confidential
Physical backups: Filesystem snapshots
☐ Filesystem snapshots☐ LVM☐ ZFS☐ XFS (xfs_freeze)☐ EBS
Confidential
Physical backups: Strata
☐ MongoRocks Strata☐ https://github.com/facebookgo/rocks-strata☐ Backs up on file level☐ Supports incremental backups☐ Queryable backups
Confidential
Percona MongoDB Consistent Backup
☐ Currently it is impossible to make a consistent backup of a sharded cluster☐ You can only make backups of individual hosts☐ Not all backups finish at the same time
Confidential
Noteworthy: Percona MongoDB Consistent Backup
Confidential
Percona MongoDB Consistent Backup
☐ Percona is working on a MongoDB consistent backup tool☐ Make a consistent backup of a sharded cluster☐ Snapshots individual hosts/replicasets in a sharded cluster☐ Oplog tailing☐ Point in time recovery possible☐ Still in development/beta
☐ Get it here:☐ https://github.com/Percona-Lab/mongodb_consistent_backup☐ Requires Python 2.7
Confidential
Why do schemas need changes?
☐ Application development☐ New features to existing modules☐ New modules☐ Data management and archiving
(partitions)☐ Schema optimization☐ Performance optimization
Confidential
Schema changes for MySQL
Confidential
Schema changes for MongoDB
Confidential
Schema Validation
☐ Meteor2 collection☐ https://github.com/aldeed/meteor-collection2☐ Ensures data gets written according to schema☐ Automatic: both client and server validate schema☐ Reactive: returns useful error messages
Confidential
Schema Validation
☐ Mongoose☐ http://mongoosejs.com/index.html☐ Node.js middleware for schema modelling and validation
Confidential
Schema Analyzer
☐ MongoDB Schema☐ https://github.com/mongodb-js/mongodb-schema☐ Schema analyzer
Confidential
Admin UIs
☐ Mongoclient☐ http://www.mongoclient.com/☐ Management / Backups
☐ PHPMoAdmin☐ http://www.phpmoadmin.com/
☐ Mongo Express☐ https://www.npmjs.com/package/mongo-express☐ Overview and data browser
☐ Robomongo☐ https://robomongo.org/download☐ Admin / data browser
Confidential
Orchestration Systems
Confidential
Orchestration systems: Mongoctl
☐ Mongoctl☐ https://github.com/mongolab/mongoctl☐ Python based CLI tool☐ Manages nodes, clusters and shards☐ Deployment☐ Configuration management☐ Scaling
Confidential
Orchestration systems: Percona Monitoring & Management
☐ PMM☐ https://www.percona.com/doc/percona-monitoring-and-management/
☐ Open Source Monitoring & Management framework☐ Can deploy, manage and monitor MySQL & MongoDB☐ Uses Prometheus and Grafana☐ Still in Beta
Confidential
Orchestration Systems: ClusterControl
☐ ClusterControl☐ http://www.severalnines.com☐ Deploy Mongo shards & replicasets☐ Monitor and trend☐ Manage configuration and backups☐ Scale
☐ Community edition
Confidential
Q & A
Confidential
Additional Resources
☐ Blog series: Become a MongoDB DBA☐ http://severalnines.com/blog-categories/mongodb
☐ Webinar series: Become a MongoDB DBA☐ http://severalnines.com/upcoming-webinars
☐ Visit our website for more resources!☐ http://www.severalnines.com
top related