mongodb performance tuning and load testing, nosql now! 2013 conference presentation
DESCRIPTION
MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentationTRANSCRIPT
Enteros, Inc.
MongoDB Performance Tuning
Load Testing
NoSQL Now! 2013 Conference San Jose
[email protected] Ron Warshawsky
August 23, 2013
2
MongoDB Overview
Agenda
Overview performance tuning
Overview load testing
3
MongoDB Areas Of Performance Tuning
OS tuning Storage tuning
Database tuning Load Testing
4
MongoDB Areas Of Performance Tuning
• OS Tuning • Follow ulimits per mongo manual
• http://docs.mongodb.org/manual/reference/ulimit/
• Follow production notes per mongo manual • http://docs.mongodb.org/manual/administration/production-notes/ • No atime on fs • No hugepages • Use NTT to sync time between the nodes
• Try “tuned”
• Best when used in staging/load testing environment under
realistic load
• CPU, Network and Storage tuning
5
MongoDB Areas Of Performance Tuning
• Storage Tuning • RAID 10
• Ext4 or XFS
• Delay sharding with better I/O
• SSD + FlashCache – shutterfly tested
• https://github.com/facebook/flashcache/
• Provisioned IOPS on AWS
6
MongoDB Areas Of Performance Tuning
• Storage Tuning • Ext4 + external journal on SSD – good I/O results
• Make sure to test • journal_async_commit is used • http://www.raid6.com.au/posts/fs_ext4_external_journal/
7
MongoDB Areas Of Performance Tuning
Softlayer study/ benchmarking “MongoDB Performance Analysis: Bare Metal v. Virtual” http://blog.softlayer.com/2012/mongodb-performance-analysis-bare-metal-v-
virtual/
Highlights:
• When a working data set is smaller than available memory, query performance increases.
• The number of clients performing queries has an impact on query performance because more data is being actively cached at a rapid rate.
• The addition of a separate Journal Mount volume significantly improves performance. Because the Small (SM) engineered server does not include a secondary mount for Journals, whenever MongoDB began to journal, the disk I/O associated with journaling was disruptive to the query and update operations performed on the Data Mount.
• The best deployments in terms of operations per second, stability and control were the configurations with a RAID10 SSD Data Mount and a RAID1 SSD Journal Mount. These configurations are available in both Medium and Large offerings, and I’d highly recommend them.
8
MongoDB Areas Of Performance Tuning
OS Performance monitoring and analysis tools:
• A) Linux utilities: iostat, vmstat, mpstat, sar, free -tm
• B) Open source monitoring / performance analysis systems: Nagios, Cacti
• http://tag1consulting.com/blog/mongodb-cacti-graphs • https://github.com/mzupan/nagios-plugin-mongodb C) Oracle OS Watcher – Free tool, performance collection and chart
generator, easy to operate, very reliable, covers CPU, processes, top processes, I/O, network, etc
OSWatcher Black Box Analyzer User Guide (Doc ID 461053.1)
9
MongoDB Areas Of Performance Tuning
Database Tuning
• Database configuration • Preallocate space (noprealloc = false)
• journalCommitInterval – range of 2 to 300, default 100. Can be
adjusted to for better performance
• Option to disable services: 1. HTTP Interface using nohttpinterface = true. 2. Scripting Engine using noscripting = true (server-side java
script). 3. REST service using rest = false.
10
MongoDB Areas Of Performance Tuning
Database Tuning
• Database monitoring tools • Mongostat – based on “eBay” “freecon” utility.
• Use “–discover” option for members of replicaSet / shared
cluster
• Displays critical performance metrics: • Flushes/Faults/locked db/idx miss/qr/qw
• Collect 24x7 and save for analysis
11
MongoDB Areas Of Performance Tuning
Database Tuning
• Database monitoring tools • Mongotop
• Per collection I/O time tracking
• Per database lock tracking (--lock option)
12
MongoDB Areas Of Performance Tuning
Database network traffic analysis
• Database network monitoring tools • Mongosniff – sniffing MongoDb requests
• Can capture invalid BSON requests from a network (--objcheck option)
• Wireshark – support for MongoDB protocol (not BSON), great options for network performance and latency analysis
13
MongoDB Areas Of Performance Tuning
Database Tuning
• Collections / Fragmentation
Collections in a MongoDB database can become fragmented. This can be a particularly serious problem if data usage patterns are relatively unstructured. In the long run, this will result in databases taking up more space on disk and in RAM to hold the same amount of data, it will make many database operations noticeably slower, and it can reduce overall query capacity significantly.
compact collection: db.runCommand ( { compact: '<collection>', paddingBytes: 100 } ) Automate compaction: http://blog.parse.com/2013/03/26/always-be-compacting/ ./mongo_compact.rb -d userdata1,userdata2,userdata3
14
MongoDB Areas Of Performance Tuning
Database Tuning
Profiler
db.setProfilingLevel(level, slowms) db.setProfilingLevel(1) – collects slow requests Output: system.profile.op – operation system.profile.ns – namespace system.profile.ntoreturn – documents returned system.profile.nscanned – documents scanned … http://docs.mongodb.org/manual/reference/database-profiler/ Profiler deep dive https://speakerdeck.com/kgorman/mongodb-profiler-deep-dive
15
MongoDB Areas Of Performance Tuning
Database Tuning
• Indexes
• Explain db.collection.find({…}).explain() Watch for scans
• Indexes + Profiler = DEX - suggest what indexesare needed
https://github.com/jwilder/mongodb-tools http://architects.dzone.com/articles/mongodb-performance-tuning-
dex
16
MongoDB Areas Of Performance Tuning
Database Monitoring and performance analysis UI
a) MongoDB monitoring service (MMS)
b) Open Source monitoring software http://docs.mongodb.org/manual/administration/monitoring/
c) Enteros High Load Capture for MongoDB
17
MongoDB Areas Of Performance Tuning
• Replication / Replica set tuning Watch for replication lag Problem: a) Weak secondary b) Write bursts c) Index build d) Secondary locked for backup e) Secondary offline Measure lag: db.printSlaveReplicationInfo() db.printReplicationInfo() Get time difference MMS provides replication lag chart and alerts
18
MongoDB Areas Of Performance Tuning
• Sharding Tuning • Shard size
• Migration of chunks
• Failed migration of chunks
19
MongoDB Load Testing
Load testing
1) Why load testing?
• Validate upgrades
• Validate fixes
• Validate hardware and platform changes (to / from cloud, etc)
• Validate multiples of production load
20
MongoDB Load Testing
2) Options for Load testing
A) Benchmark Load Testing • Mongo-Perf - Create standard TPCC-like load on MongoDB • https://github.com/mongodb/mongo-perf
B) Disk I/O load testing
• Mongoperf
• Iozone http://www.iozone.org/
21
MongoDB Load Testing
C) Real traffic Load Testing
Parse PCAP files and generate production-like load http://docs.mongodb.org/meta-driver/latest/legacy/mongodb-wire-
protocol/
Create custom tool
Enteros Load2Test (pre-GA)
Understand business requirements and it’s impact on production system design
Design for performance, scalability and availability
Implement monitoring and performance analysis across layers
Implement reliable production-like load testing
MongoDB Summary
Enteros, Inc http://www.enteros.com/
Enteros is an innovative software company specializing in Performance Management and Load Testing Software for Production Databases - RDBMS and NOSQL/Big Data Enteros solutions enable IT professionals to identify and remediate performance problems in business-critical databases with unprecedented speed, accuracy and scope. Ron Warshawsky; [email protected] 408-207-8408
MongoDB Summary