scalebase webinar: strategies for scaling mysql
DESCRIPTION
Matt Aslett of 451 Research joins ScaleBase to discuss: scaling-out your MySQL DB, new high availability strategies, centrally managing a distributed MySQL environment.TRANSCRIPT
© 2013 by The 451 Group. All rights reserved
Strategies for scaling MySQL
Matthew Aslett, 451 Research
Doron Levari & Paul Campaniello, ScaleBase
© 2013 by The 451 Group. All rights reserved
Agenda
1. 451 Research – The MySQL SPRAIN
2. ScaleBase - Scale-Out your MySQL Database
3. Customer Case Studies
4. Q & A
(please type questions directly into the GoToWebinar side panel)
© 2013 by The 451 Group. All rights reserved
Company Overview
One company with 3 operating divisions
Syndicated research, advisory, professional services, datacenter certification, and events
Global focus
200+ staff 1,300+ client organizations:
enterprises, vendors, service providers, and investment firms
Organic and growth through acquisition
© 2013 by The 451 Group. All rights reserved
Matthew Aslett
• Research Manager, Data Management and Analytics
www.twitter.com/maslett
Responsible for data management and analytics research agenda
Focus on operational and analytic databases, including NoSQL, NewSQL, and Hadoop
With 451 Research since 2007
© 2013 by The 451 Group. All rights reserved
In a nutshell
MySQL became the default database of choice for Web applications
Success breeds complexity: • Greater scalability demands • Predictable performance problems
New requirements:
• Proliferation of cloud • Geo-distributed data
Multiple strategies for dealing with scalability issues
• Each with their plusses and minuses
The emergence of new vendors aimed at masking the complexity
© 2013 by The 451 Group. All rights reserved
Previously on ScaleBase/451 webinars…
The traditional relational database has been stretched beyond its normal capacity by the needs of high-volume, highly distributed or highly complex applications.
There are workarounds – such as DIY sharding – but manual,
homegrown efforts can result in database administrators being stretched beyond their normal capacity in terms of managing complexity.
Scalability Performance Relaxed consistency Increased willingness to look towards Agility emerging alternatives Intricacy Necessity
© 2013 by The 451 Group. All rights reserved
Focus on MySQL
MySQL filled a gap in the market:
• Low cost and freely available with source code
• Better suited to emerging application workloads
MySQL benefits • Lightweight architecture
• Fast read capabilities
• Ease of use
MySQL primarily adopted for:
• in-house-developed applications
• single-function applications
• customer-facing Web applications
• development-and-test environments
© 2013 by The 451 Group. All rights reserved
Early adoption drivers: 2008 survey
© 2013 by The 451 Group. All rights reserved
Rapid increase in adoption: 2012 database survey
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
Survey conducted: Jan/Feb 2012 Sample: 205
© 2013 by The 451 Group. All rights reserved
Scalability, and other challenges
As usage of MySQL has grown, so has the usage of applications that depend on MySQL
This has highlighted a number of challenges
• Scalability of master-slave architecture
• Performance
• Predictability of performance at scale
• Manageability of increasing database/app sprawl
External factors driving greater complexity:
• Distributed computing architectures
• Proliferation of cloud, and elasticity requirements
• Geo-distributed application requirements
© 2013 by The 451 Group. All rights reserved
The road to SPRAIN: approaches to scaling MySQL
database
users
Example:
Photo sharing website
Started with single MySQL database
© 2013 by The 451 Group. All rights reserved
The road to SPRAIN: approaches to scaling MySQL
database
users users users
Example:
Photo sharing website
Started with single MySQL database
Growth through acquisition and customer adoption
© 2013 by The 451 Group. All rights reserved
The road to SPRAIN: approaches to scaling MySQL
master
slave slave
users users users
Example:
Photo sharing website
Started with single MySQL database
Growth through acquisition and customer adoption
Expanded its MySQL footprint every two years
© 2013 by The 451 Group. All rights reserved
The road to SPRAIN: approaches to scaling MySQL
master
slave slave
users users users
Example:
Photo sharing website
Started with single MySQL database
Growth through acquisition and customer adoption
Expanded its MySQL footprint every two years
Larger master
• Pros: Add more CPUs, disk and memory to cope with increased data
• Cons: Doesn’t necessarily help with performance, temporary fix
© 2013 by The 451 Group. All rights reserved
The road to SPRAIN: approaches to scaling MySQL
slave slave
master
slave slave
users users users
Example:
Photo sharing website
Started with single MySQL database
Growth through acquisition and customer adoption
Expanded its MySQL footprint every two years
More slaves
• Pros: Good for scaling our read queries
• Cons: Asynchronous replication can lead to write slave lag
By 2009 had 16 slaves and was beginning to hit the wall
© 2013 by The 451 Group. All rights reserved
The road to SPRAIN: approaches to scaling MySQL
database database database
Example:
Photo sharing website
Started with single MySQL database
Growth through acquisition and customer adoption
Expanded its MySQL footprint every two years
Considered sharding:
• Pros: increases scalability to serve high-traffic websites
• Cons: Horizontal (across masters), required application re-write
• Cons: Vertical (across the slaves), considered a temporary fix
users users users
© 2013 by The 451 Group. All rights reserved
The road to SPRAIN: approaches to scaling MySQL
slave slave slave slave
master
users users users
Example:
Photo sharing website
Started with single MySQL database
Growth through acquisition and customer adoption
Expanded its MySQL footprint every two years
Read/write splitting
• Pros: Better utilization of available resources
• Cons: Asynchronous replication can lead to slave lag
© 2013 by The 451 Group. All rights reserved
The road to SPRAIN: approaches to scaling MySQL
master
memcached
users users users
slave slave
Example:
Photo sharing website
Started with single MySQL database
Growth through acquisition and customer adoption
Expanded its MySQL footprint every two years
Memcached
• Pros: Improves read performance for frequently accessed data
• Cons: Not persistent, doesn’t solve write performance issues
• Looked instead for alternatives
© 2013 by The 451 Group. All rights reserved
Alternatives
NewSQL – retaining support for ACID transactions and SQL
• New databases
• Advanced storage engines, particularly for MySQL
• Advanced clustering/shard management approaches
© 2013 by The 451 Group. All rights reserved
Comparing NewSQL approaches
New databases • Pros: Designed specifically to support distributed architecture • Cons: May lack compatibility with existing applications
Advanced storage engines, particularly for MySQL
• Pros: Retain familiarity with with MySQL skills, tools • Cons: Re-architecting from the inside out.
Advanced clustering/shard management approaches
• Pros: Retain application compatibility while adding scalability • Cons: An extra layer of complexity?
Issues to consider: • Does it require a forklift move of your entire application ecosystem • Can you continue to leverage your existing MySQL skill set? • Is there a risk for your data, e.g. memory reliability?
© 2013 by The 451 Group. All rights reserved
Spotlight on ScaleBase
Creates a shared nothing architecture from standard databases
Elastic load balancing for MySQL (other databases on the roadmap)
Scale Out via read/write splitting or automatic data distribution
Data Traffic Manager serves as a proxy between the apps and DB
Provides a single point for administering the shared nothing cluster (for performance, HA, change management)
And the ability to add scalability without the need to migrate to a new database architecture or make any changes to existing apps.
© 2013 by The 451 Group. All rights reserved
Conclusion
Scalability isn’t simply a matter of a theoretical number of nodes
It’s also about the ability to deliver predictable performance
And to do so without adding management complexity
Distributed computing, proliferation of cloud, and geo-distributed applications are adding to the complexity
MySQL has never been under so much strain
The combination of advanced clustering/load balancing and management technologies provides a potential solution
© 2013 by The 451 Group. All rights reserved
Thank you. [email protected]
@maslett
Photo credit: swisscan on Flickr http://bit.ly/HCZCRQ
How to Scale Out your MySQL Database
25
Who We Are
Presenters: Paul Campaniello,
VP of Global Marketing 25 year technology veteran with marketing experience at Mendix, Lumigent, Savantis and Precise.
Doron Levari, Founder & CTO A technologist and long-time
veteran of the database industry. Prior to founding ScaleBase, Doron
was CEO to Aluna.
26
ScaleBase Mission
“MySQL can cost-effectively
scale to support an infinite
number of application users,
with NO disruption to the existing infrastructure”
27
ScaleBase: Scale-Out MySQL - Fix the SPRAIN
• Keep your MySQL - keep your InnoDB
• Ecosystem compatibility, preserve skills
• 100% application compatibility
• Smoother migration, no down-time, no forklift
• Your data is safe!
• No “in-memory” magic
• No “in-memory” size limit
Don’t throw out the baby with the bath water!
28
Single-Box Scalability Improvements in MySQL 5.6
Partitioning Improvements
– Explicit Partition Selection:
SELECT * FROM employees
PARTITION (p0, p2);
– Import / Export for Partitioned Tables:
Bring a new data set into a partitioned
table, or export a partition to manage it
as a regular table ALTER TABLE e
EXCHANGE PARTITION p0 WITH
TABLE e2;
http://dev.mysql.co/tech-resources/articles/whats-new-in-mysql-5.6.html
Replication Improvements
– Optimizations to Row-Based
Replication
– Multi-Threaded Slaves
– Improvements to Data Integrity
– Crash-Safe Slaves
– Replication Checksums
SCALABILITY issues remain due to the limitations of a single box:
To ensure ACID, you still face limitations with:
- Memory management - Thread management
- Semaphores - Locking
- Recovery tasks
No new functionality for sharing workloads across multiple boxes
Two Approaches to Scale-Out MySQL
30
Scale Out
Write
Read
Replication
R/W Splitting
Data Distribution
31
Read/Write Splitting: Application Experience
Read Replicas
Current: App sees master plus multiple slaves
With ScaleBase: App sees ONE database
32
Read/Write Splitting: Application Experience
ScaleBase Benefits: • Reads/Writes throughput increases • Better resource utilization/load balancing • Improved data consistency/transaction isolation with:
• Replication state lag awareness • Read stickiness after writes • Handling of edge cases
Problems: • Writing code, maintaining code • Maintaining DB ops and IPs in the app • Master DB is too occupied • Slaves are not balanced • Replication could break • Stale data may be read
Current: App sees master plus multiple slaves
With ScaleBase: App sees ONE database
Read Replicas
33
Data Distribution: Application Experience
Current: App is customized to support multiple shards
With ScaleBase: App sees ONE database…
…and does not require any customization
34
Data Distribution: Application Experience
Problems: • Major app rewrite, maintaining code • Maintaining DB ops and IPs in the app • Administration/3rd party tools are blind • Non-optimized data distribution policy
ScaleBase Benefits: • Unlimited scalability while improving performance • Real time elasticity • Verified data consistency • Real time monitoring, analysis • Carefully analyze distribution policy
Current: App is customized to support multiple shards
With ScaleBase: App sees ONE database…
…and does not require any customization
35
Typical ScaleBase Data Traffic Manager Deployment
Application
Servers
BI
Management
Database A Replica A
Database B Replica B
Database C Replica C
Database D Replica D
Unlimited Scale
Fault Tolerant Architecture
Centrally Manage Complex DB Environment
36
ScaleBase Features and Benefits
Feature Benefit
Automatic data distribution Scale data-, read-, write- intensive applications
Expert System Analysis Optimization of data distribution policy
Parallel query execution Best option for cross-db queries & maintenance commands
Query result aggregation Support of sophisticated cross-db queries, including ORDER BY, GROUP BY, LIMIT, Aggregate functions…
Online data redistribution Flexibility: no need to over-provision No downtime
Read/Write splitting Optimal for scaling of read-intensive applications
Replication lag-based routing Improves data consistency and isolation
Read stickiness after writes Ensure consistent and isolated database operation
100% compatible MySQL proxy Applications unmodified Standard MySQL tools and interfaces
MySQL databases unmodified Data is safe within MySQL InnoDB/MyISAM/etc.
Data consistency verifier Data consistency validation
Real-time monitoring and alerts Simplify management, reduce TCO
37
Scale…without SPRAIN
6000 12000
24000
36000
48000
60000
84000
500 500 1000
1500 1500 2000
2500
0
20000
40000
60000
80000
100000
120000
140000
160000
1 2 4 6 8 10 14
Thro
ugh
pu
t
Number of Databases
Throughput (TPM)
Total DB Size (MB)
# Connections
Scale Your Database With Your Business
38
ScaleBase Delivers Scalability
Scale to
Unlimited Users
Real-time DB Traffic Visibility
Automated Failover
…all with NO App Rewrites
39
Detailed Scale-Out Case Studies
Large Chip Co
• Scalability
• Multiple Apps
• Multiple growing users
• Availability
• MySQL DB
Solar Edge
• Next Gen Monitoring App
• Massive Scale
• Monitors real time data from thousands of distributed systems
Mozilla
• New Product/ Next Gen App/ AppStore
• Scalability
• Geo-clustering
AppDynamics
• Next gen APM company
• Scalability for the Netflix implementation
40
Economic Impact
Customer Experience
– Faster, more responsive apps,
even during peak loads
– Always on: 24/7
– Improve customer acquisition
(reduce churn) and brand
Scale
– Use commodity hardware vs. scale-
up hardware
– Cloud ready – deploys easily
Non-MySQL Options – Protect existing investment with no
ecosystem forklift
– Leverage existing MySQL skill set
Internal Development
– Developers focus on app, not
writing complex sharding code
– Eliminate development and
ongoing maintenance costs
Centralized Management
– Troubleshoot problems and
investigate incidents in seconds
with real-time monitoring
High Availability
– Eliminate database downtime
with automated fail-over
– Improve business continuity
41
ScaleBase Deployment
Environments
– Public Cloud
– AWS, Rackspace, any
– Private cloud
– Hosted / on-premise
Databases Supported
– MySQL 5.1, 5.5, 5.6 (under
certification)
– AWS RDS MySQL 5.1, 5.5
– Maria DB 10.0 (under
certification)
Path to Scale-Out:
1. Data Distribution
Policy Analysis
2. Functional Test
3. Load Test
4. Production Migration
(safe, online)
42
Summary
• MySQL is vast and adoption is growing
• MySQL scalability is a significant problem (SPRAIN)
– App explosion (web & mobile), Big Data transactions…
• ScaleBase provides cost-effective Scale-Out solutions
– Seamlessly scale to an infinite number of users
– Improve DB visibility with real-time monitoring
– Enhance business continuity with
automated failover
– Expert analysis and simple deployment
– No application rewrites
– Leverage existing MySQL ecosystem/skills
43
Questions (please enter directly into the GTW side panel)
@scalebase
www.ScaleBase.com
617.630.2800
@maslett
@451research
www.451research.com
44
Thank You