High Load on Alibaba MySQLXiaobin Lin

About MeXiaobin LinRDS @ Aliyun, AlibabaDatabase source code teamWork around MySQL more than 7 years

Cataloghotspot updating

unexpected long transaction

high concurrency

slave latency

thread holding

performance enhancement

hotspot updatingProblemreduce stock of hot commoditieshotspot updating on a single InnoDB rowsdead lock detecting

hotspot updatingThreadtps1610061916012140002410000489800967000192360038420007686001536150

hotspot updating

Dead lock detection

hotspot updatingDisable dead lock detection

hotspot updatingSolution: Queue on primary key

hotspot updating

unexpected long transactionsimplest transactionbegin;

update table_name set where id=N;

commit;

unexpected long transactionProblemunexpected long transaction on high load

long time waiting for commitbusy clientnetwork

rowlock causes problem

hotspot updatingLast Case: Queue on primary key

unexpected long transactionSolutionCommit/rollback transactions as early as possible

start transaction;

update /*! commit_on_success rollback_on_fail*/ table_name set

high concurrencyProblemmore than 5w active thread

cpu cost on switch

lot of timeout in client

out of MySQLs capability

high concurrencySolution: actions on Low/High level watermark

Slave latencyProblemHigh load on master means the same on slave

speed of applying event on slave VS generating event on master

Slave latencySolutionmulti-thread applying event, based on table and rowrelay fetchbulk apply

Thread holdingProblemVersion upgrade / machine maintenance need to shutdown master

user connection broken

Thread holdingSolutionproxy layerthread holding

performance enhancementasync redo log copy and writeinnodb redo crc32 checksumreuse dummy index on change bufferremove read-only transaction listcache transaction read-view reduce overload of read-only transactionsplit lock_grant lock

80% improvement on TPS/QPS

Q&A