intro to deadlocks
DESCRIPTION
What are deadlocks and how to debug and fix them.TRANSCRIPT
Intro to Deadlocks
February 4th, 2013Panupan Sriautharawong
Monday, February 11, 13
What are Deadlocks?
A situation that occurs in concurrent systems.
Happens when two or more competing actions get stuck because they’re waiting on one another to finish.
A bug or design flaw in most cases.
Preventable!
Monday, February 11, 13
InnoDB Storage Engine
Monday, February 11, 13
InnoDB Storage Engine
TransactionsAre used to group of database operations
Are atomic, meaning they can’t broken into smaller pieces. Either everything goes through or nothing does.
## Bank Transfer ExampleBEGIN;UPDATE account SET balance=balance+100 WHERE id=1;UPDATE account SET balance=balance-100 WHERE id=2;COMMIT;
Monday, February 11, 13
InnoDB Storage Engine
Shared vs Exclusive LocksA lock must be acquired (either implicitly or explicitly) before retrieving or modifying any data. There are two types of locks: - Shared (S) = Read only - Exclusive (X) = Read/Write
Table vs Row-Level LockingProperly defined indexes prevent toomany rows from unintentionally getting locked and also ensures optimal performance.
Monday, February 11, 13
Deadlocked Doggies
Tony starts with 5 bones
Jake starts with 1 bone
Action #1: Jake steals a bone from Tony
Action #2: Tony, being a good doggie, gives Jake a bone so they’ll both have 3
Jake
Tony
Monday, February 11, 13
Deadlocked DoggiesJake
Tony
## INITIAL SETUPCREATE TABLE `doggies` (`id` INT NOT NULL AUTO_INCREMENT,`name` VARCHAR(255) NOT NULL,`bones` INT NOT NULL,PRIMARY KEY (`id`)) ENGINE=InnoDB;
INSERT INTO doggies (id, name, bones) VALUES (1, "Tony", 5);INSERT INTO doggies (id, name, bones) VALUES (2, "Jake", 1);
## ACTION #1 # Jake steals a boneBEGIN;UPDATE doggies SET bones=bones-1 WHERE id=1; # X lock acquired on 1
## ACTION #2# Tony gives Jake a boneBEGIN;UPDATE doggies SET bones=bones+1 WHERE id=2; # X lock acquired on 2UPDATE doggies SET bones=bones-1 WHERE id=1;! # LOCK WAIT!COMMIT;
## ACTION #1 (continued)UPDATE doggies SET bones=bones+1 WHERE id=2;! # DEADLOCK!COMMIT;
Monday, February 11, 13
How to Debug’emInspecting MySQL Status
Grepping Rails Logs
$ cat log/production.log | grep -B 40 Deadlock
mysql> SHOW ENGINE INNODB STATUS;
mysql> SHOW PROCESSLIST;
mysql> EXPLAIN SELECT ...;
Monday, February 11, 13
How to Debug’em------------------------LATEST DETECTED DEADLOCK------------------------130204 13:10:32*** (1) TRANSACTION:TRANSACTION 214FC, ACTIVE 12 sec starting index readmysql tables in use 1, locked 1LOCK WAIT 3 lock struct(s), heap size 376, 2 row lock(s), undo log entries 1MySQL thread id 182, OS thread handle 0x111c8c000, query id 120976 localhost 127.0.0.1 root UpdatingUPDATE doggies SET bones=bones-1 WHERE id=1*** (1) WAITING FOR THIS LOCK TO BE GRANTED:RECORD LOCKS space id 0 page no 1130 n bits 72 index `PRIMARY` of table `deadlocks`.`doggies` trx id 214FC lock_mode X locks rec but not gap waitingRecord lock, heap no 2 PHYSICAL RECORD: n_fields 5; compact format; info bits 0 0: len 4; hex 80000001; asc ;; 1: len 6; hex 0000000214fb; asc ;; 2: len 7; hex 7900000de72774; asc y 't;; 3: len 4; hex 546f6e79; asc Tony;; 4: len 4; hex 80000004; asc ;;
*** (2) TRANSACTION:TRANSACTION 214FB, ACTIVE 44 sec starting index readmysql tables in use 1, locked 13 lock struct(s), heap size 376, 2 row lock(s), undo log entries 1MySQL thread id 184, OS thread handle 0x111e2b000, query id 120977 localhost 127.0.0.1 root Updating## CONNECTION #1# Jake steals a bone (cont..)UPDATE doggies SET bones=bones+1 WHERE id=2*** (2) HOLDS THE LOCK(S):RECORD LOCKS space id 0 page no 1130 n bits 72 index `PRIMARY` of table `deadlocks`.`doggies` trx id 214FB lock_mode X locks rec but not gapRecord lock, heap no 2 PHYSICAL RECORD: n_fields 5; compact format; info bits 0 0: len 4; hex 80000001; asc ;; 1: len 6; hex 0000000214fb; asc ;; 2: len 7; hex 7900000de72774; asc y 't;; 3: len 4; hex 546f6e79; asc Tony;; 4: len 4; hex 80000004; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:RECORD LOCKS space id 0 page no 1130 n bits 72 index `PRIMARY` of table `deadlocks`.`doggies` trx id 214FB lock_mode X locks rec but not gap waitingRecord lock, heap no 3 PHYSICAL RECORD: n_fields 5; compact format; info bits 0 0: len 4; hex 80000002; asc ;; 1: len 6; hex 0000000214fc; asc ;; 2: len 7; hex 7a00000cf122d8; asc z \" ;; 3: len 4; hex 4a616b65; asc Jake;; 4: len 4; hex 80000002; asc ;;
*** WE ROLL BACK TRANSACTION (2)
Monday, February 11, 13
Best PracticesDefine proper indexes and use EXPLAIN SELECT to verify. For example, make sure to define multi-column indexes across [id, type] on our polymorphic models.
Most common method of deadlock exception handling is to retry but this should only be used a last resort.
Easiest fix is to rearrange queries to make sure updates occur before selects (not always possible).
You can also lock records manually at the beginning of a transaction when you know they’ll need updating.
Monday, February 11, 13
Helpful Resources
http://dev.mysql.com/doc/refman/5.0/en/innodb-lock-modes.html
http://rhnh.net/2010/07/01/debugging-deadlocks-in-rails
http://www.chriscalender.com/?p=426
Monday, February 11, 13