cmu 15-445/645 database systems (fall 2018) :: logging …shadow paging organize the database pages...
TRANSCRIPT
![Page 1: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/1.jpg)
Database Systems
15-445/15-645
Fall 2018
Andy PavloComputer Science Carnegie Mellon Univ.AP
Lecture #20
Logging Schemes
![Page 2: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/2.jpg)
CMU 15-445/645 (Fall 2018)
ADMINISTRIVIA
Homework #4: Today @ 11:59pm
Project #3: Monday Nov 19th @ 11:59am
Homework #5: Monday Dec 3rd @ 11:59pm
2
![Page 3: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/3.jpg)
CMU 15-445/645 (Fall 2018)
TIM
E
Schedule
MOTIVATION
3
BEGINR(A)W(A)⋮
COMMIT
Buffer Pool
A=1
Pa
ge
A=1
T1
![Page 4: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/4.jpg)
CMU 15-445/645 (Fall 2018)
TIM
E
Schedule
MOTIVATION
3
BEGINR(A)W(A)⋮
COMMIT
Buffer Pool
A=1
Pa
ge
A=1A=2
T1
![Page 5: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/5.jpg)
CMU 15-445/645 (Fall 2018)
TIM
E
Schedule
MOTIVATION
3
BEGINR(A)W(A)⋮
COMMIT
Buffer Pool
A=1
Pa
ge
A=1A=2
T1
![Page 6: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/6.jpg)
CMU 15-445/645 (Fall 2018)
TIM
E
Schedule
MOTIVATION
3
BEGINR(A)W(A)⋮
COMMIT
Buffer Pool
A=1
Pa
ge
T1
![Page 7: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/7.jpg)
CMU 15-445/645 (Fall 2018)
CRASH RECOVERY
Recovery algorithms are techniques to ensure database consistency, transaction atomicity, and durability despite failures.
Recovery algorithms have two parts:→ Actions during normal txn processing to ensure that the
DBMS can recover from a failure.→ Actions after a failure to recover the database to a state
that ensures atomicity, consistency, and durability.
4
Today
![Page 8: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/8.jpg)
CMU 15-445/645 (Fall 2018)
Failure Classification
Buffer Pool Policies
Shadow Paging
Write-Ahead Log
Checkpoints
Logging Schemes
5
![Page 9: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/9.jpg)
CMU 15-445/645 (Fall 2018)
CRASH RECOVERY
DBMS is divided into different components based on the underlying storage device.
We must also classify the different types of failures that the DBMS needs to handle.
6
![Page 10: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/10.jpg)
CMU 15-445/645 (Fall 2018)
STORAGE T YPES
Volatile Storage:→ Data does not persist after power is cut.→ Examples: DRAM, SRAM
Non-volatile Storage:→ Data persists after losing power.→ Examples: HDD, SDD
Stable Storage:→ A non-existent form of non-volatile storage that survives
all possible failures scenarios.
7
![Page 11: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/11.jpg)
CMU 15-445/645 (Fall 2018)
FAILURE CL ASSIFICATION
Type #1 – Transaction Failures
Type #2 – System Failures
Type #3 – Storage Media Failures
8
![Page 12: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/12.jpg)
CMU 15-445/645 (Fall 2018)
TRANSACTION FAILURES
Logical Errors:→ Transaction cannot complete due to some internal error
condition (e.g., integrity constraint violation).
Internal State Errors:→ DBMS must terminate an active transaction due to an
error condition (e.g., deadlock).
9
![Page 13: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/13.jpg)
CMU 15-445/645 (Fall 2018)
SYSTEM FAILURES
Software Failure:→ Problem with the DBMS implementation (e.g., uncaught
divide-by-zero exception).
Hardware Failure:→ The computer hosting the DBMS crashes (e.g., power
plug gets pulled).→ Fail-stop Assumption: Non-volatile storage contents are
assumed to not be corrupted by system crash.
10
![Page 14: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/14.jpg)
CMU 15-445/645 (Fall 2018)
STORAGE MEDIA FAILURE
Non-Repairable Hardware Failure:→ A head crash or similar disk failure destroys all or part of
non-volatile storage.→ Destruction is assumed to be detectable (e.g., disk
controller use checksums to detect failures).
No DBMS can recover from this! Database must be restored from archived version.
11
![Page 15: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/15.jpg)
CMU 15-445/645 (Fall 2018)
OBSERVATION
The primary storage location of the database is on non-volatile storage, but this is much slower than volatile storage.
Use volatile memory for faster access:→ First copy target record into memory.→ Perform the writes in memory.→ Write dirty records back to disk.
12
![Page 16: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/16.jpg)
CMU 15-445/645 (Fall 2018)
OBSERVATION
The DBMS needs to ensure the following guarantees:→ The changes for any txn are durable once the DBMS has
told somebody that it committed.→ No partial changes are durable if the txn aborted.
13
![Page 17: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/17.jpg)
CMU 15-445/645 (Fall 2018)
UNDO VS. REDO
Undo: The process of removing the effects of an incomplete or aborted txn.
Redo: The process of re-instating the effects of a committed txn for durability.
How the DBMS supports this functionality depends on how it manages the buffer pool…
14
![Page 18: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/18.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
BUFFER POOL
15
A=1 B=9 C=7
A=1 B=9 C=7
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
![Page 19: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/19.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
BUFFER POOL
15
A=1 B=9 C=7
A=1 B=9 C=7
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
![Page 20: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/20.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
BUFFER POOL
15
A=1 B=9 C=7
A=1 B=9 C=7
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
![Page 21: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/21.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
BUFFER POOL
15
A=1 B=9 C=7
A=1 B=9 C=7B=8
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
![Page 22: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/22.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
BUFFER POOL
15
A=1 B=9 C=7
A=1 B=9 C=7B=8
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
Do we force T2’s changes to be written to disk?Is T1 allowed to overwrite A even
though it has not committed?
B=8A=3
![Page 23: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/23.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
BUFFER POOL
15
A=1 B=9 C=7
A=1 B=9 C=7B=8
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
What happens when we need to rollback T1?
B=8A=3
![Page 24: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/24.jpg)
CMU 15-445/645 (Fall 2018)
STEAL POLICY
Whether the DBMS allows an uncommitted txn to overwrite the most recent committed value of an object in non-volatile storage.
STEAL: Is allowed.
NO-STEAL: Is not allowed.
16
![Page 25: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/25.jpg)
CMU 15-445/645 (Fall 2018)
FORCE POLICY
Whether the DBMS requires that all updates made by a txn are reflected on non-volatile storage before the txn is allowed to commit.
FORCE: Is enforced.
NO-FORCE: Is not enforced.
17
![Page 26: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/26.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
NO-STEAL + FORCE
18
A=1 B=9 C=7
A=1 B=9 C=7
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
![Page 27: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/27.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
NO-STEAL + FORCE
18
A=1 B=9 C=7
A=1 B=9 C=7
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
![Page 28: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/28.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
NO-STEAL + FORCE
18
A=1 B=9 C=7
A=1 B=9 C=7
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
![Page 29: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/29.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
NO-STEAL + FORCE
18
A=1 B=9 C=7
A=1 B=9 C=7B=8
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
![Page 30: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/30.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
NO-STEAL + FORCE
18
A=1 B=9 C=7
A=1 B=9 C=7B=8
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
FORCE means that T2 changes must be written to disk at this point.
![Page 31: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/31.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
NO-STEAL + FORCE
18
A=1 B=9 C=7
A=1 B=9 C=7B=8
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
FORCE means that T2 changes must be written to disk at this point.
NO-STEAL means that T1 changes cannot be written to disk yet.
![Page 32: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/32.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
NO-STEAL + FORCE
18
A=1 B=9 C=7
A=1 B=9 C=7B=8
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
FORCE means that T2 changes must be written to disk at this point.
NO-STEAL means that T1 changes cannot be written to disk yet.
B=8
![Page 33: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/33.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
T1 T2
NO-STEAL + FORCE
18
A=1 B=9 C=7
A=1 B=9 C=7B=8
BEGINR(A)W(A)
⋮ABORT
BEGINR(B)W(B)COMMIT
A=3
Now it’s trivial to rollback T1
B=8
![Page 34: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/34.jpg)
CMU 15-445/645 (Fall 2018)
NO-STEAL + FORCE
This approach is the easiest to implement:→ Never have to undo changes of an aborted txn because
the changes were not written to disk.→ Never have to redo changes of a committed txn because
all the changes are guaranteed to be written to disk at commit time.
19
![Page 35: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/35.jpg)
CMU 15-445/645 (Fall 2018)
SHADOW PAGING
Maintain two separate copies of the database (master, shadow)
Updates are only made in the shadow copy.
When a txn commits, atomically switch the shadow to become the new master.
Buffer Pool Policy: NO-STEAL + FORCE
20
![Page 36: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/36.jpg)
CMU 15-445/645 (Fall 2018)
SHADOW PAGING
Organize the database pages in a tree structure where the root is a single disk page.
There are two copies of the tree, the master and shadow→ The root points to the master copy.→ Updates are applied to the shadow copy.
21
![Page 37: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/37.jpg)
CMU 15-445/645 (Fall 2018)
DiskMemory
SHADOW PAGING EXAMPLE
22
MasterPage Table
1234
DB Root
![Page 38: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/38.jpg)
CMU 15-445/645 (Fall 2018)
SHADOW PAGING
To install the updates, overwrite the root so it points to the shadow, thereby swapping the master and shadow:→ Before overwriting the root, none of the transaction’s
updates are part of the disk-resident database → After overwriting the root, all of the transaction’s updates
are part of the disk-resident database.
23
Source: The Great Phil Bernstein
![Page 39: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/39.jpg)
CMU 15-445/645 (Fall 2018)
DiskMemory
SHADOW PAGING EXAMPLE
24
MasterPage Table
1234
DB Root
Txn T1
![Page 40: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/40.jpg)
CMU 15-445/645 (Fall 2018)
DiskMemory
SHADOW PAGING EXAMPLE
24
MasterPage Table
1234
DB Root
ShadowPage Table
1234
Txn T1
![Page 41: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/41.jpg)
CMU 15-445/645 (Fall 2018)
DiskMemory
SHADOW PAGING EXAMPLE
24
MasterPage Table
1234
DB Root
ShadowPage Table
1234
Txn T1
Read-only txns access the current master.
Active modifying txnupdates shadow pages.
![Page 42: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/42.jpg)
CMU 15-445/645 (Fall 2018)
DiskMemory
SHADOW PAGING EXAMPLE
24
MasterPage Table
1234
DB Root
ShadowPage Table
1234
Txn T1
Read-only txns access the current master.
Active modifying txnupdates shadow pages.
![Page 43: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/43.jpg)
CMU 15-445/645 (Fall 2018)
COMMIT
DiskMemory
SHADOW PAGING EXAMPLE
24
MasterPage Table
1234
DB Root
ShadowPage Table
1234
Txn T1
Read-only txns access the current master.
Active modifying txnupdates shadow pages.
![Page 44: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/44.jpg)
CMU 15-445/645 (Fall 2018)
COMMIT
DiskMemory
SHADOW PAGING EXAMPLE
24
MasterPage Table
1234
DB Root
ShadowPage Table
1234
Txn T1
Read-only txns access the current master.
Active modifying txnupdates shadow pages.
![Page 45: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/45.jpg)
CMU 15-445/645 (Fall 2018)
COMMIT
DiskMemory
SHADOW PAGING EXAMPLE
24
MasterPage Table
1234
DB Root
ShadowPage Table
1234
Txn T1
Read-only txns access the current master.
Active modifying txnupdates shadow pages.
XXX
X
![Page 46: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/46.jpg)
CMU 15-445/645 (Fall 2018)
SHADOW PAGING UNDO/REDO
Supporting rollbacks and recovery is easy.
Undo: Remove the shadow pages. Leave the master and the DB root pointer alone.
Redo: Not needed at all.
25
![Page 47: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/47.jpg)
CMU 15-445/645 (Fall 2018)
SHADOW PAGING DISADVANTAGES
Copying the entire page table is expensive:→ Use a page table structured like a B+tree.→ No need to copy entire tree, only need to copy paths in
the tree that lead to updated leaf nodes.
Commit overhead is high:→ Flush every updated page, page table, and root.→ Data gets fragmented.→ Need garbage collection.
26
![Page 48: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/48.jpg)
CMU 15-445/645 (Fall 2018)
CORRECTION SQLITE
27
The older version of SQLite did notmake a complete copy of the database file at start of each new txn.
It writes old pages to a separate journal file before overwriting master version.
SQLite's default is now to use WAL.
![Page 49: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/49.jpg)
CMU 15-445/645 (Fall 2018)
OBSERVATION
Flushing non-contiguous pages on disk is slow (aka random writes).
We need a way to convert random writes into sequential writes.
28
![Page 50: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/50.jpg)
CMU 15-445/645 (Fall 2018)
WRITE-AHEAD LOG
Record the changes made to the database in a log file before the change is made.→ Assume that the log is on stable storage.→ Log contains sufficient information to perform the
necessary undo and redo actions to restore the database after a crash.
Buffer Pool Policy: STEAL + NO-FORCE
29
![Page 51: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/51.jpg)
CMU 15-445/645 (Fall 2018)
WAL PROTOCOL
The DBMS stages all of a txn's log records in volatile storage (usually backed by buffer pool).
All log records pertaining to an updated page are written to non-volatile storage before the page itself is over-written in non-volatile storage.
A txn is not considered committed until all its log records have been written to stable storage.
30
![Page 52: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/52.jpg)
CMU 15-445/645 (Fall 2018)
WAL PROTOCOL
Write a <BEGIN> record to the log for each txn to mark its starting point.
When a txn finishes, the DBMS will:→ Write a <COMMIT> record on the log→ Make sure that all log records are flushed before it
returns an acknowledgement to application.
31
![Page 53: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/53.jpg)
CMU 15-445/645 (Fall 2018)
WAL PROTOCOL
Each log entry contains information about the change to a single object:→ Transaction Id→ Object Id→ Before Value (UNDO)→ After Value (REDO)
32
![Page 54: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/54.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
A=1 B=5 C=7
TIM
E
Schedule
BEGINW(A)W(B)⋮
COMMIT
T1WAL
WAL EXAMPLE
33
<T1 BEGIN>
A=1 B=9 C=7
![Page 55: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/55.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
A=1 B=5 C=7A=8
TIM
E
Schedule
BEGINW(A)W(B)⋮
COMMIT
T1WAL
WAL EXAMPLE
33
<T1 BEGIN><T1, A, 1, 8>
A=1 B=9 C=7
1
2
![Page 56: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/56.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
A=1 B=5 C=7A=8
TIM
E
Schedule
BEGINW(A)W(B)⋮
COMMIT
T1WAL
WAL EXAMPLE
33
<T1 BEGIN><T1, A, 1, 8><T1, B, 5, 9>
B=9
A=1 B=9 C=7
![Page 57: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/57.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
A=1 B=5 C=7A=8
TIM
E
Schedule
BEGINW(A)W(B)⋮
COMMIT
T1WAL
WAL EXAMPLE
33
<T1 BEGIN><T1, A, 1, 8><T1, B, 5, 9><T1 COMMIT>
B=9
A=1 B=9 C=7
<T1 BEGIN><T1, A, 1, 8><T1, B, 5, 9><T1 COMMIT>
![Page 58: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/58.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
A=1 B=5 C=7A=8
TIM
E
Schedule
BEGINW(A)W(B)⋮
COMMIT
T1WAL
WAL EXAMPLE
33
<T1 BEGIN><T1, A, 1, 8><T1, B, 5, 9><T1 COMMIT>
Txn result is now safe to return to application.
B=9
A=1 B=9 C=7
<T1 BEGIN><T1, A, 1, 8><T1, B, 5, 9><T1 COMMIT>
![Page 59: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/59.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
BEGINW(A)W(B)⋮
COMMIT
T1WAL
WAL EXAMPLE
33
Txn result is now safe to return to application.
A=1 B=9 C=7
<T1 BEGIN><T1, A, 1, 8><T1, B, 5, 9><T1 COMMIT>
X
X
![Page 60: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/60.jpg)
CMU 15-445/645 (Fall 2018)
Buffer Pool
TIM
E
Schedule
BEGINW(A)W(B)⋮
COMMIT
T1WAL
WAL EXAMPLE
33
Txn result is now safe to return to application.
A=1 B=9 C=7
<T1 BEGIN><T1, A, 1, 8><T1, B, 5, 9><T1 COMMIT>
X
X
Everything we need to restore T1 is in the log!
![Page 61: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/61.jpg)
CMU 15-445/645 (Fall 2018)
WAL IMPLEMENTATION
When should the DBMS write log entries to disk?
When should the DBMS write dirty records to disk?
34
![Page 62: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/62.jpg)
CMU 15-445/645 (Fall 2018)
WAL IMPLEMENTATION
When should the DBMS write log entries to disk?→ When the transaction commits.→ Can use group commit to batch multiple log flushes
together to amortize overhead.
When should the DBMS write dirty records to disk?
34
![Page 63: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/63.jpg)
CMU 15-445/645 (Fall 2018)
WAL IMPLEMENTATION
When should the DBMS write log entries to disk?→ When the transaction commits.→ Can use group commit to batch multiple log flushes
together to amortize overhead.
→ Every time the txn executes an update?→ Once when the txn commits?
34
![Page 64: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/64.jpg)
CMU 15-445/645 (Fall 2018)
WAL DEFERRED UPDATES
If we prevent the DBMS from writing dirty records to disk until the txn commits, then we don’t need to store their original values.
35
WAL
<T1 BEGIN><T1, A, 1, 8><T1, B, 5, 9><T1 COMMIT>
XX
![Page 65: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/65.jpg)
CMU 15-445/645 (Fall 2018)
WAL DEFERRED UPDATES
If we prevent the DBMS from writing dirty records to disk until the txn commits, then we don’t need to store their original values.
35
WAL
<T1 BEGIN><T1, A, 1, 8><T1, B, 5, 9><T1 COMMIT>
XX
WAL
<T1 BEGIN><T1, A, 8><T1, B, 9><T1 COMMIT>CRASH!
WAL
<T1 BEGIN><T1, A, 8><T1, B, 9>CRASH!
Replay the log and redo each update.
Simply ignore all of T1's updates.
![Page 66: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/66.jpg)
CMU 15-445/645 (Fall 2018)
WAL DEFERRED UPDATES
This won’t work if the change set of a txn is larger than the amount of memory available.
The DBMS cannot undo changes for an aborted txn if it doesn’t have the original values in the log.
We need to use the STEAL policy.
36
![Page 67: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/67.jpg)
CMU 15-445/645 (Fall 2018)
BUFFER POOL POLICIES
Almost every DBMS uses NO-FORCE + STEAL
37
NO-STEAL STEAL
NO-FORCE Fastest
FORCE Slowest
Runtime Performance
NO-STEAL STEAL
NO-FORCE Slowest
FORCE Fastest
Recovery PerformanceUndo + Redo
No Undo + No Redo
![Page 68: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/68.jpg)
CMU 15-445/645 (Fall 2018)
LOGGING SCHEMES
Physical Logging→ Record the changes made to a specific location in the
database.→ Example: "Diff"
Logical Logging→ Record the high-level operations executed by txns.→ Not necessarily restricted to single page.→ Example: The UPDATE, DELETE, and INSERT queries
invoked by a txn.
43
![Page 69: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/69.jpg)
CMU 15-445/645 (Fall 2018)
PHYSICAL VS. LOGICAL LOGGING
Logical logging requires less data written in each log record than physical logging.
Difficult to implement recovery with logical logging if you have concurrent txns.→ Hard to determine which parts of the database may have
been modified by a query before crash.→ Also takes longer to recover because you must re-execute
every txn all over again.
44
![Page 70: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/70.jpg)
CMU 15-445/645 (Fall 2018)
PHYSIOLOGICAL LOGGING
Hybrid approach where log records target a single page but do not specify data organization of the page.
This is the most popular approach.
45
![Page 71: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/71.jpg)
CMU 15-445/645 (Fall 2018)
LOGGING SCHEMES
46
UPDATE foo SET val = XYZ WHERE id = 1;
Physical
<T1,Table=X,Page=99,Offset=4,Before=ABC,After=XYZ>
<T1,Index=X_PKEY,Page=45,Offset=9,Key=(1,Record1)>
Logical
<T1,Query="UPDATE foo
SET val=XYZWHERE id=1">
Physiological
<T1,Table=X,Page=99,ObjectId=1,Before=ABC,After=XYZ>
<T1,Index=X_PKEY,IndexPage=45,Key=(1,Record1)>
![Page 72: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/72.jpg)
CMU 15-445/645 (Fall 2018)
CHECKPOINTS
The WAL will grow forever.
After a crash, the DBMS has to replay the entire log which will take a long time.
The DBMS periodically takes a checkpoint where it flushes all buffers out to disk.
38
![Page 73: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/73.jpg)
CMU 15-445/645 (Fall 2018)
CHECKPOINTS
Output onto stable storage all log records currently residing in main memory.
Output to the disk all modified blocks.
Write a <CHECKPOINT> entry to the log and flush to stable storage.
39
![Page 74: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/74.jpg)
CMU 15-445/645 (Fall 2018)
WAL
<T1 BEGIN><T1, A, 1, 2><T1 COMMIT><T2 BEGIN><T2, A, 2, 3><T3 BEGIN><CHECKPOINT><T2 COMMIT><T3, A, 3, 4>
⋮CRASH!
CHECKPOINTS
40
![Page 75: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/75.jpg)
CMU 15-445/645 (Fall 2018)
WAL
<T1 BEGIN><T1, A, 1, 2><T1 COMMIT><T2 BEGIN><T2, A, 2, 3><T3 BEGIN><CHECKPOINT><T2 COMMIT><T3, A, 3, 4>
⋮CRASH!
CHECKPOINTS
Any txn that committed before the checkpoint is ignored (T1).
40
![Page 76: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/76.jpg)
CMU 15-445/645 (Fall 2018)
WAL
<T1 BEGIN><T1, A, 1, 2><T1 COMMIT><T2 BEGIN><T2, A, 2, 3><T3 BEGIN><CHECKPOINT><T2 COMMIT><T3, A, 3, 4>
⋮CRASH!
CHECKPOINTS
Any txn that committed before the checkpoint is ignored (T1).
T2 + T3 did not commit before the last checkpoint.→ Need to redo T2 because it committed after
checkpoint.→ Need to undo T3 because it did not commit
before the crash.
40
![Page 77: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/77.jpg)
CMU 15-445/645 (Fall 2018)
CHECKPOINTS CHALLENGES
We have to stall all txns when take a checkpoint to ensure a consistent snapshot.
Scanning the log to find uncommitted txns can take a long time.
Not obvious how often the DBMS should take a checkpoint…
41
![Page 78: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/78.jpg)
CMU 15-445/645 (Fall 2018)
CHECKPOINTS FREQUENCY
Checkpointing too often causes the runtime performance to degrade.→ System spends too much time flushing buffers.
But waiting a long time is just as bad:→ The checkpoint will be large and slow.→ Makes recovery time much longer.
42
![Page 79: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/79.jpg)
CMU 15-445/645 (Fall 2018)
CONCLUSION
Write-Ahead Log to handle loss of volatile storage.
Use incremental updates (STEAL + NO-FORCE) with checkpoints.
On recovery: undo uncommitted txns + redocommitted txns.
47
![Page 80: CMU 15-445/645 Database Systems (Fall 2018) :: Logging …SHADOW PAGING Organize the database pages in a tree structure where the root is a single disk page. There are two copies of](https://reader033.vdocument.in/reader033/viewer/2022051913/6004c175fe294c2a4d087b77/html5/thumbnails/80.jpg)
CMU 15-445/645 (Fall 2018)
NEXT CL ASS
Recovery with ARIES.
48