information resources management april 10, 2001. agenda n administrivia n database design n...

Information Resources Information Resources ManagementManagement

April 10, 2001April 10, 2001

AgendaAgenda

AdministriviaAdministrivia Database DesignDatabase Design

DenormalizationDenormalization Database AdministrationDatabase Administration

SecuritySecurity Backup & RecoveryBackup & Recovery Concurrency ControlsConcurrency Controls

AdministriviaAdministrivia

Schema Tuning - Schema Tuning - Staying NormalStaying Normal Split Tables - Vertical PartitioningSplit Tables - Vertical Partitioning Highly used vs. infrequently used Highly used vs. infrequently used

columnscolumns

Don’t partition if result will be more joinsDon’t partition if result will be more joins

Keys are duplicatedKeys are duplicated

Schema Tuning -Schema Tuning -Staying NormalStaying Normal Variable length fields (VARCHAR, others)Variable length fields (VARCHAR, others) Indeterminant record lengthsIndeterminant record lengths Row locations varyRow locations vary

Vertically partition row into two tables, Vertically partition row into two tables, one with fixed and one with variable one with fixed and one with variable columnscolumns

Schema Tuning -Schema Tuning -Leaving NormalLeaving Normal NormalizationNormalization

Eliminates duplicationEliminates duplication Reduces anomaliesReduces anomalies

Does not result in efficiencyDoes not result in efficiency

Denormalize for performanceDenormalize for performance

Denormalization WarningsDenormalization Warnings

Increases chance of errors or inconsistenciesIncreases chance of errors or inconsistencies May result in reprogramming if business rules May result in reprogramming if business rules

changechange Optimizes based on Optimizes based on currentcurrent transaction mix transaction mix Increases duplication and space requiredIncreases duplication and space required Increases programming complexityIncreases programming complexity

Always normalize first then denormalizeAlways normalize first then denormalize

DenormalizationDenormalization

Partition RowsPartition Rows Combine TablesCombine Tables Combine and PartitionCombine and Partition Replicate DataReplicate Data

Combining OpportunitiesCombining Opportunities

One-to-one (optional)One-to-one (optional) allow nullsallow nulls

Many-to-many (assoc. entity)Many-to-many (assoc. entity) 2 tables instead of 32 tables instead of 3

Reference data (one-to-many)Reference data (one-to-many) ““one” not use elsewhereone” not use elsewhere few of “many”few of “many”

Combining ExamplesCombining Examples

EmployeeEmployee-Spouse (name and SSN -Spouse (name and SSN only)only)

Owner-PctOwnedOwner-PctOwned - - PropertyProperty few owners with multiple propertiesfew owners with multiple properties

PropertyProperty-Type (description)-Type (description) one type per propertyone type per property

PartitioningPartitioning

HorizontalHorizontal By row typeBy row type Separate processing by typeSeparate processing by type Supertype/subtype decisionSupertype/subtype decision

Vertical (already seen)Vertical (already seen) BothBoth

ReplicationReplication

Intentionally repeating dataIntentionally repeating data

Example: Owner-PctOwned-PropertyExample: Owner-PctOwned-Property Owner includes PctOwned & Owner includes PctOwned &

PropertyIDPropertyID Property includes majority OwnerSSN Property includes majority OwnerSSN

and and PctOwnedPctOwned

Performance TuningPerformance Tuning

Not a one-time eventNot a one-time event Monitoring probably more importantMonitoring probably more important Things changeThings change

applications, database (table) sizes, applications, database (table) sizes, data characteristicsdata characteristics

hardware, operating system, DBMShardware, operating system, DBMS

Database AdministrationDatabase Administration

SecuritySecurity Backup & RecoveryBackup & Recovery Concurrency ControlsConcurrency Controls

Security - AuthorizationSecurity - Authorization

Row OperationsRow Operations ReadRead InsertInsert UpdateUpdate DeleteDelete

Table OperationsTable Operations IndexIndex

Creation/RemovalCreation/Removal ResourceResource

New TablesNew Tables AlterationAlteration DropDrop

Authorization GranularityAuthorization Granularity

Table-level onlyTable-level only View is the same as a tableView is the same as a table

ViewsViews

Select statement that is given a table Select statement that is given a table namename

Views can select from other viewsViews can select from other viewsCREATE VIEW OfficeEmps ASCREATE VIEW OfficeEmps AS

(SELECT O.OfficeNbr, E1.EmpID, E1.Name, M.EmpID, (SELECT O.OfficeNbr, E1.EmpID, E1.Name, M.EmpID, E2.Name AS MgrName) FROM Office AS O, Manager E2.Name AS MgrName) FROM Office AS O, Manager AS M, Employee AS E1, Employee as E2AS M, Employee AS E1, Employee as E2

WHERE O.OfficeNbr = M.OfficeNbr AND M.EmpID = WHERE O.OfficeNbr = M.OfficeNbr AND M.EmpID = E2.EmpID AND O.OfficeNbr = E1.OfficeNbr and E2.EmpID AND O.OfficeNbr = E1.OfficeNbr and E1.EmpID <> E2.EmpID)E1.EmpID <> E2.EmpID)

Enhancing Granularity Enhancing Granularity Through ViewsThrough Views Specific Columns - SELECT xxxxSpecific Columns - SELECT xxxx Specific Rows - WHERE xxxx=yyyySpecific Rows - WHERE xxxx=yyyy BothBoth

SQLSQL

GRANT priviledge ON table TO userGRANT priviledge ON table TO user (WITH GRANT OPTION)(WITH GRANT OPTION)

REVOKE priviledge ON table FROM REVOKE priviledge ON table FROM useruser (RESTRICT or CASCADE)(RESTRICT or CASCADE)

GRANTS by that user on that tableGRANTS by that user on that table

Types of FailuresTypes of Failures

TransactionTransaction LogicalLogical SystemSystem

SystemSystem Operating SystemOperating System HardwareHardware NetworkNetwork

DiskDisk

Recovery ApproachesRecovery Approaches

Switch - mirror DB needed (RAID-1)Switch - mirror DB needed (RAID-1) Restore/RerunRestore/Rerun

Previous backupPrevious backup Rerun all transactions (needed)Rerun all transactions (needed)

Log-BasedLog-Based Rollback - undo incompleteRollback - undo incomplete Rollforward - previous backupRollforward - previous backup

RequirementsRequirements

Permanently write changes without changing Permanently write changes without changing the databasethe database

Transaction StatesTransaction States Partially Committed - transaction is donePartially Committed - transaction is done Fully Committed - changes have been madeFully Committed - changes have been made

can fail from partially committed statecan fail from partially committed state

Log-Based RecoveryLog-Based Recovery

Log - record of all database activityLog - record of all database activity Log RecordsLog Records

Transaction startTransaction start Transaction write (update)Transaction write (update)

new and old valuesnew and old values Transaction abortTransaction abort Transaction commitTransaction commit

Log-Based RecoveryLog-Based Recovery

DeferredDeferred ImmediateImmediate

Deferred LogDeferred Log

Trans Log DB

Database modification occurs after transaction commits

Deferred LogDeferred Log

Only new values kept in update log Only new values kept in update log recordrecord

Only committed changes need to be Only committed changes need to be reapplied at recoveryreapplied at recovery

Uncommitted changes can be removed Uncommitted changes can be removed from the logfrom the log

Deferred Log ExampleDeferred Log ExampleTransaction Log Database

<T1 START>READ(EMP=75)GRADE=11WRITE(EMP=75)

<T1 EMP=75, GRADE=11>COMMIT

<T1 COMMIT>**committed**

EMP=75, GRADE=11<T2 START>

READ(EMP=75)READ(GRADE=11)SALARY = 26500WRITE(EMP=75)

<T2 EMP=75, SALARY=26500>COMMIT


EMP=75, SALARY=26500










Recovery only deletes from log










REDO(T1) - commit vs. actual database update

Transaction Log Database<T1 START>

READ(EMP=75)GRADE=11WRITE(EMP=75)

















REDO(T1); Delete T2 from Log



















REDO(T1); REDO(T2)










Failure During RecoveryFailure During Recovery

Recovery from recovery must be Recovery from recovery must be possiblepossible

Redo must be executable multiple times Redo must be executable multiple times without any differences from a single without any differences from a single executionexecution

Immediate ModificationImmediate Modification

Trans Log DB

Database modified as transaction proceeds

Immediate ModificationImmediate Modification

Update log records require old and new Update log records require old and new valuesvalues

Recovery requires either a REDO or an Recovery requires either a REDO or an UNDO based on whether or not each UNDO based on whether or not each transaction was committedtransaction was committed

Immediate ExampleImmediate ExampleTransaction Log Database


<T1 EMP=75, GRADE=10,11>EMP=75, GRADE=11

COMMIT<T1 COMMIT>

**committed**<T2 START>


<T2 EMP=75,SALARY=25000,26500>

EMP=75, SALARY=26500COMMIT





COMMIT<T1 COMMIT>



<T2 EMP=75,SALARY=25000,26500>



UNDO(T1)




COMMIT<T1 COMMIT>



<T2 EMP=75,SALARY=25000,26500>



REDO(T1)




COMMIT<T1 COMMIT>



<T2 EMP=75,SALARY=25000,26500>






COMMIT<T1 COMMIT>



<T2 EMP=75,SALARY=25000,26500>



UNDO(T2); REDO(T1) -- order can be important




COMMIT<T1 COMMIT>



<T2 EMP=75,SALARY=25000,26500>






COMMIT<T1 COMMIT>



<T2 EMP=75,SALARY=25000,26500>



REDO(T1); REDO(T2)




COMMIT<T1 COMMIT>



<T2 EMP=75,SALARY=25000,26500>



Logging RequirementsLogging Requirements

Log must always be in “stable storage”Log must always be in “stable storage” All log writes must be successfulAll log writes must be successful Log kept separate from databaseLog kept separate from database Backup copy of database that coincides with Backup copy of database that coincides with

start of a new logstart of a new log Recovery needed dependent on type of failureRecovery needed dependent on type of failure Database restart must recovery completely Database restart must recovery completely

before allowing new transactionsbefore allowing new transactions

CheckpointsCheckpoints

Recovery has to search entire logRecovery has to search entire log Many REDOs are unnecessaryMany REDOs are unnecessary Recovery can be a lengthy processRecovery can be a lengthy process

Checkpoints are used to limit the Checkpoints are used to limit the recovery action that is neededrecovery action that is needed


1. Flush all log records to permanent storage1. Flush all log records to permanent storage

2. Flush all data buffers to permanent storage2. Flush all data buffers to permanent storage

3. Write a <checkpoint> to the permanent3. Write a <checkpoint> to the permanent

storage copy of the logstorage copy of the log

No updates are allowed while No updates are allowed while checkpointing checkpointing

Checkpoint RecoveryCheckpoint Recovery

1. Search from end of log to most recent 1. Search from end of log to most recent <checkpoint><checkpoint>

2. Continue searching backward until the first 2. Continue searching backward until the first transaction <START> before the transaction <START> before the <checkpoint><checkpoint>

3. From that <START> onward, UNDO and 3. From that <START> onward, UNDO and REDO all transactionsREDO all transactions

(Serial execution only)(Serial execution only)

Advantages of LoggingAdvantages of Logging

Less Overhead at CommitLess Overhead at Commit No Data FragmentationNo Data Fragmentation No Need for Garbage CollectionNo Need for Garbage Collection Faster recoveryFaster recovery Support for ConcurrencySupport for Concurrency

TransactionsTransactions

ConceptConcept StateState SerializabilitySerializability Maintaining SerializabilityMaintaining Serializability

TransactionTransaction

Single Unit of Work - Single Unit of Work - User’s PerspectiveUser’s Perspective Multiple OperationsMultiple Operations

Required Properties (ACID)Required Properties (ACID) Atomicity - all or noneAtomicity - all or none Consistency - database consistency Consistency - database consistency

maintainedmaintained Isolation - appearance of being aloneIsolation - appearance of being alone Durability - changes persistDurability - changes persist

Transaction StateTransaction State

ActiveActive Partially CommittedPartially Committed FailedFailed AbortedAborted CommittedCommitted

Active

Partially Committed

Committed

Failed Aborted

Implementing Transactions in Implementing Transactions in SQLSQL COMMIT WORKCOMMIT WORK

ROLLBACK WORKROLLBACK WORK

Atomicity & DurabilityAtomicity & Durability

EasiestEasiest Completely new copy of databaseCompletely new copy of database Update new copyUpdate new copy Don’t update pointer until commitDon’t update pointer until commit Recoverable from failure at any point Recoverable from failure at any point

provided the acknowledgement of the provided the acknowledgement of the commit and the update of the pointer commit and the update of the pointer occur simultaneously.occur simultaneously.

ConcurrencyConcurrency

Multiple TransactionsMultiple Transactions Serial (one at a time) is best butSerial (one at a time) is best but

combination of slow & fast in single combination of slow & fast in single transactiontransaction

short and long transactionsshort and long transactions

Concurrency must be handled carefullyConcurrency must be handled carefully

ExampleExample

Employee (Employee (EmpIDEmpID, Grade, Salary), Grade, Salary)

Grade (Grade (GradeGrade, Midpoint), Midpoint)

Employee:Employee: 75, 10, 2500075, 10, 25000

Grade:Grade: 10, 2000010, 20000

11, 3000011, 30000

ExampleExample

T1 - Change employee #75 to grade 11T1 - Change employee #75 to grade 11READ (Employee)READ (Employee)

Grade = 11Grade = 11

WRITE (Employee)WRITE (Employee)

T2 - Update salaries by 5% of midpointT2 - Update salaries by 5% of midpointREAD (Employee)READ (Employee)

READ (Grade)READ (Grade)

Salary = Salary + (0.05 * Midpoint)Salary = Salary + (0.05 * Midpoint)

WRITE (Employee)WRITE (Employee)

Example - Serial ExecutionExample - Serial Execution

T1 then T2T1 then T2 Result: Salary = 26500 (25000 Result: Salary = 26500 (25000

+ .05*30000)+ .05*30000)

T2 then T1T2 then T1 Result: Salary = 26000 (25000 Result: Salary = 26000 (25000

+ .05*20000)+ .05*20000)

Concurrent ExecutionConcurrent Execution

T1 T2READ (Employee)

READ (Employee)Grade = 11

WRITE (Employee)READ (Grade)

Salary =WRITE (Employee)

Result?

Concurrent ExecutionConcurrent Execution

T1 T2READ (Employee)

READ (Grade)READ (Employee)

Grade = 11WRITE (Employee)

Salary =WRITE (Employee)

Result?

Recoverable SchedulesRecoverable Schedules

If T2 reads an item updated by T1, T1 If T2 reads an item updated by T1, T1 must commit before T2must commit before T2

Cascadeless ScheduleCascadeless Schedule If T2 reads an item updated by T1, T1 If T2 reads an item updated by T1, T1

must commit before T2 readsmust commit before T2 reads

Not RecoverableNot Recoverable

T1 T2READ (Employee)WRITE (Employee)

READ (Employee)READ (Grade)

WRITE (Employee)COMMIT

ROLLBACK

Result?

RecoverableRecoverable



WRITE (Employee)COMMIT

COMMIT

Result?

RecoverableRecoverable



WRITE (Employee)ROLLBACK

?????

Result?

CascadelessCascadeless


COMMITREAD (Employee)

READ (Grade)WRITE (Employee)

COMMIT

Result?

Ensuring SerializabilityEnsuring Serializability

Concurrency Control SchemesConcurrency Control Schemes

Can’t analyze transactionsCan’t analyze transactions some in progresssome in progress analysis longer than transactionanalysis longer than transaction already running continue to runalready running continue to run

Concurrency Control - LocksConcurrency Control - Locks

Shared - Read onlyShared - Read only LOCK-SLOCK-S Exclusive - Read/WriteExclusive - Read/Write LOCK-XLOCK-X

Compatibility of LocksCompatibility of Locks multiple transactions can have the multiple transactions can have the

same locksame lock shared locks onlyshared locks only

DeadlocksDeadlocks

T1: READ(A), READ(B), WRITE(A)

T2: READ(B), READ(A), WRITE(B)

T1 T2LOCK-X(A)READ(A)

LOCK-X(B)READ(B)

LOCK-S(B)WRITE(A)

LOCK-S(A)WRITE(B)

UNLOCK(A)UNLOCK(B)

UNLOCK(A)UNLOCK(B)

Locking ProtocolLocking Protocol

Set of RulesSet of Rules Reduce Possibility of DeadlocksReduce Possibility of Deadlocks Create “appearance” of serial executionCreate “appearance” of serial execution

to each transactionto each transaction

Two-Phase Locking ProtocolTwo-Phase Locking Protocol

Growing PhaseGrowing Phase Can obtain but not release locksCan obtain but not release locks

Shrinking PhaseShrinking Phase Can release but not obtain locksCan release but not obtain locks

First release of a lock is the transition First release of a lock is the transition between phases (between phases (lock pointlock point))

Two-Phase LockingTwo-Phase Locking

StrictStrict Prevent cascading rollbacksPrevent cascading rollbacks Exclusive locks (LOCK-X) held until Exclusive locks (LOCK-X) held until

commitcommit

RigorousRigorous AllAll locks held until commit locks held until commit

Lock ConversionLock Conversion

Changing a LockChanging a Lock Upgrade - shared to exclusiveUpgrade - shared to exclusive Downgrade - exclusive to sharedDowngrade - exclusive to shared

Can only upgrade in growing phaseCan only upgrade in growing phase Can only downgrade in shrinking Can only downgrade in shrinking

phasephase

Most Used Locking SchemeMost Used Locking Scheme

Read Read LOCK-S(A), READ(A) LOCK-S(A), READ(A) WriteWrite

If LOCK-S(A), If LOCK-S(A), UPGRADE(A), UPGRADE(A), WRITE(A)WRITE(A)

If no lock, If no lock, LOCK-X(A), WRITE(A) LOCK-X(A), WRITE(A) Locks held until COMMIT or ROLLBACKLocks held until COMMIT or ROLLBACK

Strict - exclusive onlyStrict - exclusive only Rigorous - all locksRigorous - all locks

GranularityGranularity

Lock only what is neededLock only what is needed Could beCould be

RowRow TableTable Set of TablesSet of Tables Entire DatabaseEntire Database

Model as a tree with the database at the root Model as a tree with the database at the root and the rows as the leavesand the rows as the leaves

Intention LockingIntention Locking

To lock a rowTo lock a row Traverse the tree from the root to the rowTraverse the tree from the root to the row Put intention locks on the nodes on the Put intention locks on the nodes on the

way downway down Intention locks provide knowledge of lower Intention locks provide knowledge of lower

level locks when a higher level lock is level locks when a higher level lock is desired -- prevents having to traverse the desired -- prevents having to traverse the entire tree to lock the databaseentire tree to lock the database

Intention LockingIntention Locking

Locks AcquiredLocks Acquired Top-DownTop-Down

Locks ReleasedLocks Released Bottom-UpBottom-Up

DeadlocksDeadlocks

PreventionPrevention RecoveryRecovery

Deadlock PreventionDeadlock Prevention

1. Acquire all locks simultaneously1. Acquire all locks simultaneously

2. Rollback instead of waiting for a lock2. Rollback instead of waiting for a lock

2a. Lock wait timeouts2a. Lock wait timeouts

Deadlock RecoveryDeadlock Recovery

If not prevented, deadlocks must be detected If not prevented, deadlocks must be detected and recoveredand recovered Detection - periodically search for problemsDetection - periodically search for problems

RecoveryRecovery Select a victim - which one?Select a victim - which one? Rollback - how far?Rollback - how far? Avoid StarvationAvoid Starvation

Always killing the same victim which Always killing the same victim which never gets to executenever gets to execute

Recovery with ConcurrencyRecovery with Concurrency

Locking ProtocolLocking Protocol Transaction RollbackTransaction Rollback CheckpointsCheckpoints RestartRestart

Recovery Locking ProtocolRecovery Locking Protocol

Recovery Dependent on LockingRecovery Dependent on Locking Multiple UNDOs may not work Multiple UNDOs may not work

correctly if a second transaction reads correctly if a second transaction reads a value updated by a prior transaction a value updated by a prior transaction before the prior transaction commitsbefore the prior transaction commits

Use Two-Phase Locking that is at least Use Two-Phase Locking that is at least StrictStrict

Transaction RollbackTransaction Rollback

Use log records to complete the Use log records to complete the rollbackrollback

Must rollMust rollbackback from most recent to from most recent to earlier updatesearlier updates

Release exclusive locks after rollback is Release exclusive locks after rollback is completedcompleted


Multiple transactions can be active at a Multiple transactions can be active at a checkpointcheckpoint Change <checkpoint> log record to Change <checkpoint> log record to

include list of all currently active include list of all currently active transactionstransactions

Still have to halt other processing while Still have to halt other processing while checkpointingcheckpointing

Checkpointing - When?Checkpointing - When?

More often checkpoints -> faster recoveryMore often checkpoints -> faster recovery Less often -> longer recoveryLess often -> longer recovery

MTBF - all componentsMTBF - all components TimingTiming Amount of ActivityAmount of Activity

# transactions# transactions # updates# updates log file sizelog file size

Restart RecoveryRestart Recovery

Redo list - commit foundRedo list - commit found Undo list - start found - not on Redo listUndo list - start found - not on Redo list

Scan log Scan log backwardsbackwards from the end from the end Stop at <checkpoint>Stop at <checkpoint>

For each transaction on the checkpoint For each transaction on the checkpoint list not on the Redo list, add it to the list not on the Redo list, add it to the Undo listUndo list

Restart RecoveryRestart Recovery

1. Starting again at the end of the log, 1. Starting again at the end of the log, Undo all transactions on the Undo listUndo all transactions on the Undo list

2. Return to the most recent checkpoint2. Return to the most recent checkpoint

3. Move forward and redo all transactions 3. Move forward and redo all transactions on the redo liston the redo list

Homework #8Homework #8

Database DesignDatabase Design Database AdministrationDatabase Administration

information resources management april 10, 2001. agenda n administrivia n database design n...

Documents