Download - P90 X Your Database!!
1
Friday, March 25, 2011
Your Database!!Denish Patel
Database ArchitectOmniTi Computer Consulting Inc.
1
Friday, March 25, 2011
2
Friday, March 25, 2011
Who am I ?• Denish Patel
• Database Architect
• 4+ Years with OmniTi
• Expertise
• Heterogeneous Databases
• PostgreSQL, Oracle, MySQL
• Very Large databases
• Performance tuning
• Application level
• SQL tuning
• Database and OS parameters
• Objects tuning2
Friday, March 25, 2011
Who am I ?• Denish Patel
• Database Architect
• 4+ Years with OmniTi
• Expertise
• Heterogeneous Databases
• PostgreSQL, Oracle, MySQL
• Very Large databases
• Performance tuning
• Application level
• SQL tuning
• Database and OS parameters
• Objects tuning2
OmniTi is hiring!
Friday, March 25, 2011
3
Friday, March 25, 2011
Agenda•Why bloat ?
•Issues
•Identify bloat
•Tables
•Indexes
•Bloat removal toolkit
•Monitoring and trending
3
Friday, March 25, 2011
4
Friday, March 25, 2011
why bloat?
•MVCC
•ACID
•Updates & Deletes
4
Friday, March 25, 2011
5
Friday, March 25, 2011
MVCC method-1
5
Friday, March 25, 2011
MVCC method-1
5
Friday, March 25, 2011
MVCC method-1!
!"#$%&'
()*+&$
'
,+-.$'
/+&+' (0/1,2'
34..-+56''
7$89$"&'
,31:7'
;<='
'!:723,'
/2;2,2'
72;2>,''
><??!,'
5
Friday, March 25, 2011
6
Friday, March 25, 2011
MVCC method-2
6
Friday, March 25, 2011
MVCC method-2
6
Friday, March 25, 2011
!
!"#$%&
'"("&
)*'+!,&
!-+./&
012&
&3./,-!&
',0,!,&
/,0,4!&&
41553!&
MVCC method-2
6
Friday, March 25, 2011
7
Friday, March 25, 2011
Issues•Unnecessary costly I/O operations
•Poor database tables access time
•Increase maintenance time
•Waste of resources
•RAM
•CPU
•Disk
7
Friday, March 25, 2011
8
Friday, March 25, 2011
Prevent Bloat• autovacuum =on
• vacuum manually between batch updates
• Truncate table
• Free Space Map
• <= PostgreSQL 8.3
• max_fsm_pages
• max_fsm_tables
• >=PostgreSQL 8.4
• auto sizing free space map
8
Friday, March 25, 2011
9
Friday, March 25, 2011
BEWARE!•Bug: long running auto-vacuum workers
•PostgreSQL Version Affected
•8.3.10
•8.3.11
•8.3.12
•8.4.4
•8.4.5
9
Friday, March 25, 2011
BEWARE!•Bug: long running auto-vacuum workers
•PostgreSQL Version Affected
•8.3.10
•8.3.11
•8.3.12
•8.4.4
•8.4.5
9
Upgrade to latest release !!
Friday, March 25, 2011
10
Friday, March 25, 2011
Measure Bloat
•Bloat Report
•pg_bloat_report (OmniTI Labs)
•Nagios Monitoring
•Check_postgres_bloat
10
Friday, March 25, 2011
11
Friday, March 25, 2011
pg_bloat_reportBloat Report Tables
1. omniti.hits 1969348 of 4255960 pages wasted (46.3%), 15 GB of 32 GB.
2. omniti.clients 279979 of 370509 pages wasted (75.6%), 2187 MB of 2895 MB.
Bloat Report Indexes
1. omniti.hits_pk 51275 of 71075 pages wasted (72.1%), 401 MB of 555 MB.
2. omniti.clients_tz 47550 of 71271 pages wasted (66.7%), 371 MB of 557 MB.11
Friday, March 25, 2011
12
Friday, March 25, 2011
check_postgres_bloat
Notification Type: PROBLEMHost: omniti
State: CRITICALDate/Time: Wed Feb 16 11:24:56 UTC 2011
Additional Info: POSTGRES_BLOAT CRITICAL: DB "omniti" (host:omniti)
(port=5432) table omniti.hits rows:23995164 pages:2858893 shouldbe:2278176 (1.3X)
wasted size:4757233664 (4 GB)
12
Friday, March 25, 2011
13
Friday, March 25, 2011
bloat removal toolkit
• Offline - with locking
• Vacuum
• Vacuum Full
• Cluster
• Cluster vs Vacuum Full
• Online - with minimal locking
• compact_table
• pg_reorg
13
Friday, March 25, 2011
14
Friday, March 25, 2011
14
Friday, March 25, 2011
15
Friday, March 25, 2011
vacuum•Auto vacuum
•VACUUM [verbose|analyze] table
•Reclaims storage occupied by dead tuples
•Reclaimed space available to re-use
•Normally free space doesn’t return to OS
•No exclusive table lock
15
Friday, March 25, 2011
16
Friday, March 25, 2011
vacuum full• VACUUM FULL table
• Compact table based on dead rows
• Starting PostgreSQL 9.0 , it’s rewrite entire table and indexes (like CLUSTER)
• Reclaimed space available to re-use
• Space returned to OS
• Table Exclusive lock
• Expensive operation
16
Friday, March 25, 2011
17
Friday, March 25, 2011
17
Friday, March 25, 2011
18
Friday, March 25, 2011
cluster•CLUSTER [verbose] table_name
[USING index_name]
•Create reorganized copy of table based on index
•Rebuild indexes too
•Requires Exclusive Lock on table
•Expensive Operation
18
Friday, March 25, 2011
19
Friday, March 25, 2011
cluster vs vacuum full
Order based on IndexIt’s just rewrite of table without
any specific order
Table needs index Doesn’t need index
Doesn’t bloat indexBloat indexes (Pre PostgreSQL
9.0)
Overall operation Faster Overall Operation Slower (Pre
PostgreSQL 9.0)
Recommended Not recommended
19
Friday, March 25, 2011
20
Friday, March 25, 2011
20
Friday, March 25, 2011
21
Friday, March 25, 2011
compact_table
•Developed by OmniTi
•Reorganize rows to empty pages at the end based on ctid
•ctid - (page number, tuple number)
•When we update a row, the row's ctid changes, because the update creates a new version of the row and leaves the old version behind
21
Friday, March 25, 2011
22
Friday, March 25, 2011
compact_tablepostgres=# select ctid,* from bar;
ctid | a | b -------+---+--- (0,1) | 1 | 1 (0,2) | 2 | 1
(2 rows)
postgres=# update bar set a=3 where a=2;UPDATE 1
postgres=# select ctid,* from bar; ctid | a | b
-------+---+--- (0,1) | 1 | 1 (0,3) | 3 | 1
(2 rows)
postgres=# vacuum verbose bar;..
INFO: "bar": found 1 removable, 2 nonremovable row versions in 1 out of 1 pagesDETAIL: 0 dead row versions cannot be removed yet.
VACUUM22
Friday, March 25, 2011
23
Friday, March 25, 2011
compact_table
23
Friday, March 25, 2011
compact_tablepostgres=# truncate table bar;
TRUNCATE TABLEpostgres=# insert into bar select generate_series(1,1000);
INSERT 0 1000postgres=# delete from bar where a %2 = 0;
DELETE 500postgres=# select max(ctid) from bar;
max -------- (4,95)(1 row)
postgres=# vacuum verbose bar;INFO: "bar": found 0 removable, 500
nonremovable row versions in 5 out of 5 pagesVACUUM23
Friday, March 25, 2011
24
Friday, March 25, 2011
compact_tablepostgres=# begin;
BEGINpostgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 161postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 161postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 48postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 48postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 34postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 0postgres=# commit;
COMMITpostgres=# vacuum verbose bar;
.
.INFO: "bar": truncated 5 to 3 pages
.VACUUM24
Friday, March 25, 2011
25
Friday, March 25, 2011
compact_table
•Pretty safe update process
•No additional changes required
•Slow process
•Generate Index bloat
•Under development
•Not tested in production
•Future consideration?25
Friday, March 25, 2011
26
Friday, March 25, 2011
26
Friday, March 25, 2011
27
Friday, March 25, 2011
pg_reorg• Developed by NTT OSS Center
• Reorganize table with very short lock
• Needs Primary Key
• Not Null Unique Key works!
• CLUSTER online
• ORDER BY Key
• VACUUM FULL Online
• Rewrite without ORDER BY
• Build and Install like other Contrib Module
27
Friday, March 25, 2011
28
!
!"#$%&'(
)"'($"($*+,($-./*0(
( 1&2*#$(
( 3,4-$*(
( 5*/*$*(
6*,/-7(
8,$%"&-/(
9"#:(
;-./*(
;*+,(
)"'((
;-./*(
;-#'*$(
;-./*(
3,4-$*(
9"#:(($-./*(
<%$=(
8,*#-$%"&-/(
)"'(
!<-,,%&'(
<%$=(>*#7(
2="#$()"?:(
(
Friday, March 25, 2011
pg_reorg
28
!
!"#$%&'(
)"'($"($*+,($-./*0(
( 1&2*#$(
( 3,4-$*(
( 5*/*$*(
6*,/-7(
8,$%"&-/(
9"#:(
;-./*(
;*+,(
)"'((
;-./*(
;-#'*$(
;-./*(
3,4-$*(
9"#:(($-./*(
<%$=(
8,*#-$%"&-/(
)"'(
!<-,,%&'(
<%$=(>*#7(
2="#$()"?:(
(
Friday, March 25, 2011
29
Friday, March 25, 2011
pg_reorg Optionspg_reorg --no-order -table test -d omniti
-n -- no-orderDo online VACUUM FULL.
-o columns [,...]--order-by=columns [,...]Do online CLUSTER ordered by specified columns.
-t table--table=table
-Z--no-analyze
-T seconds--wait-timeout=seconds
29
Friday, March 25, 2011
30
Friday, March 25, 2011
pg_reorg in action$ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test
LOG: (query) SET statement_timeout = 0LOG: (query) SET search_path = pg_catalog, pg_temp, publicLOG: (query) SET client_min_messages = warningLOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclassLOG: (param:0) = public.testLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.conflicted_triggers($1)LOG: (param:0) = 7441636LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer)LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk reorg.pk_7441636, row pulbic.test)LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE reorg.reorg_trigger('INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END, $2)')LOG: (query) SELECT reorg.disable_autovacuum('reorg.log_7441636')LOG: (query) COMMIT
30
Friday, March 25, 2011
pg_reorg in action$ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test
LOG: (query) SET statement_timeout = 0LOG: (query) SET search_path = pg_catalog, pg_temp, publicLOG: (query) SET client_min_messages = warningLOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclassLOG: (param:0) = public.testLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.conflicted_triggers($1)LOG: (param:0) = 7441636LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer)LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk reorg.pk_7441636, row pulbic.test)LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE reorg.reorg_trigger('INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END, $2)')LOG: (query) SELECT reorg.disable_autovacuum('reorg.log_7441636')LOG: (query) COMMIT
SET variables
30
Friday, March 25, 2011
pg_reorg in action$ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test
LOG: (query) SET statement_timeout = 0LOG: (query) SET search_path = pg_catalog, pg_temp, publicLOG: (query) SET client_min_messages = warningLOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclassLOG: (param:0) = public.testLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.conflicted_triggers($1)LOG: (param:0) = 7441636LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer)LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk reorg.pk_7441636, row pulbic.test)LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE reorg.reorg_trigger('INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END, $2)')LOG: (query) SELECT reorg.disable_autovacuum('reorg.log_7441636')LOG: (query) COMMIT
SET variables
Place Triggers
30
Friday, March 25, 2011
31
Friday, March 25, 2011
pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE
LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'), true)LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE locktype = 'virtualxid' AND pid <> pg_backend_pid()LOG: (query) DELETE FROM reorg.log_7441636LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE pg_default AS SELECT * FROM ONLY public.test ORDER BY test_idLOG: (query) SELECT reorg.disable_autovacuum('reorg.table_7441636')LOG: (query) COMMITLOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM pg_index WHERE indrelid = $1LOG: (param:0) = 7441636LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING btree (test_id)
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 1000 31
Friday, March 25, 2011
pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE
LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'), true)LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE locktype = 'virtualxid' AND pid <> pg_backend_pid()LOG: (query) DELETE FROM reorg.log_7441636LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE pg_default AS SELECT * FROM ONLY public.test ORDER BY test_idLOG: (query) SELECT reorg.disable_autovacuum('reorg.table_7441636')LOG: (query) COMMITLOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM pg_index WHERE indrelid = $1LOG: (param:0) = 7441636LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING btree (test_id)
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 1000
CTASCreate Indexes
31
Friday, March 25, 2011
pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE
LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'), true)LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE locktype = 'virtualxid' AND pid <> pg_backend_pid()LOG: (query) DELETE FROM reorg.log_7441636LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE pg_default AS SELECT * FROM ONLY public.test ORDER BY test_idLOG: (query) SELECT reorg.disable_autovacuum('reorg.table_7441636')LOG: (query) COMMITLOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM pg_index WHERE indrelid = $1LOG: (param:0) = 7441636LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING btree (test_id)
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 1000
CTASCreate Indexes
Apply Operational
logs
31
Friday, March 25, 2011
32
Friday, March 25, 2011
pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 0
LOG: (query) SELECT reorg.reorg_swap($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SELECT reorg.reorg_drop($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTEDLOG: (query) ANALYZE publoc.testLOG: (query) COMMIT
32
Friday, March 25, 2011
pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 0
LOG: (query) SELECT reorg.reorg_swap($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SELECT reorg.reorg_drop($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTEDLOG: (query) ANALYZE publoc.testLOG: (query) COMMIT
Lock table and
apply operational logs
last time
32
Friday, March 25, 2011
pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 0
LOG: (query) SELECT reorg.reorg_swap($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SELECT reorg.reorg_drop($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTEDLOG: (query) ANALYZE publoc.testLOG: (query) COMMIT
Lock table and
apply operational logs
last time
Swapping
32
Friday, March 25, 2011
pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 0
LOG: (query) SELECT reorg.reorg_swap($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SELECT reorg.reorg_drop($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTEDLOG: (query) ANALYZE publoc.testLOG: (query) COMMIT
Lock table and
apply operational logs
last time
Swapping
Drop work and temp tables
Analyze new table
32
Friday, March 25, 2011
33
Friday, March 25, 2011
pg_reorg in Production
•Database Size 540GB
•OLTP database 2-3K transactions/sec
•Largest pre-rebuild table Size was 127GB
•Took 5.75 hours to rebuild
•Reclaimed 52GB
•No outage reported for website!
33
Friday, March 25, 2011
34
Friday, March 25, 2011
pg_reorg in Production
34
Table Size Before Reorg
Pg_Reorg Execution time
Bloat Removed
35GB 24m 10sec 13GB
27GB 116m 8 Sec 6GB
31GB 64m 32sec 4GB
44GB 128m 44sec 3GB
Friday, March 25, 2011
35
Friday, March 25, 2011
35
Friday, March 25, 2011
36
Friday, March 25, 2011
36
Friday, March 25, 2011
37
Friday, March 25, 2011
Test Case
• Database : PostgreSQL 9.0.2
• OS: Solaris 11
• RAM : 32GB
• work_mem=16MB
• Table Name : Users
• Table Size : 1180 MB
• Indexes Size : 256 MB
• Rows : 6M
postgres=# \d users Table "public.users"
Column | Type | Modifiers --------------+--------------------------+----------- user_id | integer | not null
name | text | creation_tsz | timestamp with time zone |
active | boolean | clicks | double precision |
Indexes: "users_pkey" PRIMARY KEY, btree (user_id) "users_creation_tsz" btree (creation_tsz)
- DELETE 3M ROWS- UPDATE 1M ROWS37
Friday, March 25, 2011
38
Friday, March 25, 2011
Comparison
CLUSTER VACUUM FULL PG_REORG
Execution Time: 2 min
Table Size: 594MBIndex Size: 128 MB
Execution Time:1 min 50 Sec
Table Size: 594MBIndex Size: 128 MB
Execution Time: 1 min 4 Sec
Table Size: 594MBIndex Size: 128 MB
offline offline online
Built - in Built-in External Tool
38
Friday, March 25, 2011
39
Friday, March 25, 2011
Other Tips
•Remove unused indexes
•Reindex indexes as needed
•Partitioned table
•Detach child table -> Cluster child table -> Attach
•Avoid vacuum full (Pre PostgrSQL 9.0)
39
Friday, March 25, 2011
40
Friday, March 25, 2011
Monitoring
•Circonus
•Nagios monitoring
•Bloat report
40
Friday, March 25, 2011
41
Friday, March 25, 2011
trending
•Cacti
•Circonus
41
Friday, March 25, 2011
42
Friday, March 25, 2011
trending
42
Friday, March 25, 2011
trending
42
Friday, March 25, 2011
43
Friday, March 25, 2011
thanks!
•OmniTi
•Robert Treat
•Depesz
•Joshua (End Point)
•PGEast Conference Committee
43
Friday, March 25, 2011
44
Friday, March 25, 2011
References•http://www.postgresql.org/docs/9.0/
•http://reorg.projects.postgresql.org/pg_reorg.html
•https://labs.omniti.com/pgtreats/trunk/tools/compact_table
•http://blog.endpoint.com/2010/09/reducing-bloat-without-locking.html
44
Friday, March 25, 2011
45
Friday, March 25, 2011
open discussion
•When PostgreSQL will have built in features to rebuild table and index online ?
•Is it worth to change MVCC handling to avoid bloating?
45
Friday, March 25, 2011
46
Friday, March 25, 2011
46
Questions?
•Email : [email protected]
•Twitter : http://twitter.com/#!/denishpatel
Friday, March 25, 2011