managing resources with postgresql · managing resources with postgresql fosdem pgday 2015 cédric...
TRANSCRIPT
![Page 1: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/1.jpg)
Managing Resources with PostgreSQL
FOSDEM PGDay 2015
Cédric Villemain [email protected]
30th January 2015
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 1 / 24
![Page 2: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/2.jpg)
Cédric Villemain
PostgreSQL Expertise
Development, Support, Training
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 2 / 24
![Page 3: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/3.jpg)
What resources ?
• CPU
• RAM
• Disk
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 3 / 24
![Page 4: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/4.jpg)
How ?
• with .... Concurrency
• with .... Quota or Limit
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 4 / 24
![Page 5: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/5.jpg)
Where ?
Operating System
• user process (limits.conf, ...)
• container (cgroup, jail, ...)
• virtualization (VMWare, Xen, ...)
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 5 / 24
![Page 6: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/6.jpg)
Where ?
PostgreSQL
• Main con�guration (postgresql.conf)
• per object
• per role
• per database
• per role in database
• pg_service
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 6 / 24
![Page 7: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/7.jpg)
GUCs with restart
• max_connections
• shared_bu�ers
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 7 / 24
![Page 8: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/8.jpg)
GUCs without restart
• temp_bu�ers
• work_mem
• synchronous_commit
• temp_tablespace
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 8 / 24
![Page 9: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/9.jpg)
More GUCs without restart
• temp_�le_limit
• statement_timeout
• lock_timeout
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 9 / 24
![Page 10: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/10.jpg)
Resources per object
• FUNCTION
• ROLE
• DATABASE
• SYSTEM
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 10 / 24
![Page 11: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/11.jpg)
Function example
ALTER FUNCTION update_datamart()
SET temp_file_limit = -1;
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 11 / 24
![Page 12: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/12.jpg)
Database example
ALTER DATABASE datawarehouse
WITH CONNECTION LIMIT 10; --hard limit
ALTER DATABASE datawarehouse
SET temp_tablespace TO dwh_tblspc;
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 12 / 24
![Page 13: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/13.jpg)
Role example
ALTER ROLE web_user
WITH CONNECTION LIMIT 10; -- hard limit but ...
ALTER ROLE dba
SET work_mem = '128MB';
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 13 / 24
![Page 14: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/14.jpg)
Role in Database example
ALTER ROLE ALL IN DATABASE devel
SET synchronous_commit TO off;
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 14 / 24
![Page 15: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/15.jpg)
System example
ALTER SYSTEM
SET temp_buffers = '12MB';
SELECT pg_reload_conf();
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 15 / 24
![Page 16: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/16.jpg)
Connection Limit
• PostgreSQL �ood & DoS
• pooling & bouncing
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 16 / 24
![Page 17: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/17.jpg)
PgBouncer
• pool_size = 20
• max_client_conn = 2000
my_db = user=web_user host=remote_host pool_size=4
• bonus: PAUSE / RESUME
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 17 / 24
![Page 18: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/18.jpg)
top-like
• pg_activity (python, system only)
• pg_top (C, PostgreSQL extension)
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 18 / 24
![Page 19: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/19.jpg)
pg_proctab
• pg_cputime()
• pg_loadavg()
• pg_memusage()
• pg_proctab()
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 19 / 24
![Page 20: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/20.jpg)
Execution depends on loadavg
if (select load1<1 from pg_loadavg())
then update_datamart();
end if;
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 20 / 24
![Page 21: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/21.jpg)
What about e�ective_cache_size ?
select pg_size_pretty(memcached * 4096)
from pg_memusage;
-- don't forget to count shared_buffers
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 21 / 24
![Page 22: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/22.jpg)
Are we writing a lot on disk?
select pg_size_pretty(wchar) as requested_write,
pg_size_pretty(writes) as really_written,
(writes * 100 / wchar) as percent_really_written
from pg_proctab()
where pid = pg_backend_pid())
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 22 / 24
![Page 23: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/23.jpg)
And next ?
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 23 / 24
![Page 24: Managing Resources with PostgreSQL · Managing Resources with PostgreSQL FOSDEM PGDay 2015 Cédric Villemain cedric@2ndQuadrant.fr 30th January 2015 ... APUSE / RESUME Cédric Villemain](https://reader036.vdocument.in/reader036/viewer/2022070723/5f01f02c7e708231d401c69f/html5/thumbnails/24.jpg)
Questions ?
Now is the time to ask!
Cédric Villemain [email protected] () Managing Resources with PostgreSQL 30th January 2015 24 / 24