scaling with postgres (robert treat)

Highload++ 2010

Scaling with PostgresRobert Treat

Monday, October 25, 2010

Who Am I?

✤ Robert Treat

✤ OmniTI

✤ Design, Development, Database, Ops


Who Am I?

✤ Robert Treat

✤ OmniTI

✤ Design, Development, DATABASE, Ops


Who Am I?

✤ Robert Treat

✤ OmniTI

✤ Design, Development, DATABASE, Ops

✤ Etsy, Allisports, National Geographic, Gilt, etc...


Who Am I?

✤ Robert Treat

✤ Postgres

✤ Web, Advocacy, phpPgAdmin

✤ Major Contributor


Who Am I?

✤ Postgres 6.5 -> 9.1alpha1

✤ Terabytes of data

✤ Millions of transactions per day

✤ OLTP, ODS, DW

✤ Perl, PHP, Java, Ruby, C#


Who Am I?

OBSERVATION == LEARNING(hopefully)


Scalability

It is the ability of a computer application or product (hardware

or software) to continue to function well when it (or its

context) is changed in size or volume in order to meet a user

need.


Scalability

Given ever increasing load


Scalability

NEVER GO DOWNALWAYS PERFORM WELL



Scalability



impossible goal, but we’ll try


Scalability



NOTE! data loss is not a goal, but ideally we won’t lose it :-)


It starts with culture...


✤ Get over schema purity

✤ add column default not null


✤ Get over schema purity

✤ add column default not null

Good performance comes from good schema design, HOWEVER, perfect relational

modeling is NOT THE GOAL


✤ Devs must own schema and queries

✤ they design, you refine


✤ Devs must own schema and queries

✤ they design, you refine

Performance and scalability cannot be managed solely within the database; both require application level knowledge. To

achieve this, application developers need to have visibility of the resources they work on


Gain Visibility


Gain Visibility

✤ Monitoring

✤ Alerts

✤ Trending

✤ Capacity Planning

✤ Performance Tuning


Gain Visibility

✤ Alerts

✤ server: out of disk space, high load, etc...

✤ database: connections, sequences, etc...

✤ business: registrations, revenue, etc...

✤ etc...

check_postgres.pl


Gain Visibility

✤ Trending

✤ server: disk usage, load, etc...

✤ database: connections, sequences, etc...

✤ business: registrations, revenue, etc...

✤ etc...

cacti, mrtg, circonus


Gain Visibility

✤ Capacity Planning

✤ disks, cpu, memory

✤ connections, vacuum, bloat

simple projections, done regularly, are good enough


Gain Visibility

✤ Performance tuning

✤ how long do queries take?

✤ how often do they run?

pgfouine


Gain Visibility

COMMITS/PUSHES


Gain Visibility

ALL alerts, graphs, query reports, etc... MUST be available to EVERYONE on

the team AT ALL TIMES


Hands on

You can’t succeed without first putting the right culture in place.

Once you are on the right path, make sure you have the right technology


Postgres Versions

✤ MINIMUM: 8.3

✤ removes xid for read only queries, significant reduction in vacuum activity


Postgres Versions

✤ MINIMUM: 8.3


seriously!


Postgres Versions

✤ MINIMUM: 8.3


✤ BETTER: 8.4

✤ revised free space map management leads to more efficient vacuuming


Postgres Versions

✤ MINIMUM: 8.3


✤ BETTER: 8.4

✤ revised free space map management leads to more efficient vacuuming

✤ WHY NOT? 9.0

✤ Hot standby / streaming replication couldn’t hurt


Speaking of replication

✤ Common practice for scaling websites

✤ Good for READ based loads

✤ We have used many:

✤ slony, rubyrep, bucardo, 9.0 built-in, mammoth, wrote-our-own



✤ No favorite system for this, evaluate based on:

✤ avoid solutions that duplicate writes at sql level (imho)

✤ how comfortable am I debugging the system?

✤ do you need automated schema changes?

✤ how much redundancy / complexity do you need?

✤ how does the system handle node failure for N nodes?


So what would you use? (tm)

✤ 2 Nodes, master + standby: Postgres 9.0

✤ Master + multiple slaves: Slony

✤ Master-Master: Bucardo

All choices subject to change!!


A word about “Sharding”

✤ Distributed computing is hard(er)

✤ we think of things in a singular global state

✤ the more we can work in that model, the better

✤ RDBM offer poor solutions for multiple masters

✤ you must manage that complexity on your own


A word about “Sharding”

✤ Splitting systems by service:

✤ separate db for login, forums, sales, etc...

✤ allows for growth

✤ provides simple interface


Pooling

✤ Postgres connections are expensive!

✤ fork new process per connection

✤ keep 1 process open per connection

✤ 1000+ processes you will notice trouble


Pooling

✤ Postgres connections are expensive!

✤ fork new process per connection

✤ keep 1 process open per connection

✤ 1000+ processes you will notice trouble

✤ POOLING

✤ JDBC, mod-perl

✤ pgbouncer ftw!


Summary

✤ Schema / Queries should be shared between dev, dba teams!

✤ Monitoring + Visibility!

✤ >= 8.3 Required!

✤ Replication, jump in it!

✤ Use connection pooling!


Thanks!

more:@robtreat2

www.xzilla.net

Oleg & CrewHighload++

OmniTIPostgres Community!

You!


scaling with postgres (robert treat)

Technology