postgres scaling opportunities and options

Postgres Scaling Opportunities

BRUCE MOMJIAN

December, 2014

Configuring Postgres for heavy workloads can take many forms.This talk explores available Postgres scaling options.Creative Commons Attribution License http://momjian.us/presentations

1 / 37

PostgreSQL the database…

◮ Open Source Object Relational DBMS since 1996

◮ Distributed under the PostgreSQL License

◮ Similar technical heritage as Oracle, SQL Server & DB2

◮ However, a strong adherence to standards (ANSI-SQL 2008)

◮ Highly extensible and adaptable design

◮ Languages, indexing, data types, etc.◮ E.g. PostGIS, JSONB, SQL/MED

◮ Extensive use throughout the world for applications andorganizations of all types

◮ Bundled into Red Hat Enterprise Linux, Ubuntu, CentOSand Amazon Linux

Postgres Scaling Opportunities 2 / 37

PostgreSQL the community…

◮ Independent community led by a Core Team of six

◮ Large, active and vibrant community

◮ www.postgresql.org

◮ Downloads, Mailing lists, Documentation

◮ Sponsors sampler:

◮ Google, Red Hat, VMWare, Skype, Salesforce, HP andEnterpriseDB

◮ http://www.postgresql.org/community/

EnterpriseDB the company…

◮ The worldwide leader of Postgres based products andservices founded in 2004

◮ Customers include 50 of the Fortune 500 and 98 of theForbes Global 2000

◮ Enterprise offerings:

◮ PostgreSQL Support, Services and Training◮ Postgres Plus Advanced Server with Oracle Compatibility◮ Tools for Monitoring, Replication, HA, Backup & Recovery

Community

◮ Citizenship

◮ Contributor of key features: Materialized Views, JSON, &more

◮ Nine community members on staff

EnterpriseDB the company…

Scaling

Database scaling is the ability to increase database throughput byutilizing additional resources such as I/O, memory, CPU, oradditional computers.However, the high concurrency and write requirements ofdatabase servers make scaling a challenge. Sometimes scaling isonly possible with multiple sessions, while other options requiredata model adjustments or server configuration changes.

Outline

Postgres scaling opportunities:

1. Multi-session

2. Single-session

3. Multi-host

Vertical/Horizontal Scaling

��

Horizontal

Vertical

��

Examples

Vertical scaling examples:

◮ More and faster CPUs

◮ More memory

◮ More and faster storage

◮ Battery-backed cache (BBU)

Horizontal scaling involves adding servers.

Hardware Components

Memory

1. Multi-Session

Clients

Database

Memory

I/O Spreading Using Tablespaces

Clients

Database

Memory

Tablespaces

Requires tables & indexes to be spread across tablespacesTablespaces should be on different storage devices

I/O Spreading Using Hash Partitions

��

Clients

Database

Memory

Hash Partitions

Requires rows to be hashed by key; one partition per device

I/O Spreading Using RAID 0

��

Clients

Database

Memory

RAID 0

Auto-hashed by storage block number

Write Spreading Using WAL Relocation

Clients

Database

Memory

WAL Tables and Indexes

Separates WAL writes from table & index I/O

Read Reduction via Increased Memory

Clients

Database

Memory

Additional memory caching reduces read requirements

Scaling Connections Using a Pooler

Clients

Database

Connection Pooler

Memory

Fewer idle connections reduces resource usage

Multi-Session CPU Scaling

Clients

Database

Memory

Multiple sessions spread across available CPUs

2. Single-Session

Client

Database

Memory

Read Parallelism Using effective_io_concurrency

Client

Database

Memory

Used during bitmap heap scansAssumes table is hashed across multiple devices

I/O Scaling via Parallelism (not implemented)

Client

Database

Memory

Involves parallel index, heap, partition, and tablespace access

CPU Scaling via Parallelism (not implemented)Database

Client

Memory

Involves parallel sorts, joins, and function execution

Sort I/O Reduction Using Work_mem

Client

Database

Memory Sorts/Hashes

Reduces temporary result reads & writes

Logical Dump Parallelism

Memory

pg_dump

Dumps tables using concurrent database connections

Logical Restore Parallelism

Memory

pg_restore

Loads tables and creates indexes using concurrent databaseconnections

3. Multi-Host

��

Read Scaling Using Pgpool & Streaming Replication

SELECTINSERT, UPDATE,

DELETE to master

Slave SlaveMasterreplication

to any host

replicationstreaming

��

pgpool

A full copy of the data exists on every node.

CPU/Memory Scaling With AsynchronousMulti-Master

SQL Queries SQL Queries

Asynchronous

Memory

Tables and Indexes Tables and Indexes

A full copy of the data exists on every node; requires conflictresolution. The asynchronous delay allows write-load buffering.

Oracle Real Application Clusters (RAC)

Synchronization

SQL Queries SQL Queries

Memory

WALWAL1 WAL2Tables and Indexes

Tables and indexes on shared storage; inter-node synchronizationrequired for cache consistency

I/O Scaling with Sharding: Challenges

◮ Multi-host write queries require two-phase commit (exceptXC)

◮ Multi-host visibility snapshots are not supported (except XC)

◮ Sharding benefits are only possible with a shardableworkload

◮ Changing the sharding layout can cause downtime

◮ Additional hosts reduce reliability; additional standbyservers might be required

Application-Based Sharding

��

Applications

SQL Queries

Data Node Data Node Data Node

Applications send queries based on the sharding layout.

Sharding Using PL/Proxy

��

Function Calls

SQL Queries

PL/Proxy

Data Node Data Node Data Node

Requires rows to be hashed by key, supports parallel-node queryexecution

Sharding Using Postgres-XC

��

SQL Queries

XC Coordinator

SQL Queries

Data NodeData Node Data Node

Joins, sorts, aggregates

ManagerGlobal Transaction

Enables hashing of large tables, replication of others; supportsparallel-node consistent transactions and DDL

Scaling Using Foreign Data Wrappers

��

SQL Queries

Foreign Server Foreign Server Foreign Server

SQL Queries

PG FDW

No joins, sorts, aggregates

Requires rows to be hashed by key

Bulk Data Scaling Using Hadoop

��

Map/Reduce Jobs

Postgres Server

Hadoop

Postgres Server Postgres Server

Additional Resources…

◮ Postgres Downloads:

◮ www.enterprisedb.com/downloads

◮ Product and Services information:

◮ info@enterprisedb.com

Conclusion

http://momjian.us/presentations thttps://www.flickr.com/photos/blieusong/

postgres scaling opportunities and options

scalingdatabase scaling

cache bbuhorizontal

trainingi postgres

worldwide leader of

postgresql support

advanced server

sql server db2i

red hat enterprise linux

Technology

pros cons of database scaling options-final...

functions, scaling and options - aucotec · 2017-08-17 ·...

configuring edb postgres advanced server streaming...

postgres demystified

next generation postgres high availability · postgres-bdr3...

koichi suzuki - postgres-xc dynamic cluster management @...

pros cons of database scaling options-final...

using vmware vfabric postgres - vfabric postgres 9.3 · pdf...

postgres plus advanced server ecpg plus...

postgres plus cloud server getting started...

visualizing postgres - 日本postgresqlユーザ会 ·...

strategies and options for scaling up and enhancing the...

postgres enterprise manager installation guide · postgres...

postgres open, september 2014: postgres interface...

scaling aws redshift concurrency with postgres€¦ ·...

using vmware vfabric postgres - vfabric postgres 9.3.5

feasibility assessment of the options for up-scaling ... ·...

postgres open

poletti, t; balabanova, d; ghazaryan, o; kamal-yanni,...

edb postgres high availability and horizontal read scaling...