Download - Scaling PostreSQL with Stado
![Page 1: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/1.jpg)
Scaling PostgreSQL with Stado
![Page 2: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/2.jpg)
Who Am I?• Jim Mlodgenski
– Founder of Cirrus Technologies– Former Chief Architect of EnterpriseDB– Co-organizer of NYCPUG
![Page 3: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/3.jpg)
Agenda
• What is Stado?• Architecture• Query Flow• Scaling• Limitations
![Page 4: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/4.jpg)
What is Stado?
• Continuation of GridSQL
• “Shared-Nothing”, distributed data architecture.
–Leverage the power of multiple commodity servers while appearing as a single database to the application
• Essentially... Open Source Greenplum, Netezza or Teradata
![Page 5: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/5.jpg)
Stado Details
• Designed for Parallel Querying• Not just “Read-Only”, can execute
UPDATE, DELETE• Data Loader for parallel loading• Standard connectivity via PostgreSQL
compatible connectors: JDBC, ODBC, ADO.NET, libpq (psql)
![Page 6: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/6.jpg)
What Stado is not?
• A replication solution like Slony or Bucardo
• A high availability solution like Synchronous Replication in PostgreSQL 9.1
• A scalable transactional solution like PostgresXC
• An elastic, eventually consistent NoSQL database
![Page 7: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/7.jpg)
Architecture
• Loosely coupled, shared-nothing architecture
• Data repositories
–Metadata database
–Stado database
• Stado processes
–Central coordinator
–Agents
![Page 8: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/8.jpg)
Configuration
• Can be configured for multiple logical “nodes” per physical server
–Take advantage of multi-core processors
• Tables may be either replicated or partitioned
• Replicated tables for static lookup data or dimensions
–Partitioned tables for large fact tables
![Page 9: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/9.jpg)
Partitioning
• Tables may simultaneously use Stado Partitioning with Constraint Exclusion Partitioning
– Large queries scan a much smaller subset of data by using subtables
– Since each subtable is also partitioned across nodes, they are scanned in parallel
– Queries execute much faster
![Page 10: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/10.jpg)
Creating Tables
• Tables can be partitioned or replicated
CREATE TABLE STATE_CODES ( STATE_CD varchar(2) PRIMARY KEY, USPS_CD varchar(2), NAME varchar(100), GNISIS varchar(8)) REPLICATED;
![Page 11: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/11.jpg)
Creating Tables
CREATE TABLE roads (
gid integer NOT NULL,
statefp character varying(2),
countyfp character varying(3),
linearid character varying(22),
fullname character varying(100),
rttyp character varying(1),
mtfcc character varying(5),
the_geom geometry)
PARTITIONING KEY gid ON ALL;
![Page 12: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/12.jpg)
Query Optimization
• Cost Based Optimizer–Takes into account Row Shipping
(expensive)• Looks for joins with replicated tables
–Can be done locally–Looks for joins between tables on
partitioned columns
![Page 13: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/13.jpg)
Two Phase Aggregation
• SUM
–SUM(stat1)
–SUM2(SUM(stat1)
• AVG
–SUM(stat1) / COUNT(stat1)
–SUM2 (SUM(stat1)) / SUM2 (COUNT(stat1))
![Page 14: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/14.jpg)
Query 1
SELECT sum(st_length_spheroid(the_geom,
'SPHEROID["GRS_1980",6378137,298.257222101]'))/1609.344
as interstate_miles
FROM roads
WHERE rttyp = 'I';
interstate_miles
------------------
84588.5425986619
(1 row)
![Page 15: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/15.jpg)
Query 1 : Results
1 4 8 12 160
20
40
60
80
100
120
LinearActual
Nodes
Tim
e (
seco
nd
s)
Nodes Actual (sec)
1 101.2080566
4 25.6410708
8 14.3321144
12 5.4738612
16 4.8214672
![Page 16: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/16.jpg)
Query 2SELECT s.name as state, c.name as county, a.population, b.road_length,
a.population/b.road_length as person_per_km
FROM (SELECT state_cd, county_cd, sum(population) as population
FROM census_tract
GROUP BY 1, 2) a,
(SELECT statefp, countyfp,
sum(st_length_spheroid(the_geom, 'SPHEROID["GRS_1980",6378137,298.257222101]'))/1000 as road_length
FROM roads
GROUP BY 1, 2) b,
state_codes s, county_codes c
WHERE a.state_cd = b.statefp
AND a.county_cd = b.countyfp
AND a.state_cd = c.state_cd
AND a.county_cd = c.county_cd
AND c.state_cd = s.state_cd
ORDER BY 5 DESC
LIMIT 20;
![Page 17: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/17.jpg)
state | county | population | road_length | person_per_km
----------------------+-----------------+------------+------------------+------------------
New York | New York | 1537195 | 1465.35561969273 | 1049.02521909483
New York | Kings | 2465326 | 2785.37685011507 | 885.096032839562
New York | Bronx | 1332650 | 1638.47925579201 | 813.345665066614
New York | Queens | 2229379 | 4343.78066667893 | 513.234707521383
New Jersey | Hudson | 608975 | 1474.86512729116 | 412.902162191933
California | San Francisco | 776733 | 2125.05706617179 | 365.51159607175
Pennsylvania | Philadelphia | 1517550 | 5067.19918355051 | 299.484970894054
District of Columbia | Washington | 572059 | 2191.33029860109 | 261.055579054054
New York | Richmond | 443728 | 1758.77468237864 | 252.293829588156
Massachusetts | Suffolk | 689807 | 2805.37242915611 | 245.887851762877
New Jersey | Essex | 793633 | 3359.22581976629 | 236.254733257324
Virginia | Alexandria City | 128283 | 577.98117468444 | 221.950135434841
Puerto Rico | San Juan | 434374 | 1994.26820504899 | 217.811224638829
Virginia | Arlington | 189453 | 967.505165121908 | 195.816008874876
New Jersey | Union | 522541 | 2827.74655887522 | 184.790605919029
Maryland | Baltimore City | 651154 | 3707.01218958787 | 175.654669231717
Puerto Rico | Catano | 30071 | 174.765650431886 | 172.064704509654
Hawaii | Honolulu | 876156 | 5098.8482067881 | 171.834101441493
Puerto Rico | Toa Baja | 94085 | 558.532996996738 | 168.450208861249
Puerto Rico | Carolina | 186076 | 1122.20560229076 | 165.812752690026
(20 rows)
![Page 18: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/18.jpg)
Query 2 : Results
1 4 8 12 160
500
1000
1500
2000
2500
3000
3500
4000
4500
LinearActual
Nodes
Tim
e (
seco
nd
s)
Nodes Actual (sec)
1 3983.1002548
4 1007.1235182
8 563.6259202
12 365.152858
16 282.7345952
![Page 19: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/19.jpg)
Scalability
![Page 20: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/20.jpg)
Limitations
•• SQL Support–Uses its own parser and optimizer
so:• No Window Functions• No Stored Procedures• No Full Text Search
![Page 21: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/21.jpg)
Transaction Performance
• Single row Insert, Update, or Delete are slow compared to a single PostgreSQL instance
–The data must make an additional network trip to be committed
–All partitioned rows must be hashed to be mapped to the proper node
–All replicated rows must be committed to all nodes
• Use “gs-loader” for bulk loading for better performance
![Page 22: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/22.jpg)
High Availability
• No heartbeat or fail-over control in the coordinator
– High Availability for each PostgreSQL node must be configured separately
– Streaming replication can be ideal for this
• Getting a consistent backup of the entire Stado database is difficult
– Must ensure there are no transaction are occurring
– Backup each node separately
![Page 23: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/23.jpg)
Adding Nodes
• Requires Downtime
–Data must be manually reloaded to partition the data to the new node
• With planning, the process can be fast with no mapping of data
–Run multiple PostgreSQL instances on each physical server and move the PostgreSQL instances to new hardware as needed
![Page 24: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/24.jpg)
Summary
• Stado can improve performance tremendously of queries
• Stado can scale linearly as more nodes are added
• Stado is open source so if the limitations are an issue, submit a patch
![Page 25: Scaling PostreSQL with Stado](https://reader034.vdocument.in/reader034/viewer/2022051514/54875305b47959ec0c8b544a/html5/thumbnails/25.jpg)
Download Stado at:http://stado.us
Jim MlodgenskiEmail: [email protected]: @jim_mlodgenski
NYC PostgreSQL User Grouphttp://nycpug.org