PostgreSQL Magazine

Interview

#01

10 awesome new features

News and stories from the Postgres Community

NoSQL : The Key Value store everyone ignored

May 2012

is out !9.1PPoossttggrreeSSQQLL

Stephan Kaltenbrunner

Opinion : Funding PostgreSQL featuresWaiting for 9.2 : Cascading Streaming Repl ication

PostgreSQL in Mac OS X LionTips & tricks

About PG Mag

PostgreSQL Magazine is edited by and for thePostgreSQL Community.

Editor: Damien Clochard

Layout Editor: Cédric Gemy

Writers:Zohaib Sibte Hassan, Marc Cousin, SimonMerrick, Joshua Drake, Hubert Lubaczewski

Reviewers:Thom Brown, Ray O'Donnel l , Osvaldo Tul ini ,Sivakumar, Matt Keranen, Ja ime Casanova,Luca Ferrari , N icolas Thauvin, Bob Hetrick, JoshKupershmidt

Tools: Scribus 1.4 / Gimp 2.6

License:

The articles contained in this magazine arereleased under the Creative CommonsAttribution-ShareAl ike 3.0 Unported l icense. Thismeans you may adapt, copy, distribute andtransmit the articles but only under thefol lowing conditions: You must attribute thework to the original author in some way (at leasta name, emai l or URL) and to this magazine byname ('PostgreSQL Magazine') and the URL(pgmag.org). You cannot attribute the article(s)in any way that suggests that they endorse youor your use of the work. If you alter, transform,or bui ld upon this work, you must distribute theresulting work under the same, simi lar or acompatible l icense.

Disclaimer:

PostgreSQL Magazine is an independent media.The views and opinions in the magazine shouldin no way be assumed to be endorsed by thePostgreSQL Global Development Group. Thismagazine is provided with absolutely nowarranty whatsoever; neither the contributorsnor PostgreSQL Magazine accept anyresponsibi l i ty or l iabi l i ty for loss or damageresulting from readers choosing to apply thiscontent to theirs or others computers andequipment.

EditorialThis is it! The first issue of the first non-profit print mediacreated by and for the Postgres community. Time went bysince our demo issue last year! Overal l we printed 2000+ hardcopies (distributed for free in events l ike PG Con 2011 inOttawa or FOSDEM 2012 in Brussels). We also know that so farthe online PDF version was read by 10000+ visitors. So wetook some time to analyze the feedback we received and welearned many things. Many thanks to every one that sent usfeedback.

For me, the great news in al l this is that we proved that wecan run a print media as an open source community-drivenproject! We all know that the PostgreSQL community canproduce robust source code and quality documentation. Butcould we use this strong energy to generate something else?Why not a media aimed at the PostgreSQL user base? In lessthan a year, what was just a theoretical question has becomereal ity.

In this first issue, we wil l talk of the wonders of PostgreSQL9.1 and we'l l get a sneak peek at the upcoming 9.2 version .We'l l take a look at Mac OSX Lion and we'l l ask StephanKaltenbrunner a bunch of questions, one of the sysadminsbehind the postgresql .org infrastructure… We really hope youwil l find ideas and insights in this magazine!

This time, dozens of benevolent contributors made this issuepossible: writers, editors, reviewers, etc. Kudos to them! Wetry to work just l ike any “regular” open source project: contentis edited in wiki , the source fi les are stored in github and wehave a contributors' mai l ing l ist. I f you want to join us, justcheck out page 35 to see how you can help.

And final ly, I would l ike to thank our benefactors… Fotol ia.comhas offered us a free subscription plan to access their stockphoto database. We also received funding from PostgreSQLEurope (PGEU) and Software in the Public Interest (SPI ). Thisfunding wil l al low us to release this issue and the next one.

For the new issue, we plan on sending more free copies tomany PostgreSQL conferences and meetings around the world.So if you're organizing a PostgreSQL related event, just send usa message at contat@pgmag.org and we'l l try to provide youwith hard copies of this magazine.

Damien ClochardEditor in Chief

4 6 8

14

28

12

26 32

PostgreSQL Magazine

AgendaConferences & Events

AnnouncementsProduct News

InterviewStephan Kaltenbrunnerthe admin behind the scenes

PPoossttggrreeSSQQLL 99.. 111100 nneeww aawweessoommee ffeeaa ttuu rreess

NoSQL ?The key value store everyoneignored

Waiting for 9.2Cascading streamingreplication

OpinionFunding PostgreSQL features

Tips & TricksElephants and l ions

01TABLE OF CONTENTS

4

PG Mag #01

Agenda Upcoming Events4 Agenda

This is it! The first issue of the first non-profit print mediacreated by and for the Postgres community. Time went bysince our demo issue last year! Overal l we printed 2000+ hardcopies (distributed for free in events l ike PG Con 2011 inOttawa or FOSDEM 2012 in Brussels). We also know that so farthe online PDF version was read by 10000+ visitors. So wetook some time to analyze the feedback we received and welearned many things. Many thanks to every one that sent usfeedback.

For me, the great news in al l this is that we proved that wecan run a print media as an open source community-drivenproject! We all know that the PostgreSQL community canproduce robust source code and quality documentation. Butcould we use this strong energy to generate something else?Why not a media aimed at the PostgreSQL user base? In lessthan a year, what was just a theoretical question has becomereal ity.

In this first issue, we wil l talk of the wonders of PostgreSQL9.1 and we'l l get a sneak peek at the upcoming 9.2 version .We'l l take a look at Mac OSX Lion and we'l l ask StephanKaltenbrunner a bunch of questions, one of the sysadminsbehind the postgresql .org infrastructure… We really hope youwil l find ideas and insights in this magazine!

This time, dozens of benevolent contributors made this issuepossible: writers, editors, reviewers, etc. Kudos to them! Wetry to work just l ike any “regular” open source project: contentis edited in wiki , the source fi les are stored in github and wehave a contributors' mai l ing l ist. I f you want to join us, justcheck out page 35 to see how you can help.

And final ly, I would l ike to thank our benefactors… Fotol ia.comhas offered us a free subscription plan to access their stockphoto database. We also received funding from PostgreSQLEurope (PGEU) and Software in the Public Interest (SPI ). Thisfunding wil l al low us to release this issue and the next one.

For the new issue, we plan on sending more free copies tomany PostgreSQL conferences and meetings around the world.So if you're organizing a PostgreSQL related event, just send usa message at contat@pgmag.org and we'l l try to provide youwith hard copies of this magazine.

Damien ClochardEditor in Chief

PG Day FranceJune 7 / Lyon, France

UUPPCCOOMMIINNGG EEVVEENNTTSSPGCon 2012May 15-18 / Ottawa, Canada

Postgres Open is a non–profit, community–runconference series in the United States focused onbusiness users, database professionals anddevelopers of PostgreSQL. Postgres Open wi l lpromote the business of PostgreSQL as wel l as itsuse and development.

http: //postgresopen.org/

Postgres Open 2012September 16-19 / Chicago, USA

PGCon is an annual conference for users anddevelopers of PostgreSQL. PGCon is the placeto meet, discuss, bui ld relationships, learnvaluable insights, and general ly chat about thework you are doing with PostgreSQL.

http: //www.pgcon.org/2012/

The major French-speaking PostgreSQLConference is back this year with some greattalks  : PostGIS 2.0, fuzzy logic in PostgreSQL, High-Avai l ibi l i ty, Benchmarking, Foreign Data Wrappersand more !

http: //www.pgday.fr/

PG Conference Europe 2012October 23-26 / Prague, Czech Rep.This year, the major European PostgreSQLconference wi l l take place at the Corinthia hotelin Prague. You can expect three days packed withpresentations about PostgreSQL and relatedtechnologies, one day of extended trainings, andthe usual hal lway and social track!

http: //2012.pgconf.eu/

For more information on local and internationalPostgreSQL user conferences, check out the wiki !

http: //wiki .postgresql .org/wiki/Events

5

PG Mag #01

What are your plans?

Dear people from PostgreSQL Magazine,

We (database people of the City of Vienna) haveread (part of) the pi lot issue and this magazinewould be interesting for us.

Do you have any ideas how often you plan to publ ishthe magazine? Wil l i t be possible to subscribe? Doyou have any estimates how much it might cost?

Even if you have no definite answers to any of theabove yet, we thought it would be a good idea togive you feedback.

Yours, Laurenz Albe

The editors reply: For now we don't know howoften we wi l l publ ish and how much it wi l l cost. I t isnot possible to subscribe to the magazine right nowbut we're working on this. Al l these questions arecurrently being discussed. Fol low us on twitter orsubscribe to our newsletter to stay tuned!

http: //twitter.com/#! /pg_maghttp: //pgmag.org/join_the_newsletter

Online version not universallyreadable…

… due to requiring some kind of proprietary plugin.How about l imiting the del ivery technology to somefreely avai lable formats?

I wi l l of course download and read the PDF, but itdoes feel l ike I 'm excluded. The only other onl inepubl ication I pay money for is LWN, and I paybecause of its *content*. In fact, given its technicalaudience its simpl icity of design is probably anattraction.

Perhaps I 'm not part of the mainstream targetaudience? Aiming more at the corporate crowd whoneed flashy colors? Dunno…

Regards,

Mark Lawrence

The webmaster replies: I think you'retalking about the onl ine PDF reader that requires aflash plugin. I know this is a proprietary format butyou should be able to read it with free alternativesto the Adobe plug-in…Have you tried gnash orl ightspark ?Sti l l the Flash format is not open. I know that too. Fornow, I don't know any way to display a magazineonl ine with HTML5. I know scribd does it for sl idesbut this is not appropriate for a magazine.I f you have any idea on how we can improve that,I 'm open to your suggestions.

PostgreSQL magazine on tablets

Hi,

I am thri l led to see a magazine for PostgreSQL. I l ikethe format and the open source experiment aroundthe magazine. I found the announcement via thehacker news. So I am sure the news about thepubl ication wi l l be viral very soon.

It would be great if the magazine is made avai lableon the iPad too. I am sure there are others whowould l ike to read on other ereaders l ike kindle ornook. I use iBooks, Fl ipboard, and Zinio to primari lyread articles and magazines on the iPad. I especial lylove the fl ipboard interface to read articles on theiPad.

I know that the PDF can be read on the iPad or othertablets. However, it’s a bit inconvenient, since I needto pul l i t manual ly. Having the magazine pushed tomy device(s) would save me a lot of trouble. Pushingit to my device wi l l a lso make it easy for me toaccess the magazine, especial ly when I am notparticularly busy and want to catch up on my newsand magazine reading.

Thanks for the great work so far.

John

The editors reply: Right now we're focusingon the paper version. We're happy to provide anonl ine version but this is not our main goal . Behindthe word “tablet”, there are so many devices andformats! Releasing the magazine for each tabletwould take us too long. This is why we're going tostick with the PDF format for a whi le.

UUPPCCOOMMIINNGG EEVVEENNTTSS

Q & A

From Our

ReadersCorrections, ideas and requests:

this section is the place for

telling us what you think of each

issue of PostgreSQL Magazine!

Send us feedback !

Send your opinion about what you read inPostgreSQL Magazine to feedback@pgmag.org.Letters may be edited for length and clarity. Weconsider that any message we receive can bepubl ished.

6

PG Mag #01

AANNNNOOUUNNCCEEMMEENNTTSSPostGIS 2.0 is out !

PP ostGIS, a project of the Open Geo-

spatial Consortium , hit version 2.0in Apri l 2012. This PostgreSQL extensionstores spatial data and al low users tosearch that data based on geographic crite-ria as well as on standard relational criteria.

TT he development process for 2.0 hasbeen very long, but has resulted in a

release with a number of exciting new fea-tures, such as: Raster data and raster/vec-tor analysis in the database, Topologicalmodels, 3D and 4D indexing, Integrationwith the PostgreSQL extension system,Multi -fi le import support in the shapefi leGUI , and more…

http://www.postgis.org/

Postgres-XC 1.0 Beta 1 released

PP ostgres-XC is a write-scalable multi-master symetric cluster based on Post-

greSQL. This beta version is based on Post-greSQL 9.1 Beta 2 and al l the fixes ofPostgreSQL 9.1 stable branch wil l be back-ported in Postgres-XC 1.0 stabi l ized version.It is distributed under PostgreSQL license.

http://postgres-xc.sourceforge.net

EnterpriseDB launches Cloud Database

EE nterpriseDB announced the generalavai labi l i ty of PostgreSQL 9.1 and

Postgres Plus® Advanced Server 9.0 on theAmazon Web Services platform.

http://cloud.enterprisedb.com

7

PG Mag #01

News

PGXN Client 1.0

TT he PGXN Cl ient is a command l ine tooldesigned to interact with the PostgreS-

QL Extension Network al lowing searching,compi l ing, instal l ing, and removing exten-sions in a PostgreSQL instal lation or data-base.

http://pgxnclient.projects.postgresql.org/

Skytools 3.0

SS kytools is developed by Skype for re-pl ication and fai lover. I t includes a ge-

neric queuing framework (PgQ) and easy-to-use repl ication implementation (Lon-diste). The major new features comparedto 2.1 are : Cascaded queuing, Paral lel co-py, Multi -database ticker and Londistehandler modules for custom processing.

http://wiki.postgresql.org/wiki/SkyTools

VMware Expands vFabric with PostgreSQL

vv Mware created vFabric Postgres as aclosed-source fork of PostgreSQL

created by VMware, with significant impro-vements for the vSphere environment. vFa-bric Postgres' “elastic database memory”dynamical ly adapts to changing workloadsto achieve greater memory efficiency andhigher consol idation ratios. vFabric Postgresis also avai lable on http: //cloudfoundry.com/.

PostgreSQL in Google Summer of Code

GG oogle has once again selected thePostgreSQL project to take part in

Summer of Code, during 2012. Google wi l lbe funding several students to work withmentors from our project in order to hackPostgreSQL code for the summer.

http://www.postgresql.org/developer/

Scalr 2.5 supports PostgreSQL

OO pen-source project Scalr is very plea-sed to announce its support for Post-

greSQL. From now on, Scalr and PostgreSQLusers can enjoy low maintenance on thecloud — including automatic backups, re-covery, and ease of administering, alongwith added auto-scal ing.

http://scalr.net/features/databases/postgres

Heroku Launches

PostgreSQL Standalone Service

SS ince 2007, PostgreSQL has been avai-lable as module inside the Heroku

Platform. But now Postgres is also avai-lable as a standalone service. Heroku isprobably the world largest PostgreSQLhosting service with more than 150 000PostgreSQL databases up and running.

https://postgres.heroku.com/

Npgsql 2.0.12 beta3 released

NN pgsql is a .Net data provider written100% in C# which al lows .Net pro-

grams to talk to PostgreSQL backends.Npgsql is l icensed under BSD.

http://www.npgsql.org

Postgres Enterprise Manager v2.1

EE nterpriseDB released a new version oftheir management tool with SNMP and

emai l a lerts, an Audit Manager and useraccess controls. Postgres Enterprise Mana-ger (PEM) 2.1 adds a series of highly confi -gurable dashboards, new features forschedul ing and organizing managementtasks and enhanced alerting capabi l i ties.

8

PG Mag #01

The admin behind the scenesYou probably don't know his name, even if you're a Post-greSQL zealot. But Stephan Kaltenbrunner (also known as“Mastermind”) is a key member of the sysadmin team thatbuilds and maintains the postgresql.org infrastructure.This tall and quiet Austrian guy has been involved in theweb platform for years now, so we've decided to ask himabout the evolution of the project through the last decadeand the future of PostgreSQL's web presence: the newPGXN network, the death of pgFoundry and the recent up-grade of the front website….

IINNTTEERRVVIIEEWW

9

PG Mag #01

Interview

PostgreSQL Mag:Who do you work for?

Stefan Kaltenbrunner: I work for ConovaCommunications GmbH in Salzburg, Austria as theteam leader for the Engineering & Consulting de-partment. Conova is a highly IT-Service companyfocusing on providing high-qual ity services out ofits own large datacenter in Salzburg, and almostany internal or customer facing-project is runningPostgreSQL in the backend.

PGM: How are you involved in thePostgreSQL project?

SK: Aside from occasional ly giving talks, proba-bly the most visible (or invisible depending on howyou look at it) thing is my involvement in the post-gresql .org sysadmin and web teams. PostgreSQLis a huge, wel l establ ished and wel l respected opensource project, and as part of being a real opensource project we also run our own infrastructure,ful ly community managed and operated.

PGM: Is it hard to be an admin ofpostgresql.org? How many hoursper week do you spend on this?

SK: Heh, “how hard is it?” — that's a real ly goodquestion. Back in the '90s postgresql .org used torun on a single FreeBSD server sponsored andoperated by hub.org. This was later expanded to3 or 4 servers but it was sti l l a smal l environmentthat was adequate to the needs of the projects 10-15 years ago.

However over the past 5-10 years with PostgreS-QL getting rapidly developed, improved and at-tracting a larger community — both in terms ofdevelopers, contributors and users — the post-gresql .org infrastructure grew. The community gotnew servers sponsored by companies other thanhub.org; sti l l inheriting a lot of the FreeBSD pastbut it helped to get new people involved with thetopic.

Over the last few years the sysadmin team grewwith even more serious server sponsors and a si-gnificant increase in what people wanted the in-frastructure to del iver. We started working on acompletely new concept for how we as a teamcould successful ly design and run the postgres-ql .org infrastructure for the next 5-10 years, loo-king at new technologies and learning from pastexperience.

The end result is, in my opinion, a nice reflectionon the rapid evolution and the technical excel-lence of PostgreSQL itself. We now run almost al lservices using cutting edge, KVM-based ful l vir-tual isation technology and we also have a very so-

phisticated custom deployment and managementframework that al lows us to do most stuff in mi-nutes. The system as a whole has a very high le-vel of automation. Newly instal led VMs or evenservices added to existing ones are ful ly functio-nal and automatical ly backed up, and monitoringand patching for security issues is now a matterof seconds which makes managing our 40+ VMsas wel l as the VM-Hosts much easier and moresustainable.

PGM: How many other sysadminswork on the infrastructure?

SK: The “core infrastructure team” — rather loo-sely defined here as the people with root on al lVMs and the VM-Hosts and actively working onthem — consists of four people: Magnus Hagan-der, Dave Page, Alvaro Herrera and me.

However there are a larger number of people withelevated permissions on specific VMs, or withcommit access to parts of the underlying code —a quick l ist of the people involved can be found onhttp: //wiki .postgresql .org/wiki/Infrastructure_team.Work on the infrastructure is — simi l iar to most ofthe PostgreSQL ecosystem — coordinated throughmai l ingl ists, IRC and for emergencies IM.

PGM: How many services are run-ning?

SK: I t depends on how one defines a service.Most of the publ icly facing services we providewhich present the “face” of PostgreSQL, l ike thewebsite, are actual ly composed of a number ofdifferent servers running dozens of services. Rightat this moment we have exactly 571 services mo-nitored in nagios on a total of 61 hosts not inclu-ding al l the stuff we have in place for long-termcapacity planning and trending that is not direct-ly alerting which adds another 250-350 services.I t is important to mention that almost al l of theseare ful ly deployed automatical ly — so if somebo-dy adds a PostgreSQL server instance on anexisting box, monitoring and service checking wi l lbe ful ly automatical ly enabled.

“At this momentwe have exactly571 services

monitored in nagioson a total of61 hosts”

10

PG Mag #01

PGM: How many servers are usedto run those services?

SK: We try keeping an up-to-date l ist of physicalservers on http: //www.postgresql .org/about/ser-vers/, so at the time of writing we had ten VM-Hosts in five different datacenters in the US andEurope. In addition to that we have a number ofadditional servers, not l isted there on purpose,used for very specific tasks l ike running our moni-toring system or our central ised backup system.

PGM: Who donates these servers?

SK: Companies using PostgreSQL or providingServices based on it basical ly. Years ago we had torely on very few companies giving us gear they nolonger needed. These days we actual ly evaluatethe hardware we are getting offered seriously; welook at the kind of datacenter the server might behosted at and what kind of support we can getfrom that place in terms of remote hands, res-ponse times and bandwidth as wel l as the hard-ware itself. In recent times we are also starting tolook into buying hardware ourselves to provide aneven better user experience and to improve therel iabi l i ty of our services by having less dependen-cy on somebody else to provide us with spareparts or a replacement in short time.

PGM: How do you monitor this?

SK: In two words: Nagios and Munin. In somemore, we use Munin for trending, Nagios for aler-ting and monitoring external ly through NRPE. Wealso have a custom-developed configuration fi letracking system for keeping critical configurationfi les across the infrastructure properly versioncontrol led. The alerting is usual ly happeningthrough emai l d irectly to the individual sysadminmembers.

PGM: You seem to know Nagios ve-ry well. How would you compare itto other monitoring software suchas Zabbix or Hyperic?

SK: Indeed I know Nagios pretty wel l , though inrecent times I have seen a lot of movement to lookinto Icinga, and also into using some of the alter-native user interfaces provided by some projects.My personal favourite there is Thruk, which ismaybe a bit boring for the Web 2.0 generation butfor me it has the right balance of simpl icity andclarity l ike the original Nagios CGI , whi le providingsome very useful capabi l i ties on top - l ike very po-werful fi l tering & view capabi l i ties that areextremly handy in instal lations having a few thou-sand or tens of thousands of services.

PGM: The sysadmin team per-formed an upgrade of the mainwww.postgresql.org platform inNovember 2011. Can you tell us alittle more about that new version?

SK: That was a huge thing; the existing websiteframework was developed l ike 10 years ago bycommunity members based on PHP4 that has nowmoved on to other things. And over the past fewyears we have only been making modest changesto it because people never did ful ly understand it,and the number of PHP-experienced developers wi-thin the community interested in hacking on it was,say, “ l imited”. I t is mostly thanks to Magnus Ha-gander — who implemented most of the new frame-work — that we now have a pretty nice and scalablePython and Django-based framework. The newframework ful ly integrates Varnish as a front-endcache, Django in the backend and a myriad of whatI would cal l “community platform services” l ike:

➥ complete community account management —providing single-sign-on for al l community ser-vices

➥ management interfaces for news, organisations,events and training for the main website (inclu-ding automatic purging of cached content onthe frontend systems)

➥ RSS-feed generation and aggregation

➥ management of dynamic content l ike surveys

➥ scalabi l i ty

We also completely revamped our downloadsystem in the process; this now means that we arehosting al l downloads, except for the one-cl ickinstal lers, by ourselves on servers operated by theinfrastructure team. The old and often hated page,with the enormous amount of flags for the manycountries which we previously used mirrors, is nowgone for good.

“we have outgrown whatpgFoundry can really dofor us and I think weshould move on”

Interview

11

PG Mag #01

PGM: One word about pgFoundry,the PostgreSQL software forge….There's been some complaintsabout it. We see lots of projets mo-ving to other code sharing plat-forms (such as github) and somecommunity members even thinkpgFoundry should be closed.What's your opinion?

SK: Once upon a time pgFoundry (and gborg forthose who remember) were a very important ser-vice for the community. When those services werestarted, getting proper hosting was expensive andcomplex to get, especial ly for the myriad of smal-ler contributors in the PostgreSQL ecosystem. Ho-wever, time has evolved, getting proper hosting isnow reasonably easy to get in most places andthere is a large set of very successful open sourcefocused hosting systems. Contributors can choosefrom those that are very actively developed andimprove at an amazing pace. The only real missingfeature most people have with those is propermai l ingl ist support. The PostgreSQL community isusing mai l ingl ists as communication and coordina-tion media in a much more intense form than otherprojects (except maybe the Linux kernel or the De-bian project as a whole) but most of the code-sha-ring and hosting platforms have no or only veryl imited support for mai l ingl ists. However we wi l l beworking with the projects affected by the upco-ming pgFoundry shutdown on this and I expectthat we wi l l be finding a solution in most cases.

To sum this up — pgFoundry was a very importantpart of the infrastructure in the past in evolvingand growing the community. However PostgreSQLis a very large and wel l establ ished project with arapidly growing community and we have now evol-ved to a point where we have outgrown whatpgFoundry can real ly do for us. And I think weshould move on and work on stuff we are real lygood at — running the core and mission critical in-frastructure for a major open source project: Post-greSQL itself.

PGM: And what do you think of thenew PostgreSQL Extension Network(PGXN)?

SK: With my community hat on I see this as agreat thing to provide a lot more exposure to thegreat ecosystem around PostgreSQL and also seeit as a showcase of one of the most powerful , butoften overlooked capabi l i ties of PostgreSQL; “Ex-tensibi l i ty”. With my hat as a manager and teamlead at Conova “on” I think that it is hard to dealwith PGXN (or the concept it is loosely modeledaround CPAN) in a complex production scenario

usual ly requiring instal lation from binary packagesor complex approval procedures for source codeinstal lations. PGXN is a clear and nice example ofwhat the community is capable of pul l ing off thesedays, but as with a lot of services in the world wideweb - only time wi l l tel l how successful it wi l l be inthe long run : )

PGM: What's the biggest featureyou'd want to see in the next Post-greSQL release?

SK: There are three large features actual ly:

➥ scalabi l ity — on large systems, thanks to a lot ofwork from Robert Haas, 9.2 wil l scale extremelywell on modern boxes so for reads I consider thismostly done, for writes we sti l l have ways to go

➥ better partitioning — the current system based oninheritence was nice to get us started and was suffi -cient for the time being, but going forward we pro-bably need a radically new design embedded muchdeeper in the backend than what we have now

➥ parallel query — the industry as a whole is movingto more and more cores per socket at an amazingpace right now (to the point that commercial ven-dors need to update ther licencing on an almostyearly basis now), and we will have to take on thechallenge of parallel ising operations over multiplecores sooner rather than later

PGM: What other things are youinterested in outside of opensource and PostgreSQL?

SK: There is something other than open sourceand PostgreSQL?! No seriously, I l ike spending timewith my fami ly as wel l as going out for cinema orplaying cards with friends and if that gets coupledwith a good glass of Austrian white wine I 'm evenhappier…

PG Conference Europe, Amsterdam 2011

12

PG Mag #01

NoSQL ?

The keyvalue store everyone ignoredYes I know you are really happy with your “persistent” Key Value store. But

did anybody notice hstore that comes with PostgreSQL ?

I find PostgreSQL to be a real ly great RDBMS thatis overlooked al l the time. It even has a greatpubl isher/subscriber system (or LISTEN/NOTIFY interms of PostgreSQL) that a lot of people may haveimplemented using Redis, RabbitMQ etc. Forpeople who have not l ived anything other thanMySQL, I would simply ask them to try outPostgres.

Instead of looking at benchmarks, I wi l l be focusingon a key value store that is ACID compl iant for real !Postgres takes advantage of its storage engine andhas an extension on top for key value storage. Sothe plan is to have a table with column(s) that havea datatype of hstore, which in turn has a structure-free storage. Thinking of this model multipleanalogies throw themselves in. I t can be a ColumnFamily Store just l ike Cassandra where row key canbe PK of the table, and each column of hstore typein the table can be thought of l ike a super columnand each key in the hstore entry can be a column

name. Simi larly you can imagine itsomewhat l ike Hash structures inRedis (HSET, HDEL) or 2 or 3 levelMongoDB store (few modificationsrequired). Despite being simi lar(when l ittle tricks are appl ied) toyour NoSQL store structures, thisgives me an opportunity todemonstrate to you some real lytrivia l examples.

Let's setup our system first. For my experiment I wi l lbe using Postgres 9.1 and I wi l l compi le it fromsource. Once inside the source directory you can:. /configure && make install to instal l yourPostgres. Don't forget to instal l the extensions in thecontrib directory: cd . /contrib && make install .Once you have it set up, you can create your owndatabase cluster and start the server (Hints: useinitdb and pg_ctl ) . Then launch psql and make sureyou instal l your hstore extension:

CREATE EXTENSION hstore;SELECT ' foo=>bar' : : hstore;

I f everything goes wel l you should be able to see atable output. Now we are ready to do some DDL. Icreated a table named my_store as defined below:

CREATE TABLE my_store(id character varying( 1024) NOT NULL,doc hstore,CONSTRAINT my_store_pkey PRIMARY KEY ( id))WITH (OIDS=FALSE) ;

CREATE INDEX my_store_doc_idx_gistON my_storeUSING gist( doc) ;

As you can see I 've created a table with a hstorecolumn type and one GiST index (for operators ?,?&, ?| etc. ) . You can check out the documentationto have a look at the different types of operatorsavai lable. Now that I have a database and tablesset up, I use a simple script to populate it withabout 115K rows from a Twitter stream. Now keepin mind that it's real l i fe data and I was interestedin querying a few basic things from the col lecteddata. For example, how many people are usinghash tags, or doing mentions, or were posting l inksin the tweets? For doing this I wrote a simplepython script using tweepy and psycopg2 and ranit for few hours. For each tweet in my store, I addeda key value pair of ‘has_hashtags⇒:t’ i f there wereany hash tags in the tweet. Simi larly I introduced‘has_urls’ and ‘has_mentions’ if they were presentin tweet. I wi l l be using these keys along with myGiST index to query my table later on.

13

PG Mag #01

NoSQL ?

“Combining the powerof relational and keyvalue style under one

store”

So, after populating my data with 115,142 tweetsthe database grew to a size of 239691780 bytes(just 228MB). Now comes the fun part. I was total lyblown away by what I can achieve by combiningthe power of relational and key value style underone store. So, for example I want to query al l thetweets tweeted at unix timestamp of 1323446095(since I stored the timestamps as a string here iswhat my query looks l ike):

SELECT doc -> ' text' as tweet, doc -> ' created_at'as created_atFROM my_storeWHERE doc @> ' created_at=>00001323446095' ;

I can add a simple count or any other wel l -knownSQL aggregate function without compl icating mydata store with anything specific l ike map reduceor the hassle of learning a new language. Note thatI padded my timestamp value with zeros since Iam only storing strings as values. Also I am uti l izingthe @> operator, and that's going to use GiST toperform a quick bitmap index scan instead of asequential scan. That's pretty good for starters.Let's try to fetch out al l the tweets that had hashtags in them:

SELECT doc -> ' text' as tweet, doc -> ' created_at'as created_atFROM my_storeWHERE doc @> ' has_hashtags=>: t' ;

Yes querying the entire database and pul l ing outthe complete data (that you won't probably dobecause you page the data) gives me 14689 rowsin just under 360ms on average. Since we haveSQL to hand let's make the query a l ittle morecompl icated, and use a different operator for thesame stuff and also sort the data by ‘created_at’ :

SELECT doc -> ' text' as tweet, doc -> ' created_at'as created_atFROM my_storeWHERE doc @> ' has_hashtags=>: t' AND doc ?' has_urls'ORDER BY doc -> ' created_at' DESC;

This sounds tasty! But there's more: Postgres hasother operators. For example, you can also pul l outhash tagged tweets with urls or mentions:

SELECT doc -> ' text' as tweet, doc -> ' created_at'as created_atFROM my_storeWHERE doc @> ' has_hashtags=>: t' AND doc ?|ARRAY[ ' has_urls' , ' has_mentions' ]

This is not al l you can do with it! hstore comes withal l sorts of operators and index systems that youcan use to query hash stores. Check them out inthe documentation. Now, despite the NoSQL boomI think we have some great examples and reasonsof why RDBMS sti l l remains a core part of manymarket giants (Facebook being something every-one knows). Postgres just gives me one morereason to not ignore RDBMS. So, if you have beenmoving to some document stores just for thereason that RDBMS doesn't provide them; thinkagain! You can get the same rock sol id durabi l i tywith structure-free systems.

About the author

Zohaib Sibte Hassan (akamaxpert) is a contributor to DooPHPand creator of micro frameworkMiMViC. He works at Bumpin Social Media asSenior Software Engineer. He's been bustingbuzzwords on NoSQL, NodeJS, and various otherstuff. Checkout his blog athttp://blog.creapptives.com/

http: //pgmag.org/0113

About the article

The originia l article is avai lable at

Note

Prior to PostgreSQL 8.2, the containmentoperators @> and <@ were cal led @ and ~,respectively. These names are sti l l avai lable,but are deprecated and wil l eventual ly beremoved. Notice that the old names arereversed from the convention formerly fol lowedby the core geometric data types! Please alsonote that the ⇒ operator is deprecated andmay be removed in a future release. Use thehstore(text, text) function instead.

14

PG Mag #01

9.110PostgreSQL 9.1

AWESOME new

features

in

PostgreSQL

14

15

PG Mag #01

In September 2011, The PostgreSQL Global Deve-lopment Group released PostgreSQL 9.1. This latestversion of the leading open source database offersinnovative technology, unmatched extensibi l i ty, andnew features.

Responding to UsersVersion 9.1 del ivers several features which usershave been requesting for years, removing road-blocks to deploying new or ported appl ications onPostgreSQL. These include synchronous repl ication,per-column col lations and unlogged tables.

Advancing the State of the ArtAs always the PostgreSQL community innovates withcutting-edge features! Version 9.1 includes 4 itemswhich are new to the database industry, such as: Knearest neighbor (KNN) indexing, serial izable snapshotisolation (SSI), writeable common table expressions(wCTE) and security-enhanced (SE) PostgreSQL.

Extending the Database EnginePostgreSQL's extensibi l i ty enables users to add newfunctional ity to a running production database, anduse them for tasks no other database system canperform. Version 9.1 adds new extensibi l i ty tools,including: SQL/MED support, Extensions and thePostgreSQL extensions network (PGXN).

In PostgreSQL's 25th year of database development,the community continues to advance databasetechnology with every annual release!

“PostgreSQL 9.1 provides some of the mostadvanced enterprise capabilities of any opensource database, and is backed by a vibrant andinnovative community with proven customersuccess. PostgreSQL is well positioned forbuilding and running applications in the cloud,”said Charles Fan, Sr. VP R&D, VMware.

SQL/MED 16

Per-column collation 17

Unlogged tables 18

Synchronous replication 19

Extensions 20

KNN indexing 21

PGXN 22

SE PostgreSQL 23

SSI 24

Writeable CTE 25

TThhee llaatteesstt mmaajjoorr vveerrssiioonn ooff PPoossttggrreeSSQQLL hhaass ssoo mmaannyy iinnnnoovvaattiioonnss aanndd

iimmpprroovveemmeennttss !! WWee rreeaallllyy ssttrruugggglleedd ttoo cchhoooossee tthhee tteenn mmoosstt ddiissrruuppttiivvee

ffeeaattuurreess aanndd sshhoowwccaassee eeaacchh ooff tthheemm,, wwiitthh aa sstteepp--bbyy--sstteepp eexxaammppllee.. LLeett'' ss

sseeee wwhhaatt PPoossttggrreeSSQQLL 99.. 11 iiss ggoonnnnaa cchhaannggee ffoorr yyoouurr ddaattaa sseevveerrss !!

PostgreSQL 9.1

PostgreSQL 9.1

Check out the wiki

This article is heavi ly based on the wiki page cal led“What's new in PostgreSQL 9.1?”.

http: //www.pgmag.org/0115

About the author

Marc Cousin has been a PostgreSQLand Oracle DBA since 1999. He isinvolved in the Postgres community in variousways, including providing help on the French-speaking forum and participating in the PG DayFrance. He works at Dal ibo as a Senior DBA.

16

PG Mag #01

SQL/MEDSQL/MED (for Management of External Data) is an

extension to the SQL:2003 standard that provides

extensions to SQL that define foreign-data wrappers

(FDW) and datalink types to allow SQL to access data

stored outside of the RDBMS.

PostgreSQL 9.1 already has a bunch of FDW avai-lable  : Oracle, MySQL, SQLl ite, anything providing anODBC driver, LDAP, couchdb, redis, Amazon S3storage, CSV fi les, Twitter, Google Search, HTMLpages, RSS feeds, etc. There is even an extensionthat aims to make FDW development easier whendeveloping them in Python, Multicorn:http: //multicorn.org/

For an almost complete l ist, check out thePostgreSQL wiki :

http: //wiki .postgresql .org/wiki/Foreign_data_wrappers

Here is an example, using the file_fdw extension.

We' l l map a CSV fi le to a table.

=# CREATE EXTENSION file_fdw WITH SCHEMA extensions;\dx+ file_fdw

Obj ects in extension " file_fdw"Obj ect Description

----------------------------------------------------foreign-data wrapper file_fdwfunction extensions. file_fdw_handler( )function extensions. file_fdw_validator(text[ ] , oid)

This next step is optional . I t's just to show the'CREATE FOREIGN DATA WRAPPER' syntax:

=# CREATE FOREIGN DATA WRAPPER file_data_wrapper-# HANDLER extensions. file_fdw_handler;CREATE FOREIGN DATA WRAPPER

The extension already creates a foreign data wrappercal led file_fdw. We' l l use it from now on.

SQL/MED makesPostgreSQL a powerfultool for enterprisedata integration

We need to create a 'server' . As we're only retrievingdata from a fi le, i t seems to be overki l l , but SQL/MEDis also capable of coping with remote databases.

=# CREATE SERVER file-# FOREIGN DATA WRAPPER file_fdw ;CREATE SERVER

Now, let's l ink a '/tmp/data.csv' fi le to a stats table:

=# CREATE FOREIGN TABLE stats (-# field1 numeric,-# field2 numeric-# )-# server file options (-# filename ' /tmp/data. csv' ,-# format ' csv' ,-# delimiter ' ; '-# ) ;CREATE FOREIGN TABLE=# SELECT * from stats ;field1 | field2

--------+--------0. 1 | 0. 20. 2 | 0. 40. 3 | 0. 90. 4 | 1. 61. 6

For now, foreign tables are SELECT-only but moreimprovements wil l be coming in the next majorversions.

Why SQL/MED is cool

For more detai ls, check out Robert Haas' blogpost on SQL/MED at:http: //pgmag.org/0116The original article is avai lable at:http: //pgmag.org/0110

PostgreSQL 9.1

17

PG Mag #01

Per-column collationIn multilingual databases, users can now set the

collation for strings on a single column. This

permits true multilingual databases, where each

text column is a different language, and indexes

and sorts correctly for that language.

Let's say you are using a Postgres 9.0 database, withan UTF8 encoding and a de_DE. utf8 col lation(alphabetical sort) order, because most of your usersspeak German. If you had to store french data too,and had to sort, some french users could have beendisappointed:

=# SELECT * from (values ( ' élève' ) , ( ' élevé' ) , ( ' élever' ) , ( ' Élève' ) )-# as tmp order by column1;

column1---------

élevéélèveÉlèveélever

I t's not that bad, but it's not the french col lationorder: accentuated (diactric) characters are consi-dered equal on first pass to the unaccentuatedcharacters. Then, on a second pass, they are consi-dered to be after the unaccentuated ones. Exceptthat on that second pass, the letters are consideredfrom the end to the beginning of the word. That's abit strange, but that's the french col lation rules…

With Postgres 9.1 , two new features are avai lable:

You can specify col lation at query time:

=# SELECT * FROM (VALUES (' élève' ) ,(' élevé' ) ,(' élever' ) ,(' Élève' ) )-# AS tmp ORDER BY column1 COLLATE " fr_FR. utf8" ;

column1---------élèveÉlèveélevéélever

The collation order is notunique in a database

anymore.

You can specify col lation at table definition time:

=# CREATE TABLE french_messages (message TEXTCOLLATE " fr_FR. utf8" ) ;=# INSERT INTO french_messages VALUES( ' élève' ) , ( ' élevé' ) , ( ' élever' ) , ( ' Élève' ) ;=# SELECT * FROM french_messages ORDER BY message;

message---------élèveÉlèveélevéélever

And of course you can create an index on themessage column, that can be used for fast frenchsorting. For instance, using a table with more data(400k rows) and without col lation defined:

=# CREATE TABLE french_messages2 (message TEXT) ;=# INSERT INTO french_messages2-# SELECT * FROM french_messages, generate_series( 1, 100000) ;=# CREATE INDEX idx_fr_ctype-# ON french_messages2 (message COLLATE " fr_FR. utf8" ) ;

=# EXPLAIN SELECT * FROM french_messages2-# ORDER BY message;

QUERY PLAN-------------------------------------------------------------------------------Sort ( cost=62134. 28. . 63134. 28 rows=400000 width=32)

Sort Key: message-> Seq Scan on french_messages2 ( cost=0. 00. . 5770. 00 rows=400000 width=32)

=# EXPLAIN SELECT * FROM french_messages2-# ORDER BY message COLLATE " fr_FR. utf8" ;

QUERY PLAN--------------------------------------------------------------------------------

Index Scan using idx_fr_ctype on french_messages2( cost=0. 00. . 17139. 15 rows=400000 width=8)

PostgreSQL 9.1

18

PG Mag #01

Unlogged tablesWhen performance is more important than durability, unlogged tables

provide a way to improve performance while keeping the data

managed within PostgreSQL. Removing logging reduces I/O overhead,

yielding performance improvements up to 10 times faster when

compared to logged tables. Scenarios expected to leverage unlogged

tables include web session data, real time logging, ETL and

temporary/intermediate tables for functions.

Introduced in PostgreSQL 9.1 , unlogged tablesare designed for ephemeral data. They are muchfaster on writes, but won't survive a crash (it wi l l betruncated at database restart in case of a crash).Currently, GiST indexes are not supported onunlogged tables, and cannot be created on them.

They don't have the WAL maintenance overhead, sothey are much faster to write to. Here is a (non-real istic) example:

First let's create an unlogged table:

# CREATE UNLOGGED TABLE testu ( a int) ;CREATE TABLE

Now for comparison purposes, we' l l create a secondtable — identical , but logged:

# CREATE TABLE test ( a int) ;CREATE TABLE

Let's put an index on each table:

# CREATE INDEX idx_test on test ( a) ;CREATE INDEX# CREATE INDEX idx_testu on testu ( a) ;CREATE INDEX

Now we' l l see how faster we can go:

=# \timingTiming is on.=# INSERT INTO test SELECTgenerate_series( 1, 1000000) ;INSERT 0 1000000Time: 17601, 201 ms=# INSERT INTO testu SELECTgenerate_series( 1, 1000000) ;INSERT 0 1000000Time: 3439, 982 ms

With this example, the unlogged table is 5 timesfaster than the regular one. Even when using theCOPY command, the write performance is muchfaster with unlogged tables.

Performances are greatlyimproved for ephemeral

data

But remember! If your PostgreSQL crashes you wi l llose al l the content, even if you force a checkpoint:

$ cat test. sqlINSERT INTO testu VALUES ( 1) ;CHECKPOINTINSERT INTO testu VALUES ( 2) ;

$ psql -f test. sql ; killall -9 postgres postmasterINSERT 0 1CHECKPOINTINSERT 0 1postmaster: no process found

The server was forced to shutdown immediatly. Andafter restart:

# select * from testu;a

-----------

( 0 rows)

Many “NoSQL” engines advertise a speed advantageover “traditional” RDBMS engines, by employing asimi lar unlogged approach. PostgreSQL 9.1 providesan option, al lowing a database designer to choosebetween performance and crash safety for tables inthe same database, without having to abandon therich feature set provided by PostgreSQL.

In other wordsunlogged tables are very efficient for cachingdata, or for anything that can be rebui lt in caseof a crash.

PostgreSQL 9.1

19

PG Mag #01

Synchronous replicationSynchronous Replication enables high-availability

with consistency across multiple nodes.

Synchronous replication supports “2-safe

replication” which ensures transactions have

been confirmed by a standby server in addition

to the master, greatly limiting the possibility of

data loss. Only PostgreSQL has transaction level

synchronous replication, allowing users to

choose per transaction between response time

and data safety.

Let's say you have set up a “classic” PostgreSQL HotStandby cluster with streaming repl ication. To getsynchronous, just change the fol lowing option in themaster's postgresql. conf :

synchronous_standby_names = ' newcluster'

This is the application_name from our primary_conninfofrom the slave. Just do a

pg_ctl reload

and this new parameter wi l l be set. Now any commiton the master wi l l only be reported as committed onthe master when the slave has written it on its ownjournal , and acknowledged it to the master.

One of the real ly great features of synchronousrepl ication is that it is control lable per session. Theparameter synchronous_commit can be setted tolocal in a session, if i t does not require this syn-chronous guarantee, or turned off completely if youwant asynchronous commit even on master. I f youdon't need it in your transaction, just type the com-mand below and you won't pay the penalty.

SET synchronous_commit TO local;

Of course, Synchronous repl ication is not the onlyimprovement of this new version. There are manyother new repl ication features for PostgreSQL 9.1!Here's a quick l ist:

➥ In 9.0, the user used for repl ication had to be asuperuser. I t's no longer the case, there is a new'repl ication' privi lege.

➥ pg_basebackup is a new tool to create a clone ofa database, or a backup, using only the streamingrepl ication features.

➥ The slaves can now ask the master not to vacuumrecords they sti l l need.

➥ Two new systems views cal led pg_stat_repl icationand pg_stat_database_confl icts

➥ Repl ication can now be easi ly paused on a slave.

➥ Restore points can now be created.

A word of warningIn synchronous mode, transactions are consi-dered committed when they are appl ied to theslave's journal , not when they are visible on theslave. I t means there wi l l sti l l be a delaybetween the moment a transaction is commit-ted on the master, and the moment it is visibleon the slave. This sti l l is synchronous repl icationbecause no data wi l l be lost if the mastercrashes.

PostgreSQL 9.1

20

PG Mag #01

ExtensionsWhile PostgreSQL has always been extensible, now users can easily create, load,

upgrade, and manage any of dozens of database extensions using the EXTENSION

database object.

This item and the fol lowing one are another occasionto present several features in one go. We' l l need toinstal l pg_trgm. In a nutshel l , pg_trgm providesfunctions and operators for determining the simi larityof ASCI I a lphanumeric text based on trigrammatching, as wel l as index operator classes thatsupport fast searching for simi lar strings. WithPostgreSQL 9.1, it is now an extension.

Let's first instal l pg_trgm. Up to 9.0, we had to run ascript manual ly; the command looked l ike this:

\i /usr/local/pgsql/share/contrib/pg_trgm. sql

This was a real maintenance problem: the createdfunctions defaulted to the publ ic schema, weredumped “as is” in pg_dump fi les, often didn't restorecorrectly as they depended on external binaryobjects, or could change definitions betweenreleases.

With 9.1, you can use the CREATE EXTENSIONcommand:

CREATE EXTENSION [ IF NOT EXISTS ] extension_name[ WITH ] [ SCHEMA schema ]

[ VERSION version ][ FROM old_version ]

The most important options are extension_name, ofcourse, and schema: extensions can be stored in aschema.

Easily create, load, andmanage new database

features

So let's instal l pg_trgm for the next example:

=# CREATE schema extensions;CREATE SCHEMA=# CREATE EXTENSION pg_trgm-# WITH SCHEMA extensions;CREATE EXTENSION

Now, pg_trgm is instal led in an 'extensions' schema.It wi l l be included in database dumps correctly, withthe CREATE EXTENSION syntax. So if anythingchanges in the extension, this extension wi l l berestored with the new definition.

You can get the l ist of extensions under psql :

\dxList of installed extensions

\dxList of installed extensions

Name | Version | Schema | Description--------+---------+------------+------------pg_trgm | 1. 0 | extensions | . . . . .plpgsql | 1. 0 | pg_catalog | . . . . .( 2 rows) )

KNN indexes provide an innovative method to avoid

expensive table scans. They enhance PostgreSQL's query

capabilities by using mathematical “distance” for indexing

and searches. These indexes can be used to improve

common text searches, text-similarity searches, geospatial

location comparisons and other queries. Text search indexes

can now be configured to provide indexing support for LIKE

'%string%' queries without changing any SQL. PostgreSQL is

among the first database systems to have KNN.

PostgreSQL 9.1

Extend ExtensionsAs a developper, you have very few new things todo in order to package your code as an extension.You must provide a "control fi le" which must benamed the same as the extension with a suffix of.control , and must be placed in the instal lation'sSHAREDIR/extension directory. There must alsobe at least one SQL script fi le, which fol lows thenaming pattern extension--version.sql . You canalso provide a Makefi le that rel ies on PGXS.Check out the documentation for more detai ls!

http: //pgmag.org/0120

21

PG Mag #01

K nearest neighbor (KNN) IndexingKNN indexes provide an innovative method to avoid

expensive table scans. They enhance PostgreSQL's query

capabilities by using mathematical “distance” for indexing

and searches. These indexes can be used to improve

common text searches, text-similarity searches, geospatial

location comparisons and other queries. Text search indexes

can now be configured to provide indexing support for LIKE

'%string%' queries without changing any SQL. PostgreSQL is

among the first database systems to have KNN.

GiST indexes can now be used to return sorted rows,if a 'distance' has a meaning and can be defined forthe data type. For now, this work has been done forthe point datatype, the pg_trgm extension, and manybtree_gist datatypes. This feature is avai lable foral l datatypes to use, so there wi l l probably be morein the near future.

Here is an example with pg_trgm. pg_trgm usestrigrams to compare strings. Here are the trigramsfor the 'hel lo' string:

SELECT show_trgm( ' hello' ) ;show_trgm

---------------------------------{" h" , " he" , ell, hel, llo, " lo " }

Trigrams are used to evaluate simi larity (between 0and 1) between strings. So there is a notion ofdistance, with distance defined as '1-simi larity' .

For our example, we need the pg_trgm extension andan indexed table. The table contains 5 mi l l ion textrecords, for 750MB.

CREATE TABLE test_trgm ( text_data text) ;CREATE INDEX test_trgm_idx ON test_trgm USING gist( text_data extensions. gist_trgm_ops) ;

Unti l 9.0, if we wanted the two closest text_data tohel lo from the table, here was the query:

SELECT text_data, similarity( text_data, ' hello' )FROM test_trgmWHERE text_data % ' hello'ORDER BY similarity( text_data, ' hello' )LIMIT 2;

On the test database, it takes around 2 seconds tocomplete.

You can now create indexon “distance”

for faster locationand text-search queries

With 9.1 and KNN, one can write:

SELECT text_data, text_data <-> ' hello'FROM test_trgmORDER BY text_data <-> ' hello'LIMIT 2;

The <-> operator is the distance operator avai lablein the btree_gist extension. I t runs in 20ms, usingthe index to directly retrieve the 2 best records.

Whi le we're talking about pg_trgm and KNN, anothernew feature is that the LIKE and ILIKE operators cannow automatical ly make use of a trgm index. Sti l lusing the same table:

SELECT text_dataFROM test_trgmWHERE text_data LIKE ' %hello%' ;

This query wi l l use the test_trgm_idx index insteadof scanning the whole table.

PostgreSQL 9.1

22

PG Mag #01

PGXNThe PostgreSQL Extension Network (PGXN)

is a central distribution system for open

source PostgreSQL extension libraries. It

is inspired by the Comprehensive Perl

Archive Network (CPAN).

We've seen that PostgreSQL 9.1 al lows deve-lopers to create extensions. With PGXN , extensionauthors can submit their work together with meta-data describing them: the packages and their docu-mentation are indexed and distributed across severalservers. The system can be accessed through a webinterface or command l ine cl ients thanks to a simpleAPI .

A comprehensive PGXN cl ient is being developed. Itcan be instal led with:

$ easy_install pgxnclientSearching for pgxnclient. . .Best match: pgxnclient 0. 2. 1Processing pgxnclient-0. 2. 1-py2. 6. egg. . .Installed pgxnclient-0. 2. 1-py2. 6. egg

Among other commands, it al lows you to search forextensions on the website:

The new site PGXN.orgoffers a repositoryfor contributing and

downloading extensions

To bui ld and instal l them on the system:

$ pgxn install pairINFO: best version: pair 0. 1. 3INFO: saving /tmp/tmpezwyEO/pair-0. 1. 3. zipINFO: unpacking: /tmp/tmpezwyEO/pair-0. 1. 3. zipINFO: building extension. . .INFO: installing extension[ sudo] password for piro:/bin/mkdir -p' /usr/local/pg91b1/share/postgresql/extension'. . .

And to load them as database extensions:

$ pgxn load -d mydb pairINFO: best version: pair 0. 1. 3CREATE EXTENSION

What can be easier than this?

$ pgxn search pairpair 0. 1. 3

. . . Usage There are two ways to construct key/value *pairs*: Via the*pair*( ) function: % SELECT *pair*( ' foo' , ' bar' ) ; *pair* ------------( foo, bar) Or by using the ~> operator: % SELECT ' foo' ~> ' bar' ;*pair*. . .

semver 0. 2. 2*pair* | 0. 1. 0 | Key/value *pair* data type Note that " 0. 35. 0b1" is lessthan " 0. 35. 0" , as required by the specification. Use ORDER BY to getmore of a feel for semantic version ordering rules: SELECT. . .

PostgreSQL 9.1

23

PG Mag #01

SE PostgresSecurity enhanced (SE) Postgres is designed for

military-grade data stores. It allows mandatory

access control with full-integration support for

Security enhanced Linux (SE Linux). SE Linux is a

NSA project that modifies the Linux kernel with

security-related functions. It has been endorsed by

and included with major Linux distributions

including, but not limited to, Red Hat, CentOS,

Debian and Ubuntu. PostgreSQL is the only database

system which has tight integration with SE Linux.

Databases are significant faci l i ties for managinginformation assets. Databases enable information tobe searched for, retrieved, and stored in more ele-gant ways than with fi le systems.

Most existing RDBMSs apply their own access con-trols, for example, GRANT and REVOKE , without col la-borating with the operating system. This can resultin inconsistent access controls compared to the oneson fi lesystem and so on.

With SE Postgres,you can deploy

military-grade securityand mandatory access

controlSome modern operating systems have enhancedaccess control features, such as SE Linux. The designof most of these features is based upon the referencemonitor model (which came from 1980's researches),that al lows al l system access to be managed by acentral ized security pol icy. The reference monitormodel assumes that object managers (for example,operating systems) can capture al l system access,and then make decisions about whether that accessis al lowed or denied.

An operating systems is not the object manager in all cases. ARelational Database Management System (RDBMS) is an objectmanager for database objects, similar to operating systemsbeing object managers for fi le system objects. Previously,RDBMSs made access control decisions independently from acentralized security policy, and as such, meticulous care isneeded to keep its consistency between OS and RDBMS.

SE PostgreSQL is a bui lt-in enhancement ofPostgreSQL, providing fine-grained mandatory accesscontrol (MAC) for database objects. SE PostgreSQLmakes access control decisions based on SELinuxsecurity pol icy, the same way user access to fi lesystem objects is managed by the operating system.It provides the fol lowing significant features:

Mandatory access controls : PostgreSQL uses theconcept of a database superuser that can bypassall access controls of native PostgreSQL. On thecontrary, SE PostgreSQL enforces its access controlon any client, without exception, even if the clientis a database superuser. Clients can only accessdatabase objects when access is allowed by bothnative PostgreSQL and SE PostgreSQL.

Fine-grained access controls : SE PostgreSQLal lows access control to be configured at thecolumn and row levels (only a few proprietaryRDBMSs support column and row level access-control options).

Consistency in access controls : SE Linux requiresall processes and objects to have a security contextthat represents its privileges and attributes.SE PostgreSQL assigns a security context on eachdatabase object (which are tuples in system/general tables), and makes access control decisionsbased on the privileges and attributes as if SE Linuxapplies them in the kernel .

1

3

2

Learn more

SE Postgres is far too complex and powerful to beshowcased on a single page. For more detai ls,please check out the documentation athttp: //pgmag.org/0123

PostgreSQL 9.1

24

PG Mag #01

Serializable snapshot isolationThis new feature allows users to enforce arbitrarily complex

user-defined business rules within the database without

blocking, by automatically detecting live race conditions in

your SQL transactions at runtime. This feature currently

exists only in PostgreSQL.

With Serial izable Snapshot Isolation (SSI ) , i f you canshow that your transaction wi l l do the right thing ifthere are no concurrent transactions, then it wi l l dothe right thing in any mix of serial izable transactionsor else be rol led back with a serial ization fai lure.

Now let's try a “Simple Write Skew” example wheretwo concurrent transactions each determine whatthey are writing based on reading a data set whichoverlaps what the other is writing. You can get astate which could not occur if either had run beforethe other. This is known as write skew, and is thesimplest form of serial ization anomaly against whichSSI protects you.

SSI keeps concurrent transactionsconsistent without blocking, using

“true serializability”

In this case there are rows with a color columncontaining 'black' or 'white' . Two users concurrentlytry to make al l rows contain matching color values,but their attempts go in opposite directions. One istrying to update al l white rows to black and the otheris trying to update al l black rows to white. I f theseupdates are run serial ly, a l l colors wi l l match. I f theyare run concurrently in REPEATABLE READ mode, thevalues wi l l be switched, which is not consistent withany serial order of runs. I f they are run concurrentlyin SERIALIZABLE mode, SSI wi l l notice the write skewand rol l back one of the transactions.

The example can be set up with these statements:

session 1 session 2begin;update dots set color = ' black' where color = ' white' ;

begin;update dots set color = ' white' where color = ' black' ;At this point one transaction or the other is doomed to fail.commit;First commit wins.select * from dots order by id;id | color----+-------

1 | white2 | white3 | white

( 3 rows)This one ran as if by itself.

commit;ERROR: could not serialize access

due to read/write dependenciesamong transactions

DETAIL: Cancelled on identificationas a pivot, during commit attempt.

HINT: The transaction might succeed if retried.A serialization failure. We roll back and try again.rollback;begin;update dots set color = ' black' where color = ' white' ;commit;No concurrent transaction to interfere.select * from dots order by id;id | color

----+-------1 | black2 | black3 | black

( 3 rows)

This transaction ran by itself, after the other.

More examples

Check out the SSI wiki page for additional reall i fe examples such as “Intersecting Data”,“Overdraft Protection” or “Deposit Report”.

http: //wiki .postgresql .org/wiki/SSI

PostgreSQL 9.1

25

PG Mag #01

Writeable common tableexpressions

This features (also known as

wCTE) supports the relational

integrity of your data by allowing

you to update multiple,

cascading, related records in a

single statement. By using the

results of one query to execute

another query, you can update

recursively, hierarchically, across

foreign keys, or even more

creatively. PostgreSQL provides

the most complete and flexible

implementation of this SQL

feature available.

Common table expressions were introduced inPostgreSQL 8.4 (see the WITH syntax). Now, datamodification queries can be put in the WITH part ofthe query, and the returned data used later.

Let's say we want to archive al l records matching%hello% from our test_trgm table:

CREATE TABLE old_text_data ( text_data text) ;

WITH deleted AS (DELETE FROM test_trgmWHERE text_data like ' %hello%'RETURNING text_data

)INSERT INTO old_text_data SELECT * FROM deleted;

Al l in one query.

wCTE execute complexmulti-stage data updates

in a single query

As a more ambitious example, the fol lowing queryupdates a pgbench database, deleting a bunch oferroneous transactions and updating al l related tel ler,branch, and account totals in a single statement:

WITH deleted_xtns AS (DELETE FROM pgbench_historyWHERE bid = 4 and tid = 9RETURNING *

) ,deleted_per_account AS (

SELECT aid, sum(delta) as baldiffFROM deleted_xtnsGROUP BY 1

) ,accounts_rebalanced as (

UPDATE pgbench_accountsSET abalance = abalance - baldiffFROM deleted_per_accountWHERE deleted_per_account. aid =

pgbench_accounts. aidRETURNING deleted_per_account. aid,

pgbench_accounts. bid,baldiff

) ,branch_adj ustment as (

SELECT bid, SUM(baldiff) as branchdiffFROM accounts_rebalancedGROUP BY bid

)UPDATE pgbench_branchesSET bbalance = bbalance - branchdiffFROM branch_adj ustmentWHERE branch_adj ustment. bid = pgbench_branches. bidRETURNING

branch_adj ustment. bid, branchdiff, bbalance;

PostgreSQL 9.1

26

PG Mag #01

Once upon a time, a newbie would come to ourcommunity and say, “ I want feature X.” The res-ponse was general ly either, “Why?” or “Where’s yourpatch?” Over the last couple of years (I would put itat around the release of 8.3), the project hasmatured and the responses to feature requests arenow “How?” instead of “Why?” In other words, “Howcan we get this done?” versus “Why in the worldwould we want that?” This is a positive step and hasled to many world class leading features withinPostgreSQL over the last few major releases.

As with most popular, mature and large open sourceprojects, our project has become primari ly driven bycommercial interests, and the majority of keycontributors to the project have financial incentive.

This works wel l within our ecosystem because wehave quite a few different companies that contribute,whether it be 2ndQuadrant, CMD, Credativ, Enter-priseDB, or PgExperts. However, these companiesare driven by a need to pay employees, preferablyin a manner that al lows them to eat more than high-sodium 25 cent noodles. This means that whi le theymay al l contribute, they also have different profitcenters and their priorities can sway based oncustomer demands.

Paying for feature development is also difficult. Lastyear, CMD was sponsored by multiple companies todevelop the FKLocks (Foreign Key Locks) patch. I cansay with certainty that we wil l lose money on thedevelopment of that patch. We underestimated theamount of work it would take to get the job done. Thefinancial loss is our responsibil ity but when you aretrying to balance the investment of reviewing manythousands of lines of code to interpret the best wayto modify that code for a new feature and do soin a fashion that does not mean you havewritten the feature before you get it funded,

it makes for a very carefulprocess. That is not to

say the developmentof the patch wasn’tworth it; but it doespoint to a particulardifficulty in getting acompany such asCMD to continuefeature development.If we lose money on

the majority of patcheswe develop, we have to

divert resources from develo-ping features to more profitable

ventures such as training or profes-sional services. As I recall , 2ndQuadrant

ran into a similar problem when theydeveloped Hot Standby.

Money in Open Source,funding PostgreSQL features

Open source is moving fast and flourishing. In most cases, free software projects

start with little or no funding at all. Money is not necessary to implement a good

idea in an open source environment. However as a project grows and reaches

commercial success, it becomes possible to raise funds directly to add new major

features.

Opinion

27

PG Mag #01

“The key is to have amultitude of avenues forpeople and companies to

fund the continueddevelopment ofPostgreSQL.”

There is no single solution to this problem becauseeach entity wil l have to find what works for them. Thekey is to have a multitude of avenues for people andcompanies to fund the continued development ofPostgreSQL. At PostgreSQL Conference West this yearwe were able to fund two small features, one from CMDand one from 2ndQuadrant. Although the CMD feature(URI connection strings) was funded by a single entity(Heroku), 2ndQuadrant’s feature was funded bymultiple sponsors from the conference itself. Thismodel works well for well defined, small ish features(<15k) as we can use the conferences as a vehicle tocommunicate the need. I know that at Postgres Openthere were some fund-raising efforts as wel l .

One option would be to have a subscription pool . I fX number of people give Y amount, the communitycan employ Z number of engineers ful l time. Onecompl ication with this is that in order for it to work,it would have to run through one of the PostgreSQLcompanies. I know if every CMD customer werewi l l ing to commit to 1000.00 per year, we couldeasi ly employ (with benefits) 5 engineers ful l time foronly PostgreSQL contribution development. Of coursethat brings about other problems, such as whodecides what features are worked on and what ifsubscriptions drop off? When companies givemoney, they want to have a say in how it isspent and just because you write a patch, itdoesn’t mean it is going to get committed.When employees are working they want tomake sure they have stabi l i ty.

The idea of using the non-profits (PgUS, PgEU, SPI)has come up on occasion,but it has a wealth of legaland administrative issuesthat could burden thecorporations and actual lyhinder development. Thatis certainly not what wewant to have happen. Itwould be simple enough tooffer grants for develop-ment, but grants are nottax-free and it would put

the tax and administrative burden on the developer,something that most developers (let’s be honest)want nothing to do with and aren’t good at anyway.The non-profits also have to raise the money fordevelopment and al l of them are operated by peoplewith day jobs (and many have fami l ies). Fund-raisingtakes a lot of time that most people do not have.

What does the community think? How could we goabout continuing to drive feature development andmeet the required financial responsibi l i ty todevelopers? Is this something that even matters anymore? Has our community grown past that? Onething that is certain, the amount of money beinginvested in PostgreSQL is only going to increase. Thisis easi ly evidenced by announcements from VMware,or just by reviewing the number of contributions thatare commercial ly sponsored.

About the author

Joshua Drake (@Linuxpoet) isthe founder of Command Prompt, the oldestdedicated PostgreSQL support provider inNorth America. Since 1997, he's beendeveloping, supporting, deploying andadvocating PostgreSQL.

Opinion

28

PG Mag #01

Cascading streaming replication

PostgreSQL's built-in replication system is simple and solid. But so far, it has

lacked a key feature: cascading replication. With PostgreSQL 9.2, you'll be able to

design sophisticated replication clusters.

On 19th of July, Simon Riggs committed this patch:

Cascading replication feature for streaming log-basedreplication. Standby servers can now have WALSenderprocesses, which can work with either WALReceiver orarchive_commands to pass data. Fully updated docs,including new conceptual terms of sending server,upstream and downstream servers. WALSenders isterminated when promoted to master.

Fuj ii Masao, review, rework and doc rewrite by SimonRiggs

Streaming replication is relatively new, added in 9.0.Since the beginning it shared a limitation with normalWAL-fi les based replication, in that there is only onesource of data: that is the master. While it makes sense,it is also pretty cool to be able to make a slave a sourceof replication for some other systems; for example, notto keep the master occupied with such tasks.

Now, with the patch, we can set upa repl ication schema l ike this:

So, let’s test it.

To make it work I wi l l need amaster database and 3 slaves,made off the master. Seems simpleenough.

=$ mkdir master=$ initdb -D master. . .=$ vim master/postgresql. conf

In postgresql .conf, I change:

port = 4001wal_level = hot_standbycheckpoint_segments = 20archive_mode = onarchive_command = ' /bin/true'max_wal_senders = 3wal_keep_segments = 100logging_collector = onlog_checkpoints = onlog_connections = onlog_line_prefix = ' %m %r %u %d %p: '

I a lso set pg_hba. conf to something that matchesmy test environment:

# TYPE DATABASE USER ADDRESS METHODlocal replication all trustlocal all all trusthost all all 127 . 0. 0. 1/32 trust

With the master prepared that way, I can start it:

=$ pg_ctl -D master startserver starting

=$ psql -p 4001 -d postgres -c " select version( ) "

version----------------------------------------------------PostgreSQL 9. 2devel on x86_64-unknown-linux-gnu,

compiled by gcc-4. 5. real (Ubuntu/Linaro 4. 5. 2-8ubuntu4) 4. 5. 2, 64-bit( 1 row)

Al l looks ok.

One note — you might not understand why I used/bin/true as archive_command. The reason is verysimple — archive_command has to be set tosomething, otherwise archive_mode cannot beenabled, and this wi l l cause problems with backups;but on the other hand I wi l l not need to use the WALarchive, since I have a pretty large value forwal_keep_segments .

Now, we’l l set up the slaves. Starting with the firstone of course:

=$ psql -p 4001 -d postgres -c " selectpg_start_backup( ' whatever' ) "pg_start_backup-----------------0/2000020( 1 row)

=$ rsync -a master/ slave/=$ psql -p 4001 -d postgres -c " selectpg_stop_backup( ) "NOTICE: pg_stop_backup complete, all required WALsegments have been archivedpg_stop_backup----------------0/20000D8

( 1 row)

Of course we need some tidying of 'slave' :

=$ rm -f slave/pg_xlog/????????????????????????slave/pg_xlog/archive_status/* slave/pg_log/*slave/postmaster. pid

=$ vim slave/postgresql. conf

port = 4002hot_standby = on

Waiting for 9.2

29

PG Mag #01

And I also create recovery. conf in slave/ , with thiscontent:

restore_command = ' /bin/false'standby_mode = ' on'primary_conninfo = ' port=4001 user=depesz'trigger_file = ' /tmp/slave. finish. recovery'

With this in place I can start the 'slave' :

=$ pg_ctl -D slave startserver starting=$ head -n 1 slave/postmaster. pid | xargs -IPG ps uwf -p PG --ppid PGUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDdepesz 13082 1. 5 0. 0 66484 7492 pts/3 S 12: 51 0: 00 /home/pgdba/work/bin/postgres -D slavedepesz 13083 0. 0 0. 0 26136 716 ? Ss 12: 51 0: 00 \_ postgres: logger processdepesz 13084 0. 0 0. 0 66556 1428 ? Ss 12: 51 0: 00 \_ postgres: startup process recovering 000000010000000000000006depesz 13087 2. 7 0. 0 81504 3064 ? Ss 12: 51 0: 00 \_ postgres: wal receiver process streaming 0/6000078depesz 13091 0. 0 0. 0 66484 1012 ? Ss 12: 51 0: 00 \_ postgres: writer processdepesz 13092 0. 0 0. 0 26132 896 ? Ss 12: 51 0: 00 \_ postgres: stats collector process

=$ head -n 1 master/postmaster. pid | xargs -IPG ps uwf -p PG --ppid PGUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDdepesz 12981 0. 2 0. 0 66456 7520 pts/3 S 12: 50 0: 00 /home/pgdba/work/bin/postgres -D masterdepesz 12982 0. 0 0. 0 26140 724 ? Ss 12: 50 0: 00 \_ postgres: logger processdepesz 12984 0. 0 0. 0 66456 1016 ? Ss 12: 50 0: 00 \_ postgres: writer processdepesz 12985 0. 0 0. 0 66456 1012 ? Ss 12: 50 0: 00 \_ postgres: wal writer processdepesz 12986 0. 0 0. 0 67296 2096 ? Ss 12: 50 0: 00 \_ postgres: autovacuum launcher processdepesz 12987 0. 0 0. 0 26136 732 ? Ss 12: 50 0: 00 \_ postgres: archiver processdepesz 12988 0. 0 0. 0 26136 1040 ? Ss 12: 50 0: 00 \_ postgres: stats collector processdepesz 13088 0. 3 0. 0 67428 2480 ? Ss 12: 51 0: 00 \_ postgres: wal sender process depesz [ local] streaming 0/6000078

Waiting for 9.2

One note — I used “user=depesz” in primary_conninfo,because I run the tests on the depesz system account,so initdb made a superuser named depesz, not postgres.

We now have repl ication between 'master' and 'slave'set up, and so we can test it:

=$ psql -p 4001 -d postgres -c “create table i ( xint4) ”; psql -p 4002 -d postgres -c ' \d i'

CREATE TABLE Table " public. i"Column | Type | Modifiers--------+---------+-----------x | integer |

Al l looks OK. Now, we can add 'slave2' and 'slave3' .Since I ’m lazy, I wi l l just stop 'slave' , copy 'slave' to'slave2' and 'slave3' and then modify them:

=$ pg_ctl -D slave stopwaiting for server to shut down. . . . doneserver stopped=$ rsync -a slave/ slave2/=$ rsync -a slave/ slave3/=$ pg_ctl -D slave startserver starting

'slave2' and 'slave3' wi l l be basical ly the same as'slave' , but with different ports, and connecting to 4002(slave) instead of 4001 (master) for their WAL. So, let’sdo the changes:

How to test this at home ?

I f you are not a code contributor, testingPostgreSQL Alphas and Betas is one of thebest things you can do for the project. Byparticipating in organized testing, you helpget the release out faster, with more featuresand less bugs.

If you are able to contribute to PostgreSQL bydoing alpha testing, please read theAlpha/Beta Testing Page in the wiki :

http: //wiki .postgresql .org/wiki/HowToBetaTest

http: //pgmag.org/0134

About the author

About the article

Hubert “Depesz” Lubaczewski isa Database Architect at OmniTI.His blog (http://depesz.com/) isdedicated to the new features of the forth-coming version of PostgreSQL.

The originia l article is avai lable at

30

PG Mag #01

Picture

=$ perl -pi -e ' s/port = 4002/port = 4003/' slave2/postgresql. co=$ perl -pi -e ' s/port = 4002/port = 4004/' slave3/postgresql. conf=$ perl -pi -e ' s/port=4001/port=4002/' slave{2, 3}/recovery. conf=$ perl -pi -e ' s/slave. finish. recovery/slave2. finish. recovery/' slave2/recovery. conf=$ perl -pi -e ' s/slave. finish. recovery/slave3. finish. recovery/' slave3/recovery. conf

Now, let’s start them and see the processes:

=$ for a in slave2 slave3; do pg_ctl -D $a/ start; doneserver startingserver starting

=$ head -n 1 -q */*. pid | xargs -IPG echo " -p PG --ppid PG" | xargs ps uwfUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDdepesz 14031 0. 0 0. 0 66488 7496 pts/3 S 13: 03 0: 00 /home/pgdba/work/bin/postgres -D slave3depesz 14032 0. 0 0. 0 26140 720 ? Ss 13: 03 0: 00 \_ postgres: logger processdepesz 14033 0. 0 0. 0 66556 1400 ? Ss 13: 03 0: 00 \_ postgres: startup process recovering000000010000000000000006depesz 14063 0. 0 0. 0 79456 2148 ? Ss 13: 03 0: 00 \_ postgres: wal receiver process streaming0/6012ED0depesz 14069 0. 0 0. 0 66488 1532 ? Ss 13: 03 0: 00 \_ postgres: writer processdepesz 14070 0. 0 0. 0 26136 900 ? Ss 13: 03 0: 00 \_ postgres: stats collector processdepesz 14026 0. 0 0. 0 66492 7496 pts/3 S 13: 03 0: 00 /home/pgdba/work/bin/postgres -D slave2depesz 14042 0. 0 0. 0 26144 720 ? Ss 13: 03 0: 00 \_ postgres: logger processdepesz 14043 0. 0 0. 0 66560 1400 ? Ss 13: 03 0: 00 \_ postgres: startup process recovering000000010000000000000006depesz 14067 0. 0 0. 0 79460 2148 ? Ss 13: 03 0: 00 \_ postgres: wal receiver process streaming0/6012ED0depesz 14071 0. 0 0. 0 66492 1532 ? Ss 13: 03 0: 00 \_ postgres: writer processdepesz 14072 0. 0 0. 0 26140 900 ? Ss 13: 03 0: 00 \_ postgres: stats collector processdepesz 14021 0. 0 0. 0 66488 7528 pts/3 S 13: 03 0: 00 /home/pgdba/work/bin/postgres -D slavedepesz 14037 0. 0 0. 0 26140 724 ? Ss 13: 03 0: 00 \_ postgres: logger processdepesz 14038 0. 0 0. 0 66560 1572 ? Ss 13: 03 0: 00 \_ postgres: startup process recovering000000010000000000000006depesz 14048 0. 0 0. 0 66488 1536 ? Ss 13: 03 0: 00 \_ postgres: writer processdepesz 14050 0. 0 0. 0 26136 904 ? Ss 13: 03 0: 00 \_ postgres: stats collector processdepesz 14052 0. 0 0. 0 79460 2136 ? Ss 13: 03 0: 00 \_ postgres: wal receiver process streaming0/6012ED0depesz 14064 0. 0 0. 0 67332 2476 ? Ss 13: 03 0: 00 \_ postgres: wal sender process depesz [ local]streaming 0/6012ED0depesz 14068 0. 0 0. 0 67452 2476 ? Ss 13: 03 0: 00 \_ postgres: wal sender process depesz [ local]streaming 0/6012ED0depesz 12981 0. 0 0. 0 66456 7524 pts/3 S 12: 50 0: 00 /home/pgdba/work/bin/postgres -D masterdepesz 12982 0. 0 0. 0 26140 724 ? Ss 12: 50 0: 00 \_ postgres: logger processdepesz 12984 0. 0 0. 0 66456 1780 ? Ss 12: 50 0: 00 \_ postgres: writer processdepesz 12985 0. 0 0. 0 66456 1012 ? Ss 12: 50 0: 00 \_ postgres: wal writer processdepesz 12986 0. 0 0. 0 67296 2156 ? Ss 12: 50 0: 00 \_ postgres: autovacuum launcher processdepesz 12987 0. 0 0. 0 26136 732 ? Ss 12: 50 0: 00 \_ postgres: archiver processdepesz 12988 0. 0 0. 0 26136 1040 ? Ss 12: 50 0: 00 \_ postgres: stats collector processdepesz 14053 0. 0 0. 0 67444 2520 ? Ss 13: 03 0: 00 \_ postgres: wal sender process depesz [ local]streaming 0/6012ED0

Waiting for 9.2

31

PG Mag #01

Please note that 'master' Postgres has only onesender process (pid 14053 ) , 'slave' Postgres has onereceiver (14052 ) and two senders (14064 and 14068 ) ,and 'slave2' and 'slave3' have only one receiver each(14067 and 14063 ) .

Now we should test if it a l l works wel l , so:

=$ psql -d postgres -p 4001 -c \' insert into i( x) values ( 123) '

for port in 4002 4003 4004do

echo " port=$port"psql -p $port -d postgres -c \

" select * from i"doneINSERT 0 1port=4002x

---( 0 rows)

port=4003x

---( 0 rows)

port=4004x

---( 0 rows)

The tables are empty. They should have some data,but it might be simply because of repl ication lag. Solet’s retry the check, without the insert this time:

=$ for port in 4002 4003 4004do

echo " port=$port"psql -p $port -d postgres -c " select * from i"

doneport=4002

x-----123

( 1 row)

port=4003

x-----123

( 1 row)

port=4004

x-----123

( 1 row)

And all works fine now. Great! The only missing featureis the abil ity to make slave sourcing from a slave stil lwork when a slave gets promoted to standalone, butunfortunately, it’s not implemented yet:

=$ touch /tmp/slave. finish. recovery; sleep 5; head -n 1 -q */*. pid | xargs -IPG echo "-p PG --ppid PG" | xargs ps uwfUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDdepesz 14896 0. 1 0. 0 66488 7524 pts/3 S 13: 18 0: 00 /home/pgdba/work/bin/postgres -D slave3depesz 14897 0. 0 0. 0 26140 720 ? Ss 13: 18 0: 00 \_ postgres: logger processdepesz 14898 0. 0 0. 0 66556 1696 ? Ss 13: 18 0: 00 \_ postgres: startup process waiting for000000010000000000000006depesz 14901 0. 0 0. 0 66488 1276 ? Ss 13: 18 0: 00 \_ postgres: writer processdepesz 14902 0. 0 0. 0 26136 900 ? Ss 13: 18 0: 00 \_ postgres: stats collector processdepesz 14883 0. 1 0. 0 66492 7528 pts/3 S 13: 18 0: 00 /home/pgdba/work/bin/postgres -D slave2depesz 14885 0. 0 0. 0 26144 724 ? Ss 13: 18 0: 00 \_ postgres: logger processdepesz 14886 0. 0 0. 0 66560 1700 ? Ss 13: 18 0: 00 \_ postgres: startup process waiting for000000010000000000000006depesz 14890 0. 0 0. 0 66492 1280 ? Ss 13: 18 0: 00 \_ postgres: writer processdepesz 14891 0. 0 0. 0 26140 904 ? Ss 13: 18 0: 00 \_ postgres: stats collector processdepesz 14021 0. 0 0. 0 66488 7528 pts/3 S 13: 03 0: 00 /home/pgdba/work/bin/postgres -D slavedepesz 14037 0. 0 0. 0 26140 724 ? Ss 13: 03 0: 00 \_ postgres: logger processdepesz 14048 0. 0 0. 0 66488 1780 ? Ss 13: 03 0: 00 \_ postgres: writer processdepesz 14050 0. 0 0. 0 26136 1032 ? Ss 13: 03 0: 00 \_ postgres: stats collector processdepesz 15018 0. 0 0. 0 66488 1016 ? Ss 13: 20 0: 00 \_ postgres: wal writer processdepesz 15019 0. 0 0. 0 67320 2100 ? Ss 13: 20 0: 00 \_ postgres: autovacuum launcher processdepesz 15020 0. 0 0. 0 26136 912 ? Ss 13: 20 0: 00 \_ postgres: archiver process last was00000002. historydepesz 12981 0. 0 0. 0 66456 7524 pts/3 S 12: 50 0: 00 /home/pgdba/work/bin/postgres -D masterdepesz 12982 0. 0 0. 0 26140 724 ? Ss 12: 50 0: 00 \_ postgres: logger processdepesz 12984 0. 0 0. 0 66456 1780 ? Ss 12: 50 0: 00 \_ postgres: writer processdepesz 12985 0. 0 0. 0 66456 1012 ? Ss 12: 50 0: 00 \_ postgres: wal writer processdepesz 12986 0. 0 0. 0 67296 2164 ? Ss 12: 50 0: 00 \_ postgres: autovacuum launcher processdepesz 12987 0. 0 0. 0 26136 732 ? Ss 12: 50 0: 00 \_ postgres: archiver processdepesz 12988 0. 0 0. 0 26136 1040 ? Ss 12: 50 0: 00 \_ postgres: stats collector process

As you can see the sender in 'slave' got ki l led, andthus 'slave2' and 'slave3' are sti l l slaves, but withouta source of WAL. Logs of 'slave2' and 'slave3'PostgreSQL show a clear reason why it doesn’t work:

2011-07-26 13: 26: 41. 483 CEST 16318: FATAL:timeline 2 of the primary does not match recoverytarget timeline 1

Clearly 'slave' is using timeline 2, while 'slave2' and'slave3' are still on timeline 1. Theoretically it should besimple to fix, since slave has pg_xlog/00000002. historyfile, but the functionality to switch timelines in recoveryis simply not there yet.

Anyway, the abi l i ty to have slaves that are receivingWAL from other slaves is pretty cool , and definitely awelcome addition.

Waiting for 9.2

32

PG Mag #01

Elephants and LionsWith the recent release of Lion and Lion Server, one noticeable absentee was

MySQL, which we later discovered was replaced with PostgreSQL. In this article

we will take a look at how to connect to Lion Server’s built-in PostgreSQL

services. We'll be focusing on the 2 most popular managment tools: an

application called pgAdmin and a web service known as PgPhpAdmin.

How to use pgAdmin…

First let's take a look at pgAdmin! We need to openServer.app and enable the web service, as in mostcases people want to use PostgreSQL to run adatabase for their web instal l . In this example, I wantto do just that! I ’ve also gone ahead and enabledPHP web appl ications. (Please note that this step isnot important to the running of PostgreSQL, but wi l lbe required for your web appl ications that needPostgreSQL).

Next we need to download pgAdmin. To downloadthe latest Mac release, open up the DMG and drop itin your Appl ications folder (you can also l ink it inyour dock to make it more accessible). Now letscheck a few things before firing up PostgreSQL. Openup Terminal andtype:

sudo serveradminlist

This will display all ofthe available servicesof Lion Server. Checkthat ‘postgres’ is inthere.

Next type in:

sudo serveradmin fullstatus postgres

This wi l l tel l you the current status of the 'postgres'service. I f you have already been using theServer.app, then it’s possible the state wi l l a lreadybe ‘running’ .

I f not, then you can start it using the fol lowingcommand:

sudo serveradmin start postgres

This should result in the postgres state being ‘RUNNING’ .

I f you were now to open pgAdmin and try to connectit would fai l . This is because 'postgres' is notl istening to connections on localhost. To fix this, weneed to edit a fi le:

/System/Library/LaunchDaemons/org. postgresql. postgres. plist

Search for the l ine:

<string>listen_addresses=</string>

Add in our localhost address:

<string>listen_addresses=127 . 0. 0. 1</string>

Tips and Tricks

33

PG Mag #01

Now lets stop and start 'postgres' . In Terminal :

sudo serveradmin stop postgressudo serveradmin start postgres

Now lets open up pgAdmin and connect toPostgreSQL. Cl ick the connect icon in the top leftcorner and enter the settings for your setup.

Enter a descriptive Name, Host (which we’ve justopened as 127.0.0.1), Username and Password (bydefault these are ‘_postgres ’ and your administratorpassword).

Voi la! You are connected and can create newdatabases.

DownloadpgAdmin is avai lable at:http: //www.pgadmin.org/

PhpPgAdmin can be found at:http: //phppgadmin.sourceforge.net/http: //pgmag.org/0132

About the article

The originia l article is avai lable at

Known IssueI f you experience issues with permissions oraccessing the pg_hba.conf fi le, you may need tochange the permissions on the pgsql folder in/private/var/ – Get Info on the pgsql folder andadd your current administrative user to haveread/write access.

Tips and Tricks

34

PG Mag #01

… or phpPgAdmin

Now let's try phpPgAdmin , just head over to thephpPgAdmin website and download the latestrelease. Unzip the contents and rename the folder tosomething l ike 'phpPgAdmin' to make it easier tobrowse to. You need to place this folder in your webdocuments folder in the fol lowing location:

/Library/Server/Web/Data/Sites/Default/

Open up Safari and browse to the 'phpPgAdmin'folder on your localhost web server:

http: //localhost/phppgadmin/

At this point if the PostgreSQL server has a red X nextto it, this means it’s not running.

Like above the PostgreSQL service must be running.An alternative way to start PostgreSQL is to openServer App and launch the 'Wiki ' service. You maywant to change the 'Wiki Creators' to be Admin’s onlyor otherwise everybody can create Wikis:

Now if you head back to the phpPgAdmin page inSafari and select the database you should now beable to login with the user '_postgres ' and youradministrator password.

About the author

Simon Merrick (@Mactasia) has worked ineducation based IT for the past 7 years,alongside offering Apple Consultancy services.He often blogs and writes “how to” articles forhis website Mactasia (mactasia.co.uk)

Tips and Tricks

35

PG Mag #01

Next

How to

ContributeThis magazine is a community-driven initiative. You can help us in various ways!!

TranslatorsYou l ike the magazine, but you'd l ike to read it in your ownlanguage? Don't worry, this is our next chal lenge! So far wehave 4 translating teams coming off the ground: Spanish,Portuguese, French and Chinese. For now, there's notranslation avai lable but hopeful ly we' l l have something topresent by the end of the year. I f you want to help one ofthese teams, check out our website and contact the teamleader. I f you want to create a new team for anotherlanguage, please send us a message at contact@pgmag.org.

http: //pgmag.org/Contribute#translating_an_issue

WritersDo you have something to say about PostgreSQL? As article?An tutoria l? Some insights? Just send us your text atarticles@pgmag.org. Our next issue wi l l ta lk about manythings, including PostGIS 2.0 and PostgreSQL 9.2… Thedeadl ine is July 24th, 2012.

Copy EditorsWe're looking for editors to help us improve the formatting,style, and accuracy of the articles we receive. Typical ly, thisinvolves correcting spel l ing, avoiding jargon, cutting articlewhen they're long and ensuring that to the text adheres styleguidel ines. This job is probably the most interesting part ofthe magazine as you need to be both an experiencedPostgreSQL DBA and a ski l led wordsmith!

Distribute PG MagI f you organize an event or meeting dedicated to PostgreSQL,please contact us. We wi l l try to send you free copies of themagazine. You can also get the PDF version here:http: //pgmag.org/01/download

Photo Credits

Front: © claudiaveja (fotol ia) /p4: © ITPUG / p6 : © wiccked (Fl ickr)¹ / p8:© MagnusHagander / p11: © ThomasVondra(Fl ickr) / p12: © r_leontiev (fl ickr) / p16: ©andrechinn (Fl ickr)² / p17: © ilovememphis

(Flickr)³ / p18: © DanBrady (Fl ickr)² / p19: ©boklm (Flickr) ² / p20: © / p21: © CarbonNYC

(Flickr) ² / p22: / p23: © SmSm (Flickr) ² / p24:/ p 25: © QUOI Media Group (Flickr)⁴ / p26:© DMM Photography Art (fotol ia) / p27: ©remik44992 (fotolia) / p30: © droolcup(Fl ickr)⁵ / p32-34 : © SimonMerrick / P34 :©Si lvercupcake1

¹ : Creative Commons BY-NC-SA ² : Creative Commons BY³ : Creative Commons BY-ND ⁴ : Creative Commons BY-SA⁵ : Creative Commons BY-NC

wwwwww..ppggmmaagg .. oorrgg