Intro to PostgreSQLIntro to PostgreSQL

Mike CallahanMike Callahan

WFO LouisvilleWFO Louisville

September 8, 2009September 8, 2009

TopicsTopics Relational database theoryRelational database theory PostgreSQL ReferencePostgreSQL Reference psqlpsql SnoopySnoopy SQL syntaxSQL syntax

SELECTSELECT UPDATEUPDATE INSERT INTOINSERT INTO DELETE FROMDELETE FROM CREATE TEMP TABLECREATE TEMP TABLE COPYCOPY

Databases in AWIPSDatabases in AWIPS

PostgreSQL

Databases

Sources Data

D2D

WHFS

OtherOtherApplicationsApplications

SQL

SQLSQ

L

psqlpg_...

Relational TheoryRelational Theory Data are stored in groups called 'tables' or 'relations'Data are stored in groups called 'tables' or 'relations' Tables consist of 'rows' or 'records' and 'columns' or Tables consist of 'rows' or 'records' and 'columns' or

'fields''fields' Rows are usually identified by a unique 'key' which Rows are usually identified by a unique 'key' which

may be a single column or a group of columnsmay be a single column or a group of columns Columns can be linked to other tables with similar Columns can be linked to other tables with similar

columnscolumns Good design of the database structure or 'schema' is Good design of the database structure or 'schema' is

critical.critical. "All the data in a row should be dependant on the key, the "All the data in a row should be dependant on the key, the

whole key, and nothing but the key."whole key, and nothing but the key."

Working with databasesWorking with databases

Most of the time, you will work with existing Most of the time, you will work with existing databasesdatabases

However, you still need to know the schemaHowever, you still need to know the schema We will concentrate on four tasksWe will concentrate on four tasks

Extracting dataExtracting data Changing dataChanging data Inserting dataInserting data Deleting dataDeleting data

Why PostgreSQLWhy PostgreSQL Switching database server from HP to LinuxSwitching database server from HP to Linux PostgreSQL is free, Informix for Linux is notPostgreSQL is free, Informix for Linux is not PostgreSQL is the most advanced open source PostgreSQL is the most advanced open source

relational database, MySQL did not have relational database, MySQL did not have triggers, PostgreSQL didtriggers, PostgreSQL did

Version on AWIPS is 8.2, latest version is 8.4Version on AWIPS is 8.2, latest version is 8.4 Documentation is excellent and is at Documentation is excellent and is at

www.postgresql.org/docs/8.2/...www.postgresql.org/docs/8.2/... We will use version 8.2 for our examplesWe will use version 8.2 for our examples

PostgreSQL DocumentationPostgreSQL Documentation

PrefacePreface Gives history, conventions, etc.Gives history, conventions, etc.

TutorialTutorial A quick introduction to SQLA quick introduction to SQL

SQL LanguageSQL Language A description of the standard and the PostgreSQL A description of the standard and the PostgreSQL

additionsadditions Reference: SQL CommandsReference: SQL Commands

Details of each commandDetails of each command

psqlpsql ……//interactive/app-psql.htmlinteractive/app-psql.html

The standard PostgreSQL application to work The standard PostgreSQL application to work with databaseswith databases

It uses a superset of standard SQLIt uses a superset of standard SQL It can be interactively like DBaccess, or in It can be interactively like DBaccess, or in

scripts like sqlcmd scripts like sqlcmd

Common psql Command Line OptionsCommon psql Command Line Options

-A, --no-align: set output to non-aligned, no padding-A, --no-align: set output to non-aligned, no padding -c -c sqlsql, --command , --command sqlsql: execute the sql command and : execute the sql command and

then exitthen exit -d -d namename, --dbname , --dbname namename: name of database, same as : name of database, same as namename as the first non-option argument as the first non-option argument

-f -f namename, --file , --file namename: use : use namename as the source of as the source of commandscommands

-o -o namename, --output , --output namename: put the output in : put the output in namename -q, --quiet: suppress welcome messages-q, --quiet: suppress welcome messages -t, --tuples-only: suppress print column names, result -t, --tuples-only: suppress print column names, result

row counters, etcrow counters, etc -?, --help: get command line help-?, --help: get command line help

Common psql Meta-commandsCommon psql Meta-commands

\a: toggle output format (aligned/unaligned)\a: toggle output format (aligned/unaligned) \c \c namename: connect to a database : connect to a database namename \copy \copy tabletable: copy a table to/from a file: copy a table to/from a file \d : list all tables (display)\d : list all tables (display) \d \d tabletable: show information about a table: show information about a table \e: edit the query buffer\e: edit the query buffer \g: execute the query buffer (go)\g: execute the query buffer (go) \h \h commandcommand: display help on : display help on commandcommand \i \i namename: read name into query buffer (input): read name into query buffer (input)

Common psql Meta-commands Common psql Meta-commands

\o \o namename: send output to : send output to namename \p: display the query buffer\p: display the query buffer \q: quit the program\q: quit the program \r: resets (clears) the query buffer\r: resets (clears) the query buffer \t: toggle the output headers on/off\t: toggle the output headers on/off \w \w namename: write the query buffer to : write the query buffer to namename \! \! commandcommand: execute the Linux : execute the Linux commandcommand \?: help on meta-commands\?: help on meta-commands

Interactive psqlInteractive psql

Typing in a query loads the query bufferTyping in a query loads the query buffer The command will not run unless terminated The command will not run unless terminated

with a semicolon (;)with a semicolon (;) Meta-commands all start with blackslash (\)Meta-commands all start with blackslash (\) Exit with \qExit with \q

SnoopySnoopy

Snoopy is an interactive, GUI front-end to psqlSnoopy is an interactive, GUI front-end to psql It was designed for the hydro database, but It was designed for the hydro database, but

will work with any AWIPS databasewill work with any AWIPS database Queries are in a scrolling window, data is in a Queries are in a scrolling window, data is in a

separate scrolling windowseparate scrolling window Help is available about tables, columns, rows, Help is available about tables, columns, rows,

SQL syntax, and hydro database data SQL syntax, and hydro database data dictionarydictionary

Available on the AWIPS LADAvailable on the AWIPS LAD

Snoopy Screen ShotSnoopy Screen Shot

Snoopy HelpSnoopy Help

Available only for the hydro database

SQL for PostgreSQLSQL for PostgreSQL…/interactive/sql-commands.html…/interactive/sql-commands.html

SQL, pronounced “sequel”, is the standard SQL, pronounced “sequel”, is the standard way of working with relational databasesway of working with relational databases

The standard is well-supported but weak so The standard is well-supported but weak so every database manager adds commands to the every database manager adds commands to the standardstandard

In these examples, SQL keywords will be In these examples, SQL keywords will be CAPITALIZEDCAPITALIZED

However, case is NOT important except in However, case is NOT important except in stringsstrings

Basic Data Types in SQLBasic Data Types in SQL Character, Varchar: A sequence of alphanumeric Character, Varchar: A sequence of alphanumeric

characters enclosed in single quotes; characters enclosed in single quotes; ‘a string 123’‘a string 123’ To include a single quote in a string double it; To include a single quote in a string double it;

'Mike''s string''Mike''s string' Integers: A number without a decimal point; Integers: A number without a decimal point; 125, -2125, -2 Real: A number with a decimal point; Real: A number with a decimal point; 3.1416, 25.003.1416, 25.00 Decimal, Numeric: Fixed decimal point, accurate but Decimal, Numeric: Fixed decimal point, accurate but

slow mathslow math Boolean: Boolean: TRUE TRUE oror FALSE FALSE

in psql TRUE is displayed as t, FALSE is displayed in psql TRUE is displayed as t, FALSE is displayed as fas f

Date/Time Data Types in SQLDate/Time Data Types in SQL Date: The word followed by a date enclosed in single Date: The word followed by a date enclosed in single

quotes; quotes; DATE ‘2006-2-18’DATE ‘2006-2-18’ Other formats are allowed but this is the standard.Other formats are allowed but this is the standard.

Time: The word followed by a time enclosed in single Time: The word followed by a time enclosed in single quotes; quotes; TIMETIME ‘02:34:00’‘02:34:00’ Seconds and colons are optional.Seconds and colons are optional.

Timestamp: Both; Timestamp: Both; TIMESTAMP ’2004-10-23 TIMESTAMP ’2004-10-23 13:20:00’13:20:00’

Interval: A time interval; Interval: A time interval; INTERVAL ‘2 days’, INTERVAL ‘2 days’, INTERVAL ‘12 hours’INTERVAL ‘12 hours’

Many times you in PostGreSQL you don't need to type Many times you in PostGreSQL you don't need to type the descriptor. For examplethe descriptor. For example:: '2006-2-18', '02:34', '2006-2-18', '02:34', etc. etc.

Special ValuesSpecial Values

TIMESTAMP ‘NOW’TIMESTAMP ‘NOW’ The current time, date, or timestampThe current time, date, or timestamp

DATE ‘TODAY’DATE ‘TODAY’ The current date or midnight timestampThe current date or midnight timestamp

TIMESTAMP ‘TOMORROW’TIMESTAMP ‘TOMORROW’ Tomorrow date or midnight timestamp Tomorrow date or midnight timestamp

DATE ‘YESTERDAY’DATE ‘YESTERDAY’ Yesterday date or midnight timestampYesterday date or midnight timestamp

TIME ‘ALLBALLS’TIME ‘ALLBALLS’ Midnight 00:00:00 UTC timeMidnight 00:00:00 UTC time

Common SQL CommandsCommon SQL Commands

SELECTSELECT Extract data from one or more tablesExtract data from one or more tables

UPDATEUPDATE Change data in a table quicklyChange data in a table quickly

INSERT INTOINSERT INTO Add new data to a tableAdd new data to a table

DELETE FROMDELETE FROM Remove data from a tableRemove data from a table

CREATE TEMP TABLECREATE TEMP TABLE Create a temporary tableCreate a temporary table

COPYCOPY Send the contents of a table to a file or vice versaSend the contents of a table to a file or vice versa

SELECTSELECT

Most SQL work involves creating SELECT Most SQL work involves creating SELECT statementsstatements

You cannot harm data using a SELECT CommandYou cannot harm data using a SELECT Command SELECT statements can be simple but can also be SELECT statements can be simple but can also be

almost incomprehensiblealmost incomprehensible SELECT statements are broken up into clausesSELECT statements are broken up into clauses

FROM, WHERE, GROUP BY, HAVING, UNION, FROM, WHERE, GROUP BY, HAVING, UNION, INTERSECT, EXCEPT, ORDER BY, LIMIT, FOR INTERSECT, EXCEPT, ORDER BY, LIMIT, FOR UPDATEUPDATE

We will only look at a few simple queriesWe will only look at a few simple queries

Basic SELECTBasic SELECT SELECTSELECT

List of columns to extractList of columns to extract FROMFROM

List of tables to work withList of tables to work with WHEREWHERE

Condition to limit selectionCondition to limit selection GROUP BYGROUP BY

Grouping expressionGrouping expression HAVINGHAVING

Condition to limit groupingCondition to limit grouping ORDER BYORDER BY

Sort orderSort order LIMITLIMIT

Number of rows to extractNumber of rows to extract

SELECT Command StructureSELECT Command Structure

SELECT FROM WHERE GROUP BY

ORDER BY LIMITHAVING

List ofcolumns

List oftables

Condition Expression

Condition Sort order Number

Building a SELECT QueryBuilding a SELECT Query

You must know the schema (structure) of your You must know the schema (structure) of your database to know the tables and the columns database to know the tables and the columns you want to extractyou want to extract

Know the condition you want to filter the Know the condition you want to filter the selectionselection

Know the sort order you want to see the data, Know the sort order you want to see the data, the order will be undefined if not specified the order will be undefined if not specified

ExampleExample

Tables and columnsTables and columns location: lid, namelocation: lid, name Table names need to be included only if column names are Table names need to be included only if column names are

ambiguous ambiguous ConditionCondition

location.state = 'KS'location.state = 'KS' OrderOrder

location.namelocation.name

SELECT lid, name FROM locationWHERE state = 'KS'ORDER BY name;

Find all IDs and names in Kansas sorted by name

Joining Two TablesJoining Two Tables

Tables and columnsTables and columns location: lid, namelocation: lid, name observer: lastnameobserver: lastname

ConditionCondition location.lid = observer.lid and location.state = 'KS'location.lid = observer.lid and location.state = 'KS'

Sort orderSort order observer.lastnameobserver.lastname

Retrieve the list of locations, ids, and observers in KS sorted byobserver's last name

SELECT lid, name, lastname FROM location, observerWHERE location.lid = observer.lidAND location.state = 'KS'ORDER BY lastname;

More on WHERE ClauseMore on WHERE Clause

Comparison operators =, != or <>, >, >=, <, <=

Logical operators AND, OR, NOT

String concatenation 'a' || 'string'

BETWEEN operator WHERE fs BETWEEN 10 AND 50

IN operator WHERE state IN ('KY', 'IN', 'OH')

IS NULL test WHERE fs IS NULL

Matching in Character ColumnsMatching in Character Columns

LIKE Used to set up search targets % - zero or more characters _ - a single character

WHERE lid LIKE '___K1' or lid LIKE '___K2'

~ Used to set up POSIX regular expression searches

WHERE lid ~ '…K[12]' Use escape notation for backslash expressions

'…\w\d' becomes e'…\\w\\d'

SubstringsSubstrings

Use to extract part of a character column based Use to extract part of a character column based on position (count from 1)on position (count from 1)

SUBSTRING(column FROM begin FOR length)SUBSTRING(column FROM begin FOR length) Ex. show phone number of observers in the Ex. show phone number of observers in the

Kansas City KS Metro area codeKansas City KS Metro area code SELECT name, hphone FROM observer WHERE SELECT name, hphone FROM observer WHERE

SUBSTRING(hphone FROM 1 FOR 3) = '913'SUBSTRING(hphone FROM 1 FOR 3) = '913'

SELECT tricksSELECT tricks Testing an expression

SELECT SUBSTRING('ABCDE' FROM 1 FOR 3) Get all the fields in a table

SELECT * FROM location Name shortcut

SELECT l.lid, lastname FROM location l, observer o WHERE l.lid=o.lid AND lastname LIKE 'a%'

Rename columns SELECT lid AS Location_ID FROM location...

Eliminate duplicate rows SELECT DISTINCT county FROM location

Order shortcut SELECT lid, name FROM location ORDER BY 2

Descending order ...ORDER BY lid DESC

Date/Time/Timestamp FunctionsDate/Time/Timestamp Functions

AGE(timestamp, [timestamp])AGE(timestamp, [timestamp]) calculate interval between two timestamps, second calculate interval between two timestamps, second

argument defaults to NOWargument defaults to NOW ……WHERE AGE(obstime) < INTERVAL '6 hours'WHERE AGE(obstime) < INTERVAL '6 hours'

EXTRACT(unit from timestamp)EXTRACT(unit from timestamp) extract the unit from a timestampextract the unit from a timestamp common units: century, day, decade, dow (day of week common units: century, day, decade, dow (day of week

Sun=0, Mon=1, etc), doy (Julian date), hour, minute, Sun=0, Mon=1, etc), doy (Julian date), hour, minute, month, quarter, second, timezone, week, yearmonth, quarter, second, timezone, week, year

……WHERE EXTRACT(dow from datcrst) = 3WHERE EXTRACT(dow from datcrst) = 3

Group FunctionsGroup Functions COUNT()

Instead of displaying the rows, just count them SELECT COUNT(*) FROM observer

AVG() Calculate the average of a column

MAX() Calculate the maximum in a column

MIN() Calculate the minimum in a column

SUM() Calculate the sum of a column

The GROUP BY ClauseThe GROUP BY Clause

When using the previous functions, the GROUP BY clause tells what to group

We want the highest crest at EVERY location, sorted by ID

If we don't use GROUP BY we will get the highest crest of ALL IDs instead of the highest crest at EVERY ID.

SELECT l.lid, MAX(stage) FROM location l, crest cWHERE l.lid = c.lid GROUP BY l.lid ORDER BY l.lid;

The UPDATE CommandThe UPDATE Command

Used to quickly change the values in a columnUsed to quickly change the values in a column This command can save a lot of work but can This command can save a lot of work but can

also be dangerousalso be dangerous Has SET, FROM, and WHERE clausesHas SET, FROM, and WHERE clauses Here is a trick:Here is a trick:

Create the WHERE clause but use SELECT *Create the WHERE clause but use SELECT * You will see the rows you are going to changeYou will see the rows you are going to change Build rest of query Build rest of query

The UPDATE ClausesThe UPDATE Clauses

SETSET The column and the value you want to assign itThe column and the value you want to assign it

FROMFROM A list of tables you used in the SET clauseA list of tables you used in the SET clause

WHEREWHERE The conditions the data must matchThe conditions the data must match

UPDATE Command StructureUPDATE Command Structure

UPDATE FROM WHERE

TableList oftables

Condition

SET

Columnand value

Using UPDATEUsing UPDATE Ex. 913 area code changes to 914Ex. 913 area code changes to 914 Table and columnTable and column

observer: hphoneobserver: hphone ValueValue

‘‘914’ || SUBSTRING(hphone FROM 4)914’ || SUBSTRING(hphone FROM 4) ConditionCondition

SUBSTRING(hphone FOR 3) = '913'SUBSTRING(hphone FOR 3) = '913'

UPDATE observer SET hphone = '914‘ || SUBSTRING(hphone FROM 4)WHERE SUBSTRING(hphone FOR 3) = '913'

SELECT * FROM observer WHERE SUBSTRING(hphone FOR 3) = '913'

Check WHERE Clause

UPDATE rows

The INSERT INTO CommandThe INSERT INTO Command

Insert data into a new rowInsert data into a new row Has only VALUES clauseHas only VALUES clause Can also use a SELECT queryCan also use a SELECT query

The INESRT ClauseThe INESRT Clause

VALUESVALUES A list of data that you want to insertA list of data that you want to insert If the order of the columns is not listed, it must If the order of the columns is not listed, it must

match the order defined in the schemamatch the order defined in the schema For safety sake, always list the columnsFor safety sake, always list the columns

SELECTSELECT A standard select query, but the schema of the A standard select query, but the schema of the

result must match the targetresult must match the target

INSERT Command StructureINSERT Command Structure

INSERT INTO VALUES

Table,column list

List ofvalues

INSERT INTO SELECT

Table,column list

Selectquery

-or-

Using INSERTUsing INSERT

Want to add a new row in datumWant to add a new row in datum datum schema isdatum schema is

lid - location idlid - location id ddate - datum dateddate - datum date elev - elevationelev - elevation

Our values are ABEK1, 2002-3-23, 3056.34Our values are ABEK1, 2002-3-23, 3056.34

INSERT INTO datum (lid, ddate, elev)VALUES ('ABEK1', '2002-3-23', 3056.34);

Using INSERT (con't)Using INSERT (con't)

Want to add data from a temporary table Want to add data from a temporary table named junk into the height tablenamed junk into the height table

junk and height have the same schemajunk and height have the same schema

INSERT INTO height SELECT * FROM junk;

The DELETE FROM CommandThe DELETE FROM Command

Remove rowsRemove rows Has only WHERE clauseHas only WHERE clause There is no quick undo so be carefulThere is no quick undo so be careful If you accidentally leave off the WHERE If you accidentally leave off the WHERE

clause, ALL the rows will be deleted!clause, ALL the rows will be deleted! Here is another trick:Here is another trick:

Create the WHERE clause but use SELECT *Create the WHERE clause but use SELECT * You will see the rows you are going to deleteYou will see the rows you are going to delete Change SELECT * to DELETE Change SELECT * to DELETE

The DELETE ClauseThe DELETE Clause

WHEREWHERE The condition the rows must match The condition the rows must match

DELETE Command StructureDELETE Command Structure

DELETE FROM WHERE

Table Condition

Using DELETEUsing DELETE

Want to remove an observer to make a Want to remove an observer to make a location inactivelocation inactive

The location ID is CANK1The location ID is CANK1

DELETE FROM observer WHERE lid = 'CANK1'

SELECT * FROM observer WHERE lid = 'CANK1'Check WHERE Clause

DELETE row

The CREATE TEMP TABLE The CREATE TEMP TABLE CommandCommand

Creates a temporary table that can be used in Creates a temporary table that can be used in future queriesfuture queries

Table is automatically deleted once psql Table is automatically deleted once psql session is completed.session is completed.

In Snoopy, table is deleted at end of last In Snoopy, table is deleted at end of last statement in query.statement in query.

It has a simple structureIt has a simple structure

CREATE TEMP TABLE Command CREATE TEMP TABLE Command StructureStructure

CREATE TEMP TABLE AS

TableSelectquery

Using CREATE TEMP TABLEUsing CREATE TEMP TABLE Want to copy gage readings for the last three days into a temp Want to copy gage readings for the last three days into a temp

table, junktable, junk

CREATE TEMP TABLE junk ASSELECT * FROM heightWHERE AGE(obstime) < INTERVAL '3 days';

The COPY CommandThe COPY Command

There is a quick and easy way to copy a table to a fileThere is a quick and easy way to copy a table to a file You could process a table without working in SQLYou could process a table without working in SQL This is similar to the Informix UNLOAD commandThis is similar to the Informix UNLOAD command It has many options but we will only look at a fewIt has many options but we will only look at a few It combines well with the CREATE TEMP TABLE It combines well with the CREATE TEMP TABLE

commandcommand You must be logged on as You must be logged on as postgrespostgres for this command for this command

to workto work

COPY Command StructureCOPY Command Structure

COPY TO WITH

File Options Table

FROM

Example COPY CommandsExample COPY Commands

COPY observer TO 'observer.dat' WITH COPY observer TO 'observer.dat' WITH DELIMITER '|';DELIMITER '|'; Same as Informix UNLOAD command Same as Informix UNLOAD command

except default delimiter is a TABexcept default delimiter is a TAB COPY crest TO 'crest.csv' WITH CSV;COPY crest TO 'crest.csv' WITH CSV;

Good way of moving data into ExcelGood way of moving data into Excel COPY observer FROM 'observer.dat' COPY observer FROM 'observer.dat'

WITH DELIMITER '|';WITH DELIMITER '|'; Moving data into a tableMoving data into a table

Combining CREATE TEMP Combining CREATE TEMP TABLE and COPYTABLE and COPY

We want to copy today's stages into a file, We want to copy today's stages into a file, stg_tdy.datstg_tdy.dat

CREATE TEMP TABLE junk ASSELECT * FROM heightWHERE AGE(obstime) < INTERVAL '1 day';COPY junk TO stg_tdy.dat;

In ConclusionIn Conclusion

PostgreSQL may not be as popular as MySQL, it will PostgreSQL may not be as popular as MySQL, it will do what we need it to dodo what we need it to do

The PostgreSQL documentation is excellent and The PostgreSQL documentation is excellent and available on-line as HTML or off-line as PDFavailable on-line as HTML or off-line as PDF

psql is good to use in scriptspsql is good to use in scripts Snoopy is good to use interactivelySnoopy is good to use interactively The best way to learn SQL is to play with itThe best way to learn SQL is to play with it You cannot harm data by SELECT queriesYou cannot harm data by SELECT queries