sql/med - doping for postgresql€¦ · this presentation “sql/med: doping for postgresql” was...
TRANSCRIPT
![Page 1: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/1.jpg)
SQL/MEDDoping for PostgreSQL
Peter Eisentraut
Senior Software EngineerLab Development
F-Secure Corporation
PGCon 2009
![Page 2: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/2.jpg)
SQL/MED: Management of External Data
I MED = Management of ExternalData
I Methods to access data storedoutside the database systemthrough normal SQL
I SQL/MED is ISO/IEC 9075-9
![Page 3: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/3.jpg)
Applications and Use Cases
I Connect to other DBMS (like DBI-Link)I Other primary data storage: Oracle, MySQL, . . .I Data warehouses etc.: Greenplum, Truviso, . . .
I Connect to other PostgreSQL instances (like dblink)I Read non-SQL data
I Files: CSV, XML, JSON, . . .I File systems: Google FS, Hadoop FS, Lustre, . . .I Databases: CouchDB, BigTable, NDB, S3, . . . (“Cloud” stuff)I Memcache
I Clustering, partitioning (think PL/Proxy)I Manage data stored in file system
I ImagesI VideoI Engineering data
![Page 4: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/4.jpg)
Applications and Use Cases
I Connect to other DBMS (like DBI-Link)I Other primary data storage: Oracle, MySQL, . . .I Data warehouses etc.: Greenplum, Truviso, . . .
I Connect to other PostgreSQL instances (like dblink)I Read non-SQL data
I Files: CSV, XML, JSON, . . .I File systems: Google FS, Hadoop FS, Lustre, . . .I Databases: CouchDB, BigTable, NDB, S3, . . . (“Cloud” stuff)I Memcache
I Clustering, partitioning (think PL/Proxy)I Manage data stored in file system
I ImagesI VideoI Engineering data
![Page 5: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/5.jpg)
Why do we care?
I Unifies existing ad-hoc solutions.I Powerful new functionalityI Makes PostgreSQL the center of data management.I Implementation has begun in PostgreSQL 8.4.I Several people have plans for PostgreSQL 8.5.I See status report later in this presentation.
![Page 6: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/6.jpg)
Advantages
Schema integration All data appears as tables.Access control Use GRANT/REVOKE for everything.Standard APIs Mix and share.Centralized control Manage all data through the DBMS.
![Page 7: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/7.jpg)
Implications for Application Design andDeployment
Before
application application application application application
Oracle MySQL MySQLCluster
PostgreSQL file system
![Page 8: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/8.jpg)
Implications for Application Design andDeployment
After
application application application application application
PostgreSQL
MySQLCluster
file system CouchDBMySQLOracle
![Page 9: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/9.jpg)
The Two Parts of SQL/MED
Wrapper interface Access other data sources, represent themas SQL tables
Datalinks Manage files stored in file system, represent filereferences as column values
![Page 10: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/10.jpg)
Wrapper Interface Concepts
I Define a foreign table . . .
I think: a dblink view
I On a foreign server . . .
I think: dblink_connect
I Accessed through a foreign-data wrapper
I think: dblink.so library
![Page 11: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/11.jpg)
Wrapper Interface Concepts
I Define a foreign table . . .I think: a dblink view
I On a foreign server . . .I think: dblink_connect
I Accessed through a foreign-data wrapperI think: dblink.so library
![Page 12: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/12.jpg)
Foreign-Data Wrappers
Foreign-data wrapper (FDW): a library that can communicatewith external data sources
CREATE FOREIGN DATA WRAPPER foosql
LIBRARY 'foosql_fdw.so'
LANGUAGE C;
I PostgreSQL communicates with foosql_fdw.so usingSQL/MED FDW API.
I foosql_fdw.so communicates with FooSQL server usingtheir own protocol.
I In theory, FooSQL, Inc. would ship foosql_fdw.so withtheir product.
I In practice, this is not so wide-spread.
![Page 13: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/13.jpg)
Foreign Servers
Foreign server: an instance of an external data sourceaccessed through a FDW
CREATE SERVER extradb
FOREIGN DATA WRAPPER foosql
OPTIONS (host 'foo.example.com', port '2345');
I Options depend on FDW.
![Page 14: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/14.jpg)
User Mappings
User mapping: additional user-specific options for a foreignserver
CREATE USER MAPPING FOR peter SERVER extradb
OPTIONS (user 'peter', password 'seKret');
I Options depend on FDW.I Putting connection options into server vs. user mapping is
a matter of convention or convenience.
![Page 15: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/15.jpg)
Foreign Tables
Foreign table: a table stored on a foreign server
CREATE FOREIGN TABLE data
SERVER extradb
OPTIONS (tablename 'DATA123');
I Now you can read and write the table as if it were local(depending on FDW features/implementation).
I Options specified for FDW, server, and user mapping areused as connection parameters (depending on FDW).
![Page 16: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/16.jpg)
Another Wrapper Interface Example
Possible setup for accessing HTML tables stored in a web siteas SQL tables:
CREATE FOREIGN DATA WRAPPER htmlfile
LIBRARY 'html_fdw.so'
LANGUAGE C;
CREATE SERVER intranetweb
FOREIGN DATA WRAPPER htmlfile
OPTIONS (baseurl 'http://intranet/data');
CREATE FOREIGN TABLE data
SERVER intranetweb
OPTIONS (path 'foo.html#//table[@id="table1"]');
![Page 17: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/17.jpg)
Routine Mappings
Routine mappings: passing a function/procedure through to aforeign server
CREATE ROUTINE MAPPING <routine mapping name>
FOR <specific routine designator>
SERVER <foreign server name>
[ <generic options> ];
![Page 18: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/18.jpg)
Routine Mappings Examples
Example like PL/Proxy:
CREATE ROUTINE MAPPING myfunc(a int, b text)
SERVER plproxydb
OPTIONS (cluster 'somecluster',
runon 'hashtext(a)');
Example XML-RPC:
CREATE ROUTINE MAPPING process(data xml)
SERVER xmlrpc
OPTIONS (request '<methodCall>...</methodCall>');
![Page 19: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/19.jpg)
Wrapper Interface Access Control
I GRANT USAGE ON FOREIGN DATA WRAPPER
I GRANT USAGE FOREIGN SERVER
I Foreign tables and routines have regular privileges.I Passwords for remote access can be managed via user
mappings.I Front-to-end Kerberos or SSL support could be cool.
![Page 20: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/20.jpg)
Importing Foreign Schemas
Automatically create foreign tables based on tables availableremotely.
IMPORT FOREIGN SCHEMA someschema
LIMIT TO (tab1, tab2, tab2)
FROM SERVER extradb
INTO myschema;
(SQL standard doesn’t support foreign routine import.)
![Page 21: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/21.jpg)
Status of SQL/MED in PostgreSQL 8.4
PostgreSQL 8.4 has:I CREATE FOREIGN DATA WRAPPER, but no library supportI CREATE SERVER
I CREATE USER MAPPING
I ACL supportI Doesn’t really do anything :-(I Plans for PL/Proxy to store connection information
![Page 22: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/22.jpg)
Status of SQL/MED Elsewhere
I IBM DB2 provides a full implementation.I MySQL and Farrago use some syntax elements.I No other known implementations.I Some vendors have had their custom remote access
functionality.
![Page 23: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/23.jpg)
Plan & Issues
PostgreSQL 8.5 and beyond . . .I Write wrapper library and foreign table supportI Supply a few foreign-data wrapper librariesI Use standard wrapper interface API or design our own
API?I Optimizations, e. g., passing query qualifications to foreign
serversI Distributed transactionsI Needs careful security evaluation (remember dblink issues)
![Page 24: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/24.jpg)
Plan & Issues
PostgreSQL 8.5 and beyond . . .I Write wrapper library and foreign table supportI Supply a few foreign-data wrapper librariesI Use standard wrapper interface API or design our own
API?I Optimizations, e. g., passing query qualifications to foreign
serversI Distributed transactionsI Needs careful security evaluation (remember dblink issues)
![Page 25: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/25.jpg)
Datalink Concepts
I Files are referenced through a new DATALINK typeI Database system has control over external filesI No need to store file contents in database systemI Access control and integrity mechanisms of DBMS can be
extended to file system
![Page 26: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/26.jpg)
Datalink Use Cases
I Certain types of data are primarily uses as files withexternal applications.
I Handling very large files (e. g., video) by DBMS isinefficient
I Use of distributed files systemsI Handle files stored on web server, FTP server, etc.
![Page 27: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/27.jpg)
Example: Simple DATALINK Type
CREATE TABLE persons (
id integer,
name text,
picture DATALINK [NO LINK CONTROL]
);
INSERT INTO persons VALUES (
1,
'Jon Doe',
DLVALUE('file://some/where/1.jpg')
);
I SQL specifies support for file: and http:.I This variant doesn’t do anything except store URLs.
![Page 28: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/28.jpg)
DATALINK Attributes: Link and Integrity Control
NO LINK CONTROL Datalink value need not reference anexisting file/URL.
FILE LINK CONTROL Datalink value must reference anexisting file/URL.
INTEGRITY ALL Referenced files can only be renamed ordeleted through SQL.
INTEGRITY SELECTIVE Referenced files can be renamed ordeleted through SQL or directly.
INTEGRITY NONE (implied for NO LINK CONTROL)
![Page 29: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/29.jpg)
DATALINK Attributes: Unlinking and RecoveryBehavior
ON UNLINK DELETE File is deleted from file system whendeleted from database.
ON UNLINK RESTORE File’s original permissions arerestored when deleted from database.
ON UNLINK NONE No change in file permissions when filereference is deleted from database.
RECOVERY YES PITR applies to referenced files.RECOVERY NO PITR does not apply to referenced files.
![Page 30: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/30.jpg)
DATALINK Attributes: Access Permissions
READ PERMISSION FS File system controls file readpermission.
READ PERMISSION DB Database system controls file readpermission.
WRITE PERMISSION FS File system controls file writepermission.
WRITE PERMISSION ADMIN Writes to the file are managedby the database system.
WRITE PERMISSION BLOCKED Writing to file is blocked.
![Page 31: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/31.jpg)
How to Implement Datalinks
Implementation challenges:I OS-dependentI File-system dependentI Application-dependent
Possibilities:I Kernel modulesI LD_PRELOADI Extended FS attributesI Lots of hocus pocus
Don’t hold your breath.
![Page 32: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/32.jpg)
Summary
SQL/MED
I Wrapper interfaceI DatalinksI Substantial support planned for PostgreSQL 8.5 and
beyond
Further reading:I http://wiki.postgresql.org/wiki/SqlMedConnectionManager
(Martin Pihlak)I http://www.sigmod.org/record/issues/0103/JM-Sta.pdf (Jim Melton et al.)I http://www.sigmod.org/record/issues/0209/jimmelton.pdf
(Jim Melton et al.)I ISO/IEC 9075-9:2008 (“SQL/MED”)
![Page 33: SQL/MED - Doping for PostgreSQL€¦ · This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under the Creative Commons Attribution-Noncommercial-Share](https://reader034.vdocument.in/reader034/viewer/2022042217/5ec062182d5bfd3bc350a2c4/html5/thumbnails/33.jpg)
Rights and Attributions
This presentation “SQL/MED: Doping for PostgreSQL” was authored by Peter Eisentraut and is licensed under theCreative Commons Attribution-Noncommercial-Share Alike 3.0 Unported license.
I The image on page 2 is from the Open Clip Art Library and is in the public domain.I The image on page 5 is from the Open Clip Art Library and is in the public domain.I The image on page 9 is “The fork in the road” by Flickr user i_yudai, available under the Creative Commons
Attribution 2.0 Generic license.I The image on page 31 is “Fork in a Steve” by Flickr user morgantepsic, available under the Creative
Commons Attribution-Share Alike 2.0 Generic license.