OverviewMethodology

CodeTest Cases

Results

Postgres Interface Testing

Joe Conwayjoe.conway@credativ.com

mail@joeconway.com

credativ USA

June 10, 2014

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

Intro

Motivation

Answer this question:

What is relative performance ofODBC, JDBC, and libpq?

Specifically:

Connection speedQuery result retrievalAlternate retrieval methodsMaterializing to file

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Approach

Seed database

Launch command line clients

Vary conditions

Collect durations

Analyze results

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Major Components

Command line clients

PG Extension

Driver, load, and result objects

Driver function

Launcher bash script

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Flow

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Clients

Individual command line clientsNative language for interface

libpq - CODBC - CJDBC - Java

As similar as possible source code structureIdentical command line argumentsHandle variety of test conditions

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Extension

C extension for command execution

Simple way to execute commandsComplete control over string escaping, etc.Allow entire test to be table drivenEasy test result (duration) data collection

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Driver Tables

alternateclientcopy ⇒ acc-n, acc-a, acc-y

conninfos ⇒ lcl:ipc, lcl:tcp, rmt:tcp

interfaces ⇒ libpq, ODBC, JDBC

printfracdest ⇒ p-all:null, p-none:null, p-all:file

sqlstratiter⇒ sql001:0:50k, sql50k:0:001, sql10m:0:001⇒ sql001:1:50k, sql50k:1:001, sql10m:1:001

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Driver View

testcases

Generates all testcases from driver tablesUses both JOIN (inner) and CROSS JOIN (Cartesian product)

iftest=# select * from testcases limit 1;

-[ RECORD 1 ]-----------------------------------------------------

testcase | ODBC:lcl:ipc:sql001:0:50k:acc-n:p-all:null

testcmd | odbctest -c DSN=pgsql_ipc \

-s "select * from t50k where id = 4242" \

-f 1 -k 0 -n 50000 > /dev/null 2>&1

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Load Tables

t10m - 10 million rowsiftest=# select * from t10m limit 1;

-[ RECORD 1 ]------------------------------------------

x1 | 1

x2 | 1aaaaaaaaaaaaa

x3 | 2bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

x4 | 3ccccccccccccc

x5 | 4aaaaaaaaaaaaa

x6 | 5bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

x7 | 6ccccccccccccc

x8 | 7aaaaaaaaaaaaa

x9 | 8bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

x10 | 9ccccccccccccc

t50k - 50 thousand rows

iftest=# select * from t50k limit 1;

(similar row but wider text)

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Results Table

testresults - aggregate test case representationiftest=# \x

Expanded display is on.

iftest=# select * from testresults limit 1;

-[ RECORD 1 ]----------------------------------------

testcase | JDBC:lcl:tcp:sql001:0:50k:acc-n:p-all:null

iter | 1

ts | 2014-01-20 06:34:09.442483-08

duration | 1.223566

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Results View

vtestresults - expanded test case representationiftest=# select * from vtestresults limit 1;

-[ RECORD 1 ]----------------------------

iface | JDBC

loc | lcl

comm | tcp

sql | sql001

connstrat | 0

numiter | 50k

extra | acc-n

prnt | p-all

dest | file

iter | 1

ts | 2014-01-20 06:34:09.442483-08

duration | 0.185309

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Driver Function

exectestcases()

Arguments

path - client binary location (command line clients)ldlibpath - linked library location (libpq.so)classpath - Java class location (JDBC and getopt jar files)outeriter - how many times to execute all test cases

Returns SETOF RECORD

iter - current outeritertcase - aggregate form test case identifierdur - duration of test case run

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

High LevelApproach Detail

Launcher script

Kicks off the entire automated test

Sets variables for exectestcases() argumentsCalls exectestcases() using psqlCurrently executes 288 test cases per runRuns for many hours and produces many GB output files

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script

libpq Client

See libpqtest.c

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script

ODBC Client

See odbctest.c, odbc.ini, odbcinst.ini

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script

JDBC Client

See jdbctest.java

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script

Extension Code

See pgiftest.*, Makefile.ext, Makefile

-- make the extension available

CREATE EXTENSION pgiftest;

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script

Object Creation SQL

See test-plan.sql

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script

Function SQL

See test-plan.sql

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script

bash Script

See runtests.sh

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

InterfacesConn TypeSizingPrintExtra

Interface Variations

SELECT * FROM interfaces;

ifname | testprog

--------+---------------

ODBC | odbctest

libpq | libpqtest

JDBC | java jdbctest

(3 rows)

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

InterfacesConn TypeSizingPrintExtra

Connection Type Variations

SELECT * FROM conninfos;

conndesc | ifname | conninfo

----------+--------+-------------------------------------------------------

lcl:ipc | ODBC | -c DSN=pgsql_ipc

lcl:tcp | ODBC | -c DSN=pgsql_lcl

rmt:tcp | ODBC | -c DSN=pgsql_rmt

lcl:ipc | libpq | -c postgresql://%2Ftmp/iftest?user=postgres

lcl:tcp | libpq | -c postgresql://localhost/iftest?user=postgres

rmt:tcp | libpq | -c postgresql://192.168.1.3/iftest?user=postgres

lcl:tcp | JDBC | -c jdbc:postgresql://localhost/iftest?user=postgres

rmt:tcp | JDBC | -c jdbc:postgresql://192.168.1.3/iftest?user=postgres

(8 rows)

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

InterfacesConn TypeSizingPrintExtra

Result Size, Conn Count, Num Executions

SELECT sqlstratiterdesc as desc,

sql,

connstrat as strat,

numiter

FROM sqlstratiter;

desc | sql | strat | numiter

--------------+-----------------------------------------+-------+----------

sql001:0:50k | -s "select * from t50k where id = 4242" | -k 0 | -n 50000

sql50k:0:001 | -s "select * from t50k" | -k 0 | -n 1

sql10m:0:001 | -s "select * from t10m" | -k 0 | -n 1

sql001:1:50k | -s "select * from t50k where id = 4242" | -k 1 | -n 50000

sql50k:1:001 | -s "select * from t50k" | -k 1 | -n 1

sql10m:1:001 | -s "select * from t10m" | -k 1 | -n 1

(6 rows)

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

InterfacesConn TypeSizingPrintExtra

Materialize (Y/N) and Destination

SELECT * FROM printfracdest;

pfddesc | printfrac | dest

-------------+--------------+-----------------

p-all:null | -p 1 | /dev/null

p-all:file | -p 1 | /tmp/iftest.out

p-none:null | -p 0 | /dev/null

(3 rows)

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

InterfacesConn TypeSizingPrintExtra

Use Alternate Fetch or COPY Interface

SELECT * FROM alternateclientcopy;

accdesc | ifname | sw

---------+--------+----

acc-n | ODBC |

acc-n | libpq |

acc-a | libpq | -a

acc-y | libpq | -y

acc-n | JDBC |

acc-y | JDBC | -y

(6 rows)

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

Raw Results

select sql || $$:$$ || numiter || $$:$$ || loc || $$:$$ ||

comm || $$:$$ || connstrat || $$:$$ || prnt || $$:$$ || dest

AS testcase

,iface || $$:$$ || extra as candidate

,round(duration::numeric, 3) as duration

,round((((duration - first_value(duration) OVER w)/

(first_value(duration) OVER w)) * 100)::numeric, 2)

AS pct_slower

,rank() OVER w

FROM

(

select iface,loc,comm,connstrat,sql,numiter,extra,prnt,dest

,avg(duration) as duration

from vtestresults group by 1,2,3,4,5,6,7,8,9

) as t

WINDOW w

AS (PARTITION BY sql,numiter,loc,comm,connstrat,prnt,dest

ORDER BY duration)

;

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql001:50k:lcl:ipc:0:p-all:file

sql001:50k ⇒ Query 1 row, 50k times

lcl:ipc ⇒ Local host via IPC connection

0 ⇒ Connect every query

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-n | 162.720

libpq:acc-a | 163.026

libpq:acc-y | 164.761

ODBC:acc-n | 236.307

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql001:50k:lcl:ipc:1:p-all:file

sql001:50k ⇒ Query 1 row, 50k times

lcl:ipc ⇒ Local host via IPC connection

1 ⇒ Connect once

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-n | 7.753

libpq:acc-a | 7.978

libpq:acc-y | 8.619

ODBC:acc-n | 14.321

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql001:50k:lcl:tcp:0:p-all:file

sql001:50k ⇒ Query 1 row, 50k times

lcl:tcp ⇒ Local host via TCP/IP connection

0 ⇒ Connect every query

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-a | 180.219

libpq:acc-n | 180.240

libpq:acc-y | 182.211

JDBC:acc-y | 195.415

JDBC:acc-n | 201.378

ODBC:acc-n | 253.118

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql001:50k:lcl:tcp:1:p-all:file

sql001:50k ⇒ Query 1 row, 50k times

lcl:tcp ⇒ Local host via TCP/IP connection

1 ⇒ Connect once

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-n | 9.733

libpq:acc-a | 9.815

libpq:acc-y | 10.628

JDBC:acc-y | 12.673

ODBC:acc-n | 19.645

JDBC:acc-n | 20.067

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql001:50k:rmt:tcp:0:p-all:file

sql001:50k ⇒ Query 1 row, 50k times

rmt:tcp ⇒ Remote host via TCP/IP connection

0 ⇒ Connect every query

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-n | 412.389

libpq:acc-a | 413.264

libpq:acc-y | 428.484

JDBC:acc-y | 449.693

JDBC:acc-n | 452.799

ODBC:acc-n | 536.782

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql001:50k:rmt:tcp:1:p-all:file

sql001:50k ⇒ Query 1 row, 50k times

rmt:tcp ⇒ Remote host via TCP/IP connection

1 ⇒ Connect once

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-n | 26.457

libpq:acc-a | 26.483

libpq:acc-y | 41.876

JDBC:acc-y | 44.093

JDBC:acc-n | 52.347

ODBC:acc-n | 53.972

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql50k:001:rmt:tcp:1:p-all:file

sql50k:001 ⇒ Query 50k rows, 1 time

rmt:tcp ⇒ Remote host via TCP/IP connection

1 ⇒ Connect once

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-a | 0.621

libpq:acc-y | 0.665

JDBC:acc-y | 0.674

libpq:acc-n | 0.736

ODBC:acc-n | 1.075

JDBC:acc-n | 2.611

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql10m:001:rmt:tcp:1:p-all:file

sql10m:001 ⇒ Query 10m rows, 1 time

rmt:tcp ⇒ Remote host via TCP/IP connection

1 ⇒ Connect once

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-a | 37.768

libpq:acc-y | 40.196

JDBC:acc-y | 41.706

libpq:acc-n | 55.658

ODBC:acc-n | 82.782

JDBC:acc-n | 424.858

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql10m:001:lcl:tcp:1:p-all:file

sql10m:001 ⇒ Query 10m rows, 1 time

lcl:tcp ⇒ Local host via TCP/IP connection

1 ⇒ Connect once

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-y | 27.021

libpq:acc-a | 28.960

JDBC:acc-y | 31.661

libpq:acc-n | 37.058

ODBC:acc-n | 65.726

JDBC:acc-n | 407.906

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

sql10m:001:lcl:tcp:1:p-all:file

sql10m:001 ⇒ Query 10m rows, 1 time

lcl:ipc ⇒ Local host via IPC connection

1 ⇒ Connect once

p-all:file ⇒ print every row to file

candidate | duration

-------------+----------

libpq:acc-y | 26.744

libpq:acc-a | 28.902

libpq:acc-n | 33.926

ODBC:acc-n | 63.34

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

Final Results

To be continued . . .

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

Future Work

Additional interfaces

Different versions of JDBC libraryAlternate JDBC librarynpgsql (.Net data provider)py-postgresql (native python interface)

Additional alternate fetch strategies

Prepared statementsServer-side COPY with client-side file fetch

Cost of SSL connection

Joe Conway AustinPUG

OverviewMethodology

CodeTest Cases

Results

PreliminaryFinalFuture

Questions?

Thank You!joe.conway@credativ.com

mail@joeconway.com

Joe Conway AustinPUG