postgresql for buoy data
TRANSCRIPT
PostgreSQL for Buoy DataSteve Ruberg, Steve Constant, Ron Muzzi, Russ
Miller, Joe Smith
Utilization of PostgreSQL Database for Real-Time Western Lake Erie Data
Storage and DisseminationRUBERG, S.A., CONSTANT, S.A., MUZZI, R.W., MILLER, R.J., and SMITH, J.P.
Many thanks!
What are we talking about today?
1. Western Lake Erie (WLE) Real-time (RT) data collection2. Database3. Interface
Map made by Kaye LaFond
Map made by Kaye LaFond
1. WLE RT-Data Collection
● Meteorological● Water Quality
○ Phosphorus○ Chlorophyll○ Blue-green Algae○ Nitrogen
Format
● Flat text● ~50 columns● Split by day● Several MBytes per season● High temporal resolution (every 15 min)
1. WLE RT-Data Collection
Problem
● Client-side computer variability● Data require intense processing to visualize● Tried complete client-side interface to this
○ Not great...
1. WLE RT-Data Collection
2. Database
● SQL database○ Relational Database Management System (RDBMS)○ Tables connected by keys○ Through PHP, server power can be leveraged
■ Data delivered in smaller, more digestible chunks
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP
2. Database - choices
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,
ACID compliance, etc, but you do care about ease and speed, you tend
to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848
2. Database - choices
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,
ACID compliance, etc, but you do care about ease and speed, you tend
to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848
● Deployment environment = Federal Laboratory
2. Database - choices
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,
ACID compliance, etc, but you do care about ease and speed, you tend
to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848
● Deployment environment = Federal Laboratory● National Buoy Data Center uses Oracle
○ According to: Handbook of Automated Data Quality Control Checks and Procedures (2009)○ http://www.ndbc.noaa.gov/NDBCHandbookofAutomatedDataQualityControl2009.pdf
2. Database - choices
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,
ACID compliance, etc, but you do care about ease and speed, you tend
to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848
● Deployment environment = Federal Laboratory● National Buoy Data Center uses Oracle
○ According to: Handbook of Automated Data Quality Control Checks and Procedures (2009)○ http://www.ndbc.noaa.gov/NDBCHandbookofAutomatedDataQualityControl2009.pdf
● Go for something a little more free
2. Database - choices
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
2. Database - PostgreSQL
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
■ Atomicity - all or nothing
2. Database - PostgreSQL
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
■ Atomicity - all or nothing■ Consistency - transactions done right
2. Database - PostgreSQL
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
■ Atomicity - all or nothing■ Consistency - transactions done right■ Isolation - result of concurrent transactions same
as if they were run separately
2. Database - PostgreSQL
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
■ Atomicity - all or nothing■ Consistency - transactions done right■ Isolation - result of concurrent transactions same
as if they were run separately■ Durability - transaction permanence
2. Database - PostgreSQL
● 2 parallel tables○ Configurations○ Data and Quality Flags
● R used to process text data, Unix for automation
2. Database - Structure
● Configuration ID (primary key)● Start Date● End Date● Notes● GMT Offset● Matrix with the configurations, 7 columns
2. Database - Configuration table
● Matrix with the configurations, 7 columns○ Variable name universal across tables○ Variable name unique to the data under config○ Data units (as applicable)○ Statistic of data (min, max, mean, etc.)○ Data type○ QARTOD inspired gross/climatological range limits
■ Min■ Max
2. Database - Configuration table
● Data and Quality Flags○ Data Label (primary key)○ Corresponding configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database - Data table
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
Rows here match with... columns here.
2. Database
https://www.glerl.noaa.gov/res/HABs_and_Hypoxia/rtMonSQL.php
https://www.glerl.noaa.gov/res/HABs_and_Hypoxia/rtMonSQL.php
https://www.glerl.noaa.gov/res/HABs_and_Hypoxia/rtMonSQL.php
Features
● FAST
Features
● FAST● Click location on map to
view by location○ Use the drop down too
Features
● FAST● Click location on map to
view by location○ Use the drop down too
● OR Select handful of variables to view across all sampling locations
Features
● FAST● Click location on map to
view by location○ Use the drop down too
● OR Select handful of variables to view across all sampling locations
● Pick central date and day range of interest, then click ‘Go!’ to refresh the page and view that time’s data
jpthesmithe.com/iaglr.html