python in civil/environmental engineering

PYTHON IN CIVIL AND ENVIRONMENTAL ENGINEERING

PAUL HOBSON GEOSYNTEC CONSULTANTS PDX PYTHON 2016-03-24

PYTHON IN CIVIL ENGINEERING

HEALTH AND SAFETY

▸ Nearest exit in case of emergency

▸ Ergonomics

▸ Right angles

▸ Head level

▸ Take breaks, look and move around


I’M AN ENGINEER (I STUDIED MUD)

▸ What is Civil Engineering?

▸ Structural

▸ Transportation

▸ Construction Management

▸ Geotechnical

▸ Environmental

▸ Water and Natural Resources


I’M AN ENGINEER (I STUDIED MUD)

▸ What is Civil Engineering?

▸ Structural

▸ Transportation

▸ Construction Management

▸ Geotechnical

▸ Environmental

▸ Water and Natural Resources

Thesis Title:

Rheologic and Flume Erosion Characteristics of Georgia Sediments at Bridge Piers


I WORK AS AN ENVIRONMENTAL CONSULTANT

▸ Help industry with compliance and remediation

▸ Litigation support for allocation of environmental damages

▸ Grant-funded research with NCHRP, FHWA, WERF



▸ Build spatial databases of contaminant concentrations




▸ Scrape the web (brute force) to meteorologic and hydrologic data





▸ Statistical analysis of pollution mitigation and remediation systems






▸ Build inputs for, run, and analyze output from vetted numerical models






▸ Build inputs for, run, and analyze output from vetted numerical models

▸ Build tools to help my colleagues do the same


SUMMARY FOR THE REST OF THE TALK

▸ My take on the general state of the practice in engineering consulting

▸ Characteristic challenges of environmental consulting

▸ Tools I like to use

▸ Tools I have built over the course of my career


STATE OF THE PRACTICE - DATABASES



▸ Overwhelming everything is in MS Access




▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together





▸ Last Client’s DB






▸ 110 tables






▸ 110 tables

▸ 200 saved insert/create/update






▸ 110 tables


▸ 100 save select and pivot queries






▸ 110 tables



▸ 25 queries that deleted stuff






▸ 110 tables



▸ 25 queries that deleted stuff

▸ Everything connected with 40 VBA forms, 15 utility modules (9000 LOC)


STATE OF THE PRACTICE - ANALYSIS

▸ Excel is the behemoth that cannot be stopped

▸ Most people manage their data with it

▸ No one follows best practices (www.datacarpentry.org/spreadsheet-ecology-lesson)

▸ Will be pried from cold dead hands, despite will published statistical errors (resolved, IIRC)

http://www.datacarpentry.org/spreadsheet-ecology-lesson


MANNING’S EQUATION


STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS



▸ GUI or cmd-line based numerical models maintained by federal agencies




▸ SWMM

▸ 1-D urban hydrology

▸ Written in very legible C

▸ Open source

▸ Maintained by EPA, transitioning to UT Austin

▸ Compiles on Linux and Windows




▸ SWMM



▸ Open source



▸ HEC-RAS

▸ 2-D River hydraulics

▸ Completely closed source

▸ Guarded over by US Army Corps of Engineers




▸ SWMM



▸ Open source



▸ HEC-RAS




▸ EFDC




▸ SWMM



▸ Open source



▸ HEC-RAS




▸ EFDC

▸ Spaghetti FORTRAN




▸ SWMM



▸ Open source



▸ HEC-RAS




▸ EFDC


▸ Compiles on Linux but results are garbage




▸ SWMM



▸ Open source



▸ HEC-RAS




▸ EFDC



▸ 3-D river hydraulics, sediment and pollutant fate and transport




▸ SWMM



▸ Open source



▸ HEC-RAS




▸ EFDC



▸ 3-D river hydraulics, sediment and pollutant fate and transport

▸ Very optimized: 1 yr simulation takes ~ 1 day


STATE OF THE PRACTICE - DOMAIN SPECIFIC TOOLS

▸ All (numerically) solve the St. Venant Equations

▸ Differential equations in time and space

▸ Very difficult to even work with symbolically

▸ Large code bases

▸ Represent a significant intellectual investment from civil/environmental community


WHERE DOES PYTHON FIT IN ALL OF THIS?

▸ Scrape model inputs from web

▸ Hack input files for batch processing

▸ Move data around between formats

▸ General data and model results analysis

▸ (Rarely) Wrap C/Fortran libraries with ctypes/cython/numpy


CIVIL/ENVIRONMENTAL CHALLENGES

▸ Arcane input file formats

▸ Left-censored (non-detect) data

▸ Non-parametric statistics

▸ Regression-on-order statistics

▸ Kaplan-Meier

▸ Project management inertia

▸ Disconnect between libraries and project needs

▸ ~5 coworkers (out of ~600) are comfortable with python


TOOLS I USE

▸ numpy/scipy

▸ Fast numeric arrays implemented in C

▸ Specialized scientific functions

▸ Matplotlib/seaborn: generic and statistical 2-D visualizations

▸ pandas/statsmodels: table-like data structures and statistical models

▸ Jupyter: interactive computing via notebooks

▸ conda: sane installation of python packages on Windows and other operating systems


SCRAPING WEATHER DATA

▸ conda install —channel=phobson cloudside

▸ Python 3 fork of python-metar by Tom Pollard (github.com/tomp/python-metar.git)

▸ ~1000 lines of regex

▸ Cloudside uses python-metar to pull 5-min and hourly data from the FAA

▸ ~1000 lines of regex

▸ 24229KPDX PDX20120101000009801/01/12 00:00:31 5-MIN KPDX 010800Z 31005KT 10SM OVC085 04/02 A3026 -280 85 -1500 290/05 RMK A02 P0005

http://github.com/tomp/python-metar.git


WEATHER DATA

▸ Simple commands for bulk data download and parsing

▸ Creates a directory structure for each station, source, and stage of processing

▸ searches for existing files before attempting to download or parse

▸ High-level plotting functions


WQIO - WATER QUALITY, INFLOW/OUTFLOW



▸ conda install --channel=phobson wqio




▸ Centered around examining the efficacy of so-called BMPs





▸ Handles left-censored data with ROS

▸ No such things as having “zero” pollution

▸ Can only say that things are “less than” the instrument’s precision








▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate confidence intervals around statistics









▸ High-level plotting interface built on seaborn









▸ High-level plotting interface built on seaborn

▸ Basis for client-specific libraries that I distribute to them through conda


PARAMNORMAL - CONVENTIONALLY PARAMETERIZED STATISTICAL DISTRIBUTIONS

▸ http://phobson.github.io/paramnormal/index.html

▸ conda install --channel=phobson paramnormal

▸ Problem: scipy distributions are incredibly flexible, but also perhaps a bit over generalized

▸ Wanted to create an API that let statisticians of all caliber specify and fit distributions using the parameters they read about in text books.

▸ The activity module provides a simple interface for creating, fitting, and plotting statistical distributions

http://phobson.github.io/paramnormal/index.html


PARANORMAL: BRIEF EXAMPLES

▸ (super fast demo)


PARAMNORMAL: ENDORSEMENTS


MPL-PROBSCALE▸ Legit probability

scales for matplotlib

▸ Similar to a quantile plot, but expressed as a probability instead of a z-score

▸ Simply import probscale and you’re set

import probscale ax1.set_xscale('prob') ax1.set_xlim(left=2, right=98) ax1.set_xlabel('Normal probability scale')

ax2.set_xscale('prob', dist=paramnormal.beta(α=3, β=2)) ax2.set_xlim(left=2, right=98) ax2.set_xlabel('Beta probability scale (α=3, β=2)')


MPL-PROBSCALE▸ Can fit linear

regression in probability space

▸ Top-level functions for easy plotting


GENERATING MODEL GRIDS

▸ Multi-corner, curvilinear-orthogonal grids are fairly difficult to compute — lots of very expensive proprietary stuff out there.

▸ Dr. Pavel Sakov (Australian Bureau of Meteorology wrote gridgen-c

▸ Built a types interface with help from:

▸ Dr. Robert Hetland (Texas A&M)

▸ Dr. Richard Signell (USGS)

▸ Mac and Linux binaries available through conda

▸ Docs at: phobson.github.io/pygridgen

http://phobson.github.io/pygridgen


GRID EXAMPLE


HIGHER LEVEL GRID OPERATIONS

▸ conda install --channel=conda-forge pygridtools

▸ geosyntec.github.io/pygridtools

▸ Interactivity with grid generation parameters

▸ Simple, general manipulations methods (merge, split, refine, transform)

▸ Took inspiration from pandas/xarray/seaborn

▸ File IO with general and GIS data formats via fiona

http://geosyntec.github.io/pygridtools'


GRID EXAMPLE - MANIPULATION

grid1, grid2, grid3 = make_test_grids()

fig = ( grid2.merge(grid3, how='vert', where='-', shift=2) .merge(grid1, how='horiz', where='-', shift=11) .refine(10, axis=1, n_points=4) .refine(13, axis=0, n_points=2) .transform(lambda x: x*5 + 2) .update_cell_mask() .plotCells(ax=ax, cell_kws=dict(facecolor=palette[2], alpha=0.75)) )


GRIDS IN THE REAL WORLD: SULLIVAN CREEK, NE WASHINGTON


PROJECTING MODEL RESULTS ON THE GRID


RESOURCES FOR GETTING STARTED AS A MEAT-SPACE ENGINEER

▸ I was lucky enough to get started with numpy and matplotlib while:

▸ I was still very familiar with MATLAB

▸ Both projects were directly trying to take market share away from MATLAB

▸ APIs and efforts have changed since then.

▸ More recently, a new employee started with MATLAB experience from college. We gave them a copy of Python For Data Analysis by Wes McKinney (author of pandas), and were pretty productive with python in a couple of week.

http://shop.oreilly.com/product/0636920023784.do


LINK DUMP

▸ Me:

▸ https://twitter.com/pmhobson

▸ https://github.com/phobson

▸ My (documented) Projects

▸ http://geosyntec.github.io/pygridtools/

▸ http://phobson.github.io/paramnormal/

▸ http://phobson.github.io/mpl-probscale

▸ https://github.com/phobson/cloudside (docs forthcoming)

https://twitter.com/pmhobson

https://github.com/phobson

http://geosyntec.github.io/pygridtools/

http://phobson.github.io/paramnormal/

http://phobson.github.io/mpl-probscale

https://github.com/phobson/cloudside

DEMOS &

QUESTIONS.

Thank you

python in civil/environmental engineering

Technology