python in civil/environmental engineering

PYTHON IN CIVIL AND ENVIRONMENTAL ENGINEERING

PAUL HOBSON GEOSYNTEC CONSULTANTS PDX PYTHON 2016-03-24

PYTHON IN CIVIL ENGINEERING

HEALTH AND SAFETY

▸ Nearest exit in case of emergency

▸ Ergonomics

▸ Right angles

▸ Head level

▸ Take breaks, look and move around

I’M AN ENGINEER (I STUDIED MUD)

▸ What is Civil Engineering?

▸ Structural

▸ Transportation

▸ Construction Management

▸ Geotechnical

▸ Environmental

▸ Water and Natural Resources

▸ Structural

▸ Transportation

▸ Geotechnical

▸ Environmental

Thesis Title:

Rheologic and Flume Erosion Characteristics of Georgia Sediments at Bridge Piers

▸ Structural

▸ Transportation

▸ Geotechnical

▸ Environmental

Thesis Title:

Rheologic and Flume Erosion Characteristics of Georgia Sediments at Bridge Piers

I WORK AS AN ENVIRONMENTAL CONSULTANT

▸ Help industry with compliance and remediation

▸ Litigation support for allocation of environmental damages

▸ Grant-funded research with NCHRP, FHWA, WERF

▸ Build spatial databases of contaminant concentrations

▸ Scrape the web (brute force) to meteorologic and hydrologic data

▸ Statistical analysis of pollution mitigation and remediation systems

▸ Build inputs for, run, and analyze output from vetted numerical models

▸ Build tools to help my colleagues do the same

SUMMARY FOR THE REST OF THE TALK

▸ My take on the general state of the practice in engineering consulting

▸ Characteristic challenges of environmental consulting

▸ Tools I like to use

▸ Tools I have built over the course of my career

STATE OF THE PRACTICE - DATABASES

▸ Overwhelming everything is in MS Access

▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together

▸ Last Client’s DB

▸ 110 tables

▸ 200 saved insert/create/update

▸ 110 tables

▸ 100 save select and pivot queries

▸ 110 tables

▸ 25 queries that deleted stuff

▸ 110 tables

▸ 25 queries that deleted stuff

▸ Everything connected with 40 VBA forms, 15 utility modules (9000 LOC)

STATE OF THE PRACTICE - ANALYSIS

▸ Excel is the behemoth that cannot be stopped

▸ Most people manage their data with it

▸ No one follows best practices (www.datacarpentry.org/spreadsheet-ecology-lesson)

▸ Will be pried from cold dead hands, despite will published statistical errors (resolved, IIRC)

MANNING’S EQUATION

STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS

▸ GUI or cmd-line based numerical models maintained by federal agencies

▸ SWMM

▸ 1-D urban hydrology

▸ Written in very legible C

▸ Open source

▸ Maintained by EPA, transitioning to UT Austin

▸ Compiles on Linux and Windows

▸ SWMM

▸ Open source

▸ HEC-RAS

▸ 2-D River hydraulics

▸ Completely closed source

▸ Guarded over by US Army Corps of Engineers

▸ SWMM

▸ Open source

▸ HEC-RAS

▸ EFDC

▸ SWMM

▸ Open source

▸ HEC-RAS

▸ EFDC

▸ Spaghetti FORTRAN

▸ SWMM

▸ Open source

▸ HEC-RAS

▸ EFDC

▸ Compiles on Linux but results are garbage

▸ SWMM

▸ Open source

▸ HEC-RAS

▸ EFDC

▸ 3-D river hydraulics, sediment and pollutant fate and transport

▸ SWMM

▸ Open source

▸ HEC-RAS

▸ EFDC

▸ 3-D river hydraulics, sediment and pollutant fate and transport

▸ Very optimized: 1 yr simulation takes ~ 1 day

STATE OF THE PRACTICE - DOMAIN SPECIFIC TOOLS

▸ All (numerically) solve the St. Venant Equations

▸ Differential equations in time and space

▸ Very difficult to even work with symbolically

▸ Large code bases

▸ Represent a significant intellectual investment from civil/environmental community

WHERE DOES PYTHON FIT IN ALL OF THIS?

▸ Scrape model inputs from web

▸ Hack input files for batch processing

▸ Move data around between formats

▸ General data and model results analysis

▸ (Rarely) Wrap C/Fortran libraries with ctypes/cython/numpy

CIVIL/ENVIRONMENTAL CHALLENGES

▸ Arcane input file formats

▸ Left-censored (non-detect) data

▸ Non-parametric statistics

▸ Regression-on-order statistics

▸ Kaplan-Meier

▸ Project management inertia

▸ Disconnect between libraries and project needs

▸ ~5 coworkers (out of ~600) are comfortable with python

TOOLS I USE

▸ numpy/scipy

▸ Fast numeric arrays implemented in C

▸ Specialized scientific functions

▸ Matplotlib/seaborn: generic and statistical 2-D visualizations

▸ pandas/statsmodels: table-like data structures and statistical models

▸ Jupyter: interactive computing via notebooks

▸ conda: sane installation of python packages on Windows and other operating systems

SCRAPING WEATHER DATA

▸ conda install —channel=phobson cloudside

▸ Python 3 fork of python-metar by Tom Pollard (github.com/tomp/python-metar.git)

▸ ~1000 lines of regex

▸ Cloudside uses python-metar to pull 5-min and hourly data from the FAA

▸ ~1000 lines of regex

▸ 24229KPDX PDX20120101000009801/01/12 00:00:31 5-MIN KPDX 010800Z 31005KT 10SM OVC085 04/02 A3026 -280 85 -1500 290/05 RMK A02 P0005

WEATHER DATA

▸ Simple commands for bulk data download and parsing

▸ Creates a directory structure for each station, source, and stage of processing

▸ searches for existing files before attempting to download or parse

▸ High-level plotting functions

WQIO - WATER QUALITY, INFLOW/OUTFLOW

▸ conda install --channel=phobson wqio

▸ Centered around examining the efficacy of so-called BMPs

▸ Handles left-censored data with ROS

▸ No such things as having “zero” pollution

▸ Can only say that things are “less than” the instrument’s precision

▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate confidence intervals around statistics

▸ High-level plotting interface built on seaborn

▸ Basis for client-specific libraries that I distribute to them through conda

PARAMNORMAL - CONVENTIONALLY PARAMETERIZED STATISTICAL DISTRIBUTIONS

▸ http://phobson.github.io/paramnormal/index.html

▸ conda install --channel=phobson paramnormal

▸ Problem: scipy distributions are incredibly flexible, but also perhaps a bit over generalized

▸ Wanted to create an API that let statisticians of all caliber specify and fit distributions using the parameters they read about in text books.

▸ The activity module provides a simple interface for creating, fitting, and plotting statistical distributions

PARANORMAL: BRIEF EXAMPLES

▸ (super fast demo)

PARAMNORMAL: ENDORSEMENTS

MPL-PROBSCALE▸ Legit probability

scales for matplotlib

▸ Similar to a quantile plot, but expressed as a probability instead of a z-score

▸ Simply import probscale and you’re set

import probscale ax1.set_xscale('prob') ax1.set_xlim(left=2, right=98) ax1.set_xlabel('Normal probability scale')

ax2.set_xscale('prob', dist=paramnormal.beta(α=3, β=2)) ax2.set_xlim(left=2, right=98) ax2.set_xlabel('Beta probability scale (α=3, β=2)')

MPL-PROBSCALE▸ Can fit linear

regression in probability space

▸ Top-level functions for easy plotting

GENERATING MODEL GRIDS

▸ Multi-corner, curvilinear-orthogonal grids are fairly difficult to compute — lots of very expensive proprietary stuff out there.

▸ Dr. Pavel Sakov (Australian Bureau of Meteorology wrote gridgen-c

▸ Built a types interface with help from:

▸ Dr. Robert Hetland (Texas A&M)

▸ Dr. Richard Signell (USGS)

▸ Mac and Linux binaries available through conda

▸ Docs at: phobson.github.io/pygridgen

GRID EXAMPLE

HIGHER LEVEL GRID OPERATIONS

▸ conda install --channel=conda-forge pygridtools

▸ geosyntec.github.io/pygridtools

▸ Interactivity with grid generation parameters

▸ Simple, general manipulations methods (merge, split, refine, transform)

▸ Took inspiration from pandas/xarray/seaborn

▸ File IO with general and GIS data formats via fiona

GRID EXAMPLE - MANIPULATION

grid1, grid2, grid3 = make_test_grids()

fig = ( grid2.merge(grid3, how='vert', where='-', shift=2) .merge(grid1, how='horiz', where='-', shift=11) .refine(10, axis=1, n_points=4) .refine(13, axis=0, n_points=2) .transform(lambda x: x*5 + 2) .update_cell_mask() .plotCells(ax=ax, cell_kws=dict(facecolor=palette[2], alpha=0.75)) )

GRIDS IN THE REAL WORLD: SULLIVAN CREEK, NE WASHINGTON

PROJECTING MODEL RESULTS ON THE GRID

RESOURCES FOR GETTING STARTED AS A MEAT-SPACE ENGINEER

▸ I was lucky enough to get started with numpy and matplotlib while:

▸ I was still very familiar with MATLAB

▸ Both projects were directly trying to take market share away from MATLAB

▸ APIs and efforts have changed since then.

▸ More recently, a new employee started with MATLAB experience from college. We gave them a copy of Python For Data Analysis by Wes McKinney (author of pandas), and were pretty productive with python in a couple of week.

LINK DUMP

▸ Me:

▸ https://twitter.com/pmhobson

▸ https://github.com/phobson

▸ My (documented) Projects

▸ http://geosyntec.github.io/pygridtools/

▸ http://phobson.github.io/paramnormal/

▸ http://phobson.github.io/mpl-probscale

▸ https://github.com/phobson/cloudside (docs forthcoming)

DEMOS &

QUESTIONS.

Thank you

python in civil/environmental engineering

Technology

civil & environmental engineering civil engineering

nv iro m journal of civil & environmental hekal civil

python - centre for environmental data analysis · python...

civil sanctions for environmental offences - fwr home · pdf...

westera civil and environmental

e d - environmental dependencies in python

unsw civil & environmental engineering 2014

department of civil and environmental engineering -...

civil & environmental engineering graduate …

civil & environmental engineering design

department of civil, structural and environmental...

civil and environmental · pdf file1 civil and...

civil & environmental engineering research

department of civil and environmental · pdf filedepartment...

civil & environmental consultants

school of civil & environmental … school of civil &...

school of civil & environmental …...

j civil environment engg civil & environmental...

civil and environmental engineering erosion …...civil and...

civil & environmental engineering program