Python in Civil/Environmental Engineering

PYTHON IN CIVIL AND
ENVIRONMENTAL ENGINEERING
PAUL HOBSON
GEOSYNTEC CONSULTANTS
PDX PYTHON 2016-03-24

PYTHON IN CIVIL ENGINEERING
HEALTH AND SAFETY
▸ Nearest exit in case of emergency
▸ Ergonomics
▸ Right angles
▸ Head level
▸ Take breaks, look and move around

I’M AN ENGINEER (I STUDIED MUD)
▸ What is Civil Engineering?
▸ Structural
▸ Transportation
▸ Construction Management
▸ Geotechnical
▸ Environmental
▸ Water and Natural Resources

I’M AN ENGINEER (I STUDIED MUD)
▸ What is Civil Engineering?
▸ Structural
▸ Transportation
▸ Construction Management
▸ Geotechnical
▸ Environmental
▸ Water and Natural Resources
Thesis Title:
Rheologic and Flume Erosion
Characteristics of Georgia
Sediments at Bridge Piers

I WORK AS AN ENVIRONMENTAL CONSULTANT
▸ Help industry with compliance and remediation
▸ Litigation support for allocation of environmental
damages
▸ Grant-funded research with NCHRP, FHWA, WERF

▸ Build spatial databases of contaminant concentrations

▸ Scrape the web (brute force) to meteorologic and
hydrologic data

hydrologic data
▸ Statistical analysis of pollution mitigation and remediation
systems

hydrologic data
systems
▸ Build inputs for, run, and analyze output from vetted
numerical models

hydrologic data
systems
▸ Build inputs for, run, and analyze output from vetted
numerical models
▸ Build tools to help my colleagues do the same

SUMMARY FOR THE REST OF THE TALK
▸ My take on the general state of the practice in engineering
consulting
▸ Characteristic challenges of environmental consulting
▸ Tools I like to use
▸ Tools I have built over the course of my career

STATE OF THE PRACTICE - DATABASES

▸ Overwhelming everything is in MS Access

▸ PROTIP: Your MS Ofﬁce, MS Access Drivers, and python all must have the
same architecture to work together

▸ Last Client’s DB

▸ 110 tables

▸ 110 tables
▸ 200 saved insert/create/update

▸ 110 tables
▸ 100 save select and pivot queries

▸ 110 tables
▸ 25 queries that deleted stuff

▸ 110 tables
▸ 25 queries that deleted stuff
▸ Everything connected with 40 VBA forms, 15 utility modules (9000 LOC)

STATE OF THE PRACTICE - ANALYSIS
▸ Excel is the behemoth that cannot be stopped
▸ Most people manage their data with it
▸ No one follows best practices (www.datacarpentry.org/
spreadsheet-ecology-lesson)
▸ Will be pried from cold dead hands, despite will published
statistical errors (resolved, IIRC)

MANNING’S EQUATION

STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS

▸ GUI or cmd-line based numerical models maintained
by federal agencies

by federal agencies
▸ SWMM
▸ 1-D urban hydrology
▸ Written in very legible C
▸ Open source
▸ Maintained by EPA, transitioning to UT Austin
▸ Compiles on Linux and Windows

by federal agencies
▸ SWMM
▸ Open source
▸ HEC-RAS
▸ 2-D River hydraulics
▸ Completely closed source
▸ Guarded over by US Army Corps of Engineers

by federal agencies
▸ SWMM
▸ Open source
▸ HEC-RAS
▸ EFDC

by federal agencies
▸ SWMM
▸ Open source
▸ HEC-RAS
▸ EFDC
▸ Spaghetti FORTRAN

by federal agencies
▸ SWMM
▸ Open source
▸ HEC-RAS
▸ EFDC
▸ Compiles on Linux but results are
garbage

by federal agencies
▸ SWMM
▸ Open source
▸ HEC-RAS
▸ EFDC
garbage
▸ 3-D river hydraulics, sediment and
pollutant fate and transport

by federal agencies
▸ SWMM
▸ Open source
▸ HEC-RAS
▸ EFDC
garbage
▸ 3-D river hydraulics, sediment and
pollutant fate and transport
▸ Very optimized: 1 yr simulation takes ~
1 day

STATE OF THE PRACTICE - DOMAIN SPECIFIC TOOLS
▸ All (numerically) solve the St. Venant Equations
▸ Differential equations in time and space
▸ Very difﬁcult to even work with symbolically
▸ Large code bases
▸ Represent a signiﬁcant intellectual investment from civil/
environmental community

WHERE DOES PYTHON FIT IN ALL OF THIS?
▸ Scrape model inputs from web
▸ Hack input ﬁles for batch processing
▸ Move data around between formats
▸ General data and model results analysis
▸ (Rarely) Wrap C/Fortran libraries with ctypes/cython/
numpy

CIVIL/ENVIRONMENTAL CHALLENGES
▸ Arcane input ﬁle formats
▸ Left-censored (non-detect) data
▸ Non-parametric statistics
▸ Regression-on-order statistics
▸ Kaplan-Meier
▸ Project management inertia
▸ Disconnect between libraries and project needs
▸ ~5 coworkers (out of ~600) are comfortable with python

TOOLS I USE
▸ numpy/scipy
▸ Fast numeric arrays implemented in C
▸ Specialized scientiﬁc functions
▸ Matplotlib/seaborn: generic and statistical 2-D visualizations
▸ pandas/statsmodels: table-like data structures and statistical models
▸ Jupyter: interactive computing via notebooks
▸ conda: sane installation of python packages on Windows and other
operating systems

SCRAPING WEATHER DATA
▸ conda install —channel=phobson cloudside
▸ Python 3 fork of python-metar by Tom Pollard (github.com/tomp/python-
metar.git)
▸ ~1000 lines of regex
▸ Cloudside uses python-metar to pull 5-min and hourly data from the FAA
▸ ~1000 lines of regex
▸ 24229KPDX PDX20120101000009801/01/12 00:00:31 5-MIN KPDX
010800Z 31005KT 10SM OVC085 04/02 A3026 -280 85 -1500 290/05
RMK A02 P0005

WEATHER DATA
▸ Simple commands for bulk data download and parsing
▸ Creates a directory structure for each station, source, and
stage of processing
▸ searches for existing ﬁles before attempting to download or
parse
▸ High-level plotting functions

WQIO - WATER QUALITY, INFLOW/OUTFLOW

▸ conda install --channel=phobson wqio

▸ Centered around examining the efﬁcacy of so-called BMPs

▸ Handles left-censored data with ROS
▸ No such things as having “zero” pollution
▸ Can only say that things are “less than” the instrument’s precision

▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate
conﬁdence intervals around statistics

▸ High-level plotting interface built on seaborn

▸ High-level plotting interface built on seaborn
▸ Basis for client-speciﬁc libraries that I distribute to them through conda

PARAMNORMAL - CONVENTIONALLY PARAMETERIZED STATISTICAL DISTRIBUTIONS
▸ http://phobson.github.io/paramnormal/index.html
▸ conda install --channel=phobson paramnormal
▸ Problem: scipy distributions are incredibly flexible, but also
perhaps a bit over generalized
▸ Wanted to create an API that let statisticians of all caliber specify
and fit distributions using the parameters they read about in text
books.
▸ The activity module provides a simple interface for creating,
fitting, and plotting statistical distributions

PARANORMAL: BRIEF EXAMPLES
▸ (super fast demo)

PARAMNORMAL: ENDORSEMENTS

MPL-PROBSCALE
▸ Legit probability
scales for matplotlib
▸ Similar to a quantile
plot, but expressed
as a probability
instead of a z-score
▸ Simply import
probscale and
you’re set
import probscale
ax1.set_xscale('prob')
ax1.set_xlim(left=2, right=98)
ax1.set_xlabel('Normal probability scale')
ax2.set_xscale('prob', dist=paramnormal.beta(α=3, β=2))
ax2.set_xlim(left=2, right=98)
ax2.set_xlabel('Beta probability scale (α=3, β=2)')

MPL-PROBSCALE
▸ Can ﬁt linear
regression in
probability space
▸ Top-level functions
for easy plotting

GENERATING MODEL GRIDS
▸ Multi-corner, curvilinear-orthogonal grids are fairly difﬁcult to
compute — lots of very expensive proprietary stuff out there.
▸ Dr. Pavel Sakov (Australian Bureau of Meteorology wrote gridgen-c
▸ Built a types interface with help from:
▸ Dr. Robert Hetland (Texas A&M)
▸ Dr. Richard Signell (USGS)
▸ Mac and Linux binaries available through conda
▸ Docs at: phobson.github.io/pygridgen

GRID EXAMPLE

HIGHER LEVEL GRID OPERATIONS
▸ conda install --channel=conda-forge pygridtools
▸ geosyntec.github.io/pygridtools
▸ Interactivity with grid generation parameters
▸ Simple, general manipulations methods (merge, split,
reﬁne, transform)
▸ Took inspiration from pandas/xarray/seaborn
▸ File IO with general and GIS data formats via ﬁona

GRID EXAMPLE - MANIPULATION
grid1, grid2, grid3 = make_test_grids()
fig = (
grid2.merge(grid3, how='vert', where='-', shift=2)
.merge(grid1, how='horiz', where='-', shift=11)
.refine(10, axis=1, n_points=4)
.refine(13, axis=0, n_points=2)
.transform(lambda x: x*5 + 2)
.update_cell_mask()
.plotCells(ax=ax, cell_kws=dict(facecolor=palette[2], alpha=0.75))
)

GRIDS IN THE REAL WORLD: SULLIVAN CREEK, NE WASHINGTON

PROJECTING MODEL RESULTS ON THE GRID

RESOURCES FOR GETTING STARTED AS A MEAT-SPACE ENGINEER
▸ I was lucky enough to get started with numpy and matplotlib while:
▸ I was still very familiar with MATLAB
▸ Both projects were directly trying to take market share away
from MATLAB
▸ APIs and efforts have changed since then.
▸ More recently, a new employee started with MATLAB experience
from college. We gave them a copy of Python For Data Analysis by
Wes McKinney (author of pandas), and were pretty productive with
python in a couple of week.

LINK DUMP
▸ Me:
▸ https://twitter.com/pmhobson
▸ https://github.com/phobson
▸ My (documented) Projects
▸ http://geosyntec.github.io/pygridtools/
▸ http://phobson.github.io/paramnormal/
▸ http://phobson.github.io/mpl-probscale
▸ https://github.com/phobson/cloudside (docs forthcoming)

Python in Civil/Environmental Engineering

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Python in Civil/Environmental Engineering

Similar to Python in Civil/Environmental Engineering (20)

Recently uploaded

Recently uploaded (20)

Python in Civil/Environmental Engineering