plotcon nyc: domain specific visualization
TRANSCRIPT
NOVEMBER 15-18, 2016
#plotcon
1
Domain Specific Visualization
Nathan GoldbaumMoore Data Fellow_________ @njgoldbaum @yt_astrohttps://dxl.ncsa.illinois.edu/
Generic Plotting Libraries are Powerful
John Hunter Plotting Contest Entries: http://scipy2016.scipy.org/ehome/115969/276538/
Creative Power Implies API Complexity
4
I’m picking on matplotlib here, but the same is true of any generic plotting library
5
● Declarative Visualization Libraries
○ Vega-lite, ggplot, altair
● Documentation by example
○ Users create plots by cargo-culting examples
● Provide shortcuts in high-level API
○ “Easy mode” can’t cover all use cases
Managing API Complexity
6
Alternate approach: make assumptions for users
APLpyAstronomical Data
https://aplpy.github.io
CartopyGeographic Data
http://scitools.org.uk/cartopy/
yt: 3D Data Analysis
7
Ingest Data Analysis/Visualization
yt-project.org
Ramses
Gadget
Ingestion
Ingestion Representation
Ingestion Representation Analysis
Ingestion Representation Analysis Visualization
12
Ingestion Representation Analysis Visualization
Science
13
Ingestion Representation Analysis Visualization
Community
Science
Ingestion Representation Analysis Visualization
Community
Science
The ice cream sandwich of open research software
15
yt: A Community of Practice
>100 Contributors as of yt 3.3.2 (Oct 2016)
Tom Abel Bili Dong Suoqing Ji Sam Leitner John Regan Geoffrey SoGabriel Altay Nicholas Earl Allyson Julian Yuan Li Mark Richardson Casey W. StarkKenza Arraki Hilary Egan Christian Karch Alex Lindsay Sherwood Richers Antoine StrugarekKirk Barrow Daniel Fenn Max Katz Chris Malone Thomas Robitaille Elizabeth TaskerRicarda Beckmann John Forbes BW Keller Jonah Miller Anna Rosen Ben ThompsonElliott Biondo Enrico Garaldi Ji-hoon Kim Joshua Moloney Chuck Rozhon Robert ThompsonAlex Bogert Sam Geen Chang-Goo Kim Christopher Moody Douglas Rudd Joseph TomlinsonRobert Bradshaw Austin Gilbert Steffen Klemer Stuart Mumford Hsi-Yu Schive Stephanie TonnesenAndre Bubel Adam Ginsburg Fabian Koller Andrew Myers Anthony Scopatz Matt TurkCorentin Cadiou Nathan Goldbaum Kacper Kowalik Jill Naiman Noel Scudder Miguel de Val-BorroYi-Hao Chen William Gray Mark Krumholz Desika Narayanan Patrick Shriwise Rick WagnerPengfei Chen Markus Haider Michael Kuhlen Kaylea Nelson Devin Silvia Mike WarrenDavid Collins Eric Hallman Meagan Lang Brian O'Shea Sam Skillman Charlie WatsonBrian Crosby David Hannasch Erwin Lau J.S. Oishi Stephen Skory Andrew WetzelWeiguang Cui Cameron Hummels Eve Lee JC Passy Aaron Smith John WiseAndrew Cunningham Anni Järvenpää Doris Lee Rafael Ruggiero Britton Smith Michael Zingale
“Scaling a code in the human dimension”: https://arxiv.org/abs/1301.7064
16
yt: A Community of Practice
Tom Abel Bili Dong Suoqing Ji Sam Leitner John Regan Geoffrey SoGabriel Altay Nicholas Earl Allyson Julian Yuan Li Mark Richardson Casey W. StarkKenza Arraki Hilary Egan Christian Karch Alex Lindsay Sherwood Richers Antoine StrugarekKirk Barrow Daniel Fenn Max Katz Chris Malone Thomas Robitaille Elizabeth TaskerRicarda Beckmann John Forbes BW Keller Jonah Miller Anna Rosen Ben ThompsonElliott Biondo Enrico Garaldi Ji-hoon Kim Joshua Moloney Chuck Rozhon Robert ThompsonAlex Bogert Sam Geen Chang-Goo Kim Christopher Moody Douglas Rudd Joseph TomlinsonRobert Bradshaw Austin Gilbert Steffen Klemer Stuart Mumford Hsi-Yu Schive Stephanie TonnesenAndre Bubel Adam Ginsburg Fabian Koller Andrew Myers Anthony Scopatz Matt TurkCorentin Cadiou Nathan Goldbaum Kacper Kowalik Jill Naiman Noel Scudder Miguel de Val-BorroYi-Hao Chen William Gray Mark Krumholz Desika Narayanan Patrick Shriwise Rick WagnerPengfei Chen Markus Haider Michael Kuhlen Kaylea Nelson Devin Silvia Mike WarrenDavid Collins Eric Hallman Meagan Lang Brian O'Shea Sam Skillman Charlie WatsonBrian Crosby David Hannasch Erwin Lau J.S. Oishi Stephen Skory Andrew WetzelWeiguang Cui Cameron Hummels Eve Lee JC Passy Aaron Smith John WiseAndrew Cunningham Anni Järvenpää Doris Lee Rafael Ruggiero Britton Smith Michael Zingale
>100 Contributors as of yt 3.3.2 (Oct 2016)“Scaling a code in the human dimension”: https://arxiv.org/abs/1301.7064
What is yt?
17
18Data on disk has no physical meaning
19
With yt you can think about data using a physically motivated interface
20
Allowing you to forget about what the data look like as a file format
21And select only the data you want to select
22And only the data you want to select
23
It understands multiresolution data as a first-class citizen
24
25
26
PlotWindow plotting system
27
import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
http://yt-project.org/doc/visualizing/plots.html
28
29http://paste.yt-project.org/show/6913/
import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
Short but powerful scripts
● Register namespace● Set up unit metadata for further
use later
30
import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
Short but powerful scripts
● Automagically determine data format
● Parse parameters● Initialize IO and data selection
interface
31
import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
Short but powerful scripts
● Set up in-memory mesh ● Determine available fields to plot ● Chunked IO● Mask out overlapping data● Convert to data CGS units● Perform projection● Pixelize from multi-resolution
projection● Initialize plot object
32
import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
Short but powerful scripts
● Calculate positioning and metadata for matplotlib plot
● Create and customize matplotlib Figure and Axes objects
● Call imshow() with appropriate keyword arguments
● Save plot to disk
33
Generic plotting libraries can’t make assumptions
34
import matplotlib.pyplot as plt
plt.imshow(image_data)
plt.savefig('plot.png')
Customizing plots increases complexityimport matplotlib
import matplotlib.pyplot as plt
plt.imshow(image_data, norm=matplotlib.colors.LogNorm(),
origin='lower', cmap='viridis',
extent=[-7.5, 7.5, -7.5, 7.5])
cb = plt.colorbar()
cb.set_label(r'Projected Density $(\rm{g}/\rm{cm}^2)$')
plt.xlabel('x (kpc)')
plt.ylabel('y (kpc)')
plt.savefig('plot.png', bbox_inches='tight')
35
36
37
Ingestion Representation Analysis Visualization
Community
Science
Build new libraries for your use cases
Add new plotting tasks to libraries that already exist
Design Tips for Plotting Libraries
38
Provide sane defaults
39
import yt
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(ds, 'z', 'density')
prj.save('./frames/0000.png')
for i in range(150):
prj.zoom(1.03)
prj.save('./frames/%04d.png' % (i+1))
import yt
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(ds, 'z', 'density')
prj.set_zlim('density', 8e-6, 3e-1)
prj.save('./frames/0000.png')
for i in range(150):
prj.zoom(1.03)
prj.save('./frames/%04d.png' % (i+1))
Customize via properties/setters, not keyword arguments
40
Don’t overcomplicate for niche use cases
import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
fields = ['density', 'temperature',
'Velocity_divergence', 'grid_level']
slc = yt.SlicePlot(ds, 'z', fields)
slc.set_width(15*kpc)
slc.set_log('grid_level', False)
slc.set_unit('velocity_divergence', 'km/s/kpc')
slc.set_unit('density', 'Msun/pc**3')
slc.save()
41
Expose base plotting library primitives slc = yt.SlicePlot(ds, 'z', 'density')
plot = slc.plots['density']
# matplotlib.figure.Figure
plot.figure
# matplotlib.axes.Axes
plot.axes
# matlotlib.colorbar.Colorbar
plot.cb
# matplotlib.image.AxesImage
plot.image
42http://yt-project.org/docs/dev/cookbook/simple_plots.html#matplotlib-primitives
Only perform expensive actions as needed
43
def invalidate_plot(f): @functools.wraps(f) def newfunc(*args, **kwargs): rv = f(*args, **kwargs) args[0]._plot_valid = False return rv return newfunc
def validate_plot(f): @wraps(f) def newfunc(*args, **kwargs): if not args[0]._plot_valid: args[0]._setup_plots() rv = f(*args, **kwargs) return rv return newfunc
class SlicePlot:
def _setup_plots(self): # make plots using matplotlib primitives
@invalidate_plot def set_axes_unit(self, unit_name): self._axes_unit_names = (unit_name, unit_name)
@validate_plot def save(self): for field in self.plots: self.plots[field].save()
Thank you for your attention!
45
Questions? Concerns? Comments?