quest for value in big earth data - university of iceland · ccl udo ccl: connected component...
TRANSCRIPT
Quest for Value in Big Earth DataIdentifying the most cost-effective way of dealing with Big Data in Geoscience
Kwo-Sen Kuo1,2,3 and colleagues
Including but not limited to:
Michael L Rilee4, Lina Yu5, Yu Pan5, Feiyu Zhu5, Hongfeng Yu5
1. NASA Goddard Space Flight Center, Greenbelt, Maryland, USA2. University of Maryland, College Park, Maryland, USA3. Bayesics LLC, Bowie, Maryland, USA4. Rilee Systems Technologies, Derwood, MD, USA5. University of Nebraska-Lincoln, Lincoln, NE, USA
The V’s of Big Data challenge
VERACITY
SCALE
VO
LUM
E
VA
RIETY
VELOCITY VALUE
Scaling Volume
❖ Parallel processing – There is no other way!
➢ Need to exploit multimodal parallelization
o Shared memory parallelization, SMP
▪ SMP has been incorporated into the fundamentals of modern scripting
languages and achieved considerable pervasiveness
• MatLab, IDL, Python, R, Julia, etc.
▪ Geoscientists can transparently leverage SMP without knowing, say, OpenMP
▪ (GPU, Quantum)
o Distributed memory parallelization, DMP
▪ Memory on a single node is not sufficient for Big Data – need a cluster of
many nodes
▪ Pervasive systematic utilization of DMP? Very challenging!
▪ We have found a way!
Observations and Insights
Observations on Data Movement Links
~+25% / year
≪ Moore’s Law
~+40% / year< Moore’s Law)
https://itblog.sandisk.com/cpu-bandwidth-the-worrisome-2020-trend/
Dedicated
~+50% / year
< Moore’s Law
Effective Internet user bandwidth
~+50% / year
< Moore’s Law
https://www.nngroup.com/articles/law-of-bandwidth/
Summary
Link BW (MB/s) Remark
DRAM ~O(5) (max) Dedicated
Network ~O(4) (max) Shared
SSD ~O(3) (max) Dedicated
HDD ~O(2) (max) Dedicated
Internet ~O(2) (eff)
❖ Network
➢ high-performance interconnect, e.g. InfiniBand or Fiber Optics
➢ Usually shared among ~10-1000 of nodes
❖ Progressively lower bandwidth further away from the CPUs
Away
from
CPU
Insights
❖ Data must be stored close to CPU
➢ Bandwidth drops ~O(1) per link away from CPU (GPU)
➢ Congestion at any link could decimate the throughput
➢ Shared nothing architecture! – This rules out Cloud, which is
based mostly on traditional HPC compute-bound architecture
❖ If data must be moved, the volume to be moved must be
commensurate with the bandwidth
➢ i.e. move larger volume using higher bandwidth
❖ Data locality is paramount to performance/efficiency
➢ High bandwidth access to all data required for analysis
A Conclusion
Tightly coupled compute-storage: The only class of
approaches to satisfy the requirements
❖ The analysis engine also manages the storage.
❖ Parallel distributed database management system
(DBMS) typifies the approach.
➢ Not the loosely coupled approaches like Spark or Hadoop
➢ Not the non-system tools like Dask
❖ Cloud becomes prohibitively expensive with IOPS
guaranty.
HPC versus SNA Architecture
HPC/Cloud Architecture
❖ Better suited for compute-bound tasks, e.g. model simulations
❖ Facilitating elasticity – spinning up a variable number of VMs
Shared Nothing Architecture
❖ Better suited for I/O-bound tasks, e.g. data analysis
❖ Improved compute-storage affinity (data locality) but reduced elasticity
Data PlacementHow data are partitioned and distributed onto cluster nodes
t
N. America is
here!
Simplistic Data Placement/Layout
Better Load Balance – Smaller Chunks
Data Placement AlignmentData Placement also known as Data Layout
Data Variety Challenges
❖ Spatial aspect (2 or 3D)
➢ Nonuniform data models
o e.g. Grid, Swath, and Point
➢ Nonuniform data resolutions
➢ Decoupling of array indices from geolocations
❖ Temporal aspect (1D)
➢ Nonuniform data resolutions
➢ Decoupling of array index from calendrical time
❖ File-centric practice
➢ Data are packaged into files of nonuniform
extents
o i.e. different spatial coverages and temporal durations
Worker
SciDBEngine
LocalStore
Coordinator
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Coordinator
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
Coordinator
SciDBEngine
LocalStore
Worker
SciDBEngine
LocalStore
1 2 3 4
5 6
lm
1
2
3
4
5
6 7
8
9
10
11
12
13 14
15
16 1
2
3
4
5
67
8
9
10
11
12
13
14
15
16
1
2
3
4
5
6 7
8
9
10
11
12
13 14
15
16
6/6/2018 EarthCube 2018 AHM (Poster 154)
Minimize Data Movement:Keep Analysis Pleasingly Parallel!
We recognize:
❖ A great proportion of Earth Science analysis requires spatiotemporal
coincidence
➢ i.e. for the same location and same time
➢ e.g. comparisons (for verification and validation) almost always require it
❖ The objective: data placement alignment
❖ It is impossible with some analysis, e.g. FFT and CCL
To align the placements of data chunks spatiotemporally on the
cluster nodes, we must thus
❖ Uniformly index all geoscience data spatiotemporally!
➢ Because Array DBMSs use index for partitioning
Scaling Variety
❖ Scaling Variety is much harder than scaling Volume.
❖ Homogeneous dataset: parallelization, even DMP, can be developed once and applied many times (to the same homogeneous dataset): reusable and scalable.➢ NASA Earth Exchange, NEX
❖ Heterogeneous datasets: necessary for integrative analysis, but the above case-by-case approach no longer scales!➢ Being a system science, Earth science demands interdisciplinary, integrative
analysis of diverse datasets from diverse subdisciplines.
➢ If parallelization must be developed for each combination of heterogeneous datasets -> piecemeal scalability!
➢ While SMP can be transparently leveraged by typical Earth scientists using modern scripting languages, it is not the case with DMP.
➢ Typical Earth scientists do not possess the programming skill for DMP.
➢ Without a better system, Earth scientists are doomed to stay in the “disarray of variety”.
STARESpatioTemporal Adaptive-Resolution Encoding
Spatial Element of STARE
❖ Hierarchical Triangular Mesh, HTM
Right-justified HTM Encoding
❖ Quadtree hierarchy
➢ Indexes geolocation – a substitute for lat-lon
➢ Contains approximate data resolution
The bit format of the STARE spatial index including geo-position and resolution.
Efficient Set Operations on Regions
Spatial intersection by comparing integers is facilitated by the encoded resolution.
Temporal Element of STARE
❖ Hierarchical Calendrical Encoding, HCE
Summary on ScalingVolume and Variety
Scaling Volume and Variety
On a parallel distributed array DBMS like SciDB:
❖ STARE homogenizes Variety by guaranteeing spatiotemporal data placement alignment on cluster nodes➢ Data chunks of the same place and time from different datasets are colocated
❖ Shared Memory Parallelization, SMP➢ Utilized on each node when analysis is pleasingly parallel
o High bandwidth from local storage to DRAM fully exploited
o Unnecessary node-to-node communication minimized
❖ Distributed Memory Parallelization, DMP➢ Utilized automatically and systematically when necessary
o The data partitioning/distribution operation performed by by DBMSs is akin to (pre-)domain decomposition
o Data are “domain decomposed” consistently using STARE into chunks and stored on disk
o Data chunks are quickly loaded into memory when needed for analysis
o DBMSs coordinate communication and execution of DMP
o SciDB has its own communication protocol but can also use MPI, e.g. SCALAPCK
A PrototypeInteractive animation
Data and Process Flow
Heterogeneous Geoscience
Data
SciDBAnalytics
STARE
Spatio-temporally Colocated
Data
Browser-based GUI
Prototype Demo Animation
Future Additions
❖ Use STARE to drastically improve (parallel) ingest performance while reducing resource requirement.
❖ Connect to metadata repository(ies)➢ Seamlessly integrating into existing data discovery practice
❖ Enhance throughput➢ Further reducing data movement for visualization
➢ Anticipating user analysis needs with behavior prediction
❖ Improve Python API to SciDB➢ Integrating highly compatible Xarray
o E.g. Both support named dimensions
Envisioned Architecture
…
Compute Cluster Parallel File System
…
Traditional HPC Simulation Cluster
C G
C G
C G
C G
C G
Data Intensive Analysis (GATE) Cluster
…
…
C G
C G
C G
C G
C G
C G
Metadata Repository
Data Centers
Users5
1
2
3
4
67
(Profound) Implications
❖ Better software quality, traceability, and reusability
➢ Programming SciDB UDXs, with DMP especially, is not for typical scientists
➢ Professional software engineers are required
➢ Once a UDX is constructed it is immediately reusable for all users
❖ Easier interdisciplinary collaboration
➢ A multiuser system with sophisticated control of roles and permissions
❖ More ensured research reproducibility
➢ More localized provenance collection
❖ Higher cost effectiveness by leveraging existing HPC facilities!
➢ Collocated with simulation hardware -> fast ingest of simulation results for analysis
➢ Thecollective cost of using Cloud is higher than it appears (since scientists do not pay for Cloud out of their own pockets), especially when the above factors are included in consideration.
CCL UDO
❖ CCL: Connected Component Labeling
➢ Non-pleasingly parallel, DMP required
❖ UDO: User Defined Operator
❖ Used to track blizzards defined by visibility
reduction due to in-air snow mass
➢ Processed 36 years (1980-2015) of MERRA reanalysis
hourly data at 0.625°×0.5° resolution in ~30 minutes
using a cluster of 28 containerized nodes (~6-yr-old).
Blizzard Track Density
Blizzard Unique Visits
Snowfall Rate
2-m Wind Speed
Momentum Roughness Length
North America 2010 Winter Animation
Phenomenon Hierarchy
❖ Phenomenon hierarchies
➢ Super phenomena containing sub phenomena, like supersets and subsets
❖ Schiermeier, Q., “The real holes in climate science.” Nature News, 2010.
➢ Regional climate projections
➢ Representation of precipitation and cloud
➢ Role of aerosols
➢ Palaeoclimatological data
❖ Process-based diagnostics
➢ The heavy-handed use of univariate averaging has reached its limit of usefulness
➢ Highly contextual/conditional approaches are needed
➢ Phenomenon hierarchies are such high-context conditional features for more targeted
model diagnostics
Acknowledgement
This research has been sponsored in part by
❖ National Science Foundation through grants ICER-
1541043, ICER-1540542, IIS-1423487 and
❖ Advanced Information Systems Technology (AIST)
program of NASA Earth Science Technology Office
(ESTO)
with supplemental funding from
❖ Advancing Collaborative Connections for Earth System
Science (ACCESS) program of NASA Earth Science Data
System program.
Thank you!