malaria atlas project: data engineering team - foss4g sotm … · 2020. 12. 7. · malaria atlas...
TRANSCRIPT
![Page 1: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/1.jpg)
Using open geospatial data and technology to support malaria risk modelling
Malaria Atlas Project: Data Engineering Team
![Page 2: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/2.jpg)
Who we are
Malaria Atlas Project
A geospatial disease modelling research group. Focussed primarily on malaria
Why malaria?- Responsible for an estimated 640,000 deaths in 2019- Disproportionally affects sub-Saharan Africa and kills children under 5
Mission- Improve understanding of global malaria risk and the impact of interventions- Disseminate free, accurate and up-to-date geographical information on malaria- Provide policy makers with information on trends and on which interventions are most effective- Reduce malaria morbidity and mortality by facilitating targeting of resources
![Page 3: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/3.jpg)
Who we are
Data Engineering Team
Support modelling work by building and using software tools across three core areas:
1. Data acquisition
Identification and ingestion of new datasets relevant to malaria
2. Data processing
Extraction of useful information into a consistent format
3. Data dissemination
Publication of input and output datasets on the web
![Page 4: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/4.jpg)
Data Sources
Administrative unit boundaries:- GADM, GAUL, country ministries of health
Routine surveillance: data associated with administrative units (polygons)- MoH reports/websites, WHO reports, privately shared MoH reports
Parasite Rate and Interventions: data associated with survey locations (points)- Surveys from academic lit. reviews, DHS Surveys, MICS surveys, SPA Surveys
Satellite imagery: environmental data- NASA MODIS data; other publicly available satellite data
![Page 5: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/5.jpg)
Routine Surveillance Data
GIS Datasets
Routine Surveillance Datasets
Scientist
UI
Annual Parasite Incidence Analytics
Annual Parasite Incidence DatasetPostGIS + PostgreSQL
Annual Parasite Incidence CalculationData Processing PipelineNotebooks
R Studio Jupyter Lab
Source MetadataPostgreSQL
Tabular DataPostgreSQL
Admin Unit BoundariesPostGIS + PostgreSQL
Document DataFile Storage
Document IngestionPlatform Services
Malaria Information Representation
Entity RecognitionNLP
Knowledge GraphGraph Database
Architecture: Surveillance Data
![Page 6: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/6.jpg)
Parasite Rate and Intervention Survey Data
Intervention Survey Datasets
Scientist
DHS Cloud Storage
NotebooksRStudio Jupyter Lab
Data Ingestion and Processing Pipeline
Architecture: Survey Data
ITN UsagePostGIS
Antimalarial UsagePostGIS
Fever TreatmentPostGIS
IRS UsagePostGIS
Parasite Rate Survey Datasets
DCFPR PointsPostGIS
MICS Cloud Storage
PubMed
![Page 7: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/7.jpg)
Parasite Rate Survey Data
![Page 8: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/8.jpg)
Covariate Data Processing
GIS Datasets
Space Time Cube Datasets
MODIS
Index MetadataPostGIS
Admin Unit BoundariesPostGIS + PostgreSQL
GeoTIFFFile Storage
Data Ingestion Processing Pipeline
Architecture: Covariates Data
NotebooksR Studio Jupyter Lab
IMERG
CHIRPS
VIIRS
SRTM
OSM
GoogleApache Beam Processing Pipeline
Scientist
![Page 9: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/9.jpg)
Covariate Products: EVI and LST Day
![Page 10: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/10.jpg)
Geostatistical Modelling
Annual Parasite IncidencePostGIS + PostgreSQL
Parasite Rate PointsPostGIS + PostgreSQL
Intervention DatasetsPostGIS + PostgreSQL
CovariatesGeoTiff Raster Files
Administrative Unit BoundariesPostGIS + PostgreSQL
Modelling Pipeline: Global Cubes
![Page 11: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/11.jpg)
Geostatistical Modelling
Annual Parasite IncidencePostGIS + PostgreSQL
Parasite Rate PointsPostGIS + PostgreSQL
Intervention DatasetsPostGIS + PostgreSQL
CovariatesGeoTiff Raster Files
Administrative Unit BoundariesPostGIS + PostgreSQL
Modelling Pipeline: High-Resolution Country-Level Time-Series
![Page 12: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/12.jpg)
Geostatistical Modelling
Annual Parasite IncidencePostGIS + PostgreSQL
Parasite Rate PointsPostGIS + PostgreSQL
Intervention DatasetsPostGIS + PostgreSQL
CovariatesGeoTiff Raster Files
Administrative Unit BoundariesPostGIS + PostgreSQL
Modelling Pipeline: Intervention Coverage Cubes
![Page 13: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/13.jpg)
Dissemination: Data
Simplified Architecture: MAP Web Portal
Admin Units BoundariesPostGIS + PostgreSQL
Pixel Level Estimates for Malaria related metrics GeoTIFF files in File Storage
Population-weighted metrics per administrative unitPostgreSQL
OGC Web ServicesGeoserver
Tiling ServerGeoWebCache
RESTful Web ServiceSpring Framework
RESTful Web ServiceNodeJS + Koa2
WMS WFS WCS WMTS Vector Tiles JSON-statWPS
MAP ExplorerAngular + OpenLayers
Trends ExplorerAngular
malariaAtlasR Package
Third-Party Applications
https://malariaatlas.org/api-docs
![Page 16: Malaria Atlas Project: Data Engineering Team - FOSS4G SotM … · 2020. 12. 7. · Malaria Atlas Project A geospatial disease modelling research group. Focussed primarily on malaria](https://reader035.vdocument.in/reader035/viewer/2022081620/610cdd21c1c5a728e32bd89f/html5/thumbnails/16.jpg)
Acknowledgements