cloud scaling of visual weather - ecmwf · aws sqs asw sns aws elastic search metadata aws dynamodb...
TRANSCRIPT
1
Cloud scaling of Visual Weather
EGOWSReading, United Kingdom, 15th-17th October 2018
Jozef MatulaCTO
2
Cloud - Separation of responsibilities
https://blogs.technet.microsoft.com/yungchou/2010/11/15/cloud-computing-primer-for-it-pros/
3
4
5
6
~10TB of “gridded”NWP data per day
Compressed to 6TBin NetCDF
NWP in Cloud idea
ServicesStorage
7
Cloud - Scalable technologies
Amazon AWS Azure Google Cloud
Object StorageS3 (Simple Storage Service) Azure Blob Storage Cloud Storage
File StorageElastic Block StorageElastic File System
Premium Storage Account
Cloud Storage + FUSE adapter
Queue/MessagingSimple Queue ServiceSimple Notification Service
Storage queuesService Bus
Task QueueCloud Pub/Sub
Database
AuroraDynamoDBElasticsearch
SQL DatabaseDocumentDB
Cloud SQLCloud Datastore
Archive/BackupGlacier Azure Backup Cloud Storage
Coldline
Serverless CodeLambda Azure Functions App Engine
8
Cloud - Scalable technologies
Amazon AWS Azure Google Cloud
Object StorageS3 (Simple Storage Service) Azure Blob Storage Cloud Storage
File StorageElastic Block StorageElastic File System
Premium Storage Account
Cloud Storage + FUSE adapter
Queue/MessagingSimple Queue ServiceSimple Notification Service
Storage queuesService Bus
Task QueueCloud Pub/Sub
Database
AuroraDynamoDBElasticsearch
SQL DatabaseDocumentDB
Cloud SQLCloud Datastore
Archive/BackupGlacier Azure Backup Cloud Storage
Coldline
Serverless CodeLambda Azure Functions App Engine
9
Gridded Elastic Database
Cloud processing plan
Object Store Service
AWS S3
WMS
WMS
WMS
..
WCS
WCS
..
Client
Client
..
Load
bal
ance
r (A
PI
Gat
ew
ay)
AWS S3
Metadata Elastic Search
AWSSNS/SQS
10
AWS Auto Scaling Group
Gridded Elastic Database (G.E.D.)
ShredderShredder Client #1
Shredder Client #2
Shredder Client #X…..
AWS S3
AWS SQS
ASW SNS
AWS Elastic Search
Metadata
AWS DynamoDB
Unique numbers
11
G.E.D. client
AWSElastic Search
Metadata AWS S3
Grid Processor
Any kind of existingVW services:
WCS, WMS, WPSRIMA
Reprojection, Time and spatial interpolation
12
AWS Auto Scaling Group
G.E.D. client scaling
AWS Elastic Search
Metadata AWS S3
Grid Processor Grid Processor Grid Processor
…..
13
GED Auto Scaling Group
Peak time examples, 4 shredder clients are running
14
GED performance monitoring
15
16
Cons of Visual Weather in the Cloud:
• Very big code base (2GB)
17
Cons of Visual Weather in the Cloud:
• Very big code base (2GB)
• Inflexible and relatively slow deployment(a lot of configurations)
18
A new product targeting:
Cloud deployments, Cloud scalability &API standardisation
19
Open Weather does/is:
• Based on Visual Weather code base• Include GED (database + client)• Include web service suite (RIMA, WCS, WMS)
Open Weather does not:
• Include majority of VW server components (native database, ingestion and telecommunication subsystem)
• Include user interface of VW• Include traditional installation and configuration process
(replaced by pure git deployment of configuration)
Open Weather...
20
Open Weather Community Edition:
• local GRIB/NetCDF ingestion• GED client• Includes WCS and RIMA web service
(IBL OGC-like grid processing API)• Limited support
Open Weather Standard Edition:
• Includes GED database (the Shredder component)• Includes full OGC web service suite (WMS, WCS, WPS)• Python API• SLA support
Open Weather editions
21
Gridded Elastic Database
Cloud processing plan
Object Store Service
AWS S3
WMS
WMS
WMS
..
WCS
WCS
..
Client
Client
..
Load
bal
ance
r (A
PI
Gat
ew
ay)
AWS S3
Metadata
Elastic Search
AWSSNS/SQS
22
Gridded Elastic Database
Cloud processing plan - Serverless
Object Store Service
AWS S3
Client
Client
Client
..
Client
Client
..
Client
Client
..
Load
bal
ance
r (A
PI
Gat
ew
ay)
AWS S3
Metadata
Elastic Search
AWSSNS/SQS
λ
λ
λ
λ
λ
λ
λ
23
• Shrank down version of Open Weather allowing serverless grid processing (because of 250MB AWS Lambda limit).
• Includes GridProcessor capabilities operating over GED client connectivity
• No Web Service APIs, Lambda is a web service over our Python API.
Open Weather Lambda
