aws machine learning · run the fv3gfs global model on amazon web services, at full resolution and...

1© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved | 1© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |

Where and when?

Matthew Greensmith, Specialist SA

AWS Machine Learning

2© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |

Weather and climate on AWS - Joint Center for Satellite Data Assimilation (JCSDA)Joint Effort for Data Integration (JEDI) is a next-generation data assimilation (DA) system fornumerical weather prediction (NWP) that is capable and flexible enough to use for both researchand operations. Run the FV3GFS global model on Amazon Web Services, at full resolution andwith the pre-operational configuration.48-node (1,728-core) compute clusters on AWS.


Our mission at AWS

Put machine learning in the hands of every developer


Where ML doesn’t work

How to recognize AI Snake Oil - https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf

• Statistical Analysis

• What has happened

• Trends

• No outliers


Agenda

3 ways we are seeing ML in weather and climate

• Meta analysis and model tuning

• Effect prediction

• Device integration – efficiency of data collection


Meta Analysis

Scher S, Messori G. Predicting weather forecast uncertainty with machine learning. Q J R Meteorol Soc. 2018;144:2830–2841. https://doi.org/10.1002/qj.3410

“…networks are shown to be useful predictors offorecast uncertainty.”


Extending this to post processing

Rasp S, Lerch S. Neural networks for post-processing ensemble weather forecasts. Monthly Weather Review 2018, 146, 3885-3900.

“Our neural network models significantly outperform state-of-the-art post-processing techniques while being computationally more efficient.”

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

ML Analysis of prediction effect : Jupiter Intelligence

Machine Learning for improving Disaster Management and Response using AWS

Hurricane Irma predicted path Hurricane Irma real path

Source: Weather Channel

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T


Importance of Social Media Data in Disasters

• Micro-blogging data from crowds

of non-professional participants

during disasters are of significant

value.

Researchers assert that bystanders “on the ground are uniquely positioned to

share information that may not yet be available elsewhere in the information

space...” [Starbird et al., 2010].


Using locations identified by Comprehend to track hurricane path

Source: Wikipedia

Source: Weather Channel


IoT is HARD!


AmazonS3 Bucket

Amazon Redshift

Amazon SageMaker

MQTT

Endpoints Gateway/PLCCloud Enterprise

Applications

Device shadow

RulesEngine

AWS IoTCore

Certificate Authority

DeviceShadow

AWS IoT Greengrass

LambdaFunctions

MessageRouter

Long-range Comms

Amazon FreeRTOS

Local Resources

IoT SDK

AWS IoT Device Management

AWS IoTAnalytics

AWS

Enterprise Users

MQTT

UDP/IP

IoTUsers

EdgeUsers

Cert

WiFi

All AWS

Over-the-air (OTA) Updates

Analytics Data Store

Data Pipelines

Templated Reports

Local Resources

IoT Solution Map

Batch Fleet Provisioning

Real-time Fleet Index & Search

Corp Apps

AWS IoTDevice Defender

Ad-hoc & In-depth Analysis

Risk Mitigation

Detection Profiles

Alerts

Scheduled or Ad-hoc

Audit

MQTT

Things

OTA

Amazon FreeRTOS

Message Broker

IntegratedClient

OTA

BLE

Timeseries

SnowballEdge

AWS IoT SiteWise

3rd Party

On-premises Historian

Secret Manager

Event Detection

OPC Server

AWS IoTEvents

Local Comms

Custom Gateway

Amazon EMR

AmazonQuickSight

AmazonS3

AWS Lambda

Amazon Kinesis

AWS Things Graph


Data and State Sync Security

Over the Air UpdatesConnectors

Operate devices offline & synchronize data

when reconnected

LocalActions

Simplify device programming

with AWS Lambda

Mutual authentication &

authorization between cloud

and devices

Easily update AWS IoT Greengrass

Core

Machine Learning Inference

Perform ML Inference

locally

Local Resource

Access

AWS Lambda functions can

access & use local resources of a given device

Extend edge devices with

connections to external services

LocalMessages

and Triggers

Enable device communication without a cloud

connection

Secrets Manager

Deploy secrets to edge devices

AWS IoT Greengrass


Bringing machine learning to all developers

A M A Z O N S A G E M A K E R

Collect and prepare

training data

Choose and optimize your ML algorithm

Set up and manage environments

for training

Train and tune model

(trial and error)

Deploy model in

production

Scale and manage the production environment


Collect and prepare

training data

Choose and optimize your ML algorithm

Set up and manage environments

for training

Train and tune model

(trial and error)

Deploy model in

production

Scale and manage the production environment

Pre-built notebooks for

common problems

Built-in, high performance algorithms

One-click training Optimization

One-click deployment

Fully managed with auto-scaling, health checks,

automatic handling of node failures,

and security checks

Bringing machine learning to all developers

A M A Z O N S A G E M A K E R


FRAMEWORKS INTERFACES INFRASTRUCTURE

AI Services

Broadest and deepest set of capabilities

T H E A W S M L S T A C K

VISION SPEECH LANGUAGE CHATBOTS FORECASTING RECOMMENDATIONS

ML Services

ML Frameworks + Infrastructure

P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D& C O M P R E H E N D

M E D I C A L

L E X F O R E C A S TR E K O G N I T I O NI M A G E

R E K O G N I T I O NV I D E O

T E X T R A C T P E R S O N A L I Z E

Ground Truth Notebooks Algorithms + Marketplace Reinforcement Learning Training Optimization Deployment HostingAmazon SageMaker

F P G A SE C 2 P 3& P 3 D N

E C 2 G 4E C 2 C 5

I N F E R E N T I AG R E E N G R A S S E L A S T I CI N F E R E N C E

D L C O N T A I N E R S

& A M I s

E L A S T I C K U B E R N E T E S

S E R V I C E

E L A S T I C C O N T A I N E R

S E R V I C E


1

Createthe loop

Connect technology initiatives with business outcomes

2

Assess your structured and unstructured data sources

Advance yourdata strategy

?

3

Put machine learning in the hands of your developers

Organize for success

S E T T I N G Y O U R O R G A N I Z A T I O N U P F O R S U C C E S S


Collaborating on scientific data in the cloud

NOAA- NEXRAD on AWS S3, usage increased 2.3x

greater scientific impact

Do go to aws.amazon.com/earth/

https://aws.amazon.com/earth/


ML.awsThank you!

aws machine learning · run the fv3gfs global model on amazon web services, at full resolution and...

Documents