astronomy in the cloud · email : meagherp@amazon.com phone : 00353 1 961 1719 september 16, 2015...

Heidelberg Institute forTheoretical Studies

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 1/54

Astronomy in the Cloudusing cloud infrastructure to cooperate, scale, and democratise e-science

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 2/54

Crowd-sourcing / Citizen Science

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 3/54

Crowd-sourcing / Citizen Science

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 4/54

Crowd-sourcing / Citizen Science

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 5/54

Crowd-sourcing / Citizen Science

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 6/54

→ Banfield, et. al. 2015

and

in the order of objects

Radio Galaxy Zoo

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 7/54

→ Banfield, et. al. 2015

and

in the order of objects

Radio Galaxy Zoo

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 8/54

Starting the Project

50 tar files220 GB

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 9/54

Preprocessing

matrix from fits

flux relative to the maximum

interesting region

fits

x

I

x

I

0

1

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 10/54

Speeding up Preprocessing

single core python = hours

with

on cores = hours

file is still the bottleneck!

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 11/54

New Images Extracted

50 tar files220 GB

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 12/54

Similarity Measure

calculate the pixel based

for all possible

and find the best matching angle via

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 13/54

Results

Polsterer et al. 2014

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 14/54

Results

Polsterer et al. 2014

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 15/54

Results and Analysis

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 16/54

Processing

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 17/54

Interesting Objects

select

based on distribution of distances

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 18/54

The Downsides of this Approach

a lot of local copies

no orchestration of work-flow

bad exchange of intermediate results

very exclusive concerninghardware requirements

fits

NVIDIA Tesla K40

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 19/54

A new Project

idefield ou lier inder (Ray Norris)

ASKAP-EMU project

from known radio sources

develop methodology to discover

science in large data sets

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 20/54

A new Concept

use commercial cloud computing providerwith data-center in Sydney

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 21/54

The Concept

centralized storage and compute

embedded repository and project management tool

script all preprocessing andprovide orchestration

VIS Script

orchestration layer

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 22/54

Why use a Company?

international project:● who is providing resources?

● who has the infrastructure?

● who provides the services?

why not run our own science cloud?● using open stack

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 23/54

Why use a Company?

international project:● who is providing resources?

● who has the infrastructure?

● who provides the services?

why not run our own science cloud?● using open stack

who does currently

provide this

for astronomy?

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 24/54

There a many Cloud Provider

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 25/54

Use Amazon Web Services

provides the larges variety of services and the best infrastructure

SKA grant from

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 26/54

What is really new?

scaling and immediately

no longer depending on

hardware requirements

no between small university institutes and

large research centers

data via credits

open to all

DOWNDOWNUPUP

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 27/54

Basics

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 28/54

Basics

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 29/54

Basics

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 30/54

Basics

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 31/54

Simple/Scalable Storage Service (S3)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 32/54

Simple/Scalable Storage Service (S3)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 33/54

Simple/Scalable Storage Service (S3)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 34/54

Simple/Scalable Storage Service (S3)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 35/54

Simple/Scalable Storage Service (S3)

aws sync RadioGalaxyZoo s3://radiogalaxyzoo

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 36/54

Simple/Scalable Storage Service (S3)

aws sync RadioGalaxyZoo s3://radiogalaxyzoo

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 37/54

Simple/Scalable Storage Service (S3)

aws sync RadioGalaxyZoo s3://radiogalaxyzoo

www.slideshare.net/AmazonWebServices/amazon-s3-masterclass

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 38/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 39/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 40/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 41/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 42/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 43/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 44/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 45/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 46/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 47/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 48/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 49/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 50/54

Elastic Cloud Computing (EC2)

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 51/54

Elastic Cloud Computing (EC2)

www.slideshare.net/AmazonWebServices/amazon-ec2-masterclass

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 52/54

Voucher

Thanks to

we hand out 50$ voucher

need assistance?Peter MeagherEMEA Account ManagerEMEA Public SectorEmail : meagherp@amazon.comPhone : 00353 1 961 1719

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 53/54

Conclusion

thanks to

we know how to access data

we still need to learn how to

and

data

September 16, 2015 AG2015 | eScience & Virtual Observatory | Astronomy in the Cloud | Kai Lars Polsterer 54/54@AstroInformatix