tectonic summit 2016: managing 6 pb of earth imagery with kubernetes

Mailiao Refinery, Taiwan May 31, 2016

Troy Toman @troytoman

troy@planet.com

Managing 6PB of Earth Imagery with Kubernetes

Dongtaijinai’er Salt Lake, China MAY 3, 2016

You think you have cloud problems? Let me show you cloud problems

Uluru, Australia, DEC 2, 2015

and some Kubernetes

SPACE!

AGENDA

• Planet’s Mission 1• Where we started• Why Kubernetes• Where we are going

To image the whole world every day, making change visible, accessible and actionable.

CAPE TOWN, SOUTH AFRICAMARCH 9, 2016

MONITORING PORT ACTIVITY

CAPE TOWN, SOUTH AFRICAMARCH 14, 2016

MONITORING PORT ACTIVITY

CAPE TOWN, SOUTH AFRICAMARCH 15, 2016

MONITORING PORT ACTIVITY

SAN GABRIEL MOUNTAINSJUNE 4, 2016

SAN GABRIEL MOUNTAINSJUNE 4, 2016

SAN GABRIEL WILDFIRES

VEGETATION INDEX MAP

VEGETATION

bare sparse densemoderate

SAN GABRIEL WILDFIRES

FIRE RETARDENT LINES ARE IN RED

JUNE 22, 2016SAN GABRIEL MOUNTAINS

JUNE 22, 2016SAN GABRIEL MOUNTAINS

FALSE COLOR COMOPOSITE

REVEALS BURN SCARS

Muir Woods & Mt. Tamalpais, California, USA DEC 23, 2015

OUR APPROACH

HISTORICAL APPROACH • Tasking• Low coverage

• Weeks to gain access

• Expensive

PLANET APPROACH • Monitoring• Global coverage• Daily online delivery

• Affordable

13 Builds in only 3 years

BUILD 1 APR 2012

BUILD 6 APR 2013

BUILD 13 JUN 2015

AGILE AEROSPACE

12+SATELLITES

3 - 5M RESOLUTION

4-BAND IMAGERY

10 GROUND STATIONS

4SITES

10,000IMAGES PER DAY

APIFOR INTERNAL DATA

PIPELINE

100SOF VIRTUAL MACHINES

TBsof STORAGE

The early days …

Early approach

• One service team• 2 basic services

• Storage• Jobs

• Everything packaged as debs• Deployed on VMs w/Ansible• Everything tested/deployed as one

Planet’s Evolving Infrastructure

JobsWorker Spot

Instances

Imagery Storage

Storage Elastic Search

Planet API v0

100+SATELLITES

3 - 5M RESOLUTION

4-BAND IMAGERY

26 GROUND STATIONS

9SITES

10+TBPER DAY

EVERY DAY

APIFOR INTERNAL DATA

PIPELINE AND PLATFORM ACCESS,

CHANGE DETECTION, BASE MAPS,

MOSAICS, and MORE

10,000sOF VIRTUAL MACHINES

PBsof STORAGE

The very near future …

PLATFORMAUTOMATED DATA

PIPELINE

DATA AND FUNCTIONAL EXPLOSION!

MOSAIC & IMAGE

PROCESSING

ANALYTICS CONTAINER

API – RESTFUL & JSON

PlanetScope~150 Mkm2/DAY

RapidEye~6 Mkm2/DAY

Processing Engine

Ingestion API WEB TILES

API

CATALOG

SCENES, MOSAIC, TILE APIANALYTICS ANALYSIS API

ARCHIVE GUIFIND, DOWNLOAD

DATA VIA WEB BROWSER

TOOLSSAMPLE CODE

3RD PARTY APPLICATIONS

More to come …

Landsat 8

Large Scale, Dynamic Environment

We’ve Seen Change

• Growing number of service teams• Dozens of services

• Different scales• Different maturity• Different markets

• Unintentional monolith and/or unmanageable fringe services

• Infrastructure as impediment

How our software world is changing• Containers

• Flexible per service packaging• Faster, simpler, consistent deploys

• Kubernetes for orchestration• Dynamic reallocation of resources• Isolate development teams from infra

changes and each other• Couple service changes

where necessary

Planet’s Evolving Infrastructure

JobsWorker Spot

Instances

Imagery Storage

Storage Elastic Search

Planet API v0

Planet API v1

Infra

Apps

Planet Kubernetes Tooling

• Klondike• Bootstraps k8s on AWS for Planet• https://github.com/planetlabs/klondike• Focused on infrastructure deployment

• Planet-Express• Environments, Services, Deployments, Secrets• Template driven manifests and artifacts

kubectl create -f build/<platform>/<service>/ kubectl replace -f build/<platform>/<service>/

• Focused on services development and deployment

Key Learnings

• Kubernetes increases our flexibility and responsiveness

• Move can be evolutionary• Tooling is leveraged across teams• More good stuff to come• More change on the way - better

prepared to exploit opportunity

London Array Wind Farm, United Kingdom, APR 17, 2016

See Change. Change the World.

(applies to software and infrastructure too!) troy@planet.com

@troytoman

London Array Wind Farm, United Kingdom, APR 17, 2016