mongodb europe 2016 - warehousing mongodb data using apache beam and bigquery

Warehousing MongoDB DataUsing Apache Beam and BigQuerySandeep ParikhHead of Solutions Architecture, Americas East@crcsmnky

Google Cloud Platform 2

About Me

Agenda

MongoDB on Google Cloud Platform

What is Data Warehousing

Tools & Technologies

Example Use Case

Show, Don’t Tell

Confidential & ProprietaryGoogle Cloud Platform 4



Manually Deploying MongoDB


Google Cloud Launcher


MongoDB Cloud Manager


MongoDB Cloud Manager

How do you automate this?


Bootstrapping MongoDB Cloud Manager

DeploymentManagerTemplate


Cloud Deployment Manager

Provision, configure your deployment

Configuration as code

Declarative approach to configuration

Template-driven

Supports YAML, Jinja, and Python

Use schemas to constrain parameters

References control order and dependencies


Bootstrapping Cloud Manager

Schema, Configuration & Template

Posted on Github https://github.com/GoogleCloudPlatform/mongodb-cloud-manager

Three Compute Engine instances, each with 500 GB PD-SSD

MongoDB Cloud Manager automation agent pre-installed and configured

$ gcloud deployment-manager deployments create mongodb-cloud-manager \

--config mongodb-cloud-manager.jinja \

--properties mmsGroupId=MMSGROUPID,mmsApiKey=MMSAPIKEY

https://github.com/GoogleCloudPlatform/mongodb-cloud-manager


What’s a Data Warehouse

Data Warehouses are central repositories of integrated data from one or more disparate

sourceshttps://en.wikipedia.org/wiki/Data_warehouse

https://en.wikipedia.org/wiki/Data_warehouse




Data Warehouse

Money

Data

Data

Data

Insights

Profit!


Tools and Technologies


Where: BigQuery

Complex, Petabyte-scale data warehousing made simple

Scales automatically; No setup or admin

Foundation for analytics and machine learning


RUN QUERY


How: Apache Beam (incubating)

MapReduce

BigTable DremelColossus

FlumeMegastoreSpanner

PubSub

MillwheelApache Beam

Google Cloud Dataflow


Understand What, Where, When, How

3Streaming

4Streaming

+ Accumulation

1Classic Batch

2Windowed

Batch


Pipelines in Beam

Pipeline p = Pipeline.create();

p.begin()

.apply(TextIO.Read.from(“gs://…”))

.apply(ParDo.of(new ExtractTags())

.apply(Count.create())

.apply(ParDo.of(new ExpandPrefixes())

.apply(Top.largestPerKey(3))

.apply(TextIO.Write.to(“gs://…”));

p.run();

Pipeline p = Pipeline.create();

p.begin()

.apply(TextIO.Read.from(“gs://…”))

.apply(ParDo.of(new ExtractTags())

.apply(Count.create())

.apply(ParDo.of(new ExpandPrefixes())

.apply(Top.largestPerKey(3))

.apply(TextIO.Write.to(“gs://…”));

p.run();

.apply(PubsubIO.Read.from(“input_topic”))

.apply(Window.<Integer>by(FixedWindows.of(5, MINUTES))

.apply(PubsubIO.Write.to(“output_topic”));

Batch to Streaming


Apache Beam Vision

Beam Model: Fn Runners

Apache Flink

Apache Spark

Beam Model: Pipeline Construction

OtherLanguagesBeam Java

Beam Python

Execution Execution

Cloud Dataflow

Execution


Running Apache Beam

Cloud Dataflow Local Runner

25

A great place for executing Beam pipelines which provides:

● Fully managed, no-ops execution environment

● Integration with Google Cloud Platform

● Java support in GA. Python in Alpha

Cloud Dataflow Service

Deploy Tear Down

Fully Managed: Worker Lifecycle Management

Fully Managed: Dynamic Worker Scaling

100 mins. 65 mins.

vs.

Fully Managed: Dynamic Work Rebalancing

Integrated: Monitoring UI

Integrated: Distributed Logging

Cloud Logs

Google App Engine

Google Analytics Premium

Cloud Pub/Sub

BigQuery Storage(tables)

Cloud Bigtable(NoSQL)

Cloud Storage(files)

Cloud Dataflow

BigQuery Analytics(SQL)

Capture Store Analyze

Batch

Cloud DataStore

Process

Stream

Cloud MonitoringCloud

Bigtable

Real time analytics and Alerts

Cloud Dataflow

Cloud Dataproc

Integrated: Part of Google Cloud Platform

Cloud Dataproc

31


Example Use Case


Sensor Data


Show, Don’t Tell

Insert Demo Here

mongodb europe 2016 - warehousing mongodb data using apache beam and bigquery

Data & Analytics