web development to big data: a journey application pure streaming system streaming computation...

36

Web Development To Big Data: A Journey @samelamin

Upload: vuongdat

Post on 05-May-2018

218 views

Category:

Documents

2 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

Web Development To Big Data: A Journey

@samelamin

Page 2: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Sam I Am

Page 3: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Why I Changed Careers?

Page 4: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

WTF Is Big Data

@samelamin

Page 5: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

The 3 Vs

@samelamin

Page 6: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

ETL??

Page 7: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Redshift Datawarehouse

Page 8: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

State Of Play

Page 9: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Immutable Infrastructure

Page 10: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Slow Updates

Page 11: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Run Every 15 Mins

Page 12: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

New Sources = Pain

Page 13: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Building A Data Pipelines

Page 14: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Make Data Queryable!

Page 15: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Architecture!

Page 16: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Handle Bad Data

Page 17: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Schema Updates!

Page 18: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Alerting!

Page 19: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Data Formats

Page 20: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Columnar Storage

Page 21: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Data Lake

Page 22: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Apache Spark

Page 23: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

DAG Ting!

Page 24: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Data Serving Layer

Page 25: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Lambda Architecture

Page 26: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

Data Streaming

@samelamin

Page 27: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Issues With Streaming

Page 28: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Structured Streaming AKA V2

Page 29: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Google BigQuery

Page 30: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Brag Alert!

https://github.com/samelamin/spark-bigquery

Page 31: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

ML Powering Recommendations

Page 32: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

DEMO!

Page 33: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Thank You!

Page 34: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Write to BQ DEMO!

Page 35: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Read From BQ DEMO!

Page 36: Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

@samelamin

Structured Streaming DEMO!

Continuous delivery mobile application development

Application-Centric Real Time Video Synthesis and Streaming

Development of Mobile Streaming Application using Multimedia

Streaming Integration to Azure SQL Data Warehouse for Azure_DataSheet_SQLDW.pdf · SQL Data Warehouse Continuous, Real-Time Data Movement with In-Flight Processing. Streaming Integration

Apache Spark Continuous Processing in Structured Streaming …cedawi.org/docs/BDDB2018-Ramin-Orujov-Structured-Streaming-in-Apache-Spark.pdfRamin Orujov 19.05.2018 Structured Streaming

Low Cost, Isolated, Streaming Continuous Output Up to 8 kS/s

Analytical Application Sets Continuous emission monitoringpisalesdatasheets.s3.amazonaws.com/siemens/gasmet-cems.pdf · Analytical Application Sets Continuous emission monitoring

Streaming Data, Continuous Queries, and Adaptive Dataflow Michael Franklin UC Berkeley NRC June 2002

APPLICATION OF CONTINUOUS

In-transit analytics on distributed Clouds: applications ... · Autonomic Streaming Pipeline • Streaming pipeline –No “blocking” semantics –Continuous data transmission

Stream Processing Essentials - Java2Days...• List the building blocks of a streaming application • Transform, match, and aggregate streaming data • Perform continuous processing

1 A Prefetching Protocol for Continuous Media Streaming in

Providing Continuous Application Availability using Oracle ... · Providing Continuous Application Availability using Oracle Application Continuity Troy Reece –Senior Director in

ScalaPipe: A Streaming Application Generatorroger/transfer/saahpc-2012.pdf · ScalaPipe: A Streaming Application Generator Joseph G. Wingbermuehle, Roger D. Chamberlain, Ron K. Cytron

Efficient and Continuous Skyline Monitoring in Two Tier Streaming Settings

HTTP Live Streaming Abstract the actions to be taken by ... · 1. Introduction to HTTP Live Streaming HTTP Live Streaming provides a reliable, cost-effective means of delivering continuous

JBoss Enterprise Application Platform Continuous Delivery ... › documentation › en-us › jboss_enterprise... · Application Platform Continuous Delivery. The Configuration Guide,

Mobility and Progressive Application Streaming

CONTINUOUS GEOMEMBRANES FOR ON SITE APPLICATION … · continuous geomembranes for on site application ... (bases) continuous geomembranes for on site application ... maximum tensile

(Application of continuous chromatographic purification ...petit.lib.yamaguchi-u.ac.jp/G0000006y2j2/file/...(Application of continuous chromatographic purification technology in the

Application Notes for Live and Stored Streaming Video

Application-Layer Protocols Peer-to-Peer Systems, …jasleen/Courses/COMP431/slides/2...Application-Layer Protocols Video streaming and content delivery networks Video streaming is

SDN-BASED APPLICATION FOR EFFICIENT VIDEO STREAMING · SDN-BASED APPLICATION FOR EFFICIENT VIDEO STREAMING RESEARCHERS: RASHA EL-JAROUDI ADITI GHAG . Researchers ... • Send a requested

Continuous Sampling from Distributed Streamsqinzhang/papers/cdsample-full.pdf · 2011. 10. 6. · Continuous distributed streaming. Many streaming applications [26] involve multiple,

Creating a Music Recommendation and Streaming Application for … · 2017-03-01 · Creating a Music Recommendation and Streaming Application for Android Elliot Jenkins1 and Yanyan

Continuous SQL with Apache Streaming

Streaming First Story Detection with application to Twitter

Streaming Video over the Interneteeweb.poly.edu/~yao/EL6123_s16/videoTransport_DapengWu.pdf · • An architecture for video streaming – Video compression – Application-layer

Streaming Services ECE7650. Streaming Services Multimedia and Quality of Service: What is it? multimedia applications: network audio and video (“continuous

Citrix Provisioning Services 7 - Product Documentation Citrix Application Firewall, Citrix Application Gateway, Citrix Provisioning Services, Citrix Streaming Profiler, Citrix Streaming

H12693 Continuous Application Availability with EMC VPLEX · CONTINUOUS APPLICATION AVAILABILITY WITH EMC VPLEX ... And a 2013 global ... H12693 Continuous Application Availability

Streaming Data, Continuous Queries, and Adaptive Dataflow

Continuous Availability - Oracle · 4 WHITE PAPER / Continuous Availability 2 Descriptions of Fast Application Notification, Application Continuity, and Transparent Application Continuity

Chapter 1: Introduction to Apache Spark...streaming count spark application. ( words this s spark S nc -1k 8888 31-16-2ø8:- streaming application. this count streaming words spark—submit

JBoss Enterprise Application Platform Continuous Delivery ... · Continuous Delivery 17 Security Architecture For Use with JBoss Enterprise Application Platform Continuous Delivery