gridgain’in3memory’data’fabric:’ · gridgain’in3memory’data’fabric:’’...
TRANSCRIPT
![Page 1: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/1.jpg)
© 2014 GridGain Systems, Inc.
DMITRIY SETRAKYAN Founder & EVP Engineering @dsetrakyan
GridGain In-‐Memory Data Fabric: UlCmate Speed and Scale for TransacCons and AnalyCcs
www.gridgain.com #gridgain
![Page 2: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/2.jpg)
© 2014 GridGain Systems, Inc.
Agenda
• EvoluCon of In-‐Memory CompuCng • GridGain In-‐Memory Data Fabric • Distributed Cluster & Compute
– Coding Example
• Distributed Data Grid – Coding Examples
• Distributed Streaming & CEP • Plug-‐n-‐Play Hadoop Accelerator
![Page 3: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/3.jpg)
© 2014 GridGain Systems, Inc.
What is In-‐Memory CompuFng
• High Performance & Low Latencies • Faster than Disk and Flash • Cost EffecCve • Distributed or Not • Caching, Streaming, ComputaCons • Data Querying – SQL or Unstructured • VolaCle and Persistent • OLAP and OLTP Use Cases
![Page 4: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/4.jpg)
© 2014 GridGain Systems, Inc.
EvoluFon of In-‐Memory CompuFng
Caching
Distributed Caching
In-‐Memory Data Grids IMDBs
Database IM opCons Hadoop
accelerators
Streaming
BI accelerators
Clustering & Compute Grid
Data Grid Streaming
Hadoop Acceleration
![Page 5: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/5.jpg)
© 2014 GridGain Systems, Inc.
ExisFng Market is Fragmented
Company Product Proprietary/ Open Source CharacterizaFon
Oracle In-Memory Option for Oracle Database Proprietary Cost Option
Oracle Times Ten Proprietary Point Solution IMDB
Oracle Coherence Proprietary Point Solution IMDG
SAP Hana Proprietary Point Solution - IMDB
Microsoft SQL Server 2014 Proprietary Feature Upgrade
DataBricks Apache Spark Open Source Point Solution - Hadoop
VoltDB VoltDB Open Source Point Solution – IMDB
Aerospike Aerospike Open Source Point Solution – NoSQL DB
IBM DB2 with BLU Acceleration Proprietary Feature Upgrade
Software AG Terracotta Open Source Point Solution - IMDG
Hazelcast Hazelcast Open Source Point Solution - IMDG
![Page 6: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/6.jpg)
GridGain In-‐Memory Data Fabric: Strategic Approach to IMC
• Supports all Apps
• Open Source – Apache 2.0 • Simple Java APIs • 1 JAR Dependency • High Performance & Scale • Automatic Fault Tolerance • Management/Monitoring • Runs on Commodity Hardware
• Supports existing & new data sources
• No need to rip & replace
Clustering & Compute Grid
Data Grid Streaming
Hadoop Acceleration
© 2014 GridGain Systems, Inc.
![Page 7: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/7.jpg)
© 2014 GridGain Systems, Inc.
Clustering & Compute • Direct API for MapReduce • Direct API for Fork/Join • Zero Deployment • Cron-‐like Task Scheduling • State Checkpoints • Early and Late Load Balancing • AutomaCc Failover • Full Cluster Management • Pluggable SPI Design
![Page 8: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/8.jpg)
© 2014 GridGain Systems, Inc.
AutomaFc Cluster Discovery
![Page 9: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/9.jpg)
© 2014 GridGain Systems, Inc.
Closure ExecuFon
![Page 10: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/10.jpg)
© 2014 GridGain Systems, Inc.
Closure ExecuFon
![Page 11: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/11.jpg)
© 2014 GridGain Systems, Inc.
In-‐Memory Caching and Data Grid
• Distributed In-‐Memory Key-‐Value Store • Replicated and ParCConed • TBs of data, of any type • On-‐Heap and Off-‐Heap Storage • Backup Replicas / AutomaCc Failover • Distributed ACID TransacCons • SQL queries and JDBC driver • CollocaCon of Compute and Data
![Page 12: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/12.jpg)
© 2014 GridGain Systems, Inc.
Cache OperaFons
![Page 13: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/13.jpg)
© 2014 GridGain Systems, Inc.
Cache TransacFon
![Page 14: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/14.jpg)
© 2014 GridGain Systems, Inc.
Distributed Java Data Structures
• Distributed Map (cache) • Distributed Set • Distributed Queue • CountDownLatch • AtomicLong • AtomicSequence • AtomicReference • Distributed ExecutorService
![Page 15: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/15.jpg)
© 2014 GridGain Systems, Inc.
Client-‐Server vs Affinity ColocaFon
Client-‐Server Affinity ColocaCon
![Page 16: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/16.jpg)
© 2014 GridGain Systems, Inc.
In-‐Memory Streaming & CEP
• Streaming Data Never Ends • Branching Pipelines • CEP Sliding Windows • Pluggable RouCng • Real Time Analysis • At Least Once Guarantee
![Page 17: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/17.jpg)
© 2014 GridGain Systems, Inc.
Plug-‐n-‐Play Hadoop Accelerator
• Up to 100x AcceleraCon • In-‐Memory NaCve MapReduce
– In-‐Process Data ColocaCon – Eager Push Scheduling
• GGFS In-‐Memory File System – Pure In-‐Memory – Write-‐Through to HDFS – Read-‐Through from HDFS
• Sync and Async Persistence
![Page 18: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/18.jpg)
© 2014 GridGain Systems, Inc.
In-‐Memory NaFve MapReduce
• In-‐Memory NaCve MapReduce – Zero Code Change – Use exisCng MR code – Use exisCng Hive queries
• No Name Node • No Network Noise • In-‐Process Data ColocaCon • Eager Push Scheduling
![Page 19: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/19.jpg)
© 2014 GridGain Systems, Inc.
DevOps Management and Monitoring
![Page 20: GridGain’In3Memory’Data’Fabric:’ · GridGain’In3Memory’Data’Fabric:’’ Strategic’Approach’to’IMC’ • Supports all Apps • Open Source – Apache 2.0 •](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3e266b0bc5d441a5c9a/html5/thumbnails/20.jpg)
© 2014 GridGain Systems, Inc.
THANK YOU
www.gridgain.com #gridgain @dsetrakyan