apache ignite - in-memory data fabric

©2014GridGainSystems,Inc.

NIKITAIVANOVFounder,ApachePMC

ApacheIgnite- In-MemoryDataFabricBeyondtheDataGrid

www.ignite.apache.org #apacheignite


Agenda

• Projecthistory• In-MemoryDataFabric• AdvancedClustering• In-MemoryComputeGrid• In-MemoryDataGrid• In-MemoryServiceGrid• In-MemoryStreaming&CEP• Plug-n-PlayHadoopAccelerator


In-MemoryDataFabricStrategicApproachtoIMC

• Supports Applications of various types and languages

• Open Source – Apache 2.0• Simple Java APIs• 1 JAR Dependency• High Performance & Scale• Automatic Fault Tolerance• Management/Monitoring• Runs on Commodity Hardware

• Supports existing & new data sources• No need to rip & replace


Performance– HighThroughput– LowLatencies

Scalability– AddClusterMembers(cores)– AddMemory(RAM)

HighAvailability– DataBackups– DatacenterReplication

In-MemoryDataFabricMainCharacteristics

Transactions– FullyACIDCompliant– Optimistic&Pessimistic

Persistence– SQL,NoSQL,Hadoop

Security– Authentication– Authorization– Tracing&Auditing


• EaseofGettingStarted– AutomaticDiscovery

• AnyEnvironment– PublicCloud– PrivateCloud– HybridCloud– LocalLaptop

• Zero-Deployment– Auto-DeployCode

• FullClusterManagement• PluggableDesign

In-MemoryClustering&Deployment


• DirectAPIforMapReduce• ZeroDeployment• Cron-likeTaskScheduling• StateCheckpoints• LoadBalancing• AutomaticFailover• FullClusterManagement• PluggableSPIDesign

In-MemoryComputeGrid


• DistributedIn-MemoryKey-ValueStore

• ReplicatedandPartitioneddata• TBsofdata,ofanytype• On-HeapandOff-HeapStorage• HighlyAvailableIn-MemoryReplicas• AutomaticFailover• DistributedACIDTransactions• SQL99 queriesandJDBCdriver• CollocationofComputeandData

In-MemoryDataGrid


• UnlimitedVerticalScale• AvoidJavaGarbageCollectionPauses• SmallOn-HeapFootprint• LargeOff-HeapFootprint• Off-HeapIndexes• FullRAMUtilization• SimpleConfiguration

In-MemoryDataFabric:Off-HeapMemory


DistributedJavaStructures

• DistributedMap(cache)• DistributedSet• DistributedQueue• CountDownLatch• AtomicLong• AtomicSequence• AtomicReference• DistributedExecutorService


Client-ServervsAffinityColocation

Client-Server

AffinityColocation


• DistributeAnyDataStructure– AvailableAnywhereontheGrid– AutomaticRemoteAccessviaProxies

• ControlledDeployment– SupportforClusterSingleton– SupportforNodeSingleton– SupportforCustomTopology– LoadBalanced

• GuaranteedAvailability– AutoRedeploymentinCaseofFailures

In-MemoryServiceGrid


• StreamingDataNeverEnds

• BranchingPipelines• PluggableRouting• SlidingWindowsfor

CEP/ContinuousQuery

• RealTimeAnalysis

In-MemoryStreamingandCEP


• PlugandPlayinstallation• 10xto100xAcceleration• In-MemoryNativeMapReduce• In-ProcessDataColocation• IgniteFSIn-MemoryFileSystem• Read-ThroughfromHDFS• Write-ThroughtoHDFS• SyncandAsyncPersistence

In-MemoryHadoopAccelerator


In-MemoryHadoopAccelerator

• ZeroCodeChange• In-MemoryNativePerformance• UseexistingMRcode• UseexistingPig/Hivequeries• NoNameNode• EagerPushScheduling


SparkIntegration– IGFS&SharedRDD


Cross-LanguageInteroperability

• C++/.NET/PHP/Java/Node.js• PortableObjects• PerformanceAcrossLanguages• ClientFeatureParity• DynamicSchemaChanges• Searchable/Indexable• VersionIndependent


• Upto32DataCenters• Active-Active&Active-Passive• SmartConflictResolution• DurablePersistentQueues• AutomaticThrottling✴ EnterpriseEditionOnly

In-MemoryDataFabric:DataCenterReplication


Management&Monitoring

✴ EnterpriseEditionOnly


LocalRestartableStore

• PersistentOn-DiskStore• FastRecovery• LocalDataReload• EliminateNetworkandDbimpacts

whenreloadingin-memorystore



• PluggableAuth&Auth– JAAS,LDAP,JNDI,Kerberos

• In-ClusterNodeAuthentication• ClientAuthentication• SecureClientSessions• Fine-GrainedAuthorization• ComprehensiveAuditing– Who?What?When?


In-MemoryDataFabric:Security


THANKYOU!

www.ignite.apache.org #apacheignite

apache ignite - in-memory data fabric

Documents