-
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
1/31
NoSQL Landscape and aSolution to Polyglot
Persistence
-
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
2/31
Impetus Technologies
Agenda
Big Data Problems
Transition from RDMS to NoSQL
NoSQL Landscape
Challenges in transition
Tools for NoSQL
Kunderaan open source polyglot solution
Recorded version available at http://bit.ly/1hfz4Tn Impetus Technologies
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
3/31
Impetus Technologies
BIG Data Problem
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
4/31
Impetus Technologies
Why not RDBMS?
Scalability
Data format
High availability Data volume in
zeta byte,
yottabyte
Horizontal scaling
would be
expensive
Data format can be
static or dynamic
Relational / Non-
relational
Data locality
No single
point of failure
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
5/31
Impetus Technologies
Non-RDBMS way
Scale out
Scale up Static schema
Dynamic schema
Centralized
Decentralized
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
6/31
Impetus Technologies
Introduction to NoSQL
An approach to storing and retrieving data with horizontal
scaling, simple design and high availability
Data format
driven processing
Distributed with No
single point of
failure(SPOF)
Thinking out of
SQL box
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
7/31 Impetus Technologies
NoSQL : A Pragmatic Solution?
With NOSQL data can be consistent, highly
available and with no SPOF!
But not 100%!
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
8/31
Impetus Technologies
CAP Theorem
Consistency
AvailabilityPartitionTolerance
N/A
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
9/31
Impetus Technologies
Thinking NoSQL?
Size
Format
VelocityFiltering
Large
Data
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
10/31
Impetus Technologies
Size
High data growth! scalability is an issue?
Traditional RDBMS based solutions will not work!
xxxxxxxxx
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
11/31
Impetus Technologies
Velocity
Near real time/Big Data analytics
Parallel processing, ready-for-read design is required
Traditional RDBMS solutions are not
fast enough to meet the SLAs !
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
12/31
Impetus Technologies
Filtering
Filtering. Fraud detection
Risk management analysis
Traditional RDBMS may work on small
scale but not with large data !
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
13/31
Impetus Technologies
Format
Non relational data format.
Different nature of data set: graph based, key-value based access
Traditional database is limited to static tables!
l
o
g
s
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
14/31
Impetus Technologies
NoSQL Landscape
NOSQL
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
15/31
Impetus Technologies
Transition to NoSQL
DatastoreSelection
API
exploration
LandscapeUnderstandin
g
Implementation
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
16/31
-
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
17/31
Impetus Technologies
High Level APIs
Kundera
Kundera Kundera
Hector
Easy
Cassandra
Datastax java
driver
Astyanax
Morphia
Data Nucleus
Jongo
Spring data
Spring data
Neo4j
Hibernate
OGM
Data nucleus
Hbase api
Spring data
Kundera
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
18/31
Impetus Technologies
Hybrid Design
Online data
streaming
Cassandra, HBase
User
information
RDBMS
In memory
cache/session
Redis
Website
advertisement
MongoDB, Couchbase
User activity
graph
Neo4J, Titan
Batch processing
& analytics
Hadoop, Spark
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
19/31
Impetus Technologies
Bumpy Ride!
Unlearn and Learn
new APIs!
Index based retrieval
over multiple NOSQL
data stores
Atomic operations
NOSQL world is still
evolving, may need to
explore among data
stores
Migration of existing
production applications
and many more
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
20/31
Impetus Technologies
One Stop Solution
Master key, possible?Lets explore!
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
21/31
Impetus Technologies
Polyglot Way
Migrating
existing
solutions
Guarantee
atomicity
Switch
databases
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
22/31
Impetus Technologies
High Level Polyglot API
Spring data
Kundera
Spring data
Kundera
Spring data
Kundera
Spring data
Kundera
Lets implement in JPA way!
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
23/31
Impetus Technologies
Kundera to the Rescue!!
Supports 8 data stores
Cassandra, Hbase,
MongoDB, Redis,
Neo4j, Oracle NoSQL,
CouchDB and any
RDBMS
CRUD / Strong Query
SupportObject Relationships
Handling
Datastore-Optimized
Persistence and Query
Approach
Interceptors / Events /
Caching
Connection Pool /
Fallback (Lucene)
Indexing Flexibility
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
24/31
Impetus Technologies
Getting Started
Repositories
sonatype-nexus
Kundera Public Repository
https://oss.sonatype.org/content/repositories/releases
true
false
Maven Dependencies
com.impetus.kundera.client
kundera-cassandra
2.11
com.impetus. kundera.client
kundera-rdbms
2.11
Building From Sourcegit clone [email protected]:impetus-opensource/Kundera.git
mvn clean install
Recorded version available at http://bit.ly/1hfz4Tn
mailto:[email protected]:impetus-opensource/Kundera.githttp://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tnmailto:[email protected]:impetus-opensource/Kundera.gitmailto:[email protected]:impetus-opensource/Kundera.gitmailto:[email protected]:impetus-opensource/Kundera.gitmailto:[email protected]:impetus-opensource/Kundera.gitmailto:[email protected]:impetus-opensource/Kundera.git -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
25/31
Impetus Technologies
User Logs Sample App
@Entity
@Table(name = "user)
@IndexCollection(columns = { @Index(name = "emailId") })public class User {
@Id
@Column(name = "user_id")
private String userId;
@Column(name = "first_name")
private String firstName;
@Column(name = "last_name")
private String lastName;
@Column(name = "emailId")
private String emailId;
@OneToMany(cascade = CascadeType.ALL, fetch =
FetchType.LAZY)
@JoinColumn(name = "user_id")
private Set logs;
@Embedded
private PersonalDetail personalDetail;
public User() {
// Default constructor.
}
//Setters and Getters
@Entity
@Table(name = logs)
@Index(columns = { "body", created_at" }, index = true)public class UserLogs {
@Id
@Column(name = log_id")
private String logId;
@Column(name = "body")
private String body;
@Column(name = created_at")
@Temporal(TemporalType.DATE)
private Date createdDate;
public UserLogs() {
// Default constructor.
}
// Setters and Getters
User Entity UserLogs Entity
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
26/31
Impetus Technologies
User Logs Sample App
Configuration : Persistence.xml
-
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
27/31
Impetus Technologies
Switching Data stores
//create entity manager factory.
EntityManagerFactory emf = Persistence.createEntityManagerFactory(logCassandra,logMongo, properties);
EntityManager em = emf.createEntityManager();
..
em.persist(user);
Configuration : Persistence.xml
Persist Data
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
28/31
Impetus Technologies
Performance & Benchmarks
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
29/31
Impetus Technologies
Technical Challenges Addressed!
Distributed indexing over multiple NOSQL database e.g. Solr,
Elastic search
Plugin Kundera powered ES or Lucene indexer
Build your own library and simply plugin
Unlearn and Learn new APIs! Based on most popular JPA 2.0 specification
Atomicity guarantee and Transaction management
Built in support for JPA/JTA transaction and batch operations
NOSQL world is evolving, plan to switch databases?
Since its a JPA powered solution, reuse same code with almost no changes
Recorded version available at http://bit.ly/1hfz4Tn
http://bit.ly/1hfz4Tnhttp://bit.ly/1hfz4Tn -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
30/31
Q A
Big Data Solutions and Services partner for Enterprises
Impetus Technologies
mailto:[email protected]:[email protected]:[email protected]:[email protected] -
8/12/2019 NoSQL Landscape and a Solution to Polyglot Persistence- Impetus Webcast
31/31
Impetus Technologies
Thank You!
Meet us at Hadoop Summit, San Jose
CIO Big Data Summit, Texas
Strata Conference + Hadoop World, New York
Gartner Symposium, Orlando
Try / Recommend Kundera
https://github.com/impetus-opensource/Kundera
@impetustech
https://github.com/impetus-opensource/Kunderahttps://github.com/impetus-opensource/Kunderahttps://github.com/impetus-opensource/Kunderahttps://github.com/impetus-opensource/Kundera