AN EFFICIENT DATA MINING SOLUTION

Hadoop?

Cassandra?

Spark?

Stratio Deep

An efficient data mining solution

“Two and two are four?

Sometimes… Sometimes they are five.”

G. Orwell

#StratioBD

Goals

• Why do you need Cassandra?

• What is the problem?

• Why do you need Spark?

• How do they work together?

#StratioBD

Cassandra

#StratioBD

• Based on DynamoDB…

• Replication, Key/Value, P2P

• And based on Big Table…

• Column oriented

ROBUST FAST EFFICENT

NO BOTTLENECK REPLICATEDDECENTRALIZED

Another Database?

Why?

One User – Lot of data

Case A

#StratioBD

Many User – Few data

Case B

#StratioBD

Many user – Lot of data

Case C

#StratioBD

Crawler app

#StratioBD

Cassandra, I choose you

100MIndexedpages

3kreads

Query time

< 1s

But…

Marketingwalks in

New query

“I need to find all the reference to the domain ACME.

I need the answer by Friday.”

#StratioBD

Problem

Cassandra is not well suited to resolved this type of

queries

You need to design the schema with the query in mind

#StratioBD

ChallengeAccepted

What options do we have?

• Run Hive Query on top of C*

• Write an ETL script and load data into another DB

• Clone the cluster

#StratioBD

What options do we have?

Run Hive Query on top of C*

Write ETL scripts and load into another DB

Clone the cluster

#StratioBD

And now… what can we do?

“We can't solve problems by using the same kind

of thinking we used when we created them”

#StratioBD

Albert Einstein

• Alternative to MapReduce• A low latency cluster computing system• For very large datasets• Create by UC Berkeley AMP Lab in 2010.• May be 100 times faster than MapReduce for:

Interactive algorithms. Interactive data mining

Spark

#StratioBD

Logistic regression inSpark vs Hadoop

SOURCE | http://spark.incubator.apache.org/

#StratioBD

WHO USES SPARK?

Spark and Cassandra

Integration points

#StratioBD

Cassandra’s HDFS abstraction layer

Advantantages:• Easily integrates with legacy systems.

Drawbacks:• Very high-level: no access to low level Cassandra’s features.

• Questionable performance.

INTEGRATION POINTS: HDFS OVER CASSANDRA

#StratioBD

Cassandra’s Hadoop Interface

• Thrift protocol

• CQL3 (our implementation)

Uses the novel Cassandra’s CqlPagingInputFormat

INTEGRATION POINTS: HDFS OVER CASSANDRA

#StratioBD

• Supports CQL3 features

• Respects data locality

• Good compromise between

performance / implementation complexity

CQL3 Integration

INTEGRATION POINTS: CASSANDRA’S HADOOP INTERFACE – CQL3

#StratioBD

CQL3 Integration (II)

Provides a Java friendly API:

• Developers map Column Families to custom serializable POJOs

• StratioDeep wraps the complexity of performing Spark calculations

directly over the user provided POJOs.

INTEGRATION POINTS: CASSANDRA’S HADOOP INTERFACE – CQL3

#StratioBD

Demo

Drawbacks:

• Still not preforming as well as we’d like

Uses Cassandra’s Hadoop Interface

• No analyst-friendly interface:

No SQL-like query features

CQL3 Integration (III)

INTEGRATION POINTS: CASSANDRA’S HADOOP INTERFACE – CQL3#StratioBD

Bring the integration to another level:

• Dump Cassandra’s Hadoop Interface

• Direct access to Cassandra’s SSTable(s) files.

• Extend Cassandra’s CQL3 to make use of Spark’s distributed

data processing power

Future extensions

What are we currently working on?

#StratioBD

#StratioBD

Conclusion

THANKS