tools and tech for big data success

Tools and tech for Big Data success

Donnie Berkholz, Ph.D.IT Industry Analyst@dberkholz

[Imagine cheesy clipart of hammers, silicon, etc.]

Tools and tech — huh?

● Languages● Infrastructure● Tooling● It's not about the what, it's about the how

Thesis: Technology adoption is increasingly bottom-up

The new kingmakers

The best DX wins

It's all about barriers to entry

Ecosystems matter

The challenges

IT pros delivering data solutions60/36 IT/biz

Talend, summer 2012

The skills gap

NVP survey of execs, summer 2012

What can we learn from book sales?

What about the data analysts?

KDNuggets, summer 2012

What about the data analysts?

KDNuggets, summer 2012

It's not all about popularity...

The real growth isn't in SQL interfaces

Developers vs. marketers

Google Trends

IT pros delivering data solutions

60/36 IT/bizTalend, summer 2012

Hadoop distributions: popularity

Google Trends: $VENDOR hadoop

Developers choose what's easiest

How to run: bare metal / DC, private cloud, public cloud

Storing Data: HDFS, Ceph, Gluster

Open Building Blocks but not Open Source: EMR on AWS

Realtime: Impala, Druid

In-memory: Redis, Memcached

Streaming: Storm, S4

Other Options: Hadoop YARN, HPCC, Cassandra, Mongo, Riak

Other Options: Spark/Shark/Mesos

Abstraction: Mortar, Continuuity, Qubole, Concurrent Lingual, etc.

Conclusion: KISS

Donnie Berkholz, Ph.D.Analyst, RedMonk@dberkholz

Disclosure: 10gen, Amazon, Basho, Cloudera, Continuuity, IBM, MapR, Microsoft, and VMware are clients.