huge data analytics: calpont infinidb columnar dbms empowers new research with the world’s first...

Calpont InfiniDB® Accelerating Data Insights

Huge Data Analytics: Calpont InfiniDB Columnar DBMS Empowers New Research with The World’s First Searchable Genotype Database Strata Conference 2012

Calpont Proprietary and Confidential

InfiniDB® Scalable. Fast. Simple. Copyright © 2012 Calpont. All Rights Reserved.

Today’s Agenda

• Introduction of today’s speakers •What is InfiniDB? •Announced today: InfiniDB 3 •Huge Data Analytics: InfiniDB Empowers New

Research with The World’s First Searchable Genotype Database

•Questions •More information and resources

2


Today’s Presenters

3

Fernanda Foertter HPC Administrator / Scientific Programmer Genus plc Jim Tommaney Chief Technology Officer Calpont Corporation

What is InfiniDB?


Calpont Corporation

• Company o Privately held and backed oOffices

Dallas (Headquarters) Silicon Valley

•Business o Scale-out MPP analytic database oMySQL Columnar + Map Reduction o Commercial Open Core model

• Products o InfiniDB Enterprise

Forthcoming 4th major release o InfiniDB Community

Modified Open Source license

5

Calpont Mission To provide a highly

scalable data platform that enables

analytic business decisions as timely as customers and markets dictate.


Innovative Companies Turning to InfiniDB

6

InfiniDB® Scalable. Fast. Simple. Copyright © 2012 Calpont. All Rights Reserved. 7

What is InfiniDB?

®

Scalable

Fast

Simple


InfiniDB

8

What is InfiniDB?

Full-Featured SQL

Familiar MySQL Look and Feel

Big Data Analytics Engine

Game Changing Performance


Focus on Analytics Workloads

InfiniDB is … Engineered for large queries Engineered for ad-hoc flexibility Analytics, not OLTP Unique combination of columnar + map-reduce


What is InfiniDB?

®

Scalable

Fast

Simple


InfiniDB – Two Tier Architecture

Purpose built for big data analytics. • User Module (UM)

Understands SQL. • Performance Module (PM)

Operates on data blocks.

or …

Single Server


InfiniDB Performance Foundations

®

The Power and Scale of Map-Reduce plus

Transformational I/O Efficiency


Power and Scalability of Map-Reduce

SQL Operations are mapped to Performance Module threads • Parallel/Distributed Data Access • Parallel/Distributed Joins (Inner, Outer) • Parallel/Distributed Sub-queries (From, Where, Select) • Parallel/Distributed Group By, Distinct, and Aggregation • Extensible with Parallel/Distributed User Defined Functions

Results are returned to User Module in Reduce Phase

Map ↓↓↓↓↓ Reduce ↑↑↑↑↑




InfiniDB is not: … a hadoop style map-reduce framework.




InfiniDB is: … custom built and highly optimized map-reduce framework for queries.



Techniques to Avoid Unnecessary I/O oVertical Partitioning: read only the columns required oHorizontal Partition: focus on the rows required oJust-in-time materialization



Techniques for Efficient I/O oColumnar compression reduces I/O from disk oGlobal data buffer cache can reduce disk I/O oReal-time decompression accelerates reads from disk oAvoidance of Random I/O


Simple - Automatic Everything

• Vertical Partitioning • Horizontal Partitioning • Compression • Compression Algorithm Selection • Distribution of data across disk resources • Distribution of work across CPU resources

Simple


InfiniDB

®

Scalable

Fast

Simple

InfiniDB 3 Announced Today


InfiniDB 3: It is Now Possible...

21

InfiniDB 3


Today’s Presenters

Fernanda Foertter HPC Administrator / Scientific Programmer Genus plc Jim Tommaney Chief Technology Officer Calpont Corporation


Where I Work


Breeding Values

Genetic Evaluation


Phenotype: Meat Quality


Selection for Lean Growth

1980 2005


Halothane Gene (1991)

•Gene is associated

oHigh carcass yield

o Stress triggers hyperthermia

o Poor meat quality

X (Nn/nn)

(NN)


DNA Marker Use

1990 2009

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

1991HAL

1994ESR

1998RN & MC4R

2003MIS

2004Large-scale SNP discovery1999

FUT1 & PRKAG3

1991 - 2002Single genes, QTLCandidate genes

Large-scale SNP discovery, genome scans,

sequencing


0

10000

20000

30000

40000

50000

60000

70000

2004 2005 2006 2007 2008 2009

Num

ber o

f SN

Ps

Sudden Data Growth

Porcine SNP Panel Density


Sudden Data Growth

Sample Collection

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

14,000,000

16,000,000

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

Year

Animals (cumulative)

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

Year

Tissue(cumulative)


EBV Lean Yield Meat Quality Robustness Feed efficiency Etc

economic weights

Index = a1 × EBV1 + a2 × EBV2 + . . .

Genetic Evaluation


Data Pipeline


Genomic Data Deluge


Project: Genotyping DB

The Need • Accumulating SNP chip data • Difficulty searching through • Next Gen Sequencing • Cheaper SNP chips • LOTS of animals • Other projects needed the

data

Other Considerations • Store large data…BIG data • Scalable • Alternative to Oracle • Minimally impact

infrastructure • Easy for scientists to use


What Do Vendors Provide for Genotype Data?

nothing


Think Outside the (Vendor’s) Box…


All Databases are Not Created Equal


All Vehicles are Not Created Equal


Genomic Data


SNP Data

Animal ID SNP1 SNP2 SNP3 … SNP65K

1 0 1 2 1 2

2 1 1 0 0 0

3

4

5 1 2 2 0 2

…

XXXX


Single Research Cohort

What about selection and cohort comparisons?


Column Bases Make More Sense


InfiniDB: Parallel Columnar DB

2

3

7

9


Complicated Searches are Faster!


Scales for a Fraction of the Cost

Compression Up 75%

Speed vs RDBMS 15X faster

Scalability 100’s TB, parallel queries/ingest

Cost vs Oracle 25%


Future Projects: Imputation

$150 $150

$15 $15


Caution: Data multiplies in a BIG way


Conclusions

•Helps to have a deep understanding of the scientific problems being solved

•Have a good understanding of the data access pattern • Tool should solve 80% of the highest use patterns •Use combination of software, hardware knowledge to

improve performance • Think “out of the vendor box”, especially where

research is cutting edge • Take the lead to show new tools users may not even be

aware they want/ need


Questions


More Information on InfiniDB

Visit us at: o www.Calpont.com o www.InfiniDB.org o Visit Booth #414 to register to win an iPad 3

http://www.calpont.com/

http://www.infinidb.org/

huge data analytics: calpont infinidb columnar dbms empowers new research with the world’s first...

Technology

calpont proprietary

performanceinfinidb

infinidbinfinidb scalable

phaseinfinidb scalable

workinfinidb scalable

simple automatic

mapreduceinfinidb scalable

calpont infinidbcolumnar