datawarehousing 101

8/12/2019 Datawarehousing 101

1/70

Data Warehousing 101Everything

you never wantedto know aboutbig databases

but were forced

to find out anyway

Josh BerkusOpen Source Bridge 2011


2/70

contents

covering

concepts of DW

some DWtechniques

databases

not covering

hardware

analtics!reportingtools


3/70


4/70

BIGDATA


5/70

190


6/70

What is a!data warehouse"#


7/70

Big Data#


8/70


9/70

$%T& vs DW man single"row

writes

current data

queries generatedb user acti#it

$ 1s responsetimes

1000%s of users

few large batchimports

ears of data

queries generatedb large reports

queries can run forminutes!hours

10%s of users


10/70

$%T& vs DW

big data formany

concurrentrequests to

small amounts

of data each

big data

for low

concurrencyrequests to very

large amountsof data each


11/70

synony's(

subc)asses


12/70

archiving


13/70

archiving

WO&' data( )write once* read ne#er+

grows indefinitel

usuall a result of regulator compliance main concern( storage efficienc


14/70

data 'ining


15/70

data 'ining

the database where you don't know what's inthere, but you want to find out

lots of data ,-B to .B/

mostl )semi"structured+

data produced as a side effect of other

business processes needs ."intensi#e processing


16/70

BI* Business Inte))igenceD++* Decision +u,,ort

$%A&* $n)ine Ana)ytica)&rocessing

Ana)ytics


17/70


18/70

BI-D++-$%A&-Ana)ytics

databases which support visualization oflarge amounts of data

data is fairl well understood

most data can be reduced to categories*geograph* and taonom

primaril about indeing


19/70

What is a!di'ension"#


20/70

di'ensions vs. facts

3act-able

customers! accounts

categorsubcategor

sub"subcategor


21/70

di'ension e/a',)es location!region!countr!quadrant

product categori4ation

&5 transaction tpe

account heirarch

6. address OS!#ersion!build


22/70

di'ension synony's

facet

taonomsecondar inde

#iew


23/70

What is ET%#


24/70

E/tract Transfor' %oad how ou turn eternal raw data into useful

database data

7pache logs 8 web analtics DB

S9 .OS files 8 financial reporting DB

O5-. ser#er 8 10"ear data warehouse

also called :5- when the transformation is

done inside the database


25/70

&ur,ose of ET%-E%T

getting data into the data warehouse

clean up garbage data

split out attributes )normali4e+ dimensional data

deduplication

calculate materiali4ed #iews ! indees


26/70

ET% Too)s

.E.T.T.%.E.


27/70

ET% Too)s


28/70

Ad2hoc scri,ting


29/70

E%T Ti,s

think volume

bulk processing or parallel processing

no row"at"a"time* document"at"a"time

insert into permanent storage should bethe last step

no updates


30/70

3ueues not E/tract


31/70

What kind of

database shou)d Iuse for DW#


32/70

4 Ty,es

1; Standard &elational

2; ;


33/70


34/70

standard re)ationa)

the all-purpose solution for not-that-big data adequate for all tasks

but not ecellent at an of them

eas to use

low resource requirements

well"supported b all software familiar

not suitable for reall big data


35/70


36/70

What5s 6&


37/70

6assive)y&ara))e)

&rocessing

)i ft


38/70

a,,)iance software

6&&


39/70

6&&

cpu-intensive data warehousing data mining* some analtics

supporting comple quer logic

moderatel big data ,1"200-B/

drawbacks( proprietar* epensi#e

now hbridi4es with other tpes


40/70

What5s a

co)u'n store#

) t


41/70

co)u'n store

) t


42/70

co)u'n store

inversion of a row store:

indexes become datadata becomes indexes

) t


43/70

co)u'n stores

co)u'n stores


44/70

co)u'n stores

for aggregations and transformations ofhighly structured data

good for B6* analtics* some archi#ing

moderatel big data ,0;?"100-B/

bad for data mining

slow to add new data ! purge data usuall support compression


45/70

What5s

'a,-reduce#

'a,-reduce


46/70

'a,-reduce

'a,-reduce


47/70

'a,-reduce

'a,-reduce


48/70

'a,-reduce

// mapfunction(doc) { for (var i in doc.links) emit([doc.parent, i], null); }}// reducefunction(keys, values) { return null;}

'a,-reduce// Mapfunction (doc) {

i (d l d l)


49/70

'a,-reduce emit(doc.val, doc.val)}// Reducefunction (keys, values, rereduce) { // !is computes t!e standard deviation of t!e mapped results

var std"eviation#$.$; var count#$; var total#$.$; var s%rotal#$.$;

if (&rereduce) { // !is is t!e reduce p!ase, 'e are reducin over emitted values from // t!e map functions. for(var i in values) { total # total values[i];

s%rotal # s%rotal (values[i] * values[i]); } count # values.lent!; } else { // !is is t!e rereduce p!ase, 'e are re+reducin previosuly // reduced values. for(var i in values) { count # count values[i].count; total # total values[i].total; s%rotal # s%rotal values[i].s%rotal; } }

var variance # (s%rotal + ((total * total)/count)) / count; std"eviation # Mat!.s%rt(variance);

// t!e reduce result. t contains enou! information to -e rereduced // 'it! ot!er reduce results. return {std"eviationstd"eviation,countcount, totaltotal,s%rotals%rotal};

};

'a,-reduce vs 6&&


50/70

'a,-reduce vs. 6&&

open source petabtes

write routines bhand

inefficient

generic cheap W ! cloud

D6C tools

proprietar terabtes

ad#anced quersupport

efficient

specific needs good W

integrated tools


51/70

What5s enter,rise

search#

enter,rise search


52/70

enter,rise search

E)astic+earch

enter,rise search


53/70

enter,rise search

when you need to do D with a huge pile ofpartly processed !documents"

does( light data mining* light B6!analtics

best )full tet+ and keword search

supports )approimate results+

lots of special features for web data


54/70


55/70

What5s a

windowing 8uery#

regu)ar aggregate


56/70

regu)ar aggregate

windowing function


57/70

windowing function


58/70

0123 events (event4id 5,event4type 36,start M370M89,duration 53R:02,event4desc 36

);

7323 M06( t)


59/70

7323 M06(concurrent)M(tally)=:3R (=R"3R 1? start)07 concurrent


60/70

strea' ,rocessing +3%

replace multiple queries with a singlequer

a#oid scanning large tables multiple times

replace pages of application code and


61/70

What5s a

'ateria)ied view#

8uery resu)ts as tab)e


62/70

8uery resu)ts as tab)e

calculate once* read man time comple!epensi#e queries

frequentl referenced

not necessaril a whole quer often part of a quer

might be manuall or automaticall

updated depends on product

non2re)ationa) 'atviews


63/70

non re)ationa) 'atviews

ouchDB 9iews cache results of map!reduce obs

updated on data read

Solr ! :lastic Search )3aceted Search+ cached indeed results of comple searches

updated on data change

'aintaining 'atviews


64/70

'aintaining 'atviews

BE+T* update mat#iewsat batch load time

G$$D* update mat#iew according

to clock!calendar:AI;* update mat#iew on data request

BAD for DW* update mat#iewsusing a trigger

'atview ti,s


65/70

,

mat#iews should be small 1!10 to E of &7< on each node

each mat#iew should support se#eral

queries or one reall reall important one

truncate F append* don%t update

inde mat#iews like cra4

if the are not indees themsel#es


66/70

What5s $%A

cubes


67/70

Site&e

peat

9isito

rs

Browse

r

dri))2down


68/70

$%A&


69/70

On5ine 7naltical .rocessing 9isuali4ation technique

all data as a multi"dimensional space

great for decision support

. G &7< intensi#e

hard to do on reall big data

Works well with column stores

7ontact


70/70

Josh Berkus( oshHpgeperts;com blog( blogs;ittoolbo;com!database!soup

twitter( Hfu44chef

.ostgreSA5( www;postgresql;org pgeperts( www;pgeperts;com

-his talk is copright 2011 Josh Berkus and is licensed under the reati#e ommons 7ttributionlicense;

datawarehousing 101

Documents