Download - Snapshot All Too Gooood
-
8/14/2019 Snapshot All Too Gooood
1/37
SNAPDIFF Comprehension for coherent measurements1
DB2 APD : Enabling Data Analysis
Dominique Evans, DB2 Avance Pro!"em Determination, IB# evans$ca%i!m%com
mailto:[email protected]:[email protected] -
8/14/2019 Snapshot All Too Gooood
2/37
2
2
&afs&tor&u&evans&pu!"ic&!in&snapiff
Presentation goals
PA'( I Ientif) the pro!"em
Demonstrate simp"e SNAPDIFF
Introuce notion of section *+roup
Fi"ters
PA'( II Comp"e- SNAPDIFF
Importe fi"es
.oine fi"es
#eta ata an formu"ae
/oa an un"oa
/afs/tor/u/evansd/public/bin/snapdiff
-
8/14/2019 Snapshot All Too Gooood
3/37
0
0
&afs&tor&u&evans&pu!"ic&!in&snapiff
The problem
1% E-periments create "ar+e amounts of measurement ata
2% Data is hetero+eneous%
0% sefu" ata *a"most a"a)s invo"ves mu"tip"e measurementcaptures over time interva"s% *e%+% snapshots%
3% (he num!er of atasets rapi") e-cees human a!i"it) to +ainmuch insi+ht or 4no"e+e from the ra fi"es
5% Automate ata capture scripts can easi") +enerate o6ens orhunres of reports
-
8/14/2019 Snapshot All Too Gooood
4/37
3
3
&afs&tor&u&evans&pu!"ic&!in&snapiff
Snapshots
Snapshots
!2p
!2p
7ther
Snapshots
Snapshots
!2p
!2p
7ther
Snapshots
Snapshots
!2p
!2p
7ther
(ime 8 (ime 91 (ime 92
:
-
8/14/2019 Snapshot All Too Gooood
5/37
5
5
&afs&tor&u&evans&pu!"ic&!in&snapiff
A Key to analysis
;hat is chan+in+ over time Q , to create name > va"ue pairs %
-
8/14/2019 Snapshot All Too Gooood
13/3710
10
&afs&tor&u&evans&pu!"ic&!in&snapiff
-
8/14/2019 Snapshot All Too Gooood
14/3713
13
&afs&tor&u&evans&pu!"ic&!in&snapiff
What are the delta &alues'
-
8/14/2019 Snapshot All Too Gooood
15/3715
15
&afs&tor&u&evans&pu!"ic&!in&snapiff
A richer e%ample. Analysis o" slo! (AD) secondary.
snapiff U%poo"K8LK8LVoin poo"si6esimport U%!K8LK8Limport U%eusnapshot8K8LK8Lmeta &(#P&evans&meta
un"oa poo"!eu
snapiff roh T/o+ +ap runnin+ avera+eT48"oa poo"!eu
on")pivotpf Tper seconprefetch%UsecDe"ta Num!er rite "o+ I7sPrimar) Stan!) E-tent /a+Interva" compute "o+ +apW
-
8/14/2019 Snapshot All Too Gooood
16/371=
1=
&afs&tor&u&evans&pu!"ic&!in&snapiff
-
8/14/2019 Snapshot All Too Gooood
17/371G
1G
&afs&tor&u&evans&pu!"ic&!in&snapiff
A di""erent &ie!, same data:
snapiff corr T/o+ +ap runnin+ avera+eT48 "oa poo"!euon")pivotpf TDe"ta%UC7DS1%U!2shre%Utota"De"ta%UC7DS1%U!2reo%Utota"De"ta%UC7DS1%U!2hars%Utota"T on")+"o!a"
-
8/14/2019 Snapshot All Too Gooood
18/371H
1H
&afs&tor&u&evans&pu!"ic&!in&snapiff
-
8/14/2019 Snapshot All Too Gooood
19/371
1
&afs&tor&u&evans&pu!"ic&!in&snapiff
Bac$ to snapdi"" basics.
Simp"est usa+e is 2 fi"es, e%+% 2 ata!ase snapshots
Defau"t isp"a) is !) atapoint ro,
#easurements in co"umnsi%e% time a"on+ - a-is *snapiff etects snapshot timestamps
-
8/14/2019 Snapshot All Too Gooood
20/3728
28
&afs&tor&u&evans&pu!"ic&!in&snapiff
*impli"y. +et rid o" sections. *ort . +et rid uninteresting data.
-
8/14/2019 Snapshot All Too Gooood
21/3721
21
&afs&tor&u&evans&pu!"ic&!in&snapiff
E%actly the same, but delta display -includes percent by de"ault
-
8/14/2019 Snapshot All Too Gooood
22/37
22
22
&afs&tor&u&evans&pu!"ic&!in&snapiff
What is /section0 ' -a simple /group by0
?arious snapshots have repeatin+ +roups
Snapiff automatica"") etects certain +roups *ata!ase, !ufferpoo",ta!"espace%
Datapoints ithin sections are qua"ifie !) the section name
Can turn off ith nosection
Can ec"are an) atapoint a section e"imiter usin+
section Qmeasurement nameW
*here Qmeasurement nameW is a re+e- that i""" !e use to ientif) aatapoint name so ma4e it unique
-
8/14/2019 Snapshot All Too Gooood
23/37
20
20
&afs&tor&u&evans&pu!"ic&!in&snapiff
Ta$e a bu""erpool snapshot "or instance:
1se a restricti&e "ilter -"or clarity.
section /Bu""erpool name0 is automatically detected.
-
8/14/2019 Snapshot All Too Gooood
24/37
23
23
&afs&tor&u&evans&pu!"ic&!in&snapiff
*o !hat is p"' -abbre&iation o" p"ilter
/Post -data ingestion "ilter0.
A fi"ter app"ie after a"" atasets are in+este *an have ha metaru"es app"ie if present%
A"so f *fi"ter , operates on incomin+ ataset *roppin+ atapointsthat o not qua"if)% Ear"iest fi"ter%
f an pf usua"") ientica" resu"ts un"ess usin+ fancier options *meta, import %
f is faster *since rops the ata much sooner , !ut "oss) for metaprocessin+%
sfi"ter Xre+e-pM , fi"ters ho"e sections%
e%+%J
-
8/14/2019 Snapshot All Too Gooood
25/37
25
25
&afs&tor&u&evans&pu!"ic&!in&snapiff
3iltering sections
B(;J formatte ata can !e turne offJ noformat
-
8/14/2019 Snapshot All Too Gooood
26/37
2=
2=
&afs&tor&u&evans&pu!"ic&!in&snapiff
import . (o! to e%tend the data.
As QneW ata to primar) atasets
e%+% import ata!asesnapshot to !ufferpoo"snapshot
Sections not reco+ni6e *mi+ht&pro!a!") i"" enhance this
A"" importe ata is Q+"o!a"W *unqua"ifie re+ar"ess of interna" repeatin++roups%
Dup"icates in import atapoints roppe
An) name > va"ue ata can !e importe
Data shou" !e coherent *i%e% re"ate
An) num!er of separate sources can !e importe at once
Can import one ataset *e%+% constants, or count* primar) atasets*coherent measuremets
-
8/14/2019 Snapshot All Too Gooood
27/37
2G
2G
&afs&tor&u&evans&pu!"ic&!in&snapiff
-
8/14/2019 Snapshot All Too Gooood
28/37
2H
2H
&afs&tor&u&evans&pu!"ic&!in&snapiff
4oin is an import !ith relationship through the section
A""o ata to !e ae to section specific ata, *rather than
+"o!a"") as ith import
oo e-amp"e is !ufferpoo" snapshot@ since the ra snapshot
oes not inc"ue pa+e si6e * %
A QBufferpoo" Pa+e si6eW atapoints usin+ Voin%
A""os correct computation of !)tes from ra snapshot ata%
-
8/14/2019 Snapshot All Too Gooood
29/37
2
2
&afs&tor&u&evans&pu!"ic&!in&snapiff
4oin : +oing bac$ to the bu""erpool snapshots e%ample
CaveatJ Do not use nosame
-
8/14/2019 Snapshot All Too Gooood
30/37
08
08
&afs&tor&u&evans&pu!"ic&!in&snapiff
Primary data duplicate processing
Choice of severa" po"iciesJ
up K i+nore ha"t rep"ace a++re+ate se+re+ate Y *efau"t se+re+ate
1% I+nore *rops ups
2% Za"t *stops ith error on up
0% 'ep"ace *so "ast up ins
3% A++re+ate *sum
5% Se+re+ate *automatica"") +enerate moifie name *DPn
-
8/14/2019 Snapshot All Too Gooood
31/37
01
01
&afs&tor&u&evans&pu!"ic&!in&snapiff
-
8/14/2019 Snapshot All Too Gooood
32/37
02
02
&afs&tor&u&evans&pu!"ic&!in&snapiff
P567T &ie!s
reat for o!servin+ "ar+e num!ers of measurements
Best ith ver) se"ective fi"ters *so to fit on the screen
-
8/14/2019 Snapshot All Too Gooood
33/37
00
00
&afs&tor&u&evans&pu!"ic&!in&snapiff
8eta measurements , and "ormulae
A""o e-pressions to !e app"ie to measurement ata
7perans are atapoint names
Eva"uate on a per ataset !asis
AF(E' f *fi"ter % BEF7'E pf *pfi"ter
Each ataset can reference any pre&iousataset
measurement*s
-
8/14/2019 Snapshot All Too Gooood
34/37
03
03
&afs&tor&u&evans&pu!"ic&!in&snapiff
-
8/14/2019 Snapshot All Too Gooood
35/37
05
05
&afs&tor&u&evans&pu!"ic&!in&snapiff
-
8/14/2019 Snapshot All Too Gooood
36/37
0=
0=
&afs&tor&u&evans&pu!"ic&!in&snapiff
-
8/14/2019 Snapshot All Too Gooood
37/37
0G
9A*T *95DE
*T7P (E)E
snapiff < for comp"ete *more or "ess he"p
!2service ;i4i comin+ soon%
/ocation of script *Per"J
&afs&tor&u&evans&pu!"ic&!in&snapiff