accelerating devops via data virtualization | delphix
Post on 16-Apr-2017
670 Views
Preview:
TRANSCRIPT
© 2015 Delphix. All Rights Reserved
Session ID:
Prepared by:
Accelera'ngDevOpsUsingDataVirtualiza'onandMasking
HowitallWorks:SomeInternalsandUseCases
1227
@TimothyJGorman
TimGormanTechnicalConsultantDelphix
© 2015 Delphix. All Rights Reserved
Foralmost25years,I’vebeenanOracledeveloperandDBA
Asadeveloper,thejobistocreatenew
funcMonalityandfixbugs
AsaDBA,thejobistocreatedatabases,protectthem,andkeepapplicaMonsrunning
opMmally
ThispresentaMonisaboutallthesethings,butmostlyitisaboutchange
© 2015 Delphix. All Rights Reserved
DatavirtualizaMon(DataAsAServiceorDaaS)isfastbecomingthenewnorm
NobodycanjusMfyprovisioningnon-prodenvironmentsforDEV/QA/etcthewaywe
diduptonow
AndsupporMngDevOpsbeYeristhewaygoingforward
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IntegraMngdevelopment,tesMng,andoperaMonstoincreasedeliveryfrequency– Fasternewfeaturestomarket– LowerleadMmeforfixes– Lowerrateoffailure
• Ge\ngthererequires…– End-to-endintegraMonofso^waredevelopment,tesMng,andoperaMons
– CulturalreorientaMontowardcooperaMonandempathy– Toolsandprocesses
4
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IntegraMngdevelopment,tesMng,andoperaMonstoincreasedeliveryfrequency– Goal:Fasternewfeaturestomarket– LowerleadMmeforfixes– Lowerrateoffailure
• Ge\ngthererequires…– End-to-endintegraMonofso^waredevelopment,tesMng,andoperaMons
– CulturalreorientaMontowardcooperaMonandempathy– Toolsandprocesses
5
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IntegraMngdevelopment,tesMng,andoperaMonstoincreasedeliveryfrequency– Goal:Fasternewfeaturestomarket– Goal:LowerleadMmeforfixes– Lowerrateoffailure
• Ge\ngthererequires…– End-to-endintegraMonofso^waredevelopment,tesMng,andoperaMons
– CulturalreorientaMontowardcooperaMonandempathy– Toolsandprocesses
6
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IntegraMngdevelopment,tesMng,andoperaMonstoincreasedeliveryfrequency– Goal:Fasternewfeaturestomarket– Goal:LowerleadMmeforfixes– Goal:Lowerrateoffailure
• Ge\ngthererequires…– End-to-endintegraMonofso^waredevelopment,tesMng,andoperaMons
– CulturalreorientaMontowardcooperaMonandempathy– Toolsandprocesses
7
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IntegraMngdevelopment,tesMng,andoperaMonstoincreasedeliveryfrequency– Goal:Fasternewfeaturestomarket– Goal:LowerleadMmeforfixes– Goal:Lowerrateoffailure
• Ge\ngthererequires…– End-to-endintegraMonofso^waredevelopment,tesMng,andoperaMons
– CulturalreorientaMontowardcooperaMonandempathy– Toolsandprocesses
8
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IntegraMngdevelopment,tesMng,andoperaMonstoincreasedeliveryfrequency– Goal:Fasternewfeaturestomarket– Goal:LowerleadMmeforfixes– Goal:Lowerrateoffailure
• Ge\ngthererequires…– End-to-endintegraMonofso^waredevelopment,funcMonalandperformancetesMng,andoperaMons
– CulturalreorientaMontowardcooperaMonandempathy– Toolsandprocesses
9
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IntegraMngdevelopment,tesMng,andoperaMonstoincreasedeliveryfrequency– Goal:Fasternewfeaturestomarket– Goal:LowerleadMmeforfixes– Goal:Lowerrateoffailure
• Ge\ngthererequires…– End-to-endintegraMonofso^waredevelopment,funcMonalandperformancetesMng,andoperaMons
– CulturalreorientaMontowardcooperaMonandempathy– Toolsandprocesses
10
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IntegraMngdevelopment,tesMng,andoperaMonstoincreasedeliveryfrequency– Goal:Fasternewfeaturestomarket– Goal:LowerleadMmeforfixes– Goal:Lowerrateoffailure
• Ge\ngthererequires…– End-to-endintegraMonofso^waredevelopment,funcMonalandperformancetesMng,andoperaMons
– CulturalreorientaMontowardcooperaMonandempathy– Toolsandprocesses
11
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IntegraMngdevelopment,tesMng,andoperaMonstoincreasedeliveryfrequency– Goal:Fasternewfeaturestomarket– Goal:LowerleadMmeforfixes– Goal:Lowerrateoffailure
• Ge\ngthererequires…– End-to-endintegraMonofso^waredevelopment,funcMonalandperformancetesMng,andoperaMons
– CulturalreorientaMontowardcooperaMonandempathy– Toolsandprocesses
12
© 2015 Delphix. All Rights Reserved
IntroducingDevOpsUsingDevOpstoachieveConMnuousDeliverytakes
morethansheerwill
Ittakesnewtoolsandnewprocesses
Andbreakingdownsilosbuiltupovertheyears
ItisnotthewaythatDevelopment,QA/TesMng,andOperaMonshavedonethingsbefore
13
© 2015 Delphix. All Rights Reserved
IntroducingDevOpsUsingDevOpstoachieveConMnuousDeliverytakes
morethansheerwill
Ittakesnewtoolsandnewprocesses
Andbreakingdownsilosbuiltupovertheyears
ItisnotthewaythatDevelopment,QA/TesMng,andOperaMonshavedonethingsbefore
14
© 2015 Delphix. All Rights Reserved
IntroducingDevOpsUsingDevOpstoachieveConMnuousDeliverytakes
morethansheerwill
Ittakesnewtoolsandnewprocesses
Andbreakingdownsilosbuiltupovertheyears
ItisnotthewaythatDevelopment,QA/TesMng,andOperaMonshavedonethingsbefore
15
© 2015 Delphix. All Rights Reserved
IntroducingDevOpsUsingDevOpstoachieveConMnuousDeliverytakes
morethansheerwill
Ittakesnewtoolsandnewprocesses
Andbreakingdownsilosbuiltupovertheyears
ItisnotthewaythatDevelopment,QA/TesMng,andOperaMonshavedonethingspreviously
16
© 2015 Delphix. All Rights Reserved
Ops:“Butwejustfinishedbuildingthesesilos!”
17
© 2015 Delphix. All Rights Reserved
Ops:“Butwejustfinishedbuildingthesesilos!”
18
Regulatorycompliance–separaMonofduMes
© 2015 Delphix. All Rights Reserved
Ops:“Butwejustfinishedbuildingthesesilos!”
19
Regulatorycompliance–separaMonofduMes
Systemcomplexity–administerbycomponent
© 2015 Delphix. All Rights Reserved
Ops:“Butwejustfinishedbuildingthesesilos!”
20
Regulatorycompliance–separaMonofduMes
Datasecurity–leastprivileges
Systemcomplexity–administerbycomponent
© 2015 Delphix. All Rights Reserved
Ops:“Butwejustfinishedbuildingthesesilos!”
21
Regulatorycompliance–separaMonofduMes
Datasecurity–leastprivileges
Systemcomplexity–administerbycomponent
Frompeopletoprocesses–newtoolsarethesoluMon
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IworktodeployatoolinorganizaMonsjustbeginningto
embraceDevOps– SomeMmestheseorganizaMonsdon’tknowtheyare
• Buttheyare…yestheyare…
• Ifthechallengestodeploymentweremerelytechnical– Itwouldtaketwohours,tops
• ButthebiggestchallengesareculturalandpoliMcal– Somanyofmydeploymentstakemonths
• Thegoodnewsis:onlythefirstdeploymentisthatpainful
22
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IworktodeployatoolinorganizaMonsjustbeginningto
embraceDevOps– SomeMmestheseorganizaMonsdon’tknowtheyareembracing
anything• Buttheyare…yestheyare…
• Ifthechallengestodeploymentweremerelytechnical– Itwouldtaketwohours,tops
• ButthebiggestchallengesareculturalandpoliMcal– Somanyofmydeploymentstakemonths
• Thegoodnewsis:onlythefirstdeploymentisthatpainful
23
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IworktodeployatoolinorganizaMonsjustbeginningto
embraceDevOps– SomeMmestheseorganizaMonsdon’tknowtheyareembracing
anything• Buttheyare…yestheyare…
• Ifthechallengestodeploymentweremerelytechnical– Itwouldtaketwohours,tops
• ButthebiggestchallengesareculturalandpoliMcal– Somanyofmydeploymentstakemonths
• Thegoodnewsis:onlythefirstdeploymentisthatpainful
24
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IworktodeployatoolinorganizaMonsjustbeginningto
embraceDevOps– SomeMmestheseorganizaMonsdon’tknowtheyareembracing
anything• Buttheyare…yestheyare…
• Ifthechallengestodeploymentweremerelytechnical– Itwouldtaketwohours,tops
• ButthebiggestchallengesareculturalandpoliMcal– Somanyofmydeploymentstakemonths
• Thegoodnewsis:onlythefirstdeploymentisthatpainful
25
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IworktodeployatoolinorganizaMonsjustbeginningto
embraceDevOps– SomeMmestheseorganizaMonsdon’tknowtheyareembracing
anything• Buttheyare…yestheyare…
• Ifthechallengestodeploymentweremerelytechnical– Itwouldtaketwohours,tops
• ButthebiggestchallengesareculturalandpoliMcal– Somanyofmydeploymentstakemonths
• Thegoodnewsis:onlythefirstdeploymentisthatpainful
26
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IworktodeployatoolinorganizaMonsjustbeginningto
embraceDevOps– SomeMmestheseorganizaMonsdon’tknowtheyareembracing
anything• Buttheyare…yestheyare…
• Ifthechallengestodeploymentweremerelytechnical– Itwouldtaketwohours,tops
• ButthebiggestchallengesareculturalandpoliMcal– Somanyofmydeploymentstakemonths
• Thegoodnewsis:onlythefirstdeploymentisthatpainful
27
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IworktodeployatoolinorganizaMonsjustbeginningto
embraceDevOps– SomeMmestheseorganizaMonsdon’tknowtheyareembracing
anything• Buttheyare…yestheyare…
• Ifthechallengestodeploymentweremerelytechnical– Itwouldtaketwohours,tops
• ButthebiggestchallengesareculturalandpoliMcal– Somanyofmydeploymentstakemonths
• Thegoodnewsis:onlythefirstdeploymentisthatpainfu
28
© 2015 Delphix. All Rights Reserved
IntroducingDevOps• IworktodeployatoolinorganizaMonsjustbeginningto
embraceDevOps– SomeMmestheseorganizaMonsdon’tknowtheyareembracing
anything• Buttheyare…yestheyare…
• Ifthechallengestodeploymentweremerelytechnical– Itwouldtaketwohours,tops
• ButthebiggestchallengesareculturalandpoliMcal– Somanyofmydeploymentstakemonths
• Thegoodnewsis:onlythefirstdeploymentisthatpainful
29
© 2015 Delphix. All Rights Reserved
Whatisdatavirtualiza'on?• Servervirtualiza'onhashadanextraordinaryimpactonIT
– ConverMnghardwareintoso^ware– ControllingvirtualizedhardwarethroughorchestraMonscripts
• ButservervirtualizaMonisnotacompletesoluMon– VirtualizingCPU,memory(RAM),andnetwork– Butwhataboutstorage???
• Fromahigh-levelperspecMve,servervirtualizaMonhascausedthewhite-hotstoragemarkettosuper-nova– Soeasytocreatevirtualmachines,andstorageischeap
• Butisstoragecheapenough?
• Dataneedstobevirtualized,justlikeservers
30
© 2015 Delphix. All Rights Reserved
Whatisdatavirtualiza'on?• ServervirtualizaMonhashadanextraordinaryimpactonIT
– ConverMnghardwareintoso^ware– ControllingvirtualizedhardwarethroughorchestraMonscripts
• ButservervirtualizaMonisnotacompletesoluMon– VirtualizingCPU,memory(RAM),andnetwork– Butwhataboutstorage???
• Fromahigh-levelperspecMve,servervirtualizaMonhascausedthewhite-hotstoragemarkettosuper-nova– Soeasytocreatevirtualmachines,andstorageischeap
• Butisstoragecheapenough?
• Dataneedstobevirtualized,justlikeservers
31
© 2015 Delphix. All Rights Reserved
Whatisdatavirtualiza'on?• ServervirtualizaMonhashadanextraordinaryimpactonIT
– ConverMnghardwareintoso^ware– ControllingvirtualizedhardwarethroughorchestraMonscripts
• ButservervirtualizaMonisnotacompletesoluMon– VirtualizingCPU,memory(RAM),andnetwork– Butwhataboutstorage???
• Fromahigh-levelperspecMve,servervirtualizaMonhascausedthewhite-hotstorageindustrytosuper-nova– Soeasytocreatevirtualmachines,andstorageischeap
• Butisstoragecheapenough?
• Dataneedstobevirtualized,justlikeservers
32
© 2015 Delphix. All Rights Reserved
Whatisdatavirtualiza'on?• ServervirtualizaMonhashadanextraordinaryimpactonIT
– ConverMnghardwareintoso^ware– ControllingvirtualizedhardwarethroughorchestraMonscripts
• ButservervirtualizaMonisnotacompletesoluMon– VirtualizingCPU,memory(RAM),andnetwork– Butwhataboutstorage???
• Fromahigh-levelperspecMve,servervirtualizaMonhascausedthewhite-hotstorageindustrytosuper-nova– Soeasytocreatevirtualmachines,andstorageischeap
• Butisstoragecheapenough?
• Dataneedstobevirtualized,justlikeservers
33
© 2015 Delphix. All Rights Reserved
Whatisdatavirtualiza'on?• ServervirtualizaMonhashadanextraordinaryimpactonIT
– ConverMnghardwareintoso^ware– ControllingvirtualizedhardwarethroughorchestraMonscripts
• ButservervirtualizaMonisnotacompletesoluMon– VirtualizingCPU,memory(RAM),andnetwork– Butwhataboutstorage???
• Fromahigh-levelperspecMve,servervirtualizaMonhascausedthewhite-hotstorageindustrytosuper-nova– Soeasytocreatevirtualmachines,andstorageischeap
• Butisstoragecheapenough?
• Dataneedstobevirtualized,justlikeserversButfirst,let’sstartfromthebeginning…
34
© 2015 Delphix. All Rights Reserved
AreyoutoobusytoInnovate?
Iner'a35
© 2015 Delphix. All Rights Reserved
TheGoal:TheoryofConstraints
• Longbuildcyclesarewasteful
• QA/Tes'ngisthefirstvicMm
• Improvementnotmadeattheconstraintisanillusion
36
© 2015 Delphix. All Rights Reserved
TheGoal:TheoryofConstraints
• Longbuildcyclesarewasteful
• QA/Tes'ngisthefirstvicMm
• Improvementnotmadeattheconstraintisanillusion
37
© 2015 Delphix. All Rights Reserved
TheGoal:TheoryofConstraints
• Longbuildcyclesarewasteful
• QA/Tes'ngisthefirstvicMm
• Improvementnotmadeattheconstraintisanillusion
38
© 2015 Delphix. All Rights Reserved
Factoryfloor
39
© 2015 Delphix. All Rights Reserved
Factoryfloor
constraint
40
© 2015 Delphix. All Rights Reserved
Tunebeforeconstraint
constraint
Tuninghere
Stockpiling
41
© 2015 Delphix. All Rights Reserved
Tunea[erconstraint
constraint
Tuninghere
Starva'on
42
© 2015 Delphix. All Rights Reserved
Factoryfloor:op'mizingattheconstraint
constraint
Goal:findconstraintandopMmizeit43
© 2015 Delphix. All Rights Reserved
DoestheTheoryofConstraintsworkforIT?
• GoalsClarify• MetricsDefine• ConstraintsIdenMfy• PrioriMesSet• IteraMonsFast
• CI• Cloud• Agile• Kanban• Kata
“ITisthefactoryfloorofthiscentury”44
© 2015 Delphix. All Rights Reserved
ThePhoenixProject:“TheGoal”appliedtoIT
“ITisthefactoryfloorofthiscentury”
WhatistheconstraintinIT?
45
© 2015 Delphix. All Rights Reserved
ThePhoenixProject:top5constraintsinIT?
1. QAsetup2. Dev/Buildsetup3. CodeArchitecture4. Development5. Productmanagement
-GeneKim,authorof“ThePhoenixProject”
“OneofthemostpowerfulthingsthatorganizaMonscandoistoenabledevelopmentandtesMngtogetenvironmenttheyneedwhentheyneedit“
46
© 2015 Delphix. All Rights Reserved
Dataistheconstraint
60% Projects Over Schedule
85% delayed waiting for data
Data is the Constraint
CIOMagazineSurvey:
…onlyge\ngworse…
Gartner:DataDoomsday,by20171/3rdITincrisis47
© 2015 Delphix. All Rights Reserved
• DataConstraint• AYemptedsoluMons• SoluMon• UseCases
Inthispresenta'on:
© 2015 Delphix. All Rights Reserved 49
• PRODiswhatwecaremostabout…
Theproblem
PROD
© 2015 Delphix. All Rights Reserved 50
• PRODiswhatwecaremostabout…– ButbehindeveryPROD…
• ThereisQATEST…
Theproblem
PROD QATEST
© 2015 Delphix. All Rights Reserved 51
• PRODiswhatwecaremostabout…– ButbehindeveryPROD…
• ThereisQATEST,DEV…
Theproblem
PROD QATEST UAT
© 2015 Delphix. All Rights Reserved 52
• PRODiswhatwecaremostabout…– ButbehindeveryPROD…
• ThereisQATEST,DEV…• …andotherproject-specificenvironmentstosupport
Theproblem
PROD QATEST UAT
© 2015 Delphix. All Rights Reserved 53
• Also,asPRODgrows…
Theproblem
PROD PROD
© 2015 Delphix. All Rights Reserved 54
UATQATEST
• Also,asPRODgrows…– ThensodothesupporMngnon-PRODenvironments…
Theproblem
PROD
© 2015 Delphix. All Rights Reserved 55
Theproblem
CEO
COO
CFO CMO CSO CIO
• AndsoinevitablythesituaMonshi^sfromthis…
© 2015 Delphix. All Rights Reserved 56
Theproblem
CEO
COO
CFO CMO CSO CIO
StorageAdministrator
• Tothis…
*InspiredbyConnorMcDonald(hYp://connormcdonald.wordpress.com/)
© 2015 Delphix. All Rights Reserved
ITindustryisgenerallyunawareoftheproblem
57
© 2015 Delphix. All Rights Reserved
Companiesunawareoftheproblem
TesterorDeveloperBoss,StorageAdmin,DBA
58
© 2015 Delphix. All Rights Reserved
• Inbuild…
1. BoYlenecksbetweenprojects
2. ExpensiveSlowQA/Test3. WaiMngforenvironments4. MoreMmespentinsetup
thantes'ng5. ProducMonBugs6. ExpensiveSlowQA
WhatProblemsdoestheDataConstraintCause?
• InoperaMons…
1. StorageexhausMon2. NetworksaturaMon3. Admincycles4. Unabletofixproblems5. Alwaysbehind
© 2015 Delphix. All Rights Reserved
Produc'onWall
60
Produc'on
© 2015 Delphix. All Rights Reserved
LifeinITwithoutQA/Test…silos
61
© 2015 Delphix. All Rights Reserved
• Dataconstraint• AYemptedsoluMons• SoluMon• UseCases
Inthispresenta'on:
© 2015 Delphix. All Rights Reserved 63
• SometrytomiMgatetheproblembycopyingsubsetsofPROD…
• …butitisdifficultandMme-consumingandexpensivetocreatereferenMally-correctsubsetsofdatabases.
Leadstobuggycode…
A^emp'ngtosolvetheproblem…
PROD QATEST DEV UAT
© 2015 Delphix. All Rights Reserved 64
• SometrytomiMgatetheproblembycopyingsubsetsofPROD…
• …butitisdifficultandMme-consumingandexpensivetocreatereferenMally-correctsubsetsofdatabases.
Leadstobuggycode…
A^emp'ngtosolvetheproblem…
PROD QATEST DEV UAT
© 2015 Delphix. All Rights Reserved 65
• SometrytomiMgatetheproblembycopyingsubsetsofPROD…
• …butitisdifficultandMme-consumingandexpensivetocreatereferenMally-correctsubsetsofdatabases.
Leadstobuggycode…
A^emp'ngtosolvetheproblem…
PROD QATEST DEV UAT
© 2015 Delphix. All Rights Reserved 66
• CloningproducMontonon-productinevitablyperformedinfrequently…
…meaningthatschemasanddataareolder,dirMerduetoprevioustesMng,andtoosmalltoaccuratelyrepresentreality
A^emp'ngtosolvetheproblem…
PROD QATEST DEV UAT
© 2015 Delphix. All Rights Reserved 67
• CloningproducMontonon-productinevitablyperformedinfrequently…
• …meaningthatschemasanddataareolder,dirMerdueto
previoustesMng,andtoosmalltoaccuratelyrepresentreality
A^emp'ngtosolvetheproblem…
PROD QATEST DEV UAT
© 2015 Delphix. All Rights Reserved 68
PROD
• CreaMngsnapshotswithmirrorsplits…– Mirrorthevolumeoneextracopy
A^emp'ngtosolvetheproblem…
PRODPROD
© 2015 Delphix. All Rights Reserved 69
PROD
• CreaMngsnapshotswithmirrorsplits…– Mirrorthevolumeoneextracopy– ThenceasewritestothatextracopyataspecificMmeandmakeitseparately
addressableasaseparatevolume
A^emp'ngtosolvetheproblem…
PRODPROD
© 2015 Delphix. All Rights Reserved 70
PROD
• CreaMngsnapshotswithmirrorsplits…– Mirrorthevolumeoneextracopy– ThenceasewritestothatextracopyataspecificMmeandmakeitseparately
addressableasaseparatevolume– Tocreateanothersnapshot,“re-silver”thevolumeintothemirrorset
A^emp'ngtosolvetheproblem…
PRODPROD
© 2015 Delphix. All Rights Reserved 71
PROD
• CreaMngsnapshotswithmirrorsplits…– Mirrorthevolumeoneextracopy– ThenceasewritestothatextracopyataspecificMmeandmakeitseparately
addressableasaseparatevolume– Tocreateanothersnapshot,“re-silver”thevolumeintothemirrorset– Andit’sreadytosplitagain
A^emp'ngtosolvetheproblem…
PRODPROD
© 2015 Delphix. All Rights Reserved 72
PROD
• CreaMngsnapshotswithmirrorsplits…– Mirrorthevolumeoneextracopy– ThenceasewritestothatextracopyataspecificMmeandmakeitseparately
addressableasaseparatevolume– Tocreateanothersnapshot,“re-silver”thevolumeintothemirrorset– Andit’sreadytosplitagain
– Whilewe’vesolvedtheproblemofquicklycreaMngnewcopies
A^emp'ngtosolvetheproblem…
PRODPROD
© 2015 Delphix. All Rights Reserved 73
PROD
• CreaMngsnapshotswithmirrorsplits…– Mirrorthevolumeoneextracopy– ThenceasewritestothatextracopyataspecificMmeandmakeitseparately
addressableasaseparatevolume– Tocreateanothersnapshot,“re-silver”thevolumeintothemirrorset– Andit’sreadytosplitagain
– Whilewe’vesolvedtheproblemofquicklycreaMngnewcopies• Wehaven’tsolvedtheproblemofusingafullimage
A^emp'ngtosolvetheproblem…
PRODPROD
© 2015 Delphix. All Rights Reserved 74
PROD
• CreaMngsnapshotswithmirrorsplits…– Mirrorthevolumeoneextracopy– ThenceasewritestothatextracopyataspecificMmeandmakeitseparately
addressableasaseparatevolume– Tocreateanothersnapshot,“re-silver”thevolumeintothemirrorset– Andit’sreadytosplitagain
– Whilewe’vesolvedtheproblemofquicklycreaMngnewcopies• Wehaven’tsolvedtheproblemofusingafullimage• Andre-silveringisnotafastoperaMon,soweneedMmeto“reload”
A^emp'ngtosolvetheproblem…
PRODPROD
© 2015 Delphix. All Rights Reserved 75
A^emp'ngtosolvetheproblem…
CEO
COO
CFO CMO CSO CIO
StorageAdministrator
We’resMllatsquareone…
…anditgetsevenworseBecausenowwehavetopredictthesizeofeverythingwithperfectaccuracy
andthenpre-allocateforcopiesaswellasvolumes
© 2015 Delphix. All Rights Reserved 76
A^emp'ngtosolvetheproblem…
CEO
COO
CFO CMO CSO CIO
StorageAdministrator
We’resMllatsquareone…
…anditgetsevenworseBecausenowwehavetopredictthesizeofeverythingwithperfectaccuracy
andthenpre-allocateforcopiesaswellasvolumes
© 2015 Delphix. All Rights Reserved 77
A^emp'ngtosolvetheproblem…
CEO
COO
CFO CMO CSO CIO
StorageAdministrator
We’resMllatsquareone…
…anditgetsevenworseBecausenowwehavetopredictthesizeofeverythingwithperfectaccuracy
andthenpre-allocateforcopiesaswellasvolumes
© 2015 Delphix. All Rights Reserved 78
A^emp'ngtosolvetheproblem…
CEO
COO
CFO CMO CSO CIO
StorageAdministrator
We’resMllatsquareone…
…anditgetsevenworseBecausenowwehavetopredictthesizeofeverythingwithperfectaccuracy
andthenpre-allocateforcopiesaswellasvolumes
© 2015 Delphix. All Rights Reserved 79
A^emp'ngtosolvetheproblem…
CEO
COO
CFO CMO CSO CIO
StorageAdministrator
We’resMllatsquareone…
…anditgetsevenworseBecausenowwehavetopredictthesizeofeverythingwithperfectaccuracy
andthenpre-allocateforcopiesaswellasvolumes
© 2015 Delphix. All Rights Reserved 80
So,inaperfectworldwhatwe’dliketoseeissomethinglikethis…
A^emp'ngtosolvetheproblem…
PROD
QATforTom
QATforDick
UATforSally
DEVforHarry
QATforBecky
© 2015 Delphix. All Rights Reserved 81
A^emp'ngtosolvetheproblem…
PROD
DEV/QAT/UAT
Tom Dick Sally Harry Becky
…but,intherealworldweseethis…
© 2015 Delphix. All Rights Reserved 82
• It’snoteasyfordevelopersandtesterstoshareenvironmentssimultaneously– Changesmadebyeachuserconflictwithoneanother
• Soitbecomesnecessarytoserializeaccesstothelimitednumberofenvironments– Eachuserqueuesattheresource,sotheirprojecttakesmore'me
elapsed– EachuserdestrucMvelychangestheresource
• Andnowweneedawaytoresetorcleanuptheenvironmentbetweeneachuse– ResetorcleanuptakesMme,whichcausesfurtherdelaysbetween
uses
A^emp'ngtosolvetheproblem…
© 2015 Delphix. All Rights Reserved 83
• It’snoteasyfordevelopersandtesterstoshareenvironmentssimultaneously– Changesmadebyeachuserconflictwithoneanother
• Soitbecomesnecessarytoserializeaccesstothelimitednumberofenvironments– Eachuserqueuesattheresource,sotheirprojecttakesmore'me
elapsed– EachuserdestrucMvelychangestheresource
• Andnowweneedawaytoresetorcleanuptheenvironmentbetweeneachuse– ResetorcleanuptakesMme,whichcausesfurtherdelaysbetween
uses
A^emp'ngtosolvetheproblem…
© 2015 Delphix. All Rights Reserved 84
• It’snoteasyfordevelopersandtesterstoshareenvironmentssimultaneously– Changesmadebyeachuserconflictwithoneanother
• Soitbecomesnecessarytoserializeaccesstothelimitednumberofenvironments– Eachuserqueuesattheresource,sotheirprojecttakesmore'me
elapsed– EachuserdestrucMvelychangestheresource
• Andnowweneedawaytoresetorcleanuptheenvironmentbetweeneachuse– ResetorcleanuptakesMme,whichcausesfurtherdelaysbetween
uses
A^emp'ngtosolvetheproblem…
© 2015 Delphix. All Rights Reserved 85
• It’snoteasyfordevelopersandtesterstoshareenvironmentssimultaneously– Changesmadebyeachuserconflictwithoneanother
• Soitbecomesnecessarytoserializeaccesstothelimitednumberofenvironments– Eachuserqueuesattheresource,sotheirprojecttakesmoreMme
elapsed– EachuserdestrucMvelychangestheresource
• Andnowweneedawaytoresetorcleanuptheenvironmentbetweeneachuse– ResetorcleanuptakesMme,whichcausesfurtherdelaysbetween
uses
A^emp'ngtosolvetheproblem…
© 2015 Delphix. All Rights Reserved 86
• It’snoteasyfordevelopersandtesterstoshareenvironmentssimultaneously– Changesmadebyeachuserconflictwithoneanother
• Soitbecomesnecessarytoserializeaccesstothelimitednumberofenvironments– Eachuserqueuesattheresource,sotheirprojecttakesmoreMme
elapsed– EachuserdestrucMvelychangestheresource
• Andnowweneedawaytoresetorcleanuptheenvironmentbetweeneachuse– ResetorcleanuptakesMme,whichcausesfurtherdelaysbetween
uses
A^emp'ngtosolvetheproblem…
© 2015 Delphix. All Rights Reserved 87
• It’snoteasyfordevelopersandtesterstoshareenvironmentssimultaneously– Changesmadebyeachuserconflictwithoneanother
• Soitbecomesnecessarytoserializeaccesstothelimitednumberofenvironments– Eachuserqueuesattheresource,sotheirprojecttakesmoreMme
elapsed– EachuserdestrucMvelychangestheresource
• Andnowweneedawaytoresetorcleanuptheenvironmentbetweeneachuse– ResetorcleanuptakesMme,whichcausesfurtherdelaysbetween
uses
A^emp'ngtosolvetheproblem…
© 2015 Delphix. All Rights Reserved 88
• It’snoteasyfordevelopersandtesterstoshareenvironmentssimultaneously– Changesmadebyeachuserconflictwithoneanother
• Soitbecomesnecessarytoserializeaccesstothelimitednumberofenvironments– Eachuserqueuesattheresource,sotheirprojecttakesmoreMme
elapsed– EachuserdestrucMvelychangestheresource
• Andnowweneedawaytoresetorcleanuptheenvironmentbetweeneachuse– ResetorcleanuptakesMme,whichcausesfurtherdelaysbetween
uses
A^emp'ngtosolvetheproblem…
© 2015 Delphix. All Rights Reserved 89
• It’snoteasyfordevelopersandtesterstoshareenvironmentssimultaneously– Changesmadebyeachuserconflictwithoneanother
• Soitbecomesnecessarytoserializeaccesstothelimitednumberofenvironments– Eachuserqueuesattheresource,sotheirprojecttakesmore'me
elapsed– EachuserdestrucMvelychangestheresource
• Andnowweneedawaytoresetorcleanuptheenvironmentbetweeneachuse– ResetorcleanuptakesMme,whichcausesfurtherdelaysbetween
uses
A^emp'ngtosolvetheproblem…
© 2015 Delphix. All Rights Reserved
• Dataconstraint• AYemptedsoluMons• SoluMon• UseCases
Inthispresenta'on:
© 2015 Delphix. All Rights Reserved 91
• ThebiggestproblemwithcloningenvironmentsanddatabasesisalltheduplicaMonMost(ifnotall)oftheblockscomprisingthedatabaseorfile-systemexactlythesame
Whatifwestoreonlyonecopyofeachversionofablock?Copy-on-write(COW)
Nevermodifyablock,butinsertanewversioninstead
Thenletthestoragefigureoutwhatwe’resupposedtobeseeing?ThestoragemaintainstherelaMonshipsandversionsonretrieval
AndthenmakeitportablesothatANYplatorm(Unix/Linux,Windows)canuseit?
NFS(networkfilesystem)
Asolu'on…
© 2015 Delphix. All Rights Reserved 92
• ThebiggestproblemwithcloningenvironmentsanddatabasesisalltheduplicaMon– Most(ifnotall)oftheblockscomprisingthedatabaseorfile-system
exactlythesame
Whatifwestoreonlyonecopyofeachversionofablock?Copy-on-write(COW)
Nevermodifyablock,butinsertanewversioninstead
Thenletthestoragefigureoutwhatwe’resupposedtobeseeing?ThestoragemaintainstherelaMonshipsandversionsonretrieval
AndthenmakeitportablesothatANYplatorm(Unix/Linux,Windows)canuseit?
NFS(networkfilesystem)
Asolu'on…
© 2015 Delphix. All Rights Reserved 93
• ThebiggestproblemwithcloningenvironmentsanddatabasesisalltheduplicaMon– Most(ifnotall)oftheblockscomprisingthedatabaseorfile-system
exactlythesame
• Whatifwestoreonlyonecopyofeachversionofablock?Copy-on-write(COW)
Nevermodifyablock,butinsertanewversioninstead
Thenletthestoragefigureoutwhatwe’resupposedtobeseeing?ThestoragemaintainstherelaMonshipsandversionsonretrieval
AndthenmakeitportablesothatANYplatorm(Unix/Linux,Windows)canuseit?
NFS(networkfilesystem)
Asolu'on…
© 2015 Delphix. All Rights Reserved 94
• ThebiggestproblemwithcloningenvironmentsanddatabasesisalltheduplicaMon– Most(ifnotall)oftheblockscomprisingthedatabaseorfile-system
exactlythesame
• Whatifwestoreonlyonecopyofeachversionofablock?– Copy-on-write(COW)
• Nevermodifyablock,butinsertanewversioninstead
Thenletthestoragefigureoutwhatwe’resupposedtobeseeing?ThestoragemaintainstherelaMonshipsandversionsonretrieval
AndthenmakeitportablesothatANYplatorm(Unix/Linux,Windows)canuseit?
NFS(networkfilesystem)
Asolu'on…
© 2015 Delphix. All Rights Reserved 95
• ThebiggestproblemwithcloningenvironmentsanddatabasesisalltheduplicaMon– Most(ifnotall)oftheblockscomprisingthedatabaseorfile-system
exactlythesame
• Whatifwestoreonlyonecopyofeachversionofablock?– Copy-on-write(COW)
• Nevermodifyablock,butinsertanewversioninstead
• Thenletthestoragefigureoutwhatwe’resupposedtobeseeing?ThestoragemaintainstherelaMonshipsandversionsonretrieval
AndthenmakeitportablesothatANYplatorm(Unix/Linux,Windows)canuseit?
NFS(networkfilesystem)
Asolu'on…
© 2015 Delphix. All Rights Reserved 96
• ThebiggestproblemwithcloningenvironmentsanddatabasesisalltheduplicaMon– Most(ifnotall)oftheblockscomprisingthedatabaseorfile-system
exactlythesame
• Whatifwestoreonlyonecopyofeachversionofablock?– Copy-on-write(COW)
• Nevermodifyablock,butinsertanewversioninstead
• Thenletthestoragefigureoutwhatwe’resupposedtobeseeing?– ThestoragemaintainstherelaMonshipsandversionsonretrieval
AndthenmakeitportablesothatANYplatorm(Unix/Linux,Windows)canuseit?
NFS(networkfilesystem)
Asolu'on…
© 2015 Delphix. All Rights Reserved 97
• ThebiggestproblemwithcloningenvironmentsanddatabasesisalltheduplicaMon– Most(ifnotall)oftheblockscomprisingthedatabaseorfile-system
exactlythesame
• Whatifwestoreonlyonecopyofeachversionofablock?– Copy-on-write(COW)
• Nevermodifyablock,butinsertanewversioninstead
• Thenletthestoragefigureoutwhatwe’resupposedtobeseeing?– ThestoragemaintainstherelaMonshipsandversionsonretrieval
• AndthenmakeitportablesothatANYplatorm(Unix/Linux,Windows)canuseit?NFS(networkfilesystem)
Asolu'on…
© 2015 Delphix. All Rights Reserved 98
• ThebiggestproblemwithcloningenvironmentsanddatabasesisalltheduplicaMon– Most(ifnotall)oftheblockscomprisingthedatabaseorfile-system
exactlythesame
• Whatifwestoreonlyonecopyofeachversionofablock?– Copy-on-write(COW)
• Nevermodifyablock,butinsertanewversioninstead
• Thenletthestoragefigureoutwhatwe’resupposedtobeseeing?– ThestoragemaintainstherelaMonshipsandversionsonretrieval
• AndthenmakeitportablesothatANYplatorm(Unix/Linux,Windows)canuseit?– NFS(networkfilesystem–Unix/Linux),iSCSI(Windows)
Asolu'on…
© 2015 Delphix. All Rights Reserved 99
• Cloningtheoldway…
Asolu'on…
PROD QAT
© 2015 Delphix. All Rights Reserved 100
• Cloningthesmartway…
Asolu'on…
PROD QATdSource
© 2015 Delphix. All Rights Reserved 101
• Cloningthesmartway…– Aschangesaremadetotheclone,onlythosechangedblocksarestored…
Asolu'on…
QATdSourcePROD
© 2015 Delphix. All Rights Reserved 102
• Cloningthesmartway…– Aschangesaremadetotheclone,onlythosechangedblocksarestored…
• Takeslessstorage,fastertocopy• Thecostoftheclonehasgonewaydownintermsofstorageand'me
Asolu'on…
QATdSourcePROD
© 2015 Delphix. All Rights Reserved 103
• Cloningthesmartway…– Aschangesaremadetotheclone,onlythosechangedblocksarestored…
• Takeslessstorage,fastertocopy• Thecostoftheclonehasgonewaydownintermsofstorageand'me
Asolu'on…
CapEx
QATdSourcePROD
© 2015 Delphix. All Rights Reserved 104
• Cloningthesmartway…– Aschangesaremadetotheclone,onlythosechangedblocksarestored…
• Takeslessstorage,fastertocopy• Thecostoftheclonehasgonewaydownintermsofstorageand'me
Asolu'on…
CapEx OpEx
QATdSourcePROD
© 2015 Delphix. All Rights Reserved 105
• Overallarchitecture• Internalfile-system
Howdoesitwork?
© 2015 Delphix. All Rights Reserved 106
Overallarchitecture
Delphix
PROD1
PROD2
PROD3
<50msnetwork
SSHJDBCDSP
© 2015 Delphix. All Rights Reserved 107
Overallarchitecture
Delphix
PROD1
PROD2
PROD3
<50msnetwork
SSHJDBCDSP
NotreplicaMon(currentpoint-in-7me),buta'meflowwithconfiguredretenMon
© 2015 Delphix. All Rights Reserved 108
Overallarchitecture
Delphix
PROD1
PROD2
PROD3
<50ms
Virtualmachine• 4,8,12,16vCPUs• 32-512GBvRAM• <8-10msI/O
latencytounderlyingstorage
network
SSHJDBCDSP
NotreplicaMon(currentpoint-in-7me),buta'meflowwithconfiguredretenMon
© 2015 Delphix. All Rights Reserved 109
VDB4
VDB3
VDB2
Overallarchitecture
Delphix
PROD1
PROD2
PROD3
VDB1
network
<50ms <1ms
Virtualmachine• 4,8,12,16vCPUs• 32-512GBvRAM• <8-10msI/O
latencytounderlyingstorage
UNIX/Linux:NFSWindows:iSCSI
network
SSHJDBCDSP
NotreplicaMon(currentpoint-in-7me),buta'meflowwithconfiguredretenMon
© 2015 Delphix. All Rights Reserved
Snapshot1–ini'alload
a b c d e f g h i
We start with a full backup. Includes transaction logs needed for recovery.
JonathanLewis©2015
Sn#1
110
© 2015 Delphix. All Rights Reserved
Snapshot2–incomingnewchanges
b' c'
a b c d e f g h i
A^eriniMallink/load,Delphixisincrementalforever
JonathanLewis©2015111
Sn#1
© 2015 Delphix. All Rights Reserved
ApplySnapshot2
a b c d e f g h ib' c'
The Delphix appliance unpacks the backup and "overwrites" the initial backup with the changed blocks. DxFS makes new copies of the changed blocks, never updates a block
112JonathanLewis©2015
Sn#1 Sn#2
© 2015 Delphix. All Rights Reserved
Snapshot3–moreincomingchanges
a b c d e f g h ib' c'
113JonathanLewis©2015
i'
Sn#1 Sn#2
© 2015 Delphix. All Rights Reserved
ApplySnapshot3
a b c d e f g h ib' c'
114JonathanLewis©2015
i'
Sn#1 Sn#2 Sn#3
© 2015 Delphix. All Rights Reserved
Reten'onpolicies–droppingsnapshots
a b c d e f g h ib' c'
115JonathanLewis©2015
i'
Sn#1 Sn#2 Sn#3
© 2015 Delphix. All Rights Reserved
Reten'onpolicies–droppingsnapshots
a d e f g h ib' c'
116JonathanLewis©2015
i'
Sn#2 Sn#3
© 2015 Delphix. All Rights Reserved
Crea'ngaVDB
a d e f g h ib' c'
117JonathanLewis©2015
i'
Sn#2 Sn#3VDB#1
© 2015 Delphix. All Rights Reserved
Crea'ngaVDB
a d e f g h ib' c'
118JonathanLewis©2015
i'
Sn#2 Sn#3VDB#1
© 2015 Delphix. All Rights Reserved
ChangestoaVDB
a d e f g h ib' c'
119JonathanLewis©2015
i'
Sn#2 Sn#3VDB#1
a‘'
© 2015 Delphix. All Rights Reserved
ChangestoaVDB
a d e f g h ib' c'
120JonathanLewis©2015
i'
Sn#2 Sn#3VDB#1
a‘'
© 2015 Delphix. All Rights Reserved
OpenZFSfeaturesemployedwithinDxFS• Copy-On-WritetransacMonmodel• 128-bitfile-system• L1ARC–RAM-basedadapMvereplacementcache(ARC)• Dynamicstriping• Lightweightfile-systemcreaMon(snapshotsandclones)• ChecksumsandMerkleTreefile-system• Block-leveldeduplicaMon• Transparentblockcompression• Variableblocksizes• ReplicaMon• AdapMveendianness
121
© 2015 Delphix. All Rights Reserved
Changingreality
122
© 2015 Delphix. All Rights Reserved
Changingreality
123
© 2015 Delphix. All Rights Reserved
Changingreality
124
© 2015 Delphix. All Rights Reserved
DatabaseVirtualiza'on
125
© 2015 Delphix. All Rights Reserved
• Dataconstraint• AYemptedsoluMons• SoluMon• Usecases
Inthispresenta'on:
© 2015 Delphix. All Rights Reserved
WorkingwithVirtualData• Unlimited• Fullsize• SelfService• Secure• Cloudonramp Developing
Tes'ngDebugging
© 2015 Delphix. All Rights Reserved
WorkingwithVirtualData:Unlimited
GIFbySteveKaram 128
© 2015 Delphix. All Rights Reserved
WorkingwithVirtualData:Fullsize
129
© 2015 Delphix. All Rights Reserved
WorkingwithVirtualData:SelfService
130
© 2015 Delphix. All Rights Reserved
WorkingwithVirtualData:Secure
131
• Encryp'onreversiblychangesdataatrest,authenMcatedandauthorizedusersmayview
• Redac'onchangesdataonretrieval
• Maskingirreversiblychangesdataatrest– maintainingrelaMonalintegrityacrossdatadistribuMonpaYerns
© 2015 Delphix. All Rights Reserved
WorkingwithVirtualData:cloudonramp
132
Thecloudislargelyano-flyzoneforproprietaryUNIX.MigraMngfromUNIXtoLinux(U2L)iso^enthefirststepofacloudmigraMon.DatavirtualizaMonmakesU2Leasierforcross-platormdevelopmentandtesMngandmigraMon
© 2015 Delphix. All Rights Reserved
TesMngwithVirtualData• Fast,parallel• Rewind• A/BtesMng• Sharing
© 2015 Delphix. All Rights Reserved
Dev
QA
Instance
Prod
DVA
• Low Resource • Find bugs Fast
Tes'ngwithVirtualData:Fast,Parallel
Produc'onTimeFlow
134
© 2015 Delphix. All Rights Reserved
Tes'ngwithVirtualData:Rewind
Instance
QA
Prod
Produc'onTimeFlow
135
© 2015 Delphix. All Rights Reserved
Tes'ngwithVirtualData:A/Btes'ng
Instance
Instance
Instance
Index1
Index2
Produc'onTimeFlow
136
© 2015 Delphix. All Rights Reserved
Tes'ngwithVirtualData:versioncontrol
137
Dev
QA
2.1
Dev
QA
2.2
2.1 2.2
Instance
Prod
DVA Produc'onTimeFlow
© 2015 Delphix. All Rights Reserved
Tes'ngwithVirtualData:sharing
Instance Jane’sVDB'meflow
138
DialoguebetweenQAtester(Jane)anddeveloper(Mary)…
Jane:Mary,I’vefoundadataprobleminversion3.2.2a.Mary:Canyoubookmarkthatandshareit?Jane:Itisnamed“Jane3.2.2a3pm”,lookfordupsintablesABCandXYZ.Mary:Hmm,IrestoreditandIseetheproblem.Thanks!
Instance Mary’sbranch
Jane3.2.2a3pm
© 2015 Delphix. All Rights Reserved
• Projects “12 months to 6 months.” – New York Life
• Insurance product “about 50 days ... to about 23 days” – Presbyterian Health
• “Can't imagine working without it” – State of California
VirtualDataquotesfromcustomers
© 2015 Delphix. All Rights Reserved
• Problem:Dataistheconstraint• SoluMon:Virtualizethedata• Results
• HalftheMmeforprojects• Higherquality,beYertesMng• Increaserevenue
Summary
© 2015 Delphix. All Rights Reserved
© 2015 Delphix. All Rights Reserved
Pleasecompletethesessionevalua'onWeappreciateyourfeedbackandinsightYoumaycompletethesessionevaluaMon(session#1227)usingthemobileapp
Email: Tim.Gorman@Delphix.comAdd’lquesMons: hYp://Community.Delphix.comBlog:hYp://EvDBT.com Mobile:+1(303)885-4526“TheGoal:AProcessOfOngoingImprovement”in1984byEliyahuGoldraY“ThePhoenixProject:ANovelAboutIT,DevOps,AndHelpingYourBusinessWin”in2012byGeneKim,KevinBehr,GeorgeSpafford
top related