devops, databases and the phoenix project ugf4042 from oow14
Post on 27-Nov-2014
210 Views
Preview:
DESCRIPTION
TRANSCRIPT
Virtual Data Platform: or
Revolutionizing Database Cloning
How can the DBA make the biggest impact on the company
1
http://kylehailey.comkyle@delphix.com
The Goal : Theory of Constraints
Improvement not made at the constraint is an illusion
factory floor optimization
Factory floor
resinMolding
Trimmer
Leak detection
Labeling
Capping/Filling
Pallet - izing
Shipping
Factory floor
resinMolding
Trimmer
Leak detection
Labeling
Capping/Filling
Pallet - izing
Shipping
constraint
Not a relay race
resinMolding
Trimmer
Leak detection
Labeling
Capping/Filling
Pallet - izing
Shipping
Tune before constraint
constraint
Tuning here
Stock piling
resinMolding
Trimmer
Leak detection
Labeling
Capping/Filling
Pallet - izing
Shipping
Tune after constraint
constraint
Tuning here
Starvation
Factory floor : straight forward
constraint
Goal: find constraint optimize it
Theory of Constraints work for IT ?
• Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast
• CI• Cloud • Agile • Kanban• Kata
“IT is the factory floor of this century”
The Phoenix Project
What is the constraint
in IT ?
What are the top 5 constraints in IT?
1. Dev environments setup2. QA setup3. Code Architecture4. Development5. Product management
- Gene Kim
“One of the most powerful things that organizations can do is to enable development and testing to get environment they need when they need it“
Data is the constraint
60% Projects Over Schedule
85% delayed waiting for data
Data is the Constraint
CIO Magazine Survey:
only getting worseGartner: Data Doomsday, by 2017 1/3rd IT in crisis
• Data Constraint• Solution• Use Cases
In this presentation :
• Data ConstraintI. strains ITII. price is hugeIII. companies unaware
• Solution• Use Cases
– Storage & Systems– Personnel – Time
moving data is hard
Typical Architecture
Production
Instance
File system
Database
Typical Architecture
Production
Instance
Backup
File system
Database
File system
Database
Typical Architecture
Production
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File system
Database
Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance Instance
Instance
File system
Database
File system
Database
Dev, QA, UAT Reporting Backup
Triple Tax
Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance Instance
Instance
File system
Database
File system
Database
Data floods infrastructure
92% of the cost of business,
in financial services business , is “data”
www.wsta.org/resources/industry-articles
Most companies have 2-9% IT spending , ½ on “data”
http://uclue.com/?xq=1133
• Data ConstraintI. strains ITII. price is hugeIII. companies unaware
• Solution• Use Cases
In this presentation :
Four Areas data tax hits
– IT Capital resources $– IT Operations personnel $– Application Development $$$– Business $$$$$$$
price is Huge
$Hardware–Servers–Storage–Network–Data center floor space, power, cooling
$ Never enough environments
• People– DBAs– SYS Admin– Storage Admin– Backup Admin – Network Admin
• Hours : 1000s just for DBAs • $100s Millions for data center modernizations
$ IT Operations
• Inefficient QA: Higher costs of QA• QA Delays : Greater re-work of code• Sharing DB Environments : Bottlenecks• Using DB Subsets: More bugs in Prod• Slow Environment Builds: Delays
$ Application Development
Ability to capture revenue
• Business Applications – Delays cause lost revenue
• Business Intelligence – Old data = less intelligence
$ Business
• Data ConstraintI. strains ITII. price is hugeIII. companies unaware
• Solution• Use Cases
In this presentation :
companies unaware
companies unaware
Developer or AnalystBoss, Storage Admin, DBA
Metrics–Time –Old Data –Storage –Analysts –Audits
companies unaware
• Data ConstraintI. strains ITII. price is hugeIII. companies unaware
• Solution• Use Cases
In this presentation :
Clone 1 Clone 3Clone 2
99% of blocks are identical
Solution
Clone 1 Clone 2 Clone 3
Thin Clone
• EMC – 16 snapshots on Symmetrix– Write performance impact– No snapshots of snapshots
• Netapp– 255 snapshots
• ZFS– Compression– Unlimited snapshots– Snapshots of Snapshots
• DxFS– “”– Storage agnostic– Shared cache in memory
Technology Core : file system snapshots
Also check out new SSD storage such as:Pure Storage, EMC XtremIO
Fuel not equal car
Challenges 1. Technical2. Bureaucracy
Bureaucracy
Developer Asks for DB Get Access
Manager approves
DBA Request system
Setup DB
System Admin
Requeststorage
Setup machine
Storage Admin
Allocate storage (take snapshot)
Why are hand offs so expensive?
1hour1 day
9 days
Bureaucracy
Technical Challenge
Database Luns
Production FilerTarget A
Target B
Target C
snapshotclones
InstanceInstance
InstanceInstance
InstanceInstance
InstanceInstance
Instance
Source
Database LUNs
snapshotclonesProduction Filer
Development Filer
Technical Challenge
Instance
Target A
Target B
Target C
InstanceInstance
InstanceInstance
InstanceInstance
Instance
Technical Challenge
Copy Time FlowPurge
Production
File System
Instance
DevelopmentStorage
21 3
Clone (snapshot)CompressShare Cache
ProvisionMount, recover, renameSelf Service, Roles & Security
Instance
– EMC + SRDF – Netapp + SMO– Oracle EM 12c + data guard + Netapp /ZFS – Delphix
2 12 1
3 1 2
How to get a Data Virtualization?Source
sync
Deployautomation
Storagesnapshots
21 3
Goal : virtualize, govern, deliver
04/09/2023 44
• Security• Masking• Chain of custody
• Self Service• Roles• Restrictions
• Developer• Data Versioning • Refresh, Rollback
• Audit:• Live Archive Snap Shots
Thin CloningData Virtualization
Data Supply Chain
Intel hardware
Unstructured Data
Install Delphix on x86 hardware
Allocate Any Storage to Delphix
Allocate StorageAny type
Pure Storage + DelphixBetter Performance for 1/10 the cost
One time backup of source database
Database
Production
File systemFile system
InstanceInstanceInstance
DxFS (Delphix) Compress Data
Database
Production
Data is compressed typically 1/3 size
File system
InstanceInstanceInstance
Incremental forever change collection
Database
Production
File system
Changes
• Collected incrementally forever• Old data purged
File system Time Window
Production
InstanceInstanceInstance
Snapshot 1 – full backup once only at link time
Jonathan Lewis © 2013 Virtual DB
50 / 30
a b c d e f g h i
We start with a full backup - analogous to a level 0 rman backup. Includes the archived redo log files needed for recovery. Run in archivelog mode.
Snapshot 2 (from SCN)
Jonathan Lewis © 2013
b' c'
a b c d e f g h i
The "backup from SCN" is analogous to a level 1 incremental backup (which includes the relevant archived redo logs). Sensible to enable BCT.
Delphix executes standard rman scripts
Apply Snapshot 2
Jonathan Lewis © 2013
a b c d e f g h ib' c'
The Delphix appliance unpacks the rman backup and "overwrites" the initial backup with the changed blocks - but DxFS makes new copies of the blocks
Drop Snapshot 1
Jonathan Lewis © 2013
b' c'a d e f g h i
The call to rman leaves us with a new level 0 backup, waiting for recovery. But we can pick the snapshot root block. We have EVERY level 0 backup
Creating a vDB
Jonathan Lewis © 2013
b' c'a d e f g h i
The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward)
My vDB(filesystem)
Your vDB(filesystem)
b' c'a d e f g h i
Creating a vDB
Jonathan Lewis © 2013
b' c'a d e f g h i
The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward)
My vDB(filesystem)
Your vDB(filesystem)
i’
b' c'a d e f g h ib' c'a d e f g h i
Database Virtualization
Three Physical CopiesThree Virtual Copies
Data Virtualization Appliance
Before Virtual Data
Production Dev, QA, UAT
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance Instance
Instance
File system
Database
File system
Database
“triple data tax”
With Virtual DataProduction
Instance
Database
Dev & QA
Instance
Database
Reporting
Instance
Database
Backup
Instance Instance Instance
Database
InstanceInstance
Database
InstanceInstance
File system
Database
Data Virtualization Appliance
• Problem in the Industry• Solution• Use Cases
In this presentation :
1. Development and QA 2. Production Support3. Business
Use Cases
1. Development and QA2. Production Support3. Business
Use Cases
Development : bottlenecks
Frustration Waiting
Old Unrepresentative Data
Development : subsets
False NegativesFalse PositivesBugs in Production
Development : bugs
http://martinfowler.com/bliki/NoDBA.html
Development : slow env build times
Development: Virtual Data
• Unlimited • Full size • Self Service
Development
Virtual Data: Easy
Instance
Instance
Instance
Instance
Source
DVA
Development Virtual Data: Parallelize
gif by Steve Karam
Development Virtual Data: Full size
Development Virtual Data: Self Service
QA : Virtual Data• Fast • Parallel• Rollback• A/B testing
QA : Long Build times
96% of QA time was building environment$.04/$1.00 actual testing vs. setup
QA Build QAQA Build QA
BugX
1 2 3 4 5 6 70
10203040506070
Delay in Fixing the bug
Cost ToCorrect
Software Engineering Economics – Barry Boehm (1981)
Dev
QA
Instance
Prod
DVATime Flow
• Low Resource• Find bugs Fast
QA Virtual Data : Fast
QA with Virtual Data: Rewind
Instance
Instance
Development
Prod
QA with Virtual Data: A/B
Instance
Instance
Instance
Index 1
Index 2
Data Version Control
04/09/2023 77
Dev
QA
2.1
Dev
QA
2.2
2.1 2.2
Instance
Prod
DVA
1. Development and QA2. Production Support3. Business
Use Cases
• Backups• Recovery• Forensics• Migration• Consolidation
Recovery
Backups
Recovery Instance
Instance
Recover VDB
Drop
Source
DVA
Forensics
Instance
Instance
Development DVA
Source
Development (the new production)
Instance Instance
Development DVA
Source
Instance
Development
Migration
Consolidation
1. Development and QA2. Production Support3. Business Intelligence
Use Cases
Business Intelligence
• ETL• Temporal• Confidence Testing• Federated Databases• Audits
Business Intelligence: ETL and Refresh Windows
1pm 10pm 8am noon
Business Intelligence: batch taking too long
1pm 10pm 8am noon20112012201320142015
20112012201320142015
1pm 10pm 8am noon
10pm 8am noon 9pm
6am 8am 10pm
Business Intelligence: ETL and DW Refreshes
Instance
Prod
Instance
DW & BI
• Collect only Changes• Refresh in minutes
Instance Instance
Prod BI and DW
ETL24x7
DVA Instance
Virtual Data: Fast Refreshes
Temporal Data
Confidence testing
Modernization: Federated
Instance
Instance
Instance
Instance
Source1
Source2
DVA
Modernization: Federated
“I looked like a hero”Tony Young, CIO Informatica
Modernization: Federated
Audit
04/09/2023 98
Instance
Prod
DVA
Live Archive
1. Development & QA2. Production Support3. Business
Use Case Summary
How expensive is the Data Constraint?
DVA at Fortune 500 :
Dev throughput increase by 2x
• Faster Financial Close• Faster BI refreshes• Faster surgical recovery• More Project tracks• Faster Projects
How expensive is the Data Constraint?
• Projects “12 months to 6 months.”– New York Life
• Insurance product “about 50 days ... to about 23 days”– Presbyterian Health
• “Can't imagine working without it”– State of California
Virtual Data Quotes
• Problem: Data is the constraint • Solution: Virtualize Data• Results:
• Half the time for projects• Higher quality• Increase revenue
Summary
105
Oaktable World & hands on labs
We are here
Oaktable World
MosconeSouth
Thank you!
• Kyle Hailey| Oracle ACE and Technical Evangelist, Delphix – Kyle@delphix.com– kylehailey.com– slideshare.net/khailey
Oracle 12c
80MB buffer cache ?
200GBCache
5000
Tnxs
/ m
inLa
tenc
y
300 ms
1 5 10 20 30 60 100 200
with
1 5 10 20 30 60 100 200Users
8000
Tnxs
/ m
inLa
tenc
y
600 ms
1 5 10 20 30 60 100 200Users
1 5 10 20 30 60 100 200
$1,000,000 1TB cache on SAN
$6,000200GB shared cache on Delphix
Five 200GB database copies are cached with :
04/09/2023 113
04/09/2023 114
Business Intelligence
a) 24x7 Batches
b) Temporal queries
c) Confidence testing
Thin Cloning
Snap Manager
SnapManagerRepository
Protection Manager
Snap Drive
Snap Manager
Snap Mirror
Flex Clone
RMANRepository
Production
Development
DBA
Storage Admin
1 tr-3761.pdf
Netapp
NetApp Filer - DevelopmentNetApp Filer - Production
Database Luns
Snap mirror
Snapshot Manager for Oracle
Flexclone
Repository Database
SnapDrive
Protection Manage
Production
Development
1 NetappTarget A
Target B
Target C
InstanceInstance
InstanceInstance
InstanceInstance
Instance
Where we want to be
Database
File system
Production
Instance
Database
Development
Instance
Database
QA
Instance
Database
UAT
Instance
Snapshots
Instance Instance Instance Instance
EM 12c: Snap Clone
Production Development
Flexclone Flexclone
Netapp Snap Manager for Oracle
II. Data constraint price is Huge : 4. Business
II. Data constraint price is Huge : 4. Business
Storage
IT Ops
Dev
Revenue
0 5000 10000 15000 20000 25000 30000Billion $
III. Data Constraint companies unaware
#1 Biggest Enemy :
IT departments believe– best processes – greatest technology– Just the way it is
Are you Innovative?
III. Data Constraint companies unaware
Why do I need an iPhone ?
Don’t we already do that ?
SQL scriptsAlter database begin backupBack up datafilesRedoArchiveAlter database end backup
RMAN
Merge Dev1 to ForkMerge to dev2
Dev2
Dev1Merge to dev1
Merge Dev2 to Fork
Trunk
Merge Dev1 to Fork
Merge Dev2 to Fork
DBVC
Fork
Fork
Fork
Fork
DBmaestro
1. Federated2. Migration3. Auditing
Modernization
If you can’t satisfy the business demands then your process is broken.
What is the constraint in IT
top related