ukoug, instant cloning: boosting application development
DESCRIPTION
TRANSCRIPT
60% Projects Over Schedule
85% delayed waiting for data
CIO Magazine Survey:
Data is the problem
Get the right dataTo the right peopleAt the right time
Problem in IT
The Phoenix Project
One of the most powerful things that IT can do is get environments to development and QA when they need it
• IT bottlenecks• Setting Priorities• Company Goals• Defining Metrics• Fast Iterations
IT version of
“The Goal”
by E. Goldratt
Production
Development
Business Intelligence
BackupData Guard
QA
UAT
Tape
Sandbox
Forensics
Analytics
Reporting
Data Flood
Part II : Solutions: Thin cloning to Databases Virtualization
Part I : Problem: Copying Databases
Part III : Project Acceleration with Virtualization
Database Cloning Challenge
If you can’t satisfy the business demands then your process is broken.
Problem
Developers
QA and UAT
Reports
First copy
Production
• CERN - European Organization for Nuclear Research
• 145 TB database• 75 TB growth each year• Dozens of developers want copies.
Tradeoff: Speed, Quality, Cost
What We’ve Seen
1. Inefficient QA: Higher costs of QA2. QA Delays : Greater re-work of code3. Sharing DB Environments : Bottlenecks4. Using DB Subsets: More bugs in Prod5. Slow Environment Builds: Delays
“if you can't measure it you can’t manage it”
1. Inefficient QA: Long Build times
Build Time
QA Test
96% of QA time was building environment$.04/$1.00 actual testing vs. setup
Build
2. QA Delays: bugs found late require more code re-work
Build QA Env QA Build QA Env QA
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
1 2 3 4 5 6 70
10203040506070
Delay in Fixing the bug
Cost ToCorrect
Software Engineering Economics – Barry Boehm (1981)
3. Full Copy Shared : Bottlenecks
Frustration Waiting
Old Unrepresentative Data
http://martinfowler.com/bliki/NoDBA.html
4. Subsets : cause bugs
Production4. Subsets : cause bugs
Classic problem is that queries that run fast on subsets hit the wall in production.
Developers are unable to test against all data
The Production ‘Wall’
5. Slow Environment Builds: 3-6 Months to Deliver Data
Management
DBA
System Admin
Storage Admin
Developers Submit Request
Disk Capacity?
Approve Request $$ (2 Weeks)
Approve Request $$
(1 Week)
RequestAdditional Storage?
ProvisionCapacity
File SystemConfigured?
Configure LUNS & Build File System
Coordinate Replication w/ Infrastructure
Re-Parameterize & Configure DB
Mount Recovery DB to
Specific PIT
Begin Work
Approve Request $$ (2 Weeks)
(3 Days)
(3 Days)
(2 Days)
(3 Days)
(3 Days)
…….1-2 Weeks of Approvals, Delays, and Provisioning……
16
5. Slow Environment Builds: culture of no
DBA Developer
Never enough environments
bottlenecks
What We’ve Seen
1. Inefficient QA: Higher costs2. QA Delays : Increased re-work3. Sharing DB : Bottlenecks4. Subset DB : Bugs5. Slow Environment Builds: Delays
Clone 1 Clone 3Clone 2
99% of blocks are identical
Clone 1 Clone 2 Clone 3
Thin Clone
I. EMC II. Netapp III. ZFS
2. Thin Cloning
Database Luns
Production FilerTarget A
Target B
Target C
snapshotclones
1. Put database in hot backup 2. Take Snapshot (limits EMC 16, Netapp ,VNX 255, ZFS unlimted)3. Take database out of hot backup4. Clone Snapshot (Netapp , ZFS, EMC VNX)5. Export Clone, Mount on target host
InstanceInstance
InstanceInstance
InstanceInstance
InstanceInstance
Instance
Source
Database LUNs
snapshotclonesProduction Filer
Development Filer
Problem: How do you get data off Production?
Instance
Target A
Target B
Target C
InstanceInstance
InstanceInstance
InstanceInstance
Instance
Three Core Parts
Production
File System
Instance
DevelopmentStorage
21 3
Copy & sync
Snapshot & Clone
Mount, recover, rename
Instance
Netapp EM snap clone
Three Core Parts
Production
File System
Instance
DevelopmentStorage
1
Copy & sync
Instance
Snap Manager
SnapManagerRepository
Protection Manager
Snap Drive
Snap Manager
Snap Mirror
Flex Clone
RMANRepository
Production
Development
DBA
Storage Admin
1 tr-3761.pdf
Netapp
NetApp Filer - DevelopmentNetApp Filer - Production
Database Luns
Snap mirror
Snapshot Manager for Oracle
Flexclone
Repository Database
SnapDrive
Protection Manage
Production
Development
1NetappTarget A
Target B
Target C
InstanceInstance
InstanceInstance
InstanceInstance
Instance
Three Core Parts
Production
File System
Instance
DevelopmentStorage
Instance
3
Mount, recover, rename
3
Oracle EM 12c Snap Clone3
EM 12c
Agents instance
• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones• Set up a pool – a pool is a set of machines where databases can be provisioned• Set up a profile – a source database that can be used for thin cloning• Set up a service template – init.ora values
Test Master
Instance
Source
? instance
CloneLinux
ZFS orNetApp
PoolProfile
ZoneTemplate
Production Development QA UAT
Instance Instance Instance InstanceInstance Instance Instance Instance
Database
File systemFile system
Database
File systemFile system
Database
File system
Database
File systemFile systemFile systemFile system
Where we are
Where we want to be
Database
File system
Production
Instance
Database
Development
Instance
Database
QA
Instance
Database
UAT
Instance
Snapshots
Instance Instance Instance Instance
EM 12c: Snap Clone
Production Development
Flexclone Flexclone
Netapp Snap Manager for Oracle
Thin Cloning
3. Database Virtualization
Three Physical CopiesThree Virtual Copies
Data Virtualization Appliance
Install Delphix on x86 hardware
Intel hardware
Allocate Any Storage to Delphix
Allocate StorageAny type
Pure Storage + DelphixBetter Performance for 1/10 the cost
One time backup of source database
Database
Production
File system
File systemUpcoming
Supports
InstanceInstanceInstance
DxFS (Delphix) Compress Data
Database
Production
Data is compressed typically 1/3 size
File system
InstanceInstanceInstance
Incremental forever change collection
Database
Production
File system
Changes
• Collected incrementally forever• Old data purged
File system Time Window
Production
InstanceInstanceInstance
Source Full Copy Source backup from SCN 1
Snapshot 1Snapshot 2
Snapshot 1Snapshot 2
Backup from SCN
Snapshot 1Snapshot 2
Snapshot 3
DropSnapshot 1
Snapshot 2Snapshot 3
Cloning
Database
Production
Instance
File systemFile system Time Window
Database
InstanceInstance
InstanceInstance
Typical Architecture
Production Development QA UAT
Instance Instance Instance InstanceInstance Instance Instance Instance
Database
File systemFile system
Database
File systemFile system
Database
File system
Database
File systemFile systemFile systemFile system
With Delphix
Database
Production
Instance
Database
Development
Instance
Database
QA
Instance
Database
UAT
InstanceInstance Instance Instance Instance
File system
Three Core Parts
Production
File System
Instance
DevelopmentStorage
21 3
Copy Sync SnapshotsTime FlowPurge
Clone (snapshot)CompressShare CacheStorage Agnostic
Mount, recover, renameSelf Service, Roles & Security Rollback & Refresh Branch & Tag
Instance
Overview of Database Virtualization
1. Fast, Fresh, Full2. Free3. Branching4. Federated5. Self Serve
Fast, Fresh, Full
Instance
Time Window
Instance
Development VDB
Source
Free
gif by Steve Karam
Instance
Instance
Instance
Instance
Source
Branching
Instance Instance
Instance
Source
Source Dev
QA branched from Dev
Federated Cloning
Federated
Instance
Instance
Instance
Instance
Source1
Source2
Source1
“I looked like a hero”Tony Young, CIO Informatica
Self Service
Use Cases
1. Development Acceleration
2. Quality
3. BI
DevOps
DevOps With Delphix
1. Efficient QA: Low cost, high utilization2. Quick QA : Fast Bug Fix3. Every Dev gets DB: Parallelized Dev4. Full DB : Less Bugs5. Fast Builds: Culture of Yes
1. Efficient QA: Lower cost
Build Time
QA Test
1% of QA time was building environment$.99/$1.00 actual testing vs. setup
Build Time
QA Test
Build
Rapid QA via Branching
2. QA Immediate: bugs found fast and fixed
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
QA QA
Build QA Env QA Build QA Env Q
A
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
3. Private Copies: Parallelize
gif by Steve Karam
4. Full Size DB : Eliminate bugs
Production
5. Self Service: Fast, Efficient. Culture of Yes!
Management
DBA
System Admin
Storage Admin
Developers Submit Request
Disk Capacity?
Approve Request $$ (2 Weeks)
Approve Request $$
(1 Week)
RequestAdditional Storage?
ProvisionCapacity
File SystemConfigured?
Configure LUNS & Build File System
Coordinate Replication w/ Infrastructure
Re-Parameterize & Configure DB
Mount Recovery DB to
Specific PIT
Begin Work
Approve Request $$ (2 Weeks)
(3 Days)
(3 Days)
(2 Days)
(3 Days)
(3 Days)
…….1-2 Weeks of Approvals, Delays, and Provisioning……
Quality
1. Forensics2. Testing 3. Recovery
1. Forensics: Investigate Production Bugs
Instance
Time Window
Instance
Development
Anomaly on ProdPossible code bugAt noon yesterday
Spin up VDB of Prod as it was during anomaly
2. Testing : Rewind for patch and QA testing
Instance
Time Window
Instance
Development
Time Window
Prod
2. Testing: A/B
Instance
Time Window
Instance
Instance
• Keep tests for compare• Production vs Virtual
– invisible index on Prod– Creating index on virtual
• Flashback vs Virtual
Test A with Index 1
Test B with Index 2
3. Recovery: Surgical recover of Production
Instance Instance
Development
Time Window
Spin VDB up Before drop
Problem on ProdDropped Table Accidently
Source
Time Window
3. Recovery Surgical or Full Recovery on VDB
Instance
Instance
Dev1 VDB
Time Window
Dev1 VDB
InstanceSource
Source
Dev2 VDB Branched
3. Recovery: Virtual to Physical
Instance Instance
VDB
Source
Time Window
Spin VDB up Before drop
Corruption
3. Recovery
Business Intelligence
ETL and Refresh Windows
1pm 10pm 8am noon
ETL and DW refreshes taking longer
1pm 10pm 8am noon20112012201320142015
ETL and Refresh Windows
20112012201320142015
1pm 10pm 8am noon
10pm 8am noon 9pm
6am 8am 10pm
ETL and DW Refreshes
Instance
Prod
Instance
DW & BI
Data Guard – requires full refresh if usedActive Data Guard – read only, most reports don’t work
Fast Refreshes
• Collect only Changes• Refresh in minutes
Instance Instance
DW
Prod
Instance
BI
ETL24x7
Temporal Data
BI
a) Fast refreshes
b) Temporal queries
c) Confidence testing
Review: Use Cases
1. Development Accelerationa) Full, Fresh, Fast , Self Serveb) QA Branchingc) Federated
2. Qualitya) Forensicsb) Testing : A/B, upgrade, patchc) Recovery: logical, physical
3. BIa) Fast refreshb) Temporal Datac) Confidence testing
Oracle 12c
80MB buffer cache ?
200GBCache
5000
Tnxs
/ m
inLa
tenc
y
300 ms
1 5 10 20 30 60 100 200
with
1 5 10 20 30 60 100 200Users
8000
Tnxs
/ m
inLa
tenc
y
600 ms
1 5 10 20 30 60 100 200Users
1 5 10 20 30 60 100 200
$1,000,000
$6,000
About Delphix• Founded in 2008, launched in 2010• CEO Jedidiah Yueh (founder of Avamar: >$1B revenue))• Based in Silicon Valley, Global Operations
Fortune #1 Walmart #1 social: Facebook #1 US Bank: Wells Fargo #1 Networking : Cisco#1 cable provider: Comcast #1 auction site: Ebay #1 insurance: New York Life #1 chip manufacture: Intel
Devv2.6 v2.6v2.6
QA UAT
v2.6
v2.6 v2.6v2.6v2.7
v2.6 v2.6v2.6v2.8
v2.6v2.6 v2.6v2.6
v2.6v2.7 v2.6v2.7
v2.6v2.8 v2.6v2.8
Devv2.6 v2.6v2.6
QA UAT
v2.6Production
v2.6 v2.6v2.6v2.7
v2.6 v2.6v2.6v2.8
Source Control for the database data
v2.6v2.6 v2.6v2.6
v2.6v2.7 v2.6v2.7
v2.6v2.8 v2.6v2.8
DevProd
2.6
Dev
QA
Prod
2.6
Dev
QA
UAT
Prod
2.6
Dev
QA
UAT
Prod
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT
Prod
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT2.8
Dev
QA
UAT
Prod
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT2.8
Data Control = Source Control for the Database
Dev
QA
UAT
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT
2.8
Data Control = Source Control for the Database
Production Time Flow
Database Virtualization
$40M
$75M
$850M
$27,000M
Storage
IT
Develop
Business
Good, Cheap, Fast : choose two
Fast
GoodCheap
FS vs. ZFS
• FS per Volume
• FS limited bandwidth
• Storage stranded
• Many FS in a pool
• Grow automatically
• All bandwidth
Storage PoolVolume
FS
Volume
FS
Volume
FS ZFS ZFS ZFS
# of database copies
IncreasedRevenueFor Company
Three Core Parts
Production
Instance
Time Window
Instance
Self Service
Development
21 3
Source Syncing Storage (DxFS)
Database going Global
Globalization Reduces Windows
20112012201320142015
1pm 10pm 8am noon
10pm 8am noon 9pm
6am 8am 10pm
I. Clonedb Oracle II. EMC
• Copy on first write (COFW)III. Netapp
• write anywhere file system (WAFL)• & EMC VNX redirect on write (ROW)
IV. ZFS
2. Thin Cloning
RMAN backup
dNFSsparse fileI. clonedb
RMAN backup
dNFSsparse fileI. clonedb
CloneDB
1. dNFS 11.2.0.2+– cd $ORACLE_HOME/rdbms/lib– make -f ins_rdbms.mk dnfs_on
2. Clonedb.pl initSOURCE.ora output.sql– MASTER_COPY_DIR="/rman_backup”– CLONE_FILE_CREATE_DEST="/nfs_mount”– CLONEDB_NAME="clone"
3. sqlplus / as sysdba @output.sql– startup nomount PFILE=initclone.ora – Create control file backup location– dbms_dnfs.clonedb_renamefile ('/backup/file.dbf' , '/clone/file.dbf');– alter database open resetlogs;
Tim Hallwww.oracle-base.com/articles/11g/clonedb-11gr2.php
I. Clonedb Oracle II. EMC
• Copy on first write (COFW)III. Netapp
• write anywhere file system (WAFL)• & EMC VNX redirect on write (ROW)
IV. ZFS
Thin Cloning
D
ActiveFile
SystemSnapshot
CBA
II. EMC Copy on Write
D
ActiveFile
SystemSnapshot
DCBA
Write penalty (read and two writes)Limit 16 snapshotsNo Branching (snapshots of snapshots)
II. EMC Copy on Write
I. Clonedb Oracle II. EMC
• Copy on first write (COFW)III. Netapp
• write anywhere file system (WAFL)• & EMC VNX redirect on write (ROW)
IV. ZFS
Data Blocks
root
III. Netapp (WAFL) and EMC (ROW)
• 255 snapshots• Branching possible
I. Clonedb Oracle II. EMC
• Copy on first write (COFW)III. Netapp
• write anywhere file system (WAFL)• EMC redirect on write (ROW)
IV. ZFS
Snapshot rootLive root
ZilIntent Log
IV. ZFS Allocate on Write
Unlimited Instantaneous SnapshotsUnlimited Instantaneous ClonesBranching easy and unlimited
Oracle EM 12c Snap Clone3
Test Master
Instance
Source
? instance
Create a “Test Master” databaseOn ZFSSA or Netapp
ZFSSA or Netapp
Oracle EM 12c Snap Clone3
EM 12c
• Register Netapp or ZFS with Storage Credentials
Test Master
Instance
Source
? instance
ZFS orNetApp
Install EM 12cInstall Snap Clone option
Oracle EM 12c Snap Clone3
EM 12c
Agents
• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage.
Test Master
Instance
Source
? instance
Linux
ZFS orNetApp
Oracle EM 12c Snap Clone3
EM 12c
Agents
• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage• Register test master database• .
Test Master
Instance
Source
? instance
Linux
ZFS orNetApp
Oracle EM 12c Snap Clone3
EM 12c
Agents
• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage• Register test master database• Enable Snap Clone for the test master database
• .
Test Master
Instance
Source
? instance
Linux
ZFS orNetApp
Enable Snap Clone
Oracle EM 12c Snap Clone3
EM 12c
Agents
• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones
Test Master
Instance
Source
? instance
Linux
ZFS orNetApp
Zone
CPU limitsMemory limits
Oracle EM 12c Snap Clone3
EM 12c
Agents instance
• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones• Set up a pool – a pool is a set of machines where databases can be provisioned
Test Master
Instance
Source
? instance
CloneLinux
ZFS orNetApp
Pool
Zone
Oracle EM 12c Snap Clone3
EM 12c
Agents instance
• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones• Set up a pool – a pool is a set of machines where databases can be provisioned• Set up a profile – a source database that can be used for thin cloning
Test Master
Instance
Source
? instance
CloneLinux
ZFS orNetApp
PoolProfile
Zone
Oracle EM 12c Snap Clone3
EM 12c
Agents instance
• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones• Set up a pool – a pool is a set of machines where databases can be provisioned• Set up a profile – a source database that can be used for thin cloning• Set up a service template – init.ora values
Test Master
Instance
Source
? instance
CloneLinux
ZFS orNetApp
PoolProfile
ZoneTemplate
Choose your virtualization Layer:• Delphix and Oracle SMU
SMU
ZFS Storage Appliance
Oracle 12c SMUOracle Snap Management Utility for ZFS Appliance
• Requires ZFS Appliance• Supports Linux , Solaris 10+, Windows
2008+• GUI
– snapshot source databases – provision virtual databases
over 10 times
perhaps the single largest storage consolidation opportunity history“
Original Snapshot Snapshot Day 2