ilm on oracle10gr2

8/8/2019 ILM on Oracle10gR2

1/61

Information Lifecycle

Management(ILM)

for Business Data

October 2006


2/61

ILM for Business Data

This talk focuses on ILM for Business Data

Financial Data

Customer DataProduct DataPeople Data


3/61

What Drives ILM?

Reduce cost to retain data

Vast amounts of data are retained enterprisesfor business and regulatory reasons

Need to optimize the cost of retaining data in thedatabase to avoid skyrocketing costs

Active Data Less Active Data Historical Data


4/61

Database Implementation without

ILM

DIGITAL DATA STORAGE

High PerformanceStorage Tier

ActiveLessActive Historical Archive

Data Lifecycle

TapeArchive


5/61

Match Lifecycle to Storage to

Optimise Cost


Low CostStorage Tier

Online ArchiveStorage Tier

Active LessActive

HistoricalOfflineArchive

Data Lifecycle


OfflineArchive


6/61

1990 2000 2004

European Data Privacy

Directive All companies doing businessin Europe handling PII

HIPAA Healthcare Insurance All U.S. businesses

handling medicalrecords

Gramm Leach Bliley

Banks and financialservices companies doingbusiness in U.S.

Business

esImpacted

Documentation Regulations

Pharmaceuticals Aerospace/Defense Petrochemical Telcos

DoD 5015.2 (Standard) U.S. DoD U.S Federal Agencies Business dealing with U.S.

Federal Agencies

UK/PRO (Standard)

All companies doingbusiness in UK publicsector

AUS/PRO (Standard)

All companies doing business inAUS public sector

Sarbanes Oxley

All U.S. publiccompanies and private

foreign issuersSEC Rule 17a-3/4

All U.S. companiesengaged in broker-dealer activities

CA Breach Law

All U.S.companies doingbusiness in CAhandling PII

NASD 3010, 3110,NYSE 342

All member companies

Key Idea Compliance

Must guarantee compliance to governmentand business mandates

Intelligent centralized dataplatform allows you to defineand enforce compliancepolicies


7/61

ILM and Oracle

From database perspective ILM is a set ofpolicies and techniques for Managing Data

Managing data is Oracles core competency

Oracle platform can be used to implement ILMpolicies and techniques for business data

Some new technology but mostly an application ofexisting data management capabilities


8/61

Storage

Database

Oracle is Ideal for Business ILM

UnderstandsBusiness Data

Hardware

Independent

Central Point of

Control Customizable

Applications

Interfaces


9/61

Oracle is Ideal for Business ILM

ActiveLessActive

Historical

Oracle Database is the best place toimplement business ILM

Application Transparent ILM

Oracle classifies business datatransparently to the applications

Fine Grained ILM

Oracle manages the lifecycle of

groups of business data down to thelevel of individual rows

Low Cost ILM

Oracle can use low cost storage to

reduce the cost of retaining data

Enforceable Compliance Policies

Oracle has sophisticated techniques todefine and enforce data policies


10/61

Implementing Business ILM

using theOracle Database Platform


11/61

4 Steps to Business ILM




1. Define DataClasses

2. Create StorageTiers for theData Classes

3. Create DataAccess andMigrationPolicies

ActiveLessActive

Historical

4. Define andEnforceCompliancePolicies

OfflineArchive



12/61

Step 1 to Business ILM

1. Define DataClasses


13/61

Define Classes of Data First understand data as part

of a business process How is it used?

How long must it be kept? How does access vary over

time?

Choose classification based

on this understanding Classification by Age is mostcommon

Others are possible

Privacy Product ID

Consider a Hybrid Classification

Classify by business attribute

AND age

A

C

TI

V

I

T

Y

D

A

T

A

V

O

L

U

M

E

0 1 1 5 10

Months Years

Active

LowVolume LessActive

HighVolume


14/61

Separate Data by Class

The goal of ILM is to apply different policies todifferent classes of data

To treat data classes differently, you mustphysically separate data by class

Data classes must be mapped to dataattributes e.g. Order date

Table Partitions enable you to separate databy data attribute

Can manage each class (partition) as a unit

Store, move, archive, search, query Partitions are transparent to the application

Use ILM Assistant to define a Lifecycle

All Orders

Q1Orders

Q2Orders

Q3

Orders

Q4Orders

PreviousOrders


15/61


2. Create Storage Tiersfor the Data Classes

High Performance

Storage Tier

Low Cost

Storage Tier

Online Archive

Storage Tier

Offline

Archive



16/61

Create Physical Storage Tiers



Create separate storage areas for high

performance and low cost storage High Performance Storage Tier uses

High performance storage arrays

Disks optimized for throughput

Low Cost Storage Tier uses

Modular arrays for reduced cost

Large capacity commodity ATA disks


17/61

Online Archive Storage Tier

The Online Archive Storage Tier is Very large

Low Activity Read-only or Read-mostly

Store in low cost storage tier Information is still online and always readable

No delay when data needed, always available Storage cost is almost same as tape

Leverage usage patterns to furtherreduce size & cost Defragment and Compress

Rows, tables, files

Declare read-only


18/61

Assign Classes to Storage Tiers

Assign datapartitions toappropriate

storage tiers

All Orders

Q1

Orders

Q2Orders

Q3Orders

Q4Orders


Low Cost Storage Tier

Online Archive Storage Tier

PreviousOrders

Active Data

LessActive Data

Historical

Data


19/61

Storage Tier VendorStorage Tier Vendor min $/GB max$/GBmin $/GB max$/GB

EMC DMX 1000 73GB,


20/61

Storage Tier Single TierStorage Tier Single Tier Multiple Tiers w/CompressionMultiple Tiers w/Compression

High Performance (2550 GB)High Performance (2550 GB) $74,300.00$74,300.00

High Performance (50 GB)High Performance (50 GB) $1,450.00$1,450.00 $1,450.00$1,450.00

$3,500.00$3,500.00 $3,500.00$3,500.00

$5,600.00$5,600.00

$10,550.00$10,550.00

$14,000.00$14,000.00

$18,950.00$18,950.00

Low Cost (500 GB)Low Cost (500 GB)

Online Archive (2000 GB)Online Archive (2000 GB)

$74,300.00$74,300.00

Usage of appropriate storage tiers reduces total costof ownership by order of magnitudes

Compression reduces TCO even more

Full application transparency

Use ILM Assistant to view and project cost savings

Tiered Storage Sample Costs


21/61


ActiveLessActive Historical

3. Create Data Accessand MigrationPolicies




OfflineArchive



22/61

Define Access Policies

Access Policies determinedata visibility

Only authorized data Only recent data

Special users or operationsaccess historical data Hiding historical data speeds up

data scans and maintenance

Implement Access Policies

using Views and VirtualPrivate Database Transparent to the application

SpecialUsers

AuthorizedUsers



Access

Policy



23/61

Migrate Data between Classes

Periodically move data betweenstorage tiers as access patterns

change e.g. MOVE PARTITION holding Q2

Orders from high performance storagetier to low cost storage tier

Move important data on demand UPDATE of partition key will cause row to

move to a new partition e.g. product warranty expires

ILM Assistant will advise when timeto move data


Low Cost

Storage Tier

Q2Orders


24/61

Step 4 to Business ILM4. Define and

EnforceCompliancePolicies




OfflineArchive


C t li d C li


25/61

Centralized Compliance

Enforcement It is impractical to manage policy and ensure

compliance on decentralized and fragmented data

Cannot trust everyone to know the rules, andabide by them

Oracle Database Platform provides an intelligentcentral location to store data and enforce datapolicies for compliance

Policies are customizable for businesses specificneeds

Centralized policy definition and enforcement

Centralized access control using Enterprise Users

Flexibility to respond to changing regulations

El t f C li


26/61

Elements of Compliance

Policy Retention

Ensure data is retained unmodified for a specific time period

Immutability

Prove to external party that data is complete and unmodified

Privacy Protect personal and other sensitive data

Auditing

Track and report changes to important data

Expiration

Expunge stale data to limit liability


27/61

Data Retention

Business and regulatory needs dictate

specific retention periods Oracle security can enforce that users cancreate specific kinds of data, but not deleteor modify the data

Privileges to create, modify, and delete canbe independently granted to specific usersfor specific tables

More granular control than file systems

Data can be periodically purged using

privileged procedures Changes in privileges can be audited

Centrally administered

Use ILM Assistant to describe a Lifecycle

Sarbanes-OxleyHIPAAEuropean DataPrivacy Directive

UK PRODOD5015.2-STD


28/61

Immutability

Security enforcement prevents unauthorized changes, butsometimes you must prove that enforcement has not been

bypassed, even by order of the CEO Must prove no data tampering at database level or below whether

intentional or accidental

Oracle can generate a digital signature to prove that data has not

not been modified Signature can be generated on the result of any query

e.g. all the orders booked last quarter

Signature will change if even a single bit of the data is changed

Signature can be time stamped and stored, or given to third party Prevents covering up illegal or unethical activity

ILM Assistant can generate & compare


29/61

Privacy

Oracle provides many methods to ensure data privacy

Virtual Private Database (VPD)

With "Column Relevance", VPD can be configured such thatthe policy is enforced only when a critical column is selected(e.g. salary or credit limit)

Centrally managed control of data access by user


30/61

Auditing

Oracle can audit any change or access to data in the database

Common problem is that auditing every access produces a floodof audit records

Fine Grained Auditing (FGA) specifies the conditions necessaryfor an audit record to be generated

Create relevant and meaningful audit trails FGA policies are bound to a table or view

Audit when a specific column has been selected or updated

E.g. any access to credit card, salary, etc.

Audit changes to closed transactions

Detect changes to financial books that have been closed

Audit actions by administrators or highly privileged users


31/61

Expiration Data eventually expires

No longer needed for legal or businessreasons

Becomes a potential liability

Permanently Remove Expired Data Periodically the partition containing

expired data can be dropped DROP PARTITION

Files containing partition can then beexpunged

Backups of partition can beindependently expunged



32/61

Additional Features for

Implementing ILM

Automatic Storage Management


33/61

Automatic Storage Management

ASM enables the use of inexpensivemodular storage for low cost storage tier

Resiliency

ASM mirrors data across inexpensive modularstorage arrays

Performance ASM Stripes data to ensure no hot spots

Availability ASM Allows online addition and removal of disks

Disk Group

ASM Disk Groups per Storage Tier &


34/61

ASM Disk Groups per Storage Tier &

Partitions

Disk Group L Disk Group HDisk Group P

High Performance

Storage Tier


CurrentMonth

Last 11months

Year2002

Year2001

Year2000

Each Tier uses ASM for load balancingwithin the tier

Partitions are in different disk groups

Data is moved between disk groups using Partition Move Operation, or

Online Reorganization of tables, or

Tablespace Copy followed by rename

Years1995-

1999

Online Archive Tier


35/61

Prevent Data Changes

Sometimes Data reaches a point when itmust never be changed e.g. the electronic

record of a bank cheque

Data can be placed in Read-OnlyTablespace which means:

Backup Only needs to be backed up once

Read-only Media

Can place data on a CDROM or

WORM Deferred Read-Only mode does not

open files unless they are actuallyaccessed

Read

Only


36/61

Table Level ILM ILM can also be done at

the table level

Measure the I/O rate and

I/O rate per gigabyte ofdatabase tables Use v$segment_statistics

Tables with low I/O densitycan go on slower and lowercost storage Use alter table move or

online redefinition to movethe table to a differentstorage tier



High I/OTables

Low I/OTables


37/61

Moving Non-Partitioned Data

Move files to a storage deviceusing datafile copy

ALTER TABLESPACERENAME DATAFILE

Move tables or partitions usingOnline Redefinition

PackageDBMS_REDEFINITION

Moving Data to a New


38/61

Moving Data to a New

External HomeData is often moved from one database to another e.g. move orders

over 12 months old from OLTP system to Data Warehouse Data can be relocated using

Oracle Streams

Changes made to the OLTP databaseare replicated in the data warehouse

Transportable Tablespaces

Move the entire partition to anotherdatabase e.g. all of this weeks orders

Data Pump Fast unload and load of data in a file

Oracle Warehouse Builder Build custom application to move data

Reducing Storage


39/61

Reducing Storage

Requirements

Remove Unused Space Online Segment Shrink

Removes unused Space within a segment

Move Partition or Table to a new tablespace toeliminate all intra or inter segment unused

space Table Compression

Use when data is no longer changing tocompress row data

Summarize (Materialized View) Long-term a summarized value may be

sufficient rather than each individual record

With so much data being retained, can it be more efficiently stored andmanaged by reducing its storage requirements

S i & P i D


40/61

Securing & Protecting Data

Oracle Database 10gprotects against:

Unauthorised access

Hardware failures Human errors and software failure

Data Corruption

Complete Site Failure

Protect from Human Error using Flashback


41/61

Protect from Human Error using Flashback

A new strategy for point in time recovery

Flashback Log captures old versions of

changed blocks Think of it as a continuous backup

Replay log to restore DB to time

Restores just changed blocks

Its fast - recover in minutes, not hours

Its easy - single command restore

Flashback Database to 2:05 PM

Rewind button for the DatabaseData Files FlashbackLog

New BlockVersion

Disk Write

Corruption Protection using Flash


42/61

Corruption Protection using FlashRecovery Area

On-disk Backups usingRMAN are an important partof ILM

Fully automatic disk based

backup and recovery Nightly incremental backup

rolls forward recovery areabackup

Changed blocks are trackedin production DB

Full scan is never needed Dramatically faster (20x)

Blocks validated to preventcorruption of backup copy

Use low cost ATA disk arrayfor recovery area

Data on all

Storage Tiers

Flash

RecoveryArea

NightlyIncremental

Applied andValidated

WeeklyTape

Archive

Database

Area

Low-CostStorage Tier

OfflineArchive

Tier


Site Failure Protection Data Guard


43/61

Site Failure Protection Data Guard

Data Guard ensures Business Continuity thru standby databases

Physical (identical copy) Logical

Transfer processing to the standby sites for routine maintenance or when adisaster occurs

ProductionDatabase

StandbyDatabase

TransactionShipping

(Real Time Apply)

NoDelay

Highest Data Protection Using


44/61

Low Cost Storage

Combine theFeatures to

Achieve Any Levelof Data Protection

ASM MirroringStorage Failure

Protection

Flashback

Human ErrorProtection

Data GuardSite FailureProtection

FlashRecovery

AreaCorruption

Protection

ILM & Oracle RAC


45/61

& O ac e CReal Applications Clusters

Run All Your Applications

Scalable

Highly Available

No Single Point of Failure

Add Low Cost Capacity on

Demand Servers or Storage

Manageable

Secure

Network

Drive and ExploitIndustry Advances inClustering

Users

No SinglePoint Of Failure

Interconnect

Storage Area Network

SharedCacheSharedCache

CentralizedManagement Console


46/61

What Happens to Data at

Lifecycle End

Data Archiving


47/61

Data Archiving

What happens to data when it is unlikely to be

needed again?

Archive to Tape

Archive to a Database

Efficiently Remove Archive Data using Drop

Partition



48/61


Online Database Archive

Central Archive for all systems

No delay when data needed, alwaysavailable

Storage cost is almost same as tape

Only backup when data is added Always Readable

No impact on Production system

Use Oracle Warehouse Builder tocreate custom loading

MainArchive

Database

Archive to Tape


49/61

Archive to Tape

Archive expired data in DatabaseFormat for rapid unload

Transportable Tablespaces

Archive Oracle data files directly

Reload files if they are later needed

Data Pump export format Uses Oracle native data types for

high performance

Archive using XML functionality builtinto the Database


ExpiredData

OfflineArchive



50/61

ILM Assistant

Oracle ILM Assistant


51/61

Oracle ILM Assistant

Manages your ILMenvironment via a GUIinterface Define Lifecycle Definitions Illustrates Storage Costs &

Savings

Manage Security & Compliance

Calendar of Events

Simulates impact of partitioningon a table

Advises how to Partition a Table

When data needs to be moved Generate Scripts to move

data

Tool downloaded from OTN

Requirements Oracle APEX

Oracle 9i or greater

Cost Savings


52/61

Cost Savings by Storage Tier


53/61

Cost Savings by Storage Tier

Lifecycle Events


54/61

Lifecycle Events

Security & Compliance


55/61

Security & Compliance


56/61

Summary

Partitioning for ILMCustomer Success: NASDAQ


57/61

Customer Success: NASDAQ

NasdaqNasdaq implemented partitioning to enable informationimplemented partitioning to enable information

lifecycle management, supporting twolifecycle management, supporting two--tier storage thattier storage thatmoves agedmoves aged--out data to less expensive storage quicklyout data to less expensive storage quicklyand easily while keeping recent data fresh and upand easily while keeping recent data fresh and up--toto--

date with better performancedate with better performance

Saved millions of dollars in storage costs and eliminatedSaved millions of dollars in storage costs and eliminated

the need to buy additional hardwarethe need to buy additional hardware

http://www.oracle.com/customers/snapshots/http://www.oracle.com/customers/snapshots/nasdaqnasdaq..pdfpdf

ILM and Oracle


58/61

ILM and Oracle

ILM is a data management strategy designed to

Reduce Cost of Retaining Data

Comply with Legal, Regulatory, and Business Mandates

From the database perspective, ILM is a set ofpolicies and techniques for Managing Data

Managing data is Oracles core competency

Oracle platform can be used to implement ILMpolicies and techniques for business data

Some new technology but mostly an application of existingdata management capabilities

Benefits of Oracle for ILM


59/61

Benefits of Oracle for ILM

Performance

Security

Safety

Consistency

Flexibility

Hardware

LongevitySimplicity

Open

- Fastest and most functional access to data

- Retained data uses same security as current data

- Full protection from corruptions, errors, disasters

- Data is transactionally consistent

- Easily adapts to changing requirements

- Total hardware independence

- Oracle Databases will be supported for decades- No specialized data stores to manage

- Standard SQL interfaces

Over 25 years of investment in Data Management

Conclusion Oracle Database for ILMI l t li i i


60/61

Active LessActive Historical


HighPerformanceStorage Tier

Financial DataCustomer DataProduct Data


OfflineArchive


Implements policies in a

application transparent fashion Stores maximum data for

lowest cost

Centralizes compliance

enforcement ILM Assistant can model future

environment

Understanding of data and

knowledge of access patternmakes it the best location toimplement business ILM

Further Information


61/61

OTN

For white papers, presentations, eSeminar

http://www.oracle.com/technology/deploy/ilm/index.html

ilm on oracle10gr2

Documents