oracle database 10g extensibility framework · oracle database 10g extensibility framework:...

Oracle Database 10gExtensibility Framework:Building, Deploying, and Managing Data Cartridges

Geoff LeePrincipal Product Manager,Oracle CorporationGeoff.Lee@oracle.com

Agenda

OverviewOracle Database 10g Extensibility FrameworkDemoSummaryQ & A

Specialty Data in Life SciencesGenomic Proteomic Chemical

Virtual ScreeningMicroarrays

Managing All Your Data

DynamicWeb PagesDynamic

Web Pages E-MailE-Mail

StructuredData

StructuredData

Consistent, universal accessEasy to search and find any contentEasy to develop applications Easy to deploy with enterpriseQuality of ServiceEasy to manage

LegacyData

LegacyDataDocumentsDocuments

VideoVideoVideo LocationsLocationsTextText

StaticWeb Pages

StaticWeb Pages

AudioAudio ImagesImagesImages

Oracle10gOracle10g

Oracle Extensible Platform

Optimizer

Query Engine

Index EngineType Manager

Extensibility

Text

TextSpatialUtilities Tools

Image

Image

Text

Spatial

Image

Data CartridgesSpatIal

Specialty Data Cartridges

Oracle8iServer

Service Interfaces

DataCartridge

Extensibility Interfaces

TypeSystem

QueryProcessing Data

IndexingServer

Execution . . .

Database Extensibility Services

Oracle10gServer

Oracle8/8i

Evolution of the Extensibility Framework

Oracle9i/9i R2

Range PartitionedLocal Domain Indexes

ODCIEnv TypeAlter IndextypeDBMS_ODCI PackagePipelined Parallel

Table FunctionsUser-Defined AggregatesGeneric SQL TypesMulti-Threaded Extproc

Parallel Domain Indexe Scan DBMS_STATS integration

Array InsertsAlter OperatorDebug Info (29891)Trans. Tablespace

for Domain IndexesParallel Index Create/

Index Rebuild

Extensible Typesystem

Extensible ServerExecution Env.

Extensible IndexingExtensible Optimizer

Agenda

OverviewOracle Database 10g Extensibility Framework

– Building a data cartridge– Deploying a data cartridge– Managing data cartridges

DemoSummaryQ & A

Building Data Cartridges

Designing the schema (data types, LOBs, etc.)User defined types, operators, indexes, and optimizer

– Using the right languages (pl/sql, Java, C/C++)Boosting performance

– Array insert– Parallel pipelined table functions– Parallel (inter-partition) local domain indexes

Create, alter rebuild, queryUser defined aggregatesDebugging trace events

Key Steps to Building a Data Cartridge

1. User-Defined Types2. User-Defined Operators3. User-Defined Indexes4. Extensible Optimizer

User-Defined Types (Step 1 of 4)

Type System– Object Types (inheritance, attributes, methods, etc.)– Methods (dynamic method dispatch, PL/SQL, Java, or C/C++) – Object References (Object ID, dot notation)– Collection Types – LOBs

Physical Model– Column objects and row objects– Object views and object tables

Type evolution, replication, import/export, SQL*Loader, etc.Versatile PL/SQL, Java, C/C++, OLE DB and XML accessSQL 2003 standard

LOBs• LOBs help capture unstructured data• Three types of LOBS supported

• Binary LOB (BLOB)• Character LOB (CLOB, NCLOB)• Binary File (BFILE)

• LOBs replace LONG,LONG RAW• Multiple LOBs per table or UDT• Piece-wise access of LOB data• Streaming Interface• LONG to LOB APILOBs can be stored in a

separate tablespace or in operating system files

LOBs can be stored in a separate tablespace or in operating system files

Table

BFILE

BLOB/CLOB

LOB Performance Improvements

5x performance gain for accessing inline (< 4KB) LOBsTemporary LOBs use reference counting to provide orders of magnitude performance gain

– Reference on Read– Copy on Write

Temporary LOBs use aggressive garbage collection at row level

Native Floating-PointData Types

Two new numeric data types BINARY_FLOAT, BINARY_DOUBLE

– IEEE 754 Standard for binary floating point arithmetic

– Part of numerous other standards (e.g, Java, XML Schema) and hardware platforms

– Prevalent in Business Intelligence, Life Sciences, Engineering/Scientific Computation, etc.

User-defined Operators(Step 2 of 4)• Domain specific operation• Evaluated using functions or indexes

CREATE FUNCTION SssMatchBlob (a BLOB, b VARCHAR2)RETURN NUMBER AS …/CREATE OR REPLACE OPERATOR sssBINDING

(BLOB, VARCHAR2) RETURN NUMBERWITH INDEX CONTEXT, SCAN CONTEXT cdc_ixmdlUSING cdcsss.SssMatchBlob,

(BLOB, CLOB) RETURN NUMBERWITH INDEX CONTEXT, SCAN CONTEXT cdc_ixmdlUSING cdcsss.SssMatchBlob ;

User-Defined Indexes (Step 3 of 4)Registers a new indexing scheme to support a set of operatorsImplements domain indexing scheme as ODCIIndexinterface functions

CREATE OR REPLACE TYPE cdc_ixmdl AS OBJECT {STATIC FUNCTION ODCIIndexCreate(..) RETURN NUMBER,STATIC FUNCTION ODCIIndexInsert(..) RETURN NUMBER,STATIC FUNCTION ODCIIndexUpdate(..) RETURN NUMBER,… };

CREATE OR REPLACE INDEXTYPE ixmdl FORsss(BLOB, VARCHAR2), sss(BLOB, CLOB) USING cdc_ixmdl;

CREATE INDEX acd2d_mol_mdlix ON acd2d_mol(ctab)INDEXTYPE IS c$mdlimol2.ixmdlPARAMETERS ('RCG_DB = ACD2D');

Extensible Indexing Architecture

Logical index

Physical index table

UserApp.

User-definedindexes/stats

CREATE INDEX

SELECT

ODCIIndexCreateODCIIndexInsert, etc.

ODCIIndexStartODCIIndexFetchODCIIndexClose

DBMS_STATS ODCIStatsCollectDMLs

Simple QuerySELECT cdbregno as cdbregno,

chime_string(ctab) structure FROM acd2d_mol WHERE sss(ctab,’querystring’)=1;

Operation Options Object--------------------------- ---------------- --------------SELECT STATEMENT Cost=1229TABLE ACCESS BY INDEX ROWID ACD2D_MOL

DOMAIN INDEX ACD2D_MOL_MDLIX

SELECT cdbregno as cdbregno, chime_string(ctab) structure

FROM acd2d_mol WHERE sss(ctab,’querystring’)=1;

Operation Options Object--------------------------- ---------------- --------------SELECT STATEMENT Cost=1229TABLE ACCESS BY INDEX ROWID ACD2D_MOL

DOMAIN INDEX ACD2D_MOL_MDLIX

Extensible Optimizer (Step 4 of 4)SELECT /*+ ORDERED */ to_char(acd.cdbregno) as ACD_regno,

ch.ch_mdlnum as MDL_number, ch.ch_casrn as CAS_Number,ch.ch_prdcnt as Product_Count, chime_string(ctab) as structure

FROM acd991.ACD2D_moltableb acd, acd991.ACD2D_mol mol,acd991.chemical ch

WHERE sss(ctab,<Chime String>)=1 AND ch.ch_prdcnt > 1AND acd.cdbregno = mol.cdbregnoAND mol.mdlnumber = ch.ch_mdlnum

Aware of user-defined indexes User can specify cost & selectivityUser can collect domain specific statisticsSimple defaults or comprehensive functions

Cost-Based Optimizer

Query

Hints

Statistics

Execution Plan:join order,Join methods,access methods,…

C(P1)

C(P2)

C(P3)

C(Pn)

CBO (min)

System State

Estimating Costs

I/O cost: number of data blocks fetched– DBMS_STATS package

CPU cost: number of machine instructions

SQL> DECLARE2 a INTEGER;3 BEGIN4 a := DBMS_ODCI.ESTIMATE_CPU_UNITS(10);5 DBMS_OUTPUT.PUT_LINE

('CPU units = '|| a*1000);6 END;7 /

CPU units = 1198114000

Complex QuerySELECT /*+ ORDERED */ to_char(acd.cdbregno) as ACD_regno,

ch.ch_mdlnum as MDL_number, ch.ch_casrn as CAS_Number,ch.ch_prdcnt as Product_Count, chime_string(ctab) as structure

FROM acd991.ACD2D_moltableb acd, acd991.ACD2D_mol mol,acd991.chemical ch

WHERE sss(ctab,<Chime String>)=1 AND ch.ch_prdcnt > 1 AND acd.cdbregno = mol.cdbregno AND mol.mdlnumber = ch.ch_mdlnum

Query Plan-------------------------------------------------SELECT STATEMENT Cost = 11689

NESTED LOOPSNESTED LOOPS

TABLE ACCESS BY INDEX ROWID ACD2D_MOLTABLEBDOMAIN INDEX ACD2D_MOLTABLEB_MDLIX

TABLE ACCESS BY INDEX ROWID ACD2D_MOLINDEX RANGE SCAN IX_ACD2D_MOL

TABLE ACCESS BY INDEX ROWID CHEMICALINDEX UNIQUE SCAN CH_MDLNUM

Agenda

OverviewOracle Database 10g Extensibility Framework

– Building a data cartridge– Deploying a data cartridge– Managing data cartridges

DemoSummaryQ & A

Deploying

Cross platform transportable tablespacesCollecting statisticsRuntime architecture (J2EE, RAC, Grid, etc.)User privileges (invokers rights)Multi-threaded agents

Transportable TablespacesCopy database subsets (Tablespaces) between databases

– Operating system file copy for data– Managed transfer of metadata (indexes,

constraints, stats, etc.) between databases

– “Plug and go”

Introduced in Oracle8iEnhanced in Oracle9i to support different block sizesResult: extremely fast Result: extremely fast bulk data transport bulk data transport between databasesbetween databases

Oracle10g Cross Platform Transportable Tablespace

Solaris (SPARC)

Open VMS (Intel IA64)

Windows (Intel IA32/64)

Linux (Intel IA32/64)

AIX (PowerPC)

HP-UX (PA-RISC)

Tru64 UNIX (Alpha)

Little Endian Platforms

Simple File Copy Within These

HP-UX (Intel IA64)

Big Endian Platforms

Simple File Copy Within These

RMANHigh SpeedFile Convert

Automatic Statistics Gathering

DBMS_STATS.GATHER_SCHEMA_STATS(‘SCOTT’,options => ‘GATHER AUTO’)

This simple statement gathers statistics for the entire schema, and implicitly determines:

– what tables need to be sampled– how much sampling is needed– which columns need histograms– what degree of parallelism to use when sampling

Dynamic and Scalable Parallel Execution

SQL

querycoordinator

queryserver

queryserver

queryserver

queryserver

queryserver

queryserver

querycoordinator

results

• Query coordinator allocates query servers, and divides query into separate subtask

• Each “set” of query servers perform different task (e.g. scanning, sorting, joining)

• Results are “pipelined” from one set of query servers to the next

RAC Configuration

Order Entry Supply Chain

High SpeedInter-connects

High Speed Switches

Agenda

OverviewOracle Database 10g Extensibility Framework

– Building a data cartridge– Deploying a data cartridge– Managing data cartridges

DemoSummaryQ & A

Managing Data CartridgesData cartridge upgrades

– Alter type/operator/indextype/indexPerformance Tuning

– Current statistics, Explain plan, Trace event– Inlined LOBs, Persistent Data interface, chunk sizes, loading,

compressionData Management

– Partitioning, Import/export, SQL*Loader, data pump– ALL_TYPES, ALL_TYPE_ATTRS, ALL_TYPE_METHODS– ALL_OPERATORS, ALL_INDEXTYPES,

ALL_INDEXTYPE_OPERATORS, ALL_INDEXES views– ALL_IND_STATISTICS, ALL_TAB_STATISTICS,

ALL_TAB_COL_STATISTICSSpace Mgmt

– ASSM/Locally Managed Tablespaces– Inlined LOBs storage

Type Evolution

• Non-structural changes– adding, dropping or modifying

methodsStructural changes

– adding, dropping or modifying attributes

ALTER TYPE person_t

ADD/MODIFY/DROP attribute (ssn

VARCHAR2(11)) CASCADE;

Alter Operator

Add bindings to or drop bindings from an existing operator

ALTER OPERATOR Ordsys.Contains ADD BINDING (music.artist, music.artist) RETURN NUMBER USING music.contains_func;

Alter Indextype

Associate the added operator binding to the indextype

ALTER INDEXTYPE TextIndexType ADD lob_contains(CLOB, CLOB);

Alter IndexODCIIndexAlter( ia ODCIIndexInfo,

parms IN OUT VARCHAR2, alter_option NUMBER, env ODCIEnv) RETURN NUMBER

alter_option is one of the following options:• AlterIndexNone: ALTER INDEX [PARTITION] PARAMETERS• AlterIndexRename: ALTER INDEX RENAME [PARTITION]• AlterIndexRebuild: ALTER INDEX REBUILD [PARTITION]

[PARALLEL (DEGREE deg)] [PARAMETERS]• AlterIndexUpdBlockRefs: ALTER INDEX [schema.]index

UPDATE BLOCK REFERENCES

Agenda

OverviewOracle Database 10g Extensibility FrameworkDemoSummaryQ & A

D E M O N S T R A T I O N

A Data Cartridge in Action

Agenda

OverviewOracle Database 10g Extensibility FrameworkDemoSummaryQ & A

Summary

Manage All of Your Data with Oracle10g Database

– Consolidate all your data– Simplify application development, deployment,

and management– Explore versatile capabilities– Ensure RAS, manageability, and security

Managing All of Your Data with Oracle10g Database!

Q U E S T I O N SQ U E S T I O N SA N S W E R SA N S W E R S