oracle rac presentation at oracle open world
DESCRIPTION
Presentation given with Imtiaz Mazhary and Bruce Carter at Oracle Open World 2007 while working at BA.TRANSCRIPT
Oracle on Linux at BA
From Evaluation To Implementation
15th November 2007
Imtiaz Mazhary Technical Architect
Paul Marden Project Manager
Oracle RAC – From Evaluation to Implementation
little teaser …SQL> create tablespace IMTEST_TS datafile '+DGTEST' size 10m;create tablespace IMTEST_TS datafile '+DGTEST' size 10m*ERROR at line 1:ORA-01119: error in creating database file '+DGTEST'ORA-17502: ksfdcre:4 Failed to create file +DGTESTORA-15001: diskgroup "DGTEST" does not exist or is not mounted
SQL> create diskgroup DGTEST 2 normal redundancy 3 failgroup DGTEST_BH 4 disk '/dev/raw/raw1001' name DGTEST_BH_DISK1001 5 failgroup DGTEST_CB 6 disk '/dev/raw/raw2001' name DGTEST_CB_DISK2001;
Diskgroup created.
Oracle RAC – From Evaluation to Implementation
little teaser …SQL> create tablespace IMTEST_TS datafile '_DGTEST' size 10m;
Tablespace created.
SQL> !ls -lart $ORACLE_HOME/dbs | grep DGTEST
-rw-rw---- 1 oracle oinstall 1056768 Sep 4 12:32 _DGTEST
Bug 6391521 RAC DATABASE DATAFILE CREATION ALLOWS SPECIFICATION OF NON-SHARED STORAGE
Oracle RAC – From Evaluation to Implementation
Oracle on Linux Evaluation
•Proposition•Cost reduction
•Improved resilience
•Simplification
•People•Infrastructure Design & Delivery
•DBA Teams
•Linux Infrastructure
•Oracle RAC Pack, IBM, Red Hat, Scaleabilities
Oracle RAC – From Evaluation to Implementation
Oracle on Linux Evaluation
•Platform Design Stage•Design Patterns
•Technology Stack
•Areas of Testing
•List of Issues
•Deliverables•BA, Oracle and IBM Validated Configuration
Oracle RAC – From Evaluation to Implementation
Design Patterns – Database Tier
DB01
Service X
SAN Storage
Basic Pattern
Application X
Oracle RAC – From Evaluation to Implementation
SAN Storage
Design Patterns – Database Tier
Application X
DB10R1
Service X
DB10R2
Service X
RAC Lite Pattern
Oracle RAC – From Evaluation to Implementation
SAN Storage
Design Patterns – Database Tier
DB10R1
Service X
DB10R2
RAC Lite Pattern for Consolidation
Application Y
Application X
Service Y Service X
Service Y
Service Y Service X
Oracle RAC – From Evaluation to Implementation
Design Patterns – Database Tier
Application X
DB20R1
Service X
DB20R4DB20R2
Service X
DB20R3
Service X Service X
Full RAC Pattern
SAN Storage
Oracle RAC – From Evaluation to Implementation
Technology Stack
IBM SDD v’s RHEL Device
Mapper
3rd Voting Disk on NFS
Oracle RAC – From Evaluation to Implementation
Areas of Testing
•Operating Environment• RHEL 4.5 – CFS, LVM2,
Device Mapper (and IBM SDD)
• SAN Attachment – QLogic vs RHEL QLogic driver
• RHEL 4.5 Cluster Suite
• BMC Patrol, EMC Legato Networker, IBM MQ Series, Control-M . . . .
•Oracle DB 10g R2 (142 tests)
• Serviceability – build, patching & configuration
• Clusterware resilience – interconnect, OCR & voting disks
• ASM configuration• ASM resilience & recovery• Stretched cluster performance• Client connectivity & failover• Backup & recovery, using EMC
Legato Networker• System Monitoring, using BMC
Patrol • EM Grid Control
Oracle RAC – From Evaluation to Implementation
`
List of key issues – all fixed in 10.2.0.3.0
CLSFMT error on CW install 5527179.993
Use of NFS for 3rd voting disk
Kernel panic caused by aio_put_req
5473171.993
885818 (RH)
Bug – RHEL libaio driver
Problems adding additional node
5449529.993
rootaddnode.sh using incorrect VIP
ASM block corruption 5556890.993
Bug - Block corruptions when dropping & adding failure groups
ASM & CW crashes/reboots on loss of one storage array
5594045.992
Bug - CSS error when processing stale member data requests
DBVerify Segmentation Fault
5682851.993
Bug – back port issue
Loss of one OCR causes CW errors
5593646.992
Feature - to prevent OCR data loss
OCR synchronisation behaviour 5621599.992
Similar to the above
Failure to replace OCR device 5698198.992
OCR storage too small when replacing the failed OCR disk
"read only filesystem" error 1768384 (RH)
Internal RAID array firmware issue?
Oracle RAC – From Evaluation to Implementation
Evaluation Project Deliverables
•Tested reusable patterns for use across the airline
•Supplier verification of our infrastructure - Oracle Validated Configuration undertaken by:
•IBM
•Oracle
•Red Hat
•OVC link: http://www.oracle.com/technology/tech/linux/validated-configurations/index.html
Oracle RAC – From Evaluation to Implementation
Database Consolidation
•Proposition •reduce the size of Oracle DB estate
•cost reduction / simplification
•first RAC Lite and Basic Pattern – learning exercise
•Process •LEAN - Runners / Repeaters
•pre-provisioning
•People - internal, RAC Pack
•IT - criteria, examples, experiences
FTD Batch(lxlhrl214)
OAW Batch(lxlhrl213)
Database Service - RAC Lite Pattern
BH CB
Batch/Feed Service (Linux HA)
Node X ConMan
Batch
Node Y
VIP X
RACDBR1
VIP Y
DGDATA_BHDGFRA_BH
Public Network
RAC Network
Storage Network
Database Consolidation - architecture
DGDATA_CHDGFRA_CH
Database Service Basic Pattern
DB1DBservice_ConMan
DBservice2DBservice3
….
DB2DBservice1DBservice2
….
ConManBatch
RACDBserviceFAST
RACDBserviceB
RACDBserviceC
RACDBserviceD
EXT3
RACDBR2
RACDBserviceC
RACDBserviceD
RACDBserviceFAST
RACDBserviceB
FASTBatch
FASTBatch
RACDBserviceFAST
RACDBserviceB
VIP X
Oracle RAC – From Evaluation to Implementation
Database Consolidation – Progress so far•Current Production Environment:
•30+ applications consolidated
•3 on RAC Lite
•Capacity provisioned for 20 more
•Experiences•Very few technical issues with infrastructure
•Mainly classic problems associated with migration
Oracle RAC – From Evaluation to Implementation
Database Consolidation - CriteriaRunner Repeater Stranger
Pattern type Basic or RAC Lite pattern
Basic or RAC Lite Pattern
Full RAC pattern or other
Hardware IBM xSeries
(BA standard model)
IBM xSeries
(BA standard model)
IBM xSeries
(standard BA standard model) or other Intel platform
Operating platform RHEL4 64 bit RHEL4 64 bit RHEL4 64 bit or other
RH release
Database Oracle 10g DB or
Oracle 10g RAC
Oracle 10g DB or
Oracle 10g RAC
Oracle 10g RAC
Data size (storage constraint) 60GB or less Greater than 60GB Any
Memory (SGA + User) 2GB or less 4GB or less Any
Overall server capacity
(cpu + memory constraint)
Less than 25% Less than 50% Any
Concurrent clients (cpu constraint)
10 or less Less than 50 Any
Business transactions Less than 3,600 per hour
Greater than 3,600 per hour
Any
Transaction type OLTP OLTP OLTP or Warehouse / DSS
3rd party supplier contract does preclude database consolidation
Yes Yes or No No
Oracle RAC – From Evaluation to Implementation
DW to Full RAC Pattern•Proposition
•first Full RAC implementation – learning exercise•important but not key business component
•Process - LEAN -- Strangers
•People - internal, NIIT, RAC Pack, Scaleabilities
•IT•Where are we coming from?•What does the cluster look like?•Gotchas / lessons learnt?
Oracle RAC – From Evaluation to Implementation
Legacy platform
•Old SUN ES6500, 16*400MHz CPUs, 20GB RAM
•Solaris 8, Oracle 9.2
•1.6 TB Database …. Operational Activity Warehouse & FTD
•300+ batch feed jobs
•Oracle Forms 6!
•MQ Series licence costs for 16 CPUs
•Severe performance problems
•No HA
FTD Batch(lxlhrl214)
OAW Batch(lxlhrl213)
FICO DCS StaffAdmin Baggage ….. …..
WA21R Database - Full RAC Pattern
VIP1
BH CB
Batch/Feed Service (Linux HA)
Node X
OAW Batch(lxlhrl213)
FTD Batch (A)
Node Y
FTD Batch(lxlhrl214)
OAW Batch (A)
VIP2
DGDATA_CBDGFRA_CB
DGDATA_BHDGFRA_BH
Public Network
RAC Network
Storage Network
Target Platform
WA21R1
WAPRD_BATCH
WAPRD_APP
FDPRD_APP
WA21R2
FDPRD_APP
WAPRD_APP
WAPRD_BATCH
VIP3
WA21R3
WAPRD_APP
FDPRD_APP
FDPRD_BATCH
VIP4
WA21R4
FDPRD_BATCH
FDPRD_APP
WAPRD_APP
Oracle RAC – From Evaluation to Implementation
DW to Full RAC Pattern
•Proposition•first Full RAC implementation – learning exercise•important but not key business component
•Process - LEAN -- Strangers
•People - internal, NIIT, RAC Pack, Scaleabilities
•IT•Where are we coming from?•What does the cluster look like?•Gotchas / lessons learnt?
Oracle RAC – From Evaluation to Implementation
Roadmap
•More consolidation on RAC•100 databases in FY0708
•Key Business Systems move to RAC•Orders Data Store Q1-2008
•ba.com FY0809
•SAP for BA Engineering ?
•Mainframe like availability•More about people than technology
•Fast Recovery – Oracle Data Guard
Oracle RAC – From Evaluation to Implementation
Contact Details
•Imtiaz Mazhary•Technical Architect, Infrastructure Design
•Paul Marden•Project Manager, Infrastructure Delivery