03 exadata architecture overview

AgendaExadata Software ArchitectureExadata Storage LayoutExadata Storage Scale-Out Architecture 2008 Oracle Corporation Proprietary and Confidential *

2009 Oracle Corporation Proprietary and Confidential *

Exadata Software Architecture 2009 Oracle Corporation Proprietary and Confidential *


2008 Oracle Corporation Proprietary and Confidential * Exadata Architecture


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewExadata Storage Server Network FabricEvaluate Disk, CPU, and Network Throughput Standalone


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server Overview Network FabricDatabase ServerRDBMS/ASM instance SGAIO ClientIO LayerASM layer?How do we connect the IO/ASM layer of ASM/DB to Exadata??Exadata Storage Server


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewDatabase ServerRDBMS/ASM instance SGA Network FabriccellsrvlibcellIO ClientIO LayerASM layerlibcell linked with DB/ASM talks to cellsrviDB protocol is bornMultiple threads in cellsrvThreads perform asynchronous IO to disks and networkExadata Storage Server


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewDatabase ServerRDBMS/ASM instance SGA Network FabriclibcellIO ClientIO LayerASM layerSmart Scan enabled in RDBMS and CELLSRVExadata Storage Servercellsrv


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewDatabase ServerRDBMS/ASM instance SGA Network Fabriccellinit.ora Local IP/etc/oracle/cell/network-configlibcellIO ClientIO LayerASM layercellinit.ora decides which network takes Storage trafficExadata Storage Servercellsrv


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewDatabase ServerRDBMS/ASM instance SGA Network Fabriccellinit.oracellip.ora Local IP CellslibcellIO ClientIO LayerASM layercellip.ora on database/ASM host maintains list of cellsNew cells can be added to cellip.ora dynamicallyExadata Storage Servercellsrv/etc/oracle/cell/network-config


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewDatabase ServerRDBMS/ASM instance dskmdiskmonSGA Network Fabriccellinit.oracellip.ora Local IP CellslibcellIO ClientIO LayerASM layerMaster diskmon (diskmon) starts with CSS and communicates with cellsrvSlave diskmon (dskm) is part of every instance and communicates with master diskmonHandles cell failures, IO fencing, IO resource management plan propagationExadata Storage Servercellsrv/etc/oracle/cell/network-config


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewDatabase ServerRDBMS/ASM instance dskmdiskmonSGA Network Fabriccellinit.oracellip.ora Local IP CellsmscellclilibcellIO ClientIO LayerASM layerCellcli allows user interaction and configurationManagement Server (MS) displays and manages creation and deletion of griddisks, changes in hardware, SNMP traps, alerts, email, metrics etcExadata Storage Servercellsrv/etc/oracle/cell/network-config


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewDatabase ServerRDBMS/ASM instance dskmdiskmonSGA Network Fabriccellinit.oracellip.ora Local IP CellsrsmscellclilibcellIO ClientIO LayerASM layerRestart Server (RS) monitors CELLSRV and MS, Backup RS monitors core RSRS monitors for process aliveness, memory usage etcExadata Storage Servercellsrv/etc/oracle/cell/network-config


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewDatabase ServerRDBMS/ASM instance dskmdiskmonSGA Network Fabriccellinit.oracellip.ora Local IP Cellsrsmscellcli ADRadrcilibcellIO ClientIO LayerASM layerTrace files and alert logs in Automatic Diagnostic Repository on cellalert.log (from RS and CELLSRV), ms-odl.log, ms-odl.trc, rs*trc, svtrc*.trcExadata Storage Servercellsrv/etc/oracle/cell/network-config


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewDatabase ServerRDBMS/ASM instance dskmdiskmonSGAInfiniband Fabriccellinit.oracellip.ora Local IP Exadata cellsrsmscellcli ADRadrcilibcellIO ClientIO LayerASM layerInfiniband fabric requires RPMs from OFED/OEL, RedHat 5.1, OEL 5.1Exadata Storage Server works only with 11.1.0.7 Database/ASMExadata Storage Servercellsrv/etc/oracle/cell/network-config


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Server OverviewFrom the EnterpriseNetwork Fabricrsmscellcli ADRadrciEM Plugin Provides central location for metrics and alerts across cellsNo agent runs on celldcli allows user to run commands across cellsEnterprise ManagerOMSAgentssh/cellclidcliExadata Storage Servercellsrv


Exadata Storage Layout 2009 Oracle Corporation Proprietary and Confidential *


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage LayoutCell DiskSys AreaSys AreaGrid Disk nGrid Disk 1ASM diskASM diskPhysical disks map to a Cell DisksCell Disks partitioned into one or multiple Grid DisksASM diskgroups created from Grid DisksTransparent above the ASM layerPhysical Disk


2008 Oracle Corporation Proprietary and Confidential * Exadata CellExadata CellExadata Storage Layout ExampleCell DisksCell Disk is the entity that represents a physical disk residing within a Exadata Storage CellAutomatically discovered and activated


2008 Oracle Corporation Proprietary and Confidential * Exadata CellExadata CellExadata Storage Layout Example Grid DisksCell Disks are logically partitioned into Grid DisksGrid Disk is the entity allocated to ASM as an ASM diskMinimum of one Grid Disk per Cell DiskCan be used to allocate hot, warm and cold regions of a Cell Disk or to separate databases sharing Exadata Cells


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Layout Example ASM Disk Groups and MirroringTwo ASM disk groups definedOne for the active, or hot portion, of the database and a second for the cold or inactive portionASM striping evenly distributes I/O across the disk groupASM mirroring is used protect against disk failuresOptional for one or both disk groupsExadata CellExadata Cell


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Layout Example ASM Mirroring and Failure GroupsASM mirroring is used protect against disk failuresASM failure groups are used to protect against cell failuresExadata CellExadata CellASMDisk Group


Exadata Scale-Out Architecture 2009 Oracle Corporation Proprietary and Confidential *


2008 Oracle Corporation * Scale-Out ArchitectureThe database, ASM, and Exadata Cells each play a role in Oracles scale-out storage architectureResponsibilities are placed in the optimal locationDB Business Data ProtectionASM Reliable Storage PoolExadata Cell Database Intelligent StorageSeamless integration simplifies managementASMReliable Storage PoolBusinessData ProtectionExadataDatabase Intelligent Storage


2008 Oracle Corporation * Exadata Scale-Out OverviewCells implement Oracles iDB data protocolExpose storage as collections of intelligent network disks called Grid Disks to ASM and the DB iDB is conceptually similar to iSCSI but has extensive DB intelligenceiDB is layered on top of ZDP network protocolTo ensure full scale-out benefits, cells never communicate with each otherCross cell operations are implemented in ASM or DB Cell independence ensures no performance bottlenecks, and no cascading failuresKey to scalability architectureGrid DisksiDB data protocolZDP net protocol


2008 Oracle Corporation * Exadata Cell Integration for Scale-OutCell appliance design eliminates storage configuration missteps and administrative overheadSimple provisioningCell grid disks are automatically made visible to ASMNo OS level LUNs or mount points to setup and manageCross cell ASM mirroring is automatically configured for grid disks ASM failure group topologyMultiple Grid disks per physical disk allow multiple ASMs to use the same cells


2008 Oracle Corporation * Automatic Storage ManagementASM creates a reliable storage pool from Exadata Grid disks Integrated cluster volume manager Flexible data distribution (striping) Mirroring Automatic data re-balancing Free


2008 Oracle Corporation * ASM Scale-out Data DistributionASM manages storage in megabyte allocation unitsEach DB file consists of a set of allocation unitsThe location of a files allocation units are individually tracked by ASMASM evenly spreads allocation units across all cells and disks in the gridAll disks evenly utilizedOptimal performanceDBFile #1DBFile #2Files automatically spread across disks to balance loadASM


2008 Oracle Corporation * ASM Data Re-DistributionCells can be easily added or removed from the storage gridASM migrates a fraction of the allocation units to the new cellOnline and transparent to applicationMinimal data movement to new or removed cellsDBFile #1DBFile #2ASMOnline Provisioning and Capacity ManagementNew Cell


2008 Oracle Corporation * Protection from Hardware FailureASM implements mirroring at the allocation unit levelPrimary and mirror copy of allocation units are placed on separate storage cellsAutomatically remirrors across all remaining cells when a disk or array fails

Failure of disk or array is transparent to databaseDBFile #1DBFile #2ASMDesigned to Tolerate Failures of Cells and Disks


2008 Oracle Corporation * Protection from BrownoutASM temporarily freezes I/O to unresponsive cells Redirect read IOs to mirror copyTrack write IOs that are missedWrites replayed when cell begins respondingFast mirror resyncNo need to remirror all the unchanged data BenefitsFast recovery from transient failurese.g Cell crash or temporary hangCan be used for planned maintenanceCell software or component upgradeDBFile #1DBFile #2ASMFreeze I/O to CellDesigned to Gracefully Tolerate Cell Brownouts


2008 Oracle Corporation Proprietary and Confidential * Data Protection SolutionsAll single points of failure eliminated by the Exadata Storage architectureHardware Assisted Resilient Data (HARD) built in to Exadata StoragePrevent data corruption before it happensData Guard provides disaster protection and data corruption protection Automatically maintained second copy of databaseFlashback provides human error protectionSnapshot-like capabilities to rewind database to before errorRecovery Manager (RMAN) provide backup to diskArchiving and corruption protectionCan be used with Oracle Secure Backup (OSB) or third party tape backup software These work just as they do for traditional non-Exadata storageUsers and database administrator use familiar tools


2008 Oracle Corporation Proprietary and Confidential * Exadata Storage Management & AdministrationEnterprise Manager Manage & administer Database and ASM Exadata Storage Plug-inEnterprise Manager Grid Control Plug-in to monitor & manage Exadata Storage CellsComprehensive CLILocal Exadata Storage cell management Distributed shell utility to execute CLI across multiple cellsIntegrated Lights Out Management - ILOMRemote management and administration of hardware


2008 Oracle Corporation Proprietary and Confidential * AQ&


***The 11g Database Server and the Exadata Storage Servers communicate with a new iDB (Intelligent Database Protocol). iDB is implemented in the database kernel and transparently maps database operations to Exadata-enhanced operations. iDB is built on the InfiniBand. ZDP (Zero-loss Zero-copy Datagram Protocol), a zero-copy implementation of RDS (Reliable Datagram Sockets v3), and is used to eliminate unnecessary copying of blocks. Multiple network interfaces can be used on the database servers and Exadata cells.Automatic Storage Management (ASM) is the storage management foundation of Exadata. ASM virtualizes the storage resources and provides the advanced volume management and file system capabilities of Exadata. The Database Resource Manager (DBRM) feature in Oracle Database 11g has been enhanced for use with Exadata. DBRM lets the user define and manage intra and inter-database I/O bandwidth in addition to CPU, undo, degree of parallelism, active sessions, and the other resources it manages. Like any storage device the Exadata Storage Server is a computer with CPUs, memory, a bus, disks, NICs, and the other components normally found in a server. It also runs an operating system (OS), which in the case of Exadata is Oracle Enterprise Linux (OEL) 5.1. The Oracle Storage Server Software resident in the Exadata cell runs under OEL. OEL is accessible in a restricted mode to administer and manage the Exadata cell. CELLSRV (Cell Services) is the primary component of the Exadata software running in the cell and provides the majority of Exadata storage services. CELLSRV is multi-threaded software that communicates with the database instance on the database server, and serves blocks to databases based on the iDB protocol. It provides the advanced SQL offload capabilities, serves Oracle blocks when SQL offload processing is not possible, and implements the DBRM I/O resource management functionality to meter out I/O bandwidth to the various databases and consumer groups issuing I/O. Two other components of Oracle software running in the cell are the Management Server (MS) and Restart Server (RS). The MS is the primary interface to administer, manage and query the status of the Exadata cell. It works in cooperation with the Exadata cell command line interface (CLI) and EM Exadata plug-in, and provides standalone Exadata cell management and configuration. For example, from the cell, CLI commands are issued to configure storage, query I/O statistics, and restart the cell. Also supplied is a distributed CLI so that commands can be sent to multiple cells to ease management across cells. Restart Server (RS) ensures the ongoing functioning of the Exadata software and services. It is used to update the Exadata software. It also ensures storage services are started and running, and services are restarted when required.

*************A Cell Disk is the virtual representation of the physical disk, minus the System Area LUN (if present), and is one of the key disk objects the administrator manages within an Exadata cell. A Cell Disk is represented by a single LUN, which is created and managed automatically by the Exadata software when the physical disk is discovered. On the first two disks, approximately 13GB of space is used for the system area. On the the other 10 disks, the system area is approximately 50MB.Cell Disks can be further virtualized into one or more Grid Disks. Grid Disks are the disk entity assigned to ASM, as ASM disks, to manage on behalf of the database for user data. The simplest case is when a single Grid Disk takes up the entire Cell But it is also possible to partition a Cell Disk into multiple Grid Disk slices. Placing multiple Grid Disks on a Cell Disk allows the administrator to segregate the storage into pools with different performance or availability requirements. Grid Disk slices can be used to allocate hot, warm and cold regions of a Cell Disk, or to separate databases sharing Exadata disks. For example a Cell Disk could be partitioned such that one Grid Disk resides on the higher performing portion of the physical disk and is configured to be triple mirrored, while a second Grid Disk resides on the lower performing portion of the disk and is used for archive or backup data, without any mirroring. Using ASM, you create Diskgroups from the Grid Disks and from that point on the Exadata Storage is transparent to the rest of the database and applications.

***In this example, two ASM disk groups are defined, one across the hot grid disks, and a second across the cold grid disks. All of the hot grid disks are placed into one ASM disk group and all the cold grid disks are placed in a separate disk group. When the data is loaded into the database, ASM will evenly distribute the data and I/O across the disk groups. ASM mirroring can be activated for these disk groups to protect against disk failures for both, either, or neither of the disk groups. Mirroring can be turned on or off independently for each of the disk groups. Exadata *For each disk group, ASM automatically creates a failure group for each Exadata Storage Server, containing the Grid Disks that belong to that server.

ASM then mirrors the data such that the mirror copies are on a different failure group and hence a different Exadata Storage Server. That way, ASM is able to protect the database from disk failure and the failure of an Exadata Storage Server.*****Allocation units may also become larger for extremely large files****Exadata has been designed to incorporate the same standards of high availability (HA) customers have come to expect for Oracle products. With Exadata, all database features and tools work just as they do with traditional non-Exadata storage. Users and database administrators will use familiar tools and be able to leverage their existing Oracle database knowledge and procedures. With the Exadata architecture, all single points of failure are eliminated. Familiar features such as mirroring, fault isolation, and protection against drive and cell failure have been incorporated into Exadata to ensure continual availability and protection of data. Other features to ensure high availability within the Exadata Storage Server are described below

Oracle's Hardware Assisted Resilient Data (HARD) Initiative is a comprehensive program designed to prevent data corruptions before they happen. Data corruptions are very rare, but when they happen, they can have a catastrophic effect on a database, and therefore a business. Exadata has enhanced HARD functionality embedded in it to provide even higher levels of protection and end-to-end data validation for your data. Exadata performs extensive validation of the data stored in it including checksums, block locations, magic numbers, head and tail checks, alignment errors, etc. Implementing these data validation algorithms within Exadata will prevent corrupted data from being written to permanent storage. Furthermore, these checks and protections are provided without the manual steps required when using HARD with conventional storage. *Exadata also has been integrated with the Oracle Enterprise Manager (EM) Grid Control to easily monitor the Exadata environment. By installing an Exadata plug-in to the existing EM system, statistics and activity on the Exadata Storage Server can be monitored and events and alerts can be sent to the administrator. The advantages of integrating the EM system with Exadata include: Monitoring Oracle Exadata storage Gathering storage configuration and performance information Raising alerts and warnings based on thresholds set Providing rich out-of-box metrics and reports based on historical data All the functions users have come to expect from the Oracle Enterprise Manager work along with Exadata. By using the EM interface, users can easily manage the Exadata environment along with other Oracle database environments traditionally used with the Enterprise Manager. DBAs can use the familiar EM interface to view reports to determine the health of the Exadata system, and manage the configurations of the Exadata storage. Exadata Storage Servers provide a comprehensive Command Line Interface (CLI) to configure, monitor, and administer the server. In addition, a distributed version of the CLI utility is provided so that commands can be sent to multiple servers to ease the management of mulitple servers.Each Exadata Storage Server has a remote management Lights-Out 100 card to perform remote hardware administration tasks, like power cycling the servers.*

03 exadata architecture overview

Documents