DB2 and OracleInfrastructure Standardization(CA IDUG Award 2009)Peter SuhnerAXA Tech SwitzerlandSession Code: D09Nov 10, 2010, 09:45 – 10:45 | Platform: DB2 for Linux, UNIX, Windows

This presentation gives an overview of how we have standardized our Unix-based database environments. Oracle and DB2 are set up, maintained and monitored in a similar way.

This allows us to reduce the operations efforts as well as the specialist knowledge required to operate the environments and to resolve everyday problems.

1

Let's start with a picture of the award AXA Tech have gained from their approach towards the implementation of a database environment which standardizes and simplifies maintenance and administration of both, DB2 LUW and Oracle.

2

Takeaways

• Cost reduction on database infrastructure and operations

• What it takes to set up and maintain Oracle and DB2 databases in an identical manner

• Unix scripting good practices• Server virtualization – what's in it for us?• Project and daily operations practical experience

These are the bullet points I intended to cover before writing the presentation and this is what you can directly take with you.

However, it wouldn't be too helpful to build the presentation along these bullet points. Therefore the agenda looks a bit different.

3

Agenda

• Where (and why) it all started• New, standardized platform• Setup of DB2 and Oracle• Standards, standards, standards• Before and after – a comparison• Considerations

We'll first look at WHY we implemented this standardization project.

An overview of the new platform, which is standardized across DBMS products will solve the question of WHAT we did.

Then let's see what things look like in such an environment.

Finally sum up with a comparison and a few important considerations.

4

We all know this…

• …type of "DB(2) Everyplace"• Historically grown / Result of mergers• Placed somewhere on a server• Even under a user's desk• Set up individually• Badly maintained (if at all…)• Existence not even known

Yes, there is a "DB2 Everyplace" – marketed by IBM for "pervasive computing".

But, don't we all know this OTHER type of "DB(2) Everyplace"? Which is also available with Oracle and any other DBMS?

The one of which we just realize one day that it exists at all? Not because it runs, but because it has failed for the first time…

Very often, company mergers lead to comparable scenarios, because infrastructures are not aligned.

We all have come across such situations, usually in our own company, right?

And the less controlled an environment is (or the less money is available), the more often such situations occur.

5

What about…

• Regular Backups• Recovery from failures• Security issues• Fixpack and version maintenance• Licensing• Availability, Performance, SLA• ...

Nobody ever thought about securing the most precious asset (apart from people): company data!• Do we have backups?• If so, can we recover from them? Quick enough to be in time?• Are we on a supported level of all underlying software?• Do we have any security vulnerabilities?• Are we correctly licensed?• Do we even know about customer requirements towards availability, performance, recoverability, etc.?

NO??? How extraordinarily astonishing!

6

What about…

• Regular Backups• Recovery from failures• Security issues• Fixpack and version maintenance• Licensing• Availability, Performance, SLA• ...

As database professionals, we know that such scenarios are time bombs…

7

Our former environments

Oracle

$$$DB2

What we actually had in our company, was two completely disparate environments. One for Oracle, and a completely separate one for DB2.

While Oracle was our time bomb environment, DB2 was in a better shape – but extremely expensive. The next slide gives you more details why.

8

Our former environments…

$$$Oracle DB2

• Server/OS hosted• DB Ops internal

• Contractors ($$)• Little control

• Server/OS hosted• DB Ops by provider

• Expensive• No control

All of our Unix servers were hosted by a professional provider. This included hardware (Servers, SAN, etc.), network and operations of the core operating system.

As IBM was our provider, this Unix service was only a side branch of what they did for us on the Mainframe side. Even though the service was quite basic, it generated huge cash-out.

Oracle databases were not included in this hosting contract (for whatever reason…). As a result, the growing number of Oracle databases had to be maintained and operated by skilled staff. Which in turn brought in a number of external Oracle specialists:• Quite skilled = very expensive• No internal specialists = no control over what they did• They brought their own tools and scripts, etc.• We became dependent on them

DB2 databases were operated by IBM as our standard provider. They did a fairly good job and provided a stable environment, but it costed us a fortune. And again, we had no control over what they did and how they operated the environment.

In short: We paid a lot of money for something over which we had little control and which seemed to provide limited reliability.

9

Is this where we

want to be????

Rhetorical question (at least I hope so)!

10

Truth

TechnologyBlueprints

Building Blocksand Standards

DB PlatformsImplementation

and Dream

So we faced our bitter truth of database individualism and allowed a dream to come to our minds:

What, if we had technology blueprints for our database environments?

Plus – as a next step – if we had concepts and standards for each one of our RDBMS products which would define how the technology blueprints should be implemented?

Wouldn't this allow us to build standardized, centralized DB towers on which we could run all of the databases in a similar way?

Would this sound feasible? And wouldn't this reduce the pressure we had on quality and cost?

11

Let's seewhat things look like

in our shop today…

We had started the standardization project in late 2005, but stumbled a bit in the beginning. This was mostly caused by too many people considering their own view as being important for the project. By early 2006, these things were sorted out. With the right people on board the project encountered smooth progress.

The next part of the presentation will show you what our environment looks like since we have standardized it.

12

Distributed Databases Landscape (DB2/Oracle)

• 350 Oracle (10g, 11g)• 60 DB2 LUW (9.1)• 6 DB2 Gateways (9.1)• DB2 Replication (9.1)

• z/OS (9) -> Oracle (10g, 11g)• Separate Federation Servers

• z/OS <-> DB2 (8.2, ~350 Servers on Windows)• Currently being upgraded to 9.7

Oracle is one of our strategic DB systems. DB2 is also, but only on the mainframe. As a result, distributed DB2 currently doesn't seem to have much of a future in our shop. But – as we all know – this can change again anytime.

Nevertheless, we still have about 60 databases in our standardized environment. And there's still some growth, not only in the size of the databases, but also in the number.

Apart from this, we run a couple of Gateways for Mainframe access.

Plus, we do DB2 SQL replication to Oracle, for which we have set up a set of DB2 Federation servers.

And then there's a different – but also standardized – DB2 environment for one single application. It involves about 350 DB2 servers on Windows, all of them replicating to and from the Mainframe. But this presentation will focus on the standardized Unix environment, where we run Oracle and DB2 in a similar way.

13

Life cycle

Our environment provides four major stages along the life cycle of a database.

• In a standard environment, production and acceptance servers are spread over different DCs on identical hardware, which provides us with spare capacity for DR case. At the same time, these environments provide HA features like log shipping, etc.

• SysTest systems are only in use where required by the applications, the same is true for Education systems

• Development and Education/SysTest systems are set up on smaller systems and also spread over DCs, they partly provide spare capacity in DR case

From development, databases are propagated to acceptance test and from there to production environments.

Systems (integration) testing can happen on versions which have left development status.

Education can take place on versions which have gone through acceptance testing.

Education and SysTest environments are dead end; there is no further propagation of such versions to other stages.

14

Standardized Environment

A schematic overview of our standard Sun hardware and setup environment shows:• Three Network interfaces (VRF layers not shown here)

• Secure Entity Network (SEN) provides Management, SAN and separate Public network layers

• All Storage SAN attached• Optical interfaces• Disks RAID5 and striped

• SAN mirroring across data centers for higher service classes• Physical boxes provide 1 .. n virtual Zones

• All zones are set up identically• Can easily be moved between servers with minimal downtime

• Specific 3rd party software to support this• Mount points for Oracle and DB2

• Set up identically• Identical names• Identical usage

15

Centralized DB Towers

• Unix based (Solaris in our shop)• SPARC scales well and linear

• Virtualization (Solaris Zones)• Optimize HW resources• Easily manageable

• Frameworks for server management• Zone moving• Monitoring, etc

The whole Unix-centric DB environment is built on Solaris.

Solaris was mainly chosen because:• their virtualization solution (Zones) is quite mature• SPARC architecture scales more linear than x86 does• Stable HW: Replacement does not directly affect the OS and infrastructure stack built above it• it's simply the best Unix dialect ;-)

For Server Management, we rely on 3rd party software, which allows us to quickly move zones from one server HW to another one. This is even true across the different HW types we have in use. As all storage is SAN attached, this allows us to:• optimize system load by moving zones around• perform quick recovery from most of the common disaster scenarios

For system monitoring, Zabbix (more Solaris-centric) and Oracle OEM (more DB centric) are in place.

16

OS Setup for DB Servers

• HW is standardized• All DB servers are virtualized zones

• Zone is for either DB2 or Oracle – no mixes• One single OS image for DB servers

• Identical Patch levels• Kernel parameters identical • Zones are capped

• Guarantee resources (SLA)• Licensing is CPU based

All DB servers are run on standardized, identically equipped hardware:• Sun M5000 for PROD and UAT (User Acceptance Testing)• Sun T5120 for DEV, SysTest, Education, Engineering and the like

DB servers are virtualized using Solaris virtualization technology (Zones). This has proven to be very stable and flexible technology, allowing for optimal availability and simple maintenance.Each of the DB zones is built for either DB2 or Oracle. We do not mix these products within a single zone. But of course we have physical servers which drive Oracle and DB2 zones.

Operating Systems are installed from a template which has been created for DB servers. All of them are identical, without any exceptions.

OS Patch levels are identical. Patching occurs during maintenance windows in a strictly scheduled plan.

Kernel parameters have been worked out to support DB2 and Oracle. They are identical for all DB servers.

To comply with service levels defined in our SLAs, the DB zones are capped. This allows at the same time to strictly adhere to our CPU based licensing models for DB2 and Oracle.

17

Directory structures

• All storage SAN attached• Identical for DB2 and Oracle

• /u00 -> DB2 or Oracle binaries• /u01 -> DB data• /u02 -> Log files (primary)• /u03 -> Log files (mirror)• /u04 -> Temp data• /u11 -> Dump files, Instance home

• Mount points standardized below these directories

All disks are SAN based. Mount points are identical for both, Oracle and DB2.

This slide shows the basic definitions below root directory, which is an adapted version of the Oracle OFA (Optimal Flexible Architecture) standard.

Underneath these basic directories, we have standardized the mount points for all Oracle and DB2 instances as follows:

/u[nn]/[db2|ora]data/[instance]/[db_name]

Example:/u01/db2data/diniunv/UDUNV -> mount point for DB2 database "UDUNV" within instance "diniunv"

18

More standards…

• Security• Oracle and DB2 admin users, instance user names• Logon via sudo (centrally maintained and distributed)

• Parameters (Instance and DB level)• Select either DWH, OLTP or General Purpose type

• Network• DNS names, IP Ports, etc.

• Monitoring• One single tool for OS, DB2 and Oracle

Apart from the OS level, there are more standards we adhere to. Prominent examples are:

SecurityAdministrative users are standardized across all DB servers and stages. They are defined locally on the servers, whereas personal accounts (standard plus privileged for DBAs) are maintained in LDAP.

Direct logon as an administrative user is not possible (passwords unknown). Instead, DBA accounts are allowed to perform "sudo su – [admin_user]" w/o password.

Sudoers files are maintained centrally and distributed to all virtual servers by the Unix crew.

ParametersInstance and DB parameters are standardized. We basically avoid individual adaptations for optimized performance and/or resource consumption as far as possible.Generally speaking, we provide three different types of databases a customer can choose from and they come with respectively adapted parameters.

NetworkOn a network level, we standardize IP ports and service names for an instance across stages. DNS names follow a standard naming scheme as well.

MonitoringAs DB2 LUW is not a strategic product in our shop, all DB monitoring is implemented with Oracle's OEM. It is definitely more important to have it all within one single tool than to provide (pseudo-)optimal, but different solutions for every product.OEM provides plug-in agents for DB2 monitoring as well as for the underlying OS (and many more products, by the way). While OEM is rather resource consuming, this still outweighs the disadvantages that would incur from using different monitoring tools.

19

DBA Monitoring

Monitors are a DBA's headlamp in the dark cave of a database environment. They are inevitable tools providing all necessary information for problem analysis and – even better – proactive avoidance of incidents.

As Oracle is the strategic RDBMs on distributed platforms in our shop (sad enough!), this was a simple decision:OEM (Oracle Enterprise Manager) provides not only monitoring for Oracle databases. This tool comes with agents for additional products which seamlessly integrate in the OEM interface. As usual, such agents are not free, but they're affordable. This allows us to gain a complete overview of DB2 and Oracle databases as well as the underlying operating system layer from within one single tool.

The screenshot provided here shows an overview of the complete database environment in the upper half of the screen. All databases are monitored, but grouped along the DBMS products and stages (e.g. DB2 production, Oracle development, etc.). Towards the right, major areas of monitoring are listed (e.g. CPU, memory, I/O, wait and response time, # connections per minute, etc.) and the basic status is shown as green, yellow and red dots.

Clicking on any of the database groups will open the next level of detail for this specific group. This allows for a comprehensive view at a glance as well as quick drill down to the source of incidents.

The lower half of the screen provides a list of current alerts – again marked in yellow and red according to their severity (green alerts do not exist - obviously).

All alerts must be acknowledged by the DBAs as part of the problem resolution. This ensures that all error situations are treated in a controlled manner.

20

Reporting

What more, OEM stores historical data within its own database. Statistical and SLA reporting are heavily simplified and standardized for the whole database environment. Our customers receive standardized reports and can even request their own, customized view into the environment.

Again, such reports look identical for both, DB2 and Oracle – providing the customer with what he really needs: a service oriented view of the database environment.

As you can see from the screenshot shown on this slide, hyperlinks underneath the charts link to more detailed information on the respective topics.

21

Monitoring Pros/Cons (DB2 view)

One single tool

Identical presentation

Customizable views

Historical reporting

Compliance reports

Oracle, Non-DB2

DB2 Health monitor on

Java agents

Agent stability

Monitoring of the whole environment using OEM is not really a standard for DB2 databases. On the other hand, it provides quite a bunch of advantages:

• Having the complete overview of databases and underlying infrastructure monitored within one single tool heavily reduces monitoring overhead• One single interface providing an identical and comprehensive view of key figures makes it much easier to understand what happens in the infrastructure• Views in OEM are customizable. Paired with a built-in, roles supporting security, it is quite easy to provide customers with their own individual overview of key figures for their specific environment• Monitoring data is stored in the OEM database and provides a comprehensive basis for historical reporting• SLA compliance reports can be generated directly out of the tool and presented to the customer. Again, the layout and presentation of the data is identical for DB2 and Oracle as well as for the OS.

From a DB2 point of view, OEM is not optimal. While it is a feasible solution for our environment, it might not be suitable for highly performance critical DB2 databases.• The fact that the tool is non-DB2 does not necessarily make you suffer, but DB2 people need to get used to it• Much worse is the fact that Health monitoring must be switched on in DB2. As you might be aware of, this can still cause performance bottlenecks even with DB2 9. For once, we can't blame Oracle for this – IBM should definitely try to improve their own product• What we can blame Oracle for is the way they have implemented their monitoring agents. The agent simply runs as a daemon and for every single poll of almost every single value, it spawns a JVM. The JVM creates it's own connection to the database, performs the respective poll, grabs the data and is destroyed again. We all know how expensive starting a JVM is – or non-pooled DB connections…• With heavy server load and longer run queues, we have encountered limited stability of the OEM agents for both, Oracle and DB2 databases. Often, the agent then reports to be up and running, but it is unreachable from the OEM database. As a result, monitoring data is missing, leaving the DBAs blind until the respective agent is restarted.

22

On top of this:

AXA Tech's DB Build

The next part of the presentation will show you what our environment looks like since we have standardized it. And what it is all about this "Database Build" stuff.

23

How would you restart an Oracle instance?

• Identical to a DB2 instance – execute the script…

• [d|o]bld_restart_instance.ksh• -i [instance_name]• -m [STOP|START|RESTART|…]• -w [wait_time]• -f(orce)

If you – as an experienced DB2 DBA – need to work on an Oracle database, you probably get stuck if you need to bring down the instance cleanly. Even more if you have to start it.

Task oriented scripts provide the desired functionality and behind the scenes, they ensure that the task is performed properly.

This scripted approach allows us cycle an instance on schedule (or only stop it and bring it back up later).

To add some more comfort, we can specify an amount of time during which the script will wait for running utilities to end.And with the force (-f) parameter, we can decide to either give up or force a STOP|RESTART if the database activity does not end within the specified grace time.

These scripts are implemented in an identical way for both, DB2 and Oracle. And therefore are simple to understand and very safe to execute.

As a result, we are able to automate the majority of our maintenance tasks and even execute them in unattended mode. Would the idea of unattended server patching, instance upgrade to new version or FixPack overnight or on weekends sound appealing to you?

24

And how do we REORG, add tablespaces, etc.?

• Execute the respective script…• Similar script names and parameters

• As far as applicable

• For Oracle and DB2• Unified output format• Identical file locations (log files, etc.)• Error codes• Identical alert mechanisms• …

We've implemented this idea as far as possible between DB2 and Oracle. Of course there are differences within the products. You will never have to define an additional database within an instance for Oracle because Oracle does not provide this functionality.

In return, we don't have scripts to define users in a DB2 database, right? Oracle does. And Oracle has got RMAN for backups. But apart from these (documented!) differences, we have achieved an acceptable similarity for the handling of Oracle and DB2 databases.

In addition, all scripts provide identical output and log file format, identical output directories and naming standards, error codes, alert mechanisms and the like.

25

More standards: You know one? – you know all!

• Check lists• For tasks without scripts• In unified format, containing four parts:

• Basic document information (version, etc.)• Task executor name and sign-off• List of all parameters and information required• Step-by-step task description with expected results

• Documentation and Processes!• Concepts, technical designs, implementation• Unified, identical or at least comparable

There are tasks which just don‘t make sense to be scripted, mostly because they are to be executed too seldomly or they are too individual and/or complex.

All of these tasks are described with detailed checklists, which provide a step-by-step installation guide. These checklists are commonly formatted, they all look similar and have the following characteristics:

• First part contains basic document information (versioning again, etc.)• Second part defines name of task executor and his sign-off• Third part is a list of all parameters which are referred to in the checklist (e.g. target server, instance name, etc.)

• Needs to be filled in before working through the checklist• Ensures that all information is available before starting to execute the task

• Fourth part is the step-by-step work description• provides commands to be executed• shows expected results• contains success|failed checkboxes for every step• contains space to describe what action was taken in case of failure

• either workaround or stop execution

A variety of additional documents (e.g. conceptual and technical documents, operations manual, etc.) are provided with this environment and – again – are written in a way that makes it easy to derive what it takes for DB2 if you are experienced with the Oracle environment and vice versa.

Processes are identical across platforms. Obviously we follow ITIL standards, with respective tool support.

26

A few more words on the Builds…

• Task oriented structure and names of scripts• Similar names, similar parameters for DB2 and Oracle

• Unix shell scripting (ksh)• Central script providing

• Standard procedures, functions, error handlers, etc.• Environment standards (paths, naming conventions,

…)• UDAL (Unix Dialect Abstraction Layer)

• Dialect insensitive command implementation, e. g.• Different location of identical commands• Different commands (e.g. packaging mechanism, etc.)

Our "DB Build" exists for each, DB2 and Oracle. Even though they are implemented separately, they provide very common basics.

Both provide a similar set of scripts which are task oriented. Similar tasks have similar names for DB2 and Oracle and they use similar parameters.

We decided to implement the functionality in shell scripting, based on ksh.All code adheres to the same standard in terms of coding, comments and documentation.

One central standard library script (called "[x]bld_stdlib.ksh") is sourced by all other scripts and provides all necessary standards. No direct calls to OS commands are implemented in the scripts; all of them are wrapped with definitions provided by stdlib.ksh.

Even though the builds were developed on Solaris, it only took us five days to adapt it to RedHat linux. The only part we had to change was the stdlib.ksh script. Major changes were with the different packaging mechanism and OS command paths (plus a few other details). As a result most of these five days were dedicated to in-depth testing.

27

Build structure

stdlib config

Install Maintenance Operations

SunOS Linux etc.

Our builds for Oracle and for DB2 are of similar structure and basically divide up in three different sets of scripts.

InstallFor product installation, we have created wrapper scripts around the original installation routines for either DB2 or Oracle. An RSP file template is automatically adapted to the specific server and then provided to the installation routines to support silent install mode.To ensure that the installation will run correctly, functionality has been added to verify the environment (kernel parameters, filesystem structures and sizing, administrative users settings, TSM client setup, etc.). Product upgrades, fixpacks, elimination of old versions or of the complete environment, etc. are also covered with similar scripts. This is completed by scripts to register/deregister the Fault Monitor Coordinater daemon (or it's Oracle counterpart) within Solaris SMF (or inittab or whatever means the respective Unix dialect provides).

As the product installation scripts require high privileges, they are only authorized to run as our master admin user, using sudo.

MaintenanceAnother set of scripts covers instance and database maintenance (creation/deletion of instances and databases, configuration, tablespaces, etc.).Instance creation and deletion is also limited to be run by the master admin user.Database creation and maintenance can be executed by the instance user (for Oracle, there is only one master admin user per server).

OperationsThe third set of scripts deals with operational issues like backups, reorgs, etc. This includes a script that supports the execution of SQL scripts provided by the customer to ensure input validation, correct execution, standardized logging, etc.

stdlib / configAll of these scripts refer to the central "stdlib" script, which provides standard functions like logfile writers, error handlers, date formatting, etc. A set of basic configuration values define the actual standards and defaults like logfile and installation paths, default backup tool (TSM in our case), etc.

This allows us to easily adapt all existing defaults to new requirements at one central place.

Unix dialect abstractionStdlib itself checks under what Unix/Linux dialect it runs (currently by analyzing the output of "uname") and sources the correct version of the dialect dependent script. There is one such script for every single Unix/Linux dialect we support and these scripts provide all particular functions in a generalized form.

Example: in our scripts, we never simply call "grep" or "awk" or the like. Instead, we code "${GREP}" (or "${AWK}" respectively). These environment variables are resolved by the Unix dialect specific scripts which contain code like "export GREP=/usr/bin/grep" or "export AWK=/bin/awk".

This also allows us to use identical commands in our scripts in the case they are implemented differently in the target Unix dialects.

Example: "ckyorn" is a Solaris command to check for "y" or "n" input. It also exists in some – but not all – other Unix dialects. RedHat does not provide it. In our scripts, we code "${CKYORN}" and for Solaris, this refers to "/usr/bin/ckyorn". For RedHat, this definition refers to "/opt/atsmesd/db2/ckyorn.ksh", a self-written script which provides the identical functionality and is simply deployed with our Build.

28

Build Development / Deployment

• Developed and tested on engineering servers• No changes to scripts on target servers!

• Versioning concept• For scripts, packages and supported DB software

• Solaris Packages / Linux RPMs• Proper deployment and installation• Dependencies clearly defined• Proper versioning and decommissioning• Ownership of scripts is set to root (avoid tampering)

A few words as to the way we handle our "Builds". A Build consists of a well defined set of scripts which are properly developed and tested in a respective environment.

We have put a lot of effort into this functionality and it would be bad advice to mess around with this code in a production environment just to apply a quick fix or a nice enhancement. With large environments, consistency across servers would very soon suffer badly.

Therefore we strictly avoid any changes to scripts on the target servers. Builds go through acceptance testing before they are finally packaged and properly deployed during maintenance windows.

To deploy these builds, we use the packaging mechanism of the target system (currently Solaris packages and Linux RPMs). This also allows us to define dependencies on other packages or specific versions thereof.Packages also solve the problem of proper deinstallation of old versions.

Last but not least, this mechanism allows us to set the root user as the owner of these scripts. No way to change this code on the target server = clean environment!

29

Scripting considerations, Portability

• Unix comes in flavors• Integrate an abstraction layer• Supporting a new flavor = ~5 MD, whereof 3 for testing

• [[ "x${KSH}" != "x${KSH}" ]]• Be careful with specific commands

• let, getopts, ckyorn, ...

• Command options may be different• Caution with 32-bit vs. 64-bit platforms

• Calculated values may differ• etc.

Of course we all know that many different dialects of Unix and Linux exist. While many of the standard shell commands are fairly identical, there are still many ways to implement these systems. And this is where the differences start.

There are different ways to manage system startup, automation, package installation and maintenance, etc. If your scripts make use of such mechanisms, it is good practice to provide an abstraction layer. We have implemented this as follows:• Scripts first check whether the script package is correctly installed• If so, they source our central "stdlib.ksh" which provides a set of standard functionality and definitions• The "stdlib.ksh" in turn first checks the current Unix dialect (via uname command) and sources a dialect specific script• The dialect specific script ("stdlib_RedHat.ksh", "stdlib_SunOS.ksh", etc.) provides a set of environment variables which abstract the physical implementation of a certain command.• "stdlib.ksh" and our main scripts only reference these environment variables

• e.g. "${GREP}" instead of "grep"• We decide within the dialect specific script, which implementation of "grep" will be used

As a result, all scripts which implement the functionality of our Builds are very stable. Supporting a new, different Unix flavor is quite simple. It only took us around 5 man days to support the full functionality of our Solaris build under RedHat. Most of this time was dedicated to testing.

One would hope that implementations of shells are identical across platforms. Actually: NO!And it's usually the tiny details which hurt in the end. Small differences like an unsupported command option (e.g. for prstat or so), a different output format you need to parse, etc. can always lead to some trouble.

Therefore: allow enough time for testing across platforms if you have to support many in parallel!

Maths and calculations are always a concern.First, we had to realize that RedHat can behave different from Solaris. As a result, we had to recode our original Solaris "let x=y+z" into "typeset -i x=`echo y+z|bc`" on RedHat. RedHat just came to different results when working with "let" (or "${LET}" as we use it) in certain loops.

"getopts" in RedHat does not accept a parameter list in double quotation marks, Solaris does. As a result, we had to adapt our scripts accordingly. This was one of the very few points we were not able to only change our "stdlib.ksh".

But for another problem, we were extremely happy we had our "stdlib.ksh" abstraction layer:RedHat (and many more Linux dialects) do not provide something like "ckyorn", a shell program which checks keyboard input for "y/n" values. Instead of rewriting the code in 20 or so scripts, we simply provided this functionality in a self-written script and changed respective environment variable definition in our "stdlib_RedHat.ksh" to "export CKYORN=/opt/atsdb2bld/bin/dbld_ckyorn.ksh". 50 lines of code and one definition change – done!

Also tricky: calculations with 32bit vs. 64bit. Number ranges are (or may be) different. Imagine you're doing some maths under 64bit Solaris, ending up with a 12 digit number. Then do the same thing under 32bit RedHat and the result is 0 (yes, a one-digit "zero"). Or even something else.

Thinking about such differences in advance will help you avoid many of these situations. Plus, of course: don't underestimate testing, testing, testing!

30

Before and after

let's compare…

The next part of the presentation will show you what our environment looks like since we have standardized it.

31

Oracle Stability Scenario 2006 2007 2008 2009 2010

Percent 100 82 77 60 51Cost per Instance per Year 41'418 33'867 31'743 24'756 21'298Average Instance cost per year 30'617 30'617 30'617 30'617 30'617

Total cost per year 4'555'963 3'911'688 3'849'620 3'152'414 2'847'723

Number of Instances 110 116 121 127 134Service typesEngineering 352'750 352'750 352'750 352'750 352'750Support 62'250 62'250 62'250 62'250 62'250Engineering by Order 0 0 0 0 0Operations 4'032'509 4'032'509 4'032'509 4'032'509 4'032'509Hardware (all currently known + Space + Backup/Tape) 2'577'879 2'577'879 2'577'879 2'577'879 2'577'879Licenses 574'630 574'630 574'630 574'630 574'630Operations Staff 880'000 880'000 880'000 880'000 880'000

Factors for cost rise 108'454 308'076 245'780 37'569 39'448Project Ora10 40'000 250'000 210'000 0 0Enhanced Support Oracle 9 0 0 0 0 0NDCC Project (Giro) 36'000 24'000 0 0 0Disk space growth 32'454 34'076 35'780 37'569 39'448License growth 0 0 0 0 0Hardware growth 0 0 0 0 0Support Effort growth 0 0 0 0 0Additional Operations Staff 0 0 0 0 0Factors for cost reduction 0 843'898 843'670 1'332'665 1'639'234Hardware Reduction (Wintel - NIAM) 0 343'498 0 0 0Ops Saving Potential by Engineering 0 70'400 76'032 82'115 88'684Ops Saving Potential by Smartsourcing 0 130'000 250'000 430'000 480'000Engineering Savings when Environment ok 0 0 17'638 70'550 70'550Additional Ops Staff Smartsourcing savings 0 0 0 0HW Internal (in-house instead of provider) 300'000 500'000 750'000 1'000'000

TCO per DB over time – Zero Growth Scenario

This was our cost prediction in 2006 for a scenario where the database environment would stabilize in size. Meaning: no additional, new databases and overall disk space growth < 10%.

50% reduction on the average cost of a database. Obviously, the investments in our own infrastructure would also have paid back heavily without growth. And even though these resources could not be used in an optimal way due to smaller economies of scale!

The spreadsheet behind this line chart is divided up in sections "cost per annum when we started", "cost rise factors" and "cost reduction factors". The relevant results therefore are derived from original cost + additional cost – savings, all of them listed per annum.

32

Oracle Scenario 1 2006 2007 2008 2009 2010

Percent 100 66 56 40 32Cost per Instance per Year 43188 28662 24114 17315 13859Average Instance cost per year 25'427 25'427 25'427 25'427 25'427

Total cost per year 4'750'685 4'256'235 4'834'252 4'686'102 5'063'506

Number of Instances 110 149 200 271 365Service typesEngineering 352'750 352'750 352'750 352'750 352'750Support 62'250 62'250 62'250 62'250 62'250Engineering by Order 0 0 0 0 0Operations 4'032'509 4'032'509 4'032'509 4'032'509 4'032'509Hardware (all currently known + Space + Backup/Tape) 2'577'879 2'577'879 2'577'879 2'577'879 2'577'879Licenses 574'630 574'630 574'630 574'630 574'630Operations Staff 880'000 880'000 880'000 880'000 880'000

Factors for cost rise 303'175 657'024 1'309'552 1'738'796 2'600'108Project Ora10 40'000 250'000 210'000 0 0Enhanced Support Oracle 9 0 40'000 80'000 80'000 0NDCC Project (Giro) 36'000 24'000 0 0 0Disk space growth 227'175 306'686 414'027 558'936 754'564License growth 0 16'000 64'000 128'000 196'000Hardware growth 0 0 200'000 200'000 300'000Support Effort growth 0 9'338 18'675 28'013 37'350Additional Staff 0 11'000 322'850 743'848 1'312'194Factors for cost reduction 0 848'298 922'810 1'500'204 1'984'111Hardware Reduction (Wintel - NIAM) 0 343'498 0 0 0Ops Saving Potential by Engineering 0 70'400 76'032 82'115 88'684Ops Saving Potential by Smartsourcing 0 130'000 200'000 300'000 300'000Engineering Savings when Environment ok 0 0 17'638 70'550 70'550Additional Ops Staff Smartsourcing savings 4'400 129'140 297'539 524'878HW Internal (in-house instead of provider) 300'000 500'000 750'000 1'000'000

Real TCO per DB over time – with Actual Growth

While the zero-growth scenario looked already interesting, databases tend to be on the expansive side of the market. Therefore we actually had a massive growth on number and size of databases.

And growth means larger overall size of the infrastructure. And larger size means better exploitation of your assets.

With this scenario – which reflects our real environment – the price per instance was even slightly higher in 2006, which was a result of the storage space (SAN) growth during the year. On the other hand, the economies of scale became much higher over the years. At a certain point, we reached a critical mass of about 150 database instances. This allowed us to optimize the use of hardware resources as well as staff.

As a result, the savings per instance are immense compared to what we had before. But not only the average cost per database dropped by the factor of three. Even much better: the level of service we can provide at this price is clearly better than what we had before.

33

SLA levels improved

Before Now

Availability ~95% >99.5%Attended service 07:00 – 18:00 07:00 – 22:0024x7 feasible Very limited YesDR capability Limited Well definedRevisable No Yes

Reducing the TCO for databases is one part. But at the same time, we were able to support more databases with less staff and to massively improve our service levels.

Newly established and clearly defined service levels include goals for disaster recovery, availability, etc.These service levels are now standardized in four categories (Bronze, Silver, Gold, Platinum) and they have respective price tags attached to them.

• Overall availability boosted from around 95% to 99% as a standard. With specific configurations (RAC, etc.), we can provide up to 99.8%.• As we need less staff to provide the same service level, we were able to provide longer attended service hours without additional resources.• With our old environment, 24 x 7 was only feasible on a limited basis for few specific databases. Now we can provide this as a standard service for anyone who needs it.• Virtualization and zone moving concept allows us to provide DR capabilities in a simpler way.• SLA reporting allows us to clearly prove whether or not we achieve these goals. With the old environment, less figures were available, which often led to uncertainties and a negative gut feeling. This is particularly bad if it's the customer who feels like that!

34

DB Cost per Service per DB – Before

This is the structure of the cost drivers of our old environment.

IBM as our provider took more than 50 % of all our expenses for Oracle and DB2 databases.

Oracle operations made 2nd – thanks to expensive external specialists.

Oracle licensing was the next factor – even though we had been underlicensed at that time.

Our internal cost for environment engineering was only a tiny part of the total sum.

Only minimal support and consultancy for developers was provided and was split up between Engineering and Operations personnel.

35

DB Cost per Service per DB – Today

Today, SAN is by far the biggest cost factor. But then, we have had an immense growth in SAN storage due to new large data warehouses plus an archiving solution which has been migrated from DB2 z/OS to Oracle (IBM Content Manager to Filenet). Currently, we are at 50 TB, partly also because a good part of it is mirrored.

SUN SPARC hardware is also quite pricey, particularly the large M5000 boxes. In a TCO view, which should also consider reliability and durability of the servers, this becomes relative again. With the short lifespan of an x86 server, you often need to replace all servers of a cluster if one machine fails because identical hardware is no more available after two or three years. With SUN boxes, you just replace one server or even only a board and you're done. And they are less likely to fail at all compared to cheaper hardware.

DC Operations and DC Base cover all DC cost including rack space, cooling, energy, security, etc.

DB engineering covers maintenance and development of our product builds, tests of new versions, etc.

DB Operations is the daily operations business which is standardized to a level where we can easily let a small team in India take care of it. One member of this team is always working on-site in Switzerland, close to the engineering team. Apart from this, they work quite autonomously. Weekly video conferences and a phone call now and then are sufficient to provide us with good quality operations work.

In our virtualized environment, the server capability can be exploited much better than with the physical servers we had before. As a result, the license cost per single instance has dropped massively. For a 250% increase in the number of database instances, we only had about 10% growth in license cost.

36

DB Cost Comparison – with Savings

Before

Today

A direct comparison of the cost per average database which also considers the savings is possibly the most convincing evidence of what our project has achieved. And there's still the improved service level that comes at a third of the original price for a database!

37

General Considerations

38

What drives Cost?

• Outsourcing / Service providers• Cash out (HW, Ops, SLA)• Expensive -> Size matters

• Hardware• Physical servers = limited scalability and flexibility

• Environments• Two environments = twice the staff

During contract negotiation with professional service providers, customers often tend to put too little attention to growth (at least we did). Contracts are based on number of servers and database instances, etc. As a result, growth becomes expensive.

The workload of physical machines cannot be controlled as fine granular as this is possible with virtualized environments.

Operating environments for two products require specialist knowledge for both of the products. While the concepts are identical (an RDBMS is an RDBMS), engineering, operations, performance tuning and the like are different. Specialists are required for all tasks for all products.

39

And the answer is…

• Insourcing• Full control over environment• SLAs shaped to customer's needs• Reduced cash-out, no VAT

• Hardware virtualization• Optimize for scalability, DR, SLA requirements, etc.

• Standardization, even across products• Reduce operations efforts and specialist knowledge

The decision to build our own, standardized and centralized DB environment was clearly driven by cost. System growth over the last years – particularly for distributed workload – had reached a level where running our own systems became an interesting alternative to buying these services.

This has been a valuable decision for our in-house customers as it eliminates cash-out. In-house also means "no VAT", which gives our services another head start compared to competitors.

Being in full control of our own data centers, we are now able to design our environments exactly to the needs of our customers. Overhead can be eliminated and SLAs can be shaped accordingly.

Virtualization adds a level of abstraction and causes a few percent overhead on resource consumption. But you gain effortless migration of zones between servers, particularly if specific software is in place that supports zone moving.

This again opens new possiblities for DR. While this may not help if no downtime can be afforded, it helps a lot with the majority of applications/databases where short downtime in case of a disaster is acceptable. We were able to eliminate Veritas clusters when we proved that zones from a failing server could be brought up again on the second site within 30 minutes.

With environment standardization, we have eliminated individualism in setup, maintenance and operations.

With standardization across products, we have driven this even further. With identical setups, maintenance through identically looking scripts and checklists plus monitoring within one single tool, we have reduced a good part of this overhead.

Oracle and DB2 specialists are only required for very specific tasks like engineering, some performance tuning and consulting. Standard daily operations runs smoothly without particular product knowledge and can easily be done by less skilled (and therefore less expensive) staff.

40

Our Steps to Success

• Design new solution• Requirements -> Concepts -> Design -> Implementation

• Standardize Oracle• Start with what hurts most

• Convince Customers• Prove that you new environment works

• Repeat it – with DB2• Take things a big step further

• Technical Implementation, Organisation, Processes, etc.

For our approach, these were the steps that led to success:

Solution designWhile it is always nice to achieve fast solutions, the success of ambitious projects very often depends on the chosen approach.Obviously, it wouldn't make sense to simply start with the implementation. • Analysis of the current requirements and working out sensible SLAs provides the basis of what an environment must be capable of.• Concepts need to be worked out which show how the requirements should be met, independent of the actual technology and products that will be used in the implementation phase• Design and technical implementation of the solution should be based on the concepts and verified against the requirements.

Start with what hurts mostOracle was our time bomb environment. At the same time, this is our strategic environment: growth in this area could be expected. But growth would lead to even more problems unless we could stabilize it. Lots of reasons to start the improvements in this area. This may be different with your environment…

Times of changeIn unknown situations, people feel uneasy. New environments provoke such situations. It was absolutely vital to actively inform about what we were doing and how we were doing it.Once the infrastructure was built and stable, we started the migration with a few smaller databases to give proof of the stability and performance as well as of our ability to handle the environment.

Gain the real savingsRepeating the same with DB2 was already easier as we had gained a lot of experience from the Oracle implementations. As a result, we could concentrate on taking things even further.We made sure we could exploit as many similarities as possible on all levels of the implementation, be it technical, organisational or process oriented.

41

Major Challenges

• Change Management• Expect reluctance, resistance, opposition…

• DB Migration to new environment• Convincing Customers• Standardization• Version Upgrades

• Environment Continuity• Continuous Improvement = Changes to Builds, etc.

• Standardization level may decrease

Change ManagementIn the beginning of our project, the major challenges were less due to technical implementation than to behavioral aspects.

People feel uneasy in times of big changes. Uncertainties lead to reluctance in supporting such a project and it is not simple to convince them as long as you can't come up with hard facts.

It is important to have trustworthy and very experienced people on your side to keep your managers convinced of the approach.

DB MigrationOf course we had investigated a lot into whether and how we could migrate the various databases into our standardized environment (and of course we had done this before we started implementing it). For certain databases, the migration also meant version upgrades and changes of certain parameters. So, when it came to doing the real work, we had to deal with customers feeling a bit uneasy. As they had no direct added value but saw a certain risk (and some downtime) with the migration, their reluctancy was fully understandable.

It was very important to start with few, relatively simple migrations. Apart from rising customer confidence, this also allowed ourselves to get acquainted to the processes and possible problems.

Environment ContinuityOne problem we continue to face is that all further development of our Builds and their functionality can impact the level of standardization in a negative way. It is important that we are aware of this fact and that we tightly coordinate. Before new functionality is implemented, we should always discuss the way we implement it. To be honest, this is not always the case.

42

Customer focus

• Think long term• Ensure that SLAs support operational requirements

• Maintenance windows• Version Upgrades, Patching ,etc.

• Special requirements, non-standard databases• 90% of your databases easily fit to standards• Optimize value-for-price for standard levels

When figuring out the standards for your service levels, make sure you think long term. You will have to provide the infrastructure and make sure things are running smoothly over time. Remember that you are dependent on software suppliers. If you run out-of-service software, no supplier will help you unless you pay for extended support. In certain companies, such configurations also are subject to audit issues.

Therefore ensure that your service levels include proper software maintenance and version upgrades. Sell this as an added value rather than letting the customer perceive such changes as an additional business risk.

Also: 24*7 is nice, but make sure that even then your SLAs include a few hours downtime per year in scheduled maintenance windows. Otherwise you will run into problems because somewhen, you will need to patch or upgrade servers. OS patching may require to shutdown the servers. And even with clusters: if a DB patch causes changes to the metadata (DD in Oracle, Catalog in DB2), a restart of all instances may be unavoidable.

In many shops, customers are used to being treated on an individual basis. Their individual requirements and needs have been supported in the past. You can continue to provide such services, but be aware of the cost this approach imposes onto you as a database service provider.

You can get almost any car in your individual colour, with extraordinary interior and a tuned engine. But this will cost you a fortune because the manufacturing process involves a lot of manual intervention, monitoring and control.

The same is true for IT infrastructure. Clearly describe the service levels you provide and attach clear price tags to them. Also propose that any non-standard service can be delivered if required (and figure out the price for it – which should be very close to or even above the next higher standard level the customer could choose from).

Provide active consultancy to show your customer that he is in good, professional hands. In most of the cases, your customer will end up deciding for one of your standard levels. This will allow you to support him with the full level of automation you have implemented.

Meaning: smooth, stable, well performing and at a good price!

43

Standardization Pros/Cons

Stability (Processes, Habitual behavior, HW, etc.)

Level of Service (Automated, fast, proven, safe, etc.)

Value for Money

Exploit features

Individualism

Dull jobs?

Advantages of Standardization The environment as we have implemented it has shown that stability has improved a lot.This is mainly due to a strictly limited set of hardware, OS and DB systems. Clear processes are in place and over the years, everybody has got acquainted to these standards. It is usually very clear how things should be implemented and handled.

Improvements in the level of service we are able to provide is a result of the standardization.We have reached a certain level of automation on the technical implementation as well as on processes. Less individualism means less amount of work. With the same number of staff, we are not only able to handle a bigger environment, but at the same time provide a better level of service and longer service hours. Plus: a good part of the staff does not require to be database specialists.

Overall, we have achieved a stable, well-performing platform with a good level of service for clearly less money than it cost before. And – in our specific case – there is much less cash out.

Disadvantages of StandardizationOf course this comes at a certain price. Exploiting the latest features of a database product may require some more engineering efforts than you would have with a non-standardized environment. To our experience, there are very few cases customers really need all these sexy new gimmicks.

Customers often come up with very individual requirements and a clever and experienced DBA will be able to exploit his environment to best support these requirements. Within a standardized environment, such individual – admittedly very often optimized – solutions are not always feasible.

For the science and research oriented, clever and experienced DBA, such an environment may not provide the playground he or she will happily spend a worker's life in. I must admit that I myself love experimenting, testing, engineering new stuff. I can clearly find this type of challenge with the further development of our environment.

I can however not find it during everyday DB administration within such a standardized environment. If things become too stable and predictable, I'm losing my interest.

But there is the other type of DBA, the one who finds satisfaction in repeating tasks, in knowing every nibble and bit in his or her area of competence and who is happy if everything runs smoothly and evenings can be dedicated to family and friends instead of emergency calls and crashing databases. Our standardized database platform has been built to support them, and they're definitely happy with it.

Even better: our customers feel the same!

44

Peter SuhnerAXA Tech Switzerlandpeter.suhner@axa-tech.com

Session D09DB2 and Oracle Infrastructure Standardization(CA-IDUG Award 2009)

Many thanks to:Carsten Peters (our Oracle engineer)Rolf Wild (Pentagate, Oracle and Solaris specialist)

This is also the point to mention two names who have put a lot of effort and knowledge into realizing this platform:• Carsten Peters, our engineer on the Oracle side• Rolf Wild from Pentagate, an Oracle and Solaris specialist who came up with much of the conceptual background for this environment

What else is left to say? Well, as usual: Thank you for your interest in this topic – and feel free to contact me in case of questions!

Speaker Bio:Peter has worked in IBM mainframe environments for about 14 years where he mainly concentrated on consultancy and support for development methods, tools and environments. During another 6 years, he broadened his horizon working in J2EE infrastructures and development on Unix and Windows platforms. He currently works as Lead Database Engineer, Teacher and Technical Consultant for DB2 and Oracle at AXA Tech Switzerland.

Peter is an IBM Certified Advanced DBA and Developer on both, z/OS and LUW platforms.You can contact him through peter.suhner@axa-tech.com.

45