perl and the oracle dba

Perl and the Oracle DBAPerl and the Oracle DBAOr, How to have your oyster, and eat it too.

RadiSys Corporation Confidential

Who Am I?Who Am I?

Jared Still RadiSys Corporation, Hillsboro Oregon, USA Oracle DBA for 10 years Perl aficionado for 10 years


What will you learn today?What will you learn today?

Why I use Perl A Brief History of Perl Origins of DBI and DBD::Oracle Why you should use Perl How you can use Perl Some examples of Perl in action with Oracle


Why I use PerlWhy I use Perl

Background DBA since 1994 Unix user since 1988 Written a few bourne and korn shell scripts in that time Also written a fair number of DOS batch scripts.

Began using Perl in 1994. Purpose for using Perl was initially as a reporting language. Soon realized it could do much more Used to create a job scheduler that ran our reports in the

background, notifying users when reports were complete.

After this I was hooked. Perl provided much more control over details than was

possible in shell scripts. Does not mean that Shell and SQL*Plus were abandoned. Perl was a powerful new tool in the toolbox.


A Brief History of Perl – A Brief History of Perl – with apologies to Dr. Hawkingwith apologies to Dr. Hawking

Perl 1.0 released on 12/18/1987 From the man page:

Larry Wall: “The language is intended to be practical (easy to use, efficient, complete) rather than beautiful (tiny, elegant, minimal). It combines (in the author's opinion, anyway) some of the best features of C, sed, awk, and sh, so people familiar with those languages should have little difficulty with it.”

Perl is actually old enough to be considered ‘Legacy’

Perl has gone through several generations Current version is 5.8.x Perl has since evolved to a much more

sophisticated language, with a huge library of available modules


A Brief History of Perl - Cont.A Brief History of Perl - Cont.

Oraperl was created by Kevin Stock in the early 1990’s Oraperl was separate version of Perl that was linked with

Oracle libraries. Worked with version 4 of Perl First interface that allowed direct access to Oracle from

Perl. This was very cool at the time. All the power of Perl to

access the OS, file management, a powerful procedural language and access to Oracle – without the code, compile, link, test cycle.

Contractors that knew C and SQL had no trouble working with Perl.


A Brief History of Perl - Cont.A Brief History of Perl - Cont.

DBI and DBD::Oracle – the Perl Gateway to Oracle Released by Tim Bunce in 1994 Created to provide a consistent API to databases. Not just Oracle, but also Sybase, DB2, Ingres, MySQL,

PostgreSql, Informix and any database accessible via ODBC.

Provides very fast operations on database Allows multiple simultaneous connections to

multiple databases. DBD::Oracle is based on Oracle OCI An easy to use OO interface. This does not require

scripts using DBI to be written as OO code.


Why you should use PerlWhy you should use Perl

Adaptability “Swiss Army Knife” of languages Have you ever had to write routines in Shell, SQL or PL/SQL

and run into language limitations?• Connections to multiple databases not known until runtime.• Complex data structures

There are virtually no limitations in Perl and DBI• Multiple simultaneous database connections• Complex data structures are easy to create• If you need to deal with LONG datatypes, you can do so in Perl with DBI ( you can’t

do so in PL/SQL with a LONG > 32k )• Virtually anything you need to do can be done in Perl.


Why You Should use Perl – cont.Why You Should use Perl – cont.

Complexity Unix users are probably scripting in ksh, or maybe bash. Finer grain of control

• Easily trap both OS and Oracle errors• Powerful command line argument parsing• Direct control of many OS features• Sockets, ftp, ssh, file handling• Language control structures

Several things that Perl can do that you cannot easily in shell scripts, SQL*Plus or PL/SQL

• Create multiple simultaneous connections to 1 or more databases• Perform complex mass transformations on data outside the database• Interact with the user – cannot do this in PL/SQL• Interact with the user while performing error and sanity checking on user inputs –

cannot do this in SQL*Plus or PL/SQL• Some of these features available in shell, but not as robust and not cross platform.

Perl give you the most of the power of C, but in an easier to use form.


Why You Should use Perl – cont.Why You Should use Perl – cont.

Compatibility Scripts that are written in Perl can work the same way

regardless of platform I write scripts on Linux; then run them on Win32. The same script will work the same way on multiple

versions of unix.

Simplicity You may hear that Perl code resembles line noise.

• This can be true of any language

What can be done in other languages can usually be done in less time and fewer lines of code in Perl.

• Perl has some powerful operators and functions that can accomplish a lot with just a little code. Not required to use them.


Example of a simple Perl ScriptExample of a simple Perl Script

simple.pl I use this, or a variation, to begin most of my Perl

scripts that connect to Oracle. Includes

Parsing of command line arguments Connecting to the database Preparing a SQL statement Executing the SQL statement Displaying the data A basic help screen


Simple.pl Simple.pl

Magic cookie, documentation and required modules1 : #!/usr/bin/perl -w2 :3 : =head1 simple.pl4 :5 : a simple (but robust) DBI script6 :7 : This script queries the table DBA_USERS and outputs8 : the results in a formatted manner.9 :10 : Here's an example:11 :12 : simple.pl -database dv01 -username scott -password tiger13 :14 : You may change the method of output by finding the 'write'15 : statement, and inserting a single '#' at the beginning of the16 : line to comment it out. Next, uncomment the 'print' statement17 : that is following it.18 :19 : =cut20 :21 :22 : use warnings;23 : use strict;24 : use DBI;25 : use Getopt::Long;


Simple.pl – cont.Simple.pl – cont.

Process command line arguments26 :27 : my %optctl = ();28 :29 : Getopt::Long::GetOptions( \%optctl,30 : "database=s",31 : "username=s",32 : "password=s",33 : "help"34 : );35 :36 : usage(0) if $optctl{help};37 :38 : my($db, $username, $password) = (39 : $optctl{database},40 : $optctl{username},41 : $optctl{password}42 : );43 :44 : usage(1) unless (45 : $db46 : and $username47 : and $password48 : );



Connect to the database 49 :50 : # create a database handle51 : my $dbh = DBI->connect(52 : 'dbi:Oracle:' . $db,53 : $username, $password,54 : {55 : RaiseError => 1,56 : AutoCommit => 057 : }58 : );59 :60 : die "Connect to $db failed \n" unless $dbh;61 :62 : # set the array fetch size63 : $dbh->{RowCacheSize} = 100;64 :65 : # set default Oracle date format66 : $dbh->do(q{alter session set nls_date_format='yyyy-mm-dd

hh24:mi:ss'});



Prepare the SQL67 :68 : my $sql=qq{select username, created, lock_date, expiry_date from

dba_users};69 :70 : # create a statement handle71 : my $sth = $dbh->prepare($sql);72 :73 : # execute the SQL74 : $sth->execute;


Simple.pl – contSimple.pl – cont

Fetch and display the data75 :76 : #print column names77 : print join(',',@{$sth->{NAME}}),"\n";78 :79 : my @data=();80 : while( my @result = $sth->fetchrow_array ) {81 :82 : # this line will cause warnings about uninitialized variables83 : # due to NULL values in SQL return set84 : #@data = @result;85 :86 : # this one will work correctly87 : @data = map ( defined($_) ? $_ : 'N/A', @result);88 :89 : # the above is the equivalent of this90 : #foreach my $el ( 0..$#result){91 : #$data[$el] = $result[$el] ? $result[$el] : 'N/A';92 : #}93 :94 : # write the data as formatted output95 : write;96 :97 : # or print it CSV style98 : #print join(',',@data), "\n";99 :100 : }



Disconnect from the database101 :102 : $sth->finish;103 : $dbh->disconnect;


Help or usage subroutine 104 :105 : sub usage {106 : my $exitVal = shift;107 : $exitVal = 0 unless defined $exitVal;108 : use File::Basename;109 : my $basename = basename($0);110 : print qq/111 :112 : usage: $basename113 :114 : -database target instance115 : -username target instance account name116 : -password target instance account password117 :118 : example:119 :120 : $basename -database dv07 -username scott -password tiger -sysdba121 : /;122 : exit $exitVal;123 : };



Data formatting124 :125 : format STDOUT =126 : @<<<<<<<<<<<<<<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<

@<<<<<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<<<127 : @data128 : .129 :


Using Perl every day - AuthenticationUsing Perl every day - Authentication

PDBA Password Server Too many passwords! Eliminate hard coded passwords Returns password to the command line for use in shell

scripts Authentication required to use server Can restrict password retrieval to certain users


Using Perl every day – contUsing Perl every day – cont

Monitoring Alert Log Monitor (chkalert.pl) Uptime monitor (dbup.pl)

Log Rotation Copy alert.log to new file and compress Truncate remainder Delete archives after 90 days. (alert_log_cleanup.pl)


Using Perl every day – EnvironmentUsing Perl every day – Environment

poraenv.pl – Oracle Home utility on Win32 Oracle Home utility on Win32 requires GUI access, not very

practical for use in scripts that run unattended Oracle Home utility on Win32 is also rather inconvenient for

cmd line use Oracle 10g does not include an Oracle Home utility Oracle Home utility included with previous versions of

Oracle do not work with 10g. This is fine, as long as 10g is the only version of Oracle on

the server, and there is only one instance of it. (demo use of oraenv.cmd. scripts are oraenv.cmd, poraenv.pl, getoraenv.cmd, and

getenvval.pl ( show the code – c:\perl\


Using Perl every day – MetricsUsing Perl every day – Metrics

Statspack produces a lot of data. This data may be useful for tuning and spotting trends when trouble arises. “Data” however is not synonymous with “information” One useful metric that can be derived from statspack data is response time.

YAPPPACK from Miracle A/S Use it to create history of response time averages Found at http://miracleas.dk/en/tech.html Yapp provides method to aggregate response time averages

from statspack data Instructions included for using data to create a chart in Excel I thought this would be a good metric to track for several

production databases. Automated the creation of the charts via Perl, DBD::Oracle and

DBD::Chart DBD::Chart is an innovative module that allows the creation of

many types of charts directly from table data. These charts are created daily for our production databases.


Using Perl every day – Metrics cont.Using Perl every day – Metrics cont.


Using Perl to simplify difficult tasksUsing Perl to simplify difficult tasks

"Perl makes the difficult things easy, and the impossible is made possible" - Larry Wall

Auditing Auditors and Managers like to see data in Excel

spreadsheets. Dump data to CSV, load, massage data, etc. Very time consuming and tedious. Lots of cut and paste involved. For repetitive tasks (such as audit reports), two modules

have proved to be a huge benefit.


Audit Reports with PerlAudit Reports with Perl

SpreadSheet::WriteExcel This Perl module allows creation of Excel Spreadsheets

with multiple worksheets. Allows most formatting capabilities of excel: colors, cell

formats, etc. Cross platform. Files created on Linux work fine on

Windows.

SpreadSheet::ParseExcel Security data may be supplied in Excel spreadsheets. This module allows the data to be parsed from the file at

runtime. Alternative is to dump to tables, but this is more work, and a may be a manual process.


Audit Reports with Perl – cont.Audit Reports with Perl – cont.

Baseline Audits Auditors want to know what has changed in the database

since the previous audit. Each change in the database should have a corresponding

Change Control Record in your Change Management System.

Generally audited from the database back. ie. Changes in the database are identified, change control records examined.

You should be able to determine changes that have occurred since the previous audit.

Rather than evaluating tools to collect baseline data and report on it, I created one.

It does exactly what I need it to do, and the time spent writing it was less than that needed to locate, evaluate, purchase and implement a canned tool.

Saving both time, and money.


Audit Reports with Perl – cont.Audit Reports with Perl – cont.Baseline ReportBaseline Report


Audit Reports with Perl – cont.Audit Reports with Perl – cont.

Permissions Audits Auditors require reports detailing permissions granted in the database. This

is known as account reconciliation. As a DBA, you will be the person asked to provide this data. This is another area where a little extra time spent to automate the task will

simplify subsequent future audits. Example types of account reconciliation

• List of known roles and purpose of each• List of known system accounts and purpose of each• List of accounts and purpose of each• List of accounts and roles and privileges assigned to each• List of roles, and each user they are assigned to• List of roles, and privileges assigned to each.

This turns out to be quite a bit of work. Some of our security data is stored in spread sheets and must be retrieved

from there, and used to validate the user accounts for employees and contractors.

This could be a very intensive manual task, and must be performed several times a year.

Spreadsheet::ParseExcel module is used to load data from excel spreadsheets into Perl data structures. These are used to validate database security information at runtime.

Unknown accounts, roles, etc. are marked as suspect and must be validated by application owners and the DBA.

Spreadsheet::WriteExcel is used to create an Excel spreadsheet that can be used for any remaining validation that must be done.

Time to do this manually, even using SQL scripts to dump CSV files, is prohibitive.

This is a major time saver for the DBA at our company (me)


Audit Reports with Perl – cont.Audit Reports with Perl – cont.Permissions ReportPermissions Report


Recalcitrant Win32 ServicesRecalcitrant Win32 Services

Do you have win32 servers? Do some services fail to automatically restart after a

reboot? After being bitten by this a couple of times when

alert log monitors didn’t start, and backups weren’t made, it was time to do something about it.

Use get_oracle_services.pl to query all servers in our domains and retrieve Oracle services

Edit resulting configuration file as needed. Use config file to drive NT_check_services.pl Script is run via Windows job scheduler or cron. An attempt to Services that are not running, and the

DBA is notified via email of the restart, and its success or failure.


More Reasons to use PerlMore Reasons to use Perl

Huge library of modules at cpan.org File handling Networking Error handling Directory walking ( File::Find ) Graphics ( Charts and Graphs) Application Specific ( Oracle, Excel)

If you are working on Win32, you have *much* more reason to use Perl. Default batch language on Win32 is somewhat lackluster There are Many Win32 Specific modules Win32::AdminMisc Win32::Daemon ( run Perl as a service ) Win32::TieRegistry Win32::NetAdmin SpreadSheet::ParseExcel SpreadSheet::WriteExcel

Many more… Q: As a DBA, why do I care about this stuff?


Tips for using PerlTips for using Perl

Cross Platform Scripting When writing cross platform scripts, some care does need

to be taken Always use the unix form of directory separator in Perl

scripts, it works just fine on Windows. Write scripts in DOS format: use line terminators of CR/LF

when writing on *nix. This makes them easier to read if using Notepad on Windows.

There are exceptions of course Such as when writing scripts that are OS Specific Writing scripts that run in the background Unix runs as a daemon Win32 runs as a Windows service There is a different mechanism for each.


Tips for using Perl – cont.Tips for using Perl – cont.

Use the Module Library Use the following resources before ‘rolling your

own’ http://search.cpan.org http://freshmeat.net And of course, the google and teoma search engines

You may find that someone has already put together a utility that does what you need.

Or at least, modules that will simplify your task.


Perl TriviaPerl Trivia

Tell me the official definition of the PERL acronym

Tell my the unofficial definition of the PERL acronym


Q & AQ & A

Questions?


ReferencesReferences

Programming the Perl DBI http://www.oreilly.com/catalog/perldbi/index.html

Perl for Oracle DBA’s http://www.oreilly.com/catalog/oracleperl/

Programming Perl http://www.oreilly.com/catalog/pperl3/index.html

The Perl Cookbook http://www.oreilly.com/catalog/perlckbk2/index.html

Win32 Perl Scripting: The Administrator’s Handbook http://www.roth.net/books/handbook/

Win32 Perl Programming: The Standard Extensions http://www.roth.net/books/extensions2/

http://www.perl.com http://www.perlmonks.org http://search.cpan.org


Contact, examples and codeContact, examples and code

Jared Still [email protected] [email protected]

Code and examples All zipped up into one convenient tarball http://www.jaredstill.com

perl and the oracle dba

Documents

s s s s perl

s s s s s s

hawking s s perl

s s s s jared

perl code

perl cont

power of perl

simple perl scripts