dba tasks supporting a sox audit of an 11i environment brian hitchcock ocp 10g dba sun microsystems...

DBA Tasks Supporting a SOX Audit of an 11i Environment

Brian HitchcockOCP 10g DBA

Sun [email protected]

[email protected]

Brian Hitchcock November 3, 2007 www.brianhitchcock.net

mailto:[email protected]

mailto:[email protected]

www.brianhitchcock.net

Brian Hitchcock November 3, 2007

Presentation is Available



SOX Audit

SOX Auditors– Have a job to do

Making you happy isn’t one of them

– Are not Oracle DBAs Are not Oracle Applications DBAs

– Have to audit many different applications No two work the same way

– Have to get something done Audit may not be perfect…



DBA and the SOX Audit

DBA– Can make things difficult for the auditor

I am a genius (Guilty Spark 343) This is how we do things here That’s a stupid question I don’t have time for this

– Can make themselves invaluable Here is what I know This is my understanding of how Oracle works

(WOW) Way Oracle Works



Auditors Want to Know

Who can update data– I’m also concerned about who can read data

Company financial data is not public

Who can access passwords in files– Typical application has password file(s)– Application reads file(s) to setup db access– This is really the first issue in disguise…

These are reasonable questions– “Were it so easy” (Arbiter)– Not easy to answer– Lots of details and exceptions



Theme

Started out with – “Who can update data?”

Turns into– “Do we know who can update data?”

We know the individuals that can update data– Link this back to approval process

Create and modify users

– Link this back to the audit process



Theme

We offer auditors– How we know who can update data– How we know who should update data– How we maintain the answers to the first two

This is the practical answer to– “Who can update data?”



Users That Can Update Data

Oracle Applications Users– SYSADMIN– Application Responsibilities

Oracle Database Users– APPS, Select any table privilege

OS level Users Remote Users

– Database links Users in Clones Password Files



Who Can Update Data?Production Oracle Applications Environment

Application Tier

End-User Browser

Database Tier

ApplicationUsers

ApplicationResponsibilities

Users Data

Data

Data

OS UsersSQL*Plus

Database

Users Data

Data

DbLinks

Oracle Applications Clone

Passwords Not Changed--same as Production

Single TierOS users

SQL*Plus

End-User Browser

Remote Users

Password Files

Database user with select any table



Oracle Applications Users

Don’t have any direct access to db data Apps access is through responsibilities

– What is a responsibility? How to link Apps user with db data access?

– Apps does this through responsibility– Apps users access changes when DBA changes

responsibilities assigned to Apps user– How to help auditors to understand this?– Document who has which responsibilities and why



Oracle Applications Users

Disabling Applications users– Can’t drop applications user once created– Do end-date, change passwords– Do not disable SYSADMIN, GUEST

Best Practices for Securing E-Business Suite– Page 23 shows users

Which can be disabled

– Doc 189367.1



Oracle Applications User

Connects to Applications– Based on responsibilities assigned– Can access specific forms

Tied to specific data in tables

– Database access done through APPS db user

APPS database user– Doesn’t own data tables– Has synonyms to schemas that own data tables

Auditors aren’t used to this architecture…



Active Applications Users

SQL to find– All applications users– Active users (no end-date)– Active users that have end-date in the future

SQL for reports– Same as above– But linked with user employee information

Identifies apps users not linked to a person

– Audit against creation tickets and approvals



Active Applications Users SQL

--Total of All Users Active and Inactiveselect count(*) from apps.fnd_user; 997

--Total of Active Users, no end dateselect count(*) from apps.fnd_user where end_date is null; 755

--Total of Inactive Users, end date has passedselect count(*) from apps.fnd_user where end_date is not null and end_date < sysdate; 241

--Total of Active users with end date specified in the futureselect count(*) from apps.fnd_user where end_date is not null and end_date > sysdate; 1




--> report with employee information spool PROD_FND_USER_02062007.txt select sysdate from dual; select user_name, employee_id, end_date, email_address from apps.fnd_user order by employee_id; spool off spool PROD_end_date_NULL_02062007.txt select sysdate from dual; select user_name, employee_id, end_date, email_address from apps.fnd_user where end_date is null order by employee_id; spool off




spool PROD_end_date_NULL_full_name_02062007.txt select sysdate from dual; select U.user_name, H.full_name, U.email_address, U.employee_id, U.end_date from apps.fnd_user U, hr.per_all_people_f H where U.employee_id=H.person_id (+) and U.end_date (+) is NULL order by U.employee_id; spool off spool PROD_end_date_in_future_02062007.txt select sysdate from dual; select U.user_name, H.full_name, U.email_address, U.employee_id, U.end_date from apps.fnd_user U, hr.per_all_people_f H where U.employee_id=H.person_id (+) and U.end_date > sysdate order by U.employee_id; spool off



Can’t Drop Users

Oracle Applications– Can’t remove a user once created– Can only end-date the user– Can end-date all the user’s responsibilities

Issues– Users can be un-end-dated– Who can do this?

Any apps user with System Administrator responsibility– Need to audit routinely– Change user password and end-date

Prevent access even if someone else un-end-dates



Diagram – APPS users accessAPPS Schema

GL Schema

AR Schema

Views

Triggers

Packages

Tables

TablesSynonyms

Synonyms

Runtime usage of Oracle Applicationsis through APPS schema only. Usersdo not connect directly to productschemas such as GL or AR.



Diagram of Apps user connecting

End-User

APPLSYSPUB

APPS

GL AR HR

connect

connect

…



SYSADMIN

Oracle Applications user– Created by default when installed– Has System Administrator responsibility

Can do anything in Oracle Applications

SYSADMIN isn’t the problem– Can lock down this user login

change password– It is the responsibility that matters

May have been assigned to other Apps users Audit?



SYSADMIN

When environment first installed– Developers had SYSADMIN password

Sad, but true – truly bad management

After go-live– All DBAs shared SYSADMIN login– SYSADMIN password changed ‘mysteriously’

All DBAs locked out

– If anyone failed 3 logins (password failure limit) All DBAs locked out How to recover?



SYSADMIN

Switch to each DBA has individual account– BH1234 with System Administrator responsibility– No further password issues

If any DBA has 3 login failures– Any other DBA can reset password– Only one DBA locked out

More robust support process Can audit DBA activity to an individual



Oracle Database Users For every database schema

– Associated database user– GL db schema

Db user can update everything in associated schema Auditors looked at list of db users

– Two letter usernames must be end-users GL – George Lucas?

This is how he paid for the Death Star? Updating GL in systems in a far away galaxy?

Auditors need your help to understand– Oracle Applications is different…

Already discussed APPS database user



Oracle Database Users

Auditors– Wanted to drop all of these ‘end-users’

Oracle Applications won’t work– How to resolve this?– Identify db users needed for

Oracle Applications Oracle Database Other

– Show auditors how we control all types of users




Oracle Applications Database Users Multiple types of database schemas

– exist in every db, Applications or not– used by shared Apps components

APPS, APPLSYS, APPLSYSPUB

– provided by individuals Apps products GL, AP, HR etc.

– optional db features, end users CTXSYS, SCOTT, BH1234




Best Practices for Securing Oracle E-Business Suite, Version 3.0.5

– Metalink 189367.1– Appendix C, page 51, Database Schemas

Shipped with E-Business Suite– lists schemas, which schema passwords can be

changed by FNDCPASS– which schema passwords should not be changed– details on how to change password for the

various db users




Oracle Database Security Checklist, January 2007, Appendix A

– 10g Enterprise Edition Default Accounts– locked at install if you used Database Creation

Assistant (DBCA)– not locked if db created manually– Appendix B, C, D for Oracle 9iR2, 9iR1, 8i

respectively



Database Users Script

Document all db users– Ties back to Oracle documents

Auditors like this Repeatable process Identify why users exist Identify end-users

– Audit specific privileges



Database Users Script####--> Create table and populate... #### ####sqlplus / set pagesize 1500 set linesize 200 alter session set nls_date_format = 'DD-Mon-YYYY hh24:mi:ss'; ####WHY_CREATED ####Apps Module -- for specific Oracle Applications products ####Database -- created when Oracle database installed ####Apps Db -- created when database installed for Oracle Applications ####Discoverer -- Oracle Discoverer End User Layer ####Database Admin -- DBA account

####--> create table create table SOX_dbuser_doc (username varchar2(20), user_id number, created date, account_status varchar2(32), Why_Created varchar2(15));



Database Users Script####--> insert data for all existing db users insert into SOX_dbuser_doc (username, user_id, account_status, created) select username, user_id, account_status, created from dba_users order by created; ####--> update db users that are Applications Module schema owners update SOX_dbuser_doc set Why_Created='Apps Module' where username in (select oracle_username from apps.FND_ORACLE_USERID); ####update Database ####--> db users created when Oracle database installed or optional ####products installed ####--> from Appendix A - Oracle Database 10G Release 1 and Release 2 Enterprise ####Edition Default Accounts and their status, from Oracle Database Security Checklist ####January 2007. update SOX_dbuser_doc set Why_Created='Database' where username in ();



Database Users Scriptupdate SOX_dbuser_doc set Why_Created='Discoverer' where username in ('EUL4_US','EUL_US'); update SOX_dbuser_doc set Why_Created='Database Admin' where username in ('OPS$ORACLE', 'DBA_PROBE', 'STBY_PROBE'); ####--> db users created for Oracle Applications in addition to Apps modules whose database ####accounts are stored in the database table FND_ORACLE_USERID ####--> from Appendix C - Database Schemas Shipped with E-Business Suite, from Best ####Practices for Securing Oracle E-Business Suite, Version 3.0.4, Metalink Note 189367.1 update SOX_dbuser_doc set Why_Created='Apps Db' where username in ();

####--> Generate report sqlplus / set pagesize 1500 set linesize 200 alter session set nls_date_format = 'DD-Mon-YYYY hh24:mi:ss'; select * from SOX_dbuser_doc order by created;



Select any Table

Why is this granted– User wants ‘read-only’ access– Hit the easy button

Why is this a bad thing?– User can see data in all tables– Violates principle of Least Privilege

Only grant what is really needed and approved

– How is this related to updating data?



Select any Table

User can select from FND_USER– Can see encrypted user passwords

For Oracle Applications users– FND_ORACLE_USERID isn’t good either

Encrypted db schema passwords Well known issues with Oracle passwords

– Doc 340240.1 Assessment of Oracle Password Hashing– Users can find methods to crack passwords

Take passwords to another Apps installation APPS user can update any/all data As well as view confidential data



Select any Table

But I know I didn’t grant this… You have to check your databases

– On-site consultant granted this to our developers– Why?

Because “they needed it”

Regular audit for ‘select any table’ grants See documents list for further issues

– Other tables, privileges that should not be granted



Cracking Passwords I’m not suggesting you do this But if I can find out how it’s done

– Your users can figure it out as well

If we trusted everyone– No need for SOX– No need for security audit

Assume someone will try to crack passwords Auditors won’t know about this

– You have to explain it to them– Have plan to explain how you will deal with this issue



Select any Table

Release 12 (and RUP6) may be better– Doc 457166.1 Non-Reversible Hash Password

Database users– Dba_users– Encrypted passwords visible– Not as bad as FND_USER

Once user has encrypted passwords Can send elsewhere for cracking Why takes this chance?



OS Users

Database user oracle identified externally– Granted DBA_ROLE– Used by DBAs– Connect using SQL*Plus

Sqlplus / as sysdba

OS user ‘oracle’ connects without db password– Then has DBA_ROLE– Anyone with ‘oracle’ password

Can connect from anywhere on your network Can’t audit who this person is



Remote Users Connecting from other servers

– Database links– Created in remote database– Won’t show up in audit of local database

Security relies on protecting passwords– Managed SQL*Net Access

Invited nodes (more later…) Most remote servers locked out Access limited to small number of remote servers

Ideally, only the 11i apps tier server



Users in Clones

Production environment– Others created as clones of prod

Cloning process– Brings production db/apps user passwords– Doesn’t change passwords automatically

If DBA doesn’t change passwords– They can be the same in dev as in production– Dev users may not know it

But they have prod access, can update prod data Doc 419475.1

– Not just user passwords…



Password Files

Auditors concerned– Who can access passwords in files

Many applications store passwords in files– Oracle Applications

No, users don’t access passwords in a file Yes, passwords may be in some files

– No for applications users connecting– Log files may have passwords



Password Files

Many Apps DBA tasks– Require APPS and SYSTEM password– Appears on screen– Adadmin, adpatch– Email screen output, save to file

Passwords in email, in files on disk Are those emails, files properly secured?

– Send output to Oracle Support Can they see your passwords?



What Would Sgt. Johnson Do?

Let’s “get tactical” Practical steps

– Oracle Applications Users– Oracle Database Users– OS Users– Remote Users– Users in Clones– Password Files



Applications Users

Who approved each of them? Who approved their responsibilities

– In Oracle Applications Who approves changes to responsibilities? Separate organization maintains

– Master list of all approved users/responsibilities– Each user created in a separate ticket– Responsibilities modified for one user in a

separate ticket



Report on Apps Users Active Users

– Active Users Concurrent Request– SQL presented earlier

Active Responsibilities– Concurrent Request(s)– Tedious, each report requires entering

responsibility Users that shouldn’t have access

– Left company for example– End-date– Change password



System Administrator

Find all apps users that have– System Administrator responsibility

Revoke from apps users except for– DBA users– SYSADMIN

change password



Database Users

SQL, generate reports Multiple types of database schemas

– exist in every db, Apps or not– used by shared Apps components

APPS, APPLSYS, APPLSYSPUB– provided by individuals Apps products

GL, AP, HR etc.– optional db features, end users

CTXSYS, SCOTT, BH1234 Change passwords

– RUP4, FNDCPASS, change all apps users at once– Prevents database link issues from remote systems



Database Users

Database end-users– Not needed for Applications or database– Examine each user

Database privileges

Dba_role_privs

Dba_sys_privs

Dba_tab_privs Roles granted

Examine privileges of each role



Database Users

SQL to examine each user What to look for

– Non-standard role (SUN_SELECT) What privileges does this role have? Who else has this role

Examine these users– Access to FND_USER– Other system tables

Who has “select any table”– Revoke select any table from all non-dba users



Database Users

Get specific approval for each– And all of their db privileges

Watch for users created before go-live– Used by developers– Everyone has the password– Was granted full privileges on all db objects

accessible by APPS– Developers ran a script while they had access



Database User With Grants

How to deal with existing user– Has access to all db objects– Granted when developers had full access

Developers don’t know what they need– “everything”– They don’t know what their code access



Select any table

Database users– Should not have ‘select any table’ privilege– Only DBA db users have this privilege

Restrict access to system information– Encrypted passwords



OS Users

Change passwords– Don’t wait for password change in clones

Prevent OS level connections– Using ‘oracle’– From anywhere in your network

RBAC roles– User must connect to host first– Su – oracle– Auditing shows individual connecting as oracle



Remote Users

Use Managed SQL*Net Access– Restrict hosts that can connect to database– Ideally only Apps tier can connect

Prevents OS connections from other hosts Database links

– Change database user passwords– Don’t wait for password changes in other systems



Managed SQL*Net Access

How it works– For 11i this means only apps tier can connect to

database Why it is important

– Without it, we can’t identify who is connecting– Anyone with a db account password can connect

from anywhere on the net– Invited nodes, forces users to connect to one (or

more) specific servers, users forced to connect as themselves and connect to database



Invited NodesProduction Oracle Applications Environment

Application Tier

End-User Browser

Database Tier

ApplicationUsers


Users Data

Data

Data

OS usersSQL*PlusTwo Tier

Sqlnet.ora1.2.3.4

DatabaseListener

IP Address1.2.3.4

Remote Host

IP Address5.6.7.8

Invited Node



Invited Nodes

Database Tier sqlnet.ora

############################################################### # # This file is automatically generated by AutoConfig. It will be read and # overwritten. If you were instructed to edit this file, or if you are not # able to use the settings created by AutoConfig, refer to Metalink document # 165195.1 for assistance. # #$Header: NetServiceHandler.java 115.50 2006/07/18 14:57:50 nsugguna ship $ # ############################################################### NAMES.DIRECTORY_PATH=(TNSNAMES, ONAMES, HOSTNAME) SQLNET.EXPIRE_TIME= 10 tcp.validnode_checking = yes tcp.invited_nodes=(<apps tier hostname>.<domain>, <other server hostname>.<domain>)



Cloning Oracle Applications

Cloned environment has same passwords– Clone prod to beta– Beta users have prod passwords

Applies to – Applications users, db users, db links etc.

FND_USER table in clone database– Contains encrypted prod passwords

Must change all passwords in clone



Cloning Oracle Applications

Cloned environment– Has lot and out files from source– Passwords?– Sensitive data?

Remove log and output files– As part of cloning process

You need a documented cloning process



Who Can Update Data Now?Production Oracle Applications Environment

Application Tier

End-User Browser

Database Tier

ApplicationUsers


Users Data

Data

Data

OS UsersSQL*Plus

Database

Users Data

Data

DbLinks

Oracle Applications Clone

Passwords Not Changed--same as Production

Single TierOS users

SQL*Plus

End-User Browser

Remote Users

Password Files

Database user with select any table

Invited Nodes

ChangePasswords

ChangePasswords

Revoke selectAny table

RBAC roles



Password Files

Oracle Applications– Doesn’t use passwords in files– Log and output files

Owned by ‘oracle’ Owner of applications software

Output that contains passwords– Don’t email– Store only on restricted servers– Don’t upload to Oracle support

Can be problematic…



Password Files

Oracle DataGuard added later– Uses oracle password file– Stores SYS password on both primary/standby– SYS password encrypted– Owned by UNIX user oracle

Even oracle can’t see the password

Auditors not concerned



Password Files

Passwords in Oracle Applications files– Latest doc for FNDCPASS 159244.1– Also in Sys Admin Guide B13925-06 page 11-7– Must run autoconfig

to put new password into specific files

– Files are listed in both docs– Tells me that passwords are in these files– These files have group, world permissions

Can be read by non-owner users



Password Files

Open SR on this– Oracle says that was the way it worked– Now, no passwords in any config files

Check these files– No passwords found

Auditors accepted this



Real-World

Are we done?– No– Still no resolution for ‘read-only’ issue

Politically– No user will be granted ‘select any table’– Any user requesting ‘read-only’ access

Must supply list of specific tables– If approved, create role

Grant select on tables to role Grant role to user



Real-World

Don’t grant privileges to users Grant role(s) to users

– Easier to audit who has access to which tables– Easier to maintain

Add privileges to roles Users automatically get added privileges

– Easy to revoke Revoke role from user



Real-World

User suggestion– Grant select any table to user– Revoke select on <table> from user

For all sensitive tables

– Doesn’t work Can’t revoke a privilege that hasn’t been granted

– Select any table doesn’t grant select on all tables It works some other way Can’t revoke select on specific tables



Real-World

User Suggestion 2 Create role

– Grant select on all objects owned by APPS

APPS user owns– Some tables– Lots of synonyms

Including synonym for FND_USER table

This isn’t a solution



Real-World

User suggestion 3 Create read-only schema in database

– No such SQL in Oracle

Sounds good– Reasonable proposal– Create read-only user (schema)

No such functionality



Real-World Oracle must have a ‘best-practice’?

– Open SR for read-only database user– Especially for Oracle Applications database

Users need to see business data– Oracle tells us not to grant ‘select any table’

They must have a better idea

What does Oracle say?– No recommendation for read-only db user– Do not grant ‘select any table’– Grant only specific tables user needs to access– No way to identify what a users needs

“I need to look at all GL data…”



It’s Not Over

Recent Request– Verify that Brian Hitchcock does not have access

Sounds very reasonable– But it can’t be done

What access?– Applications– Database– OS– Remote– Knows another user’s password

Perhaps from dev period



The End

“It’s Finished” (Cortana) “No, I think we’re just getting started” (S117)

– None of these issues are static– Auditing must be done regularly



Editorial Opinion

Auditors start with rules They work with you Accept workable solutions Want to be reasonable

– You may not think so…

You need to offer solutions– Can’t remove GL db account– But we can change the password regularly



Editorial Opinion

Worth what you paid… This sort of task is hard

– And hard to out-source

Requires communication skills– DBA needs to be on-site

Smartest DBA doesn’t win– Best DBA is the DBA that can communicate– Patience required to understand auditors needs



Conclusion

SOX Audit is challenging– Auditors have simple questions

But the answers are complicated

For everything you tell the auditors– Provide evidence

Tie it back to Oracle docs, db tables

Develop repeatable process– This is what YOU want from the SOX audit



Conclusion

SOX Audit Do

– Develop repeatable process– Use each audit to validate, refine your process– Make each audit easier, quicker than the last

Don’t– Wonder how you passed the last audit– Recreate this process for the next audit



Documents

Oracle Applications 11i Password Decryption– Google ‘oracle applications 11i password

decryption’– Integrigy Corp, Stephen Kost and Jack Kanter

Oracle Database Security Checklist, January 2007

– http://www.oracle.com/technology/deploy/security/pdf/twp_security_checklist_db_database.pdf

– Google ‘Oracle Database Security Checklist’



Documents

Metalink– 419475.1 Removing Credentials from a Cloned EBS Prod Db– 189367.1 Best Practices for Securing Oracle E-Business Suite– 340240.1 An Assessment of Oracle Password Hashing Algorithm– 398942.1 FNDCPASS Utility New Feature ALLORACLE– 396537.1 Guest account password– 396537.1 Guest account password– 189367.1 Best Practices for Securing EBS– 398942.1 FNDCPASS ALLORACLE option– 291897.1 Managed SQL*Net Access– 457166.1 Non-Reversible Hash Password