2011 10 ebs concurrent processing best practices


My Oracle Support – Advisor Webcast Program

• EBS Concurrent Manager Performance - Best Practices

Tuesday, October 11, 2011 10:00 AM CET (GMT +2) Teleconference Access: North America: 1866 966 9439 International (UK): +44 (0) 1452 562 665 Conference ID : 95984161

Upcoming Webcasts in EBS Technology area :

o November 2011 : None planned at this stage

Do you have any requests for future EBS - Technology

Webcast Events? Please email your suggestions to

[email protected] ,subject: Topics of Interest.

mailto:[email protected]


AGENDA

• Presentation and Demo

approximately 60 - 75 minutes

• Q&A Session

maximum 15 minutes

Web attendees can ask questions via Q&A panel

Phone attendees can ask questions via Q&A panel or phone


Q&A panel

type your question here 1

Send

your question

3

Who to ask? 2


your question pops-up here


ATTENTION – AUDIO INFORMATION

Voice streaming/Audio broadcast is available.

For full audio access, please join the telephone conference.

Teleconference Connect details:

1. Conference ID: 95984161

2. International dial in: +44 (0) 1452 555 566 (UK) / 1866 966 9439 (US)

3. List with national toll free numbers is available in Note 1354461.1

Note:

You can view this info anytime using WebEx menu from your WebEx-Session :

Select „Communicate‟ --> „Join Teleconference‟.


The following is intended to outline our general product direction.

It is intended for information purposes only, and may not be

incorporated into any contract. It is not a commitment to deliver any

material, code, or functionality, and should not be relied upon in making

purchasing decisions. The development, release, and timing of any

features or functionality described for Oracle‟s products remains at the

sole discretion of Oracle.

Safe Harbor Statement


EBS Concurrent Manager Performance - Best Practices

Presenters: Kandasamy Nainamalai (India) and Kalaivani Murugesan (India)

Q&A Panel Experts : Amira Abdelkader (Egypt), Rakesh Chithuluri (India),

Michael Costa (US), Brian Kerr (US), Mario Bagnara (Chile), David Heisler (US)


Program Agenda • Overview of Concurrent Manager

• Enhancing Performance - Best Practices

Tuning Concurrent Queue Parameters

Specialized Concurrent manager

Output Post Processor – OPP

Tuning PCP/RAC

Transaction Manager (TM)

• Common / Typical issues and resolution

Purging Concurrent records

Running cmclean.sql

OPP and TM related issues.

Resolving concurrent requests

• Trouble-shooting Concurrent Manager







Tuning PCP/RAC




Running cmclean.sql





EBS Release 12 Architecture


Concurrent Manager - Introduction

• Batch Processing Sub-System

• Programs are run as operating system background processes.

• These programs may be written using a variety of Oracle tools, programming languages for executables, or OS scripts.

The Internal Concurrent Manager (ICM) controls all other concurrent managers. It administers the startup and shutdown of managers as defined by their work shift, monitors for process failure, and cleans up if a failure occurs.

The Conflict Resolution Manager (CRM) enforces rules designed to ensure that incompatible concurrent requests do not run in the same conflict domain.

The Standard Manager as shipped with Oracle Applications will accept and run any concurrent requests.



Concurrent > Manager > Administer

Are the

Managers up

and running ?


Concurrent Requests, Programs, and Processes

Concurrent

manager

User requests

Concurrent

Program

Run program ...

Run program X

Run program ...

Queue Table

System maintains list of

requests to start Concurrent

Programs

Concurrent Managers Read

applicable Requests and start

Concurrent programs

Program X

Started


Concurrent Manager - Tables

Table Content

FND_CONCURRENT_REQUESTS Details of user requests, including status,

start date, and completion date.

FND_CONCURRENT_PROGRAMS Details of concurrent programs, including

execution method, whether the program is

constrained, and whether it must be run

alone.

FND_CONCURRENT_PROCESSES Cross-references between concurrent

requests and queues, and a history of

concurrent manager processes.

FND_CONCURRENT_QUEUES Information about each of the concurrent

manager queues.


Concurrent Request - Lifecycle

Phase Activity

Pending / Normal The request is waiting to be run

Pending / Standby The request is waiting at CRM

Running / Normal The request is running

Completed / Normal The request has finished

Inactive / No Manager The request cannot be run. No manager

available to process it.







Tuning PCP/RAC




Running cmclean.sql





Enhancing Performance - Best Practices


Sleep Seconds

Cache Size

Purging

Gather Schema Statistics

Workload Management

Specialized Concurrent managers


Java Heap Size

Tuning PCP/RAC

Transaction Managers



Sleep Seconds - is the number of seconds your Concurrent manager

waits between checking the list of pending concurrent requests

(concurrent requests waiting to be started).

Tips: Set the sleep time to be very brief, during periods when the number

of requests submitted is expected to be high. Otherwise set the sleep time

to a high number (e.g. 2 minutes). This avoids constant polls to check for

new requests.

Concurrent Manager will not sleep at all if there are pending requests.


Tuning Sleep - Parameters

Reduce the

Sleep Time

Concurrent manager > define > Workshifts



Cache Size - Increase it (number of requests cached) to at least

twice the number of target processes.

For example, if a manager's work shift has 1 target process and a

cache value of 3. Then it will read three requests, and try to run those

three requests before reading any new requests.

Tips (Assume you defined one Standard Manager process)

Set a value of „1‟ ‟ if the managers run „long running‟ jobs,

Set „3‟ or „4‟ if the managers run “short running jobs”.


Tuning Parameters – Cache Concurrent > Manager > Define

Increase

Cache Size



Process – Increase it if the pending/normal request are high.

For example, if the Standard Manager has a high number of pending

request in the queue, then we can explore whether we can increase

the process.

Tip: Need to increase after examine the server capacity. If the server

is not capable of handing such number of concurrent requests in

parallel, then it can even downgrade the performance.


Tuning Parameters – Processes Concurrent > Manager > Define > Workshifts

Increase the

Number of

Processes


Purging Log Files and Tables

Use the Purge Concurrent Request and/or Manager Data program to purge Request Log files, Concurrent Manager Log Files, and report output files. Schedule as needed for each instance.

To conserve space you should periodically delete log and output files.


Purging Concurrent Requests

At regular intervals, Purge backend tables that are holding Concurrent

Request related information and delete log/out files from concurrent

manager tier.

To purge, It is recommended to run the request the "Purge Concurrent

Request and/or Manager Data“ (FNDCPPUR) with below Parameters

at regular interval.

Entity =All

Mode=Age

Mode Value=10 (Number of days to keep)

Tip Discuss with Functional team for to fix the retention policy)



Purging Log/out files

Ensure that the log/out files are removed from the below directory as

you run "Purge Concurrent Request and/or Manager Data program".

$APPLCSF/$APPLLOG

$APPLCSF/$APPLOUT

Apply Patch 7834670 (11i) 7530490 (R12). Refer to Note 822368.1

for more details.

Truncate the reports.log file in log directory. Refer to Note 844976.1

for more details.

https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=822368.1



Concurrent Processing Performance

Defragment the tables to improve Performance

As purging is done periodically, it deletes rows from the

FND_CONCURRENT tables . Hence tables get fragmented over a period.

It degrades the performance when full table scan is performed

De-fragment the below tables periodically to reclaim unused space. It

improve performance. (Refer How to defragment)

• FND_CONCURRENT_REQUESTS

• FND_CONCURRENT_PROCESSES

• FND_CRM_HISTORY

• FND_ENV_CONTEXT

• FND_TEMP_FILES


Defragment the tables to improve Performance

Table defragmentation

alter table <owner>.<table_name> move;

Index check

select owner, index_name, status from dba_indexes

where table_owner = upper('&OWNER') and

table_name = upper('&SEGMENT_NAME');

As some Indexes might become unusable for the table move command, it is needed to check the Indexes.

Rebuild Index – if needed

alter index <owner>.<index_name> rebuild online;

Note: Ensure the tablespace in which the object currently exists has got sufficient space before you move or defragment. Always make a

backup before moving the data. It is recommended to test these steps before going into PRODUCTION !

You will need to collect the statistics for the tables.

For example:

exec fnd_stats.gather_table_stats ('APPLSYS','FND_CONCURRENT_REQUESTS',PERCENT=>99);




• Gather Schema Statistics to improve performance overall

• At regular interval, run “Gather Schema Statistics” - use below parameter to speed

up the request.

Schema Name ALL

Estimate Percent 10-40

Backup Flag NOBACKUP

History Mode NONE

Gather Options GATHER AUTO


Enhancing Performance - Best Practices

Specialized / Dedicated Concurrent Managers

Create specialized concurrent managers which is dedicated to some

specific concurrent requests which need to be processed in a different way.

Benefits

1. Speed up Critical Short Running Requests which has long wait time

2. To avoid Long Running Request during Peak time

Tip

Avoid creating an excessive specialized managers. It can degrade the

performance due polling on queue tables. You need to create specialized

managers only if there is a real need.



1. Speed up Critical Short Running Requests

Assume you have a Business Critical "Short Running" (ex. Invoice Printing)

Concurrent Request, but it is waiting for the Standard Manager for a long

time before the Processing is getting picked up.

You can then define a new Specialized Concurrent Manager. And include

the specific critical short run Concurrent Request to the new Manager that

you created. This will ensure all your critical short run Concurrent Requests

are not waiting in the queue for longer time.



2. Avoid Long Running Request during Peak time

This helps you to prevents users from executing such „Long Running‟

Request during Peak time. This improves overall performance of the system.

Create a Specialized Manager to process the specific 'Long Running'

requests.

Assign the Specialized Manager with a work shift with Off peak time.

Assume, the Users submit the Concurrent Requests during peak time, then

it will get assigned to the Specialized Manager.

But Specialized Manager will not process it during peak time.


Enhancing Performance - Best Practices Output Post Processor - OPP

Concurrent Processing uses the Output Post Processor (OPP) to enforce Post Processing

actions for concurrent requests.

Tip :

Ensure that enough OPP process existing to serve the incoming request load. Otherwise

increase “process”.

Thread Vs Process

• You can either increase process (or) threads. It depends on the load .

Concurrent -> Manager -> Define > "Output Post Processor“

oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5

• Each PCP node should have one OPP defined. Refer to Note 563233.1 for more details.



Output Post Processor - OPP

Increase Number of

OPP processes

Increase the

number of

Threads

OR



Heap Size

To prevent the error "java.lang.OutOfMemoryError: Java heap space“ check the current value :

SELECT service_id, service_handle, developer_parameters

FROM fnd_cp_services

WHERE service_id =

(SELECT manager_type FROM fnd_concurrent_queues

WHERE concurrent_queue_name = 'FNDCPOPP');

Increase the maximum Java heap size for the OPP to 1024MB (1GB):

UPDATE fnd_cp_services SET developer_parameters =

'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'

WHERE service_id = (SELECT manager_type FROM fnd_concurrent_queues

WHERE concurrent_queue_name = 'FNDCPOPP');



Bi Publisher Enhancement Parameters

Use XML Publisher's XSLT processor set to True

Enable XSLT runtime optimization set to True

Enable scalable feature of XSLT processor set to False.

– Small Report - It is still recommended this remain false at Site level. This has the

advantage of faster performance by not using temporary files for XSLT processing for

small to medium reports.

– Large Reports - with Patch 7599031,can be safely set to True at the data definition

level, enabling improved JAVA heap utilization along with ScalableFlag parameter and

'Options' field with the value to -Xmx1024M (or higher).

Refer to Note 737311.1 – „How to Configure the Account Analysis Report in Release 12

for Large Reports‟



RAC & Parallel Concurrent Processing

• Set profile option 'Concurrent: PCP Instance Check' to 'OFF' if Instance-

sensitive failover is not required.

• If the failover of Managers is taking too long, then refer Note:551895.1 -

„Failover of Concurrent Manager Processes takes more than 30 Minutes.‟

• Run a concurrent program against a specific RAC Instance with PCP/RAC

setup. (Instance Affinity)

Note 1129203.1 - How to run a concurrent program against a specific RAC

Instance with PCP/RAC setup ?




RAC & Parallel Concurrent Processing



• Ensure enough number of TMs exist to serve the incoming request load.

• Profile “Concurrent: Wait for Available TM” - Total time to wait for a TM before

switchover to next available TM.

– To improve performance , try setting this to 1- 2.

– In case all TM are busy, then increase number of TM.

• Use Advance Queue(AQ) with ATG RUP3 by setting System Profile „Concurrent: TM

Transport Type' to „QUEUE'

• Pipes Vs Queue

– Pipes are more efficient than Queue. To use „Pipes‟, it requires Transaction

Manager defined for every node. However you might want to use "Queue" for easy

maintenance.





Specialized Concurrent Manager


Tuning PCP/RAC



Purging Concurrent Records

Running cmclean.sql


Resolving Concurrent Requests



Script cmclean.sql

Why to run CMCLEAN.sql ?

Cleaning out the tables is a useful method of making sure that there are no invalid

statuses that can prevent the managers from starting.

See Note 134007.1 – „Concurrent Processing - CMCLEAN.SQL - Non Destructive

Script to Clean Concurrent Manager Tables‟. It will make sure the relevant status

codes are valid without deleting any information.

Tip

• Doc ID 152209.1 – „What is the Meaning of the Codes in the STATUS_CODE and

PHASE_CODE Columns of FND_CONCURRENT_REQUESTS Table?‟

• Doc ID 155925.1 – „How to Clean Out Concurrent Manager Tables?‟






Typical issues during Purging Concurrent Requests and resolution

Log/out files are not removed from OS file system

Performance issue

Apply Patch to fix above issues.

Purge Concurrent Request Hangs post cloning.

Update FND Table.



• Release 11i

• Apply Patch 7834670 (on top of 11.5.10.2)

• Ensure afppur.lpc is in Release 115.36 available

• Set the system profile "Concurrent: Force Local Output File Mode

Refer to Note 822368.1 for more details.

Most of your issues should get fixed with Patch 7834670.

To set the Profile Option "Concurrent: Force Local Output File Mode“ to "Yes”,

It is needed that ALL Concurrent Manager nodes must be able to access the

output file locationvia the local filesystem.




• Release 12 - Patch 7530490

• Ensure afppur.lpc is higher than in Release 120.4.12000000.4 available.

• Release 12.1 - Patch 10106197

• Ensure afppur.lpc is in Release 120.6.12010000.8 available.

Version details can be identified using this command

strings -a $FND_TOP/bin/FNDCPPUR|grep 'Header'|egrep 'afppur.lpc'


Purging Requests hangs after Cloning

Cloning is not updating the host- and Instance specific information used by

Oracle Workflow

Review the following tables of the Target Instance:

FND_CONCURRENT_REQUESTS

Update LOGFILE_NAME with the correct path to the logfile directory.


Update OUTFILE_NAME with the new directory path on the Target System.

For more Information, please see the Cloning documentation :

• Release 12 : Note 406982.1 – ‚Cloning Oracle Applications Release 12 with Rapid Clone‟

• Release 11i : Note 230672.1 – ‚Cloning Oracle Applications Release 11i with Rapid Clone‟




Pending Requests

Pending / Normal

• The request is waiting for the next available concurrent manager to be picked

and it will get processed.

Increase number “processes” of the Concurrent Manager.

Pending / Standby

• The Request is waiting at CRM due to Conflict Request which is already

being run.

Check the “Incompatibilities “ defined for the request.

Check any request is running with “Run Alone “ flag.


Concurrent Request Lifecycle


Incompatible Programs

Logical database

Program A Program B

Logical database

Program A

Domain 1 Domain 2

Program B

Conflict Domains is to prevent two programs from concurrently

accessing or updating the same data.



Running / Normal

If the request is taking longer time, enable Database Trace and tkprof and find

out expensive DML. Steps given below.

• Completed / Error

Check the request log file and take action accordingly

• Inactive / No Manager

If “Standard Manager” has “Specialization Rules” defined which has rows with

“Include”. You must only define exclude. If you add any program with “Include”

options, other programs get excluded automatically.





Specialized Concurrent Manager


Tuning PCP/RAC



Purging Concurrent Records

Running cmclean.sql


Resolving Concurrent Requests



Troubleshooting Concurrent Manager

• How To Enable Tracing ( Request Running/Normal)

• Output Post Processor -OPP Timeout

• How to Report Performance Issues


How to Enable the Tracing

Enable Tracing On Program Level (Not suggested)

System Administrator Responsibility

• Navigation: Concurrent -> Program -> Define

• Click the “Enable Trace” button

Note:- This generates trace for all the request submitted by all end- users.

Hence this could generate lots of trace files which is an additional

overhead and degrade performance.


Enable the trace on Program level


How to Enable the Tracing Enable Tracing On Request Level (Recommended)

• Responsibility: System Administrator

• Navigate: Profiles > System

• Query Profile Option Concurrent: Allow Debugging

• Set profile to Yes

• Logon to the Responsibility that runs the Concurrent Program

• In the Submit Request Screen click on Debug Options (B)

• Select the Checkbox for SQL Trace and select desired Trace level.

• Submit the Request.


Enable the trace on Request level


How to enable a trace (continued)

Once you have obtained the Raw trace file you need to format the file using TKPROF :

tkprof raw_trace_file.trc output_file explain=apps/<passwd> \

sort=(exeela,fchela, prsela)

If you do not know the path for those directories, you can run a command like the following

query in SQL :

select name, value from v$parameter where name = 'user_dump_dest';



Error

The Output Post-processor is running but has not picked up this request. No further

attempts will be made to post-process this request, and the request will be marked

with Warning status.

Action

Setting the Profile Option „Concurrent: OPP Response Timeout‟ to a higher value

may be necessary.

If the value of the Profile Option „Concurrent:OPP Response Timeout‟ is 120 then

increase it to 240. (Value * 2).

Increase the number of processes or threads (or both) .



Error

The Concurrent Manager has timed out waiting for the Output Post Processor

to finish this Request.

Check that there are enough Output Post Processor Service Processes running.

More information may be found in the Service Process logfile.

Action

Check the Profile Option "Concurrent:OPP Process Timeout".

Increase the setting to a higher value.

For example: Set the value to 10800 sec (3 hours).

After this change, the Concurrent Request will wait maximum 3 hours,

for the Output Post Processor to finish the Request.


Logging a SR for Concurrent Processing Issues

Please let Support know following details :

• Was there any recent change(s)

• Describe if requests stay in Pending (or) Running phase for longer time

• Is it specific to some „Concurrent Programs‟ (or) all ?

• Was CMCLEAN.sql script run ? (Note 134007.1 )

• Run CCM.sql and upload the output (Note 171855.1 )

• Provide total number of records in the below tables


FND_CONCURRENT_PROCESSES

FND_CRM_HISTORY

FND_ENV_CONTEXT

FND_TEMP_FILES


Logging a SR for Concurrent Processing Issues

• Upload Concurrent Request log, if its specific to a program

• Upload Database Trace with tkprof output

• Upload ICM (or) Standard manager log relevant

• Backend queue concurrent manager details

Select rpad(user_concurrent_queue_name ,50,'-')

"Queue Name",TARGET_NODE, MAX_PROCESSES,running_processes,

nvl(to_char(SLEEP_SECONDS), 'Not Set') Sleep_Second,

nvl(to_char(cache_size),'Not Set')Cache_size

from FND_CONCURRENT_QUEUES_VL fcq

order by 1;


For More Information

1. Best Practices for Performance for Concurrent Managers in E-Business Suite

(Doc ID 1057802.1)

2. Product Information Center (PIC) (Doc ID 1304305.1)

3. CCM.sql Diagnostic Script to Diagnose Common Concurrent Manager

Issues (Doc ID 171855.1 )

4. CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager

Tables (Doc ID 134007.1)

5. E-Business Suite Diagnostics References for R12 (Doc ID 421245.1)







November 2011 : None planned at this stage

Additional Webcasts are planned – they will be promoted soon.

For complete details on all upcoming Oracle Advisor Webcast Events, please see

Note 740966.1, Oracle Advisor Webcast Schedule.

For EBS Technology Specific Webcasts please check Note 1186338.1.

Do you have any requests for future ATG Advisor Webcast Events ?

Please email your suggestions to me :

[email protected], subject: Topics of Interest.

Applications Technology Group Webcasts

https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=740966.1



Communities in My Oracle Support


ATG Communities in MOS available Following Communities are available in My Oracle Support

Oracle E-Business Suite

...

BI Publisher - Business Intelligence Products (not only EBS)

Core Concurrent Processing - Anything around Concurrent Processing and Concurrent Managers

Core Workflow - Any Workflow isse not only E-Business Suite

Diagnostic Tools - Anything around EBS Diagnostics

E-Business Customizations - Your Customizations

Installation - Fresh Install of the E-Business Suite

Patch Review EBS - Review of Patches around the E-Business Suite

Performance - EBS Performance

Upgrade - EBS Upgrade

User Produktivity Kit - User Productivity Kit (UPK) available for the E-Business Suite

Utilities - Utilities / generic EBS DBA issues

...

This is the current list for the E-Business Suite – Applications Technology Group

https://communities.oracle.com/portal/server.pt/community/bi_publisher/323

https://communities.oracle.com/portal/server.pt/community/core_concurrent_processing/493

https://communities.oracle.com/portal/server.pt/community/core_workflow/244

https://communities.oracle.com/portal/server.pt/community/diagnostic_tools/242

https://communities.oracle.com/portal/server.pt/community/e-business_customizations/322



https://communities.oracle.com/portal/server.pt/community/installation/257

https://communities.oracle.com/portal/server.pt/community/patch_reviews_-_ebs/358

https://communities.oracle.com/portal/server.pt/community/performance/333

https://communities.oracle.com/portal/server.pt/community/upgrade/258

https://communities.oracle.com/portal/server.pt/community/user_productivity_kit/235

https://communities.oracle.com/portal/server.pt/community/utilities/259


EBS ATG Product Information Center Note 1160285.1



Q&A


THANK YOU

for attending our Advisor Webcast !

2011 10 ebs concurrent processing best practices

Documents

oracle andor

concurrent records

concurrent queue parameters

concurrent requests

sole discretion of oracle

qa panel phone attendees

minutes qa session maximum

teleconference access