cob2[1].close of business – batch.job.control, errors-r10.01.pdf

8/11/2019 COB2[1].Close Of Business BATCH.JOB.CONTROL, Errors-R10.01.pdf

1/56

1. Welcome to the Close Of Business BATCH.JOB.CONTROL & Errors learning unit.

s earn ng un t w e p you un erstan , t e erent stages o an t eapplications in T24 associated with COB.

COB2.Close Of Business BATCH.JOB.CONTROL & Errors-R10.01


2/56

After completing this learning unit/course, you will be able to:

Understand the internal working of COB

Understand the working of tSM & tSA

Visualize failure scenarios

Understand the types of errors generated during COB



3/56

What have you seen till now -

1. You have learnt the various stages of COB

2. The fields in BATCH record

3. tSM and tSA

4. TSA.SERVICE

5. TSA.WORKLOAD.PROFILE

6. TSA.PARAMETER

7. Running COB in phantom and Interactive mode

8. Date Change

9. You learnt what is the NS Module

10. Monitoring COB using enquiries

11. EB.EOD.ERROR



4/56

1. Default Printer specifies the default printer to which all output is to be directed for the

process. no pr nter name as een spec e n e . or t e n v uajobs, then this field allows the user to define a default printer to direct the output. If this

field is left blank and no printer has been specified for the individual jobs, then thedefault SYSTEM printer defined in DE.FORM.TYPE will be used.

2. Printer Name field holds the Name of the printer where the output of the job needs to besent to. If left blank, the printer specified in the Default Printer field will be used

3. Data field holds the Input parameters for the job specified in JOB NAME. Usually

contains ENQUIRY REPORT ids or REPGEN ids. In this screen shot when you seeENQ followed by EU.FX.PL.TODAY you might think that this is the name of an enquiry

NO. This is the ID of a ENQUIRY.REPORT application and should be specified in thismanner in the DATA field

4. Job Message : In case the job results in an error, the error message is stored in thisfield. The error message gets cleared once the job error has been corrected and the

jobs Job Status changes to 2



5/56

1. In T24, most of the COB routines are multi-threaded (Not OS multi threading but a

s mu ate mu t -t rea ng

2. A routine is broken up into 3 parts - LOAD, SELECT, RECORD

1. LOAD routine - initialization of common variables and parameters

2. SELECT routine - selects IDs from file

3. RECORD routine - contains actual processing logic

3. LOAD and RECORD routine are executed by all tSA and SELECT routine is executedby only ONE tSA. The main job of the SELECT routine is to select all the IDs that needto be processed. Now, imagine a scenario, where in interest needs to be accrued for

1000 accounts as part of a COB job. Would you like all tSAs to perform a select on andget the list of account ids and want each of them to accrue interest for all the accounts?

Of course not. It would suffice if one agent performs a select and stores the selected idsin a common file so that all agents can share and process the ids. This is the reason

why only one agent performs the select while the others wait until the select is complete.

4. Instead of executing one long routine (by only one tSA), breaking it up into logical partsand then executing it (by multiple tSA), is faster



6/56

1. As you know a multi threaded routine comprises of 3 parts. Each part is a separate

rout ne t at oes a spec c tas .

For example: If we look at a sample routine ROUTINE1, it will be made up of three partsROUTINE1.LOAD, ROUTINE1.SELECT and ROUTINE1 (This is called the record

routine).

2. Once the tSAs are started, each of the tSAs execute the .LOAD routine.3. Then only 1 of the tSAs will execute the .SELECT routine while the others wait. As a

result of the .SELECT routine a LIST file will be populated. This list file will contain theactual list of IDs to be processed from the database (as a result of the select statement

inside the .SELECT routine)

4. Once the LIST is ready, all the tSAs will pick up IDs from the list file, and startexecuting the IDs one by one by calling the record routine, viz the routine name itself

i.e. ROUTINE1 (does not have any suffix to it)

Q. Now, how is this different from single threaded routine?

Ans . In case of a single threaded routine, there would be just one routine that would havethe logic of .LOAD, .SELECT and execution built into it. Therefore, only 1 tSA wouldprocess the whole routine. However, in case of a multi threaded routine, the logic is split

and written in the 3 separate routines, and multiple tSAs can execute it at the sametime, thereby decreasing the total time taken to execute a routine, and increasing

throughput.



7/56

1. How will I know if a routine is single threaded or multi threaded?

Any job (routine) that will be executed during COB, will contain an entry in the application

PGM.FILE. The id of the record in this application will be the name of the job and thefield TYPE will be set to B

2. If the field BATCH.JOB contains a NULL value or @BATCH.JOB.CONTROL, then theroutine is a multi threaded and will have 3 parts to it.

For e.g.: AZ.CYCLE.DATES.LOAD, AZ.CYCLE.DATES.SELECT and AZ.CYCLE.DATES

NOTE that the .LOAD and the .SELECT routines need not have an entry in thePGM.FILE, they will automatically picked up by the T24 system while executing theroutine.

3. If the field BATCH.JOB for that particular job contains the name of a routine (can be the

same name as that of the job) then the routine is single threaded. In other words, if thefield BATCH.JOB is not NULL or does not contain @BATCH.JOB.CONTROL then the

routine is single threaded



8/56

1. The first agent (tSA) to be started is called the tSM (T24 Service Manager). tSM monitors one or moreagent processes called tSA (T24 Service Agent). Every time an agent is started, it will f irst check if it isthe tSM. If not, then it will continue to work as an agent.

2. tSA invokes a routine called S.JOB.RUN, that in turn invokes a routine EB.SORT.BATCH only ONCE,

which sorts the COB jobs in A, S, R, D, O order.

3. Assume that today is 04-Jan-2010 and the table here contains sample records from the BATCH

application. EB.SORT.BATCH fetches all the processes from the BATCH application whosePROCESS.STATUS field is not equal to 2, i.e. it picks up all the processes that are not complete. It thensorts the jobs based on the Batch Stage and the Frequency. Once EB.SORT.BATCH returns the list

of jobs to be processed, S.JOB.RUN invokes a core T24 subroutine called BATCH.JOB.CONTROL,about which you will learn in the next few slides

4. In what order do you think will EB.SORT.BATCH give the jobs to each tSA for execution?

5. Each job is given in the form of a dynamic array to a tSA containing the details that you see on thescreen now

PROCESS.NAME:'_':JOB.NAME:'_':RTN:'_':JOB.DATA:'_':COMPANY.ID:'_':NEXT.RUN.DATE:'_':

ACTIVATION.FILE:'_':BATCH.STAGE:'_':JIDX:'_':BATCH.PROCESS.STATUS

E.g.: BNK/AC.START.OF.DAY_ACCOUNT.DEBIT.LIMIT.UPD_BATCH.JOB.CONTROL_ _GB0010001_ __D110_1_0

1. PROCESS.NAME - BNK/AC.START.OF.DAY (@ID of BATCH record)

2. JOB.NAME - ACCOUNT.DEBIT.LIMIT.UPD (JOB NAME in the above BATCH record)

3. RTN - BATCH.JOB.CONTROL (Contents of the field BATCH.JOB in the PGM.FILE entry for the aboveJOB.NAME)

4. JOB.DATA - (Contents of field JOB.DATA for this job in the BATCH record)

5. COMPANY.ID - GB0010001

6. NEXT.RUN.DATE - (not needed for a job with FREQUENCY D)

7. ACTIVATION.FILE -

8. BATCH.STAGE - D110 ; JIDX - 1 (multi value position of the above job in the BATCH record)

9. BATCH.PROCESS.STATUS 0



9/56

Slide 8

SG2 Sujoy Ghosh 08/05/2008Concentrate on explaining How the jobs are sorted. Make the trainees write it on a piece of paper firstand then check with the slide. The slide is animated, so the result of sorting is on a mouse click, wait forthe trainees to complete before showing the animationMake sure that they understand that within a BATCH stage the jobs CANNOT be swapped or sorted, for

eg: JOB2 cannot be executed before JOB1Sujoy Ghosh, 2008/05/22


10/56

1. In the previous screen, each of the agents were given a list of Jobs to execute according to the

Batch Stage. Now lets understand how each tSA processes these jobs.

. rs , e . e s up a e n e . . e o eans e s nowgoing to invoke a core T24 routine called BATCH.JOB.CONTROL). BATCH.JOB.CONTROL is

the heart of the COB process and takes care of the entire execution logic. This routine is invokedonce per job. This tSA first executes the .LOAD routine. In our example the name of the job isJOB5, therefore the name of the load routine will be JOB5.LOAD. As a result of the LOAD routine

all the common variables required by this routine will be initialized.3. Since, all the tSAs execute the same job, they need to know where to pick up contract ids in

order to process them. The contract ids are all stored in a LIST FILE.



11/56

1. What is a LIST FILE?

A LIST file is one that holds all the ids of the contracts from the database

2. How did the tSA know which LIST file to use?

Within BATCH.JOB.CONTROL, there is logic to find out a free LIST file

3. This routine will loop through F.LOCKING with @ID starting with F.JOB.LIST.1, and will

return the one (List file name) that is first available

4. If this LIST file does not exist then BATCH JOB CONTROL will create the LIST file



12/56

Slide 10

s1 F.JOB.LIST.1 exists by default when T24 is shipped. Every time the LIST file does not exist and BJC hasto create the LIST file, it creates it by using the properties of F.JOB.LIST.1salamelu, 2010/02/27


13/56

1. Before deciding which LIST FILE to use, it updates the JOB.PROGRESS field in

. . to . e va ue means t at t e t s manag ng a recor n a e ca eF.BATCH.STATUS.

What is F.BATCH.STATUS?

This is a file that contains the status of a particular job (within a batch record). A record ID in

this file is the FLAG.ID, viz. the ProcessName-JobName-Multivalue Position of the job in theBatch record.

2. All the tSA try to Read and Lock a record with @ID as Flag ID in F.BATCH.STATUS. OnetSA succeeds while the other tSAs wait for the Lock to be released. First time this record

will not exist in F.BATCH.STATUS, we use a jBASE command to read and lock a record;this command will lock a record even if the record does not exist.

3. The tSA which has acquired the lock on F.BATCH.STATUS record with @ID as Flad Id will

perform the List File allocation (refer to next slide for list file allocation logic)

4. The value in the second position of the record is checked. If it not processing, then theSELECT has to be performed. A value of Processing signifies that the select is over and theprocessing of contract IDs should begin. Since the .SELECT has not yet been executed for

this job, this position will be empty.



14/56

Step 1: Read a record in F.LOCKING with @ID equal to Process.Name-Job.Name-

o . os t on. rocess ame s t e name o t e recor . o ame s t e o w t nthe process and Job. Position is the position of the job in the BATCH record. This is

collectively called as FLAG.ID (P.N-J.N-M.Vpos). If the record exists, then read the list filename that has been assigned for the job and continue with the select logic.

Step 2: Start a transaction block

Step 3: If the record does not exist, then Read and Lock a record in F.LOCKING with @ID equalto Process.Name-Job.Name-Job.Position. The first time this record will not exist in

F.LOCKING. We use a jBASE command to read and lock a record. This command will locka record even if the record does not exist. Therefore, the tSA that came in first will try to

read a record in F.LOCKING and will retain the lock till the transaction ends.

Step 4: Get the name of the LIST FILE into which the .SELECT routine will populate values into.

Step 5: The name of the FILE in this case is F.JOB.LIST. e.g.: F.JOB.LIST.2. If thefile (LIST FILE) does not exist, then the routine will automatically create it, thereby ensuring

that the file physically exists.

We create a record in F.LOCKING with @ID as LIST FILE NAME. The contents of thisrecord is the FLAG.ID. This is done to stop other threads from picking up the LIST FILE to

populate data into it.We also create another record in F.LOCKING with @ID as the FLAG ID. The contents of

this record is the name of the LIST FILE. This is done so that we can publish to other agentsthat this particular LIST FILE is to be used for this JOB

Step 6: End the transaction block



15/56

5. Step 1: Start a transaction Block

6. Step 2: Update the JOB.PROGRESS field in F.TSA.STATUS to 2, which stands for

selecting contracts to populate into the LIST FILE record.

Execute the Job5.SELECT routine. This routine will take care of populating the contracts inthe LIST FILE record based on the logic described earlier

7. Step 3: The .SELECT routine can be called multiple times based on a values in a commonvariable CONTROL.LIST. The values here are separated by FM. For each value a select isperformed. The contents of CONTROL.LIST are written to F.BATCH.STATUS. Therefore,Update the record in F.BATCH.STATUS with the value of CONTROL.LIST along with the

first line containing VMPROCESSINGVMMAX.IST.IDVMKEYS.PROCESSED. MAX.LIST.ID is a

variable that contains the total number of records in the LIST FILE. KEYS.PROCESSED is avariable that contains the total contracts present in the LIST FILE.

8. Step 4: End the transaction Block

Once the transaction block ends, the lock that was held on F.BATCH.STATUS by the first

tSA will be released. The other tSA that was waiting for the lock on F.BATCH.STATUS, willnow get the lock on the record in F.BATCH.STATUS with @id equal to FLAG.ID. However,

now this tSA will find that the contents of the record is already set to PROCESSING, whichmeans that some other tSA has already completed executing the .SELECT routine, and willcontinue to the next step viz. executing the record routine for each contract.



16/56

A question which you should try and answer at this juncture

Q. What is the output of a select statement?

A. A list of contract ids that are selected from the database.

These contract ids are written into a record of a LIST FILE. During the selection process,

there is a call to a core T24 routine called BATCH.BUILD.LIST. This routine decides

how to write data into the LIST FILE



17/56

1. Lets assume the select routine retrieved 5 contract ids. It called BATCH.BUILD.LIST

w c n turn create recor s n t e , one or eac contract

2. For e.g.: The multithreaded routine that we have written is FT.PROCESS.EOD.Therefore, this routine will have three parts - FT.PROCESS.EOD.LOAD,

FT.PROCESS.EOD.SELECT and FT.PROCESS.EOD.

As a result of FT.PROCESS.EOD.SELECT routine, a list of FT contract ids will befetched from the database. These contract ids will be written into a LIST FILE record.The diagrams above show you the content of each LIST File record



18/56

1. For conceptual view, the LIST FILE as a result of the previous select will look like this.



19/56

1. To improve performance we can club a group of contracts and write it into the LIST FILE

e m te y s a ue ar ers .

2. This process is called bulking

Lets go back to our example of the multi threaded routine FT.PROCESS.EOD. In the.SELECT part of the routine (FT.PROCESS.EOD.SELECT) we can tell the system how

many contract IDs do we want per row in a list record. Till now we have seen that everyrow in the LIST record has only 1 contract ID in it. We can have up to a maximum of 200contract IDs per row in a LIST record. Each contract in the row is delimited by VMs

(Value Markers). In the above example the bulk count has been set as 5

How does this improve performance?

We will understand this when we look at how contract IDs are picked up for processinglater in this section.



20/56

1. For conceptual view, the LIST FILE as a result or the previous select will look like this if

t e u num er as een set to



21/56

1. The client might want to Bulk records in the a particular job to improve performance. In

t at case, e can open t e . entry or t at o an enter t e num er ocontracts to bulk in the field BULK.NO. It also possible to specify the bulk no in the

parameter to BATCH.BUILD.LIST. However, the value configured in PGM.FILE takesprecedence. The value in this field should be numeric and between 1 and 200 as the

maximum bulk limit is 200.



22/56

Continuing from the previous slides, a tSA has completed the .SELECT routine. The

ot er t ; wa t ng on t e oc to e re ease on . . w now o ta n t. twill find out that the value in the F.BATCH.STATUS record has been set to

PROCESSING, which means that the .SELECT routine has been executed. So, boththe tSAs are ready to execute the record routine.

1. STEP 1:Both the tSAs will check if the content of F.BATCH.STATUS is set toPROCESSED, if not, it means there are contract IDs to be executed. The tSA will

update the JOB.STATUS field in F.TSA.STATUS record to 1. This value stands forProcessing contracts.

2. STEP 2: Initialise a variable, which will hold all the LIST records to be processed. The

name of this variable is FULL.LIST.

3. STEP 3: Set the JOB.PROGRESS field in F.TSA.STATUS to 4 which means selectingfrom the LIST FILE.

4. STEP 4: The tSA that executed the select, will store all the LIST record ids separated

by FM (field markers). All the other tSAs that didnt execute the .SELECT routine willperform a select on the LIST FILE, extract the LIST record ids and store them in a

variable. This is done to improve performance. Every select statement executed is anI/O on the database. So, the tSA that executed .SELECT routine need not again executea select statement to pick up LIST records from the LIST FILE.



23/56

This slide explains the working of one tSA only.

The tSA has to now extract its portion of the work and start executing the RECORD routine.

To do so the tSA-

1. STEP 1: Populates the LIST records it has to execute in a variable. This variablecontains the LIST record ids separated by FM (Field Markers). For e.g.: This tSA has to

execute LIST record 12. STEP 2: Update JOB.PROGRESS field in F.TSA.STATUS to 3. This means that the tSA

has started processing the contracts one by one.

3. STEP 3: Extracts the first LIST record id from variable.

4. STEP 4: Reads and lock the corresponding record in the LIST FILE. Remember eachLIST record contains contract IDs.

5. STEP 5: Starts a Transaction Block

6. STEP 6: Extracts the first contract ID. Executes the RECORD routine for this particular

contract ID

7. STEP 7: Checks if all the contract IDs within the particular LIST record have beenprocessed. If not, it deletes the particular contact ID from the LIST FILE. Then it ends

the transaction block and goes on to pick on next contract ID within the same LISTrecord.

However if this contract processed was the last contract to be processed in the LISTrecord, then it deletes the LIST record from the LIST FILE directly. Then it ends the

transaction block. It proceeds to pick up the next LIST record ID to process.

The important thing to note here is that the transaction block is around each contract IDinside the LIST record separated by FM (field markers). Both, the deletion of contract

IDs within the LIST record or of the LIST record itself, happens within the sametransaction block. This implies that even if the tSA dies after it has executed theRECORD routine, and removed the LIST record from the LIST FILE the changes will not

be committed till the transaction block ends.



24/56

This slide explains the working of another tSA (tSA 3)

The tSA has to now extract its portion of the work and start executing the RECORD routine.

In this case, the LIST record that the tSA has to process is 2. To do so the tSA-

1. STEP 1: Populates the LIST records it has to execute in a variable. This variablecontains the LIST record ids separated by FM (Field Markers).

2. STEP 2: Update JOB.PROGRESS field in F.TSA.STATUS to 3. This means that the tSAhas started processing the contracts one by one.

3. STEP 3: Extracts the first LIST record id from the variable.

4. STEP 4: Reads and lock the corresponding record in the LIST FILE. Remember eachLIST record contains contract IDs.

5. STEP 5: Starts a Transaction Block

6. STEP 6: Extracts the first contract ID. Executes the RECORD routine for this particularcontract ID

7. STEP 7: Checks if all the contract IDs within the particular LIST record have been

processed. If not, it deletes the particular contact ID from the LIST FILE. Then it endsthe transaction block and goes on to pick on next contract ID within the same LIST

record.However if this contract processed was the last contract to be processed in the LISTrecord, then it deletes the LIST record from the LIST FILE directly. Then it ends thetransaction block. It proceeds to pick up the next LIST record ID from ID.LIST.

The important thing to note here is that the transaction block is around each contract ID

inside the LIST record separated by FM (field markers). Both, the deletion of contractIDs within the LIST record or of the LIST record itself, happens within the same

transaction block. This implies that even if the tSA dies after it has executed theRECORD routine, and removed the LIST record from the LIST FILE the changes will notbe committed till the transaction block ends.



25/56

In case of Bulking, the concept we saw earlier, each LIST record contains a group of

contract s n every row rat er t an ust one. ese group o contract s areseparated by VM (Value Marker). In the above screenshot, the bulk count has been set

to 9.

1. When a tSA extracts contract IDs in this case, it extracts all IDs till the first FM (FieldMarker). In the above example, tSA 3 extract contract1, contract2 and contract3together i.e. till it encounters the first FM. After starting the transaction block it processes

all the 3 contracts within the same transaction block. The record routine is executedthrice (once for each contract ID) within a loop. Once the loop is over, the tSA removes

the group of contract IDs from the LIST FILE in one go.

Q. How does this improve performance?

A. As you know already, once a contract is processed, it is removed from the LIST FILE.Every removal of a contract is 1 I/O (input/output) on the disk. So, now instead of 3 write

statements (1 for each contract) on the disk only 1 takes place. This quickens the entireprocessing of COB. Thus bulking helps improve performance. However, it can be used

only if we know that the result of select statement will retrieve lots of contract IDs



26/56

Once the tSA(s) have executed all the IDs in its ID.LIST it will return to read

. . . e o ow ng steps ta e p ace.

1. Step 1: Start a transaction Block

2. Step 2: Both the tSA(s) try to read and lock a record in F.BATCH.STATUS with @ID asFLAG.ID. One tSA will be successful in doing so. If the tSA finds that the record is

locked then, it will end the transaction block and exit.3. Step 3: The tSA that succeeded in the reading the record and obtaining the lock will

check if the record contains any value.

4. If yes, delete the first line of the record and write it back to F.BATCH.STATUS. This isbecause the .SELECT routine has to be called again for the next value in

CONTROL.LIST.

5. If no, then it (tSA) will update the contents of F.BATCH.STATUS record with a stringPROCESSED. This is done to tell all the tSAs that all the contract IDs have been

processed and this Batch Job is complete.

6. End the transaction block.

The following steps are done to clean up F.LOCKING and send the control back toS.JOB.RUN, so that the next Batch Job can be picked up for execution

7. Step 4 : Start a transaction block

8. Step 5: Read and lock a record in F.LOCKING with @id as FLAG.ID. Only 1 tSA will besuccessful in doing so.

9. Step 6: Delete the record in F.LOCKING with @ID as FLAG.ID

10. Step 7: Read and lock a record in F.LOCKING with @id as LIST FILE name. Only 1 tSA

will be successful in doing so.

11. Step 8: Delete the record in F.LOCKING with @ID as LIST FILE name

12. Step 9 : End the transaction block.



27/56

1. Multiple batch records with the same batch stage

2. Randomize batch records with the same batch stage

3. Never randomize jobs

4. Ideal when multiple single threaded routines need to be executed simultaneously



28/56

1. Take a look at this table that contains a set of records from the BATCH application. For

eac o t e recor s, a atc stage as een spec e . ote t at t ey are a t e same.This being the case, when EB.SORT.BATCH is internally called to sort the records,

what EB.SORT.BATCH will do is, it will pick up all BATCH records with the sameBATCH.STAGE and will randomize them and at the same time ensuring that the

sequence of jobs within the BATCH records dont change.



29/56

1. When single threaded routines get executed, as you are aware, they will neither have a

. nor a . component. ey w on y ave one rout ne w c w conta nthe entire logic. The following are the steps that will happen internally when a single

threaded routine is executed.

1. BATCH.JOB.CONTROL (BJC) will realize that it is a single threaded routine andhence will not search for a .LOAD routine and hence no LOAD routine will getexecuted.

2. BJC running on the agent that holds a lock on F.BATCH.STATUS will write a

value SingleThreaded on to the list file

3. The agent that gets the lock on the ID in the LIST file will be the one whichexecutes the single threaded routine

4. The entire single threaded routine is within a transaction block



30/56

1. Assume that tSA 2 is executing a single threaded routine

. n agen recogn zes a s execu ng a s ng e rea e rou ne us ng e e . n ecorresponding PGM.FILE record. This field will have the name of the routine to be executed prefixed

with @. Once it does, it displays that it is executing a single threaded job along with the job name



31/56

1. tSA will update JOB.PROGRESS field to 3 in F.TSA.STATUS to denote that it is processing the

o

2. Next, it will read and lock a record in F.BATCH.STATUS with @ID as Process.Name-Job.Name-Job.Position. This ID is called FLAG.ID

3. It proceeds with list file allocation. Follows the same logic as that for a multi threaded routine.

4. Then, it checks if the content of the record is Processed or Processing or NULL. NULL denotesthat the job is yet to be executed. Processing denotes that the LIST file is ready for processingand Processed denotes that the LIST file has been processed (meaning the job is done)

5. Since, it is only now that the job is being executed, tSA2 updates JOB.PROGRESS field to 2 inF.TSA.STATUS. This denotes that contracts are to be selected.



32/56


33/56

Now that the LIST file is ready with the string SingleThreaded, it is now time to execute

t e actua s ng e t rea e rout ne. s s w at appens nterna y.

1. Read and lock the F.BATCH.STATUS record. Which ever agent gets the lock , is theone that updates JOB.PROGRESS field to 1 in F.TSA.STATUS to denote that it is

processing the contents of the LIST file. Note that the LIST file doesnt have anycontracts in it, rather it just has the value SingleThreaded written in it

2. Update JOB.PROGRESS field to 7 in F.TSA.STATUS to denote that it is executing asingle threaded routine. The agent also locks the one record in the LIST file with ID 1

and contents as SingleThreaded.

3. Start a transaction block

4. Execute the routine specified in the field BATCH.JOB in the appropriate PGM.FILErecord

5. End the transaction block. Once a transaction block ends, it deletes the record from the

LIST file. Now there are no more records to be processed.

6. Update F.BATCH.STATUS record to PROCESSED to denote that the LIST file hasbeen processed

7. Start a transaction block

8. Delete the records in F.LOCKING with ID as FLAG.ID and ID as F.JOB.LIST.

9. End the transaction block



34/56

Slide 31

s3 salamelu, 2010/02/27


35/56

It is vital to understand that the tSM is itself a background process like the tSA but the only

erence s t at t w not execute o s ut w mon tor t s. ere ore, w en weinitiate the tSM, it is as good as initiating the first tSA. This tSA is like a master agent

and will control all the other agents. At any point in time, there can be only one tSMrunning on one T24 server but we can have as many numbers of tSAs as required

on a T24 server. Therefore, TSA.WORKLOAD.PROFILE record used by the TSM

service should never contain more than one agent.

1. Execute the command START.TSM with or without the DEBUG option depending onhow you wish to start tSM (Interactive or phantom mode)

2. Once tSM is started, it will check to see which agent is available to perform the role of a

manager. For this, it scans through a file named F.TSA.STATUS to see if there are anyrecords with STATUS set to STOPPED or DEAD. If you are running COB for the first

time, then, this file will be empty.

3. Depending on which agent is available, it sets the field NEXT.SERVICE in theF.TSA.STATUS file to TSM for that particular agent.

4. Then, it internally starts the agent allocated to be the tSM (1 in this case)

5. As soon as an agent starts, the first thing that it checks for is to see if it is a tSM or a

normal agent. This, it checks based on the value in NEXT.SERVICE field.



36/56

6. Once done, it updates the F.TSA.STATUS file with the details of the agent that is

runn ng t e t . eta s suc as server name, process , t e ast t me t e agent wr teto this file, status and the current service that is being executed by the agent are

updated on to the F.TSA.STATUS file. Note that the value in NEXT.SERVICE is movedto CURRENT.SERVICE.

7. Since this agent (Agent 1) is designated to be a tSM, it starts the tSM process by

internally executing the command tSM -DEBUG

8. The next step is to build the service profile. Service profile refers to a list of services that

need to be started and the number of agents required to be started for each service. Atthis juncture, it will update the field highest agent in the TSA.PARAMATER file with the

maximum number of agents running on a server

9. Assuming that the required number of agents for COB is 2, it sets the fieldNEXT.SERVICE to COB for 2 agents (Agents 2 and 3 in this case), tSM will prompt you

to start the necessary agents

10. Apart from this, it will also update the F.TSA.STATUS file with the information aboutthese 2 agents.



37/56

Slide 33

A1 This is a hidden slide and hence will not appear in a slide show mode. This slide has been placed as thenotes page contents of the previous slide did not fit into a single slide.

Alagammai, 2008/09/30


38/56

This explains the internal working of an agent that is designated to perform a role other than

t at o a manager t

1. Execute the command tSA followed by the agent number followed by DEBUG(optional) to start an agent

2. Next, it checks if any service has been assigned to the agent (2 in this case) that has

been started by referring to the F.TSA.STATUS file.3. This internally executes the tSA program followed by the agent number

4. Checks if it has to perform the role of a tSM. This is the first check that all tSAs performwhen they are started

5. Since the field NEXT.SERVICE for agent 2 is set to COB, it realizes that it has toexecute COB

6. Then, it updates the F.TSA.STATUS file with details of agent 2 that has been started

7. Next, it internally calls a routine named S.JOB.RUN to trigger off COB processing

8. Next, it invokes a routine named EB.SORT.BATCH which is the one that will sort all jobs

in the BATCH application based on BATCH.STAGE and FREQUENCY.

9. Once done, jobs start getting processed.



39/56

This illustration will help you to understand how multiple agents help perform a job and how

a ure o a s ng e agent oes not stop

1. Each server can have only 1 tSM. tSM, when started will launch the required number ofagents for that server

2. Note that this tSM and the agents have been launched in Server A. On Server B,

another tSM has been launched, which internally launches the required number ofagents on that server

3. Let us assume that all agents are helping in executing a particular COB job. Assumethat one of the tSAs on Server A is the one which has executed the .SELECT routine

has hence has populated the contents of the LIST file

4. As you can see, all agents pick up IDs to be processed from the same LIST file thusenabling to achieve multithreading

5. If you wish, you could add in one more server(s), start agents on that server and this

agent can also participate and process IDs from the LIST file

6. Even if one of the agents crash, since all other agents are active, they will continue topick up data from the LIST file.

7. This process will continue until the LIST file becomes empty.



40/56

1. How does the tSA tell the tSM that it is alive?

2. From time to time within the flow of BATCH.JOB.CONTROL, we call a routine

SERVICE.HEARTBEAT

3. This routine takes care of updating the following fields in F.TSA.STATUS every 60sec

3.1 LAST.CONTACT.TIME

3.2 JOB PROGESS

3.3 LAST MESSAGE



41/56

1. How do we ensure that TSM is alive?

2. On every call to SERVICE.HEARTBEAT, the tSAs(any agent other than the one

allocated for TSM) checks if the TSM is alive.

3. In Phantom mode, agents re-launch the manager, if TSM has not reported for aspecified period of time(an update of the last contact time of TSM is maintained in

F.LOCKING with records key as TSM:. If difference between currenttime and last contact time as updated for TSM is greater than 120 seconds, then the tSAwill re-launch the TSM)



42/56

1. Log files get stored under a directory &COMO&

2. LIST &COMO& to get a list of log files

3. JED &COMO& to view the contents of the log file

4. Log file ID : tSA___

5. Remember

5.1 If an agent is restarted, there will be 2 log file for that agent as the log file ID is

based on date and time

5.2 Log files do not get cleared automatically

From time to time within COB processing, mainly within BATCH.JOB.CONTROL, a file

called &COMO& is updated. This file resides in the T24 Home directory (bnk.run) andcontains the log of what every tSA has done. There is one record per tSA per day in

this file. The @ID of the record in this file is tSA___.The date format is YYYYMMDD and the time format is HH-MM-SS

E.g.: tSA_2_20100324_17-32-10.

We can view the log stored in &COMO& through the JED editor. At the jshell prompt, type

JED &COMO& tSA___ to view the log of a particulartSA.

E.g.: jsh ---> JED &COMO& tSA_2_20100324_17-32-10



43/56

1.What are the different types of errors that could be encountered during execution of COB?

1.1 Errors that are caused due to jBASE level problems, like error reading a file, error

opening a file etc., are called jBASE level errors.

1.2 Errors that are caused due to O/S level problems, like inappropriate permissions ona file, memory inadequate, etc., are called O/S level errors.

1.3 Within the individual COB jobs we can raise two types of errors viz. Non FatalErrors, Fatal Errors and Critical Errors. You will look at these 3 types of errors in thenext slides.

2. What happens when a jBASE/OS level error occurs?

The agent executing COB will crash out, by either going to the jBASE debugger promptor the jsh (jshell prompt). The error has to corrected before restarting COB. The

PROCESS.STATUS and the JOB.STATUS field in the BATCH record will be 1(meaning running). There will be no record created in the application EB.EOD.ERROR

and EB.EOD.ERROR.DETAIL, this is because the error was not raised by the COB job.An error will be logged in these applications only when the error raised can be correctedbefore starting COB during the next working day i.e. the error is not critical.



44/56

1. What is a Non Fatal Error?

,

job/routine we would have called a core T24 API called FATAL.ERROR. To this routinewe can pass a parameter telling it not to fatal out. This means that the agent executing

COB will not crash out, instead in this case the agent will write the error to applicationscalled EB.EOD.ERROR and EB.EOD.ERROR.DETAIL, and then continue executing

COB. A field in EB.EOD.ERROR called FIX.REQUIRED is set to YES indicating thatthe error/errors are fixed before COB is started next day. The PROCESS.STATUS fieldand the JOB.STATUS field in the BATCH record will be 1, signifying that the current

job and therefore the process is running.

2. What is a Fatal Error?

If an error is encountered during the execution of a job then, within the individual COBjob/routine we would have called a core T24 API called FATAL.ERROR. To this routine

we can pass a parameter telling it to fatal out. This means that the agent executingCOB WILL crash out, writing the error to applications called EB.EOD.ERROR andEB.EOD.ERROR.DETAIL. However, before crashing out, it (agent) removes the

contractID from the LIST FILE and then crashes. This means that the agent can be

restarted and will start processing from where it last ended, minus the contract thatcaused it to fatal out. The PROCESS.STATUS field and JOB.STATUS field in thecorresponding BATCH record remains 1 (which stands for Running)

3. When a Fatal Error is encountered, the problem needs to be reported to the TemenosHelp Desk to get it fixed



45/56

1. When a FATAL.ERROR occurs, and you are running TSM in Interactive/Debug mode.

e error w e sp aye on t e screen an w e wr tten on t e e ore t eagent crashes out, and returns to the jshell prompt. The contract that caused the error

will also be removed from the LIST FILE. If you are running TSM in Phantom mode, theagent will still crash out however it will not be displayed on the screen. The

AGENT.STATUS enquiry will show that the agent has been stopped and new agent will

automatically be launched by TSM once the REVIEW.TIME has been reached

2. The error is written on to the EB.EOD.ERROR and EB.EOD.ERROR.DETAIL file. Ifthere are multiple agents running, it will start COB from where this agent left off. You will

have to manually start the agent again in Interactive Mode.



46/56

Q. How do you tell T24 that a job is a CRITICAL job?

A. Set the ADDITIONAL.INFO field in PGM.FILE entry for that job to .CRITICAL

1. What happens when a critical job crashes?

The error is treated as a Critical Error.

If an error is encountered during the execution of a critical job then, the agent executing

COB will crash out in this case and return to the jBASE prompt (In Interactive Mode). Inphantom mode you will be able to see that the agent has stopped in the AGENT.STATUSenquiry output. It (the agent) DOES NOT write the error to applications called

EB.EOD.ERROR and EB.EOD.ERROR.DETAIL, DOES NOT remove the contractID fromthe LIST FILE and then crashes. The PROCESS.STATUS field and JOB.STATUS field in

the corresponding BATCH record is updated to 3 (which stands for Error/Hold). This meansthat if a critical job fails, the cause of error needs to be corrected immediately before COB is

restarted again.



47/56

1. Repeated crashes at system level i.e., crashes in LOAD and SELECT , will stop the

serv ce un e e ore w en cras es n rout nes w not stop t eservice rather other agents will try and complete this task i.e. repeated crashes at

system level was not detected by tSM and it tried to achieve the task through otheragents.

2. Repeated crashes on the same job within a time interval stops the service. Duration andthe number of crashes can be configured at TSA.PARAMETER table through fields

(STOPPAGE TIME & STOP COUNT Number of times the crash happens for a servicei.e. for the agents within a specified time).

The first time the agent is marked as DEAD, TSM updates a record with key as

-STOP in F.LOCKING with the stop time and stop count as one. If notthe first crash, then the time difference between the first stop time and the current time is

checked and if it has exceeded the stop time period specified in TSA.PARAMETER,then the current time is updated as the first stop time. Otherwise if the stop count hasexceeded the limit specified, then the service is stopped.



48/56

1. Date specifies the bank date on which the service was started.

2. Started specifies the time on which the service was started, in the format DD/MM/YYYY

HH:MM:SS

3. Stopped specifies the time on which the service was stopped, in the formatDD/MM/YYYY HH:MM:SS

4. Elapsed specifies the elapsed time of the service, i.e., the difference between the startand stop time for the service in seconds.

5. Transactions specifies the number of transaction processed by the service

6. Stoppage Time specifies the time interval for a service to be monitored for the numberof crashes allowed as specified in the field STOP.COUNT. If no value is specified here,the value is taken from TSA.PARAMETER. The time given in this field must be grater

than TIME.OUT/DEATH.WATCH.

7. Stop Count specifies the number of crashes allowed for a service in a time period asspecified in the field STOPPAGE.TIME. If no value is given for this field, the value is

taken from TSA.PARAMETER

e.g., STOPPAGE.TIME = 100 STOP.COUNT = 3

Here the Service can only crash 3 times in a time period of 100 seconds. At the fourthcrash within 100 seconds, the service will be stopped.



49/56

1. Fatal errors(both Online and during COB) provide more useful information.

2. Call stack & core dump are written to a new file EB.SYSTEM.STATUS along with a

transaction reference. Call stack will contain all calls made before the crash happened.

3. The above changes will be packaged as a part of TEC framework.

4. Key to be transaction reference & unique number.

5. All variables and their values at the time of crash are also stored.

6. The core dump uses jBASE system functions to obtain the required information.



50/56

1. What happens when a tSA crashes while processing the LOAD routine?

Ans. Since there are multiple tSAs executing the LOAD routine, if this tSA fails another

tSA will execute it anyways

2. What happens when a tSA crashes just after executing and populating the LIST file

Ans. The LIST FILE to be used is written onto F.LOCKING for a particular job, therefore

even if this tSA crashes, other tSAs will read F.LOCKING and pick up the LIST FILE tobe used

3. What happens if a tSA crashes while processing the SELECT routine?

Ans. The SELECT routine is executed only by one tSA and is wrapped around jBASEtransaction management. So, if the tSA crashes while doing the select any partialupdates will be rolled back. The other tSA waiting for the lock to be released on

F.BATCH.STATUS will perform the select

4. What happens if a tSA crashes while executing a contract ID in the LIST record?

Ans. Each contract ID(s) being processed in the LIST file is wrapped around atransaction block, therefore any failure in tSA will result in a rollback. The contract ID will

not be removed from the LIST record. Any other agent will pick up that particular LISTrecord and execute the contract ID(s).



51/56

So, since the TSM is still running, it will prompt the user to start the 3 agents required to run

.

The user starts the first tSA. It (tSA) calls EB.SORT.BATCH. This in turn readsF.BATCH and picks up all the processes with PROCESS.STATUS ready (0) or

running(1). So, it leaves out the process BNK/PL.CLOSE.EOD as itsPROCESS.STATUS is already 2 (completed).

It reads the JOB.STATUS of Job7 and finds that its set to 2. This means that Job7 hasbeen completed successfully.

The tSA then executes the Job8.LOAD routine and tries to read a record in F.LOCKING

with @ID equal to FLAG.ID. (Process.Name-Job.Name-Multi value position of that jobwithin the process, e.g.: BNK/FT.EOD-Job8-2). It finds out that such a record exists as

the previous tSA (that crashed) already wrote this info into F.LOCKING file.

The tSA now has the name of the LIST FILE into which the LIST records should bepopulated into. E.g.: F.JOB.LIST.2

It now tries to read a record in F.BATCH.STATUS with @ID equal to FLAG.ID. However

such a record does not exist as the previous tSA crashed before writing the informationinto F.BATCH.STATUS.



52/56

The tSA gets the lock on F.BATCH.STATUS and executes the Job8.SELECT routine.

y t s t me t e ot er t s are a so starte t en t ey w wa t t t s t s oneexecuting the Job8.SELECT routine, as this tSA is holding a lock on F.BATCH.STATUS

It will execute the Job8.SELECT routine, populate the LIST records into the LIST FILE,

F.JOB.LIST.2

Suppose as a result of Job8.SELECT , 90 contract IDs are picked up from the databasethen the LIST FILE will have 90 LIST Records each record having 1 contract IDs storedin it.

Write the updated information into F.BATCH.STATUS with value of

VMPROCESSINGVM90VM90 and end the transaction block.

Now the other tSA(s) waiting for the lock on F.BATCH.STATUS to be released, will readthe information in it and find that the value is set to PROCESSING. This means that

.SELECT routine has been executed. All the tSAs will now proceed to execute theRECORD routine.



53/56

Slide 48

A2 This is a hidden slide and will not appear when run in slide show mode. This slide has been placed asthe notes page in the previous slide did not fit into 1 slide.

Alagammai, 2008/09/30


54/56

1. False

2. False

3. True

4. False

5. True



55/56

In this learning unit, you learnt about the internal working of COB

You will now be able to:

1. Understand the internal working of COB

2. Understand the working of tSM & tSA3. Visualize failure scenarios

4. Understand the types of errors generated during COB



56/56

cob2[1].close of business – batch.job.control, errors-r10.01.pdf

Documents