Unit 1a - Overview
Unit 1a - 1
© 2013 IBM CorporationIBM Advanced Technical Skills
IBM System z and z/OS
Java Batch SolutionsVersion: September 10, 2013
IBM System z and z/OS
Java Batch SolutionsVersion: September 10, 2013
IBM z/OS Java Batch Solutions
2
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD2
Version: September 10, 2013
This page intentionally left blank
IBM z/OS Java Batch Solutions
3
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD3
Version: September 10, 2013
Agenda / Table of Contents
Material Available Online …
Overview
Java on z/OS
JVM Launcher Solutions
WebSphere Java Batch
Java Batch in CICS, DB2
Batch and Decision Management
Open Standard Java Batch
This is the agenda for the session.
IBM z/OS Java Batch Solutions
4
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD4
Version: September 10, 2013
This Presentation Available Online
Overview …
ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783
PDF in Chart FormatShowing the charts as you're seeing them here
PDF in Speaker Notes FormatSame charts, but printed so the speaker notes associated with each chart appear.
A quick reminder that the presentation is available online under "WP101783" on Techdocs.
Two formats are provided -- chart format and speaker note format. The speaker notes provide a further
explanation of the key points being made on each chart.
IBM z/OS Java Batch Solutions
5
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD5
Version: September 10, 2013
OverviewSetting the stage for the discussion of modern batch solutions
IBM z/OS Java Batch Solutions
6
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD6
Version: September 10, 2013
Batch (or Bulk) Processing
Examples of bulk processing …
Many definitions exist ... they all have in common the relative lack of human interaction, and the expectation of results at a future time rather than immediately:
Request
Response
Request
Response
In general:
● Interaction is one-for-one ... that is, request with matching response
● Expectation is for response to follow request in a near-immediate span time frame
Online Processing
Submit
Complete
Start
EndProcess
In general:
● Interaction is one-for-many... that is, initial request results in many results from processing
● Expectation is for results to finish within some determined non-immediate time frame
Bulk Processing
We start with a compare-and-contrast between online processing and the focus of this presentation, batch
processing.
The point of contrast we wish to focus on is the interaction between the request and the processing that takes
place. Online processing has, in general, a one-for-one relationship. In addition, there is an expectation of the
response being returned in a timely fashion.
Batch processing on the other hand has, in general, a one-for-many relationship between the request and the
result. The request initiates some background processing against a set of data. The result is the completion of
the processing against the data. The expectation is that the processing will be completed in some timely but non-
immediate fashion.
This is why we introduce the term "bulk" to this discussion. The focus is on processing bulk data in some form.
IBM z/OS Java Batch Solutions
7
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD7
Version: September 10, 2013
Some Examples of Batch (or Bulk) Processing
What's Behind This? …
Just to set some context for the upcoming discussion
Month-End Tax or Fee Calculation and BillingCustomer records are processed with tax or other calculations processed against activity. This may be part of a larger process of calculating amount owed and formulating billing.
Period-End Statements and ReportsAn example is investment portfolio results and reporting.
Data TransformationRaw data records are processed with goal of transforming some aspect of the data content or layout. Result is a second set of data with the intended formatting.
Data AnalysisData records are analyzed to determine trends or patterns. Data mining to find new potential markets is one example. Analyzing vast quantities of seismic data is another.
Point is these are activities where processing in bulk is a better way to maximize efficiency of data access and computer resources. The
completion time is determined but not immediate.
Here we offer a few examples of batch processing.
Note: conduct a short discussion of batch processing examples seen in the environment of the presentation
audience. That will allow the discussion that follows to relate to the specific examples.
Ultimately the point of bulk processing is that it allows more efficient use of people and data processing equipment. The nature of the processing does not demand an immediate response (though, as the chart
indicates, the completion time does need to understood and predictable).
Batch processing exists because the nature of the work suggests doing the work in bulk. It's one part of the
broader spectrum of overall information processing.
IBM z/OS Java Batch Solutions
8
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD8
Version: September 10, 2013
What's Behind This?
Approaches to Batch Modernization …
Every business has different motivators. The common ones we've seen:
Batch Window CompressionThe window in time for batch processing is shrinking. There's a need to better manage online and batch processing concurrently within the same system.
Java Skills and Common ToolingJava skills are more common that traditional programming skills. Further, Java tooling for online work is powerful and capable of being used for batch programming as well.
GP cycles and Offload to Specialty EnginesCost pressures are creating a need to explore ways of offloading processing cycles from GP to specialty engines such as zAAP or zAAP-on-zIIP.
Other motivators may exist. The key is that these motivators are real and they are driving exploration of modern batch.
The movement towards Java batch on the mainframe has several components to it. The points made on this
chart represent some of the more common things we hear.
Note: A discussion with the presentation audience on the business motivators behind their investigation of Java
batch is recommended. There may be business drivers not outlined here. That will allow better discussion of the
relevance of the technology to the business needs as the presentation unfolds.
The three issues outlined on the chart are:
● Batch Window -- in the past the "batch window" was a dedicated time during which batch processing was
done and online processing was suspended. But those days are largely behind us -- online is 24 x 7 and in
general batch must be done concurrent with online.
Note: with proper capacity planning and system goal management it's quite possible operate COBOL batch concurrent
with online processing. So there's nothing magical about Java that allow concurrent processing. The point here is that
Java batch processing within the same runtime framework as Java online processing facilitates concurrency a bit better.
That's one of the key value statements of WebSphere Java Batch.
● Skills and Tooling -- COBOL is an excellent programming language, and very good development tooling
exists for it. But in today's world the prevalence of Java skills exceeds COBOL skills, and the trend continues in that direction. Further, many have very large investments in Java development tooling and the desire is to
leverage that tooling investment into the world of batch development as well.
● CPU Offload -- batch executing in COBOL runs on general processors, and general processor cycles are
seen as having a much higher cost profile than System z specialty engines such as zAAP, or zAAP-on-zIIP.
So the motivation here is to reduce the cost profile of batch processing by moving some batch to Java.
Note: the question of what batch processing to consider a candidate for Java batch is an important discussion
to have. Many factors play into the decision -- Java for new batch vs. rewrite existing; and if rewrite, then the
complexity and inter-dependencies of the existing. The complexity / inter-dependency question gets to the
cost and risk profile of a rewrite project. In general, when starting out, tackling complex batch that's on the
critical path is not recommended.
IBM z/OS Java Batch Solutions
9
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD9
Version: September 10, 2013
Approaches to "Batch Modernization"
Framework for discussion …
Generally speaking, we see two basic approaches:
Preserve Existing ... Java Batch for NewLeave existing batch processes as they are today, but as new requirements come up then engineer them into the Java batch model
Re-Engineer Some Existing Batch ProcessesIdentify existing non-Java as candidates and then re-engineer them to operate in a Java batch environment
Typical starting approach: identify batch processes with fewer interdependencies and then work out from there
What we don't see is a "rip-and-replace" strategy. That's too costly and too risky.
Always a reasoned incremental approach.
Others who have understaken the "batch modernization" exercise have tended to approach this in one of two
general ways:
1. Java for new batch -- that is, fence off existing batch processes and preserve them as they're operating
today, but make a point to engineer any new batch processes using Java.
2. Incremental Re-engineering -- which implies identifying candidates for rewrite to Java and then undertaking
a software development project to perform the re-write and test of the new code.
With respect to the second approach, we find most approach this with a reasonable degree of caution: the initial
candidates are typically batch processes with relative few interdependencies, and batch processes off the batch
critical path. When skills and experience are gained, then other batch processes are worked on.
IBM z/OS Java Batch Solutions
10
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD10
Version: September 10, 2013
Framework for Discussion That Follows
Java on z/OS …
The following diagram illustrates the points we will touch on as we review the modern batch technologies available on System z and z/OS:
Modern Batch Technology
What it isAn explanation of its intended purpose
How it worksAn explanation of its functions and features
DifferentiatorsA discussion of how the technology compares to others discussed, and the particular value this technology offers
How it takes advantage of System zAn explanation of System z and z/OS exploitation that may occur with the technology
How it integrates with existing processesAn exploration of integration and usage patterns when other components in your overall architecture are involved
This chart provides an understanding of the discussion framework to follow.
Note: this presentation was structured on the premise that what is desired by the audience is an understanding of
the technology and help with the positioning considerations between technologies. That's what allows for an
informed decision. This presentation is designed to be a reasoned walk-through of the technology with points
along the way to discuss the potential business value.
IBM z/OS Java Batch Solutions
11
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD11
Version: September 10, 2013
Java on z/OSA story of taking advantage of the platform
IBM z/OS Java Batch Solutions
12
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD12
Version: September 10, 2013
IBM Java on z/OS
Specialty Engines …
It's important to understand that while the Java APIs are industry standards, the implementation below the APIs becomes increasingly platform-aware:
ProgrammingAPIs
High-Level JVM Functions
Low-Level JVM Functions
SDK conforms to the accepted standards● IBM SDK provides all the required APIs according to the
specification at the level being discussed● IBM z/OS SDK provides additional APIs to take advantage
of z/OS platform specific functions (such as SAF security)
JVM Functions common across IBM SDKs● The JVM is entirely IBM's ... first delivered in 2005● Many features: generational GC, shared classes● High-level JVM functions common across IBM Java
System z and z/OS functions● Takes specific advantage of platform, including exploitation
of new CISC instructions available with new System z: z10, z196, EC12
● Big Decimal, Large Page, Out of Order execution, transactional execution, flash paging ... equals performance
● Work with z/OS dispatcher to offload to specialty engines
There's a slight tension in the Java story that needs to be understood before going into technical detail -- that is,
from an application perspective Java is common across all platforms, but below that application interface line the
Java implementation is very much platform aware.
So three broad points are made on this chart:
1. IBM Java, regardless of platform, is Java ... it complies fully with the Java specification.
2. The implementation of the Java Virtual Machine (JVM) under the application specification level is fully IBM's design and development, based on IBM's many years of experience in operating system and virtual OS
environments.
3. IBM Java on z/OS is platform aware and takes advantage of specific platform features to provide either
additional function or improved performance.
The first point is important because it establishes IBM Java as being specification compliant ... applications are written to the particular Java specification level, not to some IBM version of Java. Java is Java at the API spec
level.
The second point is important because it establishes the JVM implementation as being something IBM has
designed, developed and continues to improve. Provided the application specification interfaces are compliant
(which they are), the underlying implementation may be done however a vendor chooses. IBM's JVM is based on
IBM's design.
The third point is important because helps to address the question of "Why run Java on z/OS?" The answer is
because (a) it performs very well, due in part to the specific exploitation of System z and z/OS features Java on
z/OS does, and (b) it offloads to specialty engines.
The rest of this section expands on these key points.
IBM z/OS Java Batch Solutions
13
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD13
Version: September 10, 2013
System z Specialty Engines
Timeline …
Specialty engines provide additional processing capacity with an attractive financial profile: lower acquisition cost, not counted towards software license charges
Java Work1 DDF and DB2 Utilities2 Other Work
z/OSDispatcher
zAAP Processors zIIP Processors General Processors
zAAP - System z Application Assist ProcessorOffload of Java and XML parsing work.
zIIP - System z Integrated Information ProcessorCertain DB2 work and XML parsing services.
Note 2 -- Plus other work, see http://www.ibm.com/systems/z/hardware/features/ziip/
Note 1 -- See http://www.ibm.com/systems/z/hardware/features/zaap/
zAAP-on-zIIPA means of more efficiently using specialty engines by defining only a pool of zIIP processors and allowing eligible zAAP work to run on the zIIPs3.
zAAP-on-zIIP
Note 3 -- EC12 planned to be the last system that supports zAAP; after that, zAAP-on-zIIP will be the offload mechanism
Note: simplified representation of
offload mechanism. Various tuning and control parameters exist.
IFL - Integrated Facility for LinuxFor running z/VM and Linux. Does not apply to z/OS, but plays strong role in Linux for System z
We start this discussion with a high-level explanation of System z "specialty engines" for those who may not have
a background in the topic. The ability to offload Java work to specialty engines is one of the key motivators to
using Java for batch on System z.
Specialty engines provide a means of dispatching work running on z/OS to engines other than the general
processor engines. That provides two key benefits:
● It allows work that must run on general processors (such as CICS, or COBOL batch) to have capacity on
those engines by not using GP cycles for Java, and
● It allows Java work to run on engines that do not count towards software licensing charges.
The net effect is that adding Java work to a z/OS system does not necessarily imply having to acquire addition GP
engines, which then implies potential increases in software licensing charges. It is possible that Java work can
have a cost-neutral effect (aside from the acquisition cost of the specialty engines, which are priced substantially
below GP engines).
Note: the dispatching of Java work to available specialty engines is transparent to the Java applications. The
decision where to dispatch is done at a level lower on the system ... down at the interface between the JVM and
the z/OS dispatching code.
There are several flavors of specialty engines and those are discussed on the chart above. With respect to Java
the key is zAAP, or zAAP-on-zIIP. (The EC12 machine is the last to support zAAP alone; going forward it will be
zAAP-on-zIIP, which is a means of consolidating non-GP work on a pool of specialty engines rather than having to
separately allocate zAAP and zIIP engines. In short, zAAP-on-zIIP is a good thing.)
In summary, the existence of specialty engines on z/OS is a key enabler for Java work. It is a factor often cited by
customers as a factor that plays a role in their decision to host Java work on the mainframe.
IBM z/OS Java Batch Solutions
14
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD14
Version: September 10, 2013
IBM System z Java Product Timeline
Performance …
This shows the level of commitment to Java and Java on System z:
SDK 1.1.1● First OS/390 Java product● Followed by 1.1.4 and 1.1.6● 31-bit
SDK 1.1.8● 31-bit
SDK 1.3.1● 31-bit● z/OS ● Linux for System z
SDK 1.4● 31-bit z/OS● 64-bit Linux for System z
SDK 1.4.2● 64-bit z/OS● First use of IBM J9 VM
and JIT Technology
SDK 5● 31-bit and 64-bit● z/OS and Linux for System z● z9 exploitation
SDK 6.0.0● Java SE 6 APIs● 31-bit and 64-bit● z/OS and Linux for System z● z10 exploitation
SDK 6.0.1● Java SE 6 APIs● 31-bit and 64-bit● z/OS and Linux for System z● z196 exploitation● Enhanced JZOS and z/OS security
SDK 7● Java SE 7 APIs● 31-bit and 64-bit● z/OS and Linux for System z● More z196 exploitation● zEC12 exploitation● OpenJDK
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
This is a busy chart, but don't let the busy nature of the chart distract you. The chart is telling a very good story.
What this chart represents is the time line of improvements and enhancements to the Java on System z product
since its beginning back in late 1997. From that point forward it is a fairly steady set of improvements ...
improvements in specification level supported and improvements in platform exploitation.
Note: this chart is included to establish the important point that IBM invests in the development of Java on the
System z platform. Java on System z is designed to be fully aware of the platform on which it runs and take
advantage of features and functions found there. Ultimately that translates into performance, as we'll see on the
next chart.
IBM z/OS Java Batch Solutions
15
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD15
Version: September 10, 2013
IBM z/OS Java Performance
Summary …
The story is one of improvement in hardware and Java software
zEC12 SDK 7 SR3
zEC12 SDK 7 SR1
z196 SDK 7 SR1z196 SDK 6 SR8
z10 SDK 6 SR4
z10 SDK 6 GM
z9 SDK 5 SR5
CPU Intensive Multi-threaded Workload on 16-way Machine
Better scaling slope leading up to CPU
over-commit
Around 12X improvement
CPU intensive workload so scaling
past over-commit not expected
Four things going on: (1) faster hardware, (2) exploitation of machine instructions, (3) exploitation of machine features (Flash Express paging with large pages), and (4) increasing efficiency of the JVM function itself
Disclaimer: controlled environment with
specific conditions during measured testing. Results may vary. This is not a promise of performance or results.
-Xaggressive and large page
This picture comes from another presentation related to Java performance on z/OS. The chart is showing the
performance improvements that have been achieved from the starting test point (IBM Java SDK 5 on a System z
z9 machine) to the most recent (SDK7 on zEC12 machine). The test design charts the performance for the
workload from 1 processing thread to 32 on a 16-way machine.
Notes: the various lines can make this chart a bit confusing at first. The main point we're making with this chart is
the performance improvement seen over time as the SDK levels have improved and the machine hardware has
improved. That's the two blue circles connected with the dashed line.
The workload for this test was very CPU-intensive, which meant that a dispatched thread would consume a CPU
pretty much exclusively while it ran. That's why the scaling goes up from 1 to 16, but then tapers off from 16 to 32
threads. We focus on the 16-thread data points because of the nature of this test workload.
Performance has improved over time ... 12 times from lower data point to higher data point. That's due to the four
factors outlined on the chart. This bears out the implied message of the previous chart -- that IBM has invested in
the SDK on z/OS, and the effect of that manifests as improved performance.
IBM z/OS Java Batch Solutions
16
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD16
Version: September 10, 2013
Section Summary
JVM Launchers …
A very broad summary of the Java on z/OS section:
IBM Java is fully compliant with Java specificationsProgramming APIs compliant with the specification
IBM Java on z/OS has programming extensionsProvide you option to code apps to extensions for added functionality
IBM Java has IBM JVM, GC and JIT TechnologyBased on decades of experience in operating systems
IBM Java on z/OS takes advantage of System zWhich manifests as performance improvements over time
IBM continues to invest in Java on z/OSCommitment to the platform is evident
We summarize this section with the points made on this chart.
Note: the objective of this section was to establish the viability of Java on System z and z/OS, and to overcome
objections that may have been based on experiences from many years ago. IBM Java on z/OS in 2013 is not like
Java on z/OS from 1999. Considerable progress has occurred in the years since initial release.
IBM z/OS Java Batch Solutions
17
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD17
Version: September 10, 2013
JVM Launcher SolutionsBPXBATCH, JZOS and the Batch Runtime
IBM z/OS Java Batch Solutions
18
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD18
Version: September 10, 2013
What is a "JVM Launcher?"
Three JVM Launchers on z/OS …
It's simply a means of initiating a JVM start and the running of Java programs in it:
Four questions arise:
1. What JVM launcher technologies are available on z/OS?
2. What does each provide in addition to the JVM?
3. Why would one be used over another?
4. If not JVM launcher, what else might be considered?
Java Program
Java Virtual Machine
Instantiation JVM tear-downProgram Execution
Launch the JVM
In this section we will discuss JVM Launcher technologies. JVM Launchers are designed to instantiate a JVM and
run a program, then take the JVM down after the Java program has completed. In this section we will discuss
three such technologies available on z/OS.
Note: later we will discuss an alternative approach, which is to use a JVM that stays active after the Java work
completes. That approach avoids the overhead of instantiating the JVM and tearing it down time and time again.
In this section we will seek to answer the four main questions shown on the chart (along with any other questions
that may arise during discussion).
IBM z/OS Java Batch Solutions
19
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD19
Version: September 10, 2013
Three JVM Launcher Technologies We'll Explore
BPXBATCH …
For the purposes of this discussion we'll focus on three:
BPXBATCHProvides a UNIX shell environment to JCL. From the UNIX shell you may then issue a "java" command to launch the JVM and run the specified program.For purposes of comparison, consider this equivalent to invoking "java" from a shell script or manually from a command line.
JZOSProvides a more comprehensive JVM launcher utility that overcomes some of the shortcomings of BPXBATCH. Includes a set of class libraries and JNI native to access z/OS functions and services.
z/OS Batch RuntimeProvides a functional subset of the WebSphere Java Batch environment for running Java batch programs. Includes such things as transactional management, integration with COBOL, and transactional integrity between Java, COBOL and DB2.
These are the three JVM Launcher technologies. All three are functions included with the z/OS operating system.
The primary difference between the three is the Java batch functionality each provides. The BPXBATCH solution
provides essentially none beyond what the JVM itself provides. JZOS expands on that with a toolkit for access to
z/OS services above and beyond what a simple JVM provides. The Batch Runtime provide more still -- a runtime
in which Java can interact with COBOL and share a JDBC Type 2 connection with transactional coordination.
IBM z/OS Java Batch Solutions
20
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD20
Version: September 10, 2013
BPXBATCH (and BPXBATSL)
JZOS …
Provides access to a UNIX shell environment from which you may instantiate a JVM and run a program. It works:
http://publib.boulder.ibm.com/infocenter/zos/v1r13/topic/com.ibm.zos.r13.bpxa500/bpxbatr.htm
//BPXBATCH JOB ...
//STEP EXEC PGM=BPXBATCH,
// PARM='SH java HelloWorld'
//STDIN DD DUMMY
//STDPARM DD DUMMY
//STDERR DD SYSOUT=*
//STDOUT DD SYSOUT=*
//STDENV DD *
CLASSPATH=/u/myclasses
/*
//*
JVM instantiated from shell with java command, with Java class file specified.
Parameters to Java program may be specified trailing class name in PARM=' ' or in location specified by STDPARM DD.
If STDIN DD specified, it must be a z/OS UNIX file.
STDERR and STDOUT may be JES spool, or UNIX file, or MVS data set
BPXBATSL is like BPXBATCH, but it allows child processes to be fork/exec'd within the same address space. (Restrictions apply, so see InfoCenter article below.)
It provides a JVM with some basic access to z/OS DD statements. It has limitations with respect to other z/OS services. That's why
JZOS was developed and IBM provides that with z/OS.
The first technology is BPXBATCH, which is a function provided with the UNIX Systems Services (USS) function
of z/OS. It is designed to provide a UNIX shell environment to JCL batch. That then provides you the ability to
launch the JVM and pass in a parameter that names the Java class file to execute.
It works, but it doesn't provide much in the way of additional services to your Java program other than access to
UNIX files for input and output. That means if your Java batch program seeks to do more sophisticated batch
processing -- such as transactional management and checkpointing -- that would the responsibility of your Java
code. That implies writing custom code above and beyond the business logic of your batch processing. The
desire to avoid such custom middeware code is precisely why IBM offers other Java batch solutions.
BPXBATSL provides essentially the same thing as BPXBATCH but avoids multiple address spaces when child
processes are forked.
Summary: BPXBATCH works, but nobody seriously advocates it as the solution for enterprise Java batch
processing. It's acceptable for small-scale ad hoc Java invocations but is not robust enough for larger-scale
processing. It's included in this presentation mostly to put a stake in the ground for the low end of the Java batch
processing scale.
IBM z/OS Java Batch Solutions
21
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD21
Version: September 10, 2013
Overview of JZOS
JZOS Toolkit …
JZOS is technology originally developed by Dovetail Technologies and acquired by IBM. It is now supplied as part of z/OS.
http://www.ibm.com/systems/z/os/zos/tools/java/products/jzos/overview.html
JVM Launcher
Toolkit
JZOS
Launch a JVM and run a Java program● One address space; all accounting data attributes to the job or started task● Mix Java and non-Java steps in same JCL● Pass condition codes between Java steps, or between Java and non-Java● Access MVS data set via DD statements in Java
Toolkit provides Java classes and native libraries● Methods for allocating, deleting, and renaming MVS data sets● Interface for reading and writing sequential data sets in record mode● Methods for issuing single and multi-line WTOs● Interface for invoking DFSORT and directing I/O into/from Java applications.● Java APIs for submitting z/OS batch jobs from Java● Java APIs that provide interface for invoking IDCAMS (VSAM)● Java APIs for many z/OS functions:
Obtaining job/step/user names, process ids, etc. Writing SMF records Reading environment variables
A more flexible JVM launcher mechanism along with a set of class and native libraries which provide Java programs access to z/OS services
Because of the limitations of BPXBATCH for running Java, the people at Dovetail Technologies developed the
JZOS function. IBM has since acquired the technology and now supplies it with the z/OS operating system. See
the URL at the bottom of the chart for the IBM web page that discusses JZOS.
JZOS provides two key things as part of its function:
1. A JVM launcher that overcomes some of the shortcomings of BPXBATCH
2. A set of Java class libraries and native z/OS libraries to provide program API access to z/OS services
Note: JZOS having two functional components -- improved JVM launcher and service access toolkit -- is not
immediately obvious. Please do not assume the audience is aware of this distinction. The distinction is important because the two may be used independently from one another:
● The JVM launcher component may be used even if none of the toolkit services are needed by Java program
you are running. So JZOS may be meaningfully used for simpler Java batch programs.
● The JZOS toolkit is not limited to just JZOS usage. It consists of a set of class libraries and native libraries,
which other Java runtimes may exploit. Keep that in mind when we discuss the Batch Runtime and the WebSphere Java Batch section.
Without delving deep into the details of JZOS at this point, the key point is that JZOS represents an improved means of launching a JVM and running a Java program. It is far more flexible in its design than BPXBATCH, and
provides a valuable means of mixing Java and non-Java in a multi-step JCL. It's part of z/OS so no additional
licensing is required. It is still a JVM launcher, so if your Java batch requirements call for many repeated
invocations you need to consider whether the overhead of all those JVM create/destroy cycles is worth it, or if you
should go to a model where the JVM stays active (WebSphere Java Batch).
IBM z/OS Java Batch Solutions
22
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD22
Version: September 10, 2013
JZOS Toolkit Usable Outside JZOS Launcher
JZOS Launcher …
The JZOS JVM launcher and the JZOS toolkit are not tightly coupled. The toolkit services may be used by other Java Batch mechanisms:
WebSphere Java Batch
z/OS Batch Runtime
Toolkit Classpath and Libpath updates
z/OS Services Offered by JZOS Toolkit
We've not yet covered WebSphere Java Batch or z/OS Batch Runtime
Key point is JZOS toolkit and those offerings are not mutually exclusiveAnd as we'll see, the z/OS Batch Runtime in fact uses the JZOS JVM launcher as well
This chart illustrates the point we made in the previous chart's speaker notes -- that is, the JZOS toolkit consists of
a set of Java class libraries and z/OS native libraries that may be used by other Java runtime environments.
The z/OS services made available by the JZOS toolkit may be of great value to your Java batch plans. But you
may need to use one of the other runtimes. That's okay ... they work together.
Consider the JZOS toolkit as a set of utility services that runs with JZOS and other Java runtimes.
IBM z/OS Java Batch Solutions
23
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD23
Version: September 10, 2013
z/OS JZOS JVM Launcher
Multi-Step Jobs …
It comes with the z/OS SDK installation. Sample JCL is provided along with the JZOS load module. Invocation involves customizing sample JCL:
/usr/lpp/java/J7.0/mvstools
JVMLDM70
/samples
/jcl
JVMJCL70
JVMPRC70
Note: JZOS is specific to the Java version and the bit-mode. Here SDK 7 31-bit is shown.
● Load module -- copy out to LIBRARY data set
● Calling JCL -- this JCL calls the proc and passes variables
● Procedure -- this receives variables and invokes JZOS
JCLJCLNames the Java class to invoke
Sets up the environment variables
Provides STDIN DD (optional)
Passes arguments (optional)
PRCPRCReceives Java class and arguments
Calls JZOS load module and specifies the Java class file and any passed-in arguments
Defines standard DD statements
Starts address space, instantiates the JVM,
loads the class, invokes the main()
method and supplies any arguments
This is chart that shows how JZOS is delivered. The key is that it's supplied as part of the z/OS Java SDK and it
comes with sample JCL you customize for your environment.
Note: this is standard z/OS JCL system programmer structure. There's nothing mysterious about it at all. JZOS
is supplied as a load module that is copied out to a LIBRARY data set, and JCL in two forms: JCL for the
submitted job, and a JCL procedure that is called from the job JCL. Your specific Java job information is coded in
the job JCL, and that is passed over to the proc, which is generic and receives the customized values as passed
arguments from the job JCL.
It's also worth noting that the JZOS load module is specific to the SDK level and bit-mode in use. So if you have
need for both 31-bit and 64-bit JVMs, then you would copy out the load modules and JCL from the 31-bit and 64-
bit directories of the z/OS Java SDK you wish to use.
The JCL launches the JZOS address space and instantiates the JVM within it. Then it loads the Java class you
named and invokes the main() method. When your Java completes the JVM is torn down and the address
space is shut down.
IBM z/OS Java Batch Solutions
24
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD24
Version: September 10, 2013
JZOS and Multi-Step Batch Jobs
Batch Runtime Overview …
JZOS allows you to mix Java and non-Java as steps in a larger batch job. Condition codes set by each step available to other steps in the job.
//MYBATCH JOB ...
//STEP1 EXEC PGM=COBOL1
: (step DD cards)
//STEP2 EXEC PROC=JVMPRC70,
// JAVACLS='com.ibm.MyClass1'
: (step DD cards)
//STEP3 EXEC PGM=ASSEMBLER
: (step DD cards)
//STEP2 EXEC PROC=JVMPRC70,
// JAVACLS='com.ibm.MyClass2'
: (step DD cards)
One job
Multiple steps
Mixed programs
One address space
Same WLM policyService Class (goals)Reporting Class (statistics)
Accounting data consolidated under one job
COBOLCOBOL
JavaJava
AssemblerAssembler
JavaJava
The purpose of this chart is to show how JZOS can be used in cooperation with non-Java in a multi-step JCL
batch job. JZOS provides the ability to do this with the attributes shown on the right side of the chart. You could
code BPXBATCH on a step, but it would run in a separate address space, operate under different WLM policies,
and be accounted for separately.
So JZOS represents a step forward from BPXBATCH for invocation of Java main() program execution. In
general JZOS is preferred over BPXBATCH for Java execution. (BPXBATCH is still very good for general UNIX
shell work; JZOS is for Java only ... it can't be used to invoke a shell script for example.)
Is JZOS the final answer? Maybe. Maybe not. It depends on what your Java batch processing requirements are.
For example, while JZOS provides some really nice z/OS access services, it does not provide much along the way
of batch functions such as transaction coordination and checkpoint processing. The next function discussed -- the
Batch Runtime -- does.
IBM z/OS Java Batch Solutions
25
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD25
Version: September 10, 2013
z/OS Batch Runtime Overview
z/OS 2.1 enhancements …
This is function supplied with z/OS 1.13 and higher. It provides a means of integrating Java and COBOL with DB2 JDBC T2 connection shared and TX coordination:
Java Program
DB2 z/OS
COBOL Program● Start TX● Call DB2● Invoke COBOL● Finish TX
● Business Logic● Call DB2● Business Logic● Return
Batch Runtime Transaction and Execution Services
RRS Shared Attach
z/OS 1.13 Batch Runtime
JCLJCL
Submit
Or initialize with COBOL and have it call Java ... it works in either case
The other JVM Launcher technologies provide nothing in the way of TX services. If your code
has TX requirements, then you write custom code.
With the z/OS Batch Runtime the TX services provided and help coordinate between Java and COBOL when using JDBC Type 2 to local DB2.
z/OS 2.1 adds PL/I
The z/OS Batch Runtime is function provided starting with z/OS 1.13 that takes JZOS a bit further by providing a
batch processing framework inside the JVM that is launched by JZOS. That batch processing framework is a
subset of that provided in the WebSphere Java Batch function we'll discuss next.
The key batch assist function provided by this Batch Runtime is the ability to cooperatively process Java and
COBOL as part of a larger batch process, and to have a JDBC Type 2 connection shared between the two with
coordinated transaction processing.
Note: pause and think about how much custom code you would need to write to perform just the functions
mentioned in that last sentence. The code to invoke COBOL from Java implies JNI function, which is not trivial to
write. The shared JDBC Type 2 connection implies writing to the RRSAF interface. The code to create and
manage a transactional context is also not trivial, particularly if proper transaction recovery is taken into account.
This is the value of the Batch Runtime -- it provides batch container services so your Java code can stay focused
on the batch business logic you wish t perform.
The Batch Runtime may be used even if you don't call COBOL, or don't need shared JDBC Type 2 with
transaction coordination. Those are the key features we highlight to show you the power of it.
We will offer a positioning chart in a few moments that will position BPXBATCH against JZOS and the Batch
Runtime function. All three are JVM launcher technologies. They operate with different levels of provided
function, and the programming model for the Batch Runtime is different from the programming model for JZOS
and BPXBATCH.
IBM z/OS Java Batch Solutions
26
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD26
Version: September 10, 2013
z/OS 2.1 - Enhancements to Batch Runtime
JZOS and z/OS Batch Runtime …
The Batch Runtime with z/OS 2.1 comes with enhancements:
z/OS 1.13 z/OS 2.1
Java main() Programs
Single Step Jobs
COBOL
DB2 and JDBC T2
Java main() Programs
Single Step and Multi-Step Jobs
WebSphere Java Batch programsExceptions:● Java SE runtime, not Java EE● No WAS functionality (EJB, MDB, JCA)● No WLM classification, no SMF 120.9● Parallel Job Manager not available
COBOL
PL/I
DB2 and JDBC T2
Transactional VSAM
More on WebSphere Java Batch coming up
The Batch Runtime planned for z/OS 2.1 adds several enhancements, including support for PL/I and Transaction
VSAM. In addition, it will add the Java batch container function from the WebSphere Java Batch included with
WAS V8.5. So some of the container services provided will be available to Java programs in this runtime, and
Java batch applications written for WebSphere Java Batch are compatible with this Batch Runtime environment.
Note: the key differentiation between this Batch Runtime and the WebSphere Java Batch function is the Java
runtime under the batch container. In WebSphere Java Batch it's WebSphere Application Server with its full Java
EE programming model; with the Batch Runtime the WAS runtime is not present and the JVM supports Java SE,
not Java EE. We'll cover the WebSphere Java Batch function in detail later in this session.
IBM z/OS Java Batch Solutions
27
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD27
Version: September 10, 2013
JZOS and z/OS Batch Runtime
Integration with Enterprise Schedulers …
The two are not mutually exclusive. In fact, the Batch Runtime makes use of JZOS to launch the JVM and load the Batch Runtime function:
com.ibm.zos.batch.container.BCDBatchContainer
BCDPROC
BCDBATCH
JZOSLauncher
JZOSLauncher
z/OS Address Space
31-Bit Java Virtual Machine
z/OS Batch Runtime
JavaCOBOL
JZOSToolkit
JZOSToolkit
z/OS Servicesz/OS Services
General Requirements:● IBM 31-bit SDK for z/OS, V6.0.1
Must be 31-bit because COBOL is a 31-bit language
● IBM Enterprise COBOL Version 4.2
● DB2 V9 with PTF UK62190 for JDBC 3.0 specification, or PTF UK62191 for JDBC 4.0 specification
● DB2 V10 with PTF UK62141 for JDBC 3.0 specification, or PTF UK62145 for JDBC 4.0 specification level
Calling JCL / procedure similar to JZOS; Batch Container
supplies samples of its own
The JZOS toolkit may also be used in conjunction with the
Batch Container
The Batch Runtime is a JVM Launcher solution, and in fact makes use of JZOS as the JVM launcher. So the
runtime construction is similar to JZOS in that there's a job JCL and a proc JCL. The difference is the Java class
that's called and loaded by JZOS is the IBM-written Java class that instantiates the Batch Runtime function in the
JVM launched by JZOS. The Java batch job that is then run is supplied in z/OS 1.13 via inline JCL control
statements that pass into the container the batch job to run.
The Batch Runtime has a set of general requirements as shown on the chart above.
IBM z/OS Java Batch Solutions
28
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD28
Version: September 10, 2013
Integration with Enterprise Schedulers
JVM Launcher Positioning …
All three JVM launcher technologies discussed may be initiated via submission of JCL. That becomes the integration point with enterprise schedulers:
Enterprise Scheduler Function
Job JCL
Java Program
Java Virtual Machine
JES SpoolOr elsewhere if JCL defines other than spool
Submit
In many ways this is business as usual ... to the Enterprise Scheduler function the submitted JCL job is the batch job. The invocation of a JVM
to run Java is a detail not of concern to the scheduler function
This is the first time in this presentation we've raised the issue of integration with Enterprise Scheduler functions
such as IBM Tivoli Workload Scheduler or other comparable vendor products. We didn't mention it with
BPXBATCH or JZOS because those were so clearly invoked with JCL, and the integration point with the
Enterprise Scheduler was at that point of JCL submission.
The same is true with the Batch Runtime. That makes sense when you step back and realize the Batch Runtime
is really an augmented instance of a JVM launched by JZOS. The Batch Runtime in fact relies on JZOS to be the
JVM launcher. So the point of integration with Enterprise Schedulers is the same as for BPXBATCH or JZOS -- at
the point of JCL submission.
This is true even if the Batch Runtime is simply one step in a larger multi-step batch job submitted by the
Enterprise Scheduler. That's one of the values of using JZOS as the JVM launcher is that it allows for coordinated
Java and non-Java steps within a multi-step JCL. To the Enterprise Scheduler the JCL that it submits is the batch
job.
Note: the reason we're bringing up the Enterprise Scheduler integration is because (a) eventually it becomes a
point of interest in most discussions about Java batch processing, and (b) it becomes an important point later
when we discuss WebSphere Java Batch. So we introduce the topic here where the integration is relatively
obvious, and we build on it with WebSphere Java Batch where the integration involves a supplied utility as you'll
see later.
IBM z/OS Java Batch Solutions
29
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD29
Version: September 10, 2013
JVM Launcher Positioning
Summary …
The following provides a high-level positioning of the alternatives discussed so far:
JVM Bit-mode
Batch Services
DB2 Access
TX Coordination
Programming Model
BPXBATCH JZOS Batch Runtime
Address Spaces
31-bit or 64-bit 31-bit or 64-bit 31-bit
Multiple(one with BPXBATSL)
One One
NoneProvides a JVM but no
other extra-JVM services
SomeToolkit provides z/OS
services. Batch services such as checkpoint not provided
Yesz/OS 2.1 - WebSphere Java
Batch batch services (checkpoint, data stream, etc.)
JDBC T2 or T4Access code is all
custom to the application
JDBC T2 or T4Access code is all
custom to the application
JDBC T2 onlyTX coordination requires RRSAF connect to DB2
z/OS 2.1 - tVSAM as well
NoneAny TX must be custom
implementation
NoneAny TX must be custom
implementation
YesProvided by Batch
Runtime and coordination with RRS
Java main()Standalone Java
Java main()Standalone Java
WebSphere Java Batchz/OS 2.1 - Based on subset of WebSphere Java Batch model; runs in J2SE not J2EE, and no
WAS classes available.
Here is a positioning chart showing the three JVM launcher technologies across the top with a set of functional
considerations down the left side of the chart.
Note: the functional considerations are by no means exhaustive. If other considerations come up during
presentation of this deck, please forward an e-mail to [email protected] and let me know. I can expand
the positioning to a second chart and continue the discussion.
BPXBATCH and JZOS appear to be very similar if you look at just this chart, but as we discussed earlier JZOS
provides a number of improvements over BPXBATCH for Java processing. The Batch Runtime offers some more
distinct differences from JZOS because a batch container is provided, and container code to allow COBOL to run
in the same address space as Java is provided. The programming model also changes: BPXBATCH and JZOS
are simple Java main() programs, whereas the Batch Runtime complies with the WebSphere Java Batch
programming model we'll discuss in the next section.
Summary: all three are JVM launcher technologies, but the Batch Runtime is what provides a Java batch
container with additional batch services. In z/OS 2.1 the batch services in the Batch Runtime are similar to the
services provided in WebSphere Java Batch. The big difference there is that WebSphere Java Batch is build on
WAS, which is a runtime model where the JVMs stay active. That alleviates the overhead of JVM instantiation and
tear-down, which may play a key factor when daily batch processing involves hundreds or thousands of such batch
cycles.
IBM z/OS Java Batch Solutions
30
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD30
Version: September 10, 2013
Section Summary
WebSphere Java Batch …
A very broad summary of the JVM Launcher section:
JVM Launchers instantiate JVMs and run programGenerally an "application per JVM" model
We showed three JVM Launcher ModelsThey range from simple (BPXBATCH) to more functional (Batch Runtime)
Which you use depends on your business needsBPXBATCH may work fine for simple ad hoc functions; JZOS provides a more
flexible launching model with z/OS access services; Batch Runtime addresses the need for Java/COBOL interaction with coordinated transaction support.
Potential downside: cost of JVM start and stopOne potential downside of JVM launchers is the overhead associated with starting and stopping the JVM. May not be an issue if cycles are few; tends to become an
issue when the number of start/stop cycles increases.
This is the summary chart for the section just completed.
IBM z/OS Java Batch Solutions
31
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD31
Version: September 10, 2013
WebSphere Java BatchJava Batch inside WebSphere Application Server
IBM z/OS Java Batch Solutions
32
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD32
Version: September 10, 2013
WebSphere Java Batch Overview
Two Delivery Mechanisms …
The following is a very high-level overview of the WebSphere Java Batch function:
WebSphere AppServer Instance
EJB Container
Web Container
Batch Container
Java Batch Applications
IBM RAD Tooling Support
Dispatcher Function
z/OS Operating System
System z HW Platform
Job Submission Interfaces
xJCLxJCL
Job Properties Declaration File
Key Points:● This JVM stays up for as long as the WAS server stays up
Days or more ... avoids overhead of JVM instantiation in environments where batch invocation is high
● All the services of WAS z/OS are available to the batch applications running thereTransaction, data access, Parallel Sysplex and clustering, WLM, SAF, SMF
● The batch container provides valuable batch services that keep batch logic focused on the business and away from writing custom middleware codeMulti-step jobs, Batch Data Stream Framework, Checkpoint, Skip record, Step retry, Parallel Job Manager
TechDoc: ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783
● Browser● Command Line● SOAP web services● RMI● MDB
The next function to discuss is WebSphere Java Batch. This is function that started out as a separate licensed
product called "Compute Grid," and has been incorporated into the WebSphere Application Server product starting
in WAS V8.5.
The first important point to stress about this solution is that it's built on the runtime model of WebSphere
Application Server, which means the JVMs are long-lived. What the WebSphere Java Batch function provides is a
batch container environment that operates alongside the Web and EJB containers of WAS.
That is a different execution model from the JVM Launcher technologies, where the JVM was instantiated and a
named program run. Here the JVM is already up, so another mechanism is needed to provide an interface to tell
the batch container to invoke a deployed Java batch program. WebSphere Java Batch provides a "dispatcher
function" for that purpose.
Finally, the ability to layer a batch container on top a Java EE runtime allows for some very interesting batch
services to be made available to your programs. That's what we'll cover in this section.
Note: The WP101783 Techdoc at the URL shown on the lower-left of the chart contains a set of documents
related to WebSphere Java Batch.
IBM z/OS Java Batch Solutions
33
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD33
Version: September 10, 2013
At Present Two Delivery Mechanisms
Container Services …
The "WebSphere Java Batch" function is delivered in two ways:
WebSphere Application Server
Version 7 or Version 8
WebSphere Application Server
Version 8.5
Compute Grid V8WebSphere Java Batch Product
Add to WAS ("Augment") WebSphere Java BatchFunctionally equal to CG V8
For those who have a standardized commitment to
WAS V7 or V8 but desire WebSphere Java Batch function
WebSphere Java Batch function provided in WAS V8.5 product
Note how both are functionally identical
At this point in time there's a bit of confusion about how to acquire the WebSphere Java batch function. That's
because there are two avenues -- one as a separately licensed product called Compute Grid, the other as part of
WebSphere Application Server V8.5.
The separate Compute Grid product is available for those who have WAS V7 or V8 (but not V8.5) and are not in a
position to migrate their application runtime up to WAS V8.5. The Compute Grid product may be added (the term
used is "augmented") to the WAS V7 or V8 runtime to provide the Java Batch function to WAS.
Starting with V8.5 the Compute Grid function was incorporated into the WAS V8.5 runtime foundation. With V8.5
there is no need to acquire or augment Compute Grid. It is included with the product.
Note: the key point is that Compute Grid V8 and WAS V8.5 represent the same Java Batch functionality. Which
you choose is more a matter of your runtime version considerations rather than functional differences.
IBM z/OS Java Batch Solutions
34
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD34
Version: September 10, 2013
Overview of Batch Container Services
Job Submission and the JMC …
The following provides a summary. The WP101783 Techdoc has many more details:
Batch job dispatching and management system● Separates job invocation and management from the
instantiation of the JVM● Provides job control: stop, restart, cancel, suspend● Multiple interfaces: Web, SOAP, RMI, command, MDB
Data record read and write support services● Abstracts low-level stream handling from business logic● Framework passes data to batch logic as Java object
Checkpoint and job restart services● Checkpoint tracking abstracted from batch logic
and performed as function of container● Commit and rollback a function of the container● Job restart from last checkpoint
Job resiliency services● Ability to skip and track records where data read
or write operation throws exception● Ability to retry job steps when unhandled
exception is thrown● Ability to utilize defined call-out points in batch
job execution to invoke custom Listener code
Parallel job management and execution services● Develop and deploy applications to run segments in
parallel based on your designation of data partitioning● Ability to programmatically partition data ranges based on
passed-in parameters● Container manages top-job and all sub-jobs to completion
COBOL Container Support● Invoke COBOL from Java● JNI stub code provided with container● Call stub generator utility to create call stubs and data
parameter code
Do you recall back during the discussion of BPXBATCH and JZOS we said that neither provided much in terms of
batch services? We indicated with those JVM launchers the responsibility would be yours to code any such
services as custom code.
The purpose of WebSphere Java Batch is to provide a container-managed environment that provides batch
services to your Java batch job. This chart provides a bullet-point summary of many of those services.
Note: the key point to stress here is that these functions are provided as programmatic interfaces to your batch
function. Your batch function calls these functions as needed, using the appropriate documented API. A review of
the bullet points on the chart will show this is a fair amount of function provided as container services, which
highlights the value of WebSphere Java Batch as a Java batch runtime.
There's not enough room in this speaker note section to provide details on every one of the bullets. The
WP101783 Techdoc at ibm.com/support/techdocs provides two-page brochures that describe many of the
functions shown above.
When presenting this chart some balance needs to be achieved -- some time spent is necessary to make the point
about the services provided, but covering in detail each bullet point is probably not necessary. A recommended
approach for discussion would be:
● Start with checkpoint services and make clear that the batch job step code handles none of this. This is all container-
managed with transaction management being provided by WAS and checkpoint intervals a configurable element of the
job declaration file.
● Point out that job resiliency functions are to let long batch jobs continue when the occasional glitch happens. Better to let
glitch pass and log problem than hold up whole job because of a small hiccup.
● Discuss at a high level the concept of partitioning data ranges so multiple batch jobs may run in parallel, each executing
against its data range. The service provided here is (a) programmatic interfaces to allow job partitioning at time of
submission, and (b) a coordination layer that tracks the sub-jobs as a logical entity to completion.
● Refer to the COBOL container as something very similar to what the Batch Runtime function provided, but with some
differences to be covered a few charts later from this chart.
IBM z/OS Java Batch Solutions
35
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD35
Version: September 10, 2013
Job Submission and Monitoring
CI Model …
The Job Management Console (JMC) is the browser interface. It allows you to submit and monitor the jobs:
Job State Flow
Jobs go through a series of states in this model. The interface to the
Dispatcher Function provides you control to submit, view, cancel, stop,
suspend, restart or remove jobs
Before going into any details, let's first see an example of the Job Management Console showing several jobs that
have been submitted. This affords an opportunity to illustrate what the "Dispatcher Function" is doing and the
"states" a submitted job goes through.
The Job Management Console (JMC) is the browser interface to the Dispatcher Function of WebSphere Java
Batch. It is one of several interfaces; the others being command line, SOAP web services, RMI and MDB. The
role of the Dispatcher Function is to provide a means of submitting jobs and controlling them. We show the
browser interface here simply because it's the one most likely used first, and it's the easiest to approach
conceptually.
A "job" consists of two key components: the Java code that implements the batch processing and a job declaration
file that tells the system how to process the job. The Java code is packaged in a standard EAR file format and is
deployed into WAS like any other EAR file. The job declaration file is called "xJCL" and it is read by the
Dispatcher Function to begin execution and control the operational behavior during execution. We'll see more on
the xJCL in a few charts.
Jobs go through "states" in this process. A typical series would be "Submitted" � "Executing" � "Ended." If
something is wrong the job may go into an "Execution Failed" state where you the batch operator would then read
the job log to see what happened and take corrective action. Once a job begins execution you may wish to cancel
or stop the job, where it will then go into a restartable state. You may then restart the job and it will pick up and
continue.
Note: We have not yet introduced the concepts of "compute intensive" (CI) or "transactional batch" so the
explanation of the difference between Cancel/Stop and "Suspend" might not be entirely clear. That said,
Cancel/Stop is for transactional batch and Suspend is for CI jobs. In a transactional batch job "Cancel" will cease operation and rollback to the last checkpoint; "Stop" will allow execution up to the next checkpoint and then stop.
The point of this chart is to begin the discussion with a brief illustration of the job interface structure. Anyone in the audience with a background in starting jobs using z/OS JES should relate to this.
IBM z/OS Java Batch Solutions
36
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD36
Version: September 10, 2013
High-Level: "Compute Intensive" Model
Transactional Batch Model …
This provides a simple execution model suitable for (a) computational work, or (b) a means of wrappering existing Java main() programs to run:
:
public class MyCIWork implements com.ibm.websphere.ci.CIWork {
:
public void run() {
Your batch code goes here
}
}
Batch Controller BeanStateless session bean provided by container framework
Success if no Exceptions
Thrown
Dispatch
Very simple model to develop and run
Very few batch runtime services available● No data read or write patterns to utilize ... roll your own● No checkpoint processing ... roll your own● No commit and rollback ... roll your own
Use tooling to wrapper Java main() into this framework and it will run in WebSphere Java Batch
Useful model for some types of applications.
Transactional Model may be better suited to your needs ...
Now we begin a high-level discussion of the programming model used by WebSphere Java Batch and, it should
be noted, the Batch Runtime function discussed earlier (since it is based on WebSphere Java Batch).
We're going to start with a programming model that is the most simple. It's called the "Compute Intensive" model,
or CI for short. The CI model provides a basic single-step framework in which a Java program may run.
Note: it is important to set some context before describing this CI model and why we start with it. It's likely most
customer Java batch programs using WebSphere Java Batch will use the transactional batch model described
next. That's the more intuitive "read a record, process the record, write the record" model.
We start with this CI model because (a) it's really simple, and (b) it's the "on-ramp" into WebSphere Java Batch for
people who have an asset base of Java main() programs they wish to run in WebSphere Java Batch. Point (b)
is the key reason why we start with this -- because saying that a Java main() can be wrappered in this CI model
and run within WebSphere Java Batch makes intuitive sense. That allows people with an inventory of Java
main() programs to understand they can easily participate in WebSphere Java Batch if they choose.
Originally the CI model was intended for just as the name implies -- computationally intensive batch programs,
such as you might find in mathematical or scientific data processing. There data I/O is not as important as
climbing on the CPU and crunching numbers for an extended period of time. Hence the name, and hence the
relatively simple one-step structure.
But as it turned out, this model was also very good for wrappering Java main() programs and running them
under the job submission and job management structure of WebSphere Java Batch. So it represents a means of
enjoying some of the benefits of the container-managed model of WebSphere Java Batch. And, as noted, it
provides an intuitively approachable way of preserving existing Java main() assets.
Note: the picture shows a "Batch Controller Bean," which then invokes the run() method. The batch controller
bean is provided as part of the packaging of the deployable EAR file. It is what the job dispatching function
described earlier interacts with to submit and manage jobs within the WebSphere Java Batch model. Consider it
part of the WebSphere Java Batch infrastructure. It's exposed here to "demystify" how things work in this model.
IBM z/OS Java Batch Solutions
37
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD37
Version: September 10, 2013
High-Level: "Transactional Batch" Model
xJCL …
This is what people usually think of when they think of "batch." At the heart of this is the job step code, which encapsulates the data record business logic:
Batch Controller BeanStateless session bean provided by container framework
Dispatch
:
public class MyJobStep implements BatchJobStepInterface {
:
public void createJobStep() {
}
public int destroyJobStep() {
}
public int processJobStep() {
Step logic for a data record here
if record not null: return BatchConstants.STEP_CONTINUE;
if record null: return BatchConstants.STEP_COMPLETE;
}
}
Controller bean invokes job step in iterative loop until STEP_COMPLETE returned
Your job step logic would process data record
according to business needs
Now we introduce the "transactional batch" programming model, which is built around the batch step which
handles one record at a time. The batch step processJobStep() method is invoked iteratively by the controller
bean until the step returns "complete." As long as "continue" is returned the controller bean knows to turn and
invoke the batch step class again.
Note: questions may come up at this point -- (a) multi-step jobs allowed? Answer: yes, discussed next chart; (b)
where does batch step get data record? Answer: either from custom code in batch step or by calling batch data
stream framework implementation, discussed in an upcoming chart.
The key here is that the looping logic is not part code you provide in the batch step. The batch step operates
against a data record and performs whatever business logic against the data record that is necessary. The
iterative looping continues until the record retrieval results in no more records, at which time the batch step returns
"complete." That signals to the batch controller to then invoke destroyJobStep() and the step completes.
As noted, the batch step implementation should not handle looping counts; it should not to to manage transactions
or do commits or rollbacks. Those are all functions externalized to the container and controlled by the job
declaration file (xJCL) which we'll describe next.
Note: it is useful to step back and view this conceptually. The transactional model is a data-record oriented
process, and the batch step class focuses on a data record. Think about what kind of processing against a data
record your batch might need -- update a field, write a new record based on data read in, etc. That's the very core
of your transactional batch processing; the rest is related to managing the iterative loop and all that surrounds that
-- starting the looping, knowing when looping is complete, maintaining the checkpoint count, performing commits
and rollbacks as needed. That's where the batch container of WebSphere Java Batch comes into play. That also
why a job declaration file is so important: it provides the container an understanding of how to manage your "job," which contains your "step." So we look at xJCL next.
IBM z/OS Java Batch Solutions
38
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD38
Version: September 10, 2013
Job Declaration File - xJCL
Checkpoint processing …
Conceptually, xJCL is just like normal // JCL -- it describes the job to be run and the context in which the job is to operate. The difference is xJCL is written in XML:
<?xml version="1.0" encoding="UTF-8" ?>
<job name="MyJob" ... ">
<substitution-props>
<prop name="inputDataStream" value="/tmp/input-text.txt" />
<prop name="outputDataStream" value="/tmp/output-text.txt" />
<prop name="checkPoint" value="10" />
</substitution-props>
<job-step name="MyStep1">
<classname>com.ibm.ws.batch.MyStep1</classname>
<batch-data-streams>
<bds>
<logical-name>inputStream</logical-name>
<props>
<prop name="PATTERN_IMPL_CLASS" value="com.ibm.websphere.batch.EchoReader" />
<prop name="FILENAME" value="${inputDataStream}" />
</props>
</bds>
</batch-data-streams>
</job-step>
:
<job-step ...>
</job-step>
</job>
Note: this is a trimmed version of actual xJCL
Comparable to the JCL "JOB" card. It sets job-level information along with some substitution properties
Substitution properties may be overridden at submission time.
Comparable to the JCL step. It names the Java class that
implements the step function.
The "Batch Data Stream" implementation is specified. In this
example it defines the class that implements the input stream.
If job consists of more steps they are specified in sequence.
This tells WebSphere Java Batch what to run and how to run it
This is a topic that can't be covered in detail on one chart. So the objective will be to establish a conceptual
framework of what xJCL is and how it's used.
Note: if the audience has z/OS people in attendance, then it is useful to start by pointing out that this is very similar
to normal "slash-slash" JCL ( // ) used over the years, except this is written in XML. That provides a way of
relating this new thing (xJCL) to what they already know (JCL), and moves the learning down to a matter of
understanding syntax rather than understanding concept.
At the very highest level, the xJCL file is used to tell the WebSphere Java Batch runtime that a job invocation is
being requested, and to provide the runtime an understanding of what job to run and to provide details about the
job. (That is exactly the same high-level purpose of regular JCL.)
Keep in mind that an objective of WebSphere Java Batch is to keep Java batch step logic focused on the business
needs. Therefore, one of the roles of WebSphere Java Batch and xJCL is to externalize details that are resolved
at time of job submission. For example, details about the location of a data input file should not be hard-coded in
the business logic. Those are externalized with symbolic references. Again, the JCL DDNAME concept is a
useful way of relating xJCL to normal JCL. The same concepts apply.
Note: it is also useful to remind people of two things about the creation of xJCL -- (1) like regular JCL, a common
practice is to take an existing file and modify it as needed. Almost nobody creates regular JCL from scratch. And
(2) the tooling support for WebSphere Java Batch provided in IBM Rational Application Developer provides an
"xJCL Wizard" to assist in the creation of this file.
The structure of xJCL breaks out like normal JCL -- a "job" section at the top, and "step" sections for as many
steps as your job requires. Properties that apply to the whole job are coded up in the job section; properties that
apply to the step are supplied in the step section.
The xJCL is a necessary part of job submission -- you supply your xJCL to the Job Dispatcher function through
one of the interfaces (web being the easiest to use when first starting out). The Job Dispatcher reads in the xJCL
and from that comes to understand what you're asking for. It then gets to work by signaling to the server where
the batch job is deployed to start up and process.
IBM z/OS Java Batch Solutions
39
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD39
Version: September 10, 2013
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Checkpoint Processing
Batch data stream framework …
Checkpoint commit and rollback is a function of the container. It relies on the transaction manager function of WAS itself for this:
Java Batch Application
Batch Container
xJCL where checkpoint declared
WAS Transaction Manager
Commit
Commit
Last good checkpoint value persisted
Checkpoint interval (record or time) specified in the xJCL
This is a function of the batch container, not your application code
As checkpoint intervals are reached, container commits and records the checkpoint attained
In the event of a failure, job may be restarted at the last good checkpoint
Set the checkpoint interval based on your knowledge of balance between recoverability and efficiency
Checkpoint processing handles the commit action against the data resources being updated by the Java batch
application. This function is abstracted away from your batch job and is handled by the batch container.
WebSphere Java Batch is built on top WebSphere Application Server, and WAS has within it a transaction
manager. So this checkpoint processing function supplied by the container is really relying on WAS's TX manager
to handle the creation of the transaction and managing the commits and rollbacks.
Two checkpoint metrics are supported: number of records or time since last commit. In either case, the value for
the checkpoint metric is specified in the xJCL. Your batch application does not get involved with this at all.
WebSphere Java Batch maintains a relational table entry with the last good checkpoint value reached for every
batch job. This persisted checkpoint value is used in cases where a restart is needed. The container reads the
last good checkpoint value and restarts record processing from that point.
What the checkpoint value should be is a a balance between recoverability and efficiency. Set the value really low
(checkpoint = 1) and you achieve very granular recoverability but at the cost of doing a lot of commit processing.
Set the value really high (checkpoint = 1,000,000) and any restarts will imply going back a long way and doing
work over again.
Note: the key point here is that the checkpoint / commit processing is using WAS transaction manager function,
and this is all abstracted away from your batch application code. Your code focuses on the business logic needed
to process the batch record. The checkpoint / commit / rollback is handled by the container based on the
declaration in the xJCL.
IBM z/OS Java Batch Solutions
40
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD40
Version: September 10, 2013
Job Step Class
High-Level: "Batch Data Stream Framework"
Skip Record …
This provides a way of abstracting the data read and write logic away from the batch step code. Several Batch Data Stream patterns are provided:
Batch Controller BeanStateless session bean provided by container framework
Dispatch
Batch Data Stream ClassLogic to read or write a record of
data to the data store
Data passed as Java object to and from the BDS implementation
Version 8.5 InfoCenter: search on cgrid_btchpttrnover
Supplied "patterns" for:● Read and write using JDBC
● Read and write JDBC and stored procedures
● Read and write byte data from file
● Read and write text file
● Read and write to z/OS data set
● Read and write using JPA
Options:● Take existing pattern and
customize to your needs
● Create your own custom pattern
● Not use BDS but rather code data record read and write handling as part of batch job step
An implied assumption in our discussion earlier of the batch job step was that the data retrieval came from
somewhere. WebSphere Java Batch provides something called "Batch Data Streams" (BDS) as a means of
externalizing this from the job step. The BDS implementations for a job step are specified as properties in the
xJCL. The container invokes the BDS implementation to get the data record, which is formatted into a Java object
and passed to the job step when invoked.
Several Batch Data Stream "patterns" are supplied with WebSphere Java Batch. They are shown on the chart on
the lower left bullet list.
So with this you have three options for data retrieval and storage -- use an existing pattern (customized as needed
to do the specific read or write your batch step requires), create your own BDS pattern, or not use BDS at all and
instead code your data read and write logic in the job step.
IBM z/OS Java Batch Solutions
41
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD41
Version: September 10, 2013
Skip Record Processing
Multi-step jobs …
Provides a container-managed way of tolerating data read or write errors so the job itself may continue on. Information about data errors may be logged.
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Java Batch Application
Batch Container
Batch Data Stream Framework Code
Exception!
Exception!
xJCL where skip record policy declared
xJCL tells BDSF:
● How many data read or write exceptions to consume
● What exceptions to consider for skiprecord processing
● Alternatively, what exceptions to exclude from skip-record processing
Objective is to provide mechanism to survive the odd data exception rather than stop the whole job
The BDSF implementation code consumes the exception rather than surfacing it up to the batch step code
A "listener" may be called to log information about the data record where exception occurredMore on this coming up
When the skip limit is reached then further exceptions are exposed up to the batch step implementationWhere it might result in job going into a restartable state. Normal checkpoint restart processing would occur.
Technical Flyer at: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783
The "Skip Record" function provides a means of surviving the occasional data read or write operation exception.
This provides a way for a much larger job to keep going rather than go into a restartable state because of some
odd exception.
Note: this is not to say the data record that had the exception is to be ignored. The listener function we'll describe
in a bit provides the ability to log information about the failure. What this provides is a way for a million record
batch job to complete even though one or two data records threw a read or write exception. Absent this the job
would go into a restartable state at the last good checkpoint value and time would pass while operators noticed the
failure and debugged the problem. If batch windows are tight, better to process 999,998 records with 2 failures
than to stop the whole job because of the 2 failures.
The properties for the Skip Record behavior are specified in the xJCL for the "Batch Data Stream Framework"
(BDSF) implementation. This function applies on a per-BDSF basis. The properties in the xJCL provide
information on how many failures to tolerate and what exceptions to consider or exclude from consideration.
What happens if the limit is reached? Then subsequent exceptions are surfaced up to your batch step
implementation. There your code either has to handle the exceptions or, if not, then it will throw the job into a
restartable state. If this happens then the normal checkpoint processing takes place, and the job is restartable
from that last checkpoint interval.
As mentioned, there's a "listener" function that allows this Skip Record process to give control to code you write to
record information about the record that threw an exception. With that information your auditors may then go back
and see what happened and take necessary corrective action.
IBM z/OS Java Batch Solutions
42
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD42
Version: September 10, 2013
High-Level: Multi-Step Jobs
Retry step …
A job may consist of multiple steps. The job step execution for each is what was shown earlier. Job steps are executed in sequence according to xJCL:
Batch Controller BeanStateless session bean provided by container framework
Dispatch
Job Step Class
Job Step Class
Job Step Class
<job ...
<job-step ... >
</job-step>
<job-step ... >
</job-step>
:
<job-step ... >
</job-step>
</job>
xJCL
STEP_COMPLETE
STEP_COMPLETE
STEP_COMPLETE
Job may consist of one to many stepsMay be mixed CI, TX, Parallel
Steps are executed in sequential order according to xJCL
Supplied results algorithm allows for passing step results to next step
Conditional step processing supported
The construction of WebSphere Java Batch jobs begins to look very much like traditional batch with "Jobs" and "Steps". This is the one of
the benefits of a container-managed Java batch environment.
Earlier we noted that a question will likely come up whether multi-step jobs are supported, and we said the answer
was yes. Now that we have the concept of a "job step class file" in place and xJCL understood at the conceptual
level, now we can paint the picture of an xJCL file that describes a multi-step job. It's really as simple as having
multiple sections in the xJCL file that describe each step. A job section is still needed to describe the overall job,
with one to many step sections to follow.
The steps are processed in sequential order according to what's described in the xJCL. The steps may be of all
the same type (all CI, or all transactional), or mixed type (CI, transactional batch, transactional using the Parallel
Job Manager).
Another function provided by the WebSphere Java Batch runtime is the ability to surface the return code of one
step so it's accessible and understood by other steps. Conditional step processing is supported, so a multi-step
job that fails on step 2 can suspend processing of step 3 until the issue is resolved.
Note: it's worth noting here also that jobs have several "states," one of which is "restartable." So if a job is
submitted and starts executing but throws an error, it will go into a restartable state. You correct the error and
then tell the Job Dispatching function to "restart." WebSphere Java Batch checks the job step last processed and
the checkpoint value last set and restarts from that point.
IBM z/OS Java Batch Solutions
43
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD43
Version: September 10, 2013
Retry Step Processing
Batch Listeners …
Provides a way of retrying a job step in the event of an exception thrown. If successful on retry then the job continues and your processing completes.
Job Step● Java class● Input / Output● Other Properties
Job Step● Java class● Input / Output● Other Properties
Batch Job
xJCL tells Container:● How many step retries may be attempted● What exceptions to consider for retry-step
processing● Alternatively, what exceptions to exclude
from retry-step processing● Whether to process a delay before
attempting a retry of the step
If exception, retry step n
times
This is at a higher level from Skip Record. This is at the "invoke batch step" level. This provides a way to retry the step for exceptions.
Batch container falls back to last good checkpoint and restarts from there
A "retry-step listener" may be called so you can perform custom action upon retry-step processingMore on "batch listeners" coming up
When limit reached, then job goes into restartable state. Normal restart from checkpoint processing.
Technical Flyer at: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783
Provides a way to survive unexpected transient
exceptions through step retry
One level up from skip record processing is "Retry Step Processing." This provides a mechanism to retry a job
step in the event an exception is thrown that would otherwise put the whole job into a restartable state.
Here again we see the declaration for the behavior of this function provided in the job xJCL. This is done on a
step-by-step basis, and the xJCL properties for this indicate how many retries to attempt, what exceptions to
include or exclude from consideration, and whether to include a delay before attempting a retry.
In the event an exception is thrown that results in this retry function taking effect, the batch container will fall back
to the last good checkpoint location and start again. A "listener" may be called to take action on a retry.
Like skip record, when the limit is reached this function stops retrying and allows the normal "restartable job"
function to take over. This is where you as operator would investigate why the step is failing beyond the retry limit
and take corrective action.
And just like skip record, the objective of this is to allow a large, long-running batch job to survive the occasional
hiccup and continue to completion without manual intervention.
IBM z/OS Java Batch Solutions
44
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD44
Version: September 10, 2013
Batch "Listeners"
Parallel Job Manager …
These are callout points where your custom "listener" code will be called when key events occur. The callouts are managed by the batch container:
Job Step● Java class● Input / Output● Other Properties
Job Step● Java class● Input / Output● Other Properties
Batch Job
Job Listener
Retry Step Listener
Skip Record Listener
Job Listener● Callouts occur:
Start of the job; Start of each step; End of each step; End of job● Register your code to container with property in xJCL● Use this to perform any special setup or cleanup actions at those
points in the lifecycle of a batch job
Retry-Step Listener● Callouts occur:
When the exception is thrown; When the retry is attempted● Register your listener with code in application createJobStep()
method● Use this to take action at these points, such as logging
information about the exception and the point in the processing where it occurred
Skip-Record Listener● Callouts occur:
On skipped read or skipped write operation● Register your listener with code in application createJobStep()
method● Use this to take action at these points, such as logging
information about the exception and the record skipped
Technical Flyer at: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783
Several times now we've mentioned "listeners" that get invoked at certain points of batch processing. Listeners are
implemented with code you write which you "register" to the batch environment so it knows what to call when those
call-out events take place.
There are three listeners we'll focus on -- Job listeners, Retry-Step listeners, and Skip-Record listeners.
● Job listeners -- this is invoked at the start of the job, and the start and end of each batch step, and at the end
of the job. You register your listener class file by coding a property in the xJCL. At those points during batch
job execution the container will call the methods for that lifecycle point. Your code then does what you
determined was appropriate for that point in the job lifecycle ... setup work, cleanup work, logging ... whatever.
● Retry-Step listeners -- this is invoked whenever a job step is retried using the new Retry-Step processing
function. The listener is called when an exception is thrown that triggers Retry-Step processing, and again just
as the step is being retried. You register your listener with a simple line of code in the batch application itself.
● Skip-Record listeners -- this is invoked whenever a record is skipped using the new Skip-Record processing
function. The listener is called when an exception is thrown that triggers Skip-Record processing. Your code
may then log information about the record that has been skipped and the exception that took place. You
register your listener with a simple line of code in the batch application itself.
Listeners provide you a way to insert custom processing logic into the batch job lifecycle at key points during
execution.
IBM z/OS Java Batch Solutions
45
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD45
Version: September 10, 2013
Parallel Job Manager
COBOL Container …
This provides a mechanism for partitioning batch activity into parallel jobs based on ranges of data you designate:
Job Dispatching
Function
Batch Endpoint
PJM
Parameterizer
Batch Appl
Batch Endpoint
Batch Appl
Batch Endpoint
Batch Appl
Top-Job xJCL
Sub-Job xJCL
Top-job and job number
Sub-jobs ... top-job number and each sub-job number
State of each job
<job ...
<run
instances="multiple"
jvm="multiple">
<prop
name="parallel.jobcount"
value="3" />
</props>
</run>
:
Job step seen as PJM eligible based on "multiple" specified in xJCL
PJM function receives "top job" xJCL and invokes Parameterizer to perform your custom data range segmentation
Job Dispatcher then dispatches "sub jobs" to eligible endpoints for execution. Each is passed the data select parameters generated by parameterizer
Job Dispatcher coordinates execution of top job and sub jobs as logical entity
Not all jobs are well suited for PJM, but those that are may benefit greatly
Parallel either in multiple endpoints, on multiple threads, or both.
Another element of the batch infrastructure provided by WebSphere Java Batch is the Parallel Job Manager, or
PJM for short. The high-level objective of this function is to provide a way to run multiple batch steps in parallel,
each working on its own data range. If the nature of the batch job lends itself to parallel processing (many batch
jobs do), then use of this function has the potential to reduce wall-clock completion time by splitting out and
running in parallel.
Note: you could create completely separate batch jobs, each designed to work on its own static range of data, and
submit them concurrently. That would be a form of "parallel processing." But that's not what the PJM is doing.
The PJM is providing a means of parallel processing such that the data ranges may be programmatically
determined and assigned, the number of parallel processes assigned at time of submission, and the sub-jobs
treated as part of a larger logical "top-job." The PJM is keeping track of the checkpoint processing for each sub-
job and the state of each sub-job. The top-job does not end until all the sub-jobs end. If a sub-job goes into a
restartable state, the top-job is also restartable. Restarting the top-job results in the affected sub-jobs being
restarted so they may run to completion.
The jobs may run in parallel in different JVMs, or within the same JVM but on different threads.
Note: it is useful to step back and imagine trying to provide this level of parallel job management and control with
custom code. It is possible, but likely not a good use of your time to write custom middleware code like this.
Instead, the batch runtime services of WebSphere Java Batch provides this.
As noted, not all batch jobs lend themselves to parallel processing. But those that do can achieve improved job
completion by splitting out and running in parallel. Naturally the data ranges chosen need to be thought out so
data access locks don't occur, and the processing capacity to run multiple jobs concurrently needs to be in place.
But given all that, then the PJM function may prove a valuable part of batch window planning.
IBM z/OS Java Batch Solutions
46
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD46
Version: September 10, 2013
Java Batch Integration with COBOL
Enterprise Schedulers …
This is similar in concept to the z/OS Batch Runtime, but with some differences. The fundamental objectives is the same: low overhead integration Java and COBOL:
Java Batch Application
Batch Container
COBOL Container JNI Code
COBOL Container
COBOL Module
PDSE or USS Directory
WAS z/OS Servant Region Address Space
STEPLIB or LIBPATH
Call Stub Generator
IBM RAD
Call Stubs
COBOL Container manifests as separate LE Enclave inside the WAS z/OS servant address spaceKeeps COBOL/Java separate, so environments stay clean
The JNI code for Java/COBOL exchanges providedKey piece of plumbing provided, which relieves you of writing complex JNI
Call Stub Generator builds Java call stubs and data bindingsBased on what's in COBOL source
Provides a very efficient means of calling COBOL modules as part of a Java Batch solution. The JNI code and Call Stub Generator make using this solution relatively simple.
Share JDBC T2 with TXSimilar to what Batch Runtime provides
Another function of WebSphere Java Batch is the COBOL Container, which provides an efficient means of calling
COBOL modules from the Java batch code. Like the comparable function found in the Batch Runtime discussed
earlier, this allows for JDBC Type 2 connections to be shared between Java and COBOL with the transactional
integrity maintained across the two environments.
Note: the key differences between this Java/COBOL integration and that provided by the Batch Runtime is this:
Batch Runtime allows Java to call COBOL or COBOL to call Java, while this COBOL container only operates Java
calling COBOL; and this COBOL container provides a Rational Application Developer based utility to generate the
call stubs and data binding classes to call from Java into COBOL, which makes using this function relatively easy.
What's the value of this? If you have COBOL batch assets that you wish to incorporate into a batch process,
calling COBOL from Java across this interface is far more efficient than other means of invoking COBOL from
Java. The code path is considerably shorter, and that implies less overhead and less latency per call. In batch
processes where perhaps millions of calls are made every little bit of latency avoided tends to reduce the overall
latency experienced.
The ability to share the JDBC Type 2 connection between Java and COBOL with transactional coordination
provided by the WAS runtime provides a much simpler means of managing TX between the two environments.
IBM z/OS Java Batch Solutions
47
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD47
Version: September 10, 2013
Integration with Enterprise Schedulers
WLM Classification …
One of the interfaces on the Job Dispatcher functions is MDB. This allows a client program to interact with Java Batch and integrate that system with Ent. Schedulers:
Dispatcher Function
MDB Interface
Output
Input
Enterprise Scheduler Function
"WSGRID"Utility
WSGRID JCL
JES SpoolOr elsewhere if JCL defines other than spool
Submit
Exact same Java Batch processing as before
● Messaging provider either MQ or SIBus component of WAS● WSGRID submits job request to dispatcher● Job output from WebSphere Java Batch flows back to WSGRID● WSGRID stays active for duration of Java Batch job
Enterprise Scheduler sees "batch job" as the WSGRID invocation. That stays up for life of Java Batch job. Job output from WebSphere Java batch flows back to WSGRID. Return code from Java Batch flows back to WGRID.
Earlier we introduced the question of integration with existing Enterprise Scheduler functions, and we explained
that for BPXBATCH, JZOS and the Batch Runtime that integration occurs at the submission of the job JCL.
With WebSphere Java Batch the function is different. The point of integration on z/OS is still with the Enterprise
Scheduler submitting JCL, but it also involves a bit of "glue function" between the submitted JCL and the job
dispatching function of WebSphere Java Batch.
Note: the Job Dispatcher function is hosted in a WAS server, and has several interfaces. The one used for this
integration is a Message Drive Bean (MDB) interface. The "glue" between Enterprise Scheduler JCL submission
and the Dispatcher is a supplied program that uses messaging -- either MQ or the built in SIBus of WAS -- to
submit Java batch jobs into WebSphere Java Batch. That "glue" utility is known as "WSGRID."
An important objective to this design was allowing the Enterprise Scheduler to operate in its normal fashion
(meaning without needing to be modified just for WebSphere Java Batch). That was accomplished because to the
Enterprise Scheduler all it sees is the JCL that submits the WSGRID utility job. The WSGRID utility job then
passes information over messaging to the Dispatcher, which dispatches the job into the WebSphere Java Batch
runtime.
Now here's the important point to keep in mind -- the WSGRID job submitted by the Enterprise Scheduler stays
active for the duration of the Java batch job running back in WebSphere Java Batch. That's how the Enterprise
Schedule "knows" that the Java batch job is still running, or when it has completed. The WSGRID job becomes a
"proxy" for the actual Java Batch job running in WebSphere Java Batch.
Another important point to keep in mind -- during the running of the Java batch job in WebSphere Java Batch, the
Java job output is streamed back over messaging to the WSGRID job, which then pipes it to the STDOUT defined
in the WSGRID JCL. That's how Java batch job output is spooled to JES for archival purposes.
If you don't have MQ you may use the version of WGRID that utilizes the Service Integration Bus (SIBus) of
WebSphere Application Server. The same concept of job submission via messaging applies, but the WSGRID
utility program is a Java program, and that Java program bootstraps into the WAS environment and accesses the
built-in messaging of the SIBus. Output is streamed back to STDOUT of the WSGRID job.
IBM z/OS Java Batch Solutions
48
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD48
Version: September 10, 2013
WLM Classification
SMF 120.9 …
The Dispatcher Function on z/OS has the ability to assign a WLM Transaction Class value to a submitted job. That allows mapping to Service Class and Reporting Class:
Dispatcher Function
xJCL
Control Region
Servant Region(s)
Batch Application
WLM
WLM
Classify Work
DispatchSubmit
If (rule) then TC=tclass● Job class from xJCL● Jobname● Application Name● Submitter ID● other criteria
TCLASS
Assigned Service Class and/or Reporting Class
Two use-cases for this:
Assign separate WLM Service Classes
Have jobs dispatched into separate servant regions where WLM can manage to the goal
Assign separate WLM Report Classes
This allows categories of Java Batch jobs to have system resource information accumulated under the report class for RMF reporting
WebSphere Application Server for z/OS has extensive integration with z/OS Workload Manager (WLM) for the
purposes of request classification and priority management. The WebSphere Java Batch function can take
advantage of this and classify batch jobs during submission and dispatch.
Note: this topic requires some familiarity with z/OS WLM to understand the details of what's being said here. If the
audience is non-z/OS people that familiarity may not exist. Explain WLM as function of z/OS that controls access
to system resources -- CPU, memory and I/O -- to manage all the work on the system to the relative priority goals
defined. The "Service Class" is what carries the goal definition. When two pieces of work are competing for the
same system resources, then WLM gets involved and works to make sure the defined goals are met. Higher
priority goals take precedence over lower priority goals, but WLM also makes sure lower priority is not starved
altogether. WLM is integrated deeply into the z/OS operating system, which is how it can control access to system
resources to manage competing work to the defined goals.
The starting point for this is the WebSphere Java Batch "Dispatcher Function" using rules you supply to assign a
"Transaction Class" name to a submitted job. Various criteria is available. The easiest to think of is "If Jobname =
XYZ, then assign Transaction Class = ABC." Once the Transaction Class (TC) name is assigned, that is passed
to the endpoint server where the batch application resides. The Controller Region works with WLM to classify the
work and the TC name is passed into WLM. That results in a Service Class and/or Report Class being assigned to the job.
Once the Service Class or Report Class has been assigned, then two use cases suggest themselves. They are
spelled out on the chart above. Of the two, the use of separate Report Class is the more common. This would allow batch jobs to be grouped into a relatively few categories of Report Class, which WLM then uses to
accumulate system resource statistics for all work operating under those Report Classes. So imagine 1000
different batch jobs, with 700 categorized under Report Class A and 300 under Report Class B. Those batch jobs
run at various times throughout the day. RMF may then be used to report on the system resource statistics for
everything that ran under A and everything that ran under B. That is a very good way to capture details on a group
of jobs, rather than having to accumulate information individually.
IBM z/OS Java Batch Solutions
49
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD49
Version: September 10, 2013
Job Usage Information -- SMF 120.9 on z/OS
Positioning …
In addition to capturing job statistics in a relational table (all platforms), on z/OS the SMF facility can be used to capture batch job information.
Job Number
Submitter ID
Total CPU used
Job state
Node/Server
Job start time
Last update time
Accounting string
Job Number
Submitter ID
Total CPU used
GP Time used
Job state
Node/Server
Job start time
Last update time
Accounting string
All Platforms z/OS SMF 120.9
The information provided is very similar between "all platforms" and "z/OS SMF 120.9"
SMF 120.9 augments CPU information with GP usage. You can derive offload: Total - GP = Offload
If z/OS SMF is part of other job accounting and chargeback purposes, then SMF 120.9 for WebSphere Java Batch is simply part of that.
An example of taking advantage of the platform services
Being able to account for what batch jobs were run, who ran them, when they ran, and how much resource they
consumed is an important part of managing any batch environment. The WebSphere Java Batch function
provides accounting information on all supported platforms by writing the accounting information into a defined
relational database table. The information captured is shown in the left side of the chart above.
Note: that includes z/OS. If you have WebSphere Java batch on different operating systems and you wish to
consolidate to a common set of accounting information, you may have the z/OS capture to a relational table in
exactly the same manner that it does on other platforms. You may use SMF 120.9 in place of or in addition to this
common accounting information capture.
SMF 120.9 in WebSphere Java Batch on z/OS provides two things:
1. A way to use SMF 120.9 for Java Batch if SMF is used for other job accounting in the mainframe environment.
In other words, by providing SMF 120.9 for Java Batch it allows accounting for this to easily merge into existing
skills and existing processes.
2. The ability to derive the offload achieved in Java Batch processing. The SMF 120.9 accounting information
extends the CPU used value to include GP used. Therefore, the offload amount is a derived value -- subtract
GP from Total to arrive at the offload amount.
IBM z/OS Java Batch Solutions
50
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD50
Version: September 10, 2013
Positioning WebSphere Java Batch
Summary …
The following provides a high-level positioning of the alternatives discussed so far:
JVM Model
Batch Runtime WebSphere Java Batch
COBOL Invocation Both WaysJava invoke COBOL or COBOL
invoke Java; z/OS 2.1 adds PL/I as wel. You write the JNI glue code
One WayJava invoke COBOL but not
the other way around. JNI glue code supplied with runtime.
Bit-mode 31-bitExpectation is it will be
used with COBOL
31-bit or 64-bit31-bit if COBOL container;
64-bit possible if not
JVM LauncherImplies JVM instantiation and tear-
down for each invocation
Long Running JVMBuilt on WAS server model where
servers stay up for as long as needed
Data Access Full Java EEAny data access protocol supported by Java EE WAS itself (JDBC, JCA,
JMS, WOLA, etc.)
JDBC and tVSAMIn z/OS 2.1 support for JDBC T2
and transactional VSAM.
Batch Services Full Java BatchEverything Batch Runtime has
plus Dispatcher and PJM
Subset of Java BatchIn z/OS 1.13 TX services; in z/OS 2.1 very close to full Java Batch, but no PJM
Programming Model WebSphere Java BatchFull Java Batch programming
model. Ability to wrapper Java main(). Full Java EE runtime.
WebSphere Java BatchIn z/OS 2.1 the support is based on Java
Batch. Difference: no PJM. Also, only JDBC T2 and tVSAM, no other Java EE access.
Here is the positioning chart for WebSphere Java Batch compared to the Batch Runtime. We are not showing
BPXBATCH and JZOS here because those are JVM launchers without a batch support infrastructure provided,
but the Batch Runtime is based on WebSphere Java Batch so a comparison is useful.
Note: the JZOS toolkit, which provides access to z/OS services, may be used by either the Batch runtime or
WebSphere Java Batch.
The general positioning is this -- if a JVM launcher technology fits the nature of your batch requirements, and you
do not need access to WAS classes or a Java EE runtime, then the Batch Runtime can work for you. But if you
wish a more comprehensive batch runtime built on a long-lived Java EE platform, then WebSphere Java Batch is
the choice.
IBM z/OS Java Batch Solutions
51
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD51
Version: September 10, 2013
Section Summary
CICS, DB2 …
A very broad summary of the WebSphere Java Batch solution:
WebSphere Java Batch represents a full-function Java Batch framework and runtime model
Much more batch function than BPXBATCH or JZOS
It is built on WebSphere Application ServerWhich means all the functional value statements of WAS accrue to the Java Batch environment. On z/OS, all the "Why WAS z/OS" value statements accrue as well.
Integration with Enterprise SchedulersIt provides a messaging-based interface to the Dispatcher function so Enterprise
Scheduler functions can treat WebSphere Java Batch as any other batch job. This allows WebSphere Java Batch to be part of larger orchestrated batch flows.
Here is the summary for this section.
IBM z/OS Java Batch Solutions
52
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD52
Version: September 10, 2013
Java Batch in CICS, DB2Exploring how Java Batch is implemented in those key data systems
IBM z/OS Java Batch Solutions
53
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD53
Version: September 10, 2013
WebSphere Java Batch and Access to CICS
Feature Pack for Modern Batch …
Imagine a WebSphere Java Batch application needing access to CICS for data required to complete the batch job. How can that CICS access be accomplished?
WebSphere AppServer Instance
EJB Container
Web Container
Batch Container
Java Batch Application
CICS Program
CICS Region
Dispatcher FunctionxJCLxJCL
Job Properties Declaration File
What options exist for this interaction?
Web ServicesUbiquitous, open-standard protocol that spans platforms
Overhead per invocation somewhat high, particularly if transaction required. Not a good solution if latency per invocation a concern.
CTG JCAProven WAS-to-CICS interaction pattern.
WAS-to-CICS global TX
If EXCI, then 32K limit
If IPIC, then TCP stack code path is invoked
WOLAHighly efficient WAS-to-CICS interaction pattern
WAS-to-CICS global TX
Same LPAR required
The objective of this chart is to imagine how a Java batch program running in WebSphere Java Batch might gain
access to a CICS program as part of the batch processing. The chart is showing three potential means of
accessing CICS. All are capable of performing the task, though each has its own advantages and disadvantages
associated with it.
Note: the purpose of this is to set the stage for the next function we'll discuss, which is a means of extending the
WebSphere Java Batch model to include batch endpoints that run in the CICS region. That puts the Java batch
program right next to the CICS resources and the ability to directly invoke the JCICS interface to access CICS
services. When a Java batch architecture calls for access to CICS resources, the next feature may well be the
right solution.
IBM z/OS Java Batch Solutions
54
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD54
Version: September 10, 2013
CICS TS Feature Pack for Modern Batch V1.0
Programming Model …
This allows a CICS region to become an job endpoint for a WebSphere Java Batch dispatching server. Puts batch logic much closer to the CICS data:
Dispatcher FunctionxJCLxJCL
Job Properties Declaration File
FP Modern Batch 1.0
Java Batch Application CICS
Program
JCICS Interface and Services
CICS Region TS4.2 or TS5.1
XML file in CICS Feature Pack setup lets that batch container communicate with the WAS Dispatcher Function host and port. That lets the Dispatcher know the
endpoint is present and what batch applications are deployed.
The application in the CICS batch container has a programming model very similar to that used by WebSphere Java Batch. Benefit is access to JCICS services and
very close proximity to the CICS program the batch application requires.
The batch job management is maintained in WebSphere Java Batch. The dispatching may take place to either WAS endpoints or CICS endpoints or both.
WAS V8.5 or aboveAny Platform
Communicates availability and applications deployed in container
xJCL submitted with application in CICS container gets dispatched
xJCL submitted with application in WAS container gets dispatched
The CICS TS Feature Pack for Modern Batch installs into the CICS region and presents itself to a WebSphere
Java Batch dispatcher as another endpoint able to dispatch work to. That provides Java batch management in the
hands of the WebSphere Java Batch runtime model with CICS being an endpoint.
The Feature Pack for Modern Batch function is configured to communicate over the network to the Dispatcher,
telling the Dispatcher about the CICS Feature Pack presence and the Java batch application deployed there.
Then when an xJCL is submitted to the Dispatcher for the Java batch program deployed in the CICS Feature
Pack, the Dispatcher communicates across the network to invoke and monitor the progress of the batch program.
Note: think of the CICS Feature Pack for Modern Batch as just like a WebSphere Java Batch endpoint function,
except it runs in CICS. It uses the same programming model as WebSphere Java Batch. The Java batch
program may then access CICS services directly. This is ideal for cases where the Java batch programs calls for
intensive interaction with CICS.
The integration with Enterprise Scheduler discussion earlier is exactly the same in this scenario since the
WebSphere Java Batch dispatcher is the job management function. WSGRID interacts with the Dispatcher, and
the Dispatcher interacts with CICS to run the batch program. From the Enterprise Scheduler point of view it's all
hidden behind the WSGRID proxy job.
IBM z/OS Java Batch Solutions
55
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD55
Version: September 10, 2013
Programming Model for CICS Modern Batch FP
Positioning …
The programming model is based on the WebSphere Java Batch model. There are a few restrictions plus some added functions:
Essential Programming ModelEverything discussed in the earlier section on WebSphere Java Batch applies to the CICS Modern Batch FP as well, except Java EE functions and those things noted below.
Some Batch Data Stream Patterns Not SupportedThe following patterns of WebSphere Java Batch are not supported in the CICS Modern Batch Feature Pack environment:
JDBCReader and JDBCWriterLocalJDBCReader and LocalJDBCWriter
See below ... Feature Pack provides its own JDBC patterns
Additional Batch Data Stream Patterns ProvidedThe CICS Modern Batch Feature Pack adds the following BDS patterns:
VsamKsdsReaderImpl -- read from a VSAM KSDS as input data to the job.
VsamKsdsUpdaterImpl -- update in a VSAM KSDS file.
CICSJDBCReader -- retrieve data from a database using a Java Database Connectivity (JDBC) connection.
CICSJDBCWriter -- write data to a database using a Java Database Connectivity (JDBC) connection.
In addition, the CICS Modern Batch Feature Pack provides access to the full set of JCICS APIs
Here we make the case that the programming model for the CICS Feature Pack for Modern Batch is compatible
with that of WebSphere Java Batch. It's not entirely compatible, but it's very close.
Note: The JVM Server inside of CICS does not provide a Java EE runtime model. It's Java SE. So in this sense
the programming model is like that provided in the Batch Runtime. So while not exactly the same programming
model, the essential structure of represented earlier -- job steps acting upon a data record, looping controlled by
the batch container, checkpoint and commit/rollback handled by the container -- is the same.
There are restrictions on some of the Batch Data Stream patterns supported. The WebSphere Java Batch JDBC
patterns are not supported, but the CICS Feature Pack supplies its own patterns for JDBC as well as VSAM. And
the JCICS APIs are also available to the Java batch program that runs in the Feature Pack.
IBM z/OS Java Batch Solutions
56
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD56
Version: September 10, 2013
Positioning CICS Modern Batch Feature Pack
DB2 Stored Procedures …
The following provides a high-level positioning of the alternatives discussed so far:
WebSphere Java Batch Modern Batch FP
JVM Model Long Running JVMBuilt on WAS server model where
servers stay up for as long as needed
Long Running JVMBuilt on CICS JVM server model; JVM
stays up for as long as needed
Programming Model WebSphere Modern BatchCI or Transactional Batch
WebSphere Modern BatchCI or Transactional Batch
Multi-Step YesMulti-step jobs supported
YesMulti-step jobs supported
Job Dispatching Job Dispatching FunctionJob dispatching is handled
through the WebSphere Java Batch "scheduler" interface
Job Dispatching FunctionJob dispatching is handled
through the WebSphere Java Batch "scheduler" interface
Parallel Job Manager YesSupports WebSphere Java Batch
PJM interfaces and operation
YesSupports WebSphere Java Batch
PJM interfaces and operation
Enterprise Schedulers YesIntegration through WSGRID and
MDB interfaces to dispatcher function of WebSphere Java Batch
YesIntegration through WSGRID and
MDB interfaces to dispatcher function of WebSphere Java Batch
This is the positioning chart for WebSphere Java Batch compared to the CICS Feature Pack for Modern Batch.
You'll notice that the two are very similar. Nearly identical, in fact.
Note: this chart does not reflect the differences highlighted on the previous chart; that is, Batch Data Stream
differences and the fact the CICS JVM Server is Java SE not Java EE. Otherwise, very close functionality
between the two.
If you pause and think about it, this makes sense -- the CICS Feature Pack for Modern Batch is designed to be an
extension of the WebSphere Java Batch runtime model. If the two endpoint designs were radically different the
"extension of WebSphere Java Batch" model would break down. But because it's designed to be highly
compatible, the model holds.
It's important to understand the CICS Feature Pack for Modern Batch is operational only if there's a WebSphere
Java Batch dispatcher function deployed somewhere that serves to dispatch and monitor the Java batch running
in CICS. That WebSphere Java Batch dispatcher may be anywhere on the network, on z/OS or non-z/OS. But it
has to be somewhere as without it there would be no way to manage and run the batch programs in the Feature
Pack.
So again, think of the CICS Feature Pack for Modern Batch as a way to extend the management domain of a
WebSphere Java Batch dispatcher. Then the design becomes clear -- Java batch running on server endpoints
where it makes the most sense, with a common control and monitoring manager (the Dispatcher) at the heart of it.
IBM z/OS Java Batch Solutions
57
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD57
Version: September 10, 2013
High Level: DB2 Stored Procedures Written in Java
Summary …
Stored procedures allow for compact and efficient SQL calls from Java. Writing them in Java allows the execution of the stored procedure to be offloadable:
DB2 for z/OSV7 or above
WebSphere AppServer Instance
EJB Container
Web Container
Batch Container
Java Batch Applications
z/OS Operating System
System z HW Platform
Java source for DB2 Stored Procedure
Compiled Java classDB2 Stored Procedure
WLMWLM
StoredProcedure
Address spaces established by WLM to run JVM and Stored Procedures
NUMTCB in proc determines how many JVMs per address space; JVMs single threaded and re-used
CREATE PROCEDURESQLAPPL ENVJCL Proc
Reference: http://www.redbooks.ibm.com/abstracts/sg247604.html
Details behind all this, obviously ... key point is DB2 supports Java as a language for Stored Procedures (User Defined Functions as
well) and those may be called by batch processes
This shows WebSphere Java Batch, but any batch environment capable of calling SP in DB2
may exercise Java-based Stored Procedures
It may seem odd to include DB2 Stored Procedures as part of this discussion because Stored Procedures are not
really batch programs in the sense we're using that term in this presentation. But Stored Procedures may well be
part of the DB2 process used by the Java Batch program calling the Stored Procedure.
Note: this topic is presented to complete the picture and show how even DB2 Stored Procedures can run using
Java with offload to the zAAP processors. The WebSphere Java Batch model (or any Java batch model we've
discussed that can call DB2) doesn't really know or care that the Stored Procedure is written and runs in Java. But
you care because of the benefits it provides -- leveraging Java skills, Java tooling, and specialty engines.
When a Java Stored Procedure is called, DB2 works with WLM to create address spaces inside of which JVMs
are established to run the code. WLM controls how many address spaces based on goals defined, and within any
given address space the parameter NUMTCB is used to determine how many JVMs to host within the address
space. Each JVM is single-threaded and is re-used after the Stored Procedure has been run.
The Redbook referenced in the URL on the chart is an excellent source of information about this topic and, in fact,
the much broader topic of Modern Batch.
IBM z/OS Java Batch Solutions
58
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD58
Version: September 10, 2013
Section Summary
Decision Management …
A very broad summary of the CICS and DB2 section:
The CICS Modern Batch Feature Pack extends WebSphere Java Batch management/execution realm
It allows CICS to participate as a WebSphere Java Batch endpoint server.
A CICS endpoint puts batch logic closer to CICSWhat work should run in the CICS endpoint is a function of the batch job and the
nature / intensity of its CICS interactions
DB2 Stored Procedures and JavaNot really "batch" in our normal sense, but it can part of a larger batch process.
Stored procedures can make batch execution more efficient, and Java stored procedures allow for further exploitation of Java on z/OS
The summary chart.
What this section set out to do was to show how "Modern Batch" can extend beyond the WebSphere Java Batch
model we showed earlier. That highlights the fact that when designing a Modern Batch environment, there are
many options available to you to use what's appropriate given your needs.
Note: what about IMS? It too has facilities for Java batch. It's not included in this presentation because the topic
is fairly extensive and experience has shown those interested in IMS are typically interested in a more detailed
treatment of the topic. So if you are interested in IMS, by all means please contact your IBM IMS expert to discuss
the matter.
IBM z/OS Java Batch Solutions
59
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD59
Version: September 10, 2013
Batch and Decision ManagementIntegrating Java Batch with WODM
IBM z/OS Java Batch Solutions
60
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD60
Version: September 10, 2013
Batch Step Decision Logic -- In Step or Outside Step?
Operational Decision Manager …
Imagine a batch job that reads data records and acts upon those records based on a set of decisions (or rules). Where would you prefer those rules be?
Job
Step
Read
Process
Write
Rules
Step
End
Decision logic written within the batch step
Job
Step
Read
Process
Write
Step
End
Rules Execution
Server
Call
Result
Rules Designer Tooling
Rules Repository
Rules
All else equal, it provides greater flexibility of design,
change and execution to externalize the rules
What happens when a batch program hits a point within its processing where a decision about how to process
needs to be made. Where is that decision logic best housed?
One place is in the batch program itself. And in fact in the past that's exactly how it's been done. Many thousands
of COBOL batch programs in existence today have business decision rules hard-coded deep into the batch
processing logic. Changing those rules when needed is complicated and time-consuming.
Which is why rules engines have become quite popular -- they provide a means of externalizing rules design and
rules execution from the mainline program. That enhances flexibility and change agility.
Note: what the chart above is showing is how a rules engine such as IBM's Operational Decision Manager can be
used with batch programs to take advantage of externalized rules. The intent is to foster a discussion of decision
logic within batch processing today and explore how a product like IBM's Operational Decision Manager may
provide additional value.
IBM z/OS Java Batch Solutions
61
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD61
Version: September 10, 2013
IBM Operational Decision Manager
RES Options …
Provides real-time handling of business rule execution and situational awareness via correlation of business events received:
Decision Model:Given a snapshot view of data, determines best course of action at a specific point in a process or application
Main Purpose:To automate a decision based on a combination of factors such as business policies, regulations, and best practices.
Example:If the customer is not primary card holder and age is less than 21 then freeze account and notify primary card holder
Time-Based Pattern Detection:Correlating events as data is in motion
Main Purpose:To determine what of interest is transpiring and coordinate one or more responses by other systems or generate alerts to people
Example:If more than 2 customer withdraws in an ATM are done in the same day and the 2 ATMs are from 2 foreign countries then Investigate possible fraud. Reduce cash redraw max amount to 100$
Business Rules Business Events
IBM's Operational Decision Manager (ODM) provides two key functions -- handling business rules (the point of the
previous chart) and business events.
Note: this chart is included to give ODM proper credit for the scope of what it does. The main focus for Java
batch is more the business rules capability of ODM. But business events management may well be of interest for
other aspects of the business. So this chart simply paints the broader picture.
IBM z/OS Java Batch Solutions
62
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD62
Version: September 10, 2013
High Level: Rules Execution Server Options
JSR 352 …
The Rules Execution Server (RES) has a few means of placement in z/OS architecture:
zRES
zRES
RES on WAS for z
COBOL
Java
COBOL
IMS
COBOL
Batch
CICS Region
WebSphere Application Server for z/OS
Started Task
The Rules Execution Server may run either in a CICS region, as a
standalone started task, or inside the Java EE runtime of WAS z/OS
Our focus is on how a Java batch application in WebSphere Java Batch may access the RES, and it's by running it in WAS as well.
Invocation patterns on WAS include EJB and remote EJB.
Depending on proximity of Java Batch to RES, one of those two
would be optimalOther interfaces: Web Services and JMS/MDB
WO
LA
This is a somewhat busy chart. It is intended to show the ways a batch program -- either COBOL or Java -- may
access the Rules Execution Server (RES) of Operational Decision Manager.
Two flavors of zRES are shown here -- one is native code (not Java), and the other is Java and runs in
WebSphere Application Server. Since our primary focus is on Java batch, the invocation path from that into RES
for WAS on z would be direct invocation of the RES EJB from the Java batch program. Other interfaces exist,
including Web Services and JMS/MDB, so there's flexibility in the design.
If the design called for RES to be run in WebSphere Application Server for z/OS as a common hosting platform for
ODM on the mainframe, then COBOL access into the WAS z/OS runtime is provided by WebSphere Optimized
Local Adatpers, or WOLA. WOLA is a high-speed cross-memory copy mechanism that is very efficient for calls
such as decision rule calls.
For more on WOLA -- ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101490
For more on ODM on z/OS -- ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102291
IBM z/OS Java Batch Solutions
63
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD63
Version: September 10, 2013
Open Standard Java BatchA look at the JSR 352 specification
IBM z/OS Java Batch Solutions
64
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD64
Version: September 10, 2013
JSR 352 - "Batch Applications for the Java Platform"
Generic Diagram …
The V1.0 final release is dated April 18th of 2013. IBM served as specification lead on this JSR. It is available for download from the web at the following URL:
http://jcp.org/aboutJava/communityprocess/final/jsr352/index.html
The document has a very nice section titled "Domain Language of Batch" which provides an overview of key concepts and key
terminologyAs well as detailed sections on the specification interfaces and other details of the specification
If interested in this JSR, download and review is encouraged
An open standard for Java batch has emerged. It is referred to as JSR-352, and that specification defines a set of
functions and interfaces a standard Java batch platform will implement. Representatives from several companies
participated in the JSR development, including IBM. IBM's Chris Vignola served as specification lead for this
project.
The JSR final release document may be downloaded at the URL presented on the chart.
Note: the JSR document is surprisingly approachable and readable, particularly the "Domain Language of Batch"
section, which provides an excellent overview of key concepts. If you're at all interested in the JSR 352 topic,
downloading this document and reading it is highly recommended.
IBM z/OS Java Batch Solutions
65
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD65
Version: September 10, 2013
Generic Diagram for Domain Language of JSR 352
Comparison …
The diagram is generic but useful in that it provides a few key conceptual building blocks on which the rest of the specification relies:
JobOperator Job Step
ItemReader
ItemProcessor
ItemWriter
JobRepository
1 * 1
1
1
1
A job has one to many steps
Retrieval of input, one item at a time
Business processing for an item
Output of the step -- item or chunk
Interface to manage aspects of job processing
Holds information about jobs currently running and jobs that have run in the past
Concepts here are applicable to batch of any flavor -- historic COBOL, WebSphere Java Batch, JSR 352.
The chart above comes from the JSR-353 final release document. It provides an abstract representation of a
batch structure with the terminology used in the JSR.
This maps well to WebSphere Java Batch because the concepts employed by WebSphere Java Batch are nothing
new. Batch processing has been around for a very long time and the concepts have fairly well settled around a
recognizable pattern. The terminology used is not always the same, but the fundamental concepts are.
Note: think of the JobOperator as relating to the Dispatcher function of WebSphere Java Batch. The Job and
Step blocks in the picture map to the Job and Step representation in xJCL we spoke of earlier. The ItemReader,
ItemProcessor and ItemWriter concepts above map to the representation of the batch step where we spoke of a
data record being retrieved and acted upon. The JobRepository concept maps to the relational tables used by
WebSphere Java Batch.
It's important to recognize that the JSR-352 standard is relatively new, and WebSphere Java Batch does not at
this time implement the standard as released. The preceding paragraph was meant simply to show how the
concepts of JSR-352 map to the structural concepts of WebSphere Java Batch.
So that naturally leads to the question ... how would WebSphere Java Batch compare to the specification as
drawn up in JSR-352? The next chart provides that.
IBM z/OS Java Batch Solutions
66
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD66
Version: September 10, 2013
JSR 352 Comparison to WebSphere Java Batch
Possible Integration …
A good deal of functional similarity with a few points where each has differential:
From Chris Vignola's IMPACT 2013 Presentation, TAW-1443
This chart shows a set of features and how the JSR 352 standard addresses those features and how WebSphere
Java Batch does. There's a good many features where "yes" applies to both columns.
The three functions supported by JSR 352 but not WebSphere Java Batch are concurrent steps (WebSphere Java
Batch step execution within a job is sequential), restart substitution (the ability to substitute in new values on job
restart; WebSphere Java Batch has a restart function, but does not allow value substitution on restart), and CDI
refers to dependency injection, which WebSphere Java Batch does not support.
On the other hand, WebSphere Java Batch supports several things the JSR 352 spec does not call out -- a Job
Dispatcher function (JSR 352 defines a JobOperator interface but how that's implemented is not declared), job
logs (JSR 352 does not have specific details on job log output formatting), and Distributed Parallelization, which is
the PJM function we discussed earlier.
Note: the objective here is not to claim one is better or worse than the other. IBM understands the value of open
standards and embraces their development and adoption (IBM lead the JSR-352 specification). The objective
here is first and foremost to stress the commonality of JSR-352 and WebSphere Java Batch at the functional
level, and then to provide an understanding of where differences exist.
IBM z/OS Java Batch Solutions
67
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD67
Version: September 10, 2013
Theoretical JSR 352/WebSphere Batch Integration
References …
A topology picture showing both models working cooperatively:
From Chris Vignola's IMPACT 2013 Presentation, TAW-1443
Common Batch Container
WebSphere Java Batch Applications
Dispatcher FunctionxJCLxJCL
WebSphere Java Batch
JSR 352 Java Batch Applications
JSLJSL
JSR 352 JSL
Common Job Repository
Common Job Logging
This implies no promise of future delivery ... that said, the model shown above is well within the possible given the nature of JSR 352 and the
similarities with the WebSphere Java Batch model.
WebSphere Java Batch as well as JSR 352
JobOperator function
The chart above provides a potential means by which existing WebSphere Java Batch can be enhanced to
provide both JSR-352 (new function) and WebSphere Java Batch (protect existing investment).
Note: normal disclaimers apply ... this does not represent a promise to provide function ... IBM reserves the right
to modify its plans at any time. Please, this chart is intended to suggest a potential path, but is not a promise of
one.
The previous chart illustrated how the two approaches are more similar than different. That means there's good
opportunity to have concurrency of supported function within the framework offered by WebSphere Java Batch.
Take for example the current Job Dispatcher function. It currently accepts only xJCL. But in theory it could be
enhanced to accept both xJCL and the JSR 352 JSL (Job Submission Language ... also XML, but with a syntax
different from xJCL). Similarly, the container function currently implemented to the WebSphere Java Batch model
could be enhanced to support both runtime models.
The key point here is that IBM was active in the formulation of JSR 352 and thus is makes sense that IBM would
be active in its adoption. How exactly that adoption is implemented and the timeline for implementation is not
spelled out here.
IBM z/OS Java Batch Solutions
68
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD68
Version: September 10, 2013
ReferencesA guide to reference information for further details
IBM z/OS Java Batch Solutions
69
© 2013 IBM CorporationIBM Americas Advanced Technical Skills
Gaithersburg, MD69
Version: September 10, 2013
References
http://www.ibm.com/developerworks/systems/library/es-java-batchz.html
JZOS vs. BPXBATCH
http://www.ibmsystemsmag.com/mainframe/trends/whatsnew/java_compiler/
SDK 7 features with EC12
http://www.ibm.com/systems/z/os/zos/tools/java/products/jzos/overview.html
JZOS
z/OS 2.1 announcement with information on Batch Runtime improvements
http://www.ibm.com/common/ssi/rep_ca/3/897/ENUS213-013/ENUS213-013.PDF
WebSphere Java Batch
http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783
CICS Transaction Server Feature Pack for Modern Batch V1.0
http://www-01.ibm.com/software/htp/cics/batch/
A few references. Other URLs were provided throughout the charts.
End of DocumentEnd of Document