ims database primer
TRANSCRIPT
IMS Database Primer
Original Material Created by:
Dennis EichelbergerIT SpecialistIMS Support – Washington System Center
Modified for the Canadian Eastern Region IMS User Groups by:
Gary Wicks, IT Specialist - IMS Lab Services
Contents
• Short history and business positioning for IMS• Architectural structural fundamentals• Value descriptions for IMS• DB hierarchical structures• Defining databases• Defining programs• ODBM• SQL access and the Catalog• IMS DB education available2
3
Information Management System (IMS)
Originally created for the NASA Apollo Projectto put a man on the moon by IBM and Client corroborations
Success!
4
Where the World Depends on IMS• Manufacturing
– Aerospace, equipment, automobile, …
• Banking• Telecommunications• Bulk Transportation: Air and Rail• Travel: Air, Car and Rail• Health Providers• Insurance• Finance/Credit Card Processing• Package delivery• Government• Retail• …
IMS Architecture Fundamental Concepts• Database management (IMS DB component)
– Separated from application programming• IMS realized that moving the definition of the data out of the program would have substantial benefits when many programs could access the same database
• Transaction management (IMS TM component)
– Separated from application programming• The sending and receiving of input and output messages was removed from programs
– Allows the same program to access the message regardless of what type of device was used to send in the message
– IMS's message queues were critical to its initial design
• Multiple region online system
– Applications run in their own regions• Also called partitions, address spaces, …
5
Possible Xmemory access from and between IMS address spacesDB2 APPC IRLM Others
DL/ISAS
CTRL
...
DRA
CCTL MPP BMP IFP
Multiple application threads per address space for each CCTL and ODBA region
Your application programs run in the green address spaces in this bottom row
JMP JBP
MQSeries RRS
Appl Region
ODBA
WLM
SPOC
TCPIP
VTAM Console
WEBSPHERE
IXGLOGR
IMS Connect(BPE)
...
Tools
...
DBRC CQS(BPE)
OMCSL(BPE)
RMCSL(BPE)
SCICSL(BPE)
ODBMCSL(BPE)
IMS Address Space Structures
6
Compatibility: The Foundation of IMS Success• Application compatibility
– Programs written in 1969 still work– IMS does not require recompiles of applications for new IMS releases
• Database compatibility– Databases do not require upgrades for new IMS releases
• Application programs are independent of target database type and access method utilized
• IMS has allowed users to grow their applications in lieu of forcing major conversions
7
Adaptability • And yet IMS also keeps up with technological trends
– IMS provides a TCP/IP socket server – IMS Connect• For access to IMS transactions and databases
– Allows IMS applications to be either clients or servers – Delivers adapters that run in JEE, SOAP, etc. environments
• Pluggable components that provide easy interfaces to IMS resources – Supports type-2 and type-4 universal drivers for access to IMS DB
• IMS provides application programming interfaces– Unique to IMS
• DL/I calls to access IMS resources: message queues and data• IMS applications can be written in: COBOL, PL/I, Assembler, C/C++, Pascal, ADA, REXX, Java
– Portable – Java classes and JDBC 8
Programming Interface• Database definitions reside as IMS meta data outside of programs
– DBDs • Defines the database structure
– PSBs• Views of application sensitivity to segments within databases• Contain access definitions (get, insert, delete, replace)
• Programs– DL/I calls used for database access
• Program not aware of data sets and physical characteristics of data
• DL/I calls used for input/output messages– Program independent of communication protocols– IMS applications get messages from a queue, process them, and
insert them back to the queue
9
What New Users of IMS Need to Understand• IMS Databases (this is todays introduction)
• IMS Online Systems and Transaction Management
• IMS Connect
• Application Programming Interfaces to IMS
• IMS in a Parallel Sysplex®
10
JJ
II
H 2
GGF
E
DDDCC
IMS Database Structure• IMS databases are hierarchical
A
B G
C
F
IE
J
H 1D
Root segment – Segments have parents and children• Segments are accessed through their parents
• Parents and Children are relatedD is accessed through A and B
– Segments have twins• Multiple B segments under one A segment• Twins are related • H2 is accessed via H1
- Segment pointers are updated by IMS at the same time the segment is updated
A segment type (e.g. A, B, or C) is similar to a relational table.A segment occurrence, one of the segments, is similar to a row in a relational table.An IMS database is similar to several tables which are joined.
Tree structure
11
IMS Database Structure …• Full Function database types
– Hierarchical indexed direct (HIDAM)– Partitioned HIDAM (PHIDAM)– Generalized Sequential (GSAM)– Hierarchical indexed sequential (HISAM)*– Hierarchical Sequential (HSAM) *– Hierarchical direct (HDAM)– Partitioned HDAM (PHDAM)
* Simple version available
• Fast Path database types– Data Entry Database (DEDB)– Main Storage Database (MSDB)
• MSDBs are root-only and stored & accessed only in memory
§ Indexed databases– HIDAM, PHIDAM, HISAM– Roots are ordered in key sequence
§ Non-indexed databases– HDAM, PHDAM, DEDB– Randomization routines are used to
order roots
Structure is chosen based on what makes most sensefor access patterns, e.g., sequential versus random
12
Value and use of Hierarchical Structures• GSAM
Usage Characteristics
Insert sequentially – time order Delete unavailable
Used for IMS BMP and Batch jobs requiring restartability
13
Value and use of Hierarchical Structures• HSAM
Usage Characteristics Not available
Insert in key sequence Secondary Indexing
Logical relationships
Variable length segments
Update not available DLET or REPL call
Used for low volume processing e.g. data archival
14
Value and use of Hierarchical Structures• HISAM
Usage CharacteristicsSame size records
Complete record fits into logical defined recordFew Deletes or Inserts Delete / Insert Available
Root Delete Space is reclaimed in primary area, not reclaimed in overflow
Dependent Segment Marked deleted, Space not available
Database reorganization required to reclaim physical space
15
Value and use of Hierarchical Structures• HDAM
Usage CharacteristicsSegments are unordered Position based on key randomization
Segment relationships via pointers Dependents may be in separate blocks
Delete removes segment Space is reclaimed
High activity Speedy segment retrieval and update
16
HDAM Database Randomization
17
RAP1 RAP2 RAP3
SEG12 SEG36 SEG82
SEG04 SEG01 SEG95
Block 3RAP1 RAP2 RAP3
SEG15 SEG06 SEG58
SEG65 SEG47 SEG13
Block 2RAP1 RAP2 RAP3
SEG02 SEG68 SEG77
SEG47 SEG02 SEG81
Block 1
RAP1 RAP2 RAP3
SEG01 SEG14 SEG55
SEG01 SEG01 SEG73
Block 4
RAP1 RAP2 RAP3
SEG71 SEG53 SEG01
SEG85 SEG84 SEG29
Block 6RAP1 RAP2 RAP3
SEG05 SEG04 SEG90
SEG48 SEG19 SEG76
Block 5
DATABAS1
RandomizerDFSHDC40
RMNAME=(DFSHDC40,3,3,25)
Value and use of Hierarchical Structures• HIDAM
Usage CharacteristicsTwo DBDs required 1 = Index 2 = Data area
Root Segments are sequentially ordered Index may be processed separately
Segment relationships via pointers Dependents may be in separate blocks
Delete removes segment Space is reclaimed
High activity Speedy segment retrieval and update
18
Large DB Size Support Through HALDB and Fast Path DEDBs• HALDB (PHDAM and PHIDAM) and DEDB databases
– Database is composed of multiple database data sets• HALDB (High Availability Large Database) calls these datasets partitions• DEDB calls these datasets areas
– HALDB may have up to 1001 partitions• Each partition may have up to 10 4GB data sets
– Over 40 terabytes!• In IMS 14 the OSAM data sets may be up to 8GB
– Over 80 terabytes
– DEDBs may have up to 2048 areas• Each area has one data set up to 4GB
– Over 8 terabytes
Application programs are unaware of the partitions or areas
The end user accesses their data regardless of position or residence
19
Data Base Secondary Index use
• IMS databases may also have Secondary indexes
– Indexes are based on different fields in any segment
– Example:• Secondary index into course database based on student ID– One could find all the classes in which a student is enrolled
Course
Instructor Class
Student
Sec. IndexStudent ID
ID
20
Data Base Logical Relationships• Full Function database may have logical relationships
– Logical databases are formed from multiple physical DBs– Example:
• Logical relationship between employee and department DBs
• Examples of logical databases
Employee
Depart. No.Salary History
Department
Member
Employee
Depart. No. | Depart.Salary History
Member
Member | Employee
Depart. No.Salary History
Department
21
Defining Databases• Database Definition - DBD
– Database definition is used to define:• Database type • Dataset type (primary / overflow)• Number of Segments• Hierarchical relationship of the segments• Fields within a Segment• Key fields• Data types of the fields• Logical relationships within or outside of the database• Indexing
22
Defining DatabasesDBD NAME=DI21PART, X
ACCESS=(HISAM,VSAM), XPASSWD=NO, XDATXEXIT=NO, X
DATASET DD1=DI21PART, XDEVICE=3380, XOVFLW=DI21PARO, XBLOCK=(100,10), XSIZE=(4096), XRECORD=(678)
SEGM NAME=PARTROOT, XEXTERNALNAME=PARTROOT, XPARENT=0, XBYTES=50, XRULES=(LLL,LAST), X
FIELD NAME=(PARTKEY,SEQ,U), XEXTERNALNAME=PARTKEY, XBYTES=17, XSTART=1, XTYPE=C, XDATATYPE=CHAR
SEGM NAME=STANINFO, XEXTERNALNAME=STANINFO, XPARENT=PARTROOT, XBYTES=85, XRULES=(LLL,LAST),… X
23
Defining ProgramsProgram Specification Block - PSB
– Program Specification is used to define Program Control Blocks - PCB:– Program Control Blocks define
• Database(s) to be accessed• Processing options to access it with
– Get, Insert, Delete, …• Segments the program is aware of• Fields within the segment the program is aware of• In general, the program view of the accessed database(s)
24
Defining DatabasesDBD NAME=DI21PART, X
ACCESS=(HISAM,VSAM), XPASSWD=NO, XDATXEXIT=NO,
DATASET DD1=DI21PART, XDEVICE=3380, XOVFLW=DI21PARO, XBLOCK=(100,10), XSIZE=(4096), XRECORD=(678)
SEGM NAME=PARTROOT, XEXTERNALNAME=PARTROOT, XPARENT=0, XBYTES=50, XRULES=(LLL,LAST),
FIELD NAME=(PARTKEY,SEQ,U), XEXTERNALNAME=PARTKEY, XBYTES=17, XSTART=1, XTYPE=C, XDATATYPE=CHAR
SEGM NAME=STANINFO, XEXTERNALNAME=STANINFO, XPARENT=PARTROOT, XBYTES=85, XRULES=(LLL,LAST), …
DBPCB00 PCB TYPE=DB,DBDNAME=DI21PART,PROCOPT=A,KEYLEN=44 SENSEG NAME=PARTROOT,PARENT=0 SENSEG NAME=STANINFO,PARENT=PARTROOT SENSEG NAME=STOKSTAT,PARENT=PARTROOT SENSEG NAME=CYCCOUNT,PARENT=STOKSTAT SENSEG NAME=BACKORDR,PARENT=STOKSTAT PSBGEN LANG=ASSEM,PSBNAME=DFHSAM04,MAXQ=1 END
DBPCB01 PCB TYPE=DB,DBDNAME=DI21PART,PROCOPT=G,KEYLEN=43 SENSEG NAME=PARTROOT,PARENT=0 SENSEG NAME=STANINFO,PARENT=PARTROOT SENSEG NAME=STOKSTAT,PARENT=PARTROOT PSBGEN LANG=COBOL,PSBNAME=DFSSAM01,MAXQ=1 END
25
Identifying Databases to Program Relationships• DataBase generation - DBD• Program Specification Block generation - PSB• Application Control Block - ACB
– Connects DBDs and PSBs– A PSB may access multiple databases– Multiple PSBS may access the same database– Application changes or additions may be implemented without an IMS restart.
• Affected resources may need to be quiesced for control block updates• The Online Change process is normally utilized for this
Program 1 DBD
Program 1 PSB
Program 2 ACBProgram 1 ACB
26
Database Access• All IMS Database access is though a standard programming
– DL/1 using DLI
• Data Language 1 through the Data Language Interface
– AIBTDLI Application Interface – REXX, Java using PCB labels– ASMTDLI Assembler programs– CTDLI C programs– CEETDLI C programs using PCB labels– CBLTDLI COBOL programs– PASTDLI Pascal programs– PLITDLI PL/1 programs
27
Database Access
• DL/1 call– CALL CBLTDLI USING GU dbpcb i/oarea argument
– Translates to:• LLZZtrancode ‘search argument’ landing on the Message Queue.• IMS sees ‘trancode’ and finds the (Scheduler Message Block) SMB related to that transaction
• Loads the program to start in a dependent region• Passes the input message to the program
28
Open Database Manager Access - ODBM • Implemented with Common Service Layer• Allows database access outside of dependent region• Some components are:
– IMS Universal Database resource adapter (DRDA)– IMS Universal JDBC driver– IMS Universal DL/I driver– Open Database Access interface
(ODBA)– ODBM CSL interface
• Security & Access throughIMS Connect
ClientApplication
z/OS
IMSCONNECT
Exit
IMS
Database
manager
SCI OperationsManager
IMSCONNECTExtensions
ODBM DLI
Universal DLI
WebSphereApplicationServer WAS
CSLDMI
JDBC Driver
ODBM
Database Database
RECON
Database
29
IMS Catalog• Database Considerations
– Traditional database definitions can ‘miss’ the entirety of the data • Define a segments total length
– Also can define a Key / Sequence field• The remainder is defined as ‘space’• Relies on the application program to map the ‘space’ using copy books
– Relational views of traditional IMS databases using SQL or JDBC• Require search fields to be defined in the meta data
– IMS meta data is stored in the IMS Catalog
30
IMS Catalog• IMS Version 12 added an IMS Catalog
– IMS meta data • Single source of trusted repository of meta data• Loaded and updated from IMS ACBLIB• Use IMS 14 with the IMS Explorer for Development to create DDL
– Send DDL directly to the IMS Catalog– IMPORT DEFN SOURCE(CATALOG) to activate (TYPE 2 Command)
– Uses HALDB for its layout • Registered to DBRC• Uses four dataset groups • Database maintenance procedures apply
– Image Copy– Reorganization– Recovery
– Loaded and updated from IMS ACBLIB31
Defining Databases for Eventual Use in the CatalogDBD NAME=DI21PART, X
ACCESS=(HISAM,VSAM), XPASSWD=NO, XDATXEXIT=NO,
DATASET DD1=DI21PART, XDEVICE=3380, XOVFLW=DI21PARO, XBLOCK=(100,10), XSIZE=(4096), XRECORD=(678)
SEGM NAME=PARTROOT, XEXTERNALNAME=PARTROOT, XPARENT=0, XBYTES=50, XRULES=(LLL,LAST),
FIELD NAME=(PARTKEY,SEQ,U), XEXTERNALNAME=PARTKEY, XBYTES=17, XSTART=1, XTYPE=C, XDATATYPE=CHAR
FIELD NAME=PARTNAME, XEXTERNALNAME=PARTNAME, XBYTES=24, XSTART=18, XTYPE=C, XDATATYPE=CHAR
SEGM NAME=STANINFO, X
Meta data update
1. Add fields needed to the DBD source to regen
2. FTP Source to DBD3. Generate the DBD4. Generate the ACB5. Perform an Online
Change
32
Expansion into SQL access• Understanding that the world has moved towards the relational view of databases and SQL to access them
– IMS also provides the ability to view the hierarchical database structure in a relational way
• The IMS Enterprise Suite Explorer– Creates an SQL View of the Hierarchical database
» Metadata mapping from the IMS Catalog
• And a set of java universal drivers that interpret the calls made by java programs– Local to z/OS– From remote environments
33
SQL calls to IMS databases• IMS Version 13 added direct SQL access using COBOL Version 5.1
– Builds upon the catalog trusted meta data– Allows a COBOL application program to make SQL calls to access IMS hierarchical databases• Most SQL calls are supported• IMS 14 adds more SQL support
PartitionA
PartitionB
PartitionC
PartitionA
PartitionB
Catalog Database
Catalog Secondary Index
Catalog
IMS
IMS ExplorerFor Development
ACBs
34
IMS Data Base Courses AvailableIn order to begin to master IMS DB concepts definitions and operational control and recovery, the following courses are available:
• CM220 IMS Physical Organization of Databases. 4 Day Lecture / Lab• CM301 IMS DB Performance and Tuning. 5 Day Lecture / Lab• CM20 DBRC. 5 Day Lecture / Lab• CM46 HALDB. 1 Day Lecture • CM50 IMS Data Sharing. 3 Day Lecture / Lab
– We will now introduce each course and give a sample of information contained within
Contact your IBM representative to learn more35
IMS Physical Organization Course Objectives• Code the Database Directories (DBD) and Program Specification Blocks (PSB) for physical databases• Code the DBDs and PSBs to implement secondary indexing• Use the appropriate IMS utilities to reorganize and recover physical databases, including those with secondary indexes• Use the DL/I test program• Prepare a job stream to load a database using any of the IMS access methods
36
Start with the Most Desirable Block (MDB): CI/BLOCK requiring least amount of I/O to perform pointer maintenance during insert activity
1. Check MDB for free space2. Check 2nd MDB for free space3. Block in buffer pool from same track4. Block in buffer pool from same cylinder5. Block on same track (bit map)6. Block on same cylinder (bit map)7. Block in buffer pool within +/- scan cylinder8. Block within +/- scan cylinder (bit map)9. Block in buffer pool at end of data set10. Block at end of data set (bit map) – then new dataset extent11. Any block in data set (bit map)
E S D S / O S A M
BIT MAP
MDB
DATABASE DATA SET
BIT MAP
MDB
BUFFER POOL
HDAM HIDAM
Sample from DB Physical Organization Course
HD SpaceSearch Algorithm
37
IMS DB Performance and Tuning Course Objectives• Analyze performance data about the IMS database environment• Choose IMS access methods and options that provide the best database performance• Improve performance by selecting database buffer pools and buffer pool options and with appropriate data set access method and storage attributes• Implement the optimum performance options for VSAM data sets at define and execution time• Evaluate the need for secondary indexes and select options to improve their performance
38
PROGRAM ELAPSED TIME
- Operating System (MVS-z/OS) PAGING delays- Operating System (MVS-z/OS) DISPATCHING delay
IWAIT TIME
DATABASE I/OSYNC PT PROC
LOCKING ACTIVITY WITHIN PROGRAM
ISOLATION OR IRLM
NOT-IWAIT TIME
DLI CALL PROC DATABASE OPENLOGGING I/OLWAWAIT: LOG BUFFERSLATCH REQUESTSBUFFER MGMTUSER EXITSOTHER ...
DLI ELAPSED TIMENOT-DLI TIME
APPL PGM PROC
DB2 CALLS
Sample from DB Performance and Tuning Course
39
DBRC Course Objectives• Install, implement, and manage the DBRC (Database Recovery Control) component of IMS (Information Management System)
• Identify and list the basic elements of DBRC • Describe the processing of DBRC records • Describe the data sharing environment• Describe the subsystem interaction and database authorization process • Register full-function, fast path, HALDB, non-recoverable databases, and their data sets
• Define and register database data set groups, data groups, and recovery groups
• Define and register change accumulation groups • Describe installation tasks for DBRC and RECON data sets
40
CHAN2
BCS
VOL456
P.RECON2
CACHECU2
CHAN3
BCS
VOL789
P.RECON3
CACHECU3
BCS
VOL123
P.RECON1
CACHECU1
CHAN1
Specify DASD fast write (DFW) for the RECON data sets
RECON Placement
Sample from DBRC Course
41
HALDB Course Objectives• Define HALDB databases and their partitions
• Understand database administrator responsibilities in managing HALDB
• Develop procedures for the support and recovery of HALDB
• Migrate databases to the use of HALDB
• Understand application design issues for HALDB
42
• If partition PB with high key 4000000 needs to be split:– Unload partition PB• HD Unload or HPUnload
– Define new partition PE • Sets PINIT flag forPB and PE
– Initialize partitions PB and PE– Reload partitions PE and PB• HD Reload or HP Load
– Partitions PA, PC, and PDare not affected
PA2000000
PB4000000
PC6000000
PD8000000
PA2000000
PE3000000
PC6000000
PD8000000
PB4000000
HD Unload
HD Reload
Define new partition
Splitting a HALDB Partition
Sample from HALDB Course
43
Data Sharing Course Objectives• Define IMS subsystems, IRLMs, and Coupling Facilities to use block level data sharing• Design recovery procedures for failures in the block level data sharing environment• Monitor the performance of a block level data sharing environment• Explain IRLM and Coupling Facility roles in supporting block level data sharing
44
1Subsystem A: GHU ACME (Holds DB record lock on 100 (RBA of Root or RAP)
2Subsystem A: REPL ACME (Holds block lock on Block2)
3Subsystem B: GU ZENITH (Holds DB record lock on 121)
4 Subsystem B: REPL ZENITH (Must wait till block lock on Block2 is released)
Database Records Database on DASD
100
ACME NAIL
121
ZENITH SCREW
Block1
Block2
Block3
100 121
ACME ZENITH
NAIL SCREW
Sample from Data Sharing Course• Full Function Locking Example
45
Summary• IMS is a database manager
– Supports hierarchical databases– Manages highly available and large database– May be accessed from IMS TM, CICS, WAS, DB2, distributed, any platform
• IMS is a transaction manager– Multi-address space architecture– Processes messages– Schedules and manages programs– May be accessed via VTAM, TCP/IP, MQ, …– May be accessed from any platform– Accesses IMS and DB2 databases– This is another ‘world’ to learn about
46