connect cdc sqdata - .net framework... · 2021. 3. 19. · connect cdc sqdata data capture guide 7...

Connect CDC SQData

Data Capture Guide

Version 4.0

2 Connect CDC SQData Data Capture Guide

Data Capture Guide

© 2001, 2021 SQData. All rights reserved.

Version 4.0

Last Update: 1/8/2021

Data Capture Guide

Contents

Introduction ............................................................................................... 4

Intended Audience .............................................................................. 5

Change Data Capture Summary .......................................................... 6

Terminology ......................................................................................... 7

Documentation Conventions ............................................................... 8

Related Documentation ...................................................................... 9

Change Data Capture Overview ............................................................. 10

CDC Features ..................................................................................... 11

Active/Active Replication .................................................................. 13

Authorization and Roles .................................................................... 14

CDC Metadata ................................................................................... 15

Selecting a Data Capture Agent .............................................................. 16

Transient Storage and Publishing ........................................................... 17

CDCStore ........................................................................................... 18

CDCzLog ............................................................................................ 19

TCP/IP ................................................................................................ 20


Introduction

Connect CDC SQData provides industrial-strength, enterprise-class real-time streaming replication, synchronization,enhanced ETL, business event and Big Data publishing. Connect CDC SQData provides multi-platform, multi-sourcehigh performance Changed Data Capture (CDC) that together with the Apply and Replicate Engines are part of thePrecisely Integrate gateway connecting your business infrastructure with our Data Integrity suite to unlock thepotential of your enterprise data.

This introductory section:

· Identifies the intended audience of this manual

· Summarizes the purpose of Change Data Capture

· Defines commonly used terms

· Defines documentation syntax conventions

· Identifies complimentary documents

5Connect CDC SQData Data Capture Guide

Introduction

Intended Audience

This guide provides an overview of Change Data Capture for people implementing Precisely's Connect CDC SQData.


Introduction

Change Data Capture Summary

Every organization has front and back office systems that together form the infrastructure supporting day to dayoperation and frequently contain the most current information about a particular business function. Precisely'sConnect CDC SQData Change Data Capture capability makes that real-time data available to other tactical andstrategic business functions without disrupting or requiring modification to both legacy and newer businessapplications. Changed Data Capture provides a gateway to connect your business infrastructure to Data Warehouses,Data Lakes and analytical tools designed to find the hidden value in all that data as well as improve the accuracy andusefulness of the information through Precisely's full Data Integrity suite.

Technically Change Data Capture Agents operates under constraints that vary from one platform to another. NearReal-Time data capture is accomplished on most platforms using the transaction log, maintained by the databasemanager for the purpose of database recovery. Historically, these logs have also been the source of data for otherpurposes including audit or replication. Generally it is necessary to configure the database manager to includeadditional information in the log for use by these non-recovery related activities.

This Guide provides describes Change Data Capture, its general operational characteristics and role in active/activereplication scenarios. It also includes a brief description of the capture agents, and publishing options supported andinformation on configuration, and operation.

· Change Data Capture Overview

· Selecting a Data Capture Agent

· Transient storage requirements and Publishing methods


Introduction

Terminology

Terms commonly used when discussing Change Data Capture:

Term Meaning

Agent Individual components of the Connect CDC SQData product architecture.

CDC Abbreviation for Changed Data Capture.

Datastore An object that contains data such as a hierarchical or relational database, VSAM file, flat file, etc.

ExitA classification for changed data capture components where the implementation utilizes asubsystem exit in IMS, CICS, etc.

File Refers to a sequential (flat) file.

JCL An abbreviation for Job Control Language that is used to execute z/OS processes.

Platform Refers to an operation system instance.

RecordA basic data structure usually consisting of fields in a file, topic or message. A row consisting ofcolumns in a Relational database table. Record may be used interchangeably with row or message.

SegmentA basic data structure consisting of fields in an IMS hierarchical database. Segments are recordshaving parent and child relationships with other records defined by a Database Description (DBD).

Source A datastore monitored for content changes by a Capture Agent.

SQDCONF A Utility that manages configuration parameters used by some data capture components.

SQDXPARMA Utility that manages a set of parameters used by some IMS and VSAM changed data capturecomponents.

TableUsed interchangeably with relational datastore. A table represents a physical structure that containsdata within a relational database management system.

Target A datastore where information is being updated/written.


Introduction

Documentation Conventions

The following conventions are used in command and configuration syntax and examples in this

document.

Convention Explanation Example

Regular type Items in regular type must be entered literally usingeither lowercase or uppercase letters. Items in Bold typeare usually "commands" or "Actions". Note, uppercase isoften used in "z/OS" objects for consistency just aslowercase is often used on other platforms where casemay be either enforced or optional.

create

CCSID

/directory

//SYSOUT DD *

<variable> Items between < and > symbols represent variables. Youmust substitute an appropriate numeric or text value forthe variable.

<file_name>

| Bar A vertical Bar indicates that a choice must be madeamong items in a list separated by bars.

'yes' | 'no'

JSON | AVRO

[ ] Brackets Brackets indicate that item is optional. A choice may bemade among multiple items contained in brackets.

[alias]

OR

[+ | -]

-- Double dash Double dashes "--" identify an option keyword. Somekeywords may be abbreviated and preceded by a singledash "-". A double dash in some contexts can be used toindicate the start of a single line comment.

--service=<port>

OR -s <port>

OR --apply

OR -- this is acomment

… Ellipsis An ellipsis indicates that the preceding argument orgroup of arguments may be repeated.

[expression…]

Sequencenumber

A sequence number indicates that a series of argumentsor values may be specified. The sequence number itselfmust never be specified.

field2

' ' Single quotes Single quotation marks that appear in the syntax must bespecified literally.

IF CODE = 'a'


Introduction

Related Documentation

Installation Guide - This publication describes the installation and maintenance procedures for Connect CDC SQDatafor z/OS and Linux/AIX and Windows products.

Product Architecture - Describes the overall architecture of the Connect CDC SQData product and how itscomponents deliver true Enterprise Data Integration.

Data Capture Guide - Provides an overview of the role capture plays in Precisely's Connect CDC SQData product, thecommon features Capture and the methods supported for store and forward transport of captured data to targetEngines running on all platforms

Data Capture References - Provides comprehensive information about the various Connect CDC SQData data captureagents for IMS, Db2, VSAM, Oracle, UDB (DB2/LUW) and SQL Server. They also detail the activities required for theconfiguration, operation and management for each of the data capture agents.

Apply and Replicator Engine References - Provide detail level references describing the operation and commandlanguage of the Connect CDC SQData Apply and Replicator Engine components, which support target datastores onz/OS, AIX, Linux and Windows.

Secure Communications Guide - Describes the Secure Communications architecture and the process used toauthenticate client-server connections.

Utility Guides - Describe each of the Connect CDC SQData utilities such as SQDCONF, SQDMON, SQDUTIL and the zOSMaster Controller.

Messages and Codes - Lists and describes the messages and associated codes issued by the Capture, Publisher,Storage agents, Parser, Apply and Replicator Engines, and Utilities in all operating environments z/OS, Linux, AIX,and Windows.

Quickstart Guides - Tutorial style walk through for some common configuration scenarios including Capture andReplication. z/OS Quickstarts make use of the ISPF interface. While each Quickstart can be viewed in WebHelp, youmay find it useful to print the PDF version of a Quickstart Guide to use as a checklist.


Change Data Capture Overview

Connect CDC SQData provides an efficient and effective mechanism for propagation and/or replication of datautilizing two components; Data Capture Agents and what we refer to as an Apply Engine. Connect CDC SQDatasupports both homogeneous and heterogeneous environments including zOS (mainframe), Linux and Windowsoperating systems. Source and target data may utilize different data structures and database management systemsincluding Db2, IMS, VSAM, UDB (Db2/LUW), Oracle and SQL Server where each may reside on different physicalsystems running different operating systems.

While the configuration and operation of the Apply Engine is virtually identical on all platforms, the Connect CDCSQData Capture Agents operates under constraints that vary from one platform to another. To accommodate thoseconstraints Capture utilizes various forms of Log Readers and Exits.

Near Real-Time data capture is accomplished on most platforms using the transaction log, maintained by thedatabase manager for the purpose of database recovery. Historically, these logs have been the source of data forother purposes, including audit or replication once the database manager has been configured to include additionalinformation required by the non-recovery related activities. The impact of larger logs is a minor inconveniencecompared to the additional processing required by database trigger capture methods.

Note, the "Endianess" of the source data platform is a consideration for Data Capture and then only when theCapture Agent runs remotely on a platform other than the database server. The nature of the API's used to capturethe data and the client side software provided for accessing those databases remotely requires both the Databaseand Capture Agent platforms to utilize the same encoding scheme. The methods used to transport captured data toApply Engines, generally running on other remote platforms, lifts this restriction making it possible to use the mostcost effective platforms for Apply Engines.

This section provides an overview of the Connect CDC SQData data capture framework and summarizes theindividual components involved in changed data capture.



CDC Features

Connect CDC SQData provides many options to control the capture process. Some of the more relevant featuresinclude the following:

· Changed data routing is the ability to specify that changes to a specific object such as a particular IMS segment,relational table or file will be published to a specific Engine via TCP/IP. This permits both processing andnetwork transport workload balancing .

· Auto-disable allows critical transactions/programs to continue processing in the event an error occurs duringthe data capture process. Typically, these errors involve one or more of the transient datastores used tomanage captured data before it is published. Note, when auto-disable is initiated, messages are written toboth the operator’s console and the program job log indicating that the auto-disable event occurred and theassociated error code. Optionally, a user specific WTO message may be written when auto-disable is triggered.

· Auto-Commit applies to batch ONLY IMS/DLI programs where a single unit-of-work may consist of millions ofupdates before a commit is performed. This feature is used to prevent large units-of-work from overloading asystem Log Stream by specifying the optional auto-commit frequency (i.e. how many changed data records toprocess between auto-commit events). Important Note: The auto-commit feature is only intended to providea temporary solution. Data integrity can be compromised if changed data is auto-committed and a DL/I batchprogram subsequently abends or rolls back uncommitted updates. In other words, there is no coordinationbetween the IMS and the Log Stream as far as the commit scope is concerned. Precisely recommends using theIMS Log Reader Capture for changes made by long running DLI batch programs. Speak with Precisely supportbefore using the Auto-Commit option.

· Program and transaction Include/Exclude provides for the inclusion or exclusion of specific programs ortransactions from the data capture process. For example, eliminating the capture of deletes performed bypurge or data cleanup programs.

· Other Database specific Include/Exclude options are also supported such as Db2 Correlation ID, Plan andAuthorization ID.

· Multi-Target Assignment allows data captured for a single change to drive multiple downstream processesrunning on one or more target platforms. This can be accomplished using a variety of techniques involvingboth capture agents, downstream Engines with multiple target datastores, multiple Engine subscriptions andintermediate splitter/distribution engines. Talk to Precisely support for assistance in determining how best toimplement Multi-Target processing.

· Changes to source data structures are inevitable. When new columns are added to tables or a new fieldsadded to IMS segments or Files already configured for captured the additional data is automatically included inthe captured data. The Engine will ignore the new column/field until the source and target Descriptionsreferenced by the Engine are revised and implemented. The impact of other types of changes in Source datastructures are summarized in the following table:

Change Impact Required for Capture Required for Engine

New Table NoneALTER table forsupplemental logging

Source DDL*, Re-Parse

New Index None None None unless key change



Change Impact Required for Capture Required for Engine

New Segment NoneNone if capture isspecified at DBD level

Updated DBD Source; IMS Segmentdescription (copybook)

IMS DBDhierarchychange

None if change at bottom ofhierarchy

None if capture isspecified at DBD level

Updated DBD Source; IMS Segmentdescription (copybook)

* See Schema Evolution



Active/Active Replication

Active/Active replication is the process of keeping two or more live databases in-sync with each other and is used forhigh availability, load balancing and performance. The simplest example would be two databases, one maintainedon System A and a second on System B. All changes made to the database on System A must also be applied to thedatabase on System B, while at the same time, all changes made to the database on system B must be replicated toSystem A.

Historically Active/Active replication could only be accomplished by the primary database manager be it IMS, Db2 orOracle using shared memory to manage concurrent updates. The fundamental issue with that approach is the latencyintroduced as the physical distance between the two replicated databases increased. With Connect CDC SQData, anActive/Active replication architecture can be implemented that is unaffected by distance.

The first challenge is to ensure that the replication architecture itself, does not create cyclic update transactions,where the updates being replicated from System A to System B are incorrectly replicated back from System B toSystem A. To address this problem, Connect CDC SQData uses information in the log to identify the source of theupdate. In the case of Db2, the Capture Agent generally uses the Db2 Plan used by the Apply Engine. Alternativelythe Db2 User or correlation-ID under which each transaction was applied to exclude from capture those transactionsapplied by the resident Engine. In the case of the Oracle Log Miner Capture, the User-ID is used and for this reason itis suggested that a dedicated User-ID is assigned to the replication Engine. This User should not used by any program,application, or user making changes to the database that need to be replicated. This includes other Connect CDCSQData components and manual edits to the database.

The second challenge is handling concurrency issues which arise from updates being applied to the same data onmultiple machines and/or delays in replication from one system to another. In our example this would occur if aspecific record was altered on System A and then altered again on System B before the change from System A isreplicated to system B. These issues can be addressed by a variety of techniques, but they generally fall into twoclasses: Avoidance and Detection.

The most common Avoidance strategy involves some form of transaction affinity within the application, routing thetransaction to a specific server based on key range, geographic location, etc. Applications that update subsets of dataon only a single system ensure that data is only replicated outward from the original source. In effect, making thereplicated data, read-only on the other systems.

Connect CDC SQData will detect out of sync conditions that have occurred due to issues outside of its control. Thereplication Engine by default is configured to compensate changes by converting updates to inserts (if the recorddoes not exist in the target), inserts to updates (if the record already exists in the target) and to drop deletes (if therecord does not exist in the target). The Engine can also be configured to do other things in response to these events,including termination of replication, exception processing or only exception reporting.

Connect CDC SQData also can also detect and compensate if the engine is attempting to update newer data witholder data. This can be accomplished by a Smart Apply technique utilizing an Connect CDC SQData managed time-stamp column added to each table. Smart Apply can also be very useful when used to re-sync out-of-sync databasesby re-capturing data from a point-in-time when the databases were known to be in a consistent state.

Finally Connect CDC SQData can also detect unintended activity, such as, an incomplete transaction affinity model orfraudulent transactions which are designed to exploit the time it takes to sync a transaction. Near Real timereplication is the best defense for closing this window, however Connect CDC SQData can monitor for such activity.This is accomplished by comparing the before image of the replicated data to the data in the target. If they match,then the record in the target has not been altered since the transaction was initiated.

Talk to Precisely support before implementing an Active/Active replication architecture to ensure that all factorsimpacting its effectiveness are addressed.



Authorization and Roles

Configuration of Capture Agents often requires the Operating System user to have specific authorizations or roles:

· An account with some administrator rights to access database server and Connect CDC SQData components.

· The Connect CDC SQData components must be executed as a members of a sufficiently elevated role.

Refer to the applicable Data Capture Reference and databases documentation for further discussion regarding therequired authorization and methods for specifying authentication and authorization for the capture agent.



CDC Metadata

The Change Data Capture (CDC) records created by the Capture Agents contain, in addition to before and afterimages of the Source Datastore content, information about the change event that may be unique to the type ofSource Datastore.

This Metadata information can be used by special Apply Engine Functions as part of the transformation logic used toimplement business rules.

More information about the following partial list of CDC specific Engines Functions can be found in the EngineReference.

Function Description

CDCBEFOREReturns the before image of the data (row, segment, record, etc.) that wasupdated. This applies only to update transactions.

CDCOP Returns the change operation (i.e. I=insert, R=replace, D=delete).

CDCRAW Returns the unaltered (raw) CDC record as generated by the capture agent.

CDC_TBNAME Returns the Relational database table name.

CDCSTCKReturns the time of the update as the MVS storeclock (STCK) value in areadable, 16 character representation of the 8 byte hex storeclock on z/OSplatform Capture Agents.

CDCTSTMPReturns the time of the update in a 26 byte timestamp format (YYYYMMDDD-HH:MM:SS.mmmmmm).

CDCTSTMPPKReturns the update time in a 20 byte ‘packed’ timestamp. This is expandedtimestamp (IMSCDCTSTMP) without the punctuation.

CDCUID Returns the Userid of the person making the data update.

DB2TBLNAME Db2 Only. Returns the name of the Db2 Table

IMSDBNAME IMS Only. Returns the name of the IMS database.

IMSSEGNAME IMS Only. Returns the name of the IMS segment.

RECNAMEThe alias (AS <name>) of the DESCRIPTION corresponding to the current CDCrecord.


Selecting a Data Capture Agent

Choosing the correct data capture agent for your installation depends on your environment and data capture latencyproperties (near-real-time vs asynchronous) as shown in the following table.

Please note that even though the capture latency is near-real-time, applying the changed data to target datastores isdependent on network latency, target datastore performance and if required, external schedulers controlling thePublisher and/or Engines. It is highly recommended that continuous stream processing whenever possible tominimize the impact on source system logging and transient data storage.

Data Capture Agent Module Name Near RealTime

Async

IMS Log Reader Capture SQDIMSC Y Y

IMS TM EXIT Capture SQDICDCL Y Y

Db2 Log Reader Capture SQDDB2C Y Y

UDB (Db2/LUW) Log Reader Capture SQDUDBC Y Y

Oracle LogMiner Capture SQDLOGM Y Y

VSAM Log Capture SQDAREQC Y Y

Keyed File Compare Capture SQDDFCDC N Y

Note: The IMS TM Exit Capture agent is a Database Exit and consequently adds NO overhead to IMS Logging. This exitprovides the highest level of performance for demanding applications delivering microsecond performance levels inthe Real World. It is the only viable solution for IMS FastPath databases involved in Active/Active Replication, wherea Log based capture would quadruple the size of the IMS Log.


Transient Storage and Publishing

While the method of capture varies based on the source of the data, only two methods are used to manage transientdata. A fundamental premise of the product architecture is to avoid "landing" the data from the point of capture tothe application of CDC data to target datastores. This is accomplished by using either z/OS system Log Streams, or aproprietary memory mapped storage system. In both cases the Publisher component utilizes TCP/IP to transport datato subscribing Engines:

· CDCzLog

· CDCStore

The Apply Engine is also capable of directly reading any source datastores supported by the Capture agents be it afile, relational or hierarchical database. The engine may be used to unload data or perform some filtering ortransformation of data and then write to all supported target datastores or using a TCP/IP socket, directly to anotherEngine on another platform.

· TCP/IP



CDCStore

Connect CDC SQData CDCStore provides both Storage and Publishing functions managing a virtual transient datastorefor captured data that does not require predetermined sizing. The transient datastore is a proprietary memorymapped storage system that can also be located on any platform, not just the platform on which source data is beingcaptured. In most non-mainframe environments, database servers are dedicated systems where third party productsare not allowed to operate. CDCStore can be located near either the source or target databases and on a commodityplatform where processing and data storage costs may be lower. CDCStore of course supports full restart/recoveryregardless of where the failure occurs, on the source database platform, the Engine platform or in CDCStore itself.

The second feature that differentiates CDCStore from other publishing frameworks is support for multiple targetEngines utilizing the same captured data without requiring its duplication. The ability to manage both transient andlonger term storage provides flexibility in configuration since target Engines need not operate on precisely the sameschedule or at the same performance levels.



CDCzLog

Connect CDC SQData also provides a publish and subscribe framework for cross and mult-platform enterpriseintegration that utilizes IBM's z/OS Log Stream service. The zLog component is similar to the transient datastoreprovided by CDCStore, in that z/OS Log Streams provide a high performance virtual datastore that does not requirepredetermined sizing. The Log Stream must however reside on the z/OS platform and is used only for the IMS, VSAMand Keyed File Compare capture agents. In single LPAR environments with only moderate update volume, a DASDonly configuration may be used. In all other environments and required in a Sysplex, Precisely recommends usingthe System Logger with the Coupling Facility for its greater performance.

The zLog Publisher agent, like the CDCStore Publisher, manages the Log Stream content without requiring itsduplication providing flexibility in configuration since target engines need not operate on the same schedule.



TCP/IP

While a TCP/IP only based framework does not provide a Transient Storage component, the Apply Engine cancommunicate with other Apply Engine instances through TCP/IP across different operating systems, within the sameoperating system instance or both. The source of data can be an existing application or virtually any other type ofdatastore due to the Engine's unique ability to read all types of data:

· IMS Database

· Relational Databases (Db2, Db2/LUW and Oracle)

· Most other databases using ODBC including , SQL Server, MySQL, etc.

· Sequential Files

· VSAM Files

The three elements within a TCP/IP cross platform framework are:

1. Port - This is a TCP/IP communication port that is used as the data transfer conduit within and betweenoperating system platforms.

2. Sender - This is an instance of the Apply Engine configured to send data to a TCP/IP port.

3. Receiver - This is an instance of the Apply Engine configured to receive data from a TCP/IP port.

In a cross platform TCP/IP environment, an Apply Engine can function as a sender of information or as a receiver ofinformation, or both. The following diagram depicts an instance of cross platform operation with two (2) active ApplyEngines. The first Engine reads an IMS database, formats a record which is then written to a TCP/IP port for an ApplyEngine on and another on z/OS which writes data to VSAM files for another application.

Illustration TBD


Index Ind

ex

AActive/Active 13Active/Active replication 13

CCDC Engine Functions 15CDCStore Framework 18Concurrency 13

cyclic Update 13

DData Capture framework 11

SSchema Evolution 11

TTCP/IP Framework 20

connect cdc sqdata - .net framework... · 2021. 3. 19. · connect cdc sqdata data capture guide 7...

Documents