Continuous Integration + Continuous Deployment

CI+CD Solution Reko!1

Content❖ Summary❖ Framework❖ Process

❖ Development❖ Dev Testing❖ Pre-Production❖ Production

❖ Feature Details & Scenarios❖ Cross-Platform Command Line and GUI❖ Jenkins + Reko CICD Docker Images❖ Customization Deployment

❖ By Project/Feature❖ Simple and Smart Installation❖ Selective packing

❖ Configuration Migration❖ Transaction and Rollback❖ By Instance/Warehouse❖ Unicode in Configuration❖ Cross-Database Type - Oracle and MS SQL Server.

❖ Create New Warehouse from Template Warehouse❖ Standard Copy❖ Advanced Copy

❖ Advanced Configuration Settings.❖ Override and Customized Configuration table.❖ Ignore, Drop, Default Values for Configuration Fields❖ Cross-Version Configuration

❖ Jenkins Integration❖ Automation Testing - Future Consideration

!2

Summary

!3

• Objective• Avoid human error and improve deployment efficiency by automating Dev,

Pre-Production, and Production build and deployment.• Create safe and stable environment by avoiding manual changes (code or

configuration) on Testing, Pre-Production, and Production server. • Improve system maintainability and support efficiency by keeping identical

Pre-Production and Production environment.

• Proposal• Leverage out of box toolsets like Jenkins, Git, etc. • Develop jobs, scripts, and tools to automate CI/CD process.• All tools can be delivered as docker image with all environment settings.

CI/CD Framework

!4

Source Code ManagementIntegrated JIRA and Git to manage code by release/version, feature/story, or support ticket.

Production & Operation Automated deployment to Production Server.

Configuration Synchronization with Pre-Production.

Continuous BuildAutomated build and deploy on Dev.WMS Configuration on Dev.Performing code review and unit testing.

Continuous TestingAutomated build and deployment to Pre-Production

Configuration Synchronization with Dev Server.Performing Pre-Production and reproduce operation error.

Version Control

Continuous Deployment Continuous Integration

Continuous Delivery

Reko CICD Infrastructure/Toolset

!5

+Source code control by Integrated Git and JIRA by

feature, story, ticket, and task.

Development

Pre-Production ServerProduction Server

JenkinsJenkins projects and jobs. Reko tools to pull and create rollouts.Reko tools to deploy/synchronize configurations between servers.

Pull and create rollout from git by JIRA numbers

Deploy by task/feature

Deploy rollouts and configuration AFTER unit testing and review is passed

Deploy rollouts and configuration AFTER Pre-Production is passed

Configuration Synchronization

No code or configuration

change is allowed

UAT ServerNot validated, deployed by minor release or feature. No on-going development, for testing purpose only.

Deploy by feature/release

Dev ServerNot stable, with on-going features developments.

1. JIRA is not required. Customer can use any tool to manage delivery, even an excel spreadsheet with unique ID for each feature is okay. As long as the ID is used for git commit message.

2. Git has many choices, can be Gitlab, Github, other git public or private cloud service.

3. Jenkins is open source software, and free for project use.

4. Reko tools and scripts will be part of deliverable with condition of no distribution to 3rd parties.

1. Features/tasks from JIRA.1.1. Code change locally.1.2. Use Reko Tool to capture configuration change1.3. Unit testing locally

2. Commit changes/configurations to Git3. Login Jenkins and execute Dev deployment job, by providing Story, JIRA#, and Release Notes number

3.1. Jenkins invokes Reko tool 3.2. Reko Tool pulls changes/configurations and deploy to Dev server, and create rollouts3.3. Jenkins distributes rollout to Dev server, and execute the rollout installation

CI/CD for Development

!6

JIRA Git Dev Server

Development TeamJenkins Server + Reko Rollout Tools

! "

#

3.1

3.2 3.3

Cross-Platform + GUI

!7

Cross-Platform Command LineReko CICD is a cross-platform executable binary which mainly takes command line arguments and setting files

Cross-Platform GUIA Electron GUI application is available for Linux, Mac, and Windows for people who are not used to use terminal.

Out-of Box Docker Image

!8

User only needs to copy from the build-in Jenkins project and change related settings for Jenkins and Reko CICD for example, database

connection, SSH credential, project folder, etc.

A Docker Image is available with Jenkins + Reko CICD, plus all necessary drivers and environment setting ready.

Customization Rollout By Feature(s)

!9

Customization can be deployed by feature(s). Just specify feature ID(s), and they will be packed together as one rollout.

If no ID specified the whole project will be packed as one rollout.

A configuration file is supported to ignore files, like documents, to be packed.

Rollout will be in .tar package so no any tools need to be installed on target instance.

A smart rollout installation script is included to copy artifacts, and handle any necessary database, safe-to-load data, interface, report, pagebuilder, MOCA command, RF, or Java changes.

Configuration Migration

!10

Support Oracle to Oracle, Oracle to SQL Server, SQL Server to Oracle, SQL Server to Oracle.

By instance configuration copy, or by warehouse.

Unicode is supported to handle Multi-Languages in configuration.

The execution time for configuration copy takes about 5 minutes depends on instance and hardware. The entire configuration is executed in one nested transaction, and will be rollback if exception happens.

By default, SQL Server is not case-sensitive. So copy from Oracle to SQL Server might cause duplicate PK error, which will not stop the process rather being saved to a csv data file for investigation.

CICD will rollback all configuration changes on target instance if exceptions happens.

Allow copying instances configuration for different version (See advanced setting in later pages).

Create Warehouse By Template - I

!11

Reko CICD can also to use any warehouse as template to create a new warehouse in NEW instance.

Source Warehouse ID - The warehouse you want to use as template.

Copy To New Warehouse ID - The new Warehouse ID you want to use. It can be same ID if you want. The new warehouse ID will be used to replace all settings include descriptions.

New ID Offset - The ID offset to generate new ID, for example, Pick Method 16 in template will become 10016 in new warehouse. All related configurations will be changed to refer to the new ID in new warehouse.

Ignore Sequence Reseed - By default, CICD template copy will reseed sequences in target instance to avoid error when WMS tries to generate new ID. By ignoring the reseed you can keep the sequence of your target instance. This should be only used when you’re sure all your sequence has been handled properly.

Create Warehouse By Template - II

!12

post_deploy - CICD has build-in post-deployment logic to do validation and clear up data on target instance. You can add additional post deployment scripts on global or project level.

copy_wh_info - Customized Copy Configration as additional

Customized ID - Let CICD to handle customized ID. For example, if you defined a new id named my_id, and values 101, 102, 103… as auto-grow sequential number for your customization. And you will use it in policy as polcod=MYPOL, rtstr1=my_id, and rtstr2=101… The your configuration can be

• new_id - The name of your id, in this example, MY_ID.

• row_id_col - The column to refer your id name. In this example, rtstr1

• row_id_val - The column to revere your ID values. In this example, rtstr2

• new_id_advanced - Advanced ID handling. You can specify special logic to handle specific configuration. Regular Expression is supported

Additional TablesAll standard configuration tables is built-in the program. User can use add_tables setting to

include more tables. Together with exclude_tables, user can override the standard table settings.

Exclusive TablesSpecify which configuration table not to be copied. ‘all’

will exclude all standard configuration tables.

Process BatchFor tables with large data, the tool process at 10% batch and less than 1000 rows. This can be overridden at table level for trouble shooting when exception happens to locate what configuration causes problem.

Ignore ColumnsIgnore_columns can be used to ignore columns during instance configuration comparison. For example, we don’t want to compare location status between instance, so locusts can be ignored. But the ignore columns will still be considered for new data.

Drop Columnsdrop_columns will be dropped from source and destination dataset. The difference with ignore_columns is drop_column won’t be there for new data either. So it must be nullable, or have DB level default value defined.This setting also can help to copy configurations between different version when the table table columns are different

Default Valuesdefault_value tells system what value should be used for new data. Be aware, the default values will ONLY be used for new data. It can be used together with ignore_columns or any column that you don’t want to copy from source instance.

Advanced Configuration Migration

!13

ConditionCondition can be used to specify configurations being un-touched by CICD. For example, if you put POLCOD != ‘XXX’ for poldat, then policy code XXX will not be changed

Important Notice

❖ By default, system always check configuration versions, and will stop if source and target instance versions are different.

❖ Warning - Needs to be very careful to copy configurations across different versions. The suggested approach is,

❖ Upgrade source instance to the same version by standard JDA upgrade from installation package

❖ Regression testing on upgraded instance.

❖ Use Reko CICD tool to copy configuration

❖ Most version differences can be handled by ignore/drop columns but huge version difference like 2007.x to 2017.x, or vise versa, cannot be handled since new concept of zone is huge change that impact too many configurations in a structural way.

Ignore Version Check -iv arguments can skip the version check.

Drop ColumnsOverride configuration table and use drop_columns to remove the new columns from new version, or obsoleted columns from old version. In this example new column shp_ctnrte_flg is dropped.

Version InfoVersion difference is ignored.

Cross-Version Copy

!14

ConditionCondition needs to specified to avoid new configuration from newer release being removed. For example, if a new policy POLCOD=‘NEWPOL’ is added in new release, then the condition should be specified, otherwise the new policy will be removed by CICD since the old version doesn’t have this policy

Integrate with Jenkins

!15

Test Automation - Future Consideration

!16

High cost for starterCost and time consuming is high at the beginning to build automation framework and test scripts.

Low cost for future CICD and go-liveTest cases can be reused and save time and cost by replacing human testing effort.

High quality and stabilityHigher testing efficiency and coverage rate contribute to better quality.

Automated CI/CDAutomation testing can be configured as precondition of release and deployment instead tracking testing result manually.

High VolumeSimulate high volume operation which is difficult for human testing.

Q&A, Thank You

!17

Contact US：

Address：#7001, Zhongchun Road, Suite D-601, Shanghai

Tel：+86-021-5113-25455113-2546

Mobile：137-642-89079eMail：roger.guo@rekosys.com

Contact: Roger Guowww.rekosys.com