idiom transaction engine -...

+64 9 6308950 | @ [email protected] | idiomsoftware.com

2-2, 93 Dominion Road, Mount Eden, Auckland 1024

IDIOM Transaction Engine

An Application to Manage Complex

Business Entities

Whitepaper March 2018

mailto:[email protected]

http://www.idiomsoftware.com/


Whitepaper

© Idiom Ltd 2018 Version 21


Background ................................................................................................................................................. 4

History ........................................................................................................................................................ 4

IDIOM Published Articles ....................................................................................................................... 5

Decisioning: a new approach to systems development ........................................................... 5

Requirements and the Beast of Complexity ................................................................................. 5

Decisioning – the Next Generation of Business Rules .................................................................. 6

The Role of SQL in Decision Centric Processing ............................................................................ 7

Taming the IT Beast with Decision Centric Processes .................................................................. 8

Where It Fits .............................................................................................................................................. 9

Form, Fit, and Function ........................................................................................................................ 10

IDIOM Transaction Engine ...................................................................................................................... 11

Purpose ................................................................................................................................................... 11

What is the Transaction Engine .......................................................................................................... 12

Transaction Engine Terminology ........................................................................................................ 14

Platform Technology ............................................................................................................................ 14

Component Overview ........................................................................................................................ 15

Built to be Secure and Scalable .................................................................................................... 16

Components ...................................................................................................................................... 17

Interpretation ..................................................................................................................................... 18

Services ................................................................................................................................................... 19

The Entity Builder Platform ................................................................................................................... 20

Generic Database ................................................................................................................................... 20

The Transaction Entity ...................................................................................................................... 21

Archetype Meta-Data ..................................................................................................................... 22

Permissioning and Workflow ........................................................................................................... 24

Batch Jobs ......................................................................................................................................... 25

Activity Log ........................................................................................................................................ 25

Putting it all Together ............................................................................................................................... 25

Data ........................................................................................................................................................ 25


Whitepaper

© Idiom Ltd 2018 Version 21

Schemas ............................................................................................................................................. 26

ITE Meta Data .................................................................................................................................... 27

Processes ................................................................................................................................................ 28

Workflow ............................................................................................................................................. 29

Task Management ........................................................................................................................... 29

Getting Started ..................................................................................................................................... 30

The Generic Transaction ..................................................................................................................... 31

Versioning and Release ...................................................................................................................... 31

Transaction versions ......................................................................................................................... 33

IDIOM Tools ................................................................................................................................................ 33

IDIOM Decision Manager ................................................................................................................... 34

IDIOM Forms ........................................................................................................................................... 37

IDIOM Dialog Manager ....................................................................................................................... 38

IDIOM Decision Tracker ....................................................................................................................... 38

IDIOM Mapper ...................................................................................................................................... 38

IDIOM Decision Manager Workbench ............................................................................................. 38

IDIOM Document Generator ............................................................................................................. 40

Conclusion ................................................................................................................................................. 41

Mark Norton | CEO and Founder | Idiom Limited

+64 9 630 8950 | +64 21 434 669 | Australia free call 1800 049 004

2-2, 93 Dominion Road, Mount Eden, Auckland 1024 | PO Box 60101, Titirangi, Auckland 0642, New Zealand

@ [email protected] | idiomsoftware.com | Skype Mark.Norton




Whitepaper

© Idiom Ltd 2018


BACKGROUND

This Whitepaper introduces the IDIOM Transaction Engine. The IDIOM Transaction Engine is a

recent IDIOM product that collates many new and existing IDIOM components into a

common enterprise scale platform that can be used to quickly and cost effectively build

transactional applications for a wide variety of complex business entities.

History

Over the past 17years IDIOM has conceived, evolved, and demonstrated the effectiveness

of its ‘decision centric’ development approach, which leverages both decisioning and agile

approaches to dramatically simplify and strengthen commercial systems development. This

whitepaper describes the IDIOM Transaction Engine and how it supports the decision centric

approach.

Development of systems using the approach, and the IDIOM tools that support it, has been

repeatedly shown to:

✓ simplify systems and their related development practices to reduce development time,

cost, and risk, by substantial amounts

✓ increase business agility

✓ improve the alignment of business policy and computer systems

The evolution of the approach has been chronicled in a series of published articles that

remain available as published.

For convenience, we have included a précis of the more important articles in the following

section. These précis’ provide insight into the evolution of the methodology and its rationale.

Links to the original articles are also provided for the interested reader.

An outline of the IDIOM Transaction Engine is then offered; this is followed by a description of

each of the IDIOM tools.

The IDIOM tools are derived from, and closely aligned with, the concepts described in the

articles, and together they provide a cost effective and risk averse path to successfully

building decision centric systems.


Whitepaper

© Idiom Ltd 2018

IDIOM Published Articles

Decisioning: a new approach to systems development1

This seminal article (published in the Business Rules Journal, Jan 2007) provided the

background and conceptual building blocks for decisioning and decision centric

development. It highlighted the importance and benefits of decision analysis to good

systems design, and at the same time, noted its absence from the then current development

methodologies. It drew important distinctions between decisions and ‘business rules’, and

noted the relatively greater importance of decisions over processes as analysis subjects.

It laid the conceptual foundation for decisions to be a fundamental unit of requirements

specification for both business and systems.

Decision: A single definitive datum that is derived by applying business knowledge to relevant data for the purpose of positively supporting or directing the activity of the business.

According to the article, these decisions are defined in the context of a decision model,

which is defined as:

Decision Model: An ordered assembly of decisions that creates new and proprietary information to further the mission of the business.

And decisioning itself was defined as:

Decisioning: The systematic discovery, definition, deployment, and execution of automated decision making.

Requirements and the Beast of Complexity2

[The ‘Requirements Article’]

This popular article (published in Modern Analyst, 2010 ~34,000 downloads) updated the

above article, and targeted requirements specifications as a significant weak link in the

traditional systems development cycle. It presented a case for a new approach to

requirements gathering and specification that is based on analysis of business policy to

identify core business entities and their respective state changes, which are then captured

and described as decision models and ultimately implemented as transactions. From the

paper’s conclusion:

1 http://www.brcommunity.com/b326a.php Decisioning: A new approach to Systems Development

2 http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/1354/Requirements-and-the-Beast-of-

Complexity.aspx


Whitepaper

© Idiom Ltd 2018

“This ‘decision-centric development approach’ . . . focuses on a ‘missing link’ between business strategy and operational systems – the Decision Model. The Decision Model is a new and important requirements artefact that is accessible to and understood by both business and development practitioners, giving rise to a previously unobtainable level of shared understanding that can help bridge the gap between business strategies and the systems that support them. The Decision Model gives the business 'hands-on' control over the definition and implementation of its most critical IP – its decision making know-how.”

Decisioning – the Next Generation of Business Rules3

[The ‘Decisioning Article’]

This article (published in both Modern Analyst and Business Rules Journal, 2013) noted that

Charles Forgy, the inventor of the rete algorithm (which powered a generation of business

rules engines), listed criteria that more or less excludes ‘transactions’ as an appropriate use-

case for the algorithm. Notwithstanding, rete aligned, constraints based rules engines

continue to be used in transactional systems, adding both cost and complexity for little

value.

This article then expanded the scope of IDIOM’s business rules/decisioning concepts to

include the ability to dynamically create data that is aligned with the ‘idiom’4 of the business;

that is, with the nouns and noun clauses of the proprietary language (the idiom) that we

assert is always present when describing business rules. By definition, the idiom is always

proprietary to the author of the business rules because it is the need to define the proprietary

business rules that gives rise to the idiom.

Dynamic transformation from raw ‘real-world’ data to data that is compliant with the

business idiom is needed because a) it is rare that the real-world data is already in the

required state for adjudication by the rules, and b) the business idiom is fluid and ever

changing – it is the essence of business rules, changing whenever the business changes,

which contrasts with the relatively static nature of the real-world data definitions).

The article also highlighted the separation of the doing and decisioning components within a

system:

“Service oriented architectures are inexorably trending towards separation of ‘doing’ components from ‘deciding’ components. The ‘doing’ components can be commoditized and outsourced. The truly proprietary code, the code that captures business knowledge and IP, is being increasingly concentrated in the ‘deciding’ components.”

3 http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/2713/Decisioning-the-next-generation-of-

Business-Rules.aspx 4 http://www.thefreedictionary.com/idiom [a. ‘A specialized vocabulary used by a group of people’]


Whitepaper

© Idiom Ltd 2018

The Role of SQL in Decision Centric Processing5

[The ‘SQL Article’]

The logical disconnect described in the above paper between the original design intent of

the technology (rete in that case) and its subsequent use, is echoed with SQL. This article

(published in Modern Analyst (2014) and the Business Rules Journal (2015)) noted that SQL

was designed from the outset to address a specific class of problem – set processing.

This class of problem does not fit easily with transactions, and is to a large extent redundant if

the data is hierarchically structured around core business entities (and is stored as such). In

essence, the article made the observation that the ubiquitous SQL ‘join’ is fundamentally

flawed when used in a transactional context: When dealing with the transaction’s subject

entity, dependent tables should be joined onto their parent entity; the parent entity should

not be joined on to the dependent tables. Again, and in apparent contrast with its original

purpose, complex ‘join’ based SQL is routinely used to underpin modern day transactions.

And to close the circle on the business idiom described in the Decisioning Article, we

observed that the labels that are used to describe data in the business idiom must describe

that data in context; that is, the labels used for the data values must include all relevant

context qualifiers. If we are going to be able to implement business rules that are described

using the idiom, then it is necessary that we also describe the data using the idiom; the

definition of the data ‘in context’ must be described using the language of the business

idiom and vice versa. This is something not easily achieved using SQL.

This correlation between the business idiom and data context is the essence of the linkage

between business policies and their implementation in computer systems. It is also the

backbone of business transactions, and in our experience, usually represents the major part

of a transaction’s internal work effort.

As a result, the SQL paper further extended the scope of IDIOM’s business rules/decisioning

concepts by highlighting data context as an integral part of a transaction’s ‘requirements

specification’.

With a decision authoring tool that understands context, we can quickly and completely

define decision models that address the full scope of policy driven decision-making,

including validation, transformation, calculation, adjudication and workflow; and which

requires only ‘simple SQL’ (i.e. no joins) to provide correct, consistent, and complete business

transactions of any size and complexity.

And these transactions will often execute much faster than solutions developed using more

traditional approaches to the use of SQL. This performance gain is primarily a reflection of the

way that SQL is used to support the transactions.

5 See, ‘The Role of SQL in Decision Centric Processing'

http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/3109/The-Role-of-SQL-in-Decision-Centric-

Processes.aspx’


Whitepaper

© Idiom Ltd 2018

Taming the IT Beast with Decision Centric Processes6

[The ‘Process Article’]

This article (published in Modern Analyst in 2015) builds on the earlier articles to address the

process perspective of the approach. It describes a process architecture and a matching

development approach for decision centric systems. It uses ‘business transactions’ as the

core application building blocks. A business transaction is defined as:

Business Transaction: The activity that is initiated by an event that changes the state of a business entity, and which when complete leaves all systems compliant with all business policies that are relevant for that entity in that state.

A ‘business transaction’ as described by the article includes all possible state changes that

collectively define the complete life cycle of the subject ‘business entity’. This single, omnibus

transaction is called and executed for each and every event that affects the entity – this

collapses the number of business processes that need to be considered.

Process complexity is reduced to a finite and known set of business policies in the decision

models that comprise the business transaction; and a finite and known set of generic and

pre-built activities (usually implemented as services) in the underlying application. Process

orchestration is achieved by ‘just-in-time’ policy-derived decision making within the

transaction to determine the ‘best-next’ activity or process. The decision-making that follows

each activity only needs to calculate the ‘best-next’ step to ensure a complete and

consistent overall process; the only time best-next can be assessed is ‘right now’, using the

current context and event data to achieve immediate, real-time, process orchestration. We

call this approach to process management ‘binary BPM’ to reflect the two steps in the

approach (this step, next step).

The benefits are substantial. The architecture and its associated development approach

dramatically reduce the number of moving parts in the system, leading to an equally

dramatic reduction in the quantity of bespoke coding required. Throughout the article there

are observations regarding the simplification of systems development and execution that are

attributable to the approach.

When the approach is used to address complex business transaction processing, the

demonstrated results include substantial improvements in the key development metrics of

time, cost, and risk. These already encouraging results are accompanied by improvements in

business flexibility and agility, better alignment between business strategy and operational

systems, and systems that are more transparent and more durable.

In the approach described by the article, business ‘Subject Matter Experts’ [SMEs] assume

hands-on responsibility for the definition, capture, and deployment of the business policies

that manage the entity life-cycles within the business transactions. By using the IDIOM tools to

6 See, ‘Taming the IT Beast with Decision Centric Processes’

http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/3291/Taming-the-IT-Beast-with-Decision-Centric-

Processes.aspx


Whitepaper

© Idiom Ltd 2018

implement the approach, these SMEs are empowered to take full and complete ownership

of the implementation of business policy within the operational business systems.

Where It Fits

It has become popular to refer to some systems as ‘systems of engagement’ and others as

‘systems of record’. Unfortunately, these two classes of systems are sometimes the only

classes of systems considered, and so they become receptacles for corporate knowledge in

the form of rules and similar knowledge artefacts simply for want of any other place to put

them. In which case, how is this knowledge managed and shared?

Figure 1 – Classes of Systems

IDIOM’s view is that there are other equally ranked classes of systems as per the adjacent

diagram [Figure 1]. Most importantly, all classes of systems should defer to the ‘system of

applied knowledge, innovation, and learning’ that sits centre-stage. This system becomes the

‘Core’ of the new IDIOM Transaction Engine.


Whitepaper

© Idiom Ltd 2018

This architectural pattern provides a formal place for corporate knowledge that is shareable

through an ‘API first’ approach. ‘API first’ is a concept that is expanding in popularity, and

which is neatly summed up by a guest commentator on the API Evangelist blog7 as follows:

“an API-first strategy is one where the API serves as both a pattern of an enterprises IT architecture as well as

catalyst for engagement. In a nutshell, an API-first strategy is a systems paradigm shift

- from a centralized, engineered mechanical system made up of subordinate parts serving as means to

deliberately designed ends

- to a decentralized, organic and evolving ecosystem where the individual actors can be spontaneously

enfranchised or co-opted by any other actor / set of actors to serve new purposes.

This allows the system to be more flexible, respond more quickly and behave more dynamically to suit

unforeseen purposes.”

API’s (especially in the form of WebAPIs) are a core feature of the Transaction Engine design

pattern, and are used to link the various classes of systems as shown in the above diagram.

Form, Fit, and Function

What we have described above can also be described by borrowing a phrase from

manufacturing viz. ‘form, fit, and function’.

In our version of ‘form, fit, and function’, form can be likened to the factory floor – the set of

activities and services that provide the core capabilities of the system (but not how it works).

Function is what is achieved with the form – that is, the application and use of the form to

achieve the ultimate purpose of the system.

And the fit describes how the activities in the form are orchestrated to achieve the function.

Form is what it is; function is what it does; and fit is how the form is orchestrated to achieve

the function.

The reframing of computer systems into form, fit, and function is the key to the improvements

that have been claimed above. The traditional systems approach does not separate form,

fit, and function; quite the opposite, the traditional computer process tightly binds the

function to the activities (the form) that support it, so that the activities are not universally

reusable, and their orchestration is not agile.

For this reason, processes are often confused with the function of the system – that is,

orthodoxy asserts that a process Is the function and includes the immediate activities that

support it. In fact, a process is an ephemeral consequence of the fit! This simple re-alignment

allows us to cleanly separate the purpose of the system (the function) from the underlying

activities that support it (the form). Process then reappears only fleetingly as the dynamic

7 http://apievangelist.com/

http://apievangelist.com/

http://apievangelist.com/


Whitepaper

© Idiom Ltd 2018

orchestration of activities to achieve the purpose, and is more correctly represented by a

decision model. The diminution of process is a key factor in reducing system complexity, and

improving flexibility and agility – a simple change to a decision can result in a new process,

quickly and safely.

Figure 2 – Business Logic is Content in an Optimally Architected System

IDIOM TRANSACTION ENGINE

Purpose

The essential purpose of the Transaction Engine is to manage the full life cycle of complex

business entities, exclusively and totally. The range of applicable entities is extensive, and

includes at least the following.

Information based products and services, for example loans, insurance policies, claims,

wealth accounts, pension accounts, etc.

People and their interactions with the organisation, for example, customers, passengers,

patients, beneficiaries, employees (including payroll).

Tangible entities; for example, property or physical assets that have conditional

maintenance and replacement cycles.

Derived concepts; for example, resource and capacity demands.

By ‘full life cycle’, we mean all events that affect the state of the entity, from acquisition

through disposition. The emphasis on ‘all’ is important – when all events are interpreted and

managed by the IDIOM decision models that sit within the Transaction Engine’s Core, then


Whitepaper

© Idiom Ltd 2018

the Transaction Engine can achieve its purpose exclusively, so that no other system or system

component needs to have any knowledge of the internal structure or state of the entity. This

exclusivity of knowledge and control that sits inside the Core allows the Transaction Engine to

be truly generic, to be configured by the business on demand as and when business

knowledge improves with regard to the management of the fundamental business entities

that define it. This provides the heart of a learning organisation.

This generic, business configured application is also both agile and business aligned, by

definition, which satisfies the primary objectives of the Transaction Engine concept.

And because the generic application components (i.e. ‘form’, prior to configuration) are

vendor supplied, we can reap the time, cost, and risk benefits of mass production; the

benefits of mass production and mass customisation together in a single ‘transaction

engine’.

What is the Transaction Engine

Figure 3 – IDIOM decision models provide the Fit between the Form (Transaction Engine, Legacy

Systems, External Systems), and the over-arching Business Purpose (the Function)


Whitepaper

© Idiom Ltd 2018

The IDIOM Transaction Engine is a set of libraries that together form the basis of a generic

application that can manage the full life-cycle of any complex business entity. The

functionality of the Transaction Engine can be described using the Form, Fit, Function

metaphor.

In the above diagram (Figure 3), we can see that the IDIOM tools (IDIOM Decision Manager

[I], IDIOM Forms [F], and IDIOM Document Generator [D]) are used to prescribe the business

functions of the system, predominantly using decision models. All of these tools interact with

the XML schema(s) that describe the entity context to provide a complete definition of the

business Function as it is seen by the business user community, at least as it relates to specific

entities.

The universe of existing systems (internal and external) are coupled together using the

Transaction Engine to provide the full set of activities that are available to support the

function, being the Form.

Finally, purpose specific IDIOM decision models operating at the Core of the Transaction

Engine provide the Fit, by orchestrating the executable form components as required to

support and achieve the function. We referred earlier to this just-in-time process orchestration

as Binary BPM, which provides a digital channel to support the function.

The Transaction Engine is not a finished application per se – it always requires configuration

with the IDIOM tools, and potentially some bespoke development in customer specific

extension libraries, to become functional in a specific client context. This is both deliberate

and beneficial. Historically, vendor applications have struggled to adapt to the many, many

variations in client business and technology requirements that are found in the real world. The

consequences of this struggle range from cumbersome, complex, and expensive ‘omnibus

systems’ (e.g. SAP); to systems that require significant customisation; to a need for multiple

systems; and/or the client simply accepts the constraints imposed by existing vendor

application(s). Or, the client goes it alone and builds entirely bespoke. Each of these options

incurs costs that can be much greater than the Transaction Engine approach to the

provisioning of business applications.

The Transaction Engine addresses the problem of variable client needs in two ways:

✓ By using the IDIOM tools to declaratively address the client’s unique business context

(‘what makes you different’) by configuration.

✓ And by completing the ‘last mile’ of application development only when the client’s

unique business context and the infrastructure context are both known; this

development is often limited to some integration.

In general, completing the ‘last mile’ of application development can be achieved in

parallel with the configuration of the business requirements using the IDIOM tools, so that any

plausible variation in the client’s technology landscape can be addressed without significant

time, cost, and risk at the same time as the client’s business context is being captured using

the IDIOM tools i.e. the entire project is completed in the time it takes to capture the

‘requirements’.


Whitepaper

© Idiom Ltd 2018

By the time the business requirements as described by the IDIOM tools are ready to be

deployed, the Transaction Engine is fully customised to the client’s specific needs. It is IDIOM’s

standard practice to limit this parallel development cycle to 100 days. Significant systems

can be developed within this timebox, which sits in the knee of an exponential curve as time,

cost, and risk start to climb exponentially with project size8. Limiting project size to a 100-day

time-box is a practical and effective way to ensure optimal development metrics and early

ROI.

Transaction Engine Terminology

We have chosen the term ‘transaction engine’ with some care.

Engine: “An agent, instrument, or means of accomplishment”

from the Latin ingenium innate quality, talent, ingenious contrivance 9

An engine in this context is a distinct software component that provides a ‘means of

accomplishment’. However, it may not be complete and ready for use in its delivered state:

It may need natively coded customisation and/or extension; and it will need the insertion of

various IDIOM tool generated artefacts that will define its ultimate business purpose.

The term transaction is used in a relatively orthodox manner to describe one or more

activities that collectively change the state of an entity (see Business Transaction definition in

the Process Article described above10). A state change is either made, or the entity is left

unchanged – there is no such thing as a partial transaction (by definition). The sum of all

state changes defines the life-cycle of the entity, completely and definitively.

Activity as described above refers to a secondary event that causes something to happen,

the details of which are unknown to the Transaction Engine Core. Each activity can be

thought of as a ‘black box’ from the perspective of the Transaction Engine; all required

‘black boxes’ can be orchestrated (on a ‘best next’ basis) by the Transaction Engine without

the Transaction Engine understanding how those ‘black boxes’ perform their tasks.

Finally, for the sake of clarity, any reference to Subject Matter Experts [SME] in this document

refers exclusively to the business experts who are responsible for the configurations that are

captured by the IDIOM tools; that is, they are the users of the IDIOM tools.

Platform Technology

The current reference version of the Transaction Engine is built on to Microsoft’s Azure

platform as a service as its reference platform. The choice of Azure as the initial target is a

matter of convenience, and is not intended to restrict IDIOM from porting the Transaction

8 See Programming Productivity, Capers Jones, Mcgraw-Hill, 1986. ISBN 978-0-07-032811-2 for an expansion on this

topic or visit https://en.wikipedia.org/wiki/Capers_Jones. 9 http://www.thefreedictionary.com/engine 10 Taming the IT Beast with Decision Centric Processes

https://en.wikipedia.org/wiki/Mcgraw-Hill

https://en.wikipedia.org/wiki/International_Standard_Book_Number

https://en.wikipedia.org/wiki/Special:BookSources/978-0-07-032811-2


Whitepaper

© Idiom Ltd 2018

Engine to other ‘PaaS’ offerings (for instance, Amazon or Google). IDIOM also offers

variations of the Transaction Engine, including: a virtual machine [VM] based solution that

runs on a traditional Microsoft technology stack, and which can be deployed in an

‘Infrastructure as a Service’[IaaS] cloud or on-premise; and a hybrid option for deployment of

the Core and its database in a VM, while the public facing components reside in the Azure

PaaS. In general, exclusively in-house (i.e. behind the firewall) applications tend towards the

VM options, while public facing applications tend towards PaaS, because of the greater

security and scalability offered by the PaaS.

Accordingly, the IDIOM Transaction Engine is currently Microsoft centric, primarily C# dotNET

Core, with Angular JS for the engine’s web user interfaces, and SQL Server database with

Azure Blob and File support. Authentication and authorisation use Identity Server, Active

Directory and/or Auth(0).

A secure token server [STS] is used to authorise every component that is used in every step of

every transaction to execute on behalf of the authenticated and authorised user.

Component Overview

The next diagram [Figure 4] provides an expanded view of the Transaction Engine, showing

the Core in its role as the dominant ‘system of knowledge, innovation, and learning’ inside

the larger application framework. Each icon in the diagram is a transaction engine

component making up the ‘form’.

The primary role of the Core is to execute the decision models that implement the

‘knowledge, innovation, and learning’ that makes the business transaction responsive to

business policies, and which orchestrates all workflow on a best, next basis.

The green icons highlight the IDIOM tool generated declarative components (schemas,

decision models, forms, documents) that inject the functional elements into the Transaction

Engine, and which differentiate the IDIOM Transaction Engine concept from a standard

application.

The XML that describes the subject entity is formally called the Entity XML, but the term ‘God

document’ better reflects its relative importance to the Transaction Engine concept. It is the

basis for what would often be a complete application in a traditional system; it usually

contains all of the data that describes all states of the subject entity, past, present, and

future – for instance, an insured and all of their insurance business; an entitlement beneficiary

and all of their entitlements; a superannuation fund member and all of their accounts and

entitlements, etc. In essence, it is a database for a single entity.

The God document is only ever created or changed by and within the Core under the

control of IDIOM decision models. For additional security, this document can be encrypted

and decrypted with a key that is only known to the Core.

When supported by the IDIOM Forms Engine and the IDIOM Document Generator, this allows

end-to-end entity life-cycle management to be built declaratively using the IDIOM tools.


Whitepaper

© Idiom Ltd 2018

The business functions defined and implemented declaratively using the IDIOM tools build on

to the balance of the Transaction Engine infrastructure as shown in Figure 4 (next page), and

when deployed, can collectively provide an end-to-end enterprise scale system of

substantial capability.

Figure 4 - The Transaction Engine, with the Core as the dominant ‘system of knowledge, innovation, and

learning’

Built to be Secure and Scalable

Each component can scale independently to service demand on virtually any scale (the

first constraint reached would be the single copy of the database).

Each component level transaction is thread-safe and stores its state in either the Blob

Store, the File Store, or the Database when complete, providing a resilient and scalable

architecture.


Whitepaper

© Idiom Ltd 2018

Logical transactions (online user transactions or externally managed batch processes

that change the state of the EntityXML) have a separate existence that is defined and

managed across component transactions using the ITE Meta Data.

Each component must separately acquire an ‘authority to operate’ token from the

Secure Token Server (STS) before it can perform operations on behalf of the user session

token (e.g. SAML).

Active Directory is typically used to authenticate domain users; Auth(0) is used to

authenticate Public Portal users (other home realms are possible).

The service calls between the major components only pass a small amount of data as a

single XML message (which is described by the ITE Meta Data schema, to be described

shortly).

Larger blocks of data are passed via the Blob Store, and may be encrypted by/for the

participating components.

The Database is Azure SQL if full PaaS, or SQL Server if the VM or hybrid. It can be

encrypted at rest, with the critical EntityXML data separately encrypted (if so configured)

at the column level.

The Forms Engine, which provides the user with access to the EntityXML indirectly via its

own proxy document, is isolated from the Core.

The user accessible Portals that comprise the balance of the user accessible application

can interact with all system components, but do not directly access the EntityXML.

The Admin Portal is permitted to access and update the database tables directly for the

various administration tasks that are not related to the EntityXML.

The net effect of the Portal and Forms Engine pairing is that any external web application

has either access to the Transaction Engine’s internal components, or to the EntityXML,

but not both. This is an intentional security consideration.

Other bespoke entry point applications can be added as required – for instance,

operational dashboards, mobile application servers, etc.

Components

The components in the above diagram include the following.

Admin Portal is a traditional menu based application for owners and administrators to

manage the system. Built using dotNet Core and Angular.

Public Portal is a separate web application redacted from the Admin Portal that has very

limited functionality, essentially limited to opening a form to provide access to the user’s

own business. Note that the reduction process from Admin to Public Portal actually

removed the relevant code; it is not just ‘switched off’.

Dashboard is a separate web application (Angular) that usually uses PowerBI to provide

dashboard style operational management information. The data needs of the

dashboard are provided by the Core through decision model configuration (see

ReportingXML).

IDIOM Forms Engine is enclosed in a separate container that provides full integration with

the Transaction Engine’s security and workflow.

Blob Store is Azure’s fast, low cost storage, or the local file system if in a VM.


Whitepaper

© Idiom Ltd 2018

SQL Server is either in a VM as the standard product, or Azure SQL if PaaS. EntityXML is the

decrypted God document in object form.

Core is a webapp that exclusively controls the EntityXML, and uses decision models to

orchestrate other components through process meta data.

Batch Job is a separate container that integrates standard and bespoke batch jobs with

the Core functionality. May be a WebJob if PaaS or a Windows Service if VM. Standard

jobs include BringUp processing and file pass.

Service Adapter is a separate application to accept and authorise all inbound service

requests. No external service has inbound access to the Core.

Service Bus is the Azure queue service (RabbitMQ if in a VM) that is used to queue batch

job requests, and Document Generator and Email requests.

Communications Handler is an application that orchestrates the response to each

‘communication request’, including document generation, email generation, and

inclusion of uploaded documents.

IDIOM Document Generator is a standalone application that has been integrated into

the Transaction Engine’s workflow. It is installed inside a WebApp.

Interpretation

With reference to Figure 1. Following is a brief description of the numbered connectors in the

diagram to give some sense of how it all works for a standard user interaction.

1. User is redirected to the Admin Portal or the Public Portal from a public website. If not

authenticated, authentication is required (perhaps Active Directory for the Admin Portal,

Auth(0) for the self-registered users).

2. Portal loads page of available entities or ‘add New’ button; if the Public Portal, only the

user’s personal entity data is available and no selection is required or offered.

3. User selects entity to work with. Portal populates a new instance of ‘ITE Meta Data’ and

passes it to the Core with the user’s request.

4. In accordance with the request from the Portal, the Core accesses the Entity XML and

loads it.

5. Within the Core, the Entity XML is decrypted (if needed) and passed to the decision

models. Rules determine the next action aka process step.

6. If a Form is required, then the Entity XML is transformed into the Form’s ‘Base Document’,

which is deposited in the Blob Store.

7. Core replies to the Portal that the base document is ready for the Forms Engine. The

Portal redirects to the Engine.

8. The Engine acquires the new Base document from the Blob Store. The form or dialog is

loaded. Periodically the Base document and ITE Meta Data are auto-saved (time based

or ‘Save’ button). When the form/dialog is ‘Submitted’, one last save is performed.

9. User interacts with the form or dialog. Note that a form can be a very large unit of work,

akin to an entire application in a traditional context. A dialog is similar but served in

smaller bites to meet the needs of other form factors.


Whitepaper

© Idiom Ltd 2018

10. The Forms Engine may execute external services using its own event handlers.

11. When the Base document is ‘submitted’ the Engine redirects back to the Portal.

12. The Portal reacquires the ITE Meta Data for the transaction and passes it back to the

Core.

13. The Core reacquires the Base document, and applies it to the Entity XML. The rules are

cycled to determine the next process step, which may include external services (27), a

batch job, and/or generation of business documents and emails.

14. Batch jobs, document generation and email requests are invoked by placing the ITE

Meta Data document onto the Service Bus.

15. The Service Bus handler reads the Service Bus and processes the request. This request

may have come from the Core for rules based requests, a Portal for manual requests, an

external service, or the scheduler.

16. A batch can interact with external services.

17. A Batch Job can interact with the Entity XML, but only via the Core. The Core will

transform the Batch Job’s inbound data (see 18) and apply it to the Entity XML before

running rules again. The Core updates the Entity XML and processes it back into the

database.

18. The Batch Job deposits data to the Blob Store for use by the Core or by a Portal. Business

documents that are generated by the Document Generator are also deposited into the

Blob Store (25), where they can be downloaded by users (21) or Portals (12).

19. Outbound batch files may be deposited into the File Store for download by users or

external processes. Inbound batch files may similarly be deposited into the File Store for

use by the Batch Job.

20. The Batch Job handler interacts with the database to maintain process meta data (job

started, job complete etc) and to update data outside of the Entity XML as required.

21. Authorised Users can access the Blob Store to download generated documents.

22. Any process can use the Email Server to send email.

23. New requests from external services are intercepted by the Service Adapter, which

validates and authorises the request before either launching an asynchronous Batch Job

(14), or a synchronous Core request (24).

24. Service Adapter can process transactions against the EntityXML via the Core.

25. All generated business documents are persisted in the Blob Store. The document URI is

registered in the database (26).

26. All communications, including each document and email, are registered against the

Entity.

Services

Integration is notoriously bespoke, and the Transaction Engine design accepts and adapts to

this reality. Therefore we expect to build web, queue and other bespoke service adapters as


Whitepaper

© Idiom Ltd 2018

required to meet the functional and integration requirements of the entity life-cycles for each

business; each new adapter that comes online can be used immediately as a new ‘activity’,

and invoked via SME authored decision models.

Any existing adapter can be reused at will. This reuse is further facilitated by separating the

generation of the data values used by the service from their formats. An IDIOM decision

model is used to map the Entity XML to each adapter’s specific schema, in both directions.

Therefore, provided that the context data and the adapter data have an appropriate

correlation, the SME designer can always read/write data to/from the adapter. In a second

and separate step, the adapter itself is then responsible for mapping that data to any

underlying technical implementation that the correspondent end-point may require (XML,

JSON, CSV etc).

The Entity Builder Platform

The Entity Builder Platform is the suite of IDIOM tools that can be used to configure the

Transaction Engine for the specific business purpose that is defined by the entities that it is

governing, for instance, an insurance product platform, a claims platform, a loan platform,

etc. Each of these tools is described later in this document, and includes:

✓ IDIOM Decision Manager

✓ IDIOM Forms Builder

✓ IDIOM Document Generator

The IDIOM Decision Manager Workbench is used to perform forward looking policy

simulations and to regression test policy changes prior to deploying new business policy. This

tool is a standalone application that is used by the SME’s to help develop and quality assure

business policy; it is not a ‘builder’ tool like the others.

All of these tools can be used by appropriately trained SMEs or their respective analysts. They

are described later in this document.

GENERIC DATABASE

The Transaction Engine uses a generic database design. By this, we mean a database that

has no awareness in its design of what business entity data it contains, nor how that data is

structured.

This database design is key to allowing the business to be able to extend and adapt entity

data as and when required, which is in turn an important element of the ‘business as

content’ design objective of the Transaction Engine itself.

The design requires that business entity data is stored as XML content within a simplified

database structure. For those familiar with data design, the following database schema

provides an approximate outline of that database, and the following discussion offers some

insight into the underlying design pattern used for the Engine. This database design is

synthesized from several databases in large scale production use today.


Whitepaper

© Idiom Ltd 2018

The complete system is only 34 tables, and is capable of servicing a broad range of business

domains as is. We can now look at the various colour coded sections of the database in

more detail.

Figure 5 – Database Design for Generic Transactions

The Transaction Entity

[Yellow Tables]

The focus of this discussion is on the Transaction Entity, which is the centre yellow table.

In this table we can see the business entity data stored as ‘EntityXML’. For commercial

domains of even modest complexity, this simple design concept can save us from having to

build CRUD (Create, Read, Update, Delete) programs for many (often 10’s) database tables;

nor are we constrained by the incredible development inertia that is implied by these tables.


Whitepaper

© Idiom Ltd 2018

As a bonus, query performance against this structure is typically better than with standard

‘many table’ database schemas because joins are rarely required. Also, ‘full text’ search on

the SearchXML column allows Google like search capability over the full extent of the

database at little cost11.

On transaction termination, the Transaction Engine will place a copy of the updated

EntityXML into the Entity History table, so that a full trace is maintained of every state change

for every entity, every time. This trace includes the latest copy of the EntityXML and the

ITEMetaDataXML. The ITEMetaDataXML is the Meta-Data that is used to control and record

the processing of and by the IDIOM Transaction Engine itself – it is a process map. Unlike the

EntityXML, the structure of the ITE Meta-Data is known to the Transaction Engine. The ITE

Meta-Data has its own IDIOM defined schema.

The Transaction Entity columns are never updated directly by a user – there is no UI for this. It

is the responsibility of the EntityModel to set the column values in most cases, and the

Transaction Engine itself for the balance. Note that complete and context rich reporting

data can be generated by the Core’s decision models within each transaction, and stored

in the ReportingXML column for use by the Dashboard.

The Service Data, Batch Entry, Payment, and Entity Log tables are all part of the overarching

audit approach, and provide supporting data for the state changes made to the EntityXML.

The Communication Table is a more complex structure that provides an audit capability for

auto generated communications, as well as a user operable communications for a modest

CRM capability.

Archetype Meta-Data

[Green Tables]

The Archetype Meta-Data is the data that describes the structure and logic of the

transactions that manage the state of the subject entities – that is, it is data that describes

the processing aspects for the type of entity being managed. This is where the business

processes and the business logic become content in the system.

The entity data describes the entity at rest; the archetype meta data describes the processes

which govern the entity state changes. Together these classes of data describe the full life

cycle of the entity.

While we refer to this meta data as ‘archetype’, in some cases ‘product’ or ‘service’, or ‘life-

cycle’, or ‘pathway’, or ‘workflow’, or some other term could be more appropriate. All such

terms describe the processes that govern the entity life cycle, and all are intended to be

covered by the term ‘archetype’.

11 Our experience with SQL Server is that the performance cost of full text search is insignificant.


Whitepaper

© Idiom Ltd 2018

As we can see in the above diagram, the Transaction Entity is described by an Archetype

Version (see also Figure 6 below), which links the Transaction Entity with all of the Meta-Data

that is required to manage that entity’s life-cycle.

The archetype concept is used to describe and manage all of the product and service

offerings that can be supported by the Archetype, each of which is described by an

Archetype Version record. The Archetype is the root of the Transaction Entity concept: It

names the EntityXML schema (aka the God Document) that describes all versions of the

entity for all Transaction Entities that are associated with the Archetype; and the EntityModel,

which defines all of the rules that govern the entity’s valid state changes.

The Archetype Version is a direct parent of the Transaction Entity. From the Archetype

Version we can reach:

The Product Version(s), which includes an (optional) pre-built blank EntityXML document

(as NewXML) for new instances of Transaction Entities; and an (optional) configuration

document (as ConfigXML). The configuration document is a document that holds various

parameters that will be used by the Archetype’s decision models12 to make the decision

processing specific to this Archetype Version.

The core decision models that implement and control the Archetype’s processes:

▪ The Archetype EntityModel, which is responsible for managing all states for all entities

that are described by the Archetype;

▪ the Reporting Model, which will generate easily extensible entity level reporting data

to be used by dashboards and other reporting tools;

▪ the SearchModel, which will select all searchable text for the SearchXML;

▪ the Enumeration Model, which dynamically generates all pick-lists used by the Forms,

according to both context and version constraints;

▪ the RulesDiffModel, which is used to ‘difference’ (used as a verb) the current and

immediate prior EntityXML for audit purposes;

▪ and the Payment Model, which is used to apply third party payments to the

EntityXML.

The Archetype itself, which describes the context data schema that describes all

instances of the EntityXML.

Forms, which are built over variations (the FormBaseSchema) of the Archetype’s

EntityXML schema, with further links to the models that support the Form. These include

the Business Model, which applies business logic to the form’s data; the Session Model,

which controls the Form’s Meta-Data; and the Transform Model, which maps the

EntityXML to/from the FormBaseSchema (which describes the form’s base document).

Service Adapters that may be needed to support each transaction. Each service is

supported by a decision model that is responsible for mapping the service data to/from

the EntityXML (a partially completed ‘skeleton XML’ can be used to shortcut this process);

the rest of the service is usually coded bespoke. The Content Format attribute of this table

12 For the sake of clarity, whenever we refer to a decision model, we may be referring to an entire family of decision

models that logically act as one under the control of a ‘control model’; control models are a supported feature of

the IDIOM Decision Manager.


Whitepaper

© Idiom Ltd 2018

is then used to transcribe the decision model defined data into the required technical

format (e.g. XML, JSON, CSV, etc).

Batch Types that may need to be processed against the Entity, both inbound and

outbound.

A list of Documents that are supported by the IDIOM Document Generator (or perhaps a

third party document generator).

The actual decision models referred to above are found in the ‘Binary’ column of the

Decision Model table [green], and are loaded directly from the database at runtime.

Permissioning and Workflow

[Red Tables]

The red tables in the model are used for workflow and process control. There are two major

classes of users:

✓ Public – Non Domain Internet Users

✓ Domain – Domain Authenticated Users

The Transaction Engine can include its own security model for managing user registration and

authentication for these users, or it can link to externally managed authentication and

authorisation.

Authorisation is not required for Public users because their very limited authorities are pre-

programmed into the Public Access Portal.

The Domain Users are authorised by the Administration Portal, which uses a role/claim based

authorisation regime that is integrated with the Azure Active Directory and Identity Server.

Within the Transaction Engine, the Domain Users are further classified into Internal and

External Domain Users. The Internal Domain Users belong to the owner of the system

(OwnerFlag = True on Party), and are not limited in the scope of the Transactions that they

can see. External Domain Users belong to a Party other than the Owner of the system; for

these users, data scope is restricted to the Transaction Entities for which they have a current

Related Party record in the Related Party table. Related Parties are generated by the Entity

Model, and are validated/updated with every transaction.

Tasks are manual workflow items that are assigned to users. They can be generated by the

Entity Model when the model needs to orchestrate human activity. The Task Code is used to

link the Task to its Task Type, which in turn links the Task to a Role Claim and thereafter to

authorised users etc. All users that are associated with a Task are tracked via the Task Role

table.

BringUps are future dated events that can be associated with different processes. The event

that the BringUpCode represents is invoked on due date, at which time the BringUp process

will pass in the Reference from the BringUp record to initiate the transaction. The normal

BringUp will simply load the parent Transaction Entity’s EntityXML into memory along with its

associated Entity Model, thereby starting a standard transaction in the Transaction Engine. A

BringUp will often generate a Task.


Whitepaper

© Idiom Ltd 2018

Batch Jobs

[Purple Tables]

Batches and Batch Jobs are related but not the same. A batch must be imported or

exported by a Batch Job; however, Batch Jobs can also conduct file passes of the

Transaction Entities without the need for an external file, BringUps being one specific

example.

Batch Jobs will maintain some process meta data including start and end times, record

counts, and lists of failed records.

There are many uses for batch data, including:

Financial records that are generated for the corporate ledger

Bordereaux style records for third parties

Management information and dashboard data for MIS use.

Batch Entries represent events that will be raised on external systems. These batch entries

must be persisted in the Transaction Engine until the target system is ready to receive them.

At this time a Batch Job is started and all Entries that have not yet been transferred are now

transferred and stamped with the Batch Job ID. During the transfer, the EntryDataXML will be

formatted as appropriate for the receiving system according to the Format attribute of the

Batch Type table.

Inbound batch data follows the inverse path, hence the ‘Direction’ column on Batch Type.

As each inbound Batch Entry is processed against its Transaction Entity (always in a new Core

‘logical’ transaction) the Batch Entry is inserted between its Batch Job and the Entity History

record that it just caused.

Activity Log

Any process can generate an activity log for any entity instance (a row) in any table. If

required, before and after images of changed data can be inserted into the log record.

PUTTING IT ALL TOGETHER

Data

The primary context data (the ‘God document’) is the data that describes the entity ‘en

toto’. The context data is described using an XML schema (the EntityXMLSchema), which for

a meaningful commercial entity often scales to several hundred thousand nodes or more,

with many layers of collection depth. This entity specific data is NOT known to the Transaction

Engine.

The ITE Meta-Data is data that is additional to and apart from the data that describes the

entity, and is intended to be understood by the Engine. It is produced by the decision models


Whitepaper

© Idiom Ltd 2018

acting as proxies for the business SMEs that manage them, and is used to control the

Transaction Engine workflow, and through it, all downstream activities.

Before and after each execution cycle of the primary decision model(s), the Engine will

inspect the Meta-Data and act accordingly (this is analogous to content based routing in an

ESB). For instance, if the decision model determines that additional data is required from

some external service, the Meta-Data would be used to initiate the service. Similarly, on

transaction termination, which occurs at the conclusion of an entity state change, the

decision models can use this mechanism to ensure that all downstream activities are

updated and aligned, by indirectly posting updates to all affected systems.

Data that is generated by the models for consumption by these external services, or which is

provided by the services for consumption by the decision models, is cached in a data store

in the form required or created by the external service. This creates an obligation on the

decision models to manage the transformation and loading of this data, which also isolates it

and implicitly validates it.

Schemas

We have referred to several XML schemas. The full set includes schemas that are defined

uniquely for each transaction; and a further set that are used to communicate with, and

support the operation of, the Transaction Engine itself. The former are named within the data

model as ‘xxxSchema’; the latter have fixed names that are known to the transaction

engine, so they are not named in the database.

The bespoke (i.e. client defined) schemas include:

EntityXMLSchema

ReportingSchema

FormBaseSchema (which is extended to create the Session Schema)

ServiceSchema

BatchSchema.

The fixed schemas are defined for and by the Transaction Engine, and include schemas that

define the:

ITEMetaData

UserConfigXML

Enumerations.


Whitepaper

© Idiom Ltd 2018

ITE Meta Data

Figure 6 – ITE Meta-Data XML schema

Within the Transaction Engine Core the decision models are able to control all elements of

workflow that are relevant to the business entity through the ITE Meta-Data; the ITE Meta-

Data is always present for every transaction, and exists for the duration of a business (or


Whitepaper

© Idiom Ltd 2018

logical) transaction, including the extended period when a user can be interacting with the

entity via Forms.

When the logical transaction has terminated, it is then persisted and retained in the same

history table record that contains the EntityXML that the transaction updated.

The Core will inspect the ITE Meta-Data on every rules cycle, and will respond to workflow

requests by initiating the relevant activities as required, including:

✓ Forms and generated Documents are requested using their respective elements in the

ITE Meta-Data schema.

✓ Real-time service based activities can be requested from the Engine by creating nodes

in the ITE Meta-Data XML that nominate the service required. The engine will then

execute the ServiceControlModel as defined by the Service Adapter (for the requested

service) to transform the context data into the data required by the Service. Any data

returned by the activity is added to the transaction’s available context data using the

same decision model to transform the inbound data.

✓ Future Dated Activities are controlled by generating Bring-Ups as ITE Meta-Data within

the primary decision cycle. The BringUps are stripped from the ITE Meta-Data XML when

the transaction terminates and persisted against the appropriate Transaction Entity in

the database. The Transaction Engine will launch a business transaction for each

business (transaction) entity that is nominated by a BringUp record on the BringUp date.

✓ Manual activities are controlled by generating Tasks as ITE Meta-Data within the primary

decision cycle. The Tasks are stripped from the ITE Meta-Data XML when the transaction

terminates and loaded against the appropriate Task Type. The Task Type implicitly

assigns the Tasks to appropriate roles by the rules that create them. The Admin Portal will

automatically present the list of outstanding Tasks and prior assignments to users

whenever they are signed on.

Processes

Processes are handled in a non-standard manner in the Transaction Engine. For the reasons

described in the Process Article, pre-defined business processes are not a primary

requirements artefact in the IDIOM approach.

Business processes are controlled by the ITE Meta Data and ‘emerge’ from the decision

models that are controlling this data. The traditional view of a process can be described as a

pre-defined and standardised sequence of activities, which in the case of the Transaction

Engine may include (at least) Decision Models, Forms, Service Requests, Tasks, BringUps, and

Document Requests. Each activity may be required in response to many conditions in the

underlying context; it is the purpose of the decision models to interpret this context data and

to translate it into the best-next activity, just-in-time, every-time.

By sequencing these component activities using the decision models, any pre-definable

process can be synthesized. Many possible variations in the activity sequences can evolve

over time, so that the ‘source of (process) truth’ is actually to be found in the decision models

rather than in any external process documentation. There is an argument that these models

are valid process documentation, albeit defining a process that is inherently more agile and


Whitepaper

© Idiom Ltd 2018

flexible than any that can be defined using more traditional BPM documentation techniques.

And these ‘rules as process’ models are also likely to satisfy audit or other regulatory

requirements because the intent of the ‘process’ can usually be more easily shown to have

been addressed.

Workflow

From Wikipedia:

A workflow consists of an orchestrated and repeatable pattern of business activity enabled by the systematic

organization of resources into processes that transform materials, provide services, or process information.[1] It can be depicted as a sequence of operations, declared as work of a person or group, [2] an organization of staff, or

one or more simple or complex mechanisms.

From a more abstract or higher-level perspective, workflow may be considered a view or representation of real

work.[3] The flow being described may refer to a document, service or archetype that is being transferred from

one step to another.

Workflow has many forms. There are four main workflow concepts within the Transaction

Engine, all managed by decision models.

Firstly, workflow within a transaction is managed by the decision models within an IDIOM

Form or Dialog, or if no form is present, then inside the decision models that are orchestrating

the state change in the Core. An IDIOM Form or Dialog is an activity (aka a process) that can

have diverse and complex internal workflows; it can be of significant size and complexity,

spanning many (dozens) of normalised entities and thousands of data elements, involving

many user interactions, and potentially raising other secondary events. In many ways a single

IDIOM Form or Dialog more closely resembles a traditional application than a single process.

Secondly, workflow across and between activities is managed by the Transaction Engine

acting in response to events raised in the Core and triggered via the ITE Meta Data. These

events are created by the attendant decision models, and cause the Transaction Engine to

execute the activities via the Meta Data (e.g. one or more Forms, external services, etc.).

Thirdly, workflow that involves human operator intervention is managed by Tasks. Tasks are

events that are raised in the ITE Meta Data and then inserted into the database to drive

human activities.

Fourthly, workflow that spans time is managed by BringUps. BringUps are future dated events

that can cause the Transaction Engine to initiate any of the activities described above.

Task Management

Any Domain User can have access to Task lists, either at sign-on or via the menus.

The Tasks displayed in the list(s) are selected and ranked according to the user’s roles, and

by relevant dates and status etc.

https://en.wikipedia.org/wiki/Workflow#cite_note-ca-1

https://en.wikipedia.org/wiki/Workflow#cite_note-2

https://en.wikipedia.org/wiki/Workflow#cite_note-3

https://en.wikipedia.org/wiki/Document

https://en.wikipedia.org/wiki/Product_(business)


Whitepaper

© Idiom Ltd 2018

Depending on the system owner’s preferences, Tasks may be assigned or re-assigned for

action through self-selection (by the user), by others, or only by specific roles or levels of

seniority, or they may be operated on by any user on a first come, first serve basis, etc.

A task that is generated by decision models will usually cause some activity against the

EntityXML. This task driven activity will then cause an update to the Task list, as well as the

BringUps.

Therefore, when a Task is created, it is common to also create a follow up BringUp; if the

BringUp is still in existence on the BringUpDate, then it will cause the EntityXML to be opened

and reviewed. This process may in turn generate a new and more urgent Task (again with a

follow up BringUp) or some other activity.

This interplay of Tasks and BringUps is how human operator workflows are integrated into the

automated workflows that are being managed by the decision models.

Getting Started

As noted in our preamble, the Transaction Engine is always incomplete when delivered.

Each instance of the Transaction Engine will require some degree of tailoring to marry it up to

the existing infrastructure. This infrastructure can be all external for a new business, and/or it

can involve multiple legacy systems for a long established business.

Often, this retro-fitting process is limited to development of new Services, and perhaps linking

into existing corporate authentication and authorisation infrastructure. However, there is no

inherent limit to the customisation, and this can include extensions to the database and all

that this implies.

While the Transaction Engine is being ‘fitted’ into the technical environment, the SME’s should

be busy configuring the business content using the IDIOM tools. This process is usually started

using IDIOM consultants in a ‘buddy’ style development mode, followed by a transition to

SME development without IDIOM assistance. The SME learning period is usually measured in

weeks.

The tailoring of the Transaction Engine, and the development and testing of the IDIOM tool

configurations, should occur more or less independently of and in parallel with the technical

customisation. IDIOM recommends a maximum development window for both of these

activities of 100 working days; beyond that, we suggest reducing scope. Our underlying

principle is that a modest but concise initial business scope is an important ingredient of

success.

When the first iteration of the Transaction Engine has been deployed, ongoing adjustments

can be made by the SME’s acting alone, save perhaps for the comparatively infrequent

addition or update to Service adaptors as may be required to maintain alignment with the

external technology landscape.


Whitepaper

© Idiom Ltd 2018

The Generic Transaction

The following diagram [Figure 7] shows a representative transaction outline for illustrative

purposes that is capable of addressing many standard entity life-cycles, including those that

we have used as examples in this document (insurance policies, claims, loans, patient

episodes, patient clinical pathways, passengers, taxpayers, entitlements, et al).

The variations in the underlying entities, and by extension, their life-cycles, are all addressed

by the IDIOM tools within the ‘Entity Builder Platform’. The SME has extensive control over the

definition of these entities, and of the actions that form their respective life-cycles. Provided

that the various service end-points are available to supply data to the transaction on

request, and to respond to the updates generated by the transaction on termination, then

the transaction itself can be a black box.

The majority of the complexity in the transaction is in the decision models, which are

compiled and executed behind the standard interface that is built into the generic business

transaction itself. Independently measured projects have assessed this declarative

development effort as being ~20 times more efficient than traditional SDLC approaches to

achieve the same end; and the decision models are transparent, auditable, individually

testable and traceable, and more easily maintained.

The IDIOM Transaction Engine uses the Archetype to look up and load the relevant decision

models based on the nature of the entity that it is managing and as the specific event and

context requires.

Note that IDIOM Forms, Dialogs, and Document Generator use IDIOM decision models to

drive their respective runtime Meta-Data, which is what makes these runtime engines truly

generic. The ongoing development of the decision models, forms, and documents requires

the builder versions of each of these tools. These are shown inside the Entity Builder Platform.

The Workbench is also available in this platform to assist with policy modelling on the one

hand, and regression testing on the other.

Versioning and Release

Nimble, continuous, perpetual, business led development

An important objective of a transaction/decision centric application is nimble, continuous,

and perpetual development and release of business policy by business owners, or more

specifically, the SMEs. This business policy is usually focused on management of complex

entity life-cycles.

Policy adjustments should be able to be developed, tested and deployed at any time. The

intent is not to actually deploy changes (say) daily, but to be able to deploy new policy at

any time the business requires, without the delay and risk inherent in traditional ‘code-drops’.

In the Transaction Engine, all parts of the business transaction, including all policy changes,

are effective dated and cumulative forever. The business transaction is always able to

operate ‘as at’ any prior date. The IDIOM approach even allows effective dating that spans

changes in the entity schemas.


Whitepaper

© Idiom Ltd 2018

With a transaction/decision centric application there is a new dichotomy in versioning: the

schemas, forms, rules, and business documents, and all of their associated reference data,

can all be deployed externally to, and independently of, the code-base of the Transaction

Engine.

The continuous versioning that is inherent in the IDIOM approach is designed to support

nimble, continuous, perpetual, change in a decision centric application.

Transaction versions

A ‘transaction version’ is the net effect of all of the rules that implement the entirety of the

business policy as it was defined at the point in time that is defined by the transaction’s

effective date. Transaction versioning is therefore implicitly an effective dating regime.

Each change in the rules or of any aspect of the declarative design within a business

transaction implies a version update for that transaction, and by extension, for the product or

service that it supports. In the IDIOM Transaction Engine, all such versions remain valid and

operational, subject to the effective date of the transaction at runtime.

Business transactions can also be versioned on criteria other than effective date, through

configuration or rules changes – for instance, the term for specific conditions might be

negotiated so that each customer might see a different version of the same transaction rules

on the same date. This is achieved by dynamically setting the effective date using rules that

apply the specific conditions.

As we have noted, the business transaction concept is generic. When implemented as

described, this means that entirely new business entities and their related Meta-Data can be

introduced at any time, potentially without changing any host application code. For the

sake of clarity, new types of entities, including new product types, can be introduced and,

unless new activities are required, can be quickly made fully functional within the Transaction

Engine.

IDIOM TOOLS

As you might expect, IDIOM has been fine-tuning its flagship IDIOM Decision Manager

product since its launch in 2001 to fully support the concepts outlined in our series of articles

and this document.

The ‘idiom’ is the proprietary language that is used to describe the essence of any business –

it is used to describe its business policies, its decision making, and its business rules. And in so

doing, the idiom refers to the data in the context required by the rules. In fact, the name of

our company was an early reflection of the importance of the business ‘idiom’, and its

relationship to both business rules and data.

The challenge lies in mirroring this context within the various executables that implement it.

Defining, capturing, validating, & automating the business ‘idiom’ is the essence of our business and of our flagship product, the IDIOM Decision Manager


Whitepaper

© Idiom Ltd 2018

The IDIOM Decision Manager Workbench is an industrial scale, ready-made, stand-alone

batch application processor for large scale, high performance batch processing of IDIOM

decision models.

A transaction often requires dialog with a human actor. A dialog is easily managed by

building an IDIOM form over the business transaction’s entity and event data – the same

data that is available to the decision models that interpret and maintain it.

An IDIOM form includes the ability to execute IDIOM decision models inside the form on a

large scale, field by field if necessary, to ensure a truly reflexive user experience.

Sometimes the form factor of the client device does not fit well with the size and complexity

of an IDIOM form. In these cases, the Dialog Manager can be used to serve equivalent data

(to that of the form) in appropriately smaller sized packages, to be eventually rendered by

form factor specific user interfaces.

Finally, the transaction end state might need to be reported via one or more business

documents. The IDIOM Document Generator can be used to generate business documents

under the control of decision models, using only Word authored text and images as

additional design input.

IDIOM Decision Manager

✓ IDIOM Decision Manager is a tool for graphically modelling, testing, and deploying

business decision-making logic – without programming!

✓ A tool for the policy maker, not the programmer.

✓ IDIOM automates complex decision-making at the enterprise level, deployable as

industrial strength stand-alone software components.

✓ In day-to-day practice it is used by SMEs, or analysts working closely with them.

✓ Together they model the business policies in terms of both data and decisions (see

Decision Model below) before moving on to define the underlying logic that binds them

together (see Formula below).

✓ The decision logic and data are usually modelled together in a single combined process

of analysis and definition.

✓ The data model and the decision model share the same ‘context awareness’, with

current-context and context boundaries visually highlighted within the development

palettes.

✓ Testing of the models is available at all times within the development palettes

themselves; extensive regression testing (incl. ‘model answer’ differencing) is available in

the on-board ‘Test Executive’ (not shown).

✓ Deployment as 100% generated software components is fully automated and ‘without

fingerprints’.

✓ Example below is a small but real model drawn from a city council implementation of

policy that calculates financial contributions to be paid by property developers.

✓ The problem domain is decomposed using a ‘mind mapping’ approach until we reach

the atomic units that we call decisions (rounded boxes).


Whitepaper

© Idiom Ltd 2018

✓ This ‘decision model’ and the adjacent data model (left hand panel below) are

demonstrably aligned and integrated through shared context – validating and

strengthening both.

✓ The data model defines the entity at rest; the decision model defines the valid state

transitions. Together they completely define the entity life-cycle and required business

policy.

✓ The atomic ‘decisions’ provide an easy entry point for specification of the underlying

rules details via the Formulas (see next).

✓ The underlying rules details are easily captured using a ‘Lego like’ drag-and-drop

approach that is ‘more fun than playing golf’ according to the CEO of one of our largest

customers – there is no scripting or coding required to build formulas.

✓ The rules can be tested immediately within the IDIOM Decision Manager palettes.

✓ When finished, IDIOM Decision Manager generates computer source code (C# or Java)

with a single button click, to be called by any application at run-time using any of a

wide variety of supplied interfaces and wrappers (in-line, dll, web service, queue service,

and many more).

✓ And at the same time, it generates the models into business readable PDF

documentation.

Figure 8 IDIOM Decision Manager - Decision palette showing a Decision Model and Schema


Whitepaper

© Idiom Ltd 2018

Key points of difference

✓ IDIOM’s decision models do for decisions what data models do for data – a powerful

abstraction that makes the underlying complexity visible and manageable.

✓ The models allow data transformations and more traditional business rules to be

intermingled. Business rules acting alone are severely limited in their ability to fully resolve

complex commercial problems – invariably, in-line data transformations are necessary to

facilitate the calculate/adjudicate/act behaviour of business rules.

✓ Context is continuously displayed and actively managed.

✓ Decision models that incorporate both data and rules behaviour enable a further critical

capability that is unique to IDIOM Decision Manager – the models can be fully tested

using real-world cases directly in the builder palettes. No external technology or

application support is required to empirically prove the completeness, consistency, and

correctness of the models.

Key points of innovation

✓ Fundamental redesign of the traditional SDLC by fully separating the development and

automation of business policy (deciding) from development of the system’s activities

that support it (doing).

✓ Use of IDIOM is effective in spawning a ‘Business Policy Development Life Cycle’ that is

managed independently of and alongside the traditional Systems Development Life

Cycle.

Key value propositions

✓ 100% alignment of systems based decision making with business policy, because the

business owners have hands-on custody and control of the policy definitions actually

used by the system.

✓ Increased agility with reduced business risk through business modelling and empirical

testing of policy definitions prior to automated generation and implementation.

✓ Significant reduction in the business cost of developing and implementing automated

business policy.

✓ Further reduction in software development time, cost, and risk through reduced system

complexity, fewer moving parts, & clear separation of concerns.

Further benefits

✓ Full auditability of policy changes, and visibility of policy implementation through the

graphical models and logical English PDF generation.

✓ Decision model artefacts can be traced to/from ‘sources of truth’ in underlying business

policy documents (Word, Excel) using the IDIOM Tracker for requirements traceability.

✓ The IDIOM Decision Manager Workbench is available to experiment with and further

develop policy independently of the underlying systems and of the SDLC.

✓ Automated, robust, industrial strength deployment on any scale that can be supported

by the ‘host application’ and its underlying platform.

✓ Simple injection into legacy systems leading to eventual legacy replacement


Whitepaper

© Idiom Ltd 2018

Figure 9 IDIOM Decision Manager - showing the Formula Palette

IDIOM Forms

IDIOM Forms is a tool for generating web2.0 forms that embed the power of IDIOM decision

models into the form itself. As the user navigates through an IDIOM Form, the integrated

decision models can be executed on an element by element basis to make various business

calculations and assessments; and to modify the form’s meta-data. Control of the form’s

meta-data allows the decision models to dynamically adjust the visible shape and content

of the form on an element by element basis.

A single IDIOM Form can be used to process complex business transactions (e.g. an

insurance application, a clinical pathway, a loan application) through to closure, including

such functions as validation, transaction acceptance, costing or pricing, up-selling, and

determination of subsequent workflow amongst others. Usually, a single form is used to

manage the full life-cycle of the subject entity (i.e. not just individual state changes). The

IDIOM Forms Engine (the runtime component of IDIOM Forms) is production hardened after

many years use on thousands of servers throughout the NZ health sector. The deployed

IDIOM Forms Engine became the basis for the NZ Health Information Standards Organization

(HISO) forms standard and is currently the most compliant under that standard. The Forms

Engine is now also widely used in the finance and government sectors.


Whitepaper

© Idiom Ltd 2018

IDIOM Dialog Manager

The IDIOM Dialog Manager leverages standardised forms definitions and platform specific

implementations to deliver tailored conversations to different form factors (e.g. iPhone,

Android Phones, Tablets, etc.). The forms are defined using an industry standard forms

schema13 that has multiple industry supplied implementations in different frameworks. The

IDIOM Dialog Manager maintains a small library of forms definitions and progressively supplies

fragments of XML data to these forms from an underlying base document. IDIOM decision

models are used to orchestrate the form sequences and to transform the inbound/outbound

data, which is mapped to/from the base document as required. The core business decision

making is implemented against the base document and executes at each step in the forms

cycle.

IDIOM Decision Tracker

The IDIOM Decision Tracker is a tool to map Microsoft Word and Excel policy documents to

IDIOM decision models for full bi-directional traceability between corporate policy definitions

in the Microsoft documents and their actual implementation as IDIOM generated decision

engines.

IDIOM Mapper

The IDIOM Mapper builds an in-memory XML document that mirrors the normalised database

structure that it is reading from. It generates ‘simple SQL’ from an XML configuration

document, and then executes a full round trip transaction cycle: from database to XML;

then rules execution (multiple decision models); and from XML back to the database if

required. All done extremely quickly, either in single transactions or in batch. The Mapper

process is thread-safe and can execute in many process streams for scalability that is limited

only by the database and its underlying platform. The Mapper is not normally used within the

Transaction Engine, which does not use the traditional fully normalised data approach, but

may be useful for tight legacy systems integration.

IDIOM Decision Manager Workbench

The IDIOM Decision Manager Workbench is a business operable application for running

decision models across enterprise databases on a large scale without the need for IT

technical support.

✓ A generic batch processing platform for use by IT and/or business operations.

13 see http://jsonschema.net/

http://jsonschema.net/


Whitepaper

© Idiom Ltd 2018

✓ A platform for the auditor, the business policy manager, the product manager, the

corporate business strategy manager and the IT manager.

✓ An audit tool for identifying, reporting, and managing anomalies, errors, and issues of

concern in any database.

✓ A simulation tool for comparing the performance of current and ‘to be’ versions of any

automated policy.

✓ A tool that can intelligently modify the inbound data to simulate changes in the make-

up of inbound transactions over time.

✓ A data masking tool, able to replicate large scale databases with all personally

identifying details masked from its own library of several million fake identities.

Relationships between identities in the source data can be maintained, notwithstanding

the use of fake identities.

✓ A system conversion tool: able to read data from one system in its proprietary format;

intelligently transform it through one or more decision models; and then output it to a

new system also in its proprietary format. Millions of entities described by hundreds of

tables can be supported.

Figure 10 IDIOM Decision Manager Workbench

Examples of use

Superannuation Fund Administrator: Perform all period end processing, including fee’s,

insurances, member adjustments, entitlements, and reporting, for several hundred

thousand fund members.

Insurer: Compare the current underwriting policy with a proposed underwriting policy

across a recent portfolio of 500,000 insurance policies to determine potential changes in

the rate of referral and its attendant costs.

Superannuation Fund Administrator: Run 100’s of distinct audit tests across 1million

member accounts on a daily basis to independently verify the production data.


Whitepaper

© Idiom Ltd 2018

City Council: Compare this year’s rating policy with next year’s proposed rating policy

for each property in a city of more than 500,000 ratepayers to identify outlier changes in

the rates actually charged.

Insurer: Dynamically modify key attributes of real transactions to simulate changes in the

make–up of in-bound business, and assess the impact of these changes across an entire

portfolio.

Enterprise class features

✓ Full authorization and audit controls down to the field level for all users of the platform.

✓ Seamless operation across multiple user definable environments, for instance

Development, UAT, Simulation, Production.

✓ High performance, including parallel processing on a large scale across multiple

machines, for instance running up to 24 Processes on an i7 class CPU, with multiple CPU’s

possible, each logging back to the Workbench for centralized management.

Measured performance

✓ Daily pass of 1million pension fund members each comprised of a join of 20+ complex

member related tables (~one billion rows in total).

✓ 10’s of decision models implementing hundreds of individual member tests.

✓ Alerts captured and reported daily for start of business.

✓ Run daily in 2 hours using one i7 class processor, with data drawn from an IBM iSeries.

IDIOM Document Generator

The IDIOM Document Generator is a tool to collect and collate Word documents (which are

used as document ‘templates’) and many associated Word text fragments [‘blurbs’], and to

assemble these into complete documents under the control of a decision model.

At document design time, the Document Generator inspects the templates and blurbs and

builds a list of all blurbs (including nested blurbs) that could be used for each candidate

document; and it also builds a list of all of the variables declared within them. It then places

these two lists (blurbs and variables) into an XML ‘control document’.

A decision model is then built that will extend this XML control document at runtime when the

specific business entity context is known. This decision model will analyse the business entity

context data in order to set flags to control the insertion (or not) of each blurb; and it will

derive the substitution values for each of the listed variables.

At runtime, when the actual business entity data is provided and the decision model has

executed, the Document Generator will read the tags in the control document to insert the

desired blurbs into the template, followed by substitution of all variables, so that large and

complex documents can be generated automatically under the control of the decision

model to reflect the exact circumstances of the underlying business entity. The Document

Generator also allows for insertion of images and for calling bespoke programs to insert

additional generated text or images. Documents can be generated into a number of

formats, including Word and PDF. In day to day use by SME’s, the Document Generator only


Whitepaper

© Idiom Ltd 2018

requires standard Microsoft Word and IDIOM Decision Manager authoring skills to define new

documents.

Figure 11 IDIOM Document Generator

CONCLUSION

This whitepaper describes the rationale, structure, and use of the IDIOM Transaction Engine.

If you are a prospective user of this innovative platform, then the next step is to do a proof-of-

concept using the IDIOM tools to abstract your business within the engine.

IDIOM will be pleased to work with you to complete these activities, after which we can fix

price the end-to-end application delivery. We are then ready to start development, and you

should be operational by the end of the next quarter 100 days away.

Finis

Mark Norton | CEO and Founder | IDIOM Limited

+64 9 630 8950 | +64 21 434 669 | Australia free call 1800 049 004

2-2, 93 Dominion Road, Mount Eden, Auckland 1024 | PO Box 60101, Titirangi, Auckland 0642, New Zealand

@ [email protected] | idiomsoftware.com | Skype Mark.Norton



+64 9 6308950 | @ [email protected] | idiomsoftware.com

2-2, 93 Dominion Road, Mount Eden, Auckland 1024

About IDIOM

Established in 2001, IDIOM Limited is a private company based in Auckland, New Zealand.

IDIOM develops and licenses decision-making software that automates business policy on a

large scale, making systems more transparent, more agile, and more durable, while reducing

development time, cost, and risk.

IDIOM’s innovative business oriented software is used by business users to graphically define,

document, and verify corporate decision-making and related business rules; it then auto-

generates these into small footprint, non-intrusive software components for use in systems of

any type or scale. IDIOM is a pioneer in the development and use of decision automation

concepts, and has applied these concepts to develop and automate business policy for

Partners around the world in local/state/central government, health admin/clinical health,

insurance/superannuation/finance, telecoms, logistics, and utilities.

IDIOM develops and licenses: IDIOM Decision Manager™, IDIOM Forms™, IDIOM Document

Generator™, IDIOM Dialog Manager™, IDIOM Mapper™, IDIOM Tracker™, IDIOM Decision

Manager Workbench™ and the IDIOM Transaction Engine™

