Download - Enterprise (Data) Architecture
![Page 1: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/1.jpg)
Enterprise (Data) Architecture
Christian NentwichModel Two Zero
![Page 2: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/2.jpg)
Caveat Emptor
a)This is an introductory overview, a detailed treatment would take at least a whole term!
b)It’s fiendlishly complex – nobody has a satisfactory answer
![Page 3: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/3.jpg)
About Model Two Zero
• Software company based near “Silicon Roundabout” (Old Street, London)
• Mission: help large businesses establish the next generation of enterprise data architectures
• Innovators in:– Executable specifications– Code generation from (restricted) natural
language– High-volume document matching
![Page 4: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/4.jpg)
About Me
• BSc and PhD in Computer Science from University College London
• Never held down a “proper job”...• Founded two startups – Model Two Zero is
number 2!• Long involvement in financial services IT
– Strategic data architecture consultancy for many big banks, fund managers, clearing houses
– Pushing forward industry-wide data standards
![Page 5: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/5.jpg)
Motivation• Large companies build or buy new systems all the
time• Nobody likes monolithic systems, but components
need to be assembled
Customer Records Invoicing System
Order Management System
Batch load or real-time query
Batch load Real-time message
![Page 6: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/6.jpg)
• Companies also buy other companies• Consolidation requirements
Company BAccounting System
Company AAccounting System
Motivation
JointAccounting System
![Page 7: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/7.jpg)
No planning Chaos
![Page 8: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/8.jpg)
In Real Life
![Page 9: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/9.jpg)
Format Diversity• Relational databases• APIs
– Java– .Net– Mainframe
• File formats– Comma-separator– Fixed width– XML– COBOL mainframe data
files– Tag/Value files– Proprietary binary
• Message formats– XML– JSON– Proprietary queueing
systems
• Taking a consistent approach is a challenge
![Page 10: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/10.jpg)
Complexities• Domain complexity• Architecture diversity• Volume of data• Complex, often out of date business processes• Key person dependencies• Lost business knowledge• Time pressure
![Page 11: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/11.jpg)
ESB – Key Idea• From this:
![Page 12: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/12.jpg)
ESB – Key Idea• To this:
Common Bus
![Page 13: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/13.jpg)
Zoomed in View
Common Bus
Proprietary Data Format
Standardised Data Format
Adapter
Proprietary System
![Page 14: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/14.jpg)
Calculation• Bilateral connections require O(n2) integration
projects for n systems• ESB with standard formats and adapters requires
2n = O(n) integration projects
• This calculation does not always work out in practice!
• Why?
![Page 15: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/15.jpg)
Key Disciplines• Data Modelling
– The data is more important than any execution technology!
– Need to define a “standard language” for interchanging data on the bus
– Very frequently XML and XML Schema– Sometimes JSON, but schema definition is problematic
• Governance– Curation of the standard– Versioning (one of the hardest problems in data
architecture)– Regression testing and rollout management
![Page 16: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/16.jpg)
Modelling - UML
![Page 17: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/17.jpg)
Modelling – XML Schema
![Page 18: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/18.jpg)
Quick Comparison• UML
– Fits lots of information on one page
– Understood by relatively technically unsophisticated users
– A huge standard, contains more than is necessary for the task
– Has no “wire format”
• XML Schema– Technically complex– Comes with a wire
format – XML– Liked by developers– Fairly big standard,
contains more than is necessary for the task
![Page 19: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/19.jpg)
Governance• Central team • Distributed teams
(open source style)
StandardModel
Changes
StandardModel
Changerequests
![Page 20: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/20.jpg)
Versioning• Versioning of standard data formats needs to deal
with two important scenarios:– Backward compatibility: a new version of a connecting
system is able to read/write an old version of the standard
– Forward compatibility: an old version of a connecting system is able to read/write a newer version of the standard
• Careful not to confuse them – most people do
![Page 21: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/21.jpg)
Versioning Example
<customerAccount> <firstName>Christian</firstName> <lastName>Nentwich</lastName> <address1>Somewhere</address1> <address2>in</address2> <address3>London</address3> <postcode>ABC CDE</postcode></customerAccount>
System A
System B
produces
consumes
V1.xsd
validates
![Page 22: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/22.jpg)
Versioning Example
<customerAccount> <firstName>Christian</firstName> <lastName>Nentwich</lastName> <dateOfBirth>1977-01-01</dateOfBirth> <address1>Somewhere</address1> <address2>in</address2> <address3>London</address3> <postcode>ABC CDE</postcode></customerAccount>
System A
System B
produces
consumes
V2.xsd
validates
![Page 23: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/23.jpg)
Changes• Adding optional
elements– Leaves senders
unaffected– Breaks receiver forward
compatibility if receivers cannot deal with unknown elements
• Adding mandatory elements– Breaks senders– Breaks receivers
• Removing mandatory elements– Breaks senders– Breaks receivers
• Removing optional elements– Breaks senders– Leaves receivers
unaffected
![Page 24: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/24.jpg)
Service-Oriented Architecture• Establish a landscape of coarse grain business
services• Establish service wrappers around applications• Build “value add” applications by composing /
orchestrating services
Reservation ServiceCustomer ManagementService
Flight bookingorchestration
![Page 25: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/25.jpg)
REST Wrapper
WSDL Wrapper
Service-Oriented Architecture
Reservation Service
Flight bookingOrchestration
(BPEL)
Customer ManagementService
![Page 26: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/26.jpg)
Event-Driven Architecture• Build an architecture build on business events
rather than data• Components of the architecture subscribe for
updates and listen for events they are interested in (sources and sinks model)
<trade> <id>abc</id> <amount>500000.00</amount> <ccy>GBP</ccy> <ticker>IBM</ticker></trade>
<tradeEntered> <id>abc</id> <amount>500000.00</amount> <ccy>GBP</ccy> <ticker>IBM</ticker></tradeEntered>
From this:
To this:
![Page 27: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/27.jpg)
Event Modelling
<tradeEntered> <id>abc</id> <amount>500000.00</amount> <ccy>GBP</ccy> <ticker>IBM</ticker></tradeEntered>
<cancellation> <id>abc</id></cancellation>
<amendment> <id>abc</id> <amount>400000.00</amount></amendment>
![Page 28: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/28.jpg)
Event-Driven Architecture• Discipline: send only data required for an event• Model things that actually happen in the business• Let participants determine how to react to events
– No central planning– Scalability benefits
![Page 29: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/29.jpg)
Semantic Integration• Instead of “syntactic” manipulation of data,
specify the full meaning of data in each system• Relate data items to an overarching “ontology”• Infer integration automatically• Immature discipline – many issues to solve
Field9234B TrdDt
StartDate
Trade Date
![Page 30: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/30.jpg)
Epilogue: What we do
Instead of writing informal specifi-
cations and then code, create executable specifications.
![Page 31: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/31.jpg)
What we do – Executable Specifications
public void validate(Transaction transaction) { if (transaction.getDebit() != null && transaction.getCredit() != null) { if (transaction.getDebit().getValue() == transaction.getCredit().getValue() && getChangeInAmount() != null) { exceptions.add(new ValidationError( "Must not specify changeInAmount if a debit is equal to a credit")); } }}
context: Transactioninv: (self->debit = self->credit) implies (self->changeInAmount->empty())
<rule context="ns:transaction"> <assert test="ns:debit != ns:credit or not(ns:changeInAmount)"> Must not specify changeInAmount if a debit is equal to a credit </assert></rule>
Java
OCL
SchematronNRLContext: TransactionValidation Rule "Our Sample Rule"If a debit is present and a credit is present then no changeInAmount is presentReport 'Must not specify changeInAmount if a debit is equal to a credit'
![Page 32: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/32.jpg)
Natural Rule Language• An open language for expressing:
– Validation rules (constraints)– Action rules (e.g. Enrichment rules)– Transformation / mapping
• Aimed at the core problem areas in integration• Goals
– Read like an English sentence wherever possible– Require no customisation to get going– Offer symbolic and textual alternatives
• Specification: http://nrl.sourceforge.net
![Page 33: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/33.jpg)
NRL Parser
• The NRL parser is free and open source, also on sourceforge
• Designed for processing / code generation
ConstraintRuleDeclaration
IfThenStatement
ExistsStatement ExistsStatement
1. Text in NRL concrete syntax 2. Parser-generated AST
3. Code generators (from AST)
![Page 34: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/34.jpg)
Possible Projects• Propose solutions to the model versioning / XML
schema versioning problem– Classify permissible changes in detail– Specify a formal logic for proving the impact of changes,
and establishing policies– Review W3C approaches to the problem and contrast
![Page 35: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/35.jpg)
Technologies to Review• Data standards /
formats– XML / XML Schema– JSON
• Semantic standards– RDF– OWL– SBVR– NRL
• Open source frameworks– Spring integration– Mule ESB
• Messaging– AMQP– SOAP / XML RPC– JMS
• Useful– REST– XPath– Xquery– Schematron– JAXB– JIBX– SAX / DOM– Python format modules
![Page 36: Enterprise (Data) Architecture](https://reader036.vdocument.in/reader036/viewer/2022062222/56816223550346895dd251e5/html5/thumbnails/36.jpg)
www.modeltwozero.com