reference architecture - highways englandassets.highways.gov.uk/specialist-informa… · web...

NOT PROTECTIVELY MARKED

Enterprise Architecture 2010

Reference Architecture

Enterprise Metadata Management

June 2010

Version: 1.0

Editor Mike Williams

Status: Issued

Date: 16 June 2010

HA Reference Architecture

_________________________________________________________________________

Document Control

Document Title Reference Architecture – Enterprise Metadata ManagementAuthor Mike WilliamsOwner Ivan WellsDistribution General – SHARE and PartnerNETDocument Status Issued

Revision History

Version Date Description Author0.1 30th December 2009 First draft. Mike Williams1.0 16th June 2010 Baseline issue. Mike Williams

Forecast Changes

Version Date Description

Reviewer List

Name Role

Ominder Bharj ReviewerIan Cornwell Mott MacDonald (ITS Metadata Registry)Dave Waite Atos Origin

Approvals

Name Title Date Version

Ivan Wells Strategy and Architecture

Document References

Document Title Document Links

[1] Technology Policy L1 – Corporate Data Catalogue Deprecated.

[2] EA White Paper – Meta-models TBA

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

CONTENTS

1 INTRODUCTION...............................................................................................................4

1.1 PREAMBLE......................................................................................................................4

1.2 RELATIONSHIP TO REFERENCE MODELS...........................................................................4

2 PLATFORM INDEPENDENT MODEL (PIM)....................................................................7

2.1 DEFINITIONS....................................................................................................................7

2.2 REGISTRIES AND REPOSITORIES......................................................................................8

2.3 META-MODELS...............................................................................................................11

2.4 CATALOGING: KEY TO ARTIFACT DISCOVERY..................................................................12

3 PLATFORM SPECIFIC MODEL (PSM).........................................................................13

3.1 INTRODUCTION..............................................................................................................13

3.2 ENTERPRISE METADATA MANAGEMENT OPEN INTEGRATION FRAMEWORK.......................13

3.3 SOA REGISTRY REPOSITORY........................................................................................16

3.4 THE BUSINESS INTELLIGENCE DOMAIN............................................................................18

3.5 THE EA MODELLING DOMAIN..........................................................................................21

3.6 Summary and Overview..............................................................................................22

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

1 INTRODUCTION

1.1 Preamble

Reference architectures describe one or more Architecture Building Blocks for architectures in a particular domain. They also provide a common vocabulary with which to discuss implementations, often with the aim of stressing commonality. In Model-Driven Architecture (MDA) terms, they equate to Platform Independent Models (PIM’s).These represent (potentially re-usable) components of business, ICT, or architectural capability that can be combined with other building blocks to deliver architectures and solutions. Building blocks can be defined at various levels of detail, depending on which stage of architecture development has been reached. For instance, at an early stage, a building block can simply consist of a name, or an outline description, in architecture models which represent a placeholder for subsequent specifications. Later on, a building block may be decomposed into multiple supporting building blocks that may then be accompanied by full specifications.

Reference Implementations are examples of software specifications. These are intended as a guide for Service Providers to develop concrete Solution Building Blocks (SBB’s). In Model-Driven Architecture (MDA) terms, they equate to Platform Specific Models (PSM’s).

These PSM’s are described as either Commercial-Off-The-Shelf (COTS) or Open Source Software (OSS). In this respect, the HA Technology Policies are aligned with Cross-Government Enterprise Architecture (xGEA) Technical Policies. These specify that OSS components should be considered as viable building blocks wherever they can be shown to meet the business requirements and offer Value for Money (VfM). Therefore, actual product selections will generally be determined through procurements and their evaluations of the Most Economically Advantageous Tenders (MEAT).

Where such selections have already been made, the Reference Implementations will be superseded by Level 2 (Physical) Technology Policies which reinforce the use of those components. Some of these components will stem from a build out, through re-use, of the HA’s more recently acquired, existing infrastructure assets and investments, such as in Business Intelligence. In all other cases, the PSM’s will be based on OSS projects which implement the relevant Open Standards.

1.2 Relationship to Reference ModelsThis reference architecture refers to the Enterprise Metadata Management capability. This is positioned within EA as depicted in Figures 1-3 for the Enterprise Architecture Reference Model (EARM), Technical Reference Model (TRM) and Data Management Framework respectively.

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

Figure 1 - EARM Context

Figure 2 - TRM Context

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

Figure 3 - Data Management Framework context

This reference architecture supercedes Ref.[1] Technology Policy L1 – Corporate Data Catalogue.

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

2 PLATFORM INDEPENDENT MODEL (PIM)

2.1 Definitions

Metadata is "data about data".

There are two distinct classes of metadata: Structural (or control) Metadata and Guide Metadata. Structural Metadata is used to describe the structure of machines such as tables, columns and indexes in a database. Guide Metadata is used to help humans find specific items and is usually expressed as keywords in a natural language search.

Types of metadata include:

Information Technology and Software Engineering metadata o General IT metadatao IT metadata management productso Relational database metadatao Data warehouse metadatao Business Intelligence metadatao File system metadatao Program metadatao Existing software metadatao Document metadata

Digital library metadata Image metadata Geospatial metadata Meta-metadata

Key structural domains for the HA are depicted in Figure 4 below.

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

Figure 4 - Metadata Domains

Metadata Management comprises the Planning (P), Development (D), Operations (O) and Control (C) activities necessary to enable easy access to high quality, integrated metadata. Specifically, it includes the following functions:

Understand Metadata Requirements (P) Define the Metadata Architecture (P) Develop & Maintain Metadata Standards (P) Implement a Managed Metadata Environment (D) Create & Maintain Metadata (O) Integrate Metadata (C) Manage Metadata Repositories (C) Distribute & Deliver Metadata (C) Support Metadata Reporting and Analysis (O)

The first three of these start with this document. The current issue of the Reference Architecture is focused mainly around the centre circle in Figure 4, however, some of the key domains are included as they have been identified as priorities with respect to current and planned projects, including:

1. Data Architecture and Modelling Tools as used by the EA and BI teams.2. Federated Third-Party Data Exchange as used by the ITS Metadata Registry project

and the Travel Information Highway (TIH) community.3. Service-Oriented Architecture (SOA) as planned for the EA To-Be architecture.4. Geospatial as planned for the forthcoming Geographical Information Systems (GIS)

Strategy.5. Data Warehouse and Business Intelligence as used by the BI Programme.

Other domains may be added as and when required or covered in other related artifacts.

2.2 Registries and Repositories

2.2.1 IntroductionA metadata registry is a central location in an organisation where metadata definitions are stored and maintained in a controlled method.

Metadata registries are used whenever data must be used consistently within an organisation or group of organisations. Examples of these situations include:

Organisations that transmit data using structures such as XML, Web Services or EDI, for example, the Travel Information Highway (TIH) organisation employs the HA’s ITS Metadata Registry (www.itsregistry.org.uk).

Organisations that need consistent definitions of data across time, between databases, between organisations or between processes, for example when the HA BI Programme builds a data warehouse.

Organisations, like the HA, that are attempting to break down "silos" of information captured within applications or proprietary file formats.

A metadata registry typically has the following characteristics:

It is a protected area where only approved individuals may make changes. It stores data elements that include both semantics and representations. The semantic areas of a metadata registry contain the meaning of a data element

with precise definitions. The representational areas of a metadata registry define how the data is represented

in a specific format such as within a database or an XML Schema.

_________________________________________________________________________06/05/2023 v1.0 of 22

http://www.itsregistry.org.uk/


_________________________________________________________________________

There are a number of international standards in this area – these are covered in the following sub-sections.

2.2.2 ISO 11179 Metadata Registry (MDR)The ISO 11179 standard provides some guidelines for the description of individual data elements. The ISO 11179 standard is broken into six parts:

ISO 11179-1 Framework for the specification and standardisation of data elements: Provides an overall view of how data elements should be annotated and classified. It basically serves as a roadmap to how the other parts of the ISO specification should be used. ISO 11179-2 Classification for data elements: Defines an ontological classification mechanism for the clear and unambiguous classification of individual data elements (values).

ISO 11179-3 Basic attributes of data elements: Provides a brief list of metadata that should be collected for data elements. These include information about the value itself (such as whether it is required or optional, how many times it may appear, and what the allowable values for the element are) as well as information about the meaning of the element (representational information from ISO 11179-2, keywords, and so on).

ISO 11179-4 Rules and guidelines for the formulation of data definitions: Provides some best practice naming and definition approaches for individual data elements. This includes simple rules such as “element names should be singular”, as well as more sophisticated guidelines such as “element definitions should not contain circular references.”

ISO 11179-5 Naming and identification principles for data elements: Provides some standardised mechanisms for the unique identification of data elements. It includes some guidelines for creating element names from the classification of those elements, as well as a mechanism for associating a universally unique identifier with each element.

ISO 11179-6 Registration of data elements: Describes a methodology for the registration of individual data elements. It describes how data elements should be versioned over time, and provides for a way to uniquely identify a data element through a combination of its registry authority identifier, its own unique identifier within that registry, and its version number.

2.2.3 ISO 15000 - ebXML registry and repository (regrep)The OASIS standards body is responsible for the Electronic Business using eXtensible Markup Language, or ebXML, standard (www.ebxml.org). This standard actually consists of several more granular standards, including a standard for a Collaborative Partner Profile Agreement (CPP/A) describing an agreement between trading partners, a Messaging Service Specification (MSS), a Business Process Specification Schema (BPSS), and a Registry Information Model (RIM). This registry model was promoted to version 3.0 in May 2005, and provides a common mechanism for the sharing of technical information (XML Schemas), business process descriptors (BPSS documents), and collaboration partner profiles and agreements (CPP/A documents) to fully describe the business relationships between two or more participants in an information sharing effort. Because the ebXML registry effort is a natural outgrowth of ebXML’s other goals (including the BPSS documents and CPP/A documents), an ebXML registry is both a technical registry (so-called “green pages”) as well as a business registry (so-called “white” and “yellow” pages).

2.2.4 Universal Description, Discovery and Integration (UDDI)

UDDI is a platform-independent, XML-based registry for businesses worldwide to list themselves on the Internet. UDDI is an open industry initiative, sponsored by OASIS, enabling

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

businesses to publish service listings and discover each other and define how the services or software applications interact over the Internet. A UDDI business registration consists of three components:

White Pages — address, contact, and known identifiers; Yellow Pages — industrial categorisations based on standard taxonomies; Green Pages — technical information about services exposed by the business.

UDDI was originally proposed as a core Web Service standard. It is designed to be interrogated by SOAP messages and to provide access to Web Services Description Language (WSDL) documents describing the protocol bindings and message formats required to interact with the web services listed in its directory.

2.2.5 The Need for an Integrated SOA Registry-RepositoryThe HA’s future state (To-Be) architecture is predicated on a Service Oriented Architecture (SOA). The need for a point of control and governance within the SOA deployment demands that service information artifacts be stored and managed in a consistent manner that allows enforcement of organisational policies. This is precisely the role served by a registry-repository service within an SOA deployment.

Earlier SOA deployments recognised that the rising complexity and diverse nature of service information artifacts demanded more formal means of management, sharing, and tracking than simple Web sites. This led to the use of a UDDI registry to manage, share, and track service information artifacts.

Though this is an improvement over the less formal web site approach, it has a major limitation: a registry can only store links or pointers to service information artifacts. The actual artifacts must reside outside the registry, using informal and inconsistent means such as web sites. This makes the actual artifacts ungovernable by the registry. What is missing in this scenario is a repository that stores the actual artifacts. A registry-repository provides an integrated solution which is able to store metadata such as links or pointers to artifacts, as well as the actual artifacts themselves.

In practice, SOA deployments tend to span organisational and governance boundaries. This can be true even among different parts of the same enterprise. Organisations large and small sometimes prefer to have local autonomy over their SOA deployments, but equally they also need to seamlessly integrate their services with those in SOA deployments of other organisations. For example, the HA will need to interoperate with Other Government Departments (OGD’s) through the G-Cloud as well as with partner organisations such as MAC’s and TechMAC’s, Service Providers and DBFO’s1. In order to meet the requirement of local autonomy while providing seamless integration and interoperability globally, SOA deployments must federate with other SOA deployments using open standards.

The trend towards open standards-based, integrated registry-repositories is largely because they allow organisations to share and link information with other organisations in a secure manner. Federated information management allows multiple registry-repositories to federate together and appear as a single, virtual registry-repository, while allowing individual organisations to retain local control over their own registry-repositories.

This also leads to developments in the Semantic Web and linked data culminating recently in the Government’s Open Data initiative, which employs standards such as the Web Ontology Language (OWL) and Resource Description Framework (RDF). These are catered for in the OASIS ebXML Registry and Repository 4.0 Profile for OWL-Lite (draft) standard.

1 Design, Build, Finance and Operate.

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

2.2.6 Choosing a Registry-RepositoryIn evaluating which registry-repository to deploy as part of the SOA infrastructure, the choices fall into the following categories:

1. A proprietary registry-repository.2. An ISO 11179 registry without a repository.3. An ISO 11179 registry with a proprietary repository.4. A UDDI registry without a repository.5. A UDDI registry with a proprietary repository.6. An ebXML registry-repository.7. A combination of UDDI registry and ebXML registry-repository.

As mentioned earlier, federated SOA deployments require a standards-based registry-repository. This rules out options 1, 3 and 5 above. The remaining standards-based choices involve three standards: ISO 11179, UDDI and ISO 15000 - ebXML registry and repository (regrep).

The ISO 11179 and UDDI registries offer only a subset of capabilities offered by an ebXML Registry. In particular, they provide only a registry and no repository. What gets published in an ISO 11179 or UDDI registry are pointers to service artifacts such as WSDL documents. What gets published to the ebXML Registry are not just pointers to service artifacts, but also the actual artifact themselves. Thus, an ebXML registry-repository can be used for governance of any type of service artifacts throughout their life cycles.

For these reasons, the chosen solution for this reference architecture is the ISO 15000 ebXML regrep.

2.3 Meta-modelsThe EA Team have adopted the TOGAF 9 meta-model. There are, however, a number of anomalies with this standard as far as the current manual goes. The first one pertains to capabilities, shown in Figure 5 below. The original TOGAF 9 diagram shows the “Capability” as an “Organisation”.

Figure 5 - TOGAF “Capability”

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

The other anomaly relates to the SOA extension which should appear as follows:

Figure 6 - SOA modification to TOGAF Metamodel

In the original manual the Data Entity and Technology Component are directly connected to the Business Service rather than the IS Service.

For further information on meta-models refer initially to Ref. [2]. Further considerations arise in terms of extensions such as ebRIM and OGC’s Catalogue Service for Web (CSW).

2.4 Cataloging: Key to Artifact DiscoveryThe cataloging of artifacts improves their discoverability and is essential in supporting artifact-specific queries – it’s very similar in concept to the indexing of tables in a relational database. In each case, information is automatically converted to metadata at the time it is published, and the metadata is used to facilitate efficient discovery of the published information.

Metadata discovery is the process of using automated tools to discover the semantics of a data element in data sets. This process usually ends with a set of mappings between the data source elements and a centralised metadata registry. Metadata discovery is also known as metadata scanning. Without this automation capability the task of manually managing metadata would become impossible – a never-ending job like painting the Forth Bridge!

The cataloging function also needs to support the following activities:

Implement a Managed Metadata Environment (D) Create & Maintain Metadata (O) Integrate Metadata (C) Manage Metadata Repositories (C) Distribute & Deliver Metadata (C) Support Metadata Reporting and Analysis (O)

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

3 PLATFORM SPECIFIC MODEL (PSM)

3.1 IntroductionSince metadata is such a far-reaching topic with multiple domains, this issue of the reference architecture will focus on a few key areas, as follows:

The top-level, Enterprise Metadata Management integration framework. The SOA registry-repository. The Business Intelligence domain. The EA modelling domain.

The last two have already been subject to some product selections. The remaining topics are just reference implementations and subject to confirmation via formal procurements and/or product selections with partners as appropriate.

3.2 Enterprise Metadata Management Open Integration FrameworkThe InfoLibrarian Metadata Open Integration Framework is the reference implementation for this top-level domain. It’s available both as a software product and as a hardware appliance. Subject to a value for money evaluation, the latter is considered most appropriate for the HA’s internal implementation whereas partner organisations may prefer a more limited adoption based on the software version.

Figure 7 - InfoLibrarian Framework

Framework features include:

Source-code is provided for a generic, vendor neutral, object oriented meta-model schema (the Universal MetaMart).

XML can be easily returned by querying the repository directly. The repository can be queried directly using very simple SQL commands. Component API fully documented and supports several popular languages.

_________________________________________________________________________06/05/2023 v1.0 of 22

http://www.infolibcorp.com/Software.html


_________________________________________________________________________

The meta-model schema can be deployed on virtually any RDBMS – Oracle is preferred in the HA context.

The InfoLibrarian portal is web-based and open source. Maintenance of the metadata collection is 100% automated. Hundreds of scanners and templates are available for most environments and are

completely plug-and-play.

Software features include:

Advanced search and comparison functions. Drill-Through-Impact-Analysis. Lineage analysis (visual in the portal). Semantics management. Business rules management. Business term definitions. Subject area and category modeling. Historical views. Code and object comparison. Change management. Object modeler. Document linking. Multi threaded auto-scanning. Auto-scanning command center with metrics and scanning progress indicators. Hundreds of scanners and templates available. Scanners are plug-and-play (drop in and scan). Automatic generation of graphical business views.

The InfoLibrarian Administrator is a PC based desktop application built on the robust InfoLibrarian Engine. It allows Administrators to:

Navigate metadata. Manage metadata catalogues and add descriptions. Connect to different InfoLibrarian repositories. Delete scanned metadata catalogues or data sources. Setup scanning options. Add, remove, and modify business rule objects. Build and manage subject areas, categories and mappings. Add, remove, and modify custom object types. Add, remove, modify, and categorise linked documents to appear in the portal. Manage Data Dictionary and synchronise with scanned metadata. Search metadata descriptions. Configure portal, log, and repository settings. Register scanners. View and navigate scanning logs. Run scanner wizards and autoscan interactively.

The Universal MetaMart is an open and simple schema that enables users to quickly understand and have direct access to their metadata. While sophisticated standards like OMG’s XMI, MOF, and CWM2, are supported the MetaMart provides a “Cooked” view of the metadata for administrators.

The Auto-Scanners refresh and maintain historical changes as they occur over time. This catalogues and safeguards the core knowledge about the local environment. Immediate value from the technical metadata provides technical personnel with reliable and accurate knowledge

2 OMG: Object Management Group XMI: XML Metadata Interchange MOF: Meta-Object Facility - a meta-model used to formally define Unified Modeling Language (UML) CWM: Common Warehouse Metamodel

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

to help them more effectively manage and maintain complex ICT systems and interrelated processes. The foundational metadata repository is populated this way.

The Object Modeller allows users to identify anything that needs to be documented and to define it. Additionally, users can organise it in a customised manner, like defining templates for the metadata to be managed within the enterprise. This can include:

Hardware inventory Business processes Key Performance Indicators (KPI’s) Technical processes Applications and components

There is Point and Click Category Modeller which enables users to easily capture, define and map business metadata, automatically displaying and categorising it in the portal with terminology that is already familiar to users.

The built-in visual mapper tool allows users to associate objects or defines hierarchies and categories of objects, build subject areas, and link documents. The portal automatically graphically materialises any pre-defined subject areas and provides the users with the ability to drill through the hierarchies to the finest level of detail. Subject areas are presented in the portal graphically.

The repository has been tested with the following databases:

Oracle, SQL Server DB2.

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

3.3 SOA Registry RepositoryThe WellGEO RegRep is the reference implementation for registry and repository since it has specialised extensions for the Geographical Information (GI) domain which is of significant importance to the HA. It enables seamless, secure management of GI services, datasets and other content. Its extensible repository manages all types of GI and non-GI content. Its registry manages standardised metadata describing the content within its repository or in external repositories. It is built using open standards and interfaces and integrates well with existing ICT infrastructure wherever it is deployed.

Figure 8 - Wellfleet RegRep Architecture

Features include:

GI Service Catalog based on OASIS ebXML Registry and Repository 4.0 (draft) and OGC WRS 1.0 Basic Package.

Cataloging of GI services, datasets and content using ISO 19115, ISO 19119 and ISO 19139.

Rapid discovery of GI services, datasets and content based on ISO 19115 and ISO 19139 metadata using flexible queries.

Spatial query support using GML 3.2.1. Full-text indexing and search. WSDL management based on OASIS ebXML Registry and Repository 4.0 Profile for

WSDL (draft). XML Schema management based on OASIS ebXML Registry and Repository 4.0

Profile for XML Schema (draft). Ontology management based on OASIS ebXML Registry and Repository 4.0 Profile for

OWL-Lite (draft). Registry interfaces available with REST and SOAP bindings. Relationship management between managed content. Pluggable authentication modules allowing leverage of existing identity management

and authentication solutions within the enterprise. Flexible role-based authorisation and access control based on OASIS XACML 1.0. Federation of multiple WellGEO RegRep instances to allow sharing and linking GI

information across organisational boundaries. Seamless federated search across multiple WellGEO RegRep instances even across

organisational boundaries. GUI and API interfaces provide human and programmatic access to WellGEO RegRep

services and content

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

A Publish-Subscribe workflow is illustrated below in Figure 9 below.

Figure 9 - Wellfleet Pub-Sub Workflow

A lifecycle workflow is illustrated below in Figure 10 below.

Figure 10 - Lifecycle Workflow

The product is tested on the following open source platforms:

Operating systems: Ubuntu Linux 7.10 and Red Hat Enterprise Linux 4 Java platforms: JDK 6 Application servers: Tomcat 5.5 Databases: PostgreSQL 8.3 Web browsers: Mozilla Firefox 2.0+

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

3.4 The Business Intelligence domain

3.4.1 IntroductionThe HA’s BI Programme has already selected the Oracle stack through its BI Service Enablement Project. This includes the Oracle Data Integrator (ODI) product.

Oracle Data Integrator is organised around modular repositories. All runtime and design-time components are Java components that store their information in metadata repositories. The components of this architecture can be installed and run on any platform. The architecture is shown in Figure 11 below.

Figure 11 - ODI Architecture

3.4.2 User Interfaces The five ODI graphical modules are:

Designer: In this interface, users can define declarative rules for data transformation and data integrity. Database and application metadata can be imported or defined. Designer uses metadata and rules to generate scenarios for production. All project development is performed through this interface, and it is the main user interface for developers and metadata administrators at design time.

Operator: In this interface, users can manage and monitor ODI jobs in production. It is designed for production operators and shows the execution logs with error counts, the number of rows processed, execution statistics, the actual code that is executed, etc. At design time, developers can also use Operator for debugging purposes. It is the main user interface at runtime.

Topology Manager: In this interface, users can define the physical and logical architecture of the infrastructure. Servers, schemas, and agents are registered in the ODI master repository through this interface, which is primarily used by the administrators of the infrastructure or project.

Security Manager: In this interface, administrators can manage user accounts and privileges. It can be used to give profiles and users access rights to Oracle Data Integrator objects and features. This interface is primarily used by security administrators.

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

Metadata Navigator/Lightweight Designer: Business users, developers, operators, and administrators can use their Web browser to access Metadata Navigator or Lightweight Designer. Through these Web interfaces, they can see flow maps, trace the source of all data, and even drill down to the field level to understand the transformations used to build the data. They can also launch and monitor scenarios and edit data mappings though these Web interfaces. This forms the basis of the strategic BI Data Catalogue or Data Dictionary and is separate to data modeling tools which are covered in §3.5.

3.4.3 Repositories The Oracle Data Integrator repository is composed of a master repository and several work repositories as shown in Figure 12 below. These repositories are pluggable in any relational database management system.

All objects configured, developed, or used with ODI modules are stored in one of these two types of repository.

There is usually only one master repository, which contains security information (user data and privileges), topology information (definitions of technologies and servers), and versions of objects.

Figure 12 - Structure of ODI Repositories

The work repository is where projects are stored. Several work repositories may coexist in a single ODI installation. This is useful for maintaining separate environments or to reflect a particular versioning lifecycle, such as development, user acceptance tests, and production environments.

A work repository stores information related to:

Models, including datastores, columns, data integrity rules, cross references, and data lineage3.

3 Data lineage is a foundation capability of metadata management which provides the functionality to determine where data comes from, how it is transformed, and where it is going. Data Lineage metadata traces the lifecycle of information between systems, including the operations that are performed upon the data.

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

Projects, including interfaces, packages, procedures, folders, knowledge modules, and variables.

Runtime information, including scenarios, scheduling information, and logs.

3.4.4 ODI User RolesAs Oracle Data Integrator relies on a centralised repository, different types of users need to access it. The list below describes how ODI is used within a BI/DW team based on best practice.

BI Business User

Business users have access to the final calculated key indicators through reports or ad-hoc queries. In some cases, they need to understand what the definition of the indicators is, how they are calculated and when they were updated. Alternatively, they need to be aware of any data quality issue regarding the accuracy of their indicators.

BI Business Analyst

Business Analysts define key indicators. They know the source applications and specify business rules to transform source data into meaningful target indicators. They are in charge of maintaining translation data from operational semantics to the unified data warehouse semantic.

BI Developer

Developers implement the business rules in respect of the specifications described by the Business Analysts. They release their work by providing executable scenarios to the production team. Developers must have both technical skills regarding the infrastructure and business knowledge of the source applications.

BI/DW Metadata Administrator ODI Metadata Administrators reverse engineer source and target applications. They guarantee the overall consistency of Metadata in the ODI Repository. They need to have an excellent knowledge of the structure of the sources and targets and they should have participated in the data modelling of key indicators. In conjunction with Business Analysts, they enrich the metadata by adding comments, descriptions and even integrity rules (such as constraints). Metadata Administrators are usually responsible for version management.

BI/DW Database Administrator

Database Administrators are in charge of defining the technical database infrastructure that supports ODI. They create the database profiles to let ODI access the data. They create separate schemas and databases to store the Staging Areas. They make the environments accessible by describing them in the Topology.

BI/DW System Administrator

System Administrators are responsible for maintaining technical resources and infrastructure for the project. For example, they may:

Install and monitor Scheduler Agents. Backup and restore Repositories. Install and monitor Metadata Navigator. Setup environments (development, test, maintenance etc.).

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

BI/DW Security Administrator

The Security Administrator is responsible for defining the security policy for the ODI Repository. He or she creates ODI users and grants them rights on models, projects and contexts.

BI/DW Service Operator

Operators are responsible for importing released and tested scenarios into the production environment. They schedule the execution of these scenarios. They monitor execution logs and restart failed sessions when needed.

ODI comes with the following standard user profiles:

Role User ProfileBusiness User CONNECT, NG REPOSITORY EXPLORERBusiness Analyst CONNECT, NG REPOSITORY EXPLORER, NG DESIGNERDeveloper CONNECT, DESIGNER, METADATA BASICMetadata Administrator CONNECT, METADATA ADMIN, VERSION ADMINDatabase Administrator CONNECT, DESIGNER, METADATA ADMIN, TOPOLOGY

ADMINSystem Administrator CONNECT, OPERATORSecurity Administrator CONNECT, SECURITY ADMINOperator CONNECT, OPERATOR

3.5 The EA modelling domain

The EA Team has selected iServer Enterprise Architect from Orbus Software as the standard EA modelling tool. The server is hosted externally and is accessible on the same basis as PartnerNET so that external consultants and Service Providers can access it when necessary.

iServer is a multi-user repository and modelling environment for Microsoft Office and Visio documents, thus making it accessible to a wide range of users. iServer's central repository stores both modelling information (diagrams and shapes) and MS Office(Word, Excel and PowerPoint) documents in a single EA store. Modelling objects such as services, applications, interfaces, data elements and activities can be directly related to textual information such as principles, policies, goals or objectives stored in Word documents. Deliverables created in Powerpoint, Excel and Word can also be created, stored, versioned and managed as individual artifacts (objects) in the same environment allowing for easier control and publication (dissemination) to stakeholders.

iServer allows users to choose a meta-model suitable for their EA approach and maturity. By using iServer's pre-packaged meta-models for key methods, such as TOGAF, EA initiatives can be up and running immediately, but by making use of iServer's extensible environment the EA Team can customise or build their own as required. Customisation includes choosing (and or building) the artifacts that may exist, their meta-definitions and the relationship types between them - all elements are easily customisable.

iServer provides out-of-the-box notations for Enterprise Architecture including TOGAF 9, but can also support any Visio stencils/notations and document templates.

Stakeholder needs can be met in a number of ways: targeted HTML publications, EA views, analysis information (matrices, comparisons), documents, spreadsheets and reports can be generated instantly from iServer's repository helping to ensure the dissemination and communication of EA content to the wider business audience. Traceability means that EA artifacts and proposed solutions can be traced back to stakeholder concerns and requirements.

_________________________________________________________________________06/05/2023 v1.0 of 22


_________________________________________________________________________

Organisations often use a variety of tools to describe their Enterprise Architecture. The EA Team has number of models from the BI and Technology Convergence Programmes in IBM/Telelogic System Architect. Similarly, the ITS Metadata Registry receives models in a number of formats, e.g. Datex II in Sparx Systems’ Enterprise Architect and UTMC UML in No Magic’s MagicDraw. Therefore, a key EA tool requirement is the ability to seamlessly transfer information between these different data sources. iServer Data Exchange provides a variety of import and synchronisation mechanisms, from a simple artifact Excel import to a daily synchronisation of artifacts stored in another tool such as InfoLibrarian.

3.6 Summary and Overview

The overall architecture is shown in Figure 13 below.

Figure 13 - Metadata Mangement Architecture

Eventually the building blocks can be connected through the Enterprise Service Bus (ESB) but in the interim they may be connected conventionally using standard network protocols.

_________________________________________________________________________06/05/2023 v1.0 of 22

reference architecture - highways englandassets.highways.gov.uk/specialist-informa… · web...

Documents