Systems and Data Integration

Susan KelleyYale UniversityJanuary 2015

Introduction

A lot of interest in – Service Oriented Architecture– Enterprise Service Bus

From a lot of people, for a very long time

Software solutions in varying degrees of use:Web Methods, Informatica, Lombardi (now IBM)

The Beginning

• 2013, IT Strategic Plan (Integration):– Develop and execute an enterprise-wide data and system integration strategy

enabling all Yale constituents (students, faculty, researchers, and staff) to safely, accurately, and securely access and transmit critical data across all environments and technology platforms according to their schedule and needs.

– Achieve high quality and repeatable results– Provide broad access to data across the Yale Community– Ensure data is accessed in a secure, controlled and consistent manner– Enable data stewards / data owners to manage and control data to ensure

appropriate level of access, quality, and accuracy– Employ the appropriate level of technology and data controls that provide

guaranteed and secure results while maintaining a user friendly approach across all platforms (web, cloud, mobile)

Progress

1. Director, Systems and Data Integration2. Workday, opportunity to re-do many integrations3. CIO, very interested in API management

(Princeton, Berkeley, Washington)

Exostructure The future belongs to the exostructure… The exostructure concept is about building an "exoskeleton" of services that support the institution from the outside rather than from the inside…In a competitive world, the winner in the long run is not the strongest but the most adaptable. “Gaining Competitive Advantage in the Education Ecosystem Requires Going Beyond Mere Infrastructure to Exostructure” June 2013

Progress

Systems and Data Integration“Rationale, Scope, Vision”--central Integration Competency Center--Integration Community of Practice--training for all app teams on new tools

Progress

Progress

Approach–Review and articulate the current state – Fully enumerate our current environment with some historical context to level set everyone and ensure that our future state accounts for all currently active interfaces.–Review drivers to re-analyze EAI Platform now – ensure we identify the forces driving change now which much be addressed in our solution–Define near term requirements – For purposes of this project, we are setting a maximum time horizon of three years out. Desirable features where we do not reasonably anticipate a requirement in the next 3 years will be stated and listed, but will not influence the platform selection–Gather information about 3rd party EAI platform offerings, platform extensions, and platform toolkits.–Identify platform options – identify viable components and group them into platform options where an option may include one or several vendor products plus customizations to satisfy the platform requirements. Winnow the options to a representative set with meaningful differences and characterize those differences –Recommendation – make forward-looking predictions around the cost and risk associated with meeting the near term requirements with each platform option. Propose the best option or identify the best course forward.

Preliminary/Straw Man Reference Architecture

Platform Reference Architecture

Files

Web Service Clients

Integration Server

File Processing

FilesFiles

Files

FilesFiles

Web Service ClientsWeb Service Clients

Yale DB Servers

University Application Integrations

SOAP

DB Connections

Data Integrations Data Management

Business Process Management

Managed File TransferSe

curi

ty

University Integration Services

SOA Gateway

Transformation Engine

RESTful

SOAP

MQ

Yale Application

Servers

Yale Application

Servers

University Application

Servers

SOAP

RESTful

ETL Engine Data Management

Data Management

DB Connections

Managed AccessSFTP Server

Identity and Access

Management

Certificate Store

BPM Engine Business Rules Engine

MQ

SOAP

Message Bus

Integration Platform Analysis – 1 of 4Fuse / Service Mix Dell - Boomi IBM – Cast Iron

Software AG - WebMethods

Integration Platform Analysis – 2 of 4

1. What is the best direction for eInvoicing?– Stay in WebMethods for now – lowest cost and risk– Outsource – a project to do and then we pay more forever– Convert to new platform – a project to deliver no benefit

2. Then what is the best path for ~450 other existing WebMethods interfaces?– Run as is until changes required – near zero cost, lowest risk– Convert to new platform – a project to deliver no benefit

3. Will adding another main integration execution product put us on a path to a likely better future than consolidating on WebMethods?

1. Cast Iron – not multi-tenant, serious functionality gaps and vendor questions, unlikely to help meet aspirational goals.

2. Boomi – multi-tenant, quicker than WebMethods for naïve build of simple interfaces, but closed and thus a non-starter to extend/improve to meet both current requirements and aspirational goals.

3. Fuse/Service Mix – open and elegant, but it is a toolkit with which one can build a platform , and would require a significant create/build effort to custom-provide what others have out of the box and vendor-supported.

Integration Platform Analysis – 3 of 4

• Is WebMethods “as is” and alone our platform?– Existing WebMethods Interfaces support + Base Product Features/Capabilities– Resiliency

• Single point of failure, restoration timeframe (4-6 hours)• Connection failure and recovery service

– Cost and Risk – Use Cases– Ease of Interface Development/Reuse/Design

• Functional Interface development and testing requires tech ability analogous with being able to code an excel macro

• Train 1 day to build simple interface in 1 hour (done, quickly achievable, theoretically doable)• Logical Interface/Data Modeling and Reporting capability (none, very limited, partial EDM, full

data modeling suite)• Open Architecture - Extensibility of Core to make interfaces faster to develop and higher quality

(narrow deep and fast channel construction)• Extensibility of Core for custom purposes (plug-ins)• Reusable Services Repository

– Portability of Functional Design and Logic– Extensibility of Platform

Integration Platform Analysis – 4 of 4

• Integration Platform Build

SOA Gateway

Detour

Detour with Contivo

From contivo.com:

Data Integration, Simplified with Contivo Data mapping is a costly, tedious process mired with human error and frustration. Enter Contivo —a proven design-time semantic integration technology that helps reduce the risk, time, cost and effort of integration projects through the use of a centralized, semantic metadata repository. Experience a Forward-Thinking Data Transformation SolutionContivo enables users to capture, manage, organize and leverage all of the integration metadata in an enterprise. By associating the metadata with a semantic "dictionary", Contivo establishes an integration vocabulary and thesaurus. Contivo then leverages the vocabulary and thesaurus to automate data transformation and reconciliation tasks that are traditionally implemented using manual mapping techniques.

Data Governance

Yale Yhuts Down Yale Bluebook+BY VIVIAN WANG Yale Daily News STAFF REPORTERTuesday, January 14, 2014 As the first day of shopping period opened at Yale College, the 1,418 students who had entered their tentative course schedules on Yale Bluebook+ found the website blocked.The application, designed by brothers and co-developers Peter Xu ’14 and Harry Yu ’14, used data from Yale’s course information database to offer students a convenient way to compare class evaluations and ratings, Xu said. Though the program has been available for students’ use for the last three semesters, it was only last Wednesday that the brothers were approached about the website. University Registrar Gabriel Olszewski sent them an email citing concerns that the website was “making YC course evaluation available to many who are not authorized to view this information,” asking how they obtained the information, who gave them permission to use it and where the information is hosted. In subsequent exchanges Olszewski raised concerns over the website’s unauthorized use of the Yale logo and the words “Yale” and “Bluebook,” the prominence of class and professor ratings, the application’s accessibility to non-undergraduates, and the fact that it wasn’t hosted on Yale’s servers. When the brothers met with Olszewski two days after the first email, they said they were told that the website had to be shut down.

Workday Integration Strategy

Integration End Game

Layer 7:•Web service proxy server•Provides a single entry point for synchronous RESTful and WSDL-based web services•Provides policy-based access control•Provides performance control (I/O or per-call throttling etc...)•Provides capture of use metrics

Talend Data Integration (DI):•Provides simplified ETL functionality•Replaces the vast bulk of Yale's functionality provided by (the much more expensive) Informatica product•Works well with Talend's MDM product if we decide we have a need for master data management

Web Methods:•Measured march, Workday Finance rollout July 2016

Systems and Data Integration Team

Merged Identity and Access Management with Systems and Data Integration– Developers– Platform Administrators– 1 stop for application developers to plan for

integrations and identity data

Background

Yale Data API Service•Being socialized by our Web team for reuse of data on websites•Ready for release as soon as Provost approves

Data Governance•Quietly beginning to get some traction

Workday Integrations•On track for Spring go-live HR/Payroll•fewer than 15 using ICC platform