- application semantics: the starting point for adopting soa.doc

Application Semantics:

The Starting Point for Adopting SOA

Neil RadenPrincipal Analyst

© 2005-2006, Hired Brains, Inc. Email: [email protected] rights reserved. No portion of this report may be reproduced or stored in any form without prior written permission

mailto:[email protected]

Hired Brains Research



ABOUT THE AUTHOR

Neil Raden is the founder of Hired Brains. Hired Brains provides consulting, systems integration and implementation services in Business Intelligence, Information Management, Advanced Analytics and Semantic Technology for clients worldwide. Hired Brains Research provides consulting, market research, product marketing and advisory services to the software industry. Based in Santa Barbara, CA, Raden is an active consultant and widely published author and speaker. He welcomes your comments at [email protected]. Hired Brains’ research is available at http://www.hiredbrains.com/knowout.html without registration.



http://www.hiredbrains.com/knowout.html


TABLE OF CONTENTS

EXECUTIVE SUMMARY.......................................................................................1

SOA IN BRIEF.......................................................................................................4

STARTING WITH SOA.........................................................................................5

WHAT IS LACKING IN SOA?................................................................................6

SEMANTIC SOLUTIONS......................................................................................8

A SEMANTIC SOLUTION...................................................................................10

CONCLUSION....................................................................................................11



EXECUTIVE SUMMARY

Service-Oriented Architecture (SOA) is not The Next Big Thing; it already is The Big Thing. Service-oriented architectures have been around for a while, but with all of the major technology vendors embracing standards such as Web Services, SOA has arrived in record time. On the surface, SOA is a relatively easy concept to grasp and the linkage to its promised benefits is plausible. Even lacking a firmer grasp on the implementation issues, initial steps are being taken without a deep understanding of the discipline. For example, a recent study by The Yankee Group reveals that 76% of CIOs said they will make an SOA investment this year.

IT budgets for new initiatives are under greater scrutiny than ever. In such an environment, a CIO has to make a pretty solid case for something so new, and that case has to involve either cost reduction guarantees or enablement of new revenue-generating opportunities. The most commonly-cited benefits for implementing an SOA are a mixture of efficiencies, capabilities and compliance:

Reduction in integration expense: Because SOA works at a higher level of abstraction, it can reduce or even eliminate point-to-point application integration. This includes the functionality of Enterprise Application Integration (EAI) and custom-built point-to-point integration. The latter is inordinately expensive to build and doubly expensive to maintain over time. Features of SOA can replace these legacy solutions.

Increase asset reuse: At the core of Web Services is the ability to publish and research available services, no matter where they are and what technology was used to develop them. This can help reduce duplication and redundancy, and aid in decommissioning obsolete resources

Increase business agility: In a SOA, applications can be composed of services that exist across a network, so-called composite applications. These can be put together without programming. Business experts - or as they are now more politely called, domain experts - can assemble applications instead of programmers. Configuring and reconfiguring business processes can take place in real time.

Reduce business risk: With a single set of rules, compliance and alignment, service creation and adoption policy are much simpler.

These promised benefits are all reasonable, but they are not automatic. SOA is just a concept, an application architecture. Web Services are actual technology specifications, but even the combination of SOA and Web Services is just a starting point. There are a lot of steps and tools needed to make it all work. CIOs are skeptical enough to see this.



Despite obvious interest, the same study by The Yankee Group shows that 44% of 473 responding CIOs said their lack of understanding of Web Services and loosely coupled1 architectures were two inhibitors in adopting an SOA. These concerns may be only the tip of the iceberg, and overlook some larger, lurking problems that go well beyond just being temporarily uninformed about the technology pieces. SOA is not an out-of-the-box solution. It requires assembling an overall strategy, implementation of new standards, layers of middleware, training, a radical rethinking of IT management, security, performance monitoring and optimization, and more.

The promise of SOA is heightened agility powered by reuse and dynamic assembly of business processes, a necessary element for competing in faster, more open markets. SOA gives form and substance to this promise through interoperability and communication of services. However, applications are not part of the picture. The functioning components of services which are, in turn, components of applications are not part of the SOA or Web Services specification.

Existing application portfolios potentially harbor hundreds or even thousands of services. Many of these may become the building blocks of a new set of composed applications. Many more will be redundant while others may no longer be useful at all. Because it's unlikely that anyone will scrap their entire application portfolio and start over at the services level, there has to be a methodology or a road map to follow. How would you get started? How would you know what level to expose, what becomes a service, what is an application? Before the first set of services are built and/or exposed in an SOA, you have to know what you have in the first place. You need an inventory, not just of the applications themselves, but the semantics of the applications – what they mean, what they do and how they relate to other resources.

Making decisions about what to expose and what to build assumes there is an inventory to work with in the first place, and that there is a convenient way to spot redundancies and measure dependencies between units of code or data. Documentation for your existing programs will have gaps, perhaps huge gaps. It may not even be correct or decipherable, not to mention being too cumbersome to read cover-to-cover. Even knowing where documentation exists is a problem.

The same is true with source code libraries and versions of code – was the current executable compiled from the most current source? Even reading the source code is out of the question. Clearly, the most important step for getting started with SOA is finding a way to catalog everything you have, and being able to query the catalog in limitless ways. Otherwise, it would be impossible to leverage and extend your assets for a completely new application architecture. The upside of this effort is that, going forward, maintenance will be streamlined.

1 Loosely coupled is a feature of SOA - components are not hard-wired but can be stitched together on the fly into "applications" or business processes



This is because a majority of maintenance costs will be spent researching the impact of proposed changes and testing them to be sure that nothing has been disturbed. Having a complete catalog of all of the information resources and detailed understanding of the all of the relationships and dependencies could take a pretty big bite out of the maintenance budget, which averages 75% of the total IT budget in FORTUNE 500 companies.2

Suppose creating this active catalog were possible and you’ve finished its initial creation. Once you've cleared that hurdle, you have to be able to continuously update it and be able to monitor it for changes.

The good news is that you don't have to understand everything. It's possible for machines to expose all of the "resources" in your portfolio, and their relationships, by reading the source code, metadata, documentation and other artifacts and assembling a comprehensive framework of meaning. These could use semantic technology without any prior understanding, including tools to "introspect" or mine that framework with conceptual search, such as, “Do we have a service that composes a report in .pdf format?” In addition, the discovery process can also highlight those instances where references or dependencies exist, but the code or documentation is not available. In other words, it even knows what it doesn’t know.

Such a technology exists. It is crucial for starting SOA development, but because it is a tool for initially understanding what you already have from a practical standpoint, it allows you to ramp up SOA activities from the maintenance budget.

2 Forrester



SOA IN BRIEF

Why has SOA garnered so much attention lately? What does it offer IT organizations already numb with wave after wave of innovations? What it offers is the solution to a problem that is as old as “data processing” itself; a problem that drains IT budgets while adding little to no benefit – custom interfaces.

From the time the first IT department wrote its second program, there was a need for programs to communicate with each other. Since programmers tend to regard programs written by other programmers as, at best, inadequate, organizations often find themselves saddled with an inventory of incompatible systems, each with its own way of communicating with other programs. As technology platforms change, the programs become not only logically incompatible, they are physically incapable of communicating without developing yet another program. When organizations acquire other organizations, the problem multiplies. When the Internet opened up business to the outside world, the problem multiplied geometrically. IT departments spend an inordinate amount of time building and maintaining one-to-one interfaces.

To deal with this rapidly worsening problem, a series of evolving practices emerged that aimed to solve the problem of custom interfaces, such as integration brokers and application interfaces (EAI). By incorporating into the design of new programs, or “wrapping” existing ones, attempts were made to standardize the way programs interact across a network by transforming standalone “programs” into “services”. These could be contacted by other services to provide some portion of functionality on an as needed basis, eliminating the need for custom interfaces. This approach was, obviously, a service-oriented one.

Why SOA is suddenly so hot, though, is not because the world woke up to a good concept. Web Services, a set of technology standards that power SOA, make widespread SOA possible by removing it from a set of competing proprietary standards and, instead, placing it in the hands of independent standards bodies. For the first time, all of the major technology vendors have lined up behind an open standard that provides platform independence and a means for service providers and consumers to find each other and communicate with non-proprietary standards. It also provides complete separation between the provider and the consumer of the service. The magnitude of this event cannot be overstated. All of the possibilities of the Internet before the bubble burst are small compared to the latent value of SOA. It is revolutionary. It is not an exaggeration to say that nothing like this has ever happened in the information technology field. It promises to open up computing once and for all.

But SOA is not technology, it is a strategy for providing decentralized computing services. Web Services is the technology that powers SOA because it is an open



standard, owned by everyone and owned by no one. SOA is an applications architecture powered by a set of standards.

At a functional level, SOA with Web Services provides the ability to:

Locate a service on any kind of platform, anywhere Invoke a service on any kind of platform with permissions Handle versions of services Identify users of services Define policy and enforce policy

The benefits of SOA are promising, though still a little unproven, but from a conceptual perspective, there is little doubt that the SOA wave will have a tremendous, positive impact on IT. The benefits include:

A rational approach to achieving real-time business processes Enabling customers, partners, employees, suppliers and a host of other

actors to participate in your business processes without a custom link The ability to disintermediate (an overused term from the pre-bubble days)

businesses, such as the way Amazon.com has exposed a number of key services allowing anyone to easily operate an Amazon business from their own site

Low cost for connection makes it possible to do business with thousands of small businesses efficiently

Creative minds will conceive of thousands of other ideas to put SOA to work. The only trick is figuring out how to get started without “boiling the ocean.”

STARTING WITH SOA

SOA combined with Web Services only provides a means for services to communicate, including the attendant facilities for discovery, description, etc. But SOA is mute on the subject of services themselves. Because the concept behind SOA is “loosely coupled” services, meaning it doesn’t matter how they were developed or what platform they use, SOA does not address how to actually build or deploy them. What is the proper level of service? Is it more expensive to invoke many small services or a few large ones?

What will it take to implement SOA? If you’re a brand-new company designing all of your systems from scratch (highly unlikely), or you’re scrapping 100% of your portfolio and starting over, not much. Just chose a systems architecture and adhere to some standards for Web Services, and you’re ready to go. Unfortunately, no one starts from scratch. Germinating a new set of services to conform to a new set of standards and architecture is the easy part. Enabling your existing portfolio of applications to participate in SOA is far more



complicated. Even if you knew where all the applications were and what they did, orchestrating them into coherent services would be difficult enough. But the truth is that no one really understands the totality of their application portfolio.

Our current research reveals that the major systems integration firms (including those captive professional services of hardware/software vendors) are proposing SOA projects equal in scale to ERP implementations of five to ten years ago – years of duration and many tens of millions of dollars, to start. The bulk of this effort is not new development, but rather, identification and cataloguing of existing portfolios of application software and exhaustive “tagging” of the “services” discovered. This is in line with recent research3 that reveals that 76% of IT spending in FORTUNE 500 companies is currently allocated to maintenance, not development, and the portion continues to grow. Existing portfolios are so diverse in terms of provenance, technology and function, that even small enhancements or fixes require three to four times as much effort to just locate and map out the impact of the changes as they do to actually develop and test them. Cutting these maintenance expenses for most organizations is like losing weight – everyone agrees it’s a good idea, but no one knows how to get started.

WHAT IS LACKING IN SOA?

What other pieces of the puzzle have to come together for SOA to be useful?

Existing Web Services standards, such as UDDI, SOAP and WSDL4, were developed for the relatively lightweight mission of Web applications. But like Victor Hugo said, “The most powerful force in the world is an idea whose time has come,” those with foresight looked at Web Services and saw a diamond in the rough. Struggling with integration issues, overwhelmed by maintenance costs and trying to keep up with each new wave of technology, Web Services appeared to offer a feasible solution. But existing legacy applications within organizations, as well as analytical and reporting tasks, are among the most complex and demanding computing applications, far more complicated than most straightforward B2B applications. In fact, the more central an application is to a business, the more likely it is a good candidate for Web Services. The applications that face external parties, such as customers, suppliers or partners are not only well-suited to an SOA architecture, they are often custom built (versus packaged applications), and the most highly dynamic in response to constantly changing business processes and conditions. Looking below the surface of this hopeful development of SOA and Web Services are major hurdles that have to be resolved.

3 Forrester4 UDDI – Universal Description, Discovery, Integration; SOAP – Simple Object Access Protocol; WSDL – Web Service Definition Language; for detailed information, see http://www.w3.org ;



http://www.w3.org/

If a requester and a provider of a service do not use the same semantics in their service description, there has to be a mediator. The existing standards for exposing and describing services, such as UDDI and WSDL are quite thin (The standards bodies are not standing still on this issue. Some progress is being made to add richer semantics to Web Services, but it will be some time before this is complete, and part of the wide acceptance of Web Services is their rather Spartan nature. Encumbering them with more features may not work over time.) Pulling all of the pieces together requires service brokers, orchestration agents and ever higher-level management tools for design (such as Business Process Management tools). The level of effort required to catalog the exposed resources to all of these consumers will overwhelm early efforts. Existing metadata approaches, using a relational repository, won’t suffice. Another often overlooked corner of resources are those that are spread around the organization in the form of personal applications. Tackling the problem of understanding all of the spreadsheets5 and other personal software, departmental applications and unsupported tools is currently beyond the capacity of any organization. Assuming that it was possible, how long would it be before it was out of date?

The same is true of data. Data is “owned” by applications, or at least, embodies its owner’s semantics, and services that are derived from existing or modified applications will inherit this ownership of data. SOA is not an off-the-shelf answer and Web Services and composite application components are not a solution to the information integration dilemma. The driving forces behind these emerging standards are to leverage the Internet for interoperating application interfaces and protocols, not data. SOA and Web Services, even the systems architectures being rolled out for SOA from vendors such as IBM®, SAP, Oracle® or Microsoft®, will not support these requirements, at least not with the basic building blocks.

For SOA to work at every point in time, an organization has to have a complete inventory of all of the resources in its portfolio: programs, data, data models, documentation, component and object libraries and applications. Before building or wrapping the first SOA-enabled service, IT has to be able to find what it has available and how everything relates to everything else. This requires much more than the usual flimsy metadata and documentation available, it has to be based precisely on what actually exists. SOA methodologies discuss the process of promote, bind and manage, but exposing services is still the tricky part. What is needed is not only an inventory of everything that exists - including inactive, mothballed and incomplete pieces, copies and replicates - what is really needed is a way to continuously catalog these resources and to provide a means to leverage the catalog by embedding intelligence into it. The promising field of semantic technology is the key to taking this step.

5 From personal experience, a very large (> $15 billion) multinational organization calculated consolidated operating cash flow from a spreadsheet. When the owner of the spreadsheet was made “redundant” in a wave of restructuring, it was discovered that the spreadsheet was missing. These “spreadsheet nightmares” are all too common.



SEMANTIC SOLUTIONS

Semantic representation differs from descriptions or definitions in that they capture the relationships between things. Using a trivial analogy, consider a set of Tinker Toys® without the connectors. A metadata repository is useful as a reference tool, but in order to determine what resources are dependent on other resources, or the context in which they interact, a metadata repository is just a database. The sticks are all there, but the connectors exist only outside the metadata – in the knowledge of the staff that constructs queries.

A semantic map, on the other hand, is a structure that allows computer programs to actually reason, to draw inferences about things, such as a parts catalog that knows to reject an order without needing a business rule, stored procedure or application program. The semantic map stores the sticks and the connectors. In addition, unlike metadata in a relational database, a semantic map is a run-time system. Because of their graph-based structure, they are dynamic and capable of actually expanding their knowledge as they work by finding implicit knowledge within its own structure.

Semantic map is a generic term. One kind of semantic map is an ontology, which is constructed according to standards from the W3C Group, the same organization that is responsible for Web Services. Ontologies are built with XML-variants RDF (Resource Description Framework) and OWL (Web Ontology Language). Ontology is a very interesting and rich subject beyond the scope of this paper and is based on the mathematical principles of Graph Theory, which is very different from Relational Theory, which is based on Set Theory. The most important thing to know about Graph Theory is that it is very useful for describing the relationships between things without burning the relationship into the circuitry. If a term is used in varying or even conflicting ways, for example, an ontology can capture all of the contexts rather than being forced to accept just one, and to apply the different variants consistently.

Relational schema is a representation of a model; an ontology is a model. A relational schema is passive; it is not capable of analyzing itself. An RDF model is a run time model, as it can introspect itself and produce new information that is not given to it directly. It is possible, perhaps, to develop a relational schema of an RDF model, but it would lack inferencing and could not be updated directly

A metadata repository can be queried if you know what you’re looking for, but a semantic map can be presented with conceptual questions, and resolve them. In fact, the context that provides meaning to the semantic map can be generated dynamically, in real-time, from the question posed to it. Consider, for example locating all instances of logic and data in your application portfolio that either store, update, view, add, modify, or delete a social security number. Semantics allows you to “find” all instances, regardless of how they are named (SSN, SS_NBR, SOC_SEC_NBR), and view all of the inter-dependencies. This is



simply not possible with a metadata repository unless you know specifically what you are looking for (by name) and where to look. It still may not be possible to understand the relationships unless they have been previously defined

Here is a practical example. Suppose your organization owns thousands of application programs, database tables, stored procedures and an uncountable number of spreadsheet files. Suppose one of your customers has fallen under the regulatory control of a legal jurisdiction as a result of an acquisition and you are given 30 days to find and remediate all references to, say, age, in your programs or your customer will no longer be able to do business with you. How would you go about, first, finding all of these references and then diagnosing the impact of removing them? Is there a team available that can review existing source code in, say, AS/400 RPG, OS/390 Cobol, a few Linux-based Java applications, databases in Oracle, DB2, Sybase IQ and Access®? Even if this team could be assembled, how would they go about their investigations? How would they represent their findings? Would they be able to identify dependencies or other relationships between them? Going forward, how would they monitor compliance to the new policy?

One thing learned in fifteen years of data warehousing is that integrating just disparate data is very difficult. But data warehousing has the relatively easy task of integrating very structured data from common business subject areas. Understanding the totality of your application portfolio is orders of increasingly difficult magnitude, best left to intelligent programs and clever algorithms. In fact, even in the relatively simple domain of data warehousing, metadata is still sporadically implemented and not designed for introspection or discovery. Enterprise metadata repositories, to the extent they exist, capture information about data but not about applications.

Something as complicated as an entire application portfolio, with all of the interdependencies, needs the capabilities of a semantic map and a conceptual search engine, which we’ll call “application semantics.” The semantics of the applications are expressed through semantic maps that allow enterprise architects to expose services (or potential services) based on enterprise-specific criteria.

The distinction between metadata repositories, as we know them, and semantic maps, is top down versus bottom up6. A metadata repository is top down, a “boiling the ocean” approach. First, find and describe everything there is. Then, query it based on what you already know. In a relational database, it is possible to create a schema that is completely cryptic. The names of tables and attributes need not, and often do not, impart any knowledge about the contents of the relationships of the entities. The knowledge in relational databases is in the mind of the query builder and as such, exists externally to the database. In an

6 Actually, semantic maps can be built in a top-down manner too, but Metallect’s IQ Server works strictly in a bottom-up fashion.



ontology, the knowledge is contained in the structure and posing a query is much easier. Application semantics can be built bottom-up, learning everything you need to know, building the framework by applying semantic meaning to what it doesn’t already understand through the context of your questions.

Tabular listings, even in relational tables, are not sufficient for solving this problem. Building a data model for a relational database requires knowing in advance everything that will be in the model. There is no discovery. A semantic solution captures all of the necessary relationships and provides a mechanism for inference that is lacking in standard metadata approaches. The Metallect solution does this by building the application semantics by itself by rapidly scanning all of the accessible resources.

A SEMANTIC SOLUTION

Metallect (http://www.metallect.com) is a unique semantic search-based solution that generates and maintains an up-to-date architectural system of record capable of locating and exposing potential services from within an enterprise’s existing application portfolio. Other semantic approaches require a manual effort of research and populating these repositories that can take months or years. This is a major differentiator in the Metallect solution.

The Metallect IQ Server software scans source code, database schemas, metadata repositories, and documentation and creates and populates an RDF metamodel providing real-time insights into the inner-workings of an organization’s enterprise applications. A simple browser-based semantic search interface provides visibility into existing software artifacts (resources) and their upstream and downstream dependencies, as well as the potential impact of changing the artifacts within and across the application infrastructure.

The Metallect IQ Server allows enterprise architects to identify, expose, and promote existing software artifacts to leverage and extend existing applications as part of a Service Oriented Architecture.

The Metallect IQ Server:

Discovers applications, databases, integration middleware, metadata repositories, and documentation to generate an RDF meta model of enterprise applications.

Generates an “as is” semantic map to provide aggregated views of software artifacts (resources) and dependencies (relationships) that exist between them.

“Semantic Search” traverses and semantically reconciles the metamodel to find, understand, leverage and extend software artifacts.

Integrated “Views” including audit, dependency-mapping and impact analysis reduce risk of unauthorized programming changes, runaway



http://www.metallect.com/

projects, and downtime associated with rolling out new versions of applications.

Promotes metadata regarding software artifacts (resources) to registries, repositories, and composite application frameworks to leverage and extend existing applications into digital business architecture.

Periodically detects changes in application logic and/or underlying databases and incrementally discovers and records those changes within the RDF metamodel.

The ramp-up time for semantic applications is a drawback for the technology. For large enterprises with thousands of application programs, the initial payback from a semantic mapping project may not be obvious for years. Discovering semantics instead of cataloging them is a dramatic innovation that can reduce payback time to months or even weeks. It is beyond dispute that ontology-based systems offer superior capabilities over existing methods, but getting there is an issue. Semantic search (and query) is a promising alternative.

CONCLUSION

Semantic technology is still a little mysterious, and most of the material written about it is too academic to be accessible to the layman. Using terms like ontology and directed graph theory will not win converts in the executive suite. But SOA is here to stay, and anyone starting out with it is going to quickly run into a buzz saw if they don’t get their arms around their technology portfolio. Semantic techniques are the best hope.

It isn’t necessary to become an expert in semantic theory and tools, but making the decision about it comes down to two questions. Do you want to describe and catalog everything before you can get a shred of value from it, and have to maintain it by hand on a continuous basis, or would you rather rely on a piece of technology that is designed to take this task off your hands? The other question you want to ask yourself is, “Is an SUV a passenger car or a truck?”7 When you know how to answer to that question, you will understand why semantics are so important.

SOA is the future for application architecture and the starting point for SOA is generating a catalog of what you have. Cataloging and exposing your existing semantics is the only hope there is for not being overwhelmed by the process of moving to SOA, short of starting over. Metallect’s IQ Server is a key component

7 The answer is “both.” A 5,000 pound vehicle equipped with leather seating for seven, a DVD system and two air conditioners is clearly a passenger car. However, US regulations for fuel consumption demand that the “average” fuel economy for an automaker’s fleet of passenger cars remain below a certain level, hence the re-categorization of SUVs as “light trucks,” which are exempt for the regulations. An SUV is a passenger car and it is also a light truck. In an ontological world, it is reasonable to be both. How and where a specific category is used is based in context.



of enterprise application infrastructure that addresses the most pressing needs of today’s CIOs, while laying the foundation for SOA.



- application semantics: the starting point for adopting soa.doc

Documents