xml in the supply chain · xml in the supply chain ... • semantic differences must be negotiated...

XML in the Supply Chain

A practical demonstration of how Suppliers can use their existing documents to generate Valid XML Messages

Release Draft: 1/22/03

White Paper

Table of Contents

I. Introduction 3

II. Exploring the Myths 4

III. A Practical Alternative 5

IV. Example Definition 7RosettaNet Notification of Invoice– 3C3 8Business Process Definition 8PIP Purpose 9

V. Step by Step 10Step One: Understanding the PIP 10Step Two: Extracting from the Invoice 11Step Three: Defining datasets 12Step Four: Transforming Data 13Step Five: Mapping Data Elements 14Step Six: Mapping Static Elements 15Step Seven: Generating the XML 16Summary of Actual Time and Costs 16

VI. Conclusion 17

V. About Datawatch Corporation 18

Confidential and Proprietary of 18

I. Introduction

The past year has seen a huge amount of discussion on the Extended Supply Chain Market andthe role of XML as a method for the Electronic Communication for business transactions. Thepotential of an integrated supply chain with the ability to communicate orders in a purelyelectronic medium not unexpectedly led to significant claims and expectations of technologies like XML and the vendors that provide solutions based on them. As with many new technologiesand solutions, the hype far exceeded the ability of vendors to deliver. This cycle accelerated withadditional promises and expectations creating several myths associated with the emergingtechnology such as very high costs and long, difficult implementations.

As the expectations of what XML could accomplish reached impossible levels, the bubble theyformed inevitably burst, leaving early adopters and advocates scrambling to defend thetechnology and distance themselves from the myths associated with it. In its wake, manyorganizations have distanced themselves not only from the solutions but also XML technologyaltogether. Others have focused on the scaled back, more manageable intermediate step thatWeb Services offers. Unfortunately, the hype cycle of exponentially growing promises andexpectations is facing Web Services. While this is understandable, when evaluating the potentialvalue of a technology it is important to first strip away any associated myths and evaluate theapplication of the technology in its pure form.

Consequently, the myths associated with the use of XML in the extended supply chain and theassumptions associated with them must be explored and evaluated. By providing a step-by-stepexample, this paper demonstrates a practical alternative for organizations with a strong fiscalfocus that would like to realize a majority of the benefits associated with an XML enabledintegrated supply chain without requiring excessive spending on technology solutions.

This white paper will focus on the myths associated with electronic transactions and demonstratethat with VorteXML, and the proven tools available to any purchasing agent or buyer in the supplychain, creating electronic transactions from existing system outputs is easy and cost effective andrequires NO programming skills.

Extended supply chain myths:• Electronic transactions require expensive enterprise software• XML requires significant programming skills and training• IT programming resources must be involved for the creation of electronic transactions• Current business documents are inappropriate because they don’t explicitly stipulate all the

fields required by the IT System• Organizations must first standardize their business processes with all of the members in

the extended supply chain in order to complete business electronically.• Semantic differences must be negotiated out of the electronic transaction to avoid erroneous

orders and mistakes


II. EXPLORING THE MYTHS

Within the supply chain, the conventional method of conducting business with hard copies ofbusiness documents such as purchase orders (PO’s) and requests for quotes (RFQ’s), was supposedto become an antiquated concept due to continuing developments in the technology arena ofelectronic commerce. Organizations would be able to achieve greater efficiencies and reducecosts by moving to a paperless method of purchasing goods and services. More significantly, thestandardization and refinement of business data formats and protocols in XML would allowcompanies the opportunity to conduct all matters of business in a paperless environment, byenabling the company’s systems to correctly interpret the information from an electronic format.The eventual goal would be to eliminate the very concept of traditional business documents as abygone artifact of a paper-based business world.

While the premise was compelling, the execution proved quite difficult. Semantic issues that werereadily apparent to the reader of a business document were not apparent to the systems thatwere operating on the electronic versions of these documents. While significant efforts areunderway to try and resolve these semantic variations, the undeniable truth is that these businessdocuments represent more than just paper and ink, they represent contractual obligationsbetween two parties that have been codified through the legal system over hundreds of years onan international basis.

The act of converting paper-based documents into an electronic format for exchange over anetwork between companies challenges the current definition of these documents and puts newdemands on the data elements available in the XML Data definition. As electronic documents,companies recognize that systems can read and interpret the data and drive internal processes to respond.

A business document has traditionally represented a legally binding contract between twocompanies during the business cycle of buying, selling, delivery, and payment. This long-termlegal standing has created a tight relationship between the business document and thedepartment that owns the business process responsible for creating, sending, and receiving it.Purchasing, Order Entry, Shipping and Invoicing are all common departments within businesstoday that exist to manage the business documents exchanged between two business partners.These departments, and the business documents they create, are directly integrated into eachcompany’s own internal processes to accept orders, commit to build product, build product, shipproduct and to send an invoice to the buyer. Each department has a fundamental role to assurethat the data communicated to the business partner is accurate and reflects the actual state of theorder. A sales order is not generated until manufacturing can commit to build. A ship date is notcommunicated until proper internal checks and balances indicate a high probability that this shipdate can be met. Many departments today exist primarily as gatekeepers that assure properprocedures are followed when communicating business documents to partners.


When people interpret business documents directly there is no need for each document to bemachine readable and cross-referenced to other related business documents. While partnerswould agree on a language and a standard for weight & measurement, it was recognized that theperson reading the business document could make basic assumptions. A Sales Order received for1000 Widgets was probably in response to the most recent Purchase Order for the same numberof Widgets. When a computer reads these documents, the standard business document mustcontain computer readable information that enables the system to identify the interrelationship of the documents so that appropriate action can be taken.

With paper-based systems, the sender and recipient of a business document agree on a languagethe document would be written in. Similarly, partners that intend to exchange businessdocuments electronically need to clearly identify how the data elements of the business documentwill be unambiguously exchanged. This can be done on a bilateral basis using highly structuredand tightly formatted machine-readable encoding, or it can be done by adhering to standardssuch as Electronic Data Interchange (EDI) X.12, or by adopting XML standards such as OAGI,RosettaNet, cXML or others. These are all standards that define a common syntax, or grammar,widely used by companies automating their supply chain to assure that each sent data element isproperly recognized by the receiving computer.

While the syntax assures each data element is clearly identified and put in a format that can beread by the receiving system, the use of computers to analyze business documents to increaseefficiency and lower costs has introduced a new problem: the need to manage the semantics ofthe data elements. A sender’s internal business process will typically influence exactly what thedata means within some fields of the business document. In slower, less analyzed days, minorvariations in these semantics mattered little. When a buyer kept four weeks of inventory, it didn’tmatter if a "shipped" data element meant the order was sent to shipping or actually loaded ontothe truck. The variation would only account for a day, or maybe two, of discrepancy. Today, abuyer may keep no inventory, and expect the supplier to ship product as the assembly linedemands it. In these highly efficient days, a day or two can mean the difference between success and failure.

III. A PRACTICAL ALTERNATIVE

The issues of semantics and business process are but two examples of how XML based-solutionscan experience significant delays during implementation and operational phase. Even aftercompletion, changes in business process and business requirements present delays associated with documenting these changes and communicating them to the programming resources.

While all of these challenges are real, Business users are left wondering why a process that was sostraightforward for them now requires all this additional effort. Between the knowledge of the


business users and the definition of the Paper report all of the information is available tocomplete the transaction. The difficulty arises when programming resources are required toimplement the semantic and process rules into the conversion. A practical alternative would be to use the business document as a source for the XML Message but this approach would requiretools that could address the key challenges of parsing the structured text in these businessdocuments. The task of using existing outputs presents it’s own set of challenges. The process ofconverting paper-based documents into an electronic format for exchange over a networkbetween companies requires the addition of semantic information not available in the currentdefinition of these documents and puts new demands on the data elements available in the XML Data definition.

In order to effectively use existing documents as a source for generating XML a valid solutionwould need to:

• Correctly parse all of the available data from the source document• Recognize date and numeric formats represented in the text document• Transform source data to the formats required by the specification• Reorder the source data to meet the nesting requirements in the specification• Map source, static and transformed data to the XML specification

If technology used to create the XML message could be adapted to allow business users map the text reports they have and augment them with the semantic and context required to create aValid XML message, it would be feasible to use these documents as a source for these electronicmessages.

VorteXML Designer and Server from Datawatch Corporation is such a tool. The VorteXML productsuite is comprised of two powerful components that work together:

• VorteXML Designer allows users to build and test profiles through a visual interface thatallows users to extract, transform and map data from existing text documents into XML. It may also be used for simple one-time conversions.

• VorteXML Server provides a scalable, high-volume solution that automates the extraction andconversion of text documents into XML. It includes scheduling facilities that perform automatedprocessing of XML conversions created by VorteXML Designer as well as API’s that can beinvoked programmatically through. Java, SOAP and .Net

This combination provides the ability to address all of the hurdles identified above. In this paperwe will walk through a typical Supply chain example and demonstrate how a supplier couldgenerate an XML message from their existing document sources using VorteXML. During theexample we will show each of the hurdles above and how they can be addressed using VorteXML.


IV. EXAMPLE DEFINITION

In order to articulate the difference between simple text conversion and what is required tosuccessfully convert text to XML it is necessary to address the issues that arise in a real-worldexample. Let's look at the communications associated with converting a printed invoice into anXML Invoice that is RosettaNet compliant. In the center of Figure 1 is the Great Sounds Inc. anational chain of music stores that has centralized purchasing and distribution. Betty Buyer worksfor Great Sounds and previously negotiated purchases directly with Sam Supplier at Classic MusicDistribution, a specialty supplier focused in rare classical soundtracks. After completing their ERPintegration, Great Sounds has decided to transmit Purchase Orders and accept Invoiceselectronically using the RosettaNet Standard. As an incentive for suppliers to use the new format,Great Sounds has agreed to Pay invoices submitted electronically in 30 days as opposed to theirusual 90 day terms.

Figure 1

Sam Supplier must convert their Invoices into the appropriate XML messages which must bevalidated against the appropriate RosettaNet PIP. While all of the required information is in Sam’sfinancial system, it would be prohibitive to map it to the PIP format. Ideally Sam would like to mapand transform the invoice generated by the system into the XML Message required by GreatSounds. Sam will use VorteXML to map the information from a sample invoice generated from hisfinancial system into the XML message. With VorteXML, Sam can combine the information in theinvoice with the semantic information he knows to generate a valid XML PIP Message. In this example we will map information from Sam’s Monthly Invoice into a RosettaNet Invoice Notification 3C3:

http://www.rosettanet.org/RosettaNet/Rooms/DisplayPages/LayoutInitial?Container=com.webridge.entity.Entity%5BOID%5B6DC47A9DA92DD411842300C04F689339%5D%5D


Sam reads the RosettaNet specification and process definition (excerpts shown below) from the3C3 PIP specification In accordance with the specification, Betty would represent the Buyer andSam the Seller. In this scenario in order for Sam to submit an invoice to Betty, he needs to be ableto create the invoice. This message has placeholders for all of the optional and requiredinformation necessary for a complete valid Notification of Invoice.

RosettaNet Notification of Invoice– 3C3

Business Process DefinitionThe “Notify of Invoice” Partner Interface Process™ (PIP) enables a provider toinvoice another party, such as a buyer or financing processor, for goods or servicesperformed. A financing processor is an organization that helps a seller obtain financing by serving as an intermediary between the seller andfinanciers.

Invoicing occurs after a purchase order is issued with PIP3A4, “Request PurchaseOrder” and a packing slip has been generated. For some transactions, the originalfinancial invoice must be attached to the shipment or sent in advance of thephysical shipment; thus, the invoice must be generated prior to the time ofshipment. The creation of a packing slip shall suffice to generate the invoice. Aninvoice can be a credit or a debit memo. If the original invoice is sent to afinancing processor, the financing processor may re-issue the invoice to the buyer.

The invoice is audited by the recipient’s internal systems. If the invoice is valid,payment may be scheduled for payment using PIP3C6, “Notify of RemittanceAdvice.” If the invoice is not valid, it may be rejected using PIP3C4, “Notify ofInvoice Reject.”

The time to perform is the same for all parties, including the financing processor.Should this transaction not complete successfully, the requesting partner executesPIPA01, “Notification of Failure.”


Copyright Statement Copyright 1999-2002 RosettaNet 800 W. Sixth St., Suite 320 Los Angeles, CA 90017 All rights reserved.

PIP PurposeThis PIP enables a provider to send an original invoice to another party, such as abuyer or financing processor.


Figure 2

V. STEP BY STEP

At first it may seem impossible to migrate from text to XML, but here is a step by step descriptionof the process.

Step One: Understanding the PIPOf these seven steps, by far the most difficult one is the first step: Understanding the RosettaNetspecification. In order to provide an invoice definition that can work for multiple industries andsituations, RosettaNet spent years defining and revising the PIP Specification. As a result, there aremany fields that are required by the PIP. This includes information about the company sending thedocument their contact information, payment information, and many identifiers that must bepopulated in the standard RosettaNet format. Elements like Global Country Code and UniversalPartner Identifier must adhere to the validation rules in the specification.

After spending several hours reading and understanding the specification, Sam determined thathe needed to include information from the original Purchase Order as part of the invoice and wasable to cull the field-list for the Purchase Order PIP to include the required elements and the datathat would be needed for the PO Since most of the terms and conditions for payment werealready established These fields were also deemed unnecessary. Figure 3 represents the corebusiness data entities from the "Guidelines for usage" Description for the AcknowledgePO PIP.(Specification available at www.rosettanet.org)


Figure 3

Step Two: Extracting from the InvoiceThe first step in the process is to extract the required data from a sample invoice file. This stepuses templates to extract datasets from the invoice file. The process is visual and easy to use.Sam would define a template for every set of data in the report. Each template would trapsections of data that represented a specific dataset. Once trapped, defining fields is a simplematter of painting the field and assigning its properties (by double clicking) (Figure 4).


Figure 4

Step Three: Defining datasetsBecause most text documents contain relationships that are easy for the reader to understand,VorteXML allows users to map the semantic relationships between the datasets as part of theirdefinition. This means that the data hierarchy can be communicated into the XML specification.Relationships like one Ship To location per Invoice and many Items per Order are automaticallydefined by the product as the datasets are created. These relationships can also be edited if theparticular document instance used to create the profile does not reflect all the possibilities of thedocument set. Figure 5 shows the properties for each dataset and associated definitions


Figure 5

Step Four: Transforming DataVorteXML provides a powerful calculation engine to facilitate the transformation of data from theinvoice. In Figure 6 you can see a sample transformation that correctly parses the bill to and shipto address block into their root elements.

VorteXML’s calculation enginealso provides lookup, stringmanipulation and logicaloperations through numerousfunctions and operators that allow for completetransformation of the sourcedata into the required elements.In Figure 7 this engine is usedto derive a global part numberfrom the supplier’s part numberby concatenation.

Figure 7


Figure 6

Step Five: Mapping Data ElementsOnce all of the data elements are mapped and transformed, the task of mapping them to theXML elements is quite easy. First the user must switch over to Output Mode. Figure 8 shows themapping of one of the data elements from the Ship to address into the Deliver to address elementin the PIP. This step would be repeated for each of the data elements in the report.


Figure 8

Step Six: Mapping Static ElementsIn addition to all of the elements that are extracted, derived or referenced from the report thereare additional required elements that must have static values. Information about the sender andterms as well as global country codes and role codes can be defined for each document type,since the results of these mappings can be saved as a reusable Profile (Figure 9). Users can createversions for each variation of the static information and document format. This last step assuresthe creation of a valid XML message by allowing the entry of static fields.


Figure 9

Step Seven: Generating the XMLThe final step is to browse the XML and ensure that all the required elements for a validdocument are populated. After a few revisions, the result is a complete XML document generatedfully from the text invoice (Figure 10).

Summary of Actual Time and CostsFunctionality aside, many of the solutions available for XML are cost prohibitive for smallerorganizations. Below is a summary of costs and times associated with the example above. These are the actual hours spent on the conversion.

Time Amount• Total Time to Create Input Definition 8 hours $800• Creation of Look-up Tables 2 Hours $200• Testing and editing 4 Hours $400• Software Costs

• Designer $599• Server $7999

Figure 10


VI. CONCLUSION

For several years, Manufacturers have looked for better ways to communicate business to businesstransactions. With EDI, they were able to define communications protocols that were highlyspecified for communicating transactions. While this approach worked for manufacturers andtheir tier one suppliers, they did not readily adapt to business process changes or allow for one-time exceptions. Additionally the VAN’s used to transmit EDI orders were extremely expensive. Asa result, Small/ Medium Businesses (SMB’s) and larger second-tier suppliers chose not toparticipate. The technology did not meet their needs as a cost-effective adaptable technology thatwould allow them to meet the numerous needs of their customers and partners without breakingthe bank.

The advent of XML with its self-describing ability which would allow for transmission of semanticinformation along with the data provided the foundation for a feasible solution. Suppliers still hadthe daunting challenge of mapping their existing data sources to XML, not to mention adding inall the semantic information associated with the processing of orders. Each of these progressionsfurther added to the costs associated with migrating existing information into XML and added tothe information technology resources needed to support the conversion effort. MeanwhileBusiness users wondered: “Why can’t I use my existing purchase orders and invoices as a source? Imean if we can manually use these documents why can’t these documents that I trust, be thestarting point for my Business to business communications?”

The response was that while XML technology allowed for the communication of semantics andchanges, the task of using existing outputs would prove difficult since much of the semanticinformation was not available from the source documents. In order to effectively use existingdocuments as a source for generating XML a valid solution would need to:

• Correctly parse all of the available data from the source document• Recognize date and numeric formats represented in the text document• Transform source data to the formats required by the specification• Reorder the source data to meet the nesting requirements in the specification• Map source, static and transformed data to the XML specification

Additionally, since the business user knew most of the contextual information, the tool wouldneed to be easy to learn and require no programming skills to use.

This white paper demonstrates that VorteXML provides a solution that meets all of therequirements of SMB’s in order to use existing documents as a source for their XML strategy. Inthis step-by-step demonstration, Sam was easily able to generate an XML Invoice from his printedinvoices. Clearly, using VorteXML and the paper output from existing systems, one can participateelectronically in the extended Supply Chain. As we demonstrated with VorteXML, it is possible toparticipate in an extended Supply Chain without mortgaging your company to purchase thesoftware to do it.


During the mapping from a invoice report to a Notify of Invoice PIP, Sam needed to overcomeseveral issues associated with mapping the information into the format Betty required. Theseissues are typical of the challenges that make using reports or existing documents difficult.

Datawatch has a long history of building solutions that take a practical approach to informationtechnology. By creating tools that leverage technology to simplify the process of creatingelectronic transactions, we allow the negotiators of the transactions to interact with the systemsof their trading partners. This drastically reduces the need for semantic reconciliation.

About Datawatch Corporation

Datawatch Corporation (NASDAQ: DWCH), is a leading provider of enterprise reporting,infrastructure and IT support solutions that help organizations increase productivity, reduce costsand gain competitive advantage. Datawatch products are used in more than 20,000 companies,institutions and government agencies worldwide.

DATAWATCH’s products include Monarch™, a report mining application that unlocks data inlegacy reports to provide business intelligence on the Windows desktop; Monarch|ES™, anenterprise reporting and business intelligence solution that transforms all of an organization’sexisting reports, data and documents into an easy to use, web-based decision support solution;Monarch Data Pump™, a data replication and migration tool; VorteXML™, a data transformationtool introduced to transform existing structured text data into valid XML; Q|SM™, a service centerand work flow software suite; VHD, a help desk application; and Redwing™, a plug-in for AdobeAcrobat that accurately extracts text and tables from PDF files.

Datawatch works with VARs, integrators, consultants and independent software vendors who selland support Datawatch products. In addition, Datawatch works with OEM customers who embedDatawatch components and technologies in their own solutions. The corporate address forDatawatch is 175 Cabot Street, Suite 503, Lowell, MA 01854-3633; telephone 978-441-2200, fax 978-441-1114; http://www.datawatch.com.

◆ ◆ ◆ ◆

This is a preliminary document and may be changed substantially prior to final commercial release. This document is provided for informationalpurposes only and Datawatch Corporation makes no warranties, either express or implied, in this document. Information in this document is subjectto change without notice. The entire risk of the use or the results of the use of this document remains with the user. The example companies,organizations, products, people and events depicted herein are fictitious. No association with any real company, organization, product, person orevent is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights undercopyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means(electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Datawatch Corporation.

Datawatch may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in thisdocument. Except as expressly provided in any written license agreement from Datawatch, the furnishing of this document does not give you anylicense to these patents, trademarks, copyrights, or other intellectual property.

©2003 Datawatch Corporation. All rights reserved.

Datawatch, VorteXML, Monarch, Monarch|ES, Monarch Data Pump, Q|SM, VDH and Redwing are either registered trademarks or trademarks ofDatawatch Corporation in the U.S.A. and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners.


xml in the supply chain · xml in the supply chain ... • semantic differences must be negotiated...

Documents