1 Twitter #NIEMNTE3
Automating Test and ValidationServices
for NIEM ExchangesWorkshop
Presenters:David RR WebberInformation ArchitectOracle Public Sector
Patrick FendtPrincipal TechnologistOracle Public Sector
2 Twitter #NIEMNTE3
The following is not intended to outline Oracle general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3 Twitter #NIEMNTE3
Introduction• A key need in working with NIEM IEPDs, is developing examples
and matching templates including rules, context, and roles for correctly managing information exchange content between partners. The W3C Schema by themselves are not sufficient.
• Learn how to use open source tooling, XPath rules, and the OASIS CAM templates standard to create automated testing tools.
• The automated validation engine can also be used in production environments to manage live information exchanges.
• Attendees will receive copies of the latest validation engine tools and examples for use in the workshop
4 Twitter #NIEMNTE3
Agenda• Part 1
– The XML Validation Framework– XPath Rules Techniques– Code lists and SQL lookups– Rules use cases
• Part 2– Example CAM templates – CAMV validation demonstration– Results handling techniques– Summary
• Q & A
5 Twitter #NIEMNTE3
XML Validation Framework
Architecture and Deployment
6 Twitter #NIEMNTE3
XML Validation Framework
CAMVXML instance
Template (compiled)
Database Lookup
(optional)
Results XML
Java Handler Errors and Warnings
Structure
Rules
Code Lists
SQL Query
7 Twitter #NIEMNTE3
CAMV Engine
CAMV Java (OneJar)
JDOM XML
JDBC Spring
XPath
API Error Handler
8 Twitter #NIEMNTE3
CAMV Validation Engine• Standalone XML validator using XPath rules• Context aware and declarative rules approach• Integrated with CAM editor menus for instant testing• External code list and SQL table lookup support• Callable via Spring framework or Java API• Configurable error levels – error, warning– Java error results object returned– Embedded validation results attributes in XML results
9 Twitter #NIEMNTE3
CAM Editor / CAMV Environment
CAM Editor CAMV
CAM Editor
XSLT Reports
CAM template
XML Test Instance
XML Validation Result
10 Twitter #NIEMNTE3
XML TestInstances
CAMV / ANT Script Environment
ANT
CAMV XML ValidationResults
XSLT
Reports
CAM template
CAMV Control Script XML
22
11
11 Twitter #NIEMNTE3
CAMV Test Control Script XML<camvTestSuite>
<testCase><Example-Test-v19-3>
<cxxfile>Test-Template-v19.cxx</cxxfile><xmlfile>Test-Instance-v19-3.xml</xmlfile><output>Test-Results-v19-3-out.xml</output><debug>true</debug><inlineErrors>true</inlineErrors><parameters>
<abc>Param1</abc><cde>Param2</cde>
</parameters></Example-Test-v19-3>
</testCase></camvTestSuite>
Validation Template
Test XML instance
Results output
Test case name
12 Twitter #NIEMNTE3
Linux GTK ANT Configuration• Linux distributions (e.g. Ubuntu) has ANT pre-installed
in /usr/share/ant/lib• Download Ant-contrib from – http://ant-contrib.sourceforge.net/
• Copy ant-contrib jar file to /lib directory inside ANT installation
13 Twitter #NIEMNTE3
Mac OS ANT Configuration• Mac OS X 1.7+ “Lion” comes pre-installed - use
command : ant –version and it will self-configure• Mac OS X 1.6 and earlier – need to install “Xcode”
development software first• Download Ant-contrib from – http://ant-contrib.sourceforge.net/
• Copy ant-contrib jar file to /lib directory inside ANT installation
14 Twitter #NIEMNTE3
ANT – Windows configuration• Download ANT from – http://ant.apache.org
• Download Ant-contrib from – http://ant-contrib.sourceforge.net/
• Copy ant-contrib jar file to /lib directory inside ANT installation
• Set the ANT_HOME environment variable to the directory where you installed ANT
• Add %ANT_HOME%/bin folder to executable path
15 Twitter #NIEMNTE3
Windows System Path Settings
22
11
33
16 Twitter #NIEMNTE3
Running CAMV with ANT• Go to CAMV location on Sourceforge– http://www.cameditor.org/#CAMV_Testing
• Download CAMV release JAR and Test example ZIP• Unpack test example from ZIP and put CAMV jar into
folder• Go to the command line (run cmd)– Switch to folder where CAMV test suite is located– ANT –f camv-test.xml runAll– Output is in folder called camvTest\results
17 Twitter #NIEMNTE3
Reviewing Validation Results HTML
18 Twitter #NIEMNTE3
XPath Rules Techniques
Quick Syntax PrimerControlling Scope, Context, Evaluation
Dynamic Structure ComponentsUseful XPath Resources
19 Twitter #NIEMNTE3
XPath Syntax Primer• XPath is a simple assertion language where expressions
reference components hierarchically within an XML instance structure – e.g.– /animals/canines/domestic/dogs/poodle[1]– /animals/canines/domestic/*
• CAMV uses the XPath v2.0 Java library• In CAM templates XPath is used extensively to provide
rules and content details for XML structure components
20 Twitter #NIEMNTE3
XPath FunctionsXPath Function sampler logical
concat(string,string,…)
substring(string,start,len)
string-length(string)
normalize-space(string)
upper-case(string)
contains(string1,string2) Yes
starts-with(string1,string2) Yes
not(arg) Yes
exists(item,item,...) Yes
count((item,item,...))
/bookstore/book[price>35]/title
21 Twitter #NIEMNTE3
CAM XPath Extensions (most used)Structure Functions Content Functions
makeOptional(xpath…) setNumberMask(xpath,arg)
makeRepeatable(xpath…) setStringMask(xpath,arg)
makeMandatory(xpath…) restrictValues(xpath,args)
excludeTree(xpath…) setDateMask(xpath,arg)
makeNillable(xpath…) datatype(xpath,arg)
setChoice(xpath…) setLength(xpath,arg)
setLimit(xpath…) lookup(xpath,args)
setValue(xpath,arg)
printMessage(xpath,arg) setNumberRange(xpath,args)
setDefault (xpath,arg)
22 Twitter #NIEMNTE3
CAM templates and XPath expressions• Within a CAM template XPath expressions are found in
the <as:BusinessUseContext> section of the template.
23 Twitter #NIEMNTE3
Rules Use Cases
Typical Business NeedsQuick Examples
24 Twitter #NIEMNTE3
Controlling Scope and Context
Condition controls if and when the content rule action applies
Condition can control structure as well; optional, mandatory, exclude
Condition can reference different component(s) for contextual rule actions
25 Twitter #NIEMNTE3
Some more examplesBusiness Rule: Items with color attributes don't need a tray number—the color is sufficient for picking the item:
<as:constraint condition="exists(//Item/@color)" action="makeOptional(//Item/TrayNumber)" />
Business Rule: If order weight exceeds 25 kg, purchase order must specify a freight carrier:
<as:constraint condition="//Item/@weight > 25" action="makeMandatory(//Item/FreightHandler)">
Using XPath axis referencing (when things are not adjacent in hierarchy):
<as:constraint condition="exists(ancestor::Item/@color)" action="makeOptional(//Item/TrayNumber)" />
* Excerpted from Michael Sorens DEVX CAM article examples
26 Twitter #NIEMNTE3
Useful XPath Resources• Michael Sorens article on using CAM from DEVX.com– http://www.devx.com/xml/Article/41150
• Martin Roberts – tutorial on writing CAM rules– http://merryflame.co.uk/jcam/site/tutorials/CAMTemplateBasics.htm
• XPath tutorial resources– Search on XPath tutorial– Books available
• Tools – XPath expression constructors
27 Twitter #NIEMNTE3
Code Lists and SQL Lookups
Introduction to CAM code listsAdvantages
Tools and Hints
28 Twitter #NIEMNTE3
Code lists• CAM lookup() function allows referencing to external
XML lists of values• Can replace use of restrictValues() static lists• Coming soon - CAM lookup() function can reference SQL
table for dynamic value checking• Where to get code lists?– Import XSD enumeration schema into CAM lookup lists– Convert UBL Genericode into CAM lookup lists– Dump from database into lookup list XML format
29 Twitter #NIEMNTE3
CAM Code list XML format
Textual valueCode value
Code list name
30 Twitter #NIEMNTE3
Combining Rules and Code lists• Real power of code lists and CAM – ability to select
code list values based on XPath rule criteria to handle different partner configurations, versioning and cross-table validations – e.g. Ford / Mustang | Chevy / Corvette– Message version id = 1.6 – use code list XYZ-1-6
31 Twitter #NIEMNTE3
Code lists examples• Load lookup list XML into CAM editor• Assign lookup() rule to XML component
• Coming soon - for SQL lookups – need to define connectivity to database and the SQL query to perform
32 Twitter #NIEMNTE3
Code Lists Summary• Lookups can be to external XML file instances• Simple XML format for optimized performance and
readability• Dynamic runtime control via XPath rules to select lists
to be applied• Lookup can reference SQL statement for database
content checking
33 Twitter #NIEMNTE3
Part 2– Example CAM templates – CAMV validation demonstration– Results handling techniques– Summary
34 Twitter #NIEMNTE3
Example CAM Templates
Examples ReviewedQuick walk through of important details
35 Twitter #NIEMNTE3
Experimental SAR Examples• Two test example templates and XML instances:– Baseline SAR with just structure and content usage
rules, schema parlance “cardinality, facets and enumerations”
– Baseline SAR advanced – logical business rules added to check consistency of information and structure components
• Each test set contains examples that pass and those that fail depending on the template validation applied
* SAR – Suspicious Activity Report NIEM exchange
36 Twitter #NIEMNTE3
CAM Templates Overview• Consists of 4 functional sections:
•Header declares properties and parameters for the CAM process to reference
•AssemblyStructure captures the business information exchange structures
•BusinessUseContext provides the rules to apply during validation
•Extension points to the annotations and external lookup tables
37 Twitter #NIEMNTE3
Experimental SAR Overview
* SAR – Suspicious Activity Report
Areas of Interest:-Drugs-Fraud-Gangs-Internet-Pornography-Property-Community-Weapons/Chemicals-Prescriptions-Travel/Immigration-Terrorism
38 Twitter #NIEMNTE3
CAMV Validation Demonstration
Illustrative examples – pass, fail, warningsDefault structure handling
Data type and Content checkingCross-field validation rules
39 Twitter #NIEMNTE3
Example Advanced Rules
<as:constraintcondition="//sbl:PrivacyDetails/sbl:PrivacyPurgeDate < //sbl:PrivacyDetails/sbl:PrivacyPurgeReviewDate"action="printMessage(//sbl:PrivacyDetails/sbl:PrivacyPurgeReviewDate,Warning: Review date not before purge date)"><as:annotation/></as:constraint>
Business Rule: Providing warning if PurgeDate is earlier than PurgeReviewDate
<as:constraintcondition="not(./nc:LocationAddress) and(not(./nc:LocationHighway)) and(not(./nc:LocationTwoDimensionalGeographicCoordinate))"action="printMessage(//j:TargetLocation/sbl:Location,Empty Location information)"><as:annotation><as:documentation type="documentation">Example of advanced rule</as:documentation></as:annotation></as:constraint>
Business Rule: Location must contain an address or a highway or coordinate set
40 Twitter #NIEMNTE3
Default Structure HandlingBy default all structure components are required and string content
Content Masks
Structure control
41 Twitter #NIEMNTE3
Demonstration
42 Twitter #NIEMNTE3
ANT script configuration
Directory
Control file
43 Twitter #NIEMNTE3
Results Handling Techniques
XSLT script post-processingXML Diff considerations
Regression Testing with CAMVMiddleware Integration
44 Twitter #NIEMNTE3
Default XSLT rendering of results
HTML output
Finds CAM validation results attributes
Formats and outputs error details
45 Twitter #NIEMNTE3
Post-Processing Tips and Tricks• XSLT post-processing not only for HTML reporting– Create response error messaging directly from CAMV
output– Create receipt notification directly from CAMV output
• Can use simple DIFF to compare generic HTML report output to detect regression test changes
• Rule based checking better than strict result content testing
• Java API and Spring supports direct integration of CAMV
46 Twitter #NIEMNTE3
Summary
ReviewNext Steps
Future enhancementsResources and Links
47 Twitter #NIEMNTE3
Review and Value Proposition• Testing and verification of NIEM exchange XML
examples• SQL database lookups and flexible code lists handling• Business rule checking• Post-processing options and reporting with XSLT scripts• Regression testing with batch processing• Flexible deployment of XML validation framework• Integration via Spring API to middleware
48 Twitter #NIEMNTE3
What are we working on?• Improvements– “CAM for web services” – currently exchange-centric– Enhancing ANT capabilities and soapUI integration
• New features– Dictionary integration with Domain Update Tool– Evaluation tools for dictionaries and components– Enhancing XML example generation with synthetic data
49 Twitter #NIEMNTE3
CAMV Summary• Open source, open public standards based, cross-platform
deployment written using Java• Uses Java APIs, Spring API, JDOM, etc.• Validation features:
– XPath V2.0 support– Looks ups– 20 built-in functions
• Reporting– Embedded CAMERROR, CAMWARN tags– Spring API– Optional sysout log
50 Twitter #NIEMNTE3
Q & A
Download the CAM editor toolkit from:
http://www.cameditor.org
Resource Center linkhttp://www.cameditor.org/#CAMV_Testing