agile nosql with xrx

Agile NoSQL with XRXAgile NoSQL with XRX

Dan McCrearyPresidentPresidentDan McCreary & [email protected](952) 931 9198

M

D (952) 931-9198D

Metadata Solutions

Session DescriptionWhat if all members of your software development team from Project Managers Business Analysts Testing and Documentation Project Managers, Business Analysts, Testing and Documentation members could create and modify web applications and web services? With traditional SQL solutions this was difficult because f th d t t b t bj t bj t t t bl of the need to convert web pages to objects, objects to tables as

well as the reverse functions. But now with native XML databases and drag-and-drop forms builders, data can flow from the XML model of a web form to the database and back again without translation. This radically simpler process combined with standardized query languages makes it easier for non-standardized query languages makes it easier for nonprogrammers to build and maintain their own applications and web services.

M

D Copyright 2011 Dan McCreary & Associates2 of

N

During this session viewers will:• Understand the challenges associated with traditional four-

translation web/object/RDBMS systemstranslation web/object/RDBMS systems• Understand the elegant simplicity of zero-translation systems• Understand the role of web standards in the XRX environment• See how each component of XRX plays key roles in the

application• See how organizations have leveraged XRX for agile web

development• See variations of XRX architectures• Create and objective process to evaluate the benefits of XRXj p• Get resources for building XRX pilot projects

M

D Copyright 2011 Dan McCreary & Associates3

Executive Summary• Schema-free, zero translation NoSQL

systems have the ability to have a bigimpact on overall system agilityp y g y

• XRX systems dramatically increase overall agility and can also empoweroverall agility and can also empower non-progammers to build and maintain NoSQL systemsNoSQL systems

M


Background for Dan McCreary• Circuit Designer at Bell Labs• UNIX/Supercomputers• UNIX/Supercomputers• NeXT Computer (Steve Jobs)• Owner of 75-person software consulting firm

with a focus on Object-Oriented systems andwith a focus on Object Oriented systems and Object-relational mapping frameworks

• US Federal data integration (National Information Exchange Model NIEM.gov)

• NativeXML/XQuery for metadata management since 2006

• Advocate of web standards, OpenSource, N SQL d XRX tNoSQL and XRX systems

• W3C invited expert (forms working group)• Likes functional programming

M

D Copyright Kelly-McCreary & Associates, LLC5

Origins: The Humble Data Dictionary

M

D 6Copyright 2011 Kelly-McCreary & Associates

Electronic Certificate of Real Estate

Summer 2006

1 Document= 44 SQL inserts

M


250 Data ElementsXFormsMockupMockup

M


Four Translations

T1 T2

T3T4

• T1 HTML into Java Objects

Object MiddleTier

RelationalDatabaseWeb Browser

• T1 – HTML into Java Objects• T2 – Java Objects into SQL Tables

T Tables into Objects• T3 – Tables into Objects• T4 – Objects into HTML

M


Kurt's SuggestionUse a

A Native XML

Web Form

A Native XMLDatabase!

Save

Web Browser

Kurt Cagle

store($collection, $file-name, $data)eXistM


eXist

Zero Translation

XForms

• XML lives in the web browser (XForms)

Web Browser XML database

• XML lives in the web browser (XForms)• REST interfaces• XML in the database (Native XML, XQuery)( , y)• XRX Web Application Architecture• No translation!

M

D 11Copyright 2010 Dan McCreary & Associates

Key Question: Impact on Agility• What impact do zero translation

N SQL t hNoSQL systems have on system agility?

• Agility: the ability to quicklyAgility: the ability to quickly react to changing business requirements at any stage of the software development lifecycleQuestion: Big impact or little• Question: Big impact or little impact?

• Answer: Big impactM

D

Answer: Big impact

Copyright 2011 Kelly-McCreary & Associates12

No-Shredding!

My FormData

• Relational databases take a single hierarchical document and shred it into many pieces so it will fit indocument and shred it into many pieces so it will fit in tabular structures

• Many document-oriented NoSQL databases prevent this shredding

M


g

Is Shredding Really Necessary?

• Every time you takeEvery time you take hierarchical data and put it into a traditionalput it into a traditional database you have to put repeating groups input repeating groups in separate tables and use SQL “joins” touse SQL joins to reassemble the data

M


Many Processes Today Are Driven By…

The constraints of yesterday…

Challenge:Ask ourselves the question…D t th d f l i bl ith t b l d tDo our current method of solving problems with tabular data…Reflect the storage of the 1950s…Or our actual business requirements?What structures best solve the actual business problem?

M


p

"Schema Free"• Systems that automatically determine

how to index data as the data is loaded into the database

• No a priori knowledge of data structureN d f f t l i l d t d li• No need for up-front logical data modeling– …but some modeling is still critical

• Adding new data elements or changing g g gdata elements is not disruptive

• Searching millions of records still has sub-second response timesecond response time

M


Monoculture and Mono-architecture

M


Image Source: Wikipedia Monoculture

Storage Architectural PatternsgTables Trees

T i l StarsTriples Stars

M


Finding the Right MatchSchema-Free

Standards Compliant

Mature Query Language

M


Use CMU's Architectural Tradeoff and Modeling (ATAM) Process

Architectural SummaryFour Translation Zero Translation

T T

HTML eb pages XForms Client

TT web browser XML databaseweb browser database

• HTML web pages• Object middle tier• RDBMS database

• XForms Client• Native XML Database

RDBMS database

Which system more agile and by how much?M


How can this help us manage enterprise metadata?

Origins: The XML Data Dictionary

M


Electronic Certificate of Real Estate

Summer 2006

1 Document= 44 SQL inserts

M


250 Data ElementsXFormsMockupMockup

M


The NO-SQL UniverseDocument StoresKey-Value Stores

XML

Graph Stores

Object Stores

M


Three Core Processes1. Add new data to the database

1. Use XML example but JSON and other formats could also be used

2 Q h d2. Query the data3. Create an XML web service

• Analyze the effort for each stepCompare SQL and NoSQL XRX systems• Compare SQL and NoSQL-XRX systems

• Analyze impact on participation of non-programmers• Rate the relative agility of each system

M


It is Easy to Import DataSQL1. Analyze data for all parent child

relationships and repeating groups

XQuery1. Drag XML files into folder

p p g g p2. Design logical and physical ER

diagrams3. For each table create a Data Definition

File using a data definition language (DDL)(DDL)

4. Create indexes using DDL5. Create one table for each set of

repeating set of data6. Run DDL on database creating tables6. Run DDL on database creating tables

using the appropriate data types7. Create indexes8. Create Insert statements9. Create separate insert statements for

each repeating group10. Run Insert statements on primary

structures in database11. Use primary keys of the first data

inserts as foreign keys of dependantM

D 26

inserts as foreign keys of dependant data structures

The XML File system• XML File system – a way of

storing information in XML thatstoring information in XML that can be quickly searched

• You can drag-and-drop almost fil t thi fil tany files onto this file system

• You access it by using a WebDAV connector

• Microsoft Windows “My Network Places” function

M

D 27

Functional Programming

y = f(x)y = f(x)• Computer programs are like mathematical functions• Developers do not manipulate states and variables (things that

change value), but focus entirely on constants and functions g ) y(things that never change) and transformations of data

• Makes it very easy to build modular programs• Software written in functional programming languages tend to

be very concise and easy to port to highly parallel systemsbe very concise and easy to port to highly parallel systems

M

D Copyright 2011 Kelly-McCreary & Associates28

http://en.wikibooks.org/wiki/Computer_programming/Functional_programming

It's Easy to Query XML Data

SELECT COL1, Col2FROM TABLE

for $r in doc(‘t.xml’)//rowwhere col1=1FROM TABLE

WHERE COL1=1where col1 1

return $r/col1, $r/col2

Col1 Col21 A

<root><row>

<col1>1</col1><col2>A</col2></row>1 A

1 B1 C

<row><col1>1</col1><col2>B</col2>

</row><row>

1 C1 D

<col1>1</col1><col2>C</col2></row><row>

<col1>1</col1> <col2>D</col2>M

D 29

</row></root>

It is Easy to Create A Web Service

1 Learn Java or find a Java Developer 1. All XQuerys are web servicesThe Java/JDBC/SQL Way The XRX Way

1. Learn Java or find a Java Developer2. Install TomCat Web Server3. Install Java AXIS Web Server4. Write a JDBC program that sends

SQL queries to a databaseSQL queries to a database5. Get the results back in Java Result

Object structures6. Go through the Java Results

Structues and use print statements to rap XML tags aro nd the stringsto wrap XML tags around the strings in the result objects

7. Rename your class files to .jws files8. Add the .jws files to the TomCat

deploy foldersdeploy folders9. The WSDL files will automatically be

generated

M

D 30

Insert/Select/Publish ComparisonSQL

SQLSQLJava

TomcatAXIS

logicaldata

modeling

XQuerySQL XQuery

JDBC

XQuery

modeling

Insert Query Web Service NoSQL

M

D 31Total Effort

Steps for Adding a New Element• Add the element to the "new

instance" a form uses and setinstance a form uses and set the default value there

• Add the UI control

Takes about one minute

• Add the UI control• Optionally – run an XQuery

update to update existing dataupdate to update existing data or add an insert to the "enricher" when you get an

Takes about 10 minutes

y ginstance

M

D Copyright 2011 Kell-McCreary & Associates32

Six Translation – Web ServiceWeb Service

T T

T5 T6

T1

T

T2

T

• T1 – HTML into Java Objects

T4T3

Object MiddleTier

RelationalDatabaseWeb Browser

• T1 – HTML into Java Objects• T2 – Java Objects into SQL Tables• T3 – Tables into Objects• T4 – Objects into HTML

T Objects to XMLM

D

• T5 – Objects to XML• T6 – XML to Objects

33Copyright 2011 Dan McCreary & Associates

XML Stored in XForms ModelBrowser Database

model save

view

update

M

D Copyright 2011 Kelly-McCreary & Associates 34

XRX Core ProcessBrowser Database

model save/edit

view

update

M


Code Table ServicesClient Server

model Form Data

viewCode Table Service

C dall-codes.xq

CodeTables

M

DCopyright 2011 Kelly McCreary & Associates 36

Code tables are separated from form instance data

XRX Dynamic Forms GenerationApplication Server

Document

Client Application

User Team

Session

Form Data

XForms Model

Application Server

Form DataCollection

DataElementRegistry

StatusGroupRole

Binding Rules

Code Tables

Views

Code Table ServicesContextfilters

Suggest Services

Required

Read-only

Data TypesBusiness Rules Editor

Submissions

Calculations

Business Rules Editor

Constraints

Calculations Inference

XF Vi

Static Controls

Dynamic Controls

XForms View

Semantic Schemas

Subschema Service

XML Schema Registry

Constraint SchemasM

DDesign Time

Run Time

Semantic SchemasConstraint Schemas

Model Driven• XForms enables the

developer to reuse businessdeveloper to reuse business rules encapsulated in XML Schemas (xsd) and XML

XFormsApplication Schemas (xsd) and XML

Transforms (XQuery Typeswitch)

• XForms reduces duplication and ensures that a change in XML

Schema

MetaData

the underlying business logic does not require rewriting in another language

Schema Registry

M


another language

View and Model are Trees• The view is a tree of a presentation

data elementModel

data element• Models are comprised of one or

more trees• XForms supplies the control layer

that moves data elements to and from the model

Control (Bind)from the model

• Users don’t have to worry about moving things to and from the

View (Presentation)g g

screen

M

D Copyright 2011 Kelly McCreary & Associates39

Models and View Are Linked with "Bind"

HTMLh d

xf:model

P

headbody

fPerson

Name

form

fieldset

first last

label

inputlabel

input<bind>

• Both the model and the views are trees of d t l t

input<bind>

M


data elements

Just “Do The Right Thing”HTML

headbody

xf:model

Person

body

form

PersonCurrentOnTaxes type="xs:boolean" fieldset

labellabel

inputlabel

input<bind>

PersonBirthDate type="xs:date"

• Data types from the model just do the right thing• Boolean variables become checkboxes

D t h d t l tM


• Dates have date selectors

Example of Automatic UI Generation

• All true/false data types (xs:boolean) automatically become a checkbox

• All dates (xs:date) have a date selector to the right of the date fieldthe date field

• All codes can be selected from listsM


from lists

Structure of a XForms File

• XForms tags are just XML Namespaces tags imbedded in a

standard XHTML file with

Namespaces

CSS Imports (View)

a different namespace• Most HTML form tags are

Model

Constraints (Bindings) gexactly the same but some attributes have been

( g )

UI (View)

promoted to be full elementsMyForm.xhtml

Submit Controls

M


y

REST• REpresentation State Transfer• Create applications based on well

designed URLs• Take advantage of web caching• Migrate toward Resource-OrientedMigrate toward Resource Oriented

Computing (ROC)• REST evangelists: RESTifarians• REST evangelists: RESTifarians

M


Five RESTFull Friends

1. In-resident memory cache in your browser

2. You local hard drive cache3. Your local enterprise cache4 The cache on the web server farm4. The cache on the web server farm5. The cache on the database

Please make sure to check with your RESTfullfriends BEFORE you bother the database.

M


Shallow REST vs. Deep REST• You can start taking advantage of ReST

b j t d i ll th ht t URLbuy just doing well thought-out URL design

• To take advantage of deep ReST you must consider the subtleties of the HTTP protocol– GET vs POST vs PUT– DELETE

M


Benefits of REST• Provides improved response time• Reduced server load• Improves server scalabilityp y• Requires less client-side software• Depends less on vendor dependencies• Depends less on vendor dependencies• Promotes discovery • Provides better long-term compatibility• Better and evolvability

M


y

Sample XForms

M


Requirements Editor

CodeTable

SelectionLists

R ti

Lists

RepeatingElements

M

D 49

Drag-and-Drop XForms Builders• Several options for

d d d fdrag-and-drop forms builder

• Open Source– Orbeon– BetterFORM

• CommercialCommercial– IBM Forms

M

DCopyright 2011 Kelly-McCreary & Associates

50

Application Modularity

M


Financial Institution

M


Federal Integrator

M


Minnesota Historical Society

M


XRX Apps Demo Site• Demo site for

danmccreary.com• Some programs written

under an hour with student data

• Several utility programs that start with template and add transformations to other formats

• Focus on metadata management

M


Structured Retrieval is Better

Introduction to Information Retrieval

by Christopher D. Manning, y p gPrabhakar Raghavan and Hinrich Schütze

C b id U i iCambridge University Press, 2008

http://nlp stanford edu/IR-book/information-retrieval-book htmlM

D 56

http://nlp.stanford.edu/IR-book/information-retrieval-book.html

Table 10.1 - Revised

RDB search unstructured retrieval

structured retrievalretrieval retrieval

objects records unstructured documents

trees with text at leaves

vector space & model relational model vector space & others XML hierarchy

main data t t table inverted index

trees with node-ids for structure table inverted index ids for document ids

queries SQL free text queries XQuery fulltext

XML - Table 10.1 and structured information retrieval. SQLRDB (relational database) search, unstructured information retrieval

M

D 57

unstructured information retrieval

Retain Document Structure in Search

"Bag of Words" "Retained Structure"

keywords

keywordsdoc-id

'love'

'hate''new'

keywordskeywords

keywordsnew

'fear' keywords

• All keywords in a single container• Only count frequencies are stored

with each word

• Keywords associated with each sub-document component

• Assign higher weight for titles andM

D

with each word Assign higher weight for titles and names

• Set by non-programmer 58

Empower the Non Programmer!Before XRX After XRX

SUPER PM, BA!,

Sorry, we have no ideawhat code 47 means.

Let me just search our XRX system…I'll have your answer in 150 milliseconds.

M


Using the Right Architecture

FinishStart Finish

Find ways to remove barriers to empoweringthe non programmers on your teamthe non programmers on your team.

M


Six "S"s of XRX Agility1. Semantics – build around shard metadata

i t iregistry services2. Search – structured search3. Standards – CSS, XML, XPath, XQuery,

XForms, XML Schemas4 S i ll XQ i REST i4. Services – all XQueries are REST services5. Solutions - that are quickly customized6. Super – Empower the non-programmers

M


XRX Resources• XRX Wikipedia Page

– http://en.wikipedia.org/wiki/XRX(web_application_architecture)p p g ( _ pp _ )• XRX Resources

– http://www.danmccreary.com/xrx/• LinkedIn XRX GroupLinkedIn XRX Group

– http://www.linkedin.com/groups/XRX-Web-Application-Architecture-1056777

• Beginner's Guide to XRX– http://www danmccreary com/xrx/beginners-guide– http://www.danmccreary.com/xrx/beginners-guide

• XRX Wikibook– http://en.wikibooks.org/wiki/XRX

Google Code• Google Code– http://code.google.com/p/xrx/

M


References

XForms

XQXQuery

XRX

Send e-mail to dan@danmccreary com for extended

A Beginner's Guide to XRX

M

D

Send e mail to [email protected] for extendedlist of "getting started" resources.

Copyright 2011 Kelly-McCreary & Associates63

Thank You!Dan McCreary, PresidentKelly-McCreary & [email protected](952) 931-9198

eXist Meeting Prague March 12th, 2010M


eXist Meeting Prague March 12 , 2010

agile nosql with xrx

Technology

xrx dan mccreary president

dan mccreary associatesn

xrx systems t w3c

benefits of xrx

leveraged xrx

component of xrx

xrx environment

web applications