webinar: processing high volume data feeds with mongodb - fix, fpml and swift in association with...

© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.

Title Text

• John Davies (@jtdavies) | CTO - C24• Trisha Gee (@trisha_gee) | 10gen

•Webinar• 4th December 2012

Processing High Volume Data Feeds with MongoDB & C24

1Tuesday, 4 December 12

https://twitter.com/trisha_gee?uid=15784290&iid=am-156194753213546123650035629&nid=23+112



Agenda...


• C24 iO - Introduction

• C24 iO - Demo


© 2012 C24 Technologies Confidential Information of C24 TechnologiesConfidential Information of C24 Technologies Ltd.

Typical Usage

© 2012 C24 Technologies Confidential Information of C24 Technologies7

Import or model any data formatsXML Schema, DTD, XML, Relax-NG, RDBMS, Java Classes, Text Files, Tag/Value, Delimited (CSV), Fixed, Binary, Proprietary Canonical Formats

Choose from hundreds of pre-modelled Standards SWIFT-MT, SWIFT-MX, ISO-20022, SEPA, FIX, FpML, DTCC, MINOS, CBI

Create and manage syntactic & semantic validation rulesFlexible technology, Java-based, Full XPath 2.0 support, Scala, Groovy, Clojure etc.

Compare models & manage versionsGraphical hierarchical comparison

For DevelopersBound Java ObjectsFully Documented APIGetters / SettersXPath & XQuery APIThousands of JUnit testsANT & Maven Build for C.I.Integration with Camel / Fuse, Spring Integration / Batch, Mule, GemFire, GigaSpace, Coherence, Oracle, MongoDB and others

ProductionFully tested JAR

For Business AnalystsA Powerful GUI for managing, modelling and transformation

Export Documents & Reports



SWIFT Example (MT541)

{1:F01DRESGB2LAXXX0548034693}{2:I541DRESDEFFXXXXN2}{3:{108:invalid:c3}}{4::16R:GENL:20C::SEME//FRTJ123REC2:23G:NEWM:99B::TOSE//100:16S:GENL:16R:TRADDET:98A::TRAD//20000519:98A::SETT//20000524:90A::DEAL//PRCT/101,001283:35B:ISIN GB0987654321:16S:TRADDET.........:95P::PSET//CRSTGB22:16S:SETPRTY:16R:AMT:19A::SETT//GBP4047151,3:16S:AMT:16S:SETDET-}



Transformation Example



Model Visualisation

•Our old friend the MT541 - Receive Against Payment



Graphical Navigation

• Some more complex than others• A Fix 5.0 SP2 Trade Capture Report (One of the

simpler messages)



So why are we here?

• Take any one of these technologies...•MongoDB, Neo4J, Coherence, HazelCast, Redis, GemFire, GigaSpaces,

Terracotta, MemcacheD etc.• Even a classic RDBMS, in fact “sadly”, usually a RDBMS

• A bunch of complex messages...• XML Schema, DTD, XML, Relax-NG, RDBMS, Java Classes, Text Files, Tag/

Value, Delimited (CSV), Fixed, Binary, Proprietary Canonical Formats• SWIFT MT/MX, ISO-20022/SEPA, FIX, FpML, DTCC, MINOS, CBI etc.

• And you have an integration problem...• How, for example, do you get your internal canonical messages into

MongoDB?• How do you store your front office Fix messages, in any way?



It’s a Very Simple Solution

• To load a CSV into an RDBMS is simple - 2D data anyway• There are loads of tools to do the job, usually from the database vendor• ETL (Extract, Transform, Load) is the classic tool for the RDBMS

• Complex (Hierarchical or 3D) data is difficult to map to a RDBMS•Minor changes in the data require awkward table redesign every time

•MongoDB is to complex data what an RDBMS is to a CSV•We are no longer restricted by table schema, multi-versions possible• But we need new ETL tools

•We could chuck everything in as a String or byte array• But we still need to understand the content if we are to be able to build

queries and extract what interests us



Technical Overview

• As I said, it’s very simple with this technology combination...• Load the messages into a pre-built C24 model and convert into JSON• The JSON format is built in•

Message queueMQ Series / JMS

Files Reader

C24 Parse

C24 Validate

Save to MongoDB



Fix

• Take a FIX message...

8=FIXT.1.1 9=1 35=AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING 57=STRING 143=STRING 144=STRING 145=STRING 52=20020101-00:00:00.000 122=20020101-00:00:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING 629=20020101-00:00:00.000 630=1 571=STRING 487=0 568=STRING 150=0 572=STRING 17=STRING 527=STRING 378=0 570=Y 55=STRING 65=CD 48=STRING 22=1 454=1 455=STRING 456=1 460=1 461=STRING 167=FAC 200=200201 541=20020101 224=20020101 225=20020101 239=CORP 226=1 227=1 228=1 255=STRING 543=STRING 470=AF 471=GB 472=STRING 240=20020101 202=1 206=0 231=1 223=1 207=XLON 106=STRING 348=1 349=D 107=STRING 350=1 351=D 32=1 31=1 15=AFA 120=AFA 194=1 195=1 30=XLON 75=20020101 60=20020101-00:00:00.000 768=1 769=20020101-00:00:00.000 1033=A 1034=1 1035=E.W 63=0 64=20020101 573=0 574=A1 552=1 54=1 453=1 448=STRING 447=B 452=1 802=1 523=STRING 1=STRING 581=1 81=0 575=Y 576=1 577=0 578=STRING 579=STRING 376=STRING 377=Y 582=1 336=STRING 625=STRING 12=1 13=1 479=AFA 497=Y 157=1 230=20020101 158=1 159=1 238=1 237=1 118=1 119=1 155=1 156=M 77=O 58=STRING 354=1 355=D 518=1 519=1 520=1 521=AFA 136=1 137=1 138=AFA 139=1 37=STRING 198=STRING 11=STRING 38=1 468=0 469=1 528=A 529=1 483=20020101-00:00:00.000 381=1 10=000

• Relatively simple as they go but to get this into a database in a useful format is not trivial

A reminder that it’s not flat, even Fix has hierarchy...



Fix

• Take a FIX message...

8=FIXT.1.1 9=1 35=AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING 57=STRING 143=STRING 144=STRING 145=STRING 52=20020101-00:00:00.000 122=20020101-00:00:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING 629=20020101-00:00:00.000 630=1 571=STRING 487=0 568=STRING 150=0 572=STRING 17=STRING 527=STRING 378=0 570=Y 55=STRING 65=CD 48=STRING 22=1 454=1 455=STRING 456=1 460=1 461=STRING 167=FAC 200=200201 541=20020101 224=20020101 225=20020101 239=CORP 226=1 227=1 228=1 255=STRING 543=STRING 470=AF 471=GB 472=STRING 240=20020101 202=1 206=0 231=1 223=1 207=XLON 106=STRING 348=1 349=D 107=STRING 350=1 351=D 32=1 31=1 15=AFA 120=AFA 194=1 195=1 30=XLON 75=20020101 60=20020101-00:00:00.000 768=1 769=20020101-00:00:00.000 1033=A 1034=1 1035=E.W 63=0 64=20020101 573=0 574=A1 552=1 54=1 453=1 448=STRING 447=B 452=1 802=1 523=STRING 1=STRING 581=1 81=0 575=Y 576=1 577=0 578=STRING 579=STRING 376=STRING 377=Y 582=1 336=STRING 625=STRING 12=1 13=1 479=AFA 497=Y 157=1 230=20020101 158=1 159=1 238=1 237=1 118=1 119=1 155=1 156=M 77=O 58=STRING 354=1 355=D 518=1 519=1 520=1 521=AFA 136=1 137=1 138=AFA 139=1 37=STRING 198=STRING 11=STRING 38=1 468=0 469=1 528=A 529=1 483=20020101-00:00:00.000 381=1 10=000

• Relatively simple as they go but to get this into a database in a useful format is not trivial



Using C24-iO

• Select or import the model (Fix in this example)

•Deploy the code (one click)• This generates a Fix endpoint jar that is capable of reading, validating and

writing out Fix messages - in multiple formats• Plus of course a rich API, XPath 2.0 and Spring Integration• XPath, like QL offers a single language to filter/query/sort any data structure

• From a small snippet of code we can read in a Fix message (into our Jar) and write it directly into MongoDB• It’s even simpler with Spring Integration & Spring Data - no code (just XML)

•With Spring Integration / Data we can load Fix messages into MongoDB without any code at all



Seconds later...

• Select the output format as JSON and..."StandardHeader":{ "SendingTime":{ "withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002" }, "OrigSendingTime":{ "withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002" }, "XmlData":{ "XmlDataLen":1, "XmlData":"D" }, "MessageEncoding":"ISO-2022-JP", "LastMsgSeqNumProcessed":1, "HopGrp":{ "NoHops":1, "HopGrpFields":{ "HopCompID":"STRING", "HopSendingTime":{ "withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002"



Fix into MongoDB

• In code (for the techies)...



Or if you prefer in Spring...

•We literally can take Fix (SWIFT, FpML, CSV etc.) messages from a queue/file and write them to MongoDB with NO CODE!



Some Important Links...

• The C24 iO and MongoDB page•White Paper, Log4J & Fix examples

• C24-iO• http://www.c24.biz• John Davies (@jtdavies) | CTO - C24

• 10gen• http://www.10gen.com• Trisha Gee (@trisha_gee) | 10gen

http://www.c24.biz/io-mongodb


http://www.c24.biz

http://www.c24.biz

http://www.10gen.com

http://www.10gen.com





webinar: processing high volume data feeds with mongodb - fix, fpml and swift in association with...

Technology