![Page 1: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/1.jpg)
REGULATORY REPORTING OF ASSET TRADING USING SPARK
Sudipto Shankar Dasgupta & Mayoor Rao – Infosys Limited
![Page 2: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/2.jpg)
Regulatory Reporting – Current Challenges
2
Financial Institutions need to report asset trade based on defined regulations
Reporting has to comply within a time window from the trade execution
Rules are subject to change and adjustments based on the change in regulations
The transaction volume is high Transactional systems are diverse
![Page 3: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/3.jpg)
Case Study 3
Leading global Financial Institution engaged in around 6 million trades a day in the financial markets
Regulatory requirements necessitates reporting in a specified reporting format within a 15 minute window
The trades are identified for reporting based on steps • Transformation • Enrichment • De-duplication checks • Business Rules
![Page 4: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/4.jpg)
Report Generation Process Flow Asset
Transactions
Transformation Rules
Transformed Asset Data
Master Data
Enriched Asset Data
Business Rules
Transformation Enrichment Report Generation
4
![Page 5: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/5.jpg)
Architecture Diagram
SQOOP
HDFS Data Lake
Tem
pora
ry S
tagi
ng
Pro
cess
ed O
utpu
t
In Memory Analytic and Processing Zone
Processing Logic
Apps Portal
5
![Page 6: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/6.jpg)
Ingestion throughput >130,000
records/second or 18.22 MB/second (per node)
Report execution time -> 35 seconds for 30,000 trade records
Overall Data Volume -> 596 million trade transaction records
Hardware Configuration -> Spark Installed in 10 node 32 GB RAM , 16 Core m/c
Statistics 6
01
02
03
04
![Page 7: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/7.jpg)
DEMO
7
![Page 8: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/8.jpg)
IIP - Leveraging Power of Spark 8
Open Source / Spark Components
Spark ML
Spark Streaming
HIVE / Spark SQL
Hadoop / FS Storage / Infra Mgmt
Infosys & Partner IP Components Tools | Data Extractors | Algorithms | Packaging &
Support
Customization, Integration & Implementation Services
Data Modeling & Cleansing | Agile App Development Data Science & Analytics | Security & Governance
Custom Data Extractors
![Page 9: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/9.jpg)
Infosys Information Platform - Architecture IIP Data Lake!
Query Interface
BI Applications
Down Stream Apps / Portal
IIP Data Lake!
1"
Rea
l Tim
e –
IIP In
gest
ion
Unstructured Data
RDBMS Data
NO SQL Data
Data Streams
JMS Queue
Social data
Machine data
Machine data
Web URL
Resource Manager
Governance
Security
Data Store
OS /LDAP/Kerberos Authentication Authorization Role Based Access
Metadata Management Data Lineage Data Audit
Cluster Resource Manager Cluster Monitor Disaster Recovery
Spark In Memory Analytic Zone
HDFS / Hive
Natural Language Processing
Machine Learning
Data Views
Data Mining and Models
Data Joins
Data Transformation
Single Click Installer Cluster Maintenance
Deployment Manager
IIP Data Explorer
2"
3"
4"
5"
9
![Page 10: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/10.jpg)
DEMO
10
![Page 11: Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao, Infosys Limited)](https://reader030.vdocument.in/reader030/viewer/2022032506/55cd181ebb61eb367a8b4606/html5/thumbnails/11.jpg)
© 2015 Infosys Limited, Bangalore, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to change without notice. Infosys acknowledges the proprietary rights of other companies to the trademarks, product names and such other intellectual property rights mentioned in this document. Except as expressly permitted, neither this documentation nor
any part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, printing, photocopying, recording or otherwise, without the prior permission of Infosys Limited and/ or any named intellectual property rights holders under this document.
Thank You