building a smarter planet: financial services
DESCRIPTION
Building a smarter planet: Financial Services. Exploding Demands for Big Data, Analytics, Risk Management, Ultra-low Latency and Compute Power Requires Optimized HPC Infrastructures. Robert Brinkman Infrastructure Architect for Banking and Financial Markets IBM Banking Center of Excellence. - PowerPoint PPT PresentationTRANSCRIPT
Exploding Demands for Big Data, Analytics, Risk Management, Ultra-low Latency and Compute Power Requires Optimized HPC Infrastructures
Building a smarter planet: Financial Services
Robert BrinkmanInfrastructure Architect for Banking and Financial MarketsIBM Banking Center of Excellence
Let’s build a smarter planet
2 © 2009 IBM Corporation2 © 2012 IBM Corporation
Panel
Dino VitaleDirector, Cross Technology Services
Morgan Stanley
Vikram MehtaVice President, IBM System Networking
IBM Corp.
Emile Werr VP, Global Data Services
Global Head of Enterprise Data Architecture NYSE Euronext
Nick WerstiukProduct Line Executive
IBM Platform Computing
Let’s build a smarter planet
3 © 2009 IBM Corporation3 © 2012 IBM Corporation
Workload Optimized Stacks
Financial Markets Industry Imperatives• Re-engineer for profitable growth: Renewed focus on the customer, Near real
time analytics• Improve the trade life cycle: Cloud and business process outsourcing• Optimize enterprise risk management: Data driven transformation and common
industry services
Appliances& Packages
ApplicationsIBM Provided, ISVs, Partners, Custom
Grid Stack
Low Latency
Stack
TransactionStack
High Message Rates
Data Value Decay
Big Data
Big Compute
High TransactionRates
Complex Data Models
Specialized Workloads
Packaged Hardware and Software
CloudStack
Discrete Components or Applications
Variable Workload
Messaging and Security
Nothing below this
point
Nothing below this
point
Dino VitaleDirector, Cross Technology Services
Morgan Stanley
Nothing below this
point
Nothing below this
point
Morgan Stanley: Road to Compute As a Service Trends
• Maximize efficiency of compute infrastructure • Cost / run-rate• Utilization – more with less, linear scale, sharing• Operational normalization
Challenges• Phasing• Dynamic provisioning and scaling on-demand of resources to applications according to
varying business needs and SLA• Multi-tenant workload protection• Application design and dependency management• Utility charge-back model options: pay-per-use, fixed allocation, hybrid approach • Sharing resources based on work load supply and demand • BCP
Convergence opportunities with “Big Data”• Increasing data volumes • Adaptive/real-time Scheduling• Resource management• Metrics / Data mining
ON-DEMAND DATA IN HIGH PERFORMANCE ENVIRONMENTEmile Werr, VP, Global Data Services
Global Head of Enterprise Data Architecture & Identity Management
Big Data (billions of transactions and multi-terabyte captured daily) Speed and business agility are essential to our business Different viewpoints and data patterns need to be analyzed Data coming out of a Trading Plant is not user-friendly Correlating disparate data & integration Moving large data around is expensive and complex System Capacity requirements need to efficiently handle 5x of our Avg daily volume. Data Spikes – the day after Flash Crash volume peaked over18.4 Bn transactions for NYSE Classic Matching engine (this excludes Options and other markets like Arca, Amex, Liffe, Euronext, etc.) Transaction volume growth sustained year-over-year Data needs to be readily available for a min of 7 years for Compliance It is too expensive to keep it all online Change is constant
Technology Challenges
Global Data Services 4
8
Data Architecture Practice
Financial Services, Regulatory & Compliance Expertise
Order arrived:BUY 10 @ 20.09
Full Quote Size- Best Quote size from the last published best quote
Price level for calculating Shares Ahead & Shares Available
Trading systems generate vast transaction volumes at high speeds The GRID is utilized to transform, normalize and enrich the time-series data using massive parallel computing. This is done as EOD or Intra-Day batch processing. Date-Level Table scans (Queries) need also massive parallel processing (MPP) Appropriate technologies need to be utilized (10gb Network, Virtualized CPU/MEM, Appliance Databases, Scalable Storage Pools)
USE CASE: Market Reconstruction for Trading Surveillance
The Electronic Book (NYSE DBK) and Market Depth needs to be reconstructed and accessible via Fast Database
Who Traded Ahead or Interpositioning ? This can be answered by a Database Query
Data Lifecycle Management Methodology
Data Capture
End-User Workflow
Data Transformation & Archive
User Analytics“Business Intelligence”
On-Demand Data (ODD)
Trading DataMarket DataRef DataUser Generated Data
Transform, Normalize, EnrichPartition, compress and archive in storage poolsCreate Metadata (mappings)
EnterpriseSystems
Secure Data Access & NavigationLoad, Extract, Stream, Filter, Transform, PurgeUser-driven Data Mart Provisioning (“Sandboxing”)Schema Change Capture (“Data Structure Lineage”)
Utilize MPP Databases & HDFSIntegrate Reporting ToolsFacilitate User CollaborationCapture Knowledge (KM)Automate Data Archive & Purge
Global Data Services 3
FeedHandler
FeedHandler
Continuous Flow (Trickle Batch)
files
Data PumpData Pump
Data Capture Data Virtualization & Abstraction Business Demand
Managed Data Services & Data Flow Automation
10
Transformation & Archive
Scale-Out Grid Fabricdistributed CPU/MEM
MessageBus
Storage Pools
AppsAdmins
AnalystsData ScientistsResearchers
Fast Processing & Data Movement
Scalable
Reliable
Simplified Access & Administration
File & Database Virtualization
Common Secured Access
Automation & Workflow
Standardization & Consistency
Agile Framework – Metadata Driven
Metering, Monitoring &Tracking
HadoopNetezzaAnalytics Data Warehouse
DataProvisionin
g
DataTools
Data Services
Let’s build a smarter planet
11 © 2009 IBM Corporation11 © 2012 IBM Corporation
Vikram MehtaVice President, IBM System Networking
IBM Corporation
Let’s build a smarter planet
12 © 2009 IBM Corporation12 © 2012 IBM Corporation
Nick WerstiukProduct Line Executive
IBM Platform Computing
Let’s build a smarter planet
13 © 2009 IBM Corporation13 © 2012 IBM Corporation
Workload Compute Intensive Data IntensiveCompute and DataIntensive
Data type StructuredRDBMS, Fixed records
UnstructuredVideo, E-Mail, Web
All – Structured + Unstructured
ApplicationUse Case
Pricing
BIReportingStreaming
Risk Analytics
AML/Fraud
Sentiment Analysis/CRM
Genomics ETL
Gaming
Trading
CEP
Simulation
Characteristics“Real Time” QuarterlyDailyIntraday Monthly
Infrastructure Dedicated servers,Appliances, FPGAs
Disk & Tape, SMP & Mainframe,
SAN/NAS InfrastructureData Warehouses
Compute grid,Data caches, In-memory grid,Shared services CPU + GPU
Commodity processors + storage
Convergence of Compute and Data
Let’s build a smarter planet
14 © 2009 IBM Corporation14 © 2012 IBM Corporation
14
Resource Orchestration
C
Workload Manager
C C C C C
C C C C C C
D
D
D
D
D
D
D
D
D
D
D
D
C C C C C C
Metadata generation,File classification,
Batch analysis
Search, Analysis, Concept Recognition
Data Intensive Apps
A A A A
A A A A
A A A A
A A A A
B
B
B
B
B
B
B
B
B
B
B
BB B B B B B
A B C DGeo-spatial integration,
Name classificationSignal processing
Support for Diverse Workloads & Platforms
Let’s build a smarter planet
15 © 2009 IBM Corporation15 © 2012 IBM Corporation
Latency
Scale
Inefficient scheduling, polling model & heavy-weight transport
protocols limit scalability.
OtherGrid Servers
Symphony
With a zero-wait time “push model” and efficient binary protocols, Symphony
scales until the “wire” is saturated
Why IBM Platform Symphony is faster and more scalable
Let’s build a smarter planet
16 © 2009 IBM Corporation16 © 2012 IBM Corporation
HPC Cloud – Multiple Approaches and Paths to Value
Infrastructure Management
Infrastructure Management
• Cluster consolidation into an HPC Cloud
• Self-service cluster provisioning and management
• Workload-driven dynamic cluster
Build out a more dynamic HPC infrastructure as their HPC Cloud
HPC “In the Cloud”
HPC “In the Cloud”
• ‘Bursting’ to Cloud Providers• Hosted HPC in the cloud• Enable HPC Cloud Service
Providers
Leverage the public cloud opportunity, either to tap into additional resources, or offer their own HPC cloud services
Questions
Building a smarter planet: Financial Services