atropos
DESCRIPTION
Atropos. Comp 415 Project In partnership with JPMorganChase. Introductory Remarks. COMP 415 Class: Brad Chelsea Dave Felipe Hubert Matt Sohum Dr. Wong Jacy Grannis. A Word From Our Sponsors. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/1.jpg)
ATROPOS
Comp 415 ProjectIn partnership with JPMorganChase
![Page 2: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/2.jpg)
INTRODUCTORY REMARKS COMP 415 Class:
Brad Chelsea Dave Felipe Hubert Matt Sohum
Dr. Wong Jacy Grannis
![Page 3: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/3.jpg)
A WORD FROM OUR SPONSORS...
Jacy Grannis has been our customer contact within JPMorganChase. He is a Rice Computer Science graduate and works in JPMorgan’s Houston office.
![Page 4: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/4.jpg)
COMP 415—SUMMARY Software engineering Real customer, real experience Team environment Communication Organization
![Page 5: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/5.jpg)
COMP 415—THE BENEFITS Team programming experience is sought by employers
and not often found in undergraduates Saves on-the-job learning time Industry exposure
![Page 6: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/6.jpg)
COMP 415—THE SEQUENCE COMP 410 COMP 415 Teaching assistant for both courses Need more courses
![Page 7: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/7.jpg)
ATROPOS
…semester in review
![Page 8: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/8.jpg)
ATROPOS—SUCCESS Correlation Network View Single Edge View Remote communication
![Page 9: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/9.jpg)
ATROPOS—MISTAKES Teams change Individuals change Team structure Small team Proxy pattern Integration
![Page 10: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/10.jpg)
SYSTEM REQUIREMENTS…an Atropos primer
![Page 11: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/11.jpg)
SYSTEM REQUIREMENTS—CORRELATION Track messages as they travel between servers Different systems have different message formats
Messages have no fixed unique ID Servers may split and merge messages
![Page 12: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/12.jpg)
SYSTEM REQUIREMENTS—SCALABILITY System must handle thousands of messages per second Components may run on separate machines Concurrency
Asynchronous communications
![Page 13: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/13.jpg)
SYSTEM REQUIREMENTS—RECOVERABILITY Failure of one component should not bring the system
down Failure detection Component restart Restore component state Replay dropped messages
![Page 14: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/14.jpg)
SYSTEM DESIGN…the view from the top
![Page 15: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/15.jpg)
•Distributed system
•Service-oriented architecture (SOA)
•Component registration
ATROPOS SYSTEM DESIGN—UNREGISTERED
![Page 16: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/16.jpg)
•Distributed system
•Service-oriented architecture (SOA)
•Component registration
ATROPOS SYSTEM DESIGN—REGISTERED
![Page 17: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/17.jpg)
DESIGN—ABSTRACTION AND THE PROXY PATTERN
Allows flexible system configuration Beneficial for testing purposes and agile development Unbiased by transfer protocol
![Page 18: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/18.jpg)
DESIGN—DYNAMIC CORRELATION Concepts of finding matches from message organization
are logically distinct Decouple algorithm from data Correlation engines organize messages and use
strategies to fine matches Changing strategies at runtime enables intelligent load
balancing and independence from JPM system configuration
![Page 19: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/19.jpg)
ATROPOS DEMO…which will totally work
![Page 20: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/20.jpg)
START CONTROLLER AND ROUTER Initialize the controller
Service that manages the system Router registers with controller
![Page 21: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/21.jpg)
START CORRELATION ENGINE, DATA CLIENT Correlation Engines and Data Clients run in their own JVMs On startup, they register with the controller Controller configures them to connect them into the system
![Page 22: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/22.jpg)
STARTING VIEW SYSTEM View Provider View Client
![Page 23: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/23.jpg)
CONNECT AND START CLIENT View Client connects to system Requests data
![Page 24: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/24.jpg)
CHANGING SOURCES AND EDGES Utilize IAdmin API to add sources Dynamically add and remove edges Check system status after updates
![Page 25: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/25.jpg)
SINGLE EDGE LATENCY VIEW Access
Dialog Clicking edge in Dashboard View
Display Time-chunk average of latency Scalable to different time periods
Static or updatable
![Page 26: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/26.jpg)
CUSTOM LATENCY VIEW See correlation during a specific time window
![Page 27: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/27.jpg)
UNCORRELATED MESSAGES Messages change, machine connections change, parts of
the topography may disappear completely Messages will stop correlating View uncorrelated messages over a period of time Updates live (currently every 30 secs)
![Page 28: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/28.jpg)
DISCONNECT VIEW CLIENT Disconnect a view client currently registered View client is unregistered
Future updates are no longer sent
![Page 29: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/29.jpg)
REMOTE COMMUNICATION…bridging gaps
![Page 30: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/30.jpg)
REMOTE COMMUNICATION No guarantee that components are on the same
machine Need a way to communicate between multiple
components Solution must be
Scalable Flexible Pluggable
![Page 31: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/31.jpg)
MULE—BENEFITS Provides a scalable package solution Open source Provides a flexible framework that can adapt to any
architecture Can tie into many messaging protocols
![Page 32: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/32.jpg)
MULE—HARDSHIPS Little support and documentation Remote method invocation can cause problems Needs to be scaled to work effectively
![Page 33: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/33.jpg)
FUNCTIONAL COMMUNICATION Proxy pattern connects components Controller as an object on the Mule server Mule support behind remote calls abstracted out
![Page 34: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/34.jpg)
DATA CLIENT…the beginning of it all
![Page 35: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/35.jpg)
DATA CLIENT—GOALS Vehicle for getting messages into the system Robust Abstract Recoverable Extensible
![Page 36: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/36.jpg)
DATA CLIENT—MESSAGE SOURCES Entry point for messages Accepts messages with a simple API
accept(message)... too easy? Next step: router
![Page 37: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/37.jpg)
DATA CLIENT—ABSTRACT Messages can come from anywhere
Log parsers Applications producing the messages Manual API calls
Abstracting out information about the source is crucial for interoperability
![Page 38: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/38.jpg)
DATA CLIENT—RECOVERABILITY Router dies...
What happens to messages that have been sent through the Data Client but are dropped?
Take a lesson from Tivo! Store information about messages that have been seen and
replay! Controller detects failure in the system
Can ynchronize all the modules in the system by replaying the Data Client
![Page 39: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/39.jpg)
DATA CLIENT—EXTENSIBILITY Multiple sources can send information to one data client
Require that incoming messages are be tagged with their source ID
Data Clients can send to other Data Clients Available for common special processing that can be added
at run time Added and removed via registration at run time with the
Controller
![Page 40: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/40.jpg)
ROUTER
…the grunt work
![Page 41: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/41.jpg)
ROUTER—ROLE Data Clients are only aware of Router
All messages simply forwarded Router responsible for mapping Data Clients to
Correlation Engines Data Client to Correlation Engine mapping not 1:1
One-to-many relationship
![Page 42: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/42.jpg)
ROUTER—EXTENSIBILITY Routers can make use of the composite pattern Updated dynamically at runtime by the Controller Whenever Data Clients or Correlation Engines register
(or unregister), the router must reflect those changes
![Page 43: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/43.jpg)
CORRELATION ENGINE…the workhorse
![Page 44: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/44.jpg)
CORRELATION ENGINE—ROLE Organizes messages received Applies correct strategies between various sources, as
determined by the system’s IAdmin Writes correlation information to a memory store Keeps a store of what messages have been received for
replay purposes
![Page 45: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/45.jpg)
CORRELATION ENGINE—DESIGN CONCERNS Configurability
Should handle addition of sources and new rules at runtime Concurrency
Thread safety Use of concurrency to take advantage of multi-cores
![Page 46: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/46.jpg)
CORRELATION ENGINE—IMPLEMENTATION DETAILS Makes messages appear to have been received in order;
facilitates strategy implementation Configurable correlation strategies
Handle rules for correlation Insulated from issues related to system configuration, message order, etc.
![Page 47: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/47.jpg)
USER INTERFACE…the pretty side
![Page 48: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/48.jpg)
VIEW CLIENT—TECHNOLOGIES Eclipse RCP Application Java Universal Network/Graph (JUNG) framework JFreeChart framework
![Page 49: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/49.jpg)
ECLIPSE RCP APPLICATION—POSITIVES Portability Integration with current JPMorgan systems Faster development on Eclipse IDE Powerful framework simplifies component connections
![Page 50: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/50.jpg)
ECLIPSE RCP APPLICATION—NEGATIVES SWT less popular than AWT Fewer resources for help Steep learning curve
![Page 51: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/51.jpg)
VIEW CLIENT—JUNG FRAMEWORK Ability to graph nodes based on set of vertices and
edges Robust decorator-based design for great customization Interactive display with the graph
![Page 52: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/52.jpg)
VIEW CLIENT—JFREECHART FRAMEWORK SWT-native chart creation package Wide variety of charts supported by default Interactive graph display Intuitive package to work with as a developer
![Page 53: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/53.jpg)
VIEW PROVIDER Serves data as requested by many concurrent view
clients Asynchronous "pull" data delivery system Designed to be able to live on a different server than the
view client Ability to handle several different types of data requests Design allows several View Providers to be active, and
for clients to connect to specific ones
![Page 54: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/54.jpg)
VIEW PROVIDER—NETWORK TOPOLOGY Provides the edges and vertices that comprise the
topology Pulls data from the PostgreSql datastore and converts to
viewable form
![Page 55: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/55.jpg)
VIEW PROVIDER—SINGLE EDGE LATENCY Provides time series of latencies between two vertices Parameterization of factors such as time range
![Page 56: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/56.jpg)
VIEW PROVIDER—UNCORRELATED MESSAGES Provides list of messages that have not been correlated
by the system Parameterization of time range allows specific messages
to be identified
![Page 57: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/57.jpg)
VIEW PROVIDER—SINGLE MESSAGE Provides data about the computed correlated paths of
specified messages Includes information about the latency at each vertex in
a given path
![Page 58: Atropos](https://reader036.vdocument.in/reader036/viewer/2022062410/5681606e550346895dcf9445/html5/thumbnails/58.jpg)
THANK YOU FOR COMING!…special thank you to Dr. Wong, Jacy Grannis, and JPMorganChase for making this semester possible!