gather those events : instrumenting everything for analysis
DESCRIPTION
About WSO2CEP and WSO2BAM on WSO2Con 2014 Europe BarcelonaTRANSCRIPT
![Page 1: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/1.jpg)
Gather those Events Instrumenting Everything for Analysis
Sriskandarajah Suhothayan (Suho)Technical Lead
WSO2 Inc.
![Page 2: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/2.jpg)
Connected Business !
![Page 3: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/3.jpg)
Connected Business !
![Page 4: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/4.jpg)
Connected Business !
● Services ● Systems ● Data ● APIs ● Cloud ● Mobile ● Workflows ● ….
![Page 5: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/5.jpg)
Connected Business !
● on-premise / cloud ● From WSO2/ 3rd party
geographical locations ● using various protocols &
message formats
![Page 6: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/6.jpg)
Issues
![Page 7: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/7.jpg)
Issues
![Page 8: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/8.jpg)
Issues
![Page 9: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/9.jpg)
Issues
![Page 10: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/10.jpg)
Issues
![Page 11: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/11.jpg)
Issues
![Page 12: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/12.jpg)
Issues
![Page 13: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/13.jpg)
Monitor ..!
Present
!
![Page 14: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/14.jpg)
Analyse ..!
Present
!
Analyse !!
!
![Page 15: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/15.jpg)
Gather Info ..!
Present
!
Analyse !!
!
Gather !!
![Page 16: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/16.jpg)
Gather Info ..!
Present ! !! ! Gather
!!
Having proper data @ proper place
@ proper time !
![Page 17: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/17.jpg)
Gather Info ..!
Present ! !! ! Gather
!!
Having proper data @ proper place
@ proper time ! !
What is Proper ?
![Page 18: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/18.jpg)
Do we know what proper is ?
![Page 19: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/19.jpg)
Do we know what proper is ?
![Page 20: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/20.jpg)
YES !
We captor them direct them for analysis & write queries to produce alerts using !
&
![Page 21: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/21.jpg)
NO & Maybe !
?
Lost !
http://aroundtheworldineightywaves.com/wp-content/uploads/lost.jpg
![Page 22: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/22.jpg)
NO & Maybe !
?
How we can solve ?
http://aroundtheworldineightywaves.com/wp-content/uploads/lost.jpg
![Page 23: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/23.jpg)
NO & Maybe !
?
How we can solve ?
http://aroundtheworldineightywaves.com/wp-content/uploads/lost.jpg
![Page 24: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/24.jpg)
How we gather ?
Gather !!
Analysis
Real time Post
Real time √ √
Batch x √
![Page 25: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/25.jpg)
How We Gather ?
Gather !!
Strategy ...
![Page 26: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/26.jpg)
From Where We Start ?
• There is always some sensitive parts of infrastructure Which we have to monitor !
• We can enrich them…. !
• When we have an infrastructure to analyse data, Add some more data
![Page 27: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/27.jpg)
From where we gather?
![Page 28: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/28.jpg)
From where we gather?
![Page 29: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/29.jpg)
From where we gather?
![Page 30: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/30.jpg)
From where we gather?
![Page 31: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/31.jpg)
From where we gather?
![Page 32: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/32.jpg)
From where we gather?
![Page 33: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/33.jpg)
From where we gather?
![Page 34: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/34.jpg)
From where we gather?
![Page 35: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/35.jpg)
From where we gather?
![Page 36: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/36.jpg)
From where we gather?
![Page 37: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/37.jpg)
From where we gather?
![Page 38: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/38.jpg)
From where we gather?
![Page 39: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/39.jpg)
From where we gather?
![Page 40: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/40.jpg)
From where we gather?
!! Is it only from WSO2 Servers ?
![Page 41: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/41.jpg)
From where we gather?
!
![Page 42: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/42.jpg)
From where we gather?
!
![Page 43: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/43.jpg)
From where we gather?
!
![Page 44: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/44.jpg)
From where we gather?
!
![Page 45: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/45.jpg)
From where we gather?
!
![Page 46: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/46.jpg)
From where we gather?
!
![Page 47: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/47.jpg)
From where we gather?
!
![Page 48: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/48.jpg)
From where we gather?
!
![Page 49: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/49.jpg)
From where we gather?
!
![Page 50: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/50.jpg)
From where we gather?
!
![Page 51: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/51.jpg)
From where we gather?
!All
![Page 52: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/52.jpg)
From where we gather?
!All
All these from Custom Data Publishers !
![Page 53: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/53.jpg)
They can do … !
• Asynchronous • Auto retry • Auto recovery • Load-balancing • Fail-over
How good the Data Publishers are ?
![Page 54: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/54.jpg)
!! Only from Thrift ?
How we gather?
![Page 55: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/55.jpg)
!!!!!
How we gather?
![Page 56: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/56.jpg)
!!!!!
How we gather?
They have many forms
• XML • JSON • Text • Map • WSO2 Event
![Page 57: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/57.jpg)
How we gather?
!!!!!
![Page 58: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/58.jpg)
How we gather?
!!!!!
!
![Page 59: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/59.jpg)
How we gather?
!
![Page 60: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/60.jpg)
How we gather?
!
![Page 61: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/61.jpg)
Performance
!!
• 1.8M Events/Sec on same JVM • About 250K over network for single node !!!
• 30K over Network to 3 node Casandra cluster
![Page 62: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/62.jpg)
How we manage ?
•Data is in different formats •Converted to a common format •Virtualised as Streams of Events •Events Stream has
Name : phone.retail.shop Version : 1.0.0 ! & also they have a Definition!
![Page 63: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/63.jpg)
Stream Definition
{ 'name':'phone.retail.shop', 'version':'1.0.0', 'nickName': 'Phone_Retail_Shop', 'description': 'Phone Sales', 'metaData':[ {'name':'clientType','type':'STRING'} ], 'correlaitonData':[ {'name':’transactionID’,'type':'STRING'} ], 'payloadData':[ {'name':'brand','type':'STRING'}, {'name':'quantity','type':'INT'}, {'name':'total','type':'INT'}, {'name':'user','type':'STRING'} ] } !!
![Page 64: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/64.jpg)
The Big Data View
![Page 65: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/65.jpg)
HA / Scalability of
![Page 66: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/66.jpg)
HA / Scalability of
![Page 67: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/67.jpg)
HA / Scalability of
![Page 68: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/68.jpg)
HA / Scalability of
![Page 69: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/69.jpg)
HA of
!
![Page 70: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/70.jpg)
HA of
!
![Page 71: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/71.jpg)
What gets measured, gets managed. !
~ Peter Drucker ~
![Page 72: Gather those events : Instrumenting everything for analysis](https://reader036.vdocument.in/reader036/viewer/2022062511/54c6a8ba4a7959f63e8b457a/html5/thumbnails/72.jpg)
Thank You