huge codebases application monitoring with hystrix...huge codebases – application monitoring with...
TRANSCRIPT
![Page 1: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/1.jpg)
Huge Codebases – Application Monitoring with Hystrix 1
Huge Codebases Application Monitoring with Hystrix
30 Jan. 2016
Roman MohrRed Hat
FOSDEM 2016
![Page 2: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/2.jpg)
Huge Codebases – Application Monitoring with Hystrix 2
About Me
Roman Mohr
Software Engineer at Red Hat
Member of the SLA team in oVirt
Mail: [email protected]
Github: https://github.com/rmohr
IRC: #ovirt irc.oftc.net
![Page 3: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/3.jpg)
Huge Codebases – Application Monitoring with Hystrix 3
oVirt
“oVirt is a powerful virtual machine manager for up to datacenter-class deployments, and provides an awesome KVM management interface for multi-node virtualization.” – http://www.ovirt.org
![Page 4: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/4.jpg)
Huge Codebases – Application Monitoring with Hystrix 4
oVirt Artitecture
ovirt-engine
Host 1
Host 2
Host n
VMs
VMs
VMs
...
Storage
![Page 5: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/5.jpg)
Huge Codebases – Application Monitoring with Hystrix 5
ovirt-engineThe Beast
Take 1
![Page 6: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/6.jpg)
Huge Codebases – Application Monitoring with Hystrix 6
Architecture of ovirt-engine
DB
REST GWT
Commands
DAL
VDSM
Broker
![Page 7: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/7.jpg)
Huge Codebases – Application Monitoring with Hystrix 7
Git Statistics of ovirt-engine
Branch: master
Generated: 2016-01-14 14:02:53 (in 370 seconds)
Generator: GitStats (version 2014-12-09), git version 2.4.3, gnuplot 5.0 patchlevel 0
Report Period: 2011-10-04 18:43:09 to 2025-03-31 23:18:53
Age: 4928 days, 1449 active days (29.40%)
Total Files: 10355
Total Lines of Code: 1123168 (2557376 added, 1434208 removed)
Total Commits: 20166 (average 13.9 commits per active day)
Authors: 174 (average 115.9 commits per author)
![Page 8: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/8.jpg)
Huge Codebases – Application Monitoring with Hystrix 8
Issues we have
● Many developers
● A lot of code
● No second level cache
● REST performance problems
● The product runs at the user/customer site
● Test coverage
● Hard to configure and run the application
![Page 9: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/9.jpg)
Huge Codebases – Application Monitoring with Hystrix 9
Where to start?
● Try to get a high level overview of the architecture
● “Data-mine your Source Control” – Greg Young*
● Gather code metrics (JArchitect, Sonar)
● Monitor your application before you change something
* How to get productive in a project in 24h
https://www.youtube.com/watch?v=KaLROwp-VDY
![Page 10: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/10.jpg)
JBug, Jan 2016 10
Java and Application Monitoring
![Page 11: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/11.jpg)
Huge Codebases – Application Monitoring with Hystrix 11
Profiler?
● You can see where your application spends its time
● Easy to get started. Just connect to the JVM in question and browse the CPU profiling graph.
● Some profilers even support JDBC, JPA, …
But:
● In general no application logic specific insights
● Many are closed source
● Not easy to collect data
![Page 12: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/12.jpg)
Huge Codebases – Application Monitoring with Hystrix 12
XRebel for Monitoring?
● Easy to integrate. Just start an additional Java agent
● Every HTTP servlet now contains an additional popup where you can access application metrics.
But:
● Closed source
● Development only
![Page 13: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/13.jpg)
Huge Codebases – Application Monitoring with Hystrix 13
NewRelic for Monitoring?
● Excellent visualization
● Supports multi-host applications
● Knows a lot about Java
Examples:
● Closed source
● Production only
● License based business model
![Page 14: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/14.jpg)
Huge Codebases – Application Monitoring with Hystrix 14
Hystrix
“Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop
cascading failure and enable resilience in complex distributed systems where failure is
inevitable.” – https://github.com/Netflix/Hystrix
![Page 15: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/15.jpg)
Huge Codebases – Application Monitoring with Hystrix 15
Hystrix
Hystrix also provides metrics!
![Page 16: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/16.jpg)
Huge Codebases – Application Monitoring with Hystrix 16
Hystrix Dashboard
![Page 17: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/17.jpg)
Huge Codebases – Application Monitoring with Hystrix 17
Hystrix Dashboard
Error percentage
Requests/s
Circuit breaker status
Statistics
Request shape
![Page 18: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/18.jpg)
Huge Codebases – Application Monitoring with Hystrix 18
Hystrix Dashboard
SuccessfulRejected (Short circuit)
Timeouts (thread isolation)
Rejected (max. concurrent invocations)
Failed executions (exception)Bad request (exception)
![Page 19: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/19.jpg)
Huge Codebases – Application Monitoring with Hystrix 19
Hystrix Dashboard
Easy to run
Easy to integrate
● Drop the WAR from maven central in your container
● Add the WAR as dependency and serve the resources folder on an endpoint.
● Add the hystrix-metrics-event-stream servlet to your application
$ git clone https://github.com/Netflix/Hystrix.git$ cd Hystrix/hystrix-dashboard$ ../gradlew jettyRun> Running at http://localhost:7979/hystrix-dashboard
![Page 20: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/20.jpg)
Huge Codebases – Application Monitoring with Hystrix 20
Hello World Hystrix Command
Setter setter = Setter.withGroupKey( HystrixCommandGroupKey.Factory.asKey("helloWorld")).andCommandKey( HystrixCommandKey.Factory.asKey("helloWorld"));
HystrixCommand<String> helloWorldCommand = new HystrixCommand<String>(setter) { @Override protected String run() throws Exception { return "Hello world!"; }};
return helloWorldCommand.execute();
![Page 21: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/21.jpg)
JBug, Jan 2016 21
ovirt-engineThe Problem
Take 2
![Page 22: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/22.jpg)
Huge Codebases – Application Monitoring with Hystrix 22
Problem description
● We have a datacenter with 1000 VMs.
● We query the /api/vms endpoint which returns all VMs.
● We need 2.5 seconds to fetch them with no additional load.
● We have a datacenter with 2000 VMs.
● We query the /api/vms endpoint which returns all VMs.
● We need 5 seconds to fetch them with no additional load.
![Page 23: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/23.jpg)
Huge Codebases – Application Monitoring with Hystrix 23
Solution: Let's curl a little bit
$> time bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 7043k 0 7043k 0 0 2774k 0 --:--:-- 0:00:02 --:--:-- 2775k
real 0m2.547suser0m0.008ssys 0m0.011s
$> time bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 13.7M 0 13.7M 0 0 2876k 0 --:--:-- 0:00:04 --:--:-- 3815k
real 0m4.900suser0m0.009ssys 0m0.014s
1000 VMs, 1 request
2000 VMs, 1 request
![Page 24: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/24.jpg)
Huge Codebases – Application Monitoring with Hystrix 24
Solution: Let's curl a little bit
2000 VMs, 10 parallel requests
$> time seq 1 10 | parallel -j 10 bash rest.sh vms > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 13.7M 0 13.7M 0 0 488k 0 --:--:-- 0:00:28 --:--:-- 3911k[...]100 13.7M 0 13.7M 0 0 487k 0 --:--:-- 0:00:28 --:--:-- 3848k
real 0m29.590suser0m0.212ssys 0m0.438s
![Page 25: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/25.jpg)
Huge Codebases – Application Monitoring with Hystrix 25
Solution: We can guess
● “That's because our database is so slow.”
● “The database can cache everything, it is because our REST application code is so slow.”
● “That's because we are keeping the database busy with status updates of Hosts and VMs.”
● “That's because our architecture is not smart enough, it is just an ordinary monolith. That must be solved with streaming and eventbuses.”
![Page 26: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/26.jpg)
Huge Codebases – Application Monitoring with Hystrix 26
Solution: Let's curl a little bit more
Let us execute the following scenario:
100 Vms, 10 parallel requests, 100 requests total
$> seq 1 100 | parallel -j 10 bash rest.sh vms > /dev/null
![Page 27: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/27.jpg)
Huge Codebases – Application Monitoring with Hystrix 27
Find the Error
![Page 28: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/28.jpg)
Huge Codebases – Application Monitoring with Hystrix 28
Find the Error
![Page 29: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/29.jpg)
Huge Codebases – Application Monitoring with Hystrix 29
With the Fix
$> time bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 13.7M 0 13.7M 0 0 7210k 0 --:--:-- 0:00:01 --:--:-- 7210k
real 0m2.218suser 0m0.079ssys 0m0.062s
2000 VMs, 1 request
$> time bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 7051k 0 7051k 0 0 8521k 0 --:--:-- --:--:-- --:--:-- 8516k
real 0m1.008suser0m0.091ssys 0m0.042s
1000 VMs, 1 request
![Page 30: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/30.jpg)
Huge Codebases – Application Monitoring with Hystrix 30
With the Fix
2000 VMs, 10 parallel requests
time seq 1 10 | parallel -j 10 bash rest.sh vms > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 13.7M 0 13.7M 0 0 1473k 0 --:--:-- 0:00:09 --:--:-- 3249k[...]100 13.7M 0 13.7M 0 0 1534k 0 --:--:-- 0:00:09 --:--:-- 3566k
real 0m10.228suser0m0.211ssys 0m0.436s
Much better but still too slow. We will see later how to avoid being overwhelmed by too much expensive calls.
![Page 31: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/31.jpg)
Huge Codebases – Application Monitoring with Hystrix 31
With the Fix
![Page 32: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/32.jpg)
Huge Codebases – Application Monitoring with Hystrix 32
Collect data from user systems
$> curl -H "Accept: application/json" -H "Content-type: application/json" -X GET --user admin@internal:engine http://localhost:8080/ovirt-engine/services/hystrix.stream
ping:
data: {"type":"HystrixCommand","name":"GetVmsInit","group":"GetVmsInit", [...] }data: {"type":"HystrixCommand","name":"VdsHostDevListByCaps", [...] }
Collect as much streaming data as you want:
Import it later in your favourite analysis tool or send the data directly to it by using Hystrix plugins.
![Page 33: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/33.jpg)
Huge Codebases – Application Monitoring with Hystrix 33
Archaius: config.properties
hystrix.command.SearchVM.execution.isolation.semaphore.maxConcurrentRequests=10
This will override the default configuration of the SearchVM command:
Default Value 10
Default Property hystrix.command.default.execution.isolation.semaphore .maxConcurrentRequests
Instance Property hystrix.command.HystrixCommandKey.execution.isolation .semaphore.maxConcurrentRequests
How to Set Instance Default
HystrixCommandProperties.Setter() .withExecutionIsolationSemaphoreMaxConcurrentRequests(int value)
How to limit concurrent invocations of a command protected by Hystrix:
Just add it to your config.properties file or set it as system property.
![Page 34: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/34.jpg)
Huge Codebases – Application Monitoring with Hystrix 34
Protected VMs Endpoint
![Page 35: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/35.jpg)
Huge Codebases – Application Monitoring with Hystrix 35
Great OSS Monitoring Ecosystem
● Hystrix
● Servo
● Dropwizard Metrics
● Cockpit
● Thermostat
● Grafana, Graphite
● Hawkular
● Prometheus
● ...
![Page 36: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/36.jpg)
Huge Codebases – Application Monitoring with Hystrix 36
Conclusion
● Hystrix provides an easy way to monitor and protect your monolith or your microservices
● An interesting open source ecosystem around monitoring exists
● From some closed source projects we can learn a lot about usability
![Page 37: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/37.jpg)
Huge Codebases – Application Monitoring with Hystrix 37
Questions?
![Page 38: Huge Codebases Application Monitoring with Hystrix...Huge Codebases – Application Monitoring with Hystrix 22 Problem description We have a datacenter with 1000 VMs. We query the](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d0be17e708231d4386990/html5/thumbnails/38.jpg)
Huge Codebases – Application Monitoring with Hystrix 38
Mail: [email protected]
Github: https://github.com/rmohr
IRC: #ovirt irc.oftc.net
Thank you!