counters and async infrastructures · counters and async infrastructures ... multiple counters...
TRANSCRIPT
Counters and AsyncInfrastructures
Proposal for Code ContributionGuy Sela, Senior Engineer, HPE
Who am I ?• Senior Engineer at HPE• Served as a developer and team leader at the Israeli NSA• Developing software since 2004
E-mail: [email protected]: http://www.linkedin.com/pub/guy-sela/70/17/9aa
AgendaCounters Infrastructure• Why do we need Counters?• Proposed Implementation
• Capabilities• Examples• Under the Hood
• Third-Party Alternatives
Async Infrastructure• Scope• Overview• Under the Hood
Why do we need Counters?
• High-Frequency events will swamp the Log.• Good alternative for Log.debug() and Log.trace()• Useful information for troubleshooting.• Powerful tool for analyzing bottlenecks.• What events happened in my system until now?• Analytics
Proposed Implementation
• Light-weight infrastructure that is very easy to integrate
into your code
• Supports the common use-cases for counters
• Found to be extremely useful for monitoring and
troubleshooting production systems
Capabilities
• Incremental & State Counters
• Configurable interval for “print counters delta to the log”
• Runtime counters silencing based on the user’s log library
• Almost no overhead on the system
• Counter querying API based on regular expressions
• REST API
Use Case – Incremental Counterpublic class UserService {
// High-Scale Eventvoid userEvent() {
log.info(“Event Called”); // Will swamp the Log… // Business Logic
}}
Use Case – Incremental Counterpublic class UserService {
// High-Scale Eventvoid userEvent() {
UserServiceCounters.user_event_called.inc();… // Business Logic
}}
Log Output Example – 1 Minute Interval[2016-02-16 14:18:27,626] INFO : (com.hpe.test.TestClassA:252) – Did Something[2016-02-16 14:18:27,967] INFO : (com.hpe.test.TestClassB:254) – Did Something Else[2016-02-16 14:18:28,028] CNT : user_event_called : +37824[2016-02-16 14:18:45,125] INFO : (com.hpe.test.TestClassC:63) – Yet Another Log Line[2016-02-16 14:19:28,028] CNT : user_event_called : +41215[2016-02-16 14:19:50,026] INFO : (com.hpe.test.TestClassC:72) – I Love Log Lines[2016-02-16 14:20:30,028] CNT : user_event_called : +51621[2016-02-16 14:20:50,026] INFO : (com.hpe.test.TestClassA:11) – Log Lines Rock!
10
Log Output Example – Multiple Counters[2016-02-16 15:18:27,626] INFO : (com.hpe.test.TestClassA:252) – Did Something[2016-02-16 15:18:27,967] INFO : (com.hpe.test.TestClassB:254) – Did Something Else[2016-02-16 15:18:28,028] CNT : user_event_called : +37824, other_counter: +72[2016-02-16 15:18:45,125] INFO : (com.hpe.test.TestClassC:63) – Yet Another Log Line[2016-02-16 15:19:28,028] CNT : user_event_called : +41215, foo_counter: +17
11
public class UserService {// Boilerplate Code to define a new Counters Groupenum UserServiceCounters {
user_event_called(), other_user_counter(), third_counter();
private OccurenceCounter counter; public void inc() {
counter.inc();}
UserServiceCounters() {counter = new OccurenceCounter(“UserServiceCounters”, name());
}}
}12
import com.hpe.counter.OccurrenceCounter;
Counter Types
• Incremental Counter – Value is manipulated using inc(). The log prints show the delta from the last log print. No delta No log print.
• State Counter – Value is manipulated using set(). The log prints show the current value of the counter in case it changed. Useful for detecting queue build-up.
13
Counters SilencingLog4J Config File // Standard mechanism to change Log Severity
log4j.logger.UserService.user_event_called=WARN
14
Use Case - Using Counters to Identify Bottlenecksvoid myBusinessLogic {
String value = DB.getKey(“key”);// …Socket.sendOnSocket(new Packet(value.toBytes()));// …invokeIntensiveMethod(value);
}Scenario: 5000 Events per second which invoke myBusinessLogic();Symptom:We only manage to handle 2000 Events per second
15
Use Case - Using Counters to Identify Bottlenecksvoid myBusinessLogic {
BLCounters.before_db.inc();String value = DB.getKey(“key”);
BLCounters.after_db.inc();// …BLCounters.before_socket.inc();Socket.sendOnSocket(new Packet(value.toBytes()));BLCounters.after_socket.inc();// …BLCounters.before_intensive.inc();invokeIntensiveMethod(value);BLCounters.after_intensive.inc();
}16
Log Output will tell us where is the Bottleneck[2016-02-16 15:18:28,028] CNT : before_db: +5000,
after_db: +2000, before_socket: +2000, after_socket:
+2000, before_intensive: +2000, after_intensive: +2000
The DB is Slowing Us!
17
Log Output will tell us where is the Bottleneck[2016-02-16 15:18:28,028] CNT : before_db: +5000,
after_db: +5000, before_socket: +5000, after_socket:
+5000, before_intensive: +5000, after_intensive: +2000
The Intensive Method is Slowing Us!
18
What events happened in my System until Now?
CountersDumperService APIString getCounters(String regexp);• Call using REST API / Other Management Console
19
Analytics
20
Under the Hood
• Counters based on AtomicLong & AtomicInteger causes
insignificant overhead.
• Using enum semantics instead of “register new counter”
implementation
• Background Thread that writes the delta of the counters to
the Log in each configurable interval.
21
Third-Party Alternatives
• Known third-party implementations with similar goals:
Metrics, Parfait, JAMon, Java Simon, Perf4J.
• We haven’t made a deep comparison between the
proposed solution to these.
22
Questions?
23
And Now to a Completely Different Subject
24
Async Infrastructure
25
Introduction
• In ODL’s Hydrogen version, every OSGI Bundle was activated using the BundleActivator pattern.
• Every OSGI Service declared its implementations and dependencies in its Activator.
• After the transformation to MD-SAL, Plugins/Applications define their dependencies, implementations and API using YANG Model.
• Applications that reside in the Controller communicate with each other using MD-SAL: RPCs and Notifications.
26
Introduction• Application can be implemented by a number of OSGI Services
and Bundles.• The presented Async Infrastructure facilitates communication
between OSGI services, and is agnostic to YANG, RPCs, etc.• The infrastructure is currently only relevant for Bundles that still
use the BundleActivator pattern. Specifically for Activators that extend ComponentActivatorAbstractBase.
• If the presented capabilities interest the ODL community, we can discuss extension of the infrastructure to support the new OSGI Bundle Activation patterns, that came along with MD-SAL
27
Scope
28
Async Infra (Unwanted?)
Application A
OSGI Service OSGI Service
MD-SAL
Async Infra
Application B
OSGI Service OSGI ServiceAsync Infra
RPC / NotificationsRPC / Notifications
Overview
29
• ServiceA implements PacketProcessingListener
Service A Service B Service CdoSomething() doSomethingElse()
onPacketReceived() Notification
MD-SAL
Application A
public class ServiceA implements IServiceA, PacketProcessingListener {private IServiceB serviceB;
public void onPacketReceived(PacketReceived notification) {serviceB.doSomething();
}}
30
public class ServiceB implements IServiceB {private IServiceC serviceC;
public void doSomething() {serviceC.doSomethingElse();
}}
31
32
Service A Service B Service CdoSomething() doSomethingElse()
• The presented infrastructure enables you to convert these Java calls to Asynchronous
Multi-Threaded calls just by changing a configuration file!
Example of the config-file/pools/OurThreadPool/poolSize=8
/handlers/com.hpe.ServiceB/PoolName=OurThreadPool
/handlers/com.hpe.ServiceB/WorkerCount=5
/handlers/com.hpe.ServiceC/PoolName=OurThreadPool
/handlers/com.hpe.ServiceC/WorkerCount=6
• We create a Thread Pool with 8 Threads, and define ServiceB and ServiceC to use it.
• ServiceB will use up to 5 concurrent Threads while ServiceC will use up to 6 concurrent Threads.
33
Glimpse Under the Hood
34
ServiceAdoSomething() ServiceB Proxy
ServiceB Proxy ServiceBDedicated
Queue
Worker Thread
Worker Thread
Worker Thread
ServiceBdoSomething()
Benefits• The developers can focus in coding the business logic of the Service.• The API remains exactly the same with the help of Proxies and Reflection.• Dynamically decide which services will be Async and control Thread Pools.• No need to write bug-prone Boilerplate Code for Async method invocation. This
includes BlockingQueues, Producer Threads, Consumer Threads, Executors, Thread Pools, synchronized blocks. Caveat: The code itself obviously must support multi-threaded execution, i.e. Concurrent data structures, etc.
• The Queue can be capped, and messages that exceed the cap will be dropped. Important for prevention of memory leaks.
35
What if I want only part of the API to be ASYNC?
public class MyService implements IMyService {public void methodWeWantAsync() {}
public void anotherMethodWeWantAsync() {}
@SyncMethodpublic void methodWeWantSync() {}
}
@SyncMethod - Will tell the infrastructure to run this method using the Caller’s Thread36
What about invoking a Service in a remote ODL?public class ServiceA implements IServiceA, PacketProcessingListener {
private IServiceB serviceB;
public void onPacketReceived(PacketReceived notification) {AsyncUtil.setTarget(new RemoteInstanceAddress(getIP()));serviceB.doSomething();
}private String getIP() {
if (…) {return “127.0.0.1”;
} else {return “1.2.3.4”;
}}
} 37
Glimpse Under the Hood
38
ServiceAdoSomething() ServiceB
Proxy
ServiceBProxy
RemoteAsyncService
call(ip, service, msg)
RemoteAsyncService
AKKA Sendertell(serializedMsg)
ODL 1
Glimpse Under the Hood
39
doSomething() ServiceB Proxy
ServiceB Proxy ServiceBDedicated
Queue
Worker Thread
Worker Thread
Worker Thread
ServiceBdoSomething()
AKKA Receiver
ODL 2
How Bi-Directional Invocations Work?public class ServiceA implements IServiceA, PacketProcessingListener,
IServiceBRepliesHandler {
private IServiceB serviceB;
public void onPacketReceived(PacketReceived notification) {AsyncUtil.setTarget(new RemoteInstanceAddress(getIP()));serviceB.doSomething();
}
public void replyFromServiceB(Reply reply) {// Do logic
}} 40
public class ServiceB implements IServiceB {private IServiceC serviceC;private IServiceBRepliesHandler replyHandler;
public void doSomething() {serviceC.doSomethingElse();AsyncUtil.setTarget(AsyncUtil.getCaller());replyHandler.replyFromServiceB(new Reply());
}}
41
Caveats of the Bi-Directional Invocations• By keeping the code clean from Async Java utils, we lose their potential
capabilities. If the API of ServiceB would have been:
Future doSomething();
We could use Future.get(long Timeout), or Future.isDone(), etc.
• Reply code must contain the AsyncUtil.setTarget(AsyncUtil.getCaller()), but this may be resolved in the future using Marker Annotation on the Reply Interface. Currently, this call looks like “magic” for a developer that doesn’t know the underlying infrastructure.
42
Under the Hood• When a Service registers using the BundleActivator, it must extend our new Activator:
AsyncAwareActivator. Example of Registration:
public class Activator extends AsyncAwareActivator {
protected void registerServices() {
registerNonAsyncService(SubscriberRegistrarDirect.class.getSimpleName(), SubscriberRegistrarDirect.class, //
impls(impl(ISubscriberRegistrar.class),
impl(ISystemLifecycle.class)), //
dep(IDatabaseService.class), //
dep(IGdms.class), //
dep(ISubscriberGlobalControllerCurator.class));}
}
43
Under the Hood• The infrastructure registers a Proxy to the Service in the OSGI Registry instead of the
Service itself.
• It creates a dedicated Queue<MethodCallMessage> for the Service’s Async Method Calls.
public class MethodCallMessage {
Method method;
Object[] args;
}
These fields are populated from the Proxy call of java.lang.reflect.InvocationHandler: Object invoke(Object proxy, Method method, Object[] args)
44
Under the Hood
• It also create dedicated WorkerThreads and assign them
to a ThreadPool according to the configuration file shown
previously.
• The WorkerThreads poll() the
Queue<MethodCallMessage> in a loop, and on each item
polled they invoke: method.invoke(handlerInstance, args)
45
Under the Hood - Summary
• Using the illustrated capabilities, we harvest Java’s Proxy
and Reflection power, to allow a seamless conversion of
normal method invocation to Async and potentially Remote
invocation.
46
Follow-Up Discussions
• Adaptions of the infrastructure to fit the community needs.
• Absence of capabilities in the current MD-SAL Async
solution
• Allow customization of Thread Pools. Applications that want to
listen to Notifications on a specific ThreadPool. Applications that
want its RPCs to run on a specific ThreadPool.
47
Questions?
48