practical non blocking microservices in java 8
Post on 12-Apr-2017
55 Views
Preview:
TRANSCRIPT
Practicalnon-blocking microservices
in Java 8
Michał BalińskiSystem Architect
Oleksandr GoldobinSystem Architect
Cloud of microservices for secure IoT
gateway backingservice
coreservice
gateway
gatewaycore
servicebackingservice
The demand and characteristics of our domain
Crowded IoT environment
Slow, long lived connections
Big amount of concurrent connections
Scalability and resilience
Rather I/O intensive than CPU intensive
OTA Gateway – Use Case
TSM
TrustedService
Manager
OTA Gateway – Use Case
TSM
TrustedService
Manager
Security Module
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB
HTTP
1. submit scripts
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB
HTTP
1. submit scripts
HTTP2. encrypt
scripts
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB
HTTP
1. submit scripts
HTTP2. encrypt
scripts
TCP
3. store scripts
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB
HTTP
1. submit scripts
HTTP2. encrypt
scripts
TCP
4. submition response
3. store scripts
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB
HTTP
1. submit scripts
HTTP2. encrypt
scripts
TCP
4. submition response
HTTP5. poll scripts
3. store scripts
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB
HTTP
1. submit scripts
HTTP2. encrypt
scripts
TCP
3. store scripts
4. submition response
HTTP5. poll scripts
6. search scripts
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB
HTTP
1. submit scripts
HTTP2. encrypt
scripts
TCP
3. store scripts
4. submition response
HTTP5. poll scripts
6. search scripts
7. response with scripts
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB
HTTP HTTP
HTTP
TCP
OTA Gateway – Use Case
OTA(Over-the-Air)
Gateway
TSM
TrustedService
Manager
Security Module
DB LogStorage
HTTP HTTP
HTTP
TCP File I/OMonitoringSystem
HTTP
Let’s test blocking approach!
OTA Gateway Blocking - technologies
May 3, 2023 17
TSM
DB LogStorage
HTTP
TCP STDOUT
OTAGateway
JAX-RS
Logback appender
Security Module
JAX-RS
Jedis
JAX-RS Client
OTA Gateway – Blocking - test
OTA(Over-the-Air)
Gateway
send 2 scripts per
request
Security Module
DB LogStorage
HTTP HTTP
HTTP
TCP File I/O
emulated latency200 ms
expected latency
< 450 ms
verified with throughput
over 7k req/s
Blocking – 1k threads
OTA(Over-the-Air)
Gateway
send 2 scripts per
request
Security Module
DB LogStorage
HTTP HTTP
HTTP
TCP File I/O
emulated latency200 ms
max 1000connections
max 1000 threads
expected latency
< 450 ms
Blocking – 1k threads500 req/s 1000 req/s 1500 req/s 2000 req/s
The drawbacks of classic synchronous I/O
One thread per connection
Threads waiting instead of running
Context switches
Resource wasting (~1 MB per thread - 64bit)
Let’s switch from blocking to non-blocking
OTA Gateway Non-blocking - technologies
May 3, 2023 23
TSM
DB LogStorage
HTTP
TCP STDOUT
OTAGateway
JAX-RS 2.0
Logback asyncappender
Async Http Client 2 (Netty)
Security Module
JAX-RS 2.0
Lettuce(Netty)
Non-blocking – 16 threads
OTA(Over-the-Air)
Gateway
send 2 scripts per
request
Security Module
DB LogStorage
HTTP HTTP
HTTP
TCP File I/O
emulated latency200 ms
no limit for connections
max 16 threads
expected latency
< 450 ms
Non-blocking – 16 threads1k req/s 2k req/s 2.5k req/s 3k req/s
Blocking Non-blocking
1000 threads 56 threads
1.2 GB 0.5 GB
~1.2k r/s 2.5k r/s
Let’s talk about challenges
OTA Gateway Blocking – sequence diagram
OTAGatewayTSM Security
Module DB Logs
loop
1. submit scripts
2. encrypt script
3a. store script
4. submition response
3b. count scripts
OTA Gateway Non-blocking – sequence diagram
OTAGatewayTSM Security
Module DB Logs
loop
1. submit scripts
2. encrypt script
3a. store script
4. submition response
3b. count scripts
OTA Non-blocking – realityOTAGateway
TSM Security Module DB Logs
„loop
ed p
roce
ssin
g ch
ain”
1. submit scripts
4. submition response
3b. count scripts
HTT
PSe
rver
2. encrypt script
3a. store script
Logg
ing
DB
Clie
nt
Secu
rity
Cl
ient
Code. Bird view
31May 3, 2023
package org.demo.ota.blocking.rest;
@Path("se")public class ScriptSubmissionResource extends Application {
private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission");
private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance();
@POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) { MDC.put("flow", "submission"); MDC.put("se", seId);
return METRICS.instrument(() -> { log.debug("Processing {} scripts submission", scripts.size(), seId);
for (int i = 0; i < scripts.size(); i++) { final Script script = scripts.get(i);
log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient.encrypt(seId, script.getPayload()); script.setPayload(encryptedPayload);
log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); }
long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); log.debug("Request processed", seId); return numberOfScripts; }); }
@Override public Set<Object> getSingletons() { return Collections.singleton(this); }}
package org.demo.ota.nonblocking.rest;
@Path("se")public class ScriptSubmissionResource extends Application {
private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission");
private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance();
@POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId);
METRICS.instrumentStage(() -> { log.debug("{} Processing {} scripts submission", diagnosticContext, scripts.size());
return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) ); }) .whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); } }); }
private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts ) { CompletionStage<Void> stage = null; // <- non final field, potential concurrent access bug!
for (int i = 0; i < scripts.size(); i++) { final int scriptIndex = i; final Script script = scripts.get(scriptIndex);
if (stage == null) { stage = encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script)); } }
return stage; }
private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script ) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex);
return secureModuleClient .encrypt(seId, script.getPayload()) .thenCompose( encryptedPayload -> { log.debug("{} Storing encrypted script {}", diagnosticContext, scriptIndex); return scriptStorageClient.storeScript(seId, new Script(encryptedPayload)); } ); }
@Override public Set<Object> getSingletons() { return new HashSet<>(Collections.singletonList(this)); }}
Code. Blocking. Submission 1
May 3, 2023 32
@POST@Path("/{seId}/scripts")@Consumes(MediaType.APPLICATION_JSON)@Produces(MediaType.TEXT_PLAIN)public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) {
MDC.put("flow", "submission"); // Setting diagnostic context MDC.put("se", seId);
return METRICS.instrument(() -> { // Instrumenting with metrics //...
Code. Blocking. Submission 2
May 3, 2023 33
log.debug("Processing {} scripts submission", scripts.size(), seId);
for (int i = 0; i < scripts.size(); i++) { Cycle through the scripts
final Script script = scripts.get(i);
log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient .encrypt(seId, script.getPayload()); Encrypting the script
script.setPayload(encryptedPayload);
log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); Saving the script into
DB}
long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); Getting current number
of scripts in DB
log.debug("Request processed", seId);return numberOfScripts;
Code. Non-blocking. Submission 1
May 3, 2023 34
@POST@Path("/{seId}/scripts")@Consumes(MediaType.APPLICATION_JSON)@Produces(MediaType.TEXT_PLAIN)public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); Creating diagnostic
context
METRICS.instrumentStage(() -> { Instrumenting with metrics
Code. Non-blocking. Submission 1
May 3, 2023 35
@POST@Path("/{seId}/scripts")@Consumes(MediaType.APPLICATION_JSON)@Produces(MediaType.TEXT_PLAIN)public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); Creating diagnostic
context
METRICS.instrumentStage(() -> { Instrumenting with metrics
Code. Non-blocking. Submission 2
May 3, 2023 36
METRICS.instrumentStage(() -> { log.debug(
"{} Processing {} scripts submission", diagnosticContext, scripts.size());
return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) );}).whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); }});
Code. Non-blocking. Submission 3
May 3, 2023 37
private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts) { CompletionStage<Void> stage = null; // <- non final field, potential // concurrent access bug!
for (int i = 0; i < scripts.size(); i++) { Cycle through the scripts
final int scriptIndex = i; final Script script = scripts.get(scriptIndex);
if (stage == null) { stage = encryptAndStoreSingleScript(
diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript(
diagnosticContext, seId, scriptIndex, script)); } } return stage;}
Code. Non-blocking. Submission 4
May 3, 2023 38
private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex);
return secureModuleClient .encrypt(seId, script.getPayload()) Encrypting the script .thenCompose( encryptedPayload -> { log.debug(
"{} Storing encrypted script {}", diagnosticContext, scriptIndex);
return scriptStorageClient.storeScript( Saving the script into seId,
the DB new Script(encryptedPayload));
} );}
Code. Blocking. Integration
May 3, 2023 39
private final JedisPool pool;
public void storeScript(String seId, Script script) { try (Jedis jedis = pool.getResource()) { jedis.rpush(seId, script.getPayload()); }}
public long numberOfScriptsForSe(String seId) { try (Jedis jedis = pool.getResource()) { return jedis.llen(seId); }}
public Optional<Script> nextScript(String seId) { try (Jedis jedis = pool.getResource()) { return Optional.ofNullable(jedis.lpop(seId)).map(Script::new); }}
public String encrypt(String keyDiversifier, String payload) {// ...}
Code. Non-Blocking. Integration
May 3, 2023 40
private final RedisAsyncCommands<String, String> commands;
public CompletionStage<Void> storeScript(String seId, Script script) { return commands .rpush(seId, script.getPayload()) .thenApply(ignore -> null);}
public CompletionStage<Long> numberOfScriptsForSe(String seId) { return commands.llen(seId);}
public CompletionStage<Optional<String>> nextScript(String seId) { return commands.lpop(seId).thenApply(Optional::ofNullable);}
public CompletionStage<String> encrypt(String keyDiversifier, String payload) {// ...}
Diagnostics in non-blocking systems
No clear stack traces, need for good logs
Name your threads properly
MDC becomes useless (thread locals)
Explicitly pass debug context to trace flows
Be prepared for debuging non-obvious errors
NIO technology landscapeJDK 1.4
NIOJDK 1.7NIO.2
JAX-RS 2.x
Servlet API 3.x
Lessons learned, part 1
Vanila Java 8 for NIO µ-services
Netty best for custom protocols in NIO
Unit tests should be synchronous
Load/stress testing is a must
Make bulkheading and plan your resources
Lessons learned, part 2
Functional programming patterns for readability
Immutability as 1-st class citizen
Scala may be good choice ;-)
Conclusion
Non-blocking processing can really save your resources (== money)
But!
Weight all pros and cons and use non-blocking processing only if you really need it.
Thank you.
Michał Balińskim.balinski@oberthur.com
Oleksandr Goldobino.goldobin@oberthur.com
goldobin
@goldobin
balonus
@MichalBalinski
Readings:• https://github.com/balonus/blocking-vs-nonblocking-demo
• C10k Problem, C10M Problem, Asynchronous I/O
• Boost application performance using asynchronous I/O (M. Tim Jones, 2006)
• Zuul 2 : The Netflix Journey to Asynchronous, Non-Blocking Systems (Netflix, 2016)
• Thousands of Threads and Blocking I/O: The Old Way to Write Java Servers Is New Again (and Way Better) (Paul Tyma, 2008)
• Why Non-Blocking? (Bozhidar Bozhanov, 2011)
top related