batch processing - processamento em lotes no mundo corporativo

Globalcode – Open4education

Batch Processing: Processamento em Lotes no

Mundo Corporativo Rodrigo Cândido da Silva

@rcandidosilva


Agenda

!  Conceitos !  Batch Domain Language !  Chunk vs. Batchlet !  Partitioned Step !  Flow, Split e Decision !  Listeners e Exceptions !  Execution !   Integration


Porque Batch?

!  É muito comum em aplicações !  Várias soluções “personalizadas” !  Produtos começaram a surgir

!  Spring Batch !  WebSphere Compute Grid

!   Ideal para sistemas ETL


Batch API

!  Chunk / Batchlet ! Implementação de um Step

!  Contexts !  Job e Step at runtime ! Persistência de metadados

!  Listeners !  Callback lifecycle events

!  Partitioning ! Processamento paralelo


Batch Domain Language

!  Batch job XML definition !  Descreve os steps como um agrupamento de

batch artifacts


Batch Domain Language <job id="adressJob” version="1.0"> <listeners> <listener ref="MyJobListener"/> </listeners> <step id="buildingData" next="adressStep"> <batchlet ref="GenerateDataBatchlet" /> </step> <step id="adressStep"> <listeners> <listener ref="MyStepListener"/> </listeners> <chunk item-count="10"> <reader ref="adressItemReader" /> <processor ref="adressItemProcessor" /> <writer ref="adressItemWriter" /> </chunk> </step> </job>


Chunk vs. Batchlet

!   Implementam step dentro do job !  Chunk

!  Encapsula padrão ETL !  Single Reader, Processor e Writer !  Executado por pedaços dados (chunk) !  Chunk output é escrito unitariamente

!  Batchlet !  Promove a execução de um único e simples processo !  Executado até o fim produzindo um código de retorno


Chunk vs. Batchlet

!  Chunk ! Batchlet


Batchlet

@Named public class MyBatchlet { @Process public String process() throws Exception {..} @Stop public void stopMe() throws Exception {..} }

//For a job <step id=”step1”> <batchlet ref=“MyBatchlet”/> </step>

public class MyBatchlet implements Batchlet {..}


Chunk

!  Step Job //For a job <step id=”sendStatements”> <chunk reader=”accountReader” processor=”accountProcessor” writer=”emailWriter” item-count=”10” /> </step>

@Named(“accountReader") ...implements ItemReader... { public Account readItem() { // read account using JPA

@Named(“accountProcessor") ...implements ItemProcessor... { public Statement processItems(Account account) { // read Account, return Statement

@Named(“emailWriter") ...implements ItemWriter... { public void writeItems(List<Statements> statements) { // use JavaMail to send email


Chunk

public interface ItemReader<T> { public void open(Externalizable checkpoint); public T readItem(); public Externalizable checkpointInfo(); public void close(); }

public interface ItemWriter<T> { public void open(Externalizable checkpoint); public void writeItems(List<T> items); public Externalizable checkpointInfo(); public void close(); }

public interface ItemProcessor<T, R> { public R processItem(T item); }


Checkpoint

!  Para tarefas intensivas, longo período de tempo !  Checkpoint/restart é um bastante utilizado

!  Basicamente… !  Armazena estado do ItemReader, ItemWriter !   Métodos chamados

!   reader.checkpointInfo()!!   writer.checkpointInfo()

public interface ItemReader<T> { public void open(Externalizable checkpoint); public Externalizable checkpointInfo(); }

public interface ItemWriter<T> { public void open(Externalizable checkpoint); public Externalizable checkpointInfo(); }

<chunk checkpoint-policy="item" commit-interval="10" item-count="10">


Partitioned Step

!  Step pode rodar particionado !   [N] instâncias do mesmo step em [N] Threads

!  Uma partição por Thread

<step id="step1" > <chunk ...> <partition> <plan partitions=“10" threads="2"/> <reducer .../> </partition> </chunk> </step>


Partitioned Step

!  Partition Mapper !  Decide dinamicamente o número de partições

!  Partition Plan

!  Partition Reducer !  Demarca a unidade lógica de trabalho

!  Partition Collector !  Enviar resultados de processamento das partições

!  Partition Analyzer !  Ponto de controle e análise dos resultados enviados


Flow, Split e Decision

Flow Step I

Task

Step II

Chunk

ItemReader

ItemWriter

15

Step III

Chunk

Deci-sion

ItemReader

ItemWriter

Step IV

Chunk

ItemReader

ItemWriter

End Start

ItemProcessor

ItemProcessor

ItemProcessor


Flow

!  Define a lista de steps a ser executado (unitário)

<flow id=”flow-1" next=“{flow, step, decision}-id” > <step id=“flow_1_step_1”> </step> <step id=“flow_1_step_2”> </step> </flow>


Split

!   Define a lista de flows a serem executados (paralelo) !   Coletores e analisadores para monitoramento

<split …> <flow …./> <!– each flow runs on a separate thread --> <flow …./> </split>


Decision

!  Possibilita a implementação de workflows


Decision @Named public class Decider { public String decide(BatchContext context) throws Exception { String exit = context.getExitStatus(); if (“SUCCESS”.equals(exit)) { return “SKIP”; } return exit; } }

//For a job <step id=”step1”> <decision id=“decision1” ref=“Decider”> <next on=“SKIP” to=“step3”/> <next on=“*” to=“step2”/> </decision> </step> <step id=“step2” next=“step3”/> <step id=“step3”/>


Lifecycle

STOPPED

20

STARTING STARTED COMPLETED

FAILED

STOPPING

ABANDONED

stop()

start()

abandon()

abandon()

abandon()

restart()

restart()


Listeners

!  Step !   StepListener, ItemReadListener, ItemProcessListener, ItemWriterListener, ChunkListener,

RetryReadListener, RetryProcessListener, RetryWriteListener, SkipReadListener, SkipProcessListener, SkipWriteListener

!   Job !   JobListener

@Named public class StepListener { @BatchContext StepContext context; @BeforeStep public void beforeStep() {..} @AfterStep public void afterStep() {..} }

//For a job <step id=”step1”> <listeners> <listener ref=“StepListener”/> </listeners> </step>


Exceptions

<job id=...> ... <chunk skip-limit=”5” retry-limit=”5”> <skippable-exception-classes> <include class="java.lang.Exception"/> <exclude class="java.io.FileNotFoundException"/> </skippable-exception-classes> <retryable-exception-classes> ... </retryable-exception-classes> <no-rollback-exception-classes> ... </no-rollback-exception-classes> </chunk> ... </job>


JobOperator e Repository

!   JobOperator !  Runtime interface para gerenciamento

!  start, stop, restart !  JobRepository interface commands

!   JobRepository !  Contém informações sobre os jobs

!  Completos e em execução


Execution

!   JobInstance !  Representação lógica

de um job runtime !   JobExecution

!  Tentativa de executar um JobInstance

!  StepExecution !  Tentativa de rodar um

step de um job


Integration

!  Suporte ao Java SE !  Application Server Runtime

!  Suporte clustering, segurança, gerenciamento de recursos

!  Dependency Injection com CDI !  XML descriptors

!  META-INF/batch-jobs/myJob.xml

!  Empacotamento !  JAR, WAR, EJB


DEMO http://goo.gl/wst7M7


Perguntas

?


Referências ! https://jcp.org/en/jsr/detail?id=352 ! https://java.net/projects/jbatch ! http://projects.spring.io/spring-batch/ ! http://docs.oracle.com/javaee/7/tutorial/doc/batch-processing.htm ! http://www.oracle.com/technetwork/articles/java/batch-1965499.html ! https://github.com/javaee-samples/javaee7-samples/ ! http://blog.arungupta.me/2014/07/schedule-javaee7-batch-jobs-

techtip36/ ! http://www.planetjones.co.uk/blog/25-05-2013/introducing-jsr-352-java-

batch-ee-7.html


Obrigado! @rcandidosilva

rodrigocandido.me

batch processing - processamento em lotes no mundo corporativo

Software

public interface itemwriter

public void stopme

globalcodeopen4education

globalcodeopen4education

public interface itemprocessor

public account readitem

public class steplistener

step job