Globalcode – Open4education
Batch Processing: Processamento em Lotes no
Mundo Corporativo Rodrigo Cândido da Silva
@rcandidosilva
Globalcode – Open4education
Agenda
! Conceitos ! Batch Domain Language ! Chunk vs. Batchlet ! Partitioned Step ! Flow, Split e Decision ! Listeners e Exceptions ! Execution ! Integration
Globalcode – Open4education
Porque Batch?
! É muito comum em aplicações ! Várias soluções “personalizadas” ! Produtos começaram a surgir
! Spring Batch ! WebSphere Compute Grid
! Ideal para sistemas ETL
Globalcode – Open4education
Batch API
! Chunk / Batchlet ! Implementação de um Step
! Contexts ! Job e Step at runtime ! Persistência de metadados
! Listeners ! Callback lifecycle events
! Partitioning ! Processamento paralelo
Globalcode – Open4education
Batch Domain Language
! Batch job XML definition ! Descreve os steps como um agrupamento de
batch artifacts
Globalcode – Open4education
Batch Domain Language <job id="adressJob” version="1.0"> <listeners> <listener ref="MyJobListener"/> </listeners> <step id="buildingData" next="adressStep"> <batchlet ref="GenerateDataBatchlet" /> </step> <step id="adressStep"> <listeners> <listener ref="MyStepListener"/> </listeners> <chunk item-count="10"> <reader ref="adressItemReader" /> <processor ref="adressItemProcessor" /> <writer ref="adressItemWriter" /> </chunk> </step> </job>
Globalcode – Open4education
Chunk vs. Batchlet
! Implementam step dentro do job ! Chunk
! Encapsula padrão ETL ! Single Reader, Processor e Writer ! Executado por pedaços dados (chunk) ! Chunk output é escrito unitariamente
! Batchlet ! Promove a execução de um único e simples processo ! Executado até o fim produzindo um código de retorno
Globalcode – Open4education
Chunk vs. Batchlet
! Chunk ! Batchlet
Globalcode – Open4education
Batchlet
@Named public class MyBatchlet { @Process public String process() throws Exception {..} @Stop public void stopMe() throws Exception {..} }
//For a job <step id=”step1”> <batchlet ref=“MyBatchlet”/> </step>
public class MyBatchlet implements Batchlet {..}
Globalcode – Open4education
Chunk
! Step Job //For a job <step id=”sendStatements”> <chunk reader=”accountReader” processor=”accountProcessor” writer=”emailWriter” item-count=”10” /> </step>
@Named(“accountReader") ...implements ItemReader... { public Account readItem() { // read account using JPA
@Named(“accountProcessor") ...implements ItemProcessor... { public Statement processItems(Account account) { // read Account, return Statement
@Named(“emailWriter") ...implements ItemWriter... { public void writeItems(List<Statements> statements) { // use JavaMail to send email
Globalcode – Open4education
Chunk
public interface ItemReader<T> { public void open(Externalizable checkpoint); public T readItem(); public Externalizable checkpointInfo(); public void close(); }
public interface ItemWriter<T> { public void open(Externalizable checkpoint); public void writeItems(List<T> items); public Externalizable checkpointInfo(); public void close(); }
public interface ItemProcessor<T, R> { public R processItem(T item); }
Globalcode – Open4education
Checkpoint
! Para tarefas intensivas, longo período de tempo ! Checkpoint/restart é um bastante utilizado
! Basicamente… ! Armazena estado do ItemReader, ItemWriter ! Métodos chamados
! reader.checkpointInfo()!! writer.checkpointInfo()
public interface ItemReader<T> { public void open(Externalizable checkpoint); public Externalizable checkpointInfo(); }
public interface ItemWriter<T> { public void open(Externalizable checkpoint); public Externalizable checkpointInfo(); }
<chunk checkpoint-policy="item" commit-interval="10" item-count="10">
Globalcode – Open4education
Partitioned Step
! Step pode rodar particionado ! [N] instâncias do mesmo step em [N] Threads
! Uma partição por Thread
<step id="step1" > <chunk ...> <partition> <plan partitions=“10" threads="2"/> <reducer .../> </partition> </chunk> </step>
Globalcode – Open4education
Partitioned Step
! Partition Mapper ! Decide dinamicamente o número de partições
! Partition Plan
! Partition Reducer ! Demarca a unidade lógica de trabalho
! Partition Collector ! Enviar resultados de processamento das partições
! Partition Analyzer ! Ponto de controle e análise dos resultados enviados
Globalcode – Open4education
Flow, Split e Decision
Flow Step I
Task
Step II
Chunk
ItemReader
ItemWriter
15
Step III
Chunk
Deci-sion
ItemReader
ItemWriter
Step IV
Chunk
ItemReader
ItemWriter
End Start
ItemProcessor
ItemProcessor
ItemProcessor
Globalcode – Open4education
Flow
! Define a lista de steps a ser executado (unitário)
<flow id=”flow-1" next=“{flow, step, decision}-id” > <step id=“flow_1_step_1”> </step> <step id=“flow_1_step_2”> </step> </flow>
Globalcode – Open4education
Split
! Define a lista de flows a serem executados (paralelo) ! Coletores e analisadores para monitoramento
<split …> <flow …./> <!– each flow runs on a separate thread --> <flow …./> </split>
Globalcode – Open4education
Decision
! Possibilita a implementação de workflows
Globalcode – Open4education
Decision @Named public class Decider { public String decide(BatchContext context) throws Exception { String exit = context.getExitStatus(); if (“SUCCESS”.equals(exit)) { return “SKIP”; } return exit; } }
//For a job <step id=”step1”> <decision id=“decision1” ref=“Decider”> <next on=“SKIP” to=“step3”/> <next on=“*” to=“step2”/> </decision> </step> <step id=“step2” next=“step3”/> <step id=“step3”/>
Globalcode – Open4education
Lifecycle
STOPPED
20
STARTING STARTED COMPLETED
FAILED
STOPPING
ABANDONED
stop()
start()
abandon()
abandon()
abandon()
restart()
restart()
Globalcode – Open4education
Listeners
! Step ! StepListener, ItemReadListener, ItemProcessListener, ItemWriterListener, ChunkListener,
RetryReadListener, RetryProcessListener, RetryWriteListener, SkipReadListener, SkipProcessListener, SkipWriteListener
! Job ! JobListener
@Named public class StepListener { @BatchContext StepContext context; @BeforeStep public void beforeStep() {..} @AfterStep public void afterStep() {..} }
//For a job <step id=”step1”> <listeners> <listener ref=“StepListener”/> </listeners> </step>
Globalcode – Open4education
Exceptions
<job id=...> ... <chunk skip-limit=”5” retry-limit=”5”> <skippable-exception-classes> <include class="java.lang.Exception"/> <exclude class="java.io.FileNotFoundException"/> </skippable-exception-classes> <retryable-exception-classes> ... </retryable-exception-classes> <no-rollback-exception-classes> ... </no-rollback-exception-classes> </chunk> ... </job>
Globalcode – Open4education
JobOperator e Repository
! JobOperator ! Runtime interface para gerenciamento
! start, stop, restart ! JobRepository interface commands
! JobRepository ! Contém informações sobre os jobs
! Completos e em execução
Globalcode – Open4education
Execution
! JobInstance ! Representação lógica
de um job runtime ! JobExecution
! Tentativa de executar um JobInstance
! StepExecution ! Tentativa de rodar um
step de um job
Globalcode – Open4education
Integration
! Suporte ao Java SE ! Application Server Runtime
! Suporte clustering, segurança, gerenciamento de recursos
! Dependency Injection com CDI ! XML descriptors
! META-INF/batch-jobs/myJob.xml
! Empacotamento ! JAR, WAR, EJB
Globalcode – Open4education
DEMO http://goo.gl/wst7M7
Globalcode – Open4education
Perguntas
?
Globalcode – Open4education
Referências ! https://jcp.org/en/jsr/detail?id=352 ! https://java.net/projects/jbatch ! http://projects.spring.io/spring-batch/ ! http://docs.oracle.com/javaee/7/tutorial/doc/batch-processing.htm ! http://www.oracle.com/technetwork/articles/java/batch-1965499.html ! https://github.com/javaee-samples/javaee7-samples/ ! http://blog.arungupta.me/2014/07/schedule-javaee7-batch-jobs-
techtip36/ ! http://www.planetjones.co.uk/blog/25-05-2013/introducing-jsr-352-java-
batch-ee-7.html
Globalcode – Open4education
Obrigado! @rcandidosilva
rodrigocandido.me