Palestra apre

1. Globalcode
–
Open4education
Batch Processing:
Processamento em Lotes no
Mundo Corporativo
Rodrigo Cândido da Silva
@rcandidosilva

2. Globalcode
–
Open4education
Agenda
! Conceitos
! Batch Domain Language
! Chunk vs. Batchlet
! Partitioned Step
! Flow, Split e Decision
! Listeners e Exceptions
! Execution
! Integration

3. Globalcode
–
Open4education
Porque Batch?
! É muito comum em aplicações
! Várias soluções “personalizadas”
! Produtos começaram a surgir
! Spring Batch
! WebSphere Compute Grid
! Ideal para sistemas ETL

4. Globalcode
–
Open4education
Batch API
! Chunk / Batchlet
! Implementação de um Step
! Contexts
! Job e Step at runtime
! Persistência de metadados
! Listeners
! Callback lifecycle events
! Partitioning
! Processamento paralelo

5. Globalcode
–
Open4education
Batch Domain Language
! Batch job XML definition
! Descreve os steps como um agrupamento de
batch artifacts

6. Globalcode
–
Open4education
Batch Domain Language
<job id="adressJob” version="1.0">
<listeners>
<listener ref="MyJobListener"/>
</listeners>
<step id="buildingData" next="adressStep">
<batchlet ref="GenerateDataBatchlet" />
</step>
<step id="adressStep">
<listeners>
<listener ref="MyStepListener"/>
</listeners>
<chunk item-count="10">
<reader ref="adressItemReader" />
<processor ref="adressItemProcessor" />
<writer ref="adressItemWriter" />
</chunk>
</step>
</job>

7. Globalcode
–
Open4education
Chunk vs. Batchlet
! Implementam step dentro do job
! Chunk
! Encapsula padrão ETL
! Single Reader, Processor e Writer
! Executado por pedaços dados (chunk)
! Chunk output é escrito unitariamente
! Batchlet
! Promove a execução de um único e simples processo
! Executado até o fim produzindo um código de retorno

8. Globalcode
–
Open4education
Chunk vs. Batchlet
! Chunk ! Batchlet

9. Globalcode
–
Open4education
Batchlet
@Named
public class MyBatchlet {
@Process
public String process() throws Exception {..}
@Stop
public void stopMe() throws Exception {..}
}
//For a job
<step id=”step1”>
<batchlet ref=“MyBatchlet”/>
</step>
public class MyBatchlet implements Batchlet {..}

10. Globalcode
–
Open4education
Chunk
! Step Job
//For a job
<step id=”sendStatements”>
<chunk reader=”accountReader”
processor=”accountProcessor”
writer=”emailWriter”
item-count=”10” />
</step>
@Named(“accountReader")
...implements ItemReader... {
public Account readItem() {
// read account using JPA
@Named(“accountProcessor")
...implements ItemProcessor... {
public Statement processItems(Account account) {
// read Account, return Statement
@Named(“emailWriter")
...implements ItemWriter... {
public void writeItems(List<Statements> statements) {
// use JavaMail to send email

11. Globalcode
–
Open4education
Chunk
public interface ItemReader<T> {
public void open(Externalizable checkpoint);
public T readItem();
public Externalizable checkpointInfo();
public void close();
}
public interface ItemWriter<T> {
public void open(Externalizable checkpoint);
public void writeItems(List<T> items);
public Externalizable checkpointInfo();
public void close();
}
public interface ItemProcessor<T, R> {
public R processItem(T item);
}

12. Globalcode
–
Open4education
Checkpoint
! Para tarefas intensivas, longo período de tempo
! Checkpoint/restart é um bastante utilizado
! Basicamente…
! Armazena estado do ItemReader, ItemWriter
! Métodos chamados
! reader.checkpointInfo()!
! writer.checkpointInfo()
public interface ItemReader<T> {
public void open(Externalizable checkpoint);
public Externalizable checkpointInfo();
}
public interface ItemWriter<T> {
public void open(Externalizable checkpoint);
public Externalizable checkpointInfo();
}
<chunk checkpoint-policy="item"
commit-interval="10" item-count="10">

13. Globalcode
–
Open4education
Partitioned Step
! Step pode rodar particionado
! [N] instâncias do mesmo step em [N] Threads
! Uma partição por Thread
<step id="step1" >
<chunk ...>
<partition>
<plan partitions=“10" threads="2"/>
<reducer .../>
</partition>
</chunk>
</step>

14. Globalcode
–
Open4education
Partitioned Step
! Partition Mapper
! Decide dinamicamente o número de partições
! Partition Plan
! Partition Reducer
! Demarca a unidade lógica de trabalho
! Partition Collector
! Enviar resultados de processamento das partições
! Partition Analyzer
! Ponto de controle e análise dos resultados enviados

15. Globalcode
–
Open4education
Flow, Split e Decision
Flow
Step I
Task
Step II
Chunk
ItemReader
ItemWriter
Step III
Chunk
Deci-
sion
ItemReader
ItemWriter
Step IV
Chunk
ItemReader
ItemWriter
EndStart
ItemProces
sor
ItemProces
sor
ItemProces
sor

16. Globalcode
–
Open4education
Flow
! Define a lista de steps a ser executado (unitário)
<flow id=”flow-1" next=“{flow, step, decision}-id” >
<step id=“flow_1_step_1”>
</step>
<step id=“flow_1_step_2”>
</step>
</flow>

17. Globalcode
–
Open4education
Split
! Define a lista de flows a serem executados (paralelo)
! Coletores e analisadores para monitoramento
<split …>
<flow …./> <!– each flow runs on a separate thread -->
<flow …./>
</split>

18. Globalcode
–
Open4education
Decision
! Possibilita a implementação de workflows

19. Globalcode
–
Open4education
Decision
@Named
public class Decider {
public String decide(BatchContext context)
throws Exception {
String exit = context.getExitStatus();
if (“SUCCESS”.equals(exit)) {
return “SKIP”;
}
return exit;
}
}
//For a job
<step id=”step1”>
<decision id=“decision1” ref=“Decider”>
<next on=“SKIP” to=“step3”/>
<next on=“*” to=“step2”/>
</decision>
</step>
<step id=“step2” next=“step3”/>
<step id=“step3”/>

20. Globalcode
–
Open4education
Lifecycle
STOPPED
20
STARTING STARTED COMPLETED
FAILED
STOPPING
ABANDONED
stop()
start()
abandon()
abandon()
abandon()
restart()
restart()

21. Globalcode
–
Open4education
Listeners
! Step
! StepListener, ItemReadListener, ItemProcessListener, ItemWriterListener, ChunkListener,
RetryReadListener, RetryProcessListener, RetryWriteListener, SkipReadListener,
SkipProcessListener, SkipWriteListener
! Job
! JobListener
@Named
public class StepListener {
@BatchContext
StepContext context;
@BeforeStep
public void beforeStep() {..}
@AfterStep
public void afterStep() {..}
}
//For a job
<step id=”step1”>
<listeners>
<listener
ref=“StepListener”/>
</listeners>
</step>

22. Globalcode
–
Open4education
Exceptions
<job id=...>
...
<chunk skip-limit=”5” retry-limit=”5”>
<skippable-exception-classes>
<include class="java.lang.Exception"/>
<exclude class="java.io.FileNotFoundException"/>
</skippable-exception-classes>
<retryable-exception-classes>
...
</retryable-exception-classes>
<no-rollback-exception-classes>
...
</no-rollback-exception-classes>
</chunk>
...
</job>

23. Globalcode
–
Open4education
JobOperator e Repository
! JobOperator
! Runtime interface para gerenciamento
! start, stop, restart
! JobRepository interface commands
! JobRepository
! Contém informações sobre os jobs
! Completos e em execução

24. Globalcode
–
Open4education
Execution
! JobInstance
! Representação lógica
de um job runtime
! JobExecution
! Tentativa de executar
um JobInstance
! StepExecution
! Tentativa de rodar um
step de um job

25. Globalcode
–
Open4education
Integration
! Suporte ao Java SE
! Application Server Runtime
! Suporte clustering, segurança, gerenciamento de
recursos
! Dependency Injection com CDI
! XML descriptors
! META-INF/batch-jobs/myJob.xml
! Empacotamento
! JAR, WAR, EJB

26. Globalcode
–
Open4education
DEMO
http://goo.gl/wst7M7

27. Globalcode
–
Open4education
Perguntas
?

28. Globalcode
–
Open4education
Referências
! https://jcp.org/en/jsr/detail?id=352
! https://java.net/projects/jbatch
! http://projects.spring.io/spring-batch/
! http://docs.oracle.com/javaee/7/tutorial/doc/batch-processing.htm
! http://www.oracle.com/technetwork/articles/java/batch-1965499.html
! https://github.com/javaee-samples/javaee7-samples/
! http://blog.arungupta.me/2014/07/schedule-javaee7-batch-jobs-
techtip36/
! http://www.planetjones.co.uk/blog/25-05-2013/introducing-jsr-352-java-
batch-ee-7.html

29. Globalcode
–
Open4education
Obrigado!
@rcandidosilva
rodrigocandido.me