A lot of projects sooner or later encounter the challenge of processing huge data sets. You might want to choose a system where you receive the requests, but process them later on to avoid a system overload, eg. during the night. During this workshop we will teach you the principles of batch processing. We will try to explain the matter in a framework independant way, but we will concretize the theory with code samples using the Spring Batch Framework.

1. Batch Processing
ORA/J

2. Agenda
▪
▪
▪
▪
▪

3. Batch principles

4. Characteristics
▪
▪
▪
▪

5. Batch processing ≠ Scheduling
Batch processing Scheduling
Process large volumes of data Scheduling tasks
eg. Spring Batch, JEE eg. Quartz, Obsidian

6. Things to consider
▪
▪
▪
▪
▪
▪

7. Essential building blocks

8. Categorize based on responsibility
▪
▪
▪
▪
▪
▪

9. Categorize based on input source
▪
▪
▪

10. Guidelines
▪
▪
▪
▪
▪

11. Transaction Management

12. ACID principle
▪
▪
▪
▪

13. Transactions are handled at the step level
▪
▪

14. Transaction management in steps (tasklets)
Step 1
Step 2
Step 3
Transaction
Transaction
Transaction x nn

15. Example: chunk-oriented step
<step id="importProductsStep">
<tasklet>
<chunk reader="reader" writer="writer" commit-interval="100" />
<transaction-attributes isolation="READ_UNCOMMITTED" />
</tasklet>
</step>
Transaction

16. Transaction management in listeners
▪
▪
▪
▪ ChunkListener
beforeChunk afterChunk

17. Transactional reader
<step id="importProductsStep">
<tasklet>
<chunk reader="reader" writer="writer" commit-interval="100"
reader-transactional-queue="true"/>
<transaction-attributes isolation="READ_UNCOMMITTED" />
</tasklet>
</step>

18. Transaction management patterns
▪

19. Transaction management patterns
▪

20. Transaction management patterns
▪ …
▪
▪

21. Global transactions implementation

22. Drawbacks to global transactions
▪
▪
▪
▪
▪
▪
▪

23. Shared resource transaction pattern
▪

24. Best effort pattern with JMS

25. What can go wrong with JMS?
▪
▪

26. Best effort pattern with JMS
commit
start
commit
start
Chunk local
transaction
JMS local
transaction
timeline
Commits are synchronized
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="connectionFactory" />
<property name="defaultDestination" ref="orderQueue" />
<property name="receiveTimeout" value="100" />
<property name="sessionTransacted" value="true" />
</bean>

27. JMS failure will result in a duplicate message
commit
start
commit
start
Chunk local
transaction
JMS local
transaction
timeline
Commits are synchronized
failed

28. Handling duplicate messages with manual detection
▪
▪

29. Handling duplicate messages with manual detection

30. Handling duplicate messages with idempotency

31. Conclusions
▪
▪
▪
▪

32. Execution flows

33. Execution flows
▪
▪

34. Standard flow

35. Conditional flow - example

36. Conditional flow

37. Conditional flow
▪
▪
▪

38. ExitStatus vs BatchStatus
▪
…
▪
▪
▪

39. Custom Exitstatus

40. Custom Batchstatus
▪
▪
▪
▪
▪
▪

41. Custom Batchstatus

42. Parallel execution
▪
▪

43. Parallel execution - types
▪
▪
▪

44. Parallel execution - Multithreaded
▪
▪
▪

45. Parallel execution - Remote chunking
▪
▪
▪
▪
▪
▪

46. Parallel execution - Step partitioning
▪
▪
▪
▪

47. Parallel execution - Step partitioning

48. Parallel execution - Step partitioning example

49. Demo: Speeder problem

50. Introduction
▪
▪
▪

51. Introduction
▪
▪

52. Introduction
▪
▪
▪
▪
▪
▪

53. Generating input
▪
▪
▪

54. Reading the input

55. Reading the input
▪
▪
▪
▪
▪

56. Processing the data

57. Processing the lines we just read
▪
▪ KmPerHourCalculator
▪ null

58. Writing the data

59. Writing the Fines
▪
▪ BeanWrapperFieldExtractor
DelimitedLineAggregator

60. Adding it all together
▪
▪ BatchConfiguration
▪
▪
▪
▪ Step
▪
▪

61. Listening in
●
●
●
○
○
○
●

62. Gathering statistics
▪
▪
▪ StepExecutionContext
JobExecutionContext

63. Gathering Statistics

64. Spring Batch and JEE

65. JSR-352
▪
▪
▪
▪

66. JSR-352 Enterprise
▪
▪

67. JSR-352 spring
▪
▪
▪

68. org.springframework.batch.core.
SkipListener#onSkipInWrite(S item, Throwable t)
javax.batch.api.chunk.listener.
SkipWriteListener#onSkipWriteItem(List<Object> items,
Exception ex)
JSR-352 Spring vs. JEE

69. Questions?